Add Agent Doctor fleet diagnostics workflow

This commit is contained in:
rcourtman
2026-07-20 19:53:34 +01:00
parent cee4f64614
commit e23f19459d
60 changed files with 2734 additions and 279 deletions
+6
View File
@@ -112,6 +112,12 @@ rm -f install.sh install.sh.sshsig
Note: this installs the Pulse **server**. Agent installs and v5-to-v6 agent upgrades use the command generated in **Settings → Infrastructure → Install on a host** (served from `/install.sh` on your Pulse server).
Server and agent updates are separate lifecycle paths. Updating the server sets
the version eligible v6 agents should reach, but it does not prove the fleet has
converged: those agents update asynchronously, while v5 agents, PVE host agents,
and agents with auto-update disabled or failed update prerequisites need the
manual per-host command from **Settings → Infrastructure**.
### Option 2: Docker
```bash
docker run -d \
+14 -2
View File
@@ -1,6 +1,14 @@
# Automatic Updates
# Pulse Server Automatic Updates
Pulse supports one-click updates for supported deployment types, making it easy to keep your monitoring system up to date.
Pulse supports one-click server updates for supported deployment types. This
document describes the Pulse server runtime, not installed Pulse Agents.
Eligible v6 agents update asynchronously through their own update client. A
server update changes their target version but does not prove fleet convergence.
For v5, PVE, disabled, or failed agent updates, use **Agent Doctor** at
`/settings/infrastructure?agentDoctor=1` or the installer in
**Settings → Infrastructure → Install on a host**. See
[Unified Agent](UNIFIED_AGENT.md#auto-update).
## Supported Deployment Types
@@ -81,6 +89,10 @@ docker pull rcourtman/pulse:vX.Y.Z
docker compose down && docker compose up -d
```
This command uses the public Community image. Private Pro runtime installs must
use the private image and credentials supplied by the private download/update
path; replacing a Pro image with `rcourtman/pulse` changes the runtime edition.
If you use the legacy `docker-compose` binary, replace `docker compose` with `docker-compose`.
### ProxmoxVE LXC (Manual)
+13
View File
@@ -24,6 +24,19 @@ If you want Pulse to find servers automatically, enable discovery in **Settings
- **Systemd**: `sudo systemctl edit pulse`, add `Environment="FRONTEND_PORT=8080"`, restart.
- **Docker**: Use `-p 8080:7655` in your run command.
### Does updating the Pulse server update every agent immediately?
No. The server and agent have separate update lifecycles. Eligible v6 agents
check for and apply the server's target version asynchronously. v5 agents, PVE
host agents, agents with auto-update disabled, and agents with failed or missing
update prerequisites need a manual command.
Open an outdated-agent notice or
`/settings/infrastructure?agentDoctor=1` to open **Agent Doctor** and
copy the correct per-host command. The surface does not remotely execute the
update. Use **Settings → Infrastructure → Install on a host** for first installs
and v5-to-v6 upgrades. See [Unified Agent](UNIFIED_AGENT.md#auto-update).
### Why can't I change settings in the UI?
If a setting is disabled with an amber warning, it's being overridden by an environment variable (e.g., `DISCOVERY_ENABLED`). Remove the env var to regain UI control.
+30 -6
View File
@@ -3,7 +3,7 @@
Pulse offers flexible installation options from Docker to enterprise-ready Kubernetes charts.
> **Paid Pulse Pro / Relay / legacy customers:** GitHub release assets and the
> public `rcourtman/pulse` Docker image are community builds. They can accept an
> public `rcourtman/pulse` Docker image are Community builds. They can accept an
> activation key, but they do not include the private Pulse Pro runtime hooks.
> Use <https://pulserelay.pro/download.html> with your activation key to get the
> private Pulse Pro Docker image or Linux archive. For Docker Compose, use the
@@ -45,7 +45,7 @@ bash install.sh --version "${PULSE_VERSION}"
rm -f install.sh install.sh.sshsig
```
> **Note**: The GitHub `install.sh` is the **server** installer. The agent installer is served from your Pulse server at `/install.sh` (see **Settings → Infrastructure → Install on a host**).
> **Note**: The GitHub `install.sh` is the **server** installer. The agent installer is served from your Pulse server at `/install.sh` (see **Settings → Infrastructure → Install on a host**). Do not use the GitHub server installer to install or update `pulse-agent`.
### Docker
Ideal for containerized environments or testing.
@@ -120,7 +120,7 @@ sudo bash install.sh --version "${PULSE_VERSION}"
rm -f install.sh install.sh.sshsig
```
> **Note**: This installs the Pulse server. Use the `/install.sh` endpoint from **Settings → Infrastructure → Install on a host** for installing `pulse-agent` on monitored hosts.
> **Note**: This installs the Pulse server. Use the `/install.sh` endpoint from **Settings → Infrastructure → Install on a host** for installing or upgrading `pulse-agent` on monitored hosts.
<details>
<summary><strong>Manual systemd install (advanced)</strong></summary>
@@ -197,12 +197,16 @@ Pulse is secure by default. On first launch, you must retrieve a **Bootstrap Tok
## 🔄 Updates
### Automatic Updates (Systemd/LXC only)
Pulse can self-update to the latest stable version.
The Pulse server and installed Pulse Agents have independent update paths.
### Pulse server updates
#### Automatic Updates (Systemd/LXC only)
Pulse can update the server runtime to the latest stable version.
**Enable via UI**: Settings → System → Updates
### Manual Update
#### Manual Update
| Platform | Command |
|----------|---------|
@@ -212,6 +216,26 @@ Pulse can self-update to the latest stable version.
Docker without Compose: `docker restart` keeps the old image running. Run `docker pull rcourtman/pulse:vX.Y.Z`, then `docker stop pulse && docker rm pulse` and re-run your original `docker run` command.
The public image and commands above install the Community runtime. If the
instance uses the private Pro runtime, keep it on the private image or archive
shown by <https://pulserelay.pro/download.html>; replacing it with a public
GitHub asset or `rcourtman/pulse` image removes the private runtime hooks.
### Pulse Agent updates
Eligible v6 agents check the Pulse server for updates and apply them
asynchronously. A current server version therefore does not prove every agent is
current. v5 agents, PVE host agents, agents with auto-update disabled, and agents
whose authentication, connection state, download, trust, or self-test checks
fail require manual handling.
Open an outdated-agent notice, or use **Agent Doctor** at
`/settings/infrastructure?agentDoctor=1`, to review the agents Pulse currently
sees and copy the platform-specific command for each host. This surface provides
commands for the operator to run on the host; it does not remotely execute the
update. Use **Settings → Infrastructure → Install on a host** for a first install
or a v5-to-v6 in-place upgrade.
### Rollback
If an update causes issues on systemd installations, backups are created automatically during the update process.
+30
View File
@@ -70,6 +70,33 @@ If you only missed the token during a fresh install (no password set yet), skip
### Monitoring Data
#### Agent fleet update or identity issue
- Open an outdated-agent notice or
`/settings/infrastructure?agentDoctor=1` to open **Agent Doctor** and
copy the platform-specific command for each reported host. This is a manual
handoff; Pulse does not remotely execute the command.
- Administrators can call the read-only Agent Fleet Doctor endpoint,
`GET /api/agents/diagnostics`, to inspect liveness, version drift, profile
deployment drift, expected telemetry gaps, and identity-split evidence. It
does not change agent configuration or enqueue a repair.
- A current Pulse server does not prove fleet convergence. Eligible v6 agents
update asynchronously; v5, PVE, disabled, and failed updates require manual
handling.
#### Removed Pulse server but `pulse-agent` still logs connection failures
Removing the Pulse server does not remove agent services installed on monitored
hosts. On a systemd host, stop and disable the orphaned service to halt retries:
```bash
sudo systemctl disable --now pulse-agent.service
```
If the Pulse server is still reachable, use its generated uninstall command so
the agent can deregister cleanly. Otherwise, stopping the service is the safe
first step before platform-local cleanup.
#### VMs show "-" for disk usage
- Install **QEMU Guest Agent** in the VM.
- Enable "QEMU Guest Agent" in Proxmox VM Options.
@@ -85,6 +112,9 @@ If you only missed the token during a fresh install (no password set yet), skip
#### Docker hosts appearing/disappearing
- **Duplicate IDs**: Cloned VMs often share `/etc/machine-id`.
- **Fix**: Run `rm /etc/machine-id && systemd-machine-id-setup` on the clone.
- **Identity note**: The displayed IP is not the durable identity. Pulse uses
the machine ID or an explicit agent ID, so two clones with the same value can
collapse into one record even when their hostnames or IP addresses differ.
### Notifications
+34 -7
View File
@@ -39,6 +39,9 @@ After an upgrade, check the relevant platform page or **Machines** view once
the agent has reported, and confirm the host-local version with
`pulse-agent --version` if the UI has not received a fresh report yet.
This is the agent installer served by your Pulse server. It is separate from the
top-level GitHub `install.sh`, which installs or updates the Pulse server itself.
### Linux (systemd)
```bash
curl -fsSL http://<pulse-ip>:7655/install.sh | \
@@ -300,7 +303,10 @@ The agent can report S.M.A.R.T. disk temperatures, health status, identity, and
## Auto-Update
The unified agent automatically checks for updates every hour. When a new version is available:
Eligible v6 agents automatically check the Pulse server for updates every hour.
The check is asynchronous: updating the Pulse server changes the target version,
but does not prove every agent is online, eligible, or already current. When a
new version is available:
1. Agent downloads the new binary from the Pulse server
2. Verifies the checksum
@@ -309,12 +315,20 @@ The unified agent automatically checks for updates every hour. When a new versio
5. Replaces itself atomically (with backup)
6. Restarts with the same configuration
When an already-installed v5 `pulse-agent` moves to v6, the first automatic hop
is performed by the v5 updater. That hop verifies TLS by default, the SHA-256
checksum, executable magic, size limits, and atomic replacement, but the newer
v6 signature and `--self-test` checks apply only after the agent has landed on
v6. Use HTTPS or a trusted local network for v5-to-v6 automatic migration. For
high-assurance environments, reinstall the v6 `pulse-agent` through the signed
Use the manual update path for v5 agents, PVE host agents, agents with
auto-update disabled, and agents blocked by authentication, missing connection
state, download, trust, or self-test failures. Open an outdated-agent notice or
`/settings/infrastructure?agentDoctor=1` to open **Agent Doctor** and
copy the command for each reported host. Pulse does not remotely execute those
commands.
If an already-installed v5 `pulse-agent` follows its legacy automatic updater
path instead of the supported manual installer path, the first hop is performed
by the v5 updater. That hop verifies TLS by default, the SHA-256 checksum,
executable magic, size limits, and atomic replacement, but the newer v6
signature and `--self-test` checks apply only after the agent has landed on v6.
Use HTTPS or a trusted local network for that legacy migration. For
high-assurance environments, install the v6 `pulse-agent` through the signed
installer path instead of relying on a plain-HTTP first hop.
To disable auto-updates:
@@ -403,6 +417,14 @@ Set `--health-addr=""` or `PULSE_HEALTH_ADDR=off` to disable the health/metrics
- Check logs: `journalctl -u pulse-agent -f`
- Verify network connectivity to Pulse server
- Ensure auto-update is not disabled
- Confirm the agent can authenticate and that its saved connection state still
identifies the Pulse URL and token.
- Open **Agent Doctor** from an outdated-agent notice or
`/settings/infrastructure?agentDoctor=1` and use the command for that reported
host. Do not substitute the public GitHub server installer.
- Administrators can query the read-only Agent Fleet Doctor endpoint,
`GET /api/agents/diagnostics`, for liveness, version, profile, telemetry, and
identity evidence. The endpoint reports repair handoffs but does not run them.
### Duplicate Agents
If cloned VMs appear as the same agent:
@@ -415,6 +437,11 @@ Or set a unique agent ID:
--agent-id my-unique-agent-id
```
The displayed or reported IP is not the durable agent identity. Pulse normally
uses the machine ID (or an explicit `--agent-id`), so cloned systems must have
unique machine and agent IDs even when their hostnames, MAC addresses, and IPs
differ.
### Permission Denied (Docker)
Ensure the agent can access the Docker socket:
```bash
+18 -11
View File
@@ -24,7 +24,7 @@ stable. Keep v5.1.35 as the explicit rollback target for the v6.0.0 cutover.
## Before You Upgrade
- Create an encrypted config backup: **Settings → System → Recovery → Create Backup** (older versions labeled this **Backups**)
- Open **Settings → System → Updates** and review the upgrade checks on the update plan. Pulse checks the server update path, current agent continuity, and agent reporting token scope before you install.
- Open **Settings → System → Updates** and review the upgrade checks on the update plan. Pulse checks the server update path, current agent continuity, and agent reporting token scope before you install. These checks describe the currently reported fleet; they do not prove every installed agent is online or already updated.
- Confirm you can access the host/container console (for rollback and bootstrap token retrieval)
- If you have any external integrations or scripts: review the **API Changes** section below
@@ -82,18 +82,25 @@ No. Upgrade the existing Pulse server installation in place.
No. Use the unified installer to upgrade existing agent deployments in place. Generate the current command from **Settings → Infrastructure → Install on a host**, then run it on the host that already has the v5 agent service. You do not need to remove the old service first.
### Does upgrading the Pulse server to v6 automatically upgrade my agents?
### Does upgrading the Pulse server prove that all agents have upgraded?
No. The server upgrade and the Unified Agent upgrade are separate operations.
After the server is on v6, use the generated install or upgrade command from
**Settings → Infrastructure → Install on a host** when you want to move agents
to v6. A v5 agent can be missing from v6 Reporting until it has upgraded,
authenticated, and sent its first v6 report.
No. The server and Unified Agent have separate update lifecycles. After the
server changes the target version, eligible v6 agents normally discover and
apply that update asynchronously during their update checks. The server being
current is not proof that every installed agent has checked in or converged.
Agent self-update still belongs to the agent update path and depends on the
agent being able to authenticate, reach a trusted update channel, and accept
the release signing key. Do not treat the server update alone as proof that
every installed agent has moved to the same v6 version.
Use the manual path for v5 agents, PVE host agents, agents with auto-update
disabled, and agents blocked by authentication, missing connection state,
download, trust, or self-test failures. For v5-to-v6 upgrades, generate the
current command from **Settings → Infrastructure → Install on a host** and run it
on the host with the existing agent service. A v5 agent can be missing from v6
Reporting until it has upgraded, authenticated, and sent its first v6 report.
For agents already visible to v6, open an outdated-agent notice or **Agent
Doctor** at `/settings/infrastructure?agentDoctor=1`. It shows per-host commands
for the operator to copy and run; it does not remotely execute fleet updates. Agent
self-update and manual update both still depend on valid authentication, a
reachable trusted update channel, and accepted release signing keys.
### Will an upgraded v5 agent keep the same identity in v6?
@@ -55,6 +55,7 @@ that binary, not separate customer-facing agent products.
22a. `frontend-modern/src/components/Settings/ConnectionEditor/CredentialSlots/AvailabilityTargetSlot.tsx`
23. `frontend-modern/src/components/Settings/InfrastructureWorkspace.tsx`
23a. `frontend-modern/src/components/Settings/InfrastructureAgentUpdatesDialog.tsx`
23b. `frontend-modern/src/components/Settings/useAgentFleetDiagnostics.ts`
24. `frontend-modern/src/components/Settings/InfrastructureSourceManager.tsx`
25. `frontend-modern/src/components/Settings/InfrastructureSourcePicker.tsx`
26. `frontend-modern/src/components/Settings/InfrastructureDiscoverySettingsDialog.tsx`
@@ -416,10 +417,26 @@ equivalent saved-state update mode.
Agent Fleet Doctor diagnostics extend that same read-only lifecycle triage
surface: `GET /api/agents/diagnostics` may explain stale versions, missing
reports, profile deployment drift, expected Docker/Kubernetes telemetry gaps,
identity splits, and removed-agent blocks, and may advertise existing repair
handoffs such as copy-upgrade-command or allow-reenroll. It must not perform
the repair, create an action plan, or replace the canonical `/api/connections`
fleet projection used by Infrastructure.
identity splits, updater/module failures, and removed-agent blocks. The
diagnostic target is the canonical agent-update target, not the Pulse server
build string, so development builds and separately versioned agent artifacts
do not create false drift. The Infrastructure workspace polls this read model
only while Agent Doctor is open, enriches canonical connection-ledger rows by
stable `connectionId`, and retains ledger-only fallback rows when structured
evidence is absent.
Repair entries remain handoffs to existing lifecycle operations:
`copy_upgrade_command` renders a local operator command and
`allow_reenroll` invokes the existing removed-agent flow. They never enqueue a
remote command or create an action plan. A stale agent with an unknown or
unsupported platform, or with FreeBSD/pfSense installer state that the server
cannot verify, must receive an unsupported handoff rather than a guessed
command. When automatic update is enabled and checking or applying, Agent
Doctor waits and reports that state instead of prematurely offering a manual
installer path. The `agentDoctor` route key is canonical; the older
`agentUpdates` deep link remains a read-side compatibility alias only. Agent
Doctor must not replace the canonical `/api/connections` fleet projection used
by Infrastructure.
Agent lifecycle and fleet-operation surfaces may consume
`POST /api/actions/plan` for resource capability planning, but the action plan
@@ -143,6 +143,7 @@ product API routes free of maintainer commercial analytics.
73b. `pkg/extensions/ai_autofix.go`
83. `scripts/generate-types.go`
83a. `internal/api/agent_fleet_doctor.go`
83b. `frontend-modern/src/api/agentDiagnostics.ts`
84. `internal/api/notification_queue.go`
The alert and notification transports expose the operational-trust contract
@@ -5563,9 +5564,23 @@ desired/applied disagreement into one enabled/disabled fact.
The adjacent Agent Fleet Doctor endpoint, `GET /api/agents/diagnostics`, is a
read-only admin `settings:read` API for deeper fleet triage. It may summarize
liveness, version drift, profile deployment drift, missing expected telemetry,
identity splits, removed-agent blocks, and supported repair handoff hints, but
it must not mutate configuration, enqueue remote commands, or become the
canonical `/api/connections` fleet row source.
identity splits, updater/module state, removed-agent blocks, and supported
repair handoff hints, but it must not mutate configuration, enqueue remote
commands, or become the canonical `/api/connections` fleet row source.
The response is a versioned additive payload. `schemaVersion` identifies the
diagnostic schema, `serverVersion` describes the running application, and
`agentUpdateTargetVersion` separately carries the canonical release target
used for agent version drift. Each row retains stable agent and connection
identity plus bounded platform, network, profile, updater, module, reason, and
repair-handoff evidence. Raw machine identity is represented only by a
one-way fingerprint; unbounded or secret-shaped updater/module errors are
redacted before serialization; malformed interface addresses are omitted.
Older consumers may ignore the additive fields, while the Agent Doctor client
must tolerate absent optional fields and preserve `/api/connections` fallback
rows. Repair objects describe whether an existing local handoff is supported
and for which normalized platform; they are not executable commands or
authorization grants.
That same shared infrastructure-settings boundary also owns install-profile
semantics surfaced by
`frontend-modern/src/components/Settings/infrastructureOperationsModel.tsx`:
@@ -5142,3 +5142,26 @@ The focused browser proofs are
`frontend-modern/src/components/Storage/__tests__/useDiskDetailModel.test.ts`,
and
`frontend-modern/src/features/alerts/thresholds/hooks/__tests__/truenasThresholdPersistence.test.tsx`.
### Agent Doctor settings framing
Settings labels the application update panel **Pulse server updates** and keeps
agent lifecycle triage in the separate **Agent Doctor** dialog. Platform update
notices, Diagnostics, and Infrastructure rows use the canonical Agent Doctor
route handoff instead of recreating installer or repair controls. The dialog
may enrich the shared connections ledger with structured diagnostics, but it
must preserve loading, unavailable, unsupported, waiting-for-auto-update,
removed, warning, and critical states rather than flattening them into a
generic update badge.
The canonical query is `agentDoctor`; `agentUpdates` remains a compatibility
alias that opens the same dialog and does not create a second settings surface.
Scoped connection IDs filter active rows without hiding removed-agent history
from the unscoped view. Copy-command controls render only for a backend- and
frontend-confirmed supported platform; unknown and unverified FreeBSD/pfSense
states show bounded guidance with no executable command. The focused proofs are
`frontend-modern/src/components/Settings/__tests__/infrastructureAgentDoctorModel.test.ts`,
`frontend-modern/src/components/Settings/__tests__/InfrastructureWorkspace.test.tsx`,
`frontend-modern/src/components/Settings/__tests__/DiagnosticsResultsPanel.test.tsx`,
`frontend-modern/src/components/Settings/__tests__/settingsHeaderMeta.branchcov0713.test.ts`,
and `frontend-modern/src/utils/__tests__/updatesPresentation.test.ts`.
@@ -158,6 +158,7 @@ resource health.
56a. `internal/monitoring/pbs_protection_observation.go`
57. `internal/monitoring/multi_tenant_monitor.go`
58. `internal/monitoring/proxmox_action_observer.go`
59. `internal/monitoring/agent_fleet_doctor.go`
## Shared Boundaries
@@ -1772,6 +1773,28 @@ transition detection consumes that status instead of inventing a fixed stale
window: a stopped guest can have fresh inventory, while a stale source cannot
authoritatively prove either a stopped transition or recovery.
### Agent fleet diagnostic derivation
Monitoring owns the read-only Agent Fleet Doctor derivation over current host,
Docker, Kubernetes, removed-agent, profile-assignment, and deployment state.
`internal/fleethealth/agent.go` supplies the shared agent connection identity,
heartbeat cutoff, and version-drift vocabulary used by both the monitoring
diagnostic and API connections ledger. Five expected reports must be missed,
with a five-minute minimum, before an agent becomes stale; missing timestamps
remain pending/never-reported rather than silently healthy. Version comparison
uses the canonical agent-update target independently from the running server
build version.
The diagnostic may derive bounded updater and module failure reasons, normalized
platform and network evidence, profile drift, and safe repair-handoff support.
It hashes raw machine IDs, filters malformed interface addresses, and redacts
unbounded error strings before returning evidence. Derivation must not mutate
monitor state, probe providers, enqueue commands, or turn a repair hint into
execution authority. Unknown updater states remain explicit warnings; unknown
platforms and unverified FreeBSD/pfSense installer state fail closed for
upgrade-command support. `internal/fleethealth/agent_test.go` and
`internal/monitoring/agent_fleet_doctor_test.go` are the focused runtime proofs.
### Unified Agent destination delivery metrics
The local agent health listener exports
@@ -1123,6 +1123,8 @@
"frontend-modern/src/components/Settings/connectionsTableModel.ts",
"frontend-modern/src/components/Settings/DiscoverySettingsForm.tsx",
"frontend-modern/src/components/Settings/discoverySettingsModel.ts",
"frontend-modern/src/components/Settings/infrastructureAgentUpdateCommandsModel.ts",
"frontend-modern/src/components/Settings/InfrastructureAgentUpdatesDialog.tsx",
"frontend-modern/src/components/Settings/InfrastructureDiscoverySettingsDialog.tsx",
"frontend-modern/src/components/Settings/InfrastructureInstallerSection.tsx",
"frontend-modern/src/components/Settings/infrastructureOperationsModel.tsx",
@@ -1138,6 +1140,7 @@
"frontend-modern/src/components/Settings/NodeModalSetupGuideSection.tsx",
"frontend-modern/src/components/Settings/NodeModalStatusFooter.tsx",
"frontend-modern/src/components/Settings/proxmoxSettingsModel.ts",
"frontend-modern/src/components/Settings/useAgentFleetDiagnostics.ts",
"frontend-modern/src/components/Settings/useAgentProfilesPanelState.ts",
"frontend-modern/src/components/Settings/useConnectionRowActions.ts",
"frontend-modern/src/components/Settings/useConnectionsLedger.ts",
@@ -1472,6 +1475,29 @@
"frontend-modern/src/components/SetupWizard/__tests__/SetupCompletionPanel.guardrails.test.ts"
]
},
{
"id": "agent-doctor-settings-surface",
"label": "Agent Doctor lifecycle and safe update-handoff proof",
"match_prefixes": [],
"match_files": [
"frontend-modern/src/components/Settings/infrastructureAgentUpdateCommandsModel.ts",
"frontend-modern/src/components/Settings/InfrastructureAgentUpdatesDialog.tsx",
"frontend-modern/src/components/Settings/InfrastructureSourceManager.tsx",
"frontend-modern/src/components/Settings/InfrastructureWorkspace.tsx",
"frontend-modern/src/components/Settings/infrastructureWorkspaceModel.ts",
"frontend-modern/src/components/Settings/useAgentFleetDiagnostics.ts",
"frontend-modern/src/components/Settings/useConnectionsLedger.ts",
"frontend-modern/src/components/Settings/useInfrastructureOperationsState.tsx"
],
"allow_same_subsystem_tests": false,
"test_prefixes": [],
"exact_files": [
"frontend-modern/src/components/Settings/__tests__/infrastructureAgentDoctorModel.test.ts",
"frontend-modern/src/components/Settings/__tests__/InfrastructureSourceManager.test.tsx",
"frontend-modern/src/components/Settings/__tests__/InfrastructureWorkspace.test.tsx",
"frontend-modern/src/components/Settings/__tests__/infrastructureWorkspaceModel.test.ts"
]
},
{
"id": "unified-agent-settings-surface",
"label": "unified agent settings lifecycle proof",
@@ -2592,6 +2618,27 @@
"internal/api/route_inventory_test.go"
]
},
{
"id": "agent-doctor-api-surface",
"label": "Agent Doctor diagnostics payload and consumer proof",
"match_prefixes": [],
"match_files": [
"frontend-modern/src/api/agentDiagnostics.ts",
"frontend-modern/src/components/Settings/useInfrastructureOperationsState.tsx",
"internal/api/agent_fleet_doctor.go",
"internal/api/connections_aggregator.go"
],
"allow_same_subsystem_tests": false,
"test_prefixes": [],
"exact_files": [
"frontend-modern/src/api/__tests__/agentDiagnostics.test.ts",
"frontend-modern/src/components/Settings/__tests__/infrastructureAgentDoctorModel.test.ts",
"internal/api/agent_fleet_doctor_test.go",
"internal/api/connections_aggregator_test.go",
"internal/api/contract_test.go",
"internal/api/update_readiness_test.go"
]
},
{
"id": "backend-payload-contracts",
"label": "backend API payload proof",
@@ -4445,6 +4492,21 @@
"frontend-modern/src/components/shared/FilterToolbar.test.tsx"
]
},
{
"id": "agent-doctor-settings-framing",
"label": "Agent Doctor settings framing and diagnostic handoff proof",
"match_prefixes": [],
"match_files": [
"frontend-modern/src/components/Settings/DiagnosticsResultsPanel.tsx",
"frontend-modern/src/components/Settings/settingsHeaderMeta.ts"
],
"allow_same_subsystem_tests": false,
"test_prefixes": [],
"exact_files": [
"frontend-modern/src/components/Settings/__tests__/DiagnosticsResultsPanel.test.tsx",
"frontend-modern/src/components/Settings/__tests__/settingsHeaderMeta.branchcov0713.test.ts"
]
},
{
"id": "settings-shell-and-framing",
"label": "settings shell framing proof",
@@ -4893,6 +4955,7 @@
"lane": "L13",
"contract": "docs/release-control/v6/internal/subsystems/monitoring.md",
"owned_prefixes": [
"internal/fleethealth/",
"internal/monitoring/",
"internal/storagehealth/",
"internal/truenas/",
@@ -5245,6 +5308,25 @@
"pkg/diskinventory/status_test.go"
]
},
{
"id": "agent-fleet-diagnostics-runtime",
"label": "Agent fleet liveness, update, identity, and redaction proof",
"match_prefixes": [
"internal/fleethealth/"
],
"match_files": [
"internal/monitoring/agent_fleet_doctor.go"
],
"allow_same_subsystem_tests": false,
"test_prefixes": [],
"exact_files": [
"internal/api/agent_fleet_doctor_test.go",
"internal/api/connections_aggregator_test.go",
"internal/fleethealth/agent_test.go",
"internal/monitoring/agent_fleet_doctor_test.go",
"internal/monitoring/monitoring_fleet_doctor_helpers_branchcov0716_test.go"
]
},
{
"id": "monitoring-runtime",
"label": "monitoring runtime proof",
@@ -1457,6 +1457,17 @@ the exact operational record and policy-shaped evidence IDs internally.
Unauthorized, stale, partial, permission-limited, ambiguous, unsupported, or
executor-unready records return no offer and no cross-resource detail.
Agent Fleet Doctor is an authenticated admin `settings:read` projection, not a
support-bundle export or command channel. Its identity evidence may expose only
a one-way machine-ID fingerprint, normalized platform metadata, and validated
IP/interface addresses. Raw machine IDs, credentials, tokens, environment
values, command text, and unbounded updater or module errors must not cross the
API boundary; secret-shaped error detail is reduced to bounded redacted
evidence. A repair entry is descriptive support metadata only and never grants
`settings:write`, local shell, agent execution, or action approval authority.
Unknown platform and unverified installer state fail closed without rendering
an executable upgrade command.
### Operational Trust evidence and mutation authorization
Evidence detail is tenant-scoped through the selected canonical record and
@@ -1232,7 +1232,14 @@ recovery scope, or a storage/recovery-owned secret source.
storage and recovery may read Agent Fleet Doctor evidence as operational
context for stale agents, version drift, profile drift, or identity split
investigation, but must not treat that endpoint as a storage/recovery
health source, repair API, or recovery-local fleet payload shape.
health source, repair API, or recovery-local fleet payload shape. Agent
updater/module failure, missing telemetry, machine fingerprint, network
evidence, and supported/unsupported installer handoffs remain lifecycle
diagnostics; none proves backup completeness, protection posture, restore
readiness, recovery-point identity, or storage mutation authority. An
Agent Doctor repair handoff may open an existing lifecycle flow or copy a
local installer command, but it must never be reinterpreted as a restore,
recovery, SMART, or cleanup action.
24. Keep backend-native platform configuration reads on the adjacent AI/runtime and platform contracts. When `internal/api/` wires native TrueNAS app config for Assistant, storage and recovery may use that runtime shape during investigation, but they must not grow a parallel recovery-local config transport or provider-shaped configuration payload.
25. Keep provider-backed poll cadence and settings-runtime health on the adjacent platform-connections contract. When shared `internal/api/` and poller wiring expose TrueNAS last-sync status, failure summaries, discovered contribution counts, manual saved-test status refresh, or platform handoff links in settings, storage and recovery may consume the resulting datasets, apps, disks, and recovery artifacts but must not redefine those settings-runtime health semantics or connection-level handoffs in storage/recovery-local transport or page flows.
26. Keep recovery filter/query state on the shared route-state parsing contract without restoring standalone recovery navigation. When platform pages or other embedded owners expose TrueNAS recovery context, they may reuse the canonical recovery query vocabulary with owned `platform` and `node` fields, but they must land inside an owning platform/runtime route instead of inventing drawer-local recovery URLs, treating PBS services as the only recovery path, or sending operators to the retired Recovery aggregate route.
@@ -0,0 +1,57 @@
import { beforeEach, describe, expect, it, vi } from 'vitest';
import { apiFetchJSON } from '@/utils/apiClient';
import { AgentDiagnosticsAPI } from '../agentDiagnostics';
vi.mock('@/utils/apiClient', () => ({ apiFetchJSON: vi.fn() }));
const mockedApiFetchJSON = vi.mocked(apiFetchJSON);
describe('AgentDiagnosticsAPI', () => {
beforeEach(() => vi.clearAllMocks());
it('loads the read-only fleet diagnostic endpoint', async () => {
mockedApiFetchJSON.mockResolvedValueOnce({
schemaVersion: 1,
generatedAt: 123,
serverVersion: '6.2.0',
agentUpdateTargetVersion: '6.2.0',
summary: { total: 1, warning: 1 },
agents: [
{
connectionId: 'agent:host-1',
rowKey: 'agent-host-1',
id: 'host-1',
name: 'host-1',
types: ['host'],
status: 'warning',
reasons: [],
},
],
});
const result = await AgentDiagnosticsAPI.getFleetDiagnostics();
expect(mockedApiFetchJSON).toHaveBeenCalledWith('/api/agents/diagnostics');
expect(result.summary).toEqual({
total: 1,
healthy: 0,
warning: 1,
critical: 0,
removed: 0,
});
expect(result.schemaVersion).toBe(1);
expect(result.agentUpdateTargetVersion).toBe('6.2.0');
expect(result.agents[0].connectionId).toBe('agent:host-1');
});
it('normalizes omitted collections for rolling upgrades', async () => {
mockedApiFetchJSON.mockResolvedValueOnce({});
await expect(AgentDiagnosticsAPI.getFleetDiagnostics()).resolves.toMatchObject({
schemaVersion: 0,
generatedAt: 0,
agents: [],
summary: { total: 0, healthy: 0, warning: 0, critical: 0, removed: 0 },
});
});
});
+120
View File
@@ -0,0 +1,120 @@
import { apiFetchJSON } from '@/utils/apiClient';
export type AgentFleetDiagnosticStatus = 'healthy' | 'warning' | 'critical' | 'removed';
export interface AgentFleetDiagnosticSummary {
total: number;
healthy: number;
warning: number;
critical: number;
removed: number;
}
export interface AgentFleetDiagnosticReason {
code: string;
severity: AgentFleetDiagnosticStatus | string;
message: string;
evidence?: string[];
}
export interface AgentFleetDiagnosticRepair {
code: string;
label: string;
description: string;
supported: boolean;
mode?: 'handoff' | string;
platform?: string;
scope?: string;
}
export interface AgentFleetDiagnosticUpdate {
state: string;
autoUpdate: boolean;
updatedFrom?: string;
availableVersion?: string;
lastCheckedAt?: string;
lastAttemptAt?: string;
lastSuccessAt?: string;
lastError?: string;
}
export interface AgentFleetDiagnosticModule {
name: string;
enabled: boolean;
state: string;
lastError?: string;
updatedAt?: string;
}
export interface AgentFleetAgentDiagnostic {
/** Canonical `/api/connections` identifier. */
connectionId?: string;
rowKey: string;
id: string;
agentId?: string;
name: string;
hostname?: string;
platform?: string;
osName?: string;
osVersion?: string;
kernelVersion?: string;
architecture?: string;
machineIdFingerprint?: string;
reportIp?: string;
interfaceAddresses?: string[];
types: string[];
status: AgentFleetDiagnosticStatus;
rawStatus?: string;
lastSeen?: number;
intervalSeconds?: number;
version?: string;
profileId?: string;
profileName?: string;
profileVersion?: number;
deployedProfileVersion?: number;
agentUpdate?: AgentFleetDiagnosticUpdate;
agentModules?: AgentFleetDiagnosticModule[];
reasons: AgentFleetDiagnosticReason[];
repairActions?: AgentFleetDiagnosticRepair[];
}
export interface AgentFleetDiagnosticsResponse {
schemaVersion: number;
generatedAt: number;
serverVersion?: string;
agentUpdateTargetVersion?: string;
summary: AgentFleetDiagnosticSummary;
agents: AgentFleetAgentDiagnostic[];
}
const EMPTY_SUMMARY: AgentFleetDiagnosticSummary = {
total: 0,
healthy: 0,
warning: 0,
critical: 0,
removed: 0,
};
interface AgentFleetDiagnosticsWireResponse {
schemaVersion?: number;
generatedAt?: number;
serverVersion?: string;
agentUpdateTargetVersion?: string;
summary?: Partial<AgentFleetDiagnosticSummary>;
agents?: AgentFleetAgentDiagnostic[];
}
export class AgentDiagnosticsAPI {
static async getFleetDiagnostics(): Promise<AgentFleetDiagnosticsResponse> {
const response =
await apiFetchJSON<AgentFleetDiagnosticsWireResponse>('/api/agents/diagnostics');
return {
schemaVersion: response.schemaVersion ?? 0,
generatedAt: response.generatedAt ?? 0,
serverVersion: response.serverVersion,
agentUpdateTargetVersion: response.agentUpdateTargetVersion,
summary: { ...EMPTY_SUMMARY, ...response.summary },
agents: response.agents ?? [],
};
}
}
@@ -15,7 +15,7 @@ import Server from 'lucide-solid/icons/server';
import Shield from 'lucide-solid/icons/shield';
import Sparkles from 'lucide-solid/icons/sparkles';
import XCircle from 'lucide-solid/icons/x-circle';
import { Button } from '@/components/shared/Button';
import { Button, ButtonLink } from '@/components/shared/Button';
import { EmptyState } from '@/components/shared/EmptyState';
import { StatusIndicatorBadge } from '@/components/shared/StatusIndicatorBadge';
import { getSemanticTonePresentation } from '@/utils/semanticTonePresentation';
@@ -385,6 +385,15 @@ export const DiagnosticsResultsPanel: Component<DiagnosticsResultsPanelProps> =
{props.diagnosticsData?.dockerAgents?.recommendedAgentVersion}
</div>
</Show>
<div class="mt-3 border-t border-border-subtle pt-3">
<ButtonLink
href="/settings/infrastructure?agentDoctor=1"
variant="outline"
size="sm"
>
Open Agent Doctor
</ButtonLink>
</div>
</Card>
</Show>
@@ -1,5 +1,5 @@
import { For, Show, createMemo, type Component } from 'solid-js';
import { X } from 'lucide-solid';
import { RefreshCw, X } from 'lucide-solid';
import { Dialog } from '@/components/shared/Dialog';
import { Button, CommandCopyButton } from '@/components/shared/Button';
import { copyToClipboard } from '@/utils/clipboard';
@@ -8,33 +8,91 @@ import {
getUnifiedAgentClipboardCopyErrorMessage,
getUnifiedAgentClipboardCopySuccessMessage,
} from '@/utils/unifiedAgentInventoryPresentation';
import type { InfrastructureAgentUpdateTarget } from './infrastructureAgentUpdateCommandsModel';
import {
summarizeInfrastructureAgentDoctorTargets,
type InfrastructureAgentDoctorStatus,
type InfrastructureAgentDoctorTarget,
} from './infrastructureAgentUpdateCommandsModel';
import { useInfrastructureOperationsContext } from './useInfrastructureOperationsState';
interface InfrastructureAgentUpdatesDialogProps {
isOpen: boolean;
targets: readonly InfrastructureAgentUpdateTarget[];
targets: readonly InfrastructureAgentDoctorTarget[];
diagnosticsLoading?: boolean;
diagnosticsError?: unknown;
onRetryDiagnostics?: () => void;
onClose: () => void;
}
const STATUS_PRESENTATION: Record<
InfrastructureAgentDoctorStatus,
{ label: string; badgeClass: string }
> = {
healthy: {
label: 'Healthy',
badgeClass: 'bg-emerald-100 text-emerald-800 dark:bg-emerald-900 dark:text-emerald-200',
},
waiting: {
label: 'Waiting for updater',
badgeClass: 'bg-blue-100 text-blue-800 dark:bg-blue-900 dark:text-blue-200',
},
warning: {
label: 'Needs attention',
badgeClass: 'bg-amber-100 text-amber-800 dark:bg-amber-900 dark:text-amber-200',
},
critical: {
label: 'Critical',
badgeClass: 'bg-rose-100 text-rose-800 dark:bg-rose-900 dark:text-rose-200',
},
removed: {
label: 'Removed',
badgeClass: 'bg-surface-alt text-muted',
},
unknown: {
label: 'Unknown',
badgeClass: 'bg-surface-alt text-base-content',
},
};
const formatLastSeen = (value?: number | string | null): string | undefined => {
if (!value) return undefined;
const timestamp = typeof value === 'number' ? value : Date.parse(value);
if (!Number.isFinite(timestamp)) return undefined;
return new Date(timestamp).toLocaleString();
};
export const InfrastructureAgentUpdatesDialog: Component<InfrastructureAgentUpdatesDialogProps> = (
props,
) => {
const operations = useInfrastructureOperationsContext();
const targetCount = createMemo(() => props.targets.length);
const summary = createMemo(() => summarizeInfrastructureAgentDoctorTargets(props.targets));
const commandTargets = createMemo(() =>
props.targets.filter(
(target) =>
target.needsUpdate &&
Boolean(target.connection) &&
Boolean(target.commandPlatform) &&
!target.commandBlockedReason,
),
);
const tokenGatedTargetCount = createMemo(
() =>
props.targets.filter((target) =>
operations.getAgentConnectionUpgradeCommandRequiresToken(target.connection),
commandTargets().filter(
(target) =>
target.connection &&
operations.getAgentConnectionUpgradeCommandRequiresToken(
target.connection,
target.commandPlatform ?? undefined,
),
).length,
);
const hasTokenGatedTargets = createMemo(() => tokenGatedTargetCount() > 0);
const targetSummary = createMemo(() =>
targetCount() === 1 ? '1 agent needs an update' : `${targetCount()} agents need updates`,
);
const commandReadyForTarget = (target: InfrastructureAgentUpdateTarget) =>
!operations.getAgentConnectionUpgradeCommandRequiresToken(target.connection) ||
operations.commandsUnlocked();
const commandReadyForTarget = (target: InfrastructureAgentDoctorTarget) =>
Boolean(target.connection && target.commandPlatform && !target.commandBlockedReason) &&
(!operations.getAgentConnectionUpgradeCommandRequiresToken(
target.connection!,
target.commandPlatform!,
) ||
operations.commandsUnlocked());
const copyCommand = async (command: string) => {
const success = await copyToClipboard(command);
@@ -46,13 +104,13 @@ export const InfrastructureAgentUpdatesDialog: Component<InfrastructureAgentUpda
};
return (
<Dialog isOpen={props.isOpen} onClose={props.onClose} ariaLabel="Update Pulse Agents">
<Dialog isOpen={props.isOpen} onClose={props.onClose} ariaLabel="Agent Doctor">
<div class="flex h-full min-h-0 flex-col">
<div class="flex items-start justify-between gap-4 border-b border-border bg-surface-alt px-4 py-4 sm:px-6">
<div class="space-y-1">
<h2 class="text-base font-semibold text-base-content">Update Pulse Agents</h2>
<h2 class="text-base font-semibold text-base-content">Agent Doctor</h2>
<p class="text-sm text-muted">
Copy the update command for each host that is behind the current Pulse Agent target.
Diagnose fleet connectivity, versions, identity, profiles, and removed-agent state.
</p>
</div>
<Button
@@ -60,33 +118,74 @@ export const InfrastructureAgentUpdatesDialog: Component<InfrastructureAgentUpda
variant="outline"
size="iconMd"
onClick={props.onClose}
aria-label="Close agent update commands"
aria-label="Close Agent Doctor"
>
<X class="h-4 w-4" />
</Button>
</div>
<div class="min-h-0 flex-1 space-y-4 overflow-y-auto p-4 sm:p-6">
<Show
when={targetCount() > 0}
fallback={
<div class="rounded-md border border-emerald-200 bg-emerald-50 px-4 py-3 text-sm text-emerald-900 dark:border-emerald-800 dark:bg-emerald-950/40 dark:text-emerald-100">
Pulse does not currently see any agents behind the target version.
<Show when={props.diagnosticsError}>
<section class="flex flex-col gap-3 rounded-md border border-amber-200 bg-amber-50 px-4 py-3 text-sm text-amber-900 dark:border-amber-800 dark:bg-amber-950/30 dark:text-amber-100 sm:flex-row sm:items-center sm:justify-between">
<div>
<div class="font-semibold">Structured diagnostics are temporarily unavailable</div>
<p class="mt-1 text-xs leading-5 text-amber-800 dark:text-amber-200">
Showing the last known connection-ledger assessment. Profile and removed-agent
details may be incomplete.
</p>
</div>
<Show when={props.onRetryDiagnostics}>
<Button
type="button"
variant="outline"
size="sm"
class="gap-2 self-start"
onClick={() => props.onRetryDiagnostics?.()}
>
<RefreshCw class="h-3.5 w-3.5" aria-hidden="true" />
Retry
</Button>
</Show>
</section>
</Show>
<Show when={props.diagnosticsLoading && props.targets.length === 0}>
<div class="rounded-md border border-border bg-surface-alt px-4 py-3 text-sm text-muted">
Checking agent fleet health
</div>
</Show>
<Show
when={props.targets.length > 0}
fallback={
<Show when={!props.diagnosticsLoading}>
<div class="rounded-md border border-emerald-200 bg-emerald-50 px-4 py-3 text-sm text-emerald-900 dark:border-emerald-800 dark:bg-emerald-950/40 dark:text-emerald-100">
No Pulse Agent connections are currently in scope.
</div>
</Show>
}
>
<div class="rounded-md border border-amber-200 bg-amber-50 px-4 py-3 text-sm text-amber-900 dark:border-amber-800 dark:bg-amber-950/30 dark:text-amber-100">
<div class="font-semibold">{targetSummary()}</div>
<p class="mt-1 text-xs leading-5 text-amber-800 dark:text-amber-200">
Run these commands on the affected hosts. The installer reuses the existing agent
connection state where supported, verifies the matching agent binary, preserves host
identity, and restarts the service after the update.
</p>
<section
aria-label="Agent Doctor summary"
class="grid grid-cols-2 gap-2 sm:grid-cols-6"
>
<SummaryItem label="Critical" value={summary().critical} tone="critical" />
<SummaryItem label="Attention" value={summary().warning} tone="warning" />
<SummaryItem label="Waiting" value={summary().waiting} />
<SummaryItem label="Healthy" value={summary().healthy} tone="healthy" />
<SummaryItem label="Unknown" value={summary().unknown} />
<SummaryItem label="Removed" value={summary().removed} />
</section>
<div class="rounded-md border border-blue-200 bg-blue-50 px-4 py-3 text-xs leading-5 text-blue-900 dark:border-blue-800 dark:bg-blue-950/30 dark:text-blue-100">
Update commands are host-local: copy one to the affected machine to update its Pulse
Agent from this server. They do not update the Pulse server runtime and Pulse does not
run them remotely.
</div>
<Show
when={
hasTokenGatedTargets() &&
tokenGatedTargetCount() > 0 &&
operations.requiresToken() &&
!operations.commandsUnlocked()
}
@@ -98,8 +197,8 @@ export const InfrastructureAgentUpdatesDialog: Component<InfrastructureAgentUpda
</h3>
<p class="text-xs leading-5 text-blue-800 dark:text-blue-200">
{tokenGatedTargetCount() === 1
? 'One target still needs a scoped install token before Pulse can show its update command.'
: `${tokenGatedTargetCount()} targets still need a scoped install token before Pulse can show their update commands.`}
? 'One Windows repair needs a scoped install token before Pulse can show its command.'
: `${tokenGatedTargetCount()} Windows repairs need a scoped install token before Pulse can show their commands.`}
</p>
</div>
<div class="flex flex-col gap-2 sm:flex-row">
@@ -129,15 +228,15 @@ export const InfrastructureAgentUpdatesDialog: Component<InfrastructureAgentUpda
<Show
when={
hasTokenGatedTargets() &&
tokenGatedTargetCount() > 0 &&
!operations.requiresToken() &&
!operations.commandsUnlocked()
}
>
<section class="space-y-3 rounded-md border border-amber-200 bg-amber-50 px-4 py-3 text-sm text-amber-900 dark:border-amber-800 dark:bg-amber-950/30 dark:text-amber-100">
<p class="text-xs leading-5">
Tokens are optional on this Pulse instance. Confirm to generate update commands
without embedding a token.
Tokens are optional on this Pulse instance. Confirm to generate Windows update
commands without embedding a token.
</p>
<button
type="button"
@@ -153,10 +252,19 @@ export const InfrastructureAgentUpdatesDialog: Component<InfrastructureAgentUpda
<div class="space-y-3">
<For each={props.targets}>
{(target) => {
const status = () => STATUS_PRESENTATION[target.status];
const lastSeen = () => formatLastSeen(target.lastSeen);
const command = () =>
operations.getAgentConnectionUpgradeCommand(
target.connection,
target.installFlags,
target.connection && target.commandPlatform
? operations.getAgentConnectionUpgradeCommand(
target.connection,
target.installFlags,
target.commandPlatform,
)
: '';
const otherRepairs = () =>
(target.diagnostic?.repairActions ?? []).filter(
(action) => action.code !== 'copy_upgrade_command',
);
return (
@@ -167,36 +275,125 @@ export const InfrastructureAgentUpdatesDialog: Component<InfrastructureAgentUpda
{target.displayName}
</h3>
<p class="text-xs text-muted">
{target.contextLabel}
<Show when={target.currentVersion && target.expectedVersion}>
{' '}
· {target.currentVersion} -&gt; {target.expectedVersion}
</Show>
{target.contextLabel} · {target.connectionId}
<Show when={lastSeen()}> · Last seen {lastSeen()}</Show>
</p>
</div>
<span class="inline-flex w-fit items-center rounded-full bg-amber-100 px-2 py-0.5 text-[11px] font-medium text-amber-800 dark:bg-amber-900 dark:text-amber-200">
Update available
<span
class={`inline-flex w-fit items-center rounded-full px-2 py-0.5 text-[11px] font-medium ${status().badgeClass}`}
>
{status().label}
</span>
</div>
<Show when={target.currentVersion || target.expectedVersion}>
<div class="grid gap-2 text-xs sm:grid-cols-2">
<Detail
label="Reported agent"
value={target.currentVersion || 'Unknown'}
/>
<Detail
label="Supported target"
value={target.expectedVersion || 'Unknown'}
/>
</div>
</Show>
<Show when={target.updaterLabel}>
<Detail label="Agent updater" value={target.updaterLabel!} />
</Show>
<Show when={target.profileLabel}>
<div class="rounded-md border border-border-subtle bg-surface-alt px-3 py-2 text-xs text-base-content">
<span class="font-medium">Profile: {target.profileLabel}</span>
<Show when={target.profileVersionLabel}>
<span class="text-muted"> · {target.profileVersionLabel}</span>
</Show>
</div>
</Show>
<Show
when={commandReadyForTarget(target)}
when={target.reasons.length > 0}
fallback={
<div class="rounded-md border border-border bg-surface-alt px-3 py-3 text-xs text-muted">
Generate a token to unlock the copyable update command.
</div>
<p class="text-xs text-muted">
{target.status === 'healthy'
? 'No fleet-health issues detected.'
: 'No structured reason is available for this connection yet.'}
</p>
}
>
<div class="relative">
<CommandCopyButton
onClick={() => void copyCommand(command())}
title="Copy update command"
label={`Copy update command for ${target.displayName}`}
/>
<pre class="overflow-x-auto rounded-md bg-base p-3 pr-12 text-xs text-base-content">
<code>{command()}</code>
</pre>
</div>
<ul class="space-y-2">
<For each={target.reasons}>
{(reason) => (
<li class="rounded-md border border-border-subtle bg-surface-alt px-3 py-2">
<div class="text-xs font-medium text-base-content">
{reason.message}
</div>
<Show when={reason.evidence?.length}>
<div class="mt-1 text-[11px] leading-4 text-muted">
{(reason.evidence ?? []).join(' · ')}
</div>
</Show>
</li>
)}
</For>
</ul>
</Show>
<Show when={target.evidence.length > 0}>
<details class="text-xs text-muted">
<summary class="cursor-pointer font-medium text-base-content">
Identity evidence
</summary>
<ul class="mt-2 list-disc space-y-1 pl-5">
<For each={target.evidence}>{(item) => <li>{item}</li>}</For>
</ul>
</details>
</Show>
<For each={otherRepairs()}>
{(repair) => (
<div class="rounded-md border border-border bg-surface-alt px-3 py-2 text-xs">
<div class="font-medium text-base-content">{repair.label}</div>
<div class="mt-0.5 text-muted">{repair.description}</div>
<Show when={repair.scope}>
<div class="mt-1 text-[11px] text-muted">
Required scope: {repair.scope}
</div>
</Show>
</div>
)}
</For>
<Show when={target.needsUpdate}>
<Show
when={!target.commandBlockedReason}
fallback={
<div class="rounded-md border border-border bg-surface-alt px-3 py-3 text-xs text-muted">
{target.commandBlockedReason}
</div>
}
>
<Show
when={commandReadyForTarget(target)}
fallback={
<div class="rounded-md border border-border bg-surface-alt px-3 py-3 text-xs text-muted">
Generate a token to unlock this host-local update command.
</div>
}
>
<div class="relative">
<CommandCopyButton
onClick={() => void copyCommand(command())}
title="Copy host-local agent update command"
label={`Copy update command for ${target.displayName}`}
/>
<pre class="overflow-x-auto rounded-md bg-base p-3 pr-12 text-xs text-base-content">
<code>{command()}</code>
</pre>
</div>
</Show>
</Show>
</Show>
</section>
);
@@ -209,3 +406,30 @@ export const InfrastructureAgentUpdatesDialog: Component<InfrastructureAgentUpda
</Dialog>
);
};
const SummaryItem: Component<{
label: string;
value: number;
tone?: 'critical' | 'warning' | 'healthy';
}> = (props) => {
const toneClass = () => {
if (props.tone === 'critical' && props.value > 0) return 'text-rose-700 dark:text-rose-300';
if (props.tone === 'warning' && props.value > 0) return 'text-amber-700 dark:text-amber-300';
if (props.tone === 'healthy' && props.value > 0)
return 'text-emerald-700 dark:text-emerald-300';
return 'text-base-content';
};
return (
<div class="rounded-md border border-border bg-surface-alt px-3 py-2">
<div class={`text-lg font-semibold ${toneClass()}`}>{props.value}</div>
<div class="text-[11px] text-muted">{props.label}</div>
</div>
);
};
const Detail: Component<{ label: string; value: string }> = (props) => (
<div class="rounded-md border border-border-subtle bg-surface-alt px-3 py-2">
<div class="text-[11px] text-muted">{props.label}</div>
<div class="mt-0.5 font-medium text-base-content">{props.value}</div>
</div>
);
@@ -65,6 +65,7 @@ interface InfrastructureSourceManagerProps {
onRunDiscovery?: () => void;
onOpenDiscoverySettings?: () => void;
onOpenConnection?: (row: InfrastructureSystemRow) => void;
onOpenAgentDoctor?: (agentIds?: string[]) => void;
onReviewDiscoveredSource?: (server: DiscoveredServer) => void;
}
@@ -231,6 +232,30 @@ const rowHasAgentCoverage = (row: InfrastructureSystemRow): boolean =>
(member) => member.source === 'agent' || member.source === 'both' || member.agentConnection,
);
export const agentConnectionIDsForInfrastructureRow = (
row: InfrastructureSystemRow,
updatesOnly = false,
): string[] => {
const connections = [
row.connection,
...row.attachedConnections,
...row.members.map((member) => member.agentConnection).filter(Boolean),
];
return Array.from(
new Set(
connections
.filter(
(connection) =>
connection?.type === 'agent' &&
(!updatesOnly ||
Boolean(connection.agentUpdateAvailable) ||
connection.fleet?.versionDrift === 'behind'),
)
.map((connection) => connection!.id),
),
).sort((left, right) => left.localeCompare(right));
};
const agentHostProfileRouteStep = (
row: InfrastructureSystemRow,
): InfrastructureSourcePickerRouteStep | null => {
@@ -409,6 +434,11 @@ export const InfrastructureSourceManager: Component<InfrastructureSourceManagerP
formatRelativeTimestamp(props.discoveryScanStatus().lastResultAt),
);
const connectedSystemCount = createMemo(() => infrastructureRows().length);
const agentConnectionIds = createMemo(() =>
Array.from(
new Set(infrastructureRows().flatMap((row) => agentConnectionIDsForInfrastructureRow(row))),
),
);
const discoveredCandidateCount = createMemo(() => props.discoveredNodes().length);
// Scoped to Proxmox VE: PBS/PMG/TrueNAS/etc. are fully covered by their
// own API and don't benefit from a paired agent in the way PVE hosts do
@@ -722,18 +752,33 @@ export const InfrastructureSourceManager: Component<InfrastructureSourceManagerP
</span>
</Show>
</div>
<Show when={!props.readOnly && Boolean(setupConfidenceAction().onClick)}>
<Button
type="button"
variant="secondary"
size="mdCompact"
onClick={() => setupConfidenceAction().onClick?.()}
disabled={setupConfidenceAction().disabled}
class="min-h-9 gap-2 self-start sm:self-center"
>
{setupConfidenceActionIcon(setupConfidenceAction().kind)}
{setupConfidenceAction().label}
</Button>
<Show when={!props.readOnly}>
<div class="flex flex-wrap gap-2 self-start sm:self-center">
<Show when={agentConnectionIds().length > 0 && props.onOpenAgentDoctor}>
<Button
type="button"
variant={fleetAttentionSystemCount() > 0 ? 'primary' : 'secondary'}
size="mdCompact"
onClick={() => props.onOpenAgentDoctor?.()}
class="min-h-9"
>
Open Agent Doctor
</Button>
</Show>
<Show when={Boolean(setupConfidenceAction().onClick)}>
<Button
type="button"
variant="secondary"
size="mdCompact"
onClick={() => setupConfidenceAction().onClick?.()}
disabled={setupConfidenceAction().disabled}
class="min-h-9 gap-2"
>
{setupConfidenceActionIcon(setupConfidenceAction().kind)}
{setupConfidenceAction().label}
</Button>
</Show>
</div>
</Show>
</div>
@@ -946,11 +991,34 @@ export const InfrastructureSourceManager: Component<InfrastructureSourceManagerP
{row.statusLabel}
</span>
<Show when={row.agentUpdateCount > 0}>
<span class="inline-flex items-center rounded-full bg-amber-100 px-2 py-0.5 text-[11px] font-medium whitespace-nowrap text-amber-800 dark:bg-amber-900 dark:text-amber-200">
{row.agentUpdateCount === 1
? 'Agent update'
: `${row.agentUpdateCount} updates`}
</span>
<Show
when={props.onOpenAgentDoctor}
fallback={
<span class="inline-flex items-center rounded-full bg-amber-100 px-2 py-0.5 text-[11px] font-medium whitespace-nowrap text-amber-800 dark:bg-amber-900 dark:text-amber-200">
{row.agentUpdateCount === 1
? 'Agent update'
: `${row.agentUpdateCount} updates`}
</span>
}
>
<button
type="button"
class="inline-flex items-center rounded-full bg-amber-100 px-2 py-0.5 text-[11px] font-medium whitespace-nowrap text-amber-800 transition-colors hover:bg-amber-200 dark:bg-amber-900 dark:text-amber-200 dark:hover:bg-amber-800"
onClick={() =>
props.onOpenAgentDoctor?.(
agentConnectionIDsForInfrastructureRow(
row,
true,
),
)
}
aria-label={`Open Agent Doctor for ${row.name}`}
>
{row.agentUpdateCount === 1
? 'Agent update'
: `${row.agentUpdateCount} updates`}
</button>
</Show>
</Show>
<span
class="truncate text-[12px] text-muted/90"
@@ -1466,11 +1534,31 @@ export const InfrastructureSourceManager: Component<InfrastructureSourceManagerP
{row.statusLabel}
</span>
<Show when={row.agentUpdateCount > 0}>
<span class="inline-flex items-center rounded-full bg-amber-100 px-2 py-0.5 text-[11px] font-medium text-amber-800 dark:bg-amber-900 dark:text-amber-200">
{row.agentUpdateCount === 1
? 'Agent update'
: `${row.agentUpdateCount} agent updates`}
</span>
<Show
when={props.onOpenAgentDoctor}
fallback={
<span class="inline-flex items-center rounded-full bg-amber-100 px-2 py-0.5 text-[11px] font-medium text-amber-800 dark:bg-amber-900 dark:text-amber-200">
{row.agentUpdateCount === 1
? 'Agent update'
: `${row.agentUpdateCount} agent updates`}
</span>
}
>
<button
type="button"
class="inline-flex items-center rounded-full bg-amber-100 px-2 py-0.5 text-[11px] font-medium text-amber-800 transition-colors hover:bg-amber-200 dark:bg-amber-900 dark:text-amber-200 dark:hover:bg-amber-800"
onClick={() =>
props.onOpenAgentDoctor?.(
agentConnectionIDsForInfrastructureRow(row, true),
)
}
aria-label={`Open Agent Doctor for ${row.name}`}
>
{row.agentUpdateCount === 1
? 'Agent update'
: `${row.agentUpdateCount} agent updates`}
</button>
</Show>
</Show>
<span
class="text-[12px] text-muted/90"
@@ -33,16 +33,18 @@ import {
} from './connectionsTableModel';
import {
buildInfrastructureOnboardingPath,
buildInfrastructureAgentDoctorPath,
buildInfrastructureWorkspacePath,
deriveAgentUpdateScopeFromLocation,
deriveAgentUpdatesFromLocation,
deriveAgentDoctorScopeFromLocation,
deriveAgentDoctorFromLocation,
deriveAddStepFromLocation,
type InfrastructureAddStep,
type InfrastructurePanelStep,
} from './infrastructureWorkspaceModel';
import { collectInfrastructureAgentUpdateTargets } from './infrastructureAgentUpdateCommandsModel';
import { collectInfrastructureAgentDoctorTargets } from './infrastructureAgentUpdateCommandsModel';
import type { InfrastructurePlatformSettingsProps } from './proxmoxSettingsModel';
import { useConnectionsLedger } from './useConnectionsLedger';
import { useAgentFleetDiagnostics } from './useAgentFleetDiagnostics';
import { useConnectionRowActions } from './useConnectionRowActions';
import {
InfrastructureOperationsStateProvider,
@@ -133,15 +135,16 @@ const InfrastructureWorkspaceContent: Component<InfrastructureWorkspaceProps> =
if (readOnly()) return null;
return deriveAddStepFromLocation(location.pathname, location.search ?? '');
});
const showAgentUpdateCommands = createMemo(
const showAgentDoctor = createMemo(
() =>
!readOnly() &&
deriveAgentUpdatesFromLocation(location.pathname, location.search ?? '') &&
deriveAgentDoctorFromLocation(location.pathname, location.search ?? '') &&
routeStep() === null,
);
const agentUpdateScope = createMemo(() =>
deriveAgentUpdateScopeFromLocation(location.pathname, location.search ?? ''),
const agentDoctorScope = createMemo(() =>
deriveAgentDoctorScopeFromLocation(location.pathname, location.search ?? ''),
);
const agentDiagnostics = useAgentFleetDiagnostics(showAgentDoctor);
const activeAddType = createMemo<ConnectionType | null>(() => {
const step = routeStep();
if (!step || step === 'pick' || step === 'detect') return null;
@@ -189,12 +192,15 @@ const InfrastructureWorkspaceContent: Component<InfrastructureWorkspaceProps> =
};
const rows = createMemo(() => ledger.rows());
const agentUpdateTargets = createMemo(() =>
collectInfrastructureAgentUpdateTargets(
rows(),
updateStore.versionInfo()?.agentUpdateTargetVersion,
agentUpdateScope(),
),
const agentDoctorTargets = createMemo(() =>
collectInfrastructureAgentDoctorTargets({
rows: rows(),
connections: ledger.connections(),
diagnostics: agentDiagnostics.data().agents,
diagnosticsAvailable: agentDiagnostics.resolvedOnce() && !agentDiagnostics.error(),
targetVersion: updateStore.versionInfo()?.agentUpdateTargetVersion,
scopedAgentIds: agentDoctorScope(),
}),
);
const visibleDiscoveredNodes = createMemo(() =>
filterRepresentedDiscoveredServers(
@@ -254,8 +260,8 @@ const InfrastructureWorkspaceContent: Component<InfrastructureWorkspaceProps> =
navigateToWorkspace(Boolean(routeStep()));
};
const closeAgentUpdateCommands = () => {
navigateToWorkspace(Boolean(showAgentUpdateCommands()));
const closeAgentDoctor = () => {
navigateToWorkspace(Boolean(showAgentDoctor()));
};
const closeEditFlow = () => {
@@ -906,6 +912,12 @@ const InfrastructureWorkspaceContent: Component<InfrastructureWorkspaceProps> =
}
onOpenDiscoverySettings={readOnly() ? undefined : () => setShowDiscoverySettings(true)}
onOpenConnection={readOnly() ? undefined : (row) => setEditingRow(row)}
onOpenAgentDoctor={
readOnly()
? undefined
: (agentIds = []) =>
navigate(buildInfrastructureAgentDoctorPath(agentIds), { scroll: false })
}
onReviewDiscoveredSource={
readOnly() ? undefined : (server) => reviewDiscoveredSource(server)
}
@@ -934,11 +946,14 @@ const InfrastructureWorkspaceContent: Component<InfrastructureWorkspaceProps> =
discoverySubnetInputRef={props.discoverySubnetInputRef}
/>
<Show when={showAgentUpdateCommands()}>
<Show when={showAgentDoctor()}>
<InfrastructureAgentUpdatesDialog
isOpen={true}
targets={agentUpdateTargets()}
onClose={closeAgentUpdateCommands}
targets={agentDoctorTargets()}
diagnosticsLoading={agentDiagnostics.loading()}
diagnosticsError={agentDiagnostics.error()}
onRetryDiagnostics={() => void agentDiagnostics.reload()}
onClose={closeAgentDoctor}
/>
</Show>
@@ -1,4 +1,5 @@
import { cleanup, render, screen } from '@solidjs/testing-library';
import { Route, Router } from '@solidjs/router';
import { afterEach, describe, expect, it } from 'vitest';
import { DiagnosticsResultsPanel } from '@/components/Settings/DiagnosticsResultsPanel';
import type { DiagnosticsData } from '@/components/Settings/diagnosticsModel';
@@ -35,11 +36,18 @@ describe('DiagnosticsResultsPanel', () => {
} as DiagnosticsData;
render(() => (
<DiagnosticsResultsPanel
diagnosticsData={diagnosticsData}
loading={false}
onRunDiagnostics={() => {}}
/>
<Router>
<Route
path="/"
component={() => (
<DiagnosticsResultsPanel
diagnosticsData={diagnosticsData}
loading={false}
onRunDiagnostics={() => {}}
/>
)}
/>
</Router>
));
expect(screen.getByText('Pulse Assistant Service')).toBeInTheDocument();
@@ -204,11 +212,18 @@ describe('DiagnosticsResultsPanel', () => {
} as DiagnosticsData;
render(() => (
<DiagnosticsResultsPanel
diagnosticsData={diagnosticsData}
loading={false}
onRunDiagnostics={() => {}}
/>
<Router>
<Route
path="/"
component={() => (
<DiagnosticsResultsPanel
diagnosticsData={diagnosticsData}
loading={false}
onRunDiagnostics={() => {}}
/>
)}
/>
</Router>
));
expect(screen.queryByText('Commercial Funnel')).not.toBeInTheDocument();
@@ -223,6 +238,10 @@ describe('DiagnosticsResultsPanel', () => {
expect(screen.queryByText('TrueNAS SCALE')).not.toBeInTheDocument();
expect(screen.getByText('Docker / Podman agents')).toBeInTheDocument();
expect(screen.getByText('Agent-backed Docker / Podman monitoring')).toBeInTheDocument();
expect(screen.getByRole('link', { name: 'Open Agent Doctor' })).toHaveAttribute(
'href',
'/settings/infrastructure?agentDoctor=1',
);
expect(screen.queryByText('Container Runtime Agents')).not.toBeInTheDocument();
});
});
@@ -1,6 +1,9 @@
import { cleanup, fireEvent, render, screen, within } from '@solidjs/testing-library';
import { afterEach, describe, expect, it, vi } from 'vitest';
import { InfrastructureSourceManager } from '../InfrastructureSourceManager';
import {
InfrastructureSourceManager,
agentConnectionIDsForInfrastructureRow,
} from '../InfrastructureSourceManager';
import {
primaryRowProblem,
type FleetGovernanceSignal,
@@ -309,6 +312,37 @@ describe('InfrastructureSourceManager setup summary', () => {
expect(screen.queryByText('Rollout pending')).toBeNull();
});
it('keeps fleet version-drift update chips scoped to the affected agent', () => {
const behindAgent = connectionFixture({
id: 'agent:behind-host',
agentUpdateAvailable: false,
fleet: { versionDrift: 'behind' } as Connection['fleet'],
});
const behindRow = row({
id: behindAgent.id,
name: 'behind-host',
agentUpdateCount: 1,
connection: behindAgent,
});
const onOpenAgentDoctor = vi.fn();
expect(agentConnectionIDsForInfrastructureRow(behindRow, true)).toEqual(['agent:behind-host']);
render(() => (
<InfrastructureSourceManager
rows={() => [behindRow]}
discoveredNodes={() => []}
discoveryEnabled={false}
discoveryScanStatus={() => ({ scanning: false })}
readOnly={false}
onOpenAgentDoctor={onOpenAgentDoctor}
/>
));
fireEvent.click(screen.getByRole('button', { name: 'Open Agent Doctor for behind-host' }));
expect(onOpenAgentDoctor).toHaveBeenCalledWith(['agent:behind-host']);
});
it('still counts actionable member posture when the cluster parent is healthy', () => {
render(() => (
<InfrastructureSourceManager
@@ -143,6 +143,20 @@ vi.mock('../useConnectionsLedger', () => ({
}),
}));
vi.mock('../useAgentFleetDiagnostics', () => ({
useAgentFleetDiagnostics: () => ({
data: () => ({
generatedAt: 0,
summary: { total: 0, healthy: 0, warning: 0, critical: 0, removed: 0 },
agents: [],
}),
error: () => null,
loading: () => false,
reload: vi.fn(),
resolvedOnce: () => false,
}),
}));
vi.mock('../InfrastructureInstallerSection', () => ({
InfrastructureInstallerSection: (props: { focus?: string }) => (
<div data-testid="install-section" data-focus={props.focus}>
@@ -343,7 +357,7 @@ describe('InfrastructureWorkspace', () => {
expect(screen.queryByRole('button', { name: /^Monitor endpoint$/i })).toBeNull();
});
it('opens stale agent update commands from the canonical route', async () => {
it('opens scoped Agent Doctor from a legacy agent update route', async () => {
routeState.search = '?agentUpdates=1&agents=agent%3Aagent-zeus';
const primaryConnection = connectionFixture();
const attachedAgent = connectionFixture({
@@ -410,15 +424,16 @@ describe('InfrastructureWorkspace', () => {
renderWorkspace();
await waitFor(() => expect(screen.getByRole('dialog')).toBeInTheDocument());
expect(screen.getByText('Update Pulse Agents')).toBeInTheDocument();
expect(screen.getByText('Agent Doctor')).toBeInTheDocument();
expect(screen.getByText('zeus')).toBeInTheDocument();
expect(screen.queryByText('other')).not.toBeInTheDocument();
expect(screen.getByText(/5\.1\.34 -> 6\.0\.0-rc\.6/)).toBeInTheDocument();
expect(screen.getByText('5.1.34')).toBeInTheDocument();
expect(screen.getByText('6.0.0-rc.6')).toBeInTheDocument();
expect(
screen.getByText(/upgrade agent:agent-zeus --enable-proxmox --proxmox-type pve/),
).toBeInTheDocument();
fireEvent.click(screen.getByRole('button', { name: 'Close agent update commands' }));
fireEvent.click(screen.getByRole('button', { name: 'Close Agent Doctor' }));
expect(navigateSpy).toHaveBeenLastCalledWith('/settings/infrastructure', {
replace: true,
scroll: false,
@@ -474,12 +489,9 @@ describe('InfrastructureWorkspace', () => {
renderWorkspace();
const dialog = await screen.findByRole('dialog');
expect(
within(dialog).getByText(
'Pulse does not currently see any agents behind the target version.',
),
).toBeInTheDocument();
expect(within(dialog).queryByText('zeus')).not.toBeInTheDocument();
expect(within(dialog).getByText('zeus')).toBeInTheDocument();
expect(within(dialog).getByText('Supported target')).toBeInTheDocument();
expect(within(dialog).getAllByText('Unknown').length).toBeGreaterThan(0);
expect(within(dialog).queryByText(/upgrade agent:agent-zeus/)).not.toBeInTheDocument();
});
@@ -0,0 +1,258 @@
import { describe, expect, it } from 'vitest';
import type { AgentFleetAgentDiagnostic } from '@/api/agentDiagnostics';
import type { Connection } from '@/api/connections';
import type { InfrastructureSystemRow } from '../connectionsTableModel';
import {
collectInfrastructureAgentDoctorTargets,
diagnosticConnectionID,
resolveKnownAgentCommandPlatform,
} from '../infrastructureAgentUpdateCommandsModel';
const connectionFixture = (overrides: Partial<Connection> = {}): Connection => ({
id: 'agent:host-1',
type: 'agent',
name: 'host-1',
address: 'host-1.lab',
state: 'active',
stateReason: '',
enabled: true,
surfaces: ['host'],
scope: { host: true },
lastSeen: '2026-07-13T09:00:00Z',
lastError: null,
source: 'agent',
agentVersion: '6.1.0',
expectedAgentVersion: '6.2.0',
agentUpdateAvailable: true,
agentIdentity: { hostname: 'host-1', platform: 'ubuntu', architecture: 'amd64' },
fleet: { versionDrift: 'behind' } as Connection['fleet'],
capabilities: { supportsPause: false, supportsScope: false, supportsTest: false },
...overrides,
});
const rowFixture = (connection: Connection): InfrastructureSystemRow =>
({
id: connection.id,
ownerType: 'agent',
name: connection.name,
subtitle: 'via Pulse Agent',
source: 'agent',
host: connection.address,
coverageLabels: ['Host telemetry'],
statusLabel: 'Active',
statusClassName: 'bg-green-100',
agentUpdateCount: connection.agentUpdateAvailable ? 1 : 0,
lastActivityText: '1m ago',
fleetSignals: [],
fleetHighlights: [],
enabled: true,
canEdit: false,
canPause: false,
canRemove: true,
isAgent: true,
isCluster: false,
attachedConnections: [],
members: [],
connection,
}) as InfrastructureSystemRow;
const diagnosticFixture = (
overrides: Partial<AgentFleetAgentDiagnostic> = {},
): AgentFleetAgentDiagnostic => ({
connectionId: 'agent:host-1',
rowKey: 'agent-host-1',
id: 'host-1',
agentId: 'host-1',
name: 'host-1',
hostname: 'host-1',
types: ['host'],
status: 'warning',
version: '6.1.0',
profileId: 'profile-linux',
profileName: 'Linux servers',
profileVersion: 4,
deployedProfileVersion: 3,
reasons: [
{
code: 'agent_version_stale',
severity: 'warning',
message: 'Agent is behind the supported server version.',
evidence: ['Reported v6.1.0; target v6.2.0'],
},
],
repairActions: [
{
code: 'copy_upgrade_command',
label: 'Copy upgrade command',
description: 'Run on the affected host.',
supported: true,
scope: 'local_admin_shell',
},
],
...overrides,
});
describe('Agent Doctor model', () => {
it('enriches authoritative connection rows with structured reasons and profile drift', () => {
const connection = connectionFixture();
const targets = collectInfrastructureAgentDoctorTargets({
rows: [rowFixture(connection)],
connections: [connection],
diagnostics: [diagnosticFixture()],
diagnosticsAvailable: true,
targetVersion: '6.2.0',
});
expect(targets).toHaveLength(1);
expect(targets[0]).toMatchObject({
connectionId: 'agent:host-1',
source: 'diagnostics',
status: 'warning',
needsUpdate: true,
commandPlatform: 'linux',
profileLabel: 'Linux servers',
profileVersionLabel: 'Expected v4 · deployed v3',
});
expect(targets[0].reasons[0].code).toBe('agent_version_stale');
expect(targets[0].evidence).toContain('Reported v6.1.0; target v6.2.0');
expect(targets[0].commandBlockedReason).toBeUndefined();
});
it('falls back to ledger classification and blocks commands for unknown platforms', () => {
const connection = connectionFixture({
state: 'stale',
stateReason: 'no heartbeat in 3m',
agentIdentity: { hostname: 'host-1', platform: 'haiku' },
});
const targets = collectInfrastructureAgentDoctorTargets({
rows: [rowFixture(connection)],
connections: [connection],
diagnosticsAvailable: false,
targetVersion: '6.2.0',
});
expect(targets[0]).toMatchObject({
source: 'ledger-fallback',
status: 'warning',
needsUpdate: true,
commandPlatform: null,
});
expect(targets[0].reasons.map((reason) => reason.code)).toEqual([
'ledger_stale',
'agent_version_stale',
]);
expect(targets[0].commandBlockedReason).toContain('will not guess');
});
it('classifies eligible v6 convergence as waiting and withholds a premature manual command', () => {
const connection = connectionFixture({
agentUpdate: {
state: 'checking',
autoUpdate: true,
lastCheckedAt: '2026-07-13T09:01:00Z',
},
});
const targets = collectInfrastructureAgentDoctorTargets({
rows: [rowFixture(connection)],
connections: [connection],
diagnostics: [
diagnosticFixture({
agentUpdate: {
state: 'checking',
autoUpdate: true,
lastCheckedAt: '2026-07-13T09:01:00Z',
},
}),
],
diagnosticsAvailable: true,
targetVersion: '6.2.0',
});
expect(targets[0]).toMatchObject({
status: 'waiting',
updaterLabel: 'Checking for an automatic update',
});
expect(targets[0].commandBlockedReason).toContain('handling the update asynchronously');
expect(targets[0].evidence).toContain('Last updater check: 2026-07-13T09:01:00Z');
});
it('withholds FreeBSD and pfSense update commands until installer state is proven', () => {
const connection = connectionFixture({
agentIdentity: { hostname: 'firewall', platform: 'pfSense', architecture: 'amd64' },
});
const targets = collectInfrastructureAgentDoctorTargets({
rows: [rowFixture(connection)],
connections: [connection],
diagnostics: [diagnosticFixture()],
diagnosticsAvailable: true,
targetVersion: '6.2.0',
});
expect(targets[0].commandPlatform).toBe('freebsd');
expect(targets[0].commandBlockedReason).toContain('cannot verify saved FreeBSD or pfSense');
});
it('includes removed endpoint records only in unscoped fleet drilldown', () => {
const connection = connectionFixture({ agentUpdateAvailable: false });
const removed = diagnosticFixture({
connectionId: 'agent:removed-host',
rowKey: 'removed-host-removed-host',
id: 'removed-host',
agentId: undefined,
name: 'removed-host',
status: 'removed',
reasons: [
{
code: 'agent_removed_blocked',
severity: 'warning',
message: 'Agent is intentionally removed.',
},
],
repairActions: [
{
code: 'allow_reenroll',
label: 'Allow re-enroll',
description: 'Use the existing removed-agent action.',
supported: true,
},
],
});
const base = {
rows: [rowFixture(connection)],
connections: [connection],
diagnostics: [removed],
diagnosticsAvailable: true,
targetVersion: '6.2.0',
};
expect(collectInfrastructureAgentDoctorTargets(base).map((target) => target.status)).toContain(
'removed',
);
expect(
collectInfrastructureAgentDoctorTargets({
...base,
scopedAgentIds: ['agent:host-1'],
}).map((target) => target.status),
).not.toContain('removed');
});
it('canonicalizes compatibility diagnostics that predate connectionId', () => {
expect(
diagnosticConnectionID(
diagnosticFixture({ connectionId: undefined, agentId: 'host-1', id: 'legacy-id' }),
),
).toBe('agent:host-1');
});
it.each(['AlmaLinux', 'Proxmox VE', 'QNAP', 'Synology DSM', 'openSUSE Leap', 'Debian GNU/Linux'])(
'recognizes supported Linux platform caption %s',
(platform) => {
expect(resolveKnownAgentCommandPlatform(platform)).toBe('linux');
},
);
it('fails closed for missing and unsupported platform captions', () => {
expect(resolveKnownAgentCommandPlatform('')).toBeNull();
expect(resolveKnownAgentCommandPlatform('Haiku')).toBeNull();
});
});
@@ -1,5 +1,6 @@
import { describe, expect, it } from 'vitest';
import {
buildInfrastructureAgentDoctorPath,
buildInfrastructureAgentUpdatesPath,
buildInfrastructureOnboardingPath,
buildInfrastructureWorkspacePath,
@@ -27,11 +28,13 @@ describe('infrastructureWorkspaceModel', () => {
expect(buildInfrastructureOnboardingPath('vmware')).toBe('/settings/infrastructure?add=vmware');
});
it('builds and derives the canonical agent update command route', () => {
expect(buildInfrastructureAgentUpdatesPath()).toBe('/settings/infrastructure?agentUpdates=1');
it('builds the canonical Agent Doctor route and accepts legacy update deep links', () => {
expect(buildInfrastructureAgentDoctorPath()).toBe('/settings/infrastructure?agentDoctor=1');
expect(buildInfrastructureAgentUpdatesPath()).toBe('/settings/infrastructure?agentDoctor=1');
expect(buildInfrastructureAgentUpdatesPath(['agent:agent-delly', 'agent-pi'])).toBe(
'/settings/infrastructure?agentUpdates=1&agents=agent%3Aagent-delly&agents=agent%3Aagent-pi',
'/settings/infrastructure?agentDoctor=1&agents=agent%3Aagent-delly&agents=agent%3Aagent-pi',
);
expect(deriveAgentUpdatesFromLocation('/settings/infrastructure', '?agentDoctor=1')).toBe(true);
expect(deriveAgentUpdatesFromLocation('/settings/infrastructure', '?agentUpdates=1')).toBe(
true,
);
@@ -307,6 +307,15 @@ describe('settings architecture guardrails', () => {
);
});
it('keeps Pulse server updates separate from Agent Doctor lifecycle triage', () => {
const updatesNavBlock = settingsNavCatalogSource.match(
/id: 'system-updates',[\s\S]*?id: 'system-recovery',/,
);
expect(updatesNavBlock?.[0]).toContain("label: 'Pulse server updates'");
expect(settingsHeaderMetaSource).toContain("title: 'Pulse server updates'");
expect(settingsNavCatalogSource).not.toContain("label: 'Agent Doctor'");
});
it('keeps resource privacy route-backed instead of sidebar-promoted', () => {
expect(settingsNavCatalogSource).toMatch(
/id: 'security-data-handling',[\s\S]*label: 'Resource Privacy',[\s\S]*hideFromSidebar: true/,
@@ -118,8 +118,9 @@ describe('getSettingsHeaderMeta', () => {
it('localizes system-updates to the English baseline', () => {
expect(en['system-updates']).toEqual({
title: 'Updates',
description: 'Manage version checks, update channels, and automatic update behavior.',
title: 'Pulse server updates',
description:
'Manage Pulse server runtime version checks, update channels, and automatic updates. Agent updates stay under Infrastructure.',
});
});
@@ -415,9 +416,9 @@ describe('getSettingsHeaderMeta', () => {
it('localizes system-updates', () => {
expect(de['system-updates']).toEqual({
title: 'Updates',
title: 'Pulse-Server-Updates',
description:
'Verwalten Sie Versionspruefungen, Update-Kanaele und automatisches Update-Verhalten.',
'Verwalten Sie Versionspruefungen, Update-Kanaele und automatische Updates der Pulse-Server-Laufzeit. Agent-Updates bleiben unter Infrastruktur.',
});
});
@@ -611,9 +612,9 @@ describe('getSettingsHeaderMeta', () => {
it('localizes system-updates', () => {
expect(es['system-updates']).toEqual({
title: 'Actualizaciones',
title: 'Actualizaciones del servidor Pulse',
description:
'Administra comprobaciones de versión, canales de actualización y comportamiento de actualización automática.',
'Administra las comprobaciones de versión, los canales y las actualizaciones automáticas del servidor Pulse. Las actualizaciones de agentes permanecen en Infraestructura.',
});
});
@@ -33,7 +33,7 @@ describe('settings localization catalog', () => {
const groups = getSettingsNavGroups('de');
expect(groups[0]?.label).toBe('Infrastruktur');
expect(groups[0]?.items[0]?.label).toBe('Infrastruktur');
expect(getSettingsNavItem('system-updates', 'de')?.label).toBe('Updates');
expect(getSettingsNavItem('system-updates', 'de')?.label).toBe('Pulse-Server-Updates');
expect(getSettingsNavItem('system-ai-assistant', 'de')?.label).toBe('Assistant');
expect(getSettingsNavItem('system-ai-discovery', 'de')).toBeUndefined();
expect(getSettingsNavItem('security-data-handling', 'de')?.label).toBe('Ressourcenschutz');
@@ -1,5 +1,11 @@
import type { Connection } from '@/api/connections';
import { compareAgentVersions, formatAgentVersionDisplay } from '@/utils/agentVersion';
import type { AgentFleetAgentDiagnostic, AgentFleetDiagnosticReason } from '@/api/agentDiagnostics';
import {
compareAgentVersions,
formatAgentVersionDisplay,
parseAgentVersion,
} from '@/utils/agentVersion';
import type { AgentCommandPlatform } from '@/utils/agentInstallCommand';
import type { InfrastructureSystemRow } from './connectionsTableModel';
export type InfrastructureAgentUpdateTarget = {
@@ -12,16 +18,54 @@ export type InfrastructureAgentUpdateTarget = {
installFlags: string[];
};
export type InfrastructureAgentDoctorStatus =
'healthy' | 'waiting' | 'warning' | 'critical' | 'removed' | 'unknown';
export type InfrastructureAgentDoctorTarget = Omit<
InfrastructureAgentUpdateTarget,
'connection'
> & {
connectionId: string;
connection?: Connection;
diagnostic?: AgentFleetAgentDiagnostic;
status: InfrastructureAgentDoctorStatus;
reasons: AgentFleetDiagnosticReason[];
evidence: string[];
needsUpdate: boolean;
commandPlatform: AgentCommandPlatform | null;
commandBlockedReason?: string;
updaterLabel?: string;
profileLabel?: string;
profileVersionLabel?: string;
lastSeen?: number | string | null;
source: 'diagnostics' | 'ledger-fallback' | 'removed';
};
export interface InfrastructureAgentDoctorOptions {
rows: readonly InfrastructureSystemRow[];
connections?: readonly Connection[];
diagnostics?: readonly AgentFleetAgentDiagnostic[];
diagnosticsAvailable: boolean;
targetVersion?: string | null;
scopedAgentIds?: readonly string[];
}
const maybeAdd = (flags: Set<string>, flag: string) => {
if (flag.trim()) flags.add(flag);
};
const normalizeAgentConnectionID = (value: string | null | undefined): string => {
export const normalizeAgentConnectionID = (value: string | null | undefined): string => {
const trimmed = (value || '').trim();
if (!trimmed) return '';
return trimmed.startsWith('agent:') ? trimmed : `agent:${trimmed}`;
};
export const diagnosticConnectionID = (diagnostic: AgentFleetAgentDiagnostic): string => {
const explicit = diagnostic.connectionId?.trim();
if (explicit) return explicit;
return normalizeAgentConnectionID(diagnostic.agentId || diagnostic.id);
};
const updateInstallFlagsForRow = (row: InfrastructureSystemRow): string[] => {
const flags = new Set<string>();
@@ -68,6 +112,71 @@ const connectionNeedsUpdate = (connection: Connection, targetVersion?: string |
const expectedVersionFor = (connection: Connection, targetVersion?: string | null) =>
connection.expectedAgentVersion?.trim() || formatAgentVersionDisplay(targetVersion) || undefined;
const KNOWN_LINUX_PLATFORMS = new Set([
'alpine',
'almalinux',
'amazon',
'arch',
'centos',
'debian',
'fedora',
'gentoo',
'linux',
'manjaro',
'nixos',
'openwrt',
'opensuse',
'oracle',
'proxmox',
'qnap',
'raspbian',
'redhat',
'rhel',
'rocky',
'sles',
'suse',
'synology',
'ubuntu',
'unraid',
]);
/**
* Agent Doctor must never turn an unknown platform into an executable command.
* The general installer defaults unknown values to Linux for legacy callers;
* this stricter resolver is intentionally limited to host-local repair output.
*/
export const resolveKnownAgentCommandPlatform = (
platform?: string | null,
): AgentCommandPlatform | null => {
const normalized = platform?.trim().toLowerCase() ?? '';
if (!normalized) return null;
if (normalized.includes('windows')) return 'windows';
if (
normalized === 'darwin' ||
normalized === 'mac' ||
normalized === 'macos' ||
normalized.includes('mac os') ||
normalized.includes('os x')
) {
return 'macos';
}
if (
normalized.includes('freebsd') ||
normalized.includes('pfsense') ||
normalized.includes('opnsense')
) {
return 'freebsd';
}
if (
normalized.includes('linux') ||
KNOWN_LINUX_PLATFORMS.has(normalized) ||
Array.from(KNOWN_LINUX_PLATFORMS).some((candidate) => normalized.startsWith(`${candidate} `))
) {
return 'linux';
}
return null;
};
const pushTarget = (
targetsByID: Map<string, InfrastructureAgentUpdateTarget>,
row: InfrastructureSystemRow,
@@ -122,3 +231,389 @@ export const collectInfrastructureAgentUpdateTargets = (
)
.sort((left, right) => left.displayName.localeCompare(right.displayName));
};
type AgentConnectionBinding = {
connection: Connection;
displayName: string;
contextLabel: string;
installFlags: string[];
};
const collectAgentConnectionBindings = (
rows: readonly InfrastructureSystemRow[],
connections: readonly Connection[],
): Map<string, AgentConnectionBinding> => {
const bindings = new Map<string, AgentConnectionBinding>();
const add = (row: InfrastructureSystemRow | undefined, connection?: Connection) => {
if (!connection || connection.type !== 'agent' || bindings.has(connection.id)) return;
bindings.set(connection.id, {
connection,
displayName: connectionDisplayName(connection),
contextLabel: row ? rowContextLabel(row) : 'Machine',
installFlags: row ? updateInstallFlagsForRow(row) : [],
});
};
for (const row of rows) {
add(row, row.connection);
for (const connection of row.attachedConnections) add(row, connection);
for (const member of row.members) add(row, member.agentConnection);
}
// Rows deliberately suppress some duplicate physical hosts. The raw ledger
// remains authoritative for fleet membership, so retain unrepresented agents.
for (const connection of connections) add(undefined, connection);
return bindings;
};
const fallbackReason = (
code: string,
severity: 'warning' | 'critical',
message: string,
evidence: string[] = [],
): AgentFleetDiagnosticReason => ({ code, severity, message, evidence });
const ledgerFallbackReasons = (
connection: Connection,
needsUpdate: boolean,
): AgentFleetDiagnosticReason[] => {
const reasons: AgentFleetDiagnosticReason[] = [];
switch (connection.state) {
case 'unauthorized':
reasons.push(
fallbackReason(
'ledger_unauthorized',
'critical',
'The agent connection is unauthorized.',
connection.stateReason ? [connection.stateReason] : [],
),
);
break;
case 'unreachable':
reasons.push(
fallbackReason(
'ledger_unreachable',
'critical',
'The agent connection is unreachable.',
connection.stateReason ? [connection.stateReason] : [],
),
);
break;
case 'stale':
reasons.push(
fallbackReason(
'ledger_stale',
'warning',
'The agent has stopped reporting recently.',
connection.stateReason ? [connection.stateReason] : [],
),
);
break;
case 'pending':
reasons.push(fallbackReason('ledger_pending', 'warning', 'The agent has not reported yet.'));
break;
case 'paused':
reasons.push(fallbackReason('ledger_paused', 'warning', 'The agent is paused.'));
break;
}
if (connection.agentUpdate?.state === 'error') {
reasons.push(
fallbackReason(
'ledger_update_error',
'warning',
'The last agent update attempt failed.',
connection.agentUpdate.lastError ? [connection.agentUpdate.lastError] : [],
),
);
}
for (const module of connection.agentModules ?? []) {
if (module.enabled && module.state !== 'running') {
reasons.push(
fallbackReason(
'ledger_module_degraded',
'warning',
`${module.name} is enabled but ${module.state}.`,
module.lastError ? [module.lastError] : [],
),
);
}
}
if (needsUpdate) {
reasons.push(
fallbackReason(
'agent_version_stale',
'warning',
'This agent is behind the supported target.',
),
);
}
return reasons;
};
const fallbackStatus = (
connection: Connection,
reasons: readonly AgentFleetDiagnosticReason[],
): InfrastructureAgentDoctorStatus => {
if (reasons.some((reason) => reason.severity === 'critical')) return 'critical';
if (reasons.length > 0) return 'warning';
if (connection.state === 'active' && connection.fleet?.versionDrift === 'current') {
return 'healthy';
}
return 'unknown';
};
const evidenceFor = (
connection: Connection | undefined,
diagnostic: AgentFleetAgentDiagnostic | undefined,
): string[] => {
const evidence = new Set<string>();
if (connection?.id) evidence.add(`Connection: ${connection.id}`);
if (connection?.agentIdentity?.hostname) {
evidence.add(`Hostname: ${connection.agentIdentity.hostname}`);
}
if (connection?.agentIdentity?.platform) {
const platform = [connection.agentIdentity.platform, connection.agentIdentity.architecture]
.filter(Boolean)
.join(' / ');
evidence.add(`Platform: ${platform}`);
}
if (connection?.agentIdentity?.reportIp) {
evidence.add(`Reported IP: ${connection.agentIdentity.reportIp}`);
}
const update = connection?.agentUpdate ?? diagnostic?.agentUpdate;
if (update?.lastCheckedAt) evidence.add(`Last updater check: ${update.lastCheckedAt}`);
if (update?.lastAttemptAt) evidence.add(`Last update attempt: ${update.lastAttemptAt}`);
if (update?.lastSuccessAt) evidence.add(`Last successful update: ${update.lastSuccessAt}`);
if (diagnostic?.machineIdFingerprint) {
evidence.add(`Machine identity: ${diagnostic.machineIdFingerprint}`);
}
for (const address of diagnostic?.interfaceAddresses ?? []) {
evidence.add(`Reported interface: ${address}`);
}
for (const reason of diagnostic?.reasons ?? []) {
for (const item of reason.evidence ?? []) evidence.add(item);
}
return Array.from(evidence);
};
const updaterPresentation = (
connection: Connection,
diagnostic: AgentFleetAgentDiagnostic | undefined,
needsUpdate: boolean,
): { label?: string; waiting: boolean } => {
const update = connection.agentUpdate ?? diagnostic?.agentUpdate;
if (!update) return { waiting: false };
const state = update.state?.trim().toLowerCase();
switch (state) {
case 'updating':
return { label: 'Updating automatically', waiting: needsUpdate };
case 'checking':
return {
label: update.autoUpdate ? 'Checking for an automatic update' : 'Checking for an update',
waiting: needsUpdate && update.autoUpdate,
};
case 'update-available':
return {
label: update.autoUpdate
? 'Update queued automatically'
: 'Update available; manual action required',
waiting: needsUpdate && update.autoUpdate,
};
case 'idle':
return {
label:
needsUpdate && update.autoUpdate
? 'Waiting for the next automatic check'
: update.autoUpdate
? 'Automatic updates ready'
: 'Manual updates only',
waiting: needsUpdate && update.autoUpdate,
};
case 'disabled':
return { label: 'Automatic updates disabled', waiting: false };
case 'error':
return { label: 'Last update attempt failed', waiting: false };
default:
return state ? { label: `Updater state: ${state}`, waiting: false } : { waiting: false };
}
};
const doctorTargetFromBinding = (
binding: AgentConnectionBinding,
diagnostic: AgentFleetAgentDiagnostic | undefined,
diagnosticsAvailable: boolean,
targetVersion?: string | null,
): InfrastructureAgentDoctorTarget => {
const connection = binding.connection;
const expectedVersion = expectedVersionFor(connection, targetVersion);
const needsUpdate = connectionNeedsUpdate(connection, expectedVersion);
const fallbackReasons = ledgerFallbackReasons(connection, needsUpdate);
const reasons = diagnosticsAvailable && diagnostic ? (diagnostic.reasons ?? []) : fallbackReasons;
const updater = updaterPresentation(connection, diagnostic, needsUpdate);
let status: InfrastructureAgentDoctorStatus =
diagnosticsAvailable && diagnostic
? diagnostic.status
: fallbackStatus(connection, fallbackReasons);
// The ledger can advance one poll ahead of diagnostics. Never hide a live
// update/error signal while the structured endpoint catches up.
if (status === 'healthy' && fallbackReasons.length > 0) status = 'warning';
const nonVersionReasons = reasons.filter((reason) => reason.code !== 'agent_version_stale');
if (
updater.waiting &&
connection.state === 'active' &&
status !== 'critical' &&
nonVersionReasons.length === 0
) {
status = 'waiting';
}
const commandPlatform = resolveKnownAgentCommandPlatform(connection.agentIdentity?.platform);
let commandBlockedReason: string | undefined;
if (needsUpdate && !expectedVersion) {
commandBlockedReason = 'No supported target version is available, so Pulse will not guess.';
} else if (needsUpdate && !parseAgentVersion(expectedVersion)) {
commandBlockedReason = 'The reported target version is not a supported release version.';
} else if (needsUpdate && !commandPlatform) {
commandBlockedReason =
'The agent did not report a recognized platform, so Pulse will not guess an update command.';
} else if (needsUpdate && commandPlatform === 'freebsd') {
commandBlockedReason =
'Pulse cannot verify saved FreeBSD or pfSense installer state yet. Open Install on a host for the reviewed manual path instead of running a guessed update command.';
} else if (updater.waiting) {
commandBlockedReason =
'This eligible v6 agent is handling the update asynchronously. Wait for its updater result before using a manual command.';
}
const hasStructuredUpgradeAction = Boolean(
diagnostic?.repairActions?.some(
(action) => action.code === 'copy_upgrade_command' && action.supported,
),
);
if (
needsUpdate &&
diagnosticsAvailable &&
diagnostic &&
!hasStructuredUpgradeAction &&
!commandBlockedReason
) {
commandBlockedReason = 'The diagnostic service did not offer a supported update repair.';
}
const profileLabel =
diagnostic?.profileName?.trim() || diagnostic?.profileId?.trim() || undefined;
const profileVersionLabel = diagnostic?.profileVersion
? `Expected v${diagnostic.profileVersion} · deployed v${diagnostic.deployedProfileVersion || 0}`
: undefined;
return {
key: connection.id,
connectionId: connection.id,
connection,
diagnostic,
displayName: binding.displayName,
contextLabel: binding.contextLabel,
currentVersion: connection.agentVersion?.trim() || diagnostic?.version?.trim() || undefined,
expectedVersion,
installFlags: binding.installFlags,
status,
reasons,
evidence: evidenceFor(connection, diagnostic),
needsUpdate,
commandPlatform,
commandBlockedReason,
updaterLabel: updater.label,
profileLabel,
profileVersionLabel,
lastSeen: connection.lastSeen ?? diagnostic?.lastSeen,
source: diagnosticsAvailable && diagnostic ? 'diagnostics' : 'ledger-fallback',
};
};
const removedDoctorTarget = (
diagnostic: AgentFleetAgentDiagnostic,
): InfrastructureAgentDoctorTarget => ({
key: `removed:${diagnostic.rowKey || diagnostic.id}`,
connectionId: diagnosticConnectionID(diagnostic),
diagnostic,
displayName: diagnostic.name || diagnostic.hostname || diagnostic.id,
contextLabel: diagnostic.types?.join(' + ') || 'Removed agent',
currentVersion: diagnostic.version?.trim() || undefined,
installFlags: [],
status: 'removed',
reasons: diagnostic.reasons ?? [],
evidence: evidenceFor(undefined, diagnostic),
needsUpdate: false,
commandPlatform: null,
profileLabel: diagnostic.profileName?.trim() || diagnostic.profileId?.trim() || undefined,
profileVersionLabel: diagnostic.profileVersion
? `Expected v${diagnostic.profileVersion} · deployed v${diagnostic.deployedProfileVersion || 0}`
: undefined,
lastSeen: diagnostic.lastSeen,
source: 'removed',
});
const DOCTOR_STATUS_RANK: Record<InfrastructureAgentDoctorStatus, number> = {
critical: 6,
warning: 5,
waiting: 4,
unknown: 3,
removed: 2,
healthy: 1,
};
export const collectInfrastructureAgentDoctorTargets = ({
rows,
connections = [],
diagnostics = [],
diagnosticsAvailable,
targetVersion,
scopedAgentIds = [],
}: InfrastructureAgentDoctorOptions): InfrastructureAgentDoctorTarget[] => {
const bindings = collectAgentConnectionBindings(rows, connections);
const diagnosticsByConnectionID = new Map(
diagnostics
.filter((diagnostic) => diagnostic.status !== 'removed')
.map((diagnostic) => [diagnosticConnectionID(diagnostic), diagnostic]),
);
const scoped = new Set(scopedAgentIds.map(normalizeAgentConnectionID).filter(Boolean));
const inScope = (connectionId: string) =>
scoped.size === 0 || scoped.has(normalizeAgentConnectionID(connectionId));
const targets = Array.from(bindings.values())
.filter((binding) => inScope(binding.connection.id))
.map((binding) =>
doctorTargetFromBinding(
binding,
diagnosticsByConnectionID.get(binding.connection.id),
diagnosticsAvailable,
targetVersion,
),
);
if (diagnosticsAvailable && scoped.size === 0) {
for (const diagnostic of diagnostics) {
if (diagnostic.status === 'removed') targets.push(removedDoctorTarget(diagnostic));
}
}
return targets.sort(
(left, right) =>
DOCTOR_STATUS_RANK[right.status] - DOCTOR_STATUS_RANK[left.status] ||
left.displayName.localeCompare(right.displayName),
);
};
export const summarizeInfrastructureAgentDoctorTargets = (
targets: readonly InfrastructureAgentDoctorTarget[],
) => ({
total: targets.length,
healthy: targets.filter((target) => target.status === 'healthy').length,
waiting: targets.filter((target) => target.status === 'waiting').length,
warning: targets.filter((target) => target.status === 'warning').length,
critical: targets.filter((target) => target.status === 'critical').length,
unknown: targets.filter((target) => target.status === 'unknown').length,
removed: targets.filter((target) => target.status === 'removed').length,
});
@@ -14,6 +14,8 @@ export type InfrastructurePanelStep = 'pick' | InfrastructureAddStep;
const INFRASTRUCTURE_BASE_PATH = '/settings/infrastructure';
export const INFRASTRUCTURE_ADD_QUERY_PARAM = 'add';
export const INFRASTRUCTURE_AGENT_DOCTOR_QUERY_PARAM = 'agentDoctor';
// Legacy deep links from platform update notices and bookmarks remain valid.
export const INFRASTRUCTURE_AGENT_UPDATES_QUERY_PARAM = 'agentUpdates';
export const INFRASTRUCTURE_AGENT_UPDATE_IDS_QUERY_PARAM = 'agents';
@@ -49,11 +51,11 @@ const normalizeAgentUpdateConnectionID = (value: string | null | undefined): str
return trimmed.startsWith('agent:') ? trimmed : `agent:${trimmed}`;
};
export function buildInfrastructureAgentUpdatesPath(
export function buildInfrastructureAgentDoctorPath(
agentIds: readonly (string | null | undefined)[] = [],
): string {
const params = new URLSearchParams();
params.set(INFRASTRUCTURE_AGENT_UPDATES_QUERY_PARAM, '1');
params.set(INFRASTRUCTURE_AGENT_DOCTOR_QUERY_PARAM, '1');
const normalizedAgentIds = Array.from(
new Set(agentIds.map(normalizeAgentUpdateConnectionID).filter(Boolean) as string[]),
).sort((left, right) => left.localeCompare(right));
@@ -63,6 +65,9 @@ export function buildInfrastructureAgentUpdatesPath(
return `${INFRASTRUCTURE_BASE_PATH}?${params.toString()}`;
}
/** @deprecated Use buildInfrastructureAgentDoctorPath. */
export const buildInfrastructureAgentUpdatesPath = buildInfrastructureAgentDoctorPath;
export function buildInfrastructureOnboardingPath(step: InfrastructurePanelStep = 'agent'): string {
const params = new URLSearchParams();
params.set(INFRASTRUCTURE_ADD_QUERY_PARAM, step);
@@ -85,17 +90,20 @@ export function deriveAddStepFromLocation(
return deriveAddStepFromSearch(search);
}
export function deriveAgentUpdatesFromLocation(pathname: string, search: string): boolean {
export function deriveAgentDoctorFromLocation(pathname: string, search: string): boolean {
if (pathname !== INFRASTRUCTURE_BASE_PATH && pathname !== `${INFRASTRUCTURE_BASE_PATH}/`) {
return false;
}
const params = new URLSearchParams(search);
return params.get(INFRASTRUCTURE_AGENT_UPDATES_QUERY_PARAM) === '1';
return (
params.get(INFRASTRUCTURE_AGENT_DOCTOR_QUERY_PARAM) === '1' ||
params.get(INFRASTRUCTURE_AGENT_UPDATES_QUERY_PARAM) === '1'
);
}
export function deriveAgentUpdateScopeFromLocation(pathname: string, search: string): string[] {
if (!deriveAgentUpdatesFromLocation(pathname, search)) {
export function deriveAgentDoctorScopeFromLocation(pathname: string, search: string): string[] {
if (!deriveAgentDoctorFromLocation(pathname, search)) {
return [];
}
@@ -109,3 +117,9 @@ export function deriveAgentUpdateScopeFromLocation(pathname: string, search: str
),
).sort((left, right) => left.localeCompare(right));
}
/** @deprecated Use deriveAgentDoctorFromLocation. */
export const deriveAgentUpdatesFromLocation = deriveAgentDoctorFromLocation;
/** @deprecated Use deriveAgentDoctorScopeFromLocation. */
export const deriveAgentUpdateScopeFromLocation = deriveAgentDoctorScopeFromLocation;
@@ -25,8 +25,9 @@ export const SETTINGS_HEADER_META: SettingsHeaderMetaMap = {
description: 'Configure the public URL, CORS, embedding, and webhook network boundaries.',
},
'system-updates': {
title: 'Updates',
description: 'Manage version checks, update channels, and automatic update behavior.',
title: 'Pulse server updates',
description:
'Manage Pulse server runtime version checks, update channels, and automatic updates. Agent updates stay under Infrastructure.',
},
'system-recovery': {
title: 'Recovery',
@@ -168,7 +168,7 @@ export const SETTINGS_NAV_GROUPS: SettingsNavGroup[] = [
},
{
id: 'system-updates',
label: 'Updates',
label: 'Pulse server updates',
icon: RefreshCw,
iconProps: { strokeWidth: 2 },
saveBehavior: 'system',
@@ -0,0 +1,36 @@
import { createNonSuspendingQuery } from '@/hooks/createNonSuspendingQuery';
import { AgentDiagnosticsAPI, type AgentFleetDiagnosticsResponse } from '@/api/agentDiagnostics';
const AGENT_DIAGNOSTICS_QUERY_KEY = 'settings-agent-fleet-diagnostics';
const POLL_INTERVAL_MS = 15000;
const EMPTY_DIAGNOSTICS: AgentFleetDiagnosticsResponse = {
schemaVersion: 0,
generatedAt: 0,
summary: {
total: 0,
healthy: 0,
warning: 0,
critical: 0,
removed: 0,
},
agents: [],
};
export const useAgentFleetDiagnostics = (enabled: () => boolean) => {
const query = createNonSuspendingQuery<AgentFleetDiagnosticsResponse, string>({
source: () => (enabled() ? AGENT_DIAGNOSTICS_QUERY_KEY : null),
fetcher: () => AgentDiagnosticsAPI.getFleetDiagnostics(),
initialValue: EMPTY_DIAGNOSTICS,
cacheKey: (key) => key,
pollMs: POLL_INTERVAL_MS,
});
return {
data: query.value,
error: query.error,
loading: query.loading,
reload: query.refetch,
resolvedOnce: query.resolvedOnce,
};
};
@@ -164,7 +164,9 @@ const sourceFor = (connections: readonly Connection[]): InfrastructureSourceKind
const agentUpdateCountFor = (connections: readonly Connection[]): number =>
connections.filter(
(connection) => connection.type === 'agent' && Boolean(connection.agentUpdateAvailable),
(connection) =>
connection.type === 'agent' &&
(Boolean(connection.agentUpdateAvailable) || connection.fleet?.versionDrift === 'behind'),
).length;
const moreSevereState = (
@@ -222,6 +224,8 @@ const connectionRowSignature = (connection: Connection): string =>
agentVersion: connection.agentVersion,
expectedAgentVersion: connection.expectedAgentVersion,
agentUpdateAvailable: connection.agentUpdateAvailable,
agentUpdate: connection.agentUpdate,
agentModules: connection.agentModules,
agentIdentity: connection.agentIdentity,
hostAliases: connection.hostAliases,
fleet: connection.fleet,
@@ -177,13 +177,14 @@ export const useInfrastructureOperationsState = (
const getAgentConnectionUpgradeCommand = (
connection: Connection,
installFlags: string[] = [],
platformOverride?: AgentPlatform,
) => {
const token = resolvedCommandToken();
const url = installState.selectedAgentUrl();
const agentId = getCanonicalConnectionAgentId(connection);
const hostname = getCanonicalConnectionHostname(connection);
const commandsEnabled = Boolean(connection.agentIdentity?.commandsEnabled);
const platform = getConnectionUpgradePlatform(connection);
const platform = platformOverride ?? getConnectionUpgradePlatform(connection);
if (platform === 'windows') {
const envAssignments = [
...getPowerShellInstallProfileEnvFromFlags(installFlags),
@@ -221,8 +222,12 @@ export const useInfrastructureOperationsState = (
return withPrivilegeEscalation(command);
};
const getAgentConnectionUpgradeCommandRequiresToken = (connection: Connection) =>
getConnectionUpgradePlatform(connection) === 'windows' && installState.requiresToken();
const getAgentConnectionUpgradeCommandRequiresToken = (
connection: Connection,
platformOverride?: AgentPlatform,
) =>
(platformOverride ?? getConnectionUpgradePlatform(connection)) === 'windows' &&
installState.requiresToken();
return {
...installState,
@@ -350,7 +350,7 @@ describe('DockerPageSurface', () => {
expect(screen.getByRole('link', { name: 'Open agent upgrade commands' })).toHaveAttribute(
'href',
'/settings/infrastructure?agentUpdates=1&agents=agent%3Aagent-docker-old',
'/settings/infrastructure?agentDoctor=1&agents=agent%3Aagent-docker-old',
);
});
@@ -236,7 +236,7 @@ describe('KubernetesPageSurface contract', () => {
expect(notice).toHaveTextContent('Kubernetes nodes, workloads, services, storage');
expect(screen.getByRole('link', { name: 'Open agent upgrade commands' })).toHaveAttribute(
'href',
'/settings/infrastructure?agentUpdates=1&agents=agent%3Aagent-k8s-node-1',
'/settings/infrastructure?agentDoctor=1&agents=agent%3Aagent-k8s-node-1',
);
});
@@ -280,7 +280,7 @@ describe('KubernetesPageSurface contract', () => {
expect(notice).toHaveTextContent('Kubernetes nodes, workloads, services, storage');
expect(screen.getByRole('link', { name: 'Open agent upgrade commands' })).toHaveAttribute(
'href',
'/settings/infrastructure?agentUpdates=1&agents=agent%3Aagent-k8s-cluster',
'/settings/infrastructure?agentDoctor=1&agents=agent%3Aagent-k8s-cluster',
);
});
@@ -159,7 +159,7 @@ describe('ProxmoxPageSurface contract', () => {
);
expect(screen.getByRole('link', { name: 'Open agent upgrade commands' })).toHaveAttribute(
'href',
'/settings/infrastructure?agentUpdates=1&agents=agent%3Aagent-delly',
'/settings/infrastructure?agentDoctor=1&agents=agent%3Aagent-delly',
);
});
@@ -225,7 +225,7 @@ describe('StandalonePageSurface', () => {
);
expect(screen.getByRole('link', { name: 'Open agent upgrade commands' })).toHaveAttribute(
'href',
'/settings/infrastructure?agentUpdates=1&agents=agent%3Aagent-tower',
'/settings/infrastructure?agentDoctor=1&agents=agent%3Aagent-tower',
);
});
@@ -158,7 +158,7 @@ describe('TrueNASPageSurface contract', () => {
);
expect(screen.getByRole('link', { name: 'Open agent upgrade commands' })).toHaveAttribute(
'href',
'/settings/infrastructure?agentUpdates=1&agents=agent%3Aagent-truenas-scale',
'/settings/infrastructure?agentDoctor=1&agents=agent%3Aagent-truenas-scale',
);
});
});
@@ -159,7 +159,7 @@ describe('VmwarePageSurface contract', () => {
expect(notice).toHaveTextContent('latest in-guest telemetry and command support on this VM');
expect(screen.getByRole('link', { name: 'Open agent upgrade commands' })).toHaveAttribute(
'href',
'/settings/infrastructure?agentUpdates=1&agents=agent%3Aagent-app-01',
'/settings/infrastructure?agentDoctor=1&agents=agent%3Aagent-app-01',
);
});
+3 -3
View File
@@ -426,8 +426,8 @@ export const DE_MESSAGE_OVERRIDES = {
'Behalten Sie Ihre Systeme von ueberall im Blick und erhalten Sie Alarm-Push-Benachrichtigungen ueber die Pulse-Mobile-App — ohne Portfreigaben oder VPN.',
'settings.header.systemRelay.title': 'Remote-Zugriff',
'settings.header.systemUpdates.description':
'Verwalten Sie Versionspruefungen, Update-Kanaele und automatisches Update-Verhalten.',
'settings.header.systemUpdates.title': 'Updates',
'Verwalten Sie Versionspruefungen, Update-Kanaele und automatische Updates der Pulse-Server-Laufzeit. Agent-Updates bleiben unter Infrastruktur.',
'settings.header.systemUpdates.title': 'Pulse-Server-Updates',
'settings.header.supportDiagnostics.description':
'Fuehren Sie Zustandspruefungen aus, validieren Sie Verbindungen und exportieren Sie Troubleshooting-Snapshots.',
'settings.header.supportDiagnostics.title': 'Diagnose & Zustand',
@@ -471,7 +471,7 @@ export const DE_MESSAGE_OVERRIDES = {
'settings.nav.item.sharing': 'Freigabe',
'settings.nav.item.singleSignOn': 'Single Sign-On',
'settings.nav.item.systemLogs': 'Systemprotokolle',
'settings.nav.item.updates': 'Updates',
'settings.nav.item.updates': 'Pulse-Server-Updates',
'settings.nav.item.users': 'Benutzer',
'settings.shell.collapseSidebarLabel': 'Einstellungsnavigation einklappen',
'settings.shell.configurationLoading': 'Konfiguration wird geladen...',
+3 -3
View File
@@ -419,8 +419,8 @@ export const ES_MESSAGE_OVERRIDES = {
'Consulta tus sistemas y recibe notificaciones push de alertas desde cualquier lugar con la aplicación Pulse Mobile — sin abrir puertos ni VPN.',
'settings.header.systemRelay.title': 'Acceso remoto',
'settings.header.systemUpdates.description':
'Administra comprobaciones de versión, canales de actualización y comportamiento de actualización automática.',
'settings.header.systemUpdates.title': 'Actualizaciones',
'Administra las comprobaciones de versión, los canales y las actualizaciones automáticas del servidor Pulse. Las actualizaciones de agentes permanecen en Infraestructura.',
'settings.header.systemUpdates.title': 'Actualizaciones del servidor Pulse',
'settings.header.supportDiagnostics.description':
'Ejecuta comprobaciones de salud, valida conectividad y exporta snapshots de resolución de problemas.',
'settings.header.supportDiagnostics.title': 'Diagnóstico y salud',
@@ -464,7 +464,7 @@ export const ES_MESSAGE_OVERRIDES = {
'settings.nav.item.sharing': 'Uso compartido',
'settings.nav.item.singleSignOn': 'Inicio de sesión único',
'settings.nav.item.systemLogs': 'Logs del sistema',
'settings.nav.item.updates': 'Actualizaciones',
'settings.nav.item.updates': 'Actualizaciones del servidor Pulse',
'settings.nav.item.users': 'Usuarios',
'settings.shell.collapseSidebarLabel': 'Contraer navegación de ajustes',
'settings.shell.configurationLoading': 'Cargando configuración...',
+3 -3
View File
@@ -411,8 +411,8 @@ export const EN_MESSAGES = {
'Check on your systems and get alert push notifications anywhere with the Pulse Mobile app — no port forwarding or VPN required.',
'settings.header.systemRelay.title': 'Remote Access',
'settings.header.systemUpdates.description':
'Manage version checks, update channels, and automatic update behavior.',
'settings.header.systemUpdates.title': 'Updates',
'Manage Pulse server runtime version checks, update channels, and automatic updates. Agent updates stay under Infrastructure.',
'settings.header.systemUpdates.title': 'Pulse server updates',
'settings.header.supportDiagnostics.description':
'Run health checks, validate connectivity, and export troubleshooting snapshots.',
'settings.header.supportDiagnostics.title': 'Diagnostics & Health',
@@ -456,7 +456,7 @@ export const EN_MESSAGES = {
'settings.nav.item.sharing': 'Sharing',
'settings.nav.item.singleSignOn': 'Single Sign-On',
'settings.nav.item.systemLogs': 'System Logs',
'settings.nav.item.updates': 'Updates',
'settings.nav.item.updates': 'Pulse server updates',
'settings.nav.item.users': 'Users',
'settings.shell.collapseSidebarLabel': 'Collapse settings navigation',
'settings.shell.configurationLoading': 'Loading configuration...',
@@ -19,9 +19,10 @@ describe('updatesPresentation', () => {
it('returns canonical updates panel framing copy', () => {
expect(UPDATES_PANEL_COPY).toEqual({
title: 'Updates',
description: 'Manage version checks and automatic update preferences.',
currentVersionLabel: 'Current Version',
title: 'Pulse server updates',
description:
'Manage the Pulse server runtime. Pulse Agent updates are diagnosed under Infrastructure.',
currentVersionLabel: 'Server version',
checkNowLabel: 'Check Now',
checkingLabel: 'Checking...',
updatePreferencesTitle: 'Update Preferences',
@@ -6,9 +6,10 @@ export interface UpdateBuildBadge {
}
export const UPDATES_PANEL_COPY = {
title: 'Updates',
description: 'Manage version checks and automatic update preferences.',
currentVersionLabel: 'Current Version',
title: 'Pulse server updates',
description:
'Manage the Pulse server runtime. Pulse Agent updates are diagnosed under Infrastructure.',
currentVersionLabel: 'Server version',
checkNowLabel: 'Check Now',
checkingLabel: 'Checking...',
updatePreferencesTitle: 'Update Preferences',
+2 -1
View File
@@ -25,9 +25,10 @@ func (r *Router) handleAgentFleetDiagnostics(w http.ResponseWriter, req *http.Re
if versionInfo, err := updates.GetCurrentVersion(); err == nil && versionInfo != nil {
serverVersion = versionInfo.Version
}
agentUpdateTargetVersion := currentAgentTargetVersion()
w.Header().Set("Content-Type", "application/json")
if err := json.NewEncoder(w).Encode(monitor.GetAgentFleetDiagnostics(serverVersion, time.Now().UTC())); err != nil {
if err := json.NewEncoder(w).Encode(monitor.GetAgentFleetDiagnosticsForTarget(serverVersion, agentUpdateTargetVersion, time.Now().UTC())); err != nil {
log.Error().Err(err).Msg("Failed to serialize agent fleet diagnostics")
}
}
+10
View File
@@ -19,6 +19,7 @@ func TestHandleAgentFleetDiagnosticsReturnsFleetPayload(t *testing.T) {
ID: "agent-1",
Hostname: "node-1",
DisplayName: "Node One",
Platform: "linux",
Status: "online",
LastSeen: now.Add(-30 * time.Second),
IntervalSeconds: 30,
@@ -45,7 +46,16 @@ func TestHandleAgentFleetDiagnosticsReturnsFleetPayload(t *testing.T) {
if payload.Summary.Total != 1 || len(payload.Agents) != 1 {
t.Fatalf("expected one agent diagnostic, summary=%+v agents=%+v", payload.Summary, payload.Agents)
}
if payload.SchemaVersion != monitoring.AgentFleetDiagnosticsSchemaVersion {
t.Fatalf("schema version = %d, want %d", payload.SchemaVersion, monitoring.AgentFleetDiagnosticsSchemaVersion)
}
if payload.Agents[0].Name != "Node One" {
t.Fatalf("agent name = %q, want Node One", payload.Agents[0].Name)
}
if payload.Agents[0].ConnectionID != "agent:agent-1" || payload.Agents[0].Platform != "linux" {
t.Fatalf("agent identity = %+v", payload.Agents[0])
}
if payload.AgentUpdateTargetVersion != currentAgentTargetVersion() {
t.Fatalf("agent update target = %q, want %q", payload.AgentUpdateTargetVersion, currentAgentTargetVersion())
}
}
+13 -14
View File
@@ -11,11 +11,11 @@ import (
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/config"
"github.com/rcourtman/pulse-go-rewrite/internal/fleethealth"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/monitoring"
"github.com/rcourtman/pulse-go-rewrite/internal/platformsupport"
"github.com/rcourtman/pulse-go-rewrite/internal/remoteconfig"
pulseutils "github.com/rcourtman/pulse-go-rewrite/internal/utils"
)
// connectionStaleThreshold is the baseline "haven't heard from this connection
@@ -501,15 +501,13 @@ func buildAgentConnection(host models.Host, expectedAgentVersion string, now tim
reason := ""
currentAgentVersion := strings.TrimSpace(host.AgentVersion)
expectedAgentVersion = strings.TrimSpace(expectedAgentVersion)
updateAvailable := false
if currentAgentVersion != "" && expectedAgentVersion != "" {
updateAvailable = pulseutils.CompareVersions(currentAgentVersion, expectedAgentVersion) < 0
}
versionDrift := fleethealth.DeriveAgentVersionDrift(currentAgentVersion, expectedAgentVersion)
updateAvailable := versionDrift == fleethealth.AgentVersionBehind
agentIdentity := connectionAgentIdentityForHost(host)
switch {
case lastSeen == nil:
switch fleethealth.DeriveAgentLiveness(host.LastSeen, now, host.IntervalSeconds) {
case fleethealth.AgentLivenessPending:
state = ConnectionStatePending
case now.Sub(*lastSeen) > connectionStaleThreshold:
case fleethealth.AgentLivenessStale:
state = ConnectionStateStale
reason = fmt.Sprintf("no heartbeat in %s", now.Sub(*lastSeen).Round(time.Second))
default:
@@ -517,7 +515,7 @@ func buildAgentConnection(host models.Host, expectedAgentVersion string, now tim
}
conn := withFleetGovernance(Connection{
ID: "agent:" + host.ID,
ID: fleethealth.AgentConnectionID(host.ID),
Type: ConnectionTypeAgent,
Name: name,
Address: address,
@@ -626,13 +624,14 @@ func connectionFleetVersionDrift(conn Connection) string {
if conn.Type != ConnectionTypeAgent {
return fleetStateNotApplicable
}
if strings.TrimSpace(conn.AgentVersion) == "" || strings.TrimSpace(conn.ExpectedAgentVersion) == "" {
switch fleethealth.DeriveAgentVersionDrift(conn.AgentVersion, conn.ExpectedAgentVersion) {
case fleethealth.AgentVersionBehind:
return fleetStateBehind
case fleethealth.AgentVersionCurrent:
return fleetStateCurrent
default:
return fleetStateUnknown
}
if conn.AgentUpdateAvailable {
return fleetStateBehind
}
return fleetStateCurrent
}
func connectionFleetAdapterHealth(conn Connection) string {
+2 -2
View File
@@ -157,8 +157,8 @@ func TestBuildConnections_AgentStateFromLastSeen(t *testing.T) {
now := time.Now()
in := aggregatorInputs{
hosts: []models.Host{
{ID: "fresh", Hostname: "h1", LastSeen: now.Add(-10 * time.Second)},
{ID: "stale", Hostname: "h2", LastSeen: now.Add(-5 * time.Minute)},
{ID: "fresh", Hostname: "h1", LastSeen: now.Add(-10 * time.Second), IntervalSeconds: 30},
{ID: "stale", Hostname: "h2", LastSeen: now.Add(-5*time.Minute - time.Second), IntervalSeconds: 30},
{ID: "never", Hostname: "h3"},
},
now: now,
+10 -5
View File
@@ -19688,14 +19688,19 @@ func TestContract_AgentFleetDiagnosticsEndpointSurfacesStableShape(t *testing.T)
!strings.Contains(routerSrc, `RequireAdmin(r.config, RequireScope(config.ScopeSettingsRead, r.handleAgentFleetDiagnostics))`) {
t.Error("agent fleet diagnostics route must remain admin settings:read only")
}
if !strings.Contains(handlerSrc, "GetAgentFleetDiagnostics(serverVersion, time.Now().UTC())") {
if !strings.Contains(handlerSrc, "GetAgentFleetDiagnosticsForTarget(serverVersion, agentUpdateTargetVersion, time.Now().UTC())") {
t.Error("agent fleet diagnostics handler must delegate to the monitoring-owned read-only producer")
}
for _, required := range []string{
"GeneratedAt int64 `json:\"generatedAt\"`",
"ServerVersion string `json:\"serverVersion,omitempty\"`",
"Summary AgentFleetDiagnosticSummary `json:\"summary\"`",
"Agents []AgentFleetAgentDiagnostic `json:\"agents\"`",
"SchemaVersion int `json:\"schemaVersion\"`",
"GeneratedAt int64 `json:\"generatedAt\"`",
"ServerVersion string `json:\"serverVersion,omitempty\"`",
"AgentUpdateTargetVersion string `json:\"agentUpdateTargetVersion,omitempty\"`",
"Summary AgentFleetDiagnosticSummary `json:\"summary\"`",
"Agents []AgentFleetAgentDiagnostic `json:\"agents\"`",
"ConnectionID string `json:\"connectionId,omitempty\"`",
"AgentUpdate *AgentFleetDiagnosticUpdate `json:\"agentUpdate,omitempty\"`",
"AgentModules []AgentFleetDiagnosticModule `json:\"agentModules,omitempty\"`",
"Reasons []AgentFleetDiagnosticReason `json:\"reasons\"`",
"RepairActions []AgentFleetDiagnosticRepair `json:\"repairActions,omitempty\"`",
} {
+5 -4
View File
@@ -130,10 +130,11 @@ func TestBuildUpdateReadiness_WarnsOnStaleAgent(t *testing.T) {
readiness := buildUpdateReadiness(updateReadinessInputs{
cfg: &config.Config{APITokens: []config.APITokenRecord{*record}},
hosts: []models.Host{{
ID: "host-1",
Hostname: "host-1",
LastSeen: now.Add(-5 * time.Minute),
AgentVersion: "6.0.0-rc.6",
ID: "host-1",
Hostname: "host-1",
LastSeen: now.Add(-5*time.Minute - time.Second),
IntervalSeconds: 30,
AgentVersion: "6.0.0-rc.6",
}},
targetVersion: "v6.0.0",
plan: updates.UpdatePlan{
+91
View File
@@ -0,0 +1,91 @@
package fleethealth
import (
"strings"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/updates"
)
const (
defaultAgentReportIntervalSeconds = 30
minimumAgentStaleThreshold = 5 * time.Minute
)
type AgentLiveness string
const (
AgentLivenessActive AgentLiveness = "active"
AgentLivenessPending AgentLiveness = "pending"
AgentLivenessStale AgentLiveness = "stale"
)
type AgentVersionDrift string
const (
AgentVersionBehind AgentVersionDrift = "behind"
AgentVersionCurrent AgentVersionDrift = "current"
AgentVersionUnknown AgentVersionDrift = "unknown"
)
// AgentConnectionID returns the stable ledger identity for a host-backed
// agent. Empty agent IDs do not produce a connection identity.
func AgentConnectionID(agentID string) string {
agentID = strings.TrimSpace(agentID)
if agentID == "" {
return ""
}
return "agent:" + agentID
}
// AgentStaleThreshold returns the canonical heartbeat cutoff shared by the
// connections ledger and agent diagnostics. Five expected reports must be
// missed, with a five-minute floor to tolerate updates and service restarts.
func AgentStaleThreshold(intervalSeconds int) time.Duration {
if intervalSeconds <= 0 {
intervalSeconds = defaultAgentReportIntervalSeconds
}
threshold := time.Duration(intervalSeconds*5) * time.Second
if threshold < minimumAgentStaleThreshold {
return minimumAgentStaleThreshold
}
return threshold
}
// DeriveAgentLiveness derives agent heartbeat state without probing or
// mutating runtime state.
func DeriveAgentLiveness(lastSeen, now time.Time, intervalSeconds int) AgentLiveness {
if lastSeen.IsZero() {
return AgentLivenessPending
}
if now.IsZero() {
now = time.Now()
}
if now.Sub(lastSeen) > AgentStaleThreshold(intervalSeconds) {
return AgentLivenessStale
}
return AgentLivenessActive
}
// DeriveAgentVersionDrift compares a reported version with the canonical
// agent update target. Missing or invalid release versions remain unknown.
func DeriveAgentVersionDrift(currentVersion, targetVersion string) AgentVersionDrift {
currentVersion = strings.TrimSpace(currentVersion)
targetVersion = strings.TrimSpace(targetVersion)
if currentVersion == "" || targetVersion == "" {
return AgentVersionUnknown
}
current, err := updates.ParseVersion(currentVersion)
if err != nil {
return AgentVersionUnknown
}
target, err := updates.ParseVersion(targetVersion)
if err != nil {
return AgentVersionUnknown
}
if target.IsNewerThan(current) {
return AgentVersionBehind
}
return AgentVersionCurrent
}
+73
View File
@@ -0,0 +1,73 @@
package fleethealth
import (
"testing"
"time"
)
func TestDeriveAgentLiveness(t *testing.T) {
now := time.Date(2026, 7, 13, 12, 0, 0, 0, time.UTC)
tests := []struct {
name string
lastSeen time.Time
interval int
want AgentLiveness
}{
{name: "never reported", want: AgentLivenessPending},
{name: "recent", lastSeen: now.Add(-30 * time.Second), want: AgentLivenessActive},
{name: "at five minute floor", lastSeen: now.Add(-5 * time.Minute), interval: 30, want: AgentLivenessActive},
{name: "past five minute floor", lastSeen: now.Add(-5*time.Minute - time.Second), interval: 30, want: AgentLivenessStale},
{name: "at five intervals", lastSeen: now.Add(-10 * time.Minute), interval: 120, want: AgentLivenessActive},
{name: "past five intervals", lastSeen: now.Add(-10*time.Minute - time.Second), interval: 120, want: AgentLivenessStale},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
if got := DeriveAgentLiveness(test.lastSeen, now, test.interval); got != test.want {
t.Fatalf("DeriveAgentLiveness() = %q, want %q", got, test.want)
}
})
}
}
func TestAgentStaleThresholdUsesFloorAndReportedInterval(t *testing.T) {
if got := AgentStaleThreshold(30); got != 5*time.Minute {
t.Fatalf("AgentStaleThreshold(30) = %s, want 5m", got)
}
if got := AgentStaleThreshold(120); got != 10*time.Minute {
t.Fatalf("AgentStaleThreshold(120) = %s, want 10m", got)
}
}
func TestDeriveAgentVersionDrift(t *testing.T) {
tests := []struct {
name string
current string
target string
want AgentVersionDrift
}{
{name: "current", current: "6.2.0", target: "6.2.0", want: AgentVersionCurrent},
{name: "ahead", current: "6.3.0", target: "6.2.0", want: AgentVersionCurrent},
{name: "behind", current: "6.1.0", target: "6.2.0", want: AgentVersionBehind},
{name: "missing", current: "", target: "6.2.0", want: AgentVersionUnknown},
{name: "invalid current", current: "latest", target: "6.2.0", want: AgentVersionUnknown},
{name: "invalid target", current: "6.2.0", target: "dev", want: AgentVersionUnknown},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
if got := DeriveAgentVersionDrift(test.current, test.target); got != test.want {
t.Fatalf("DeriveAgentVersionDrift() = %q, want %q", got, test.want)
}
})
}
}
func TestAgentConnectionIDRejectsEmptyIdentity(t *testing.T) {
if got := AgentConnectionID(" "); got != "" {
t.Fatalf("AgentConnectionID(empty) = %q, want empty", got)
}
if got := AgentConnectionID(" agent-1 "); got != "agent:agent-1" {
t.Fatalf("AgentConnectionID(agent-1) = %q", got)
}
}
+429 -43
View File
@@ -1,29 +1,51 @@
package monitoring
import (
"crypto/sha256"
"encoding/hex"
"fmt"
"net/netip"
"sort"
"strconv"
"strings"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/fleethealth"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/platformsupport"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/rcourtman/pulse-go-rewrite/internal/updates"
"github.com/rs/zerolog/log"
)
const (
AgentFleetDiagnosticsSchemaVersion = 1
AgentFleetStatusHealthy = "healthy"
AgentFleetStatusWarning = "warning"
AgentFleetStatusCritical = "critical"
AgentFleetStatusRemoved = "removed"
AgentFleetReasonUpdateDisabled = "agent_update_disabled"
AgentFleetReasonUpdateFailed = "agent_update_failed"
AgentFleetReasonUpdateStateUnknown = "agent_update_state_unknown"
AgentFleetReasonUpdatePlatformUnknown = "agent_update_platform_unknown"
AgentFleetReasonUpdateStateUnverified = "agent_update_installer_state_unverified"
AgentFleetReasonModuleFailed = "agent_module_failed"
AgentFleetReasonModuleDegraded = "agent_module_degraded"
AgentFleetActionAllowReenroll = "allow_reenroll"
AgentFleetActionCopyUpgradeCommand = "copy_upgrade_command"
AgentFleetRepairModeHandoff = "handoff"
)
type AgentFleetDiagnostics struct {
GeneratedAt int64 `json:"generatedAt"`
ServerVersion string `json:"serverVersion,omitempty"`
Summary AgentFleetDiagnosticSummary `json:"summary"`
Agents []AgentFleetAgentDiagnostic `json:"agents"`
SchemaVersion int `json:"schemaVersion"`
GeneratedAt int64 `json:"generatedAt"`
ServerVersion string `json:"serverVersion,omitempty"`
AgentUpdateTargetVersion string `json:"agentUpdateTargetVersion,omitempty"`
Summary AgentFleetDiagnosticSummary `json:"summary"`
Agents []AgentFleetAgentDiagnostic `json:"agents"`
}
type AgentFleetDiagnosticSummary struct {
@@ -38,8 +60,17 @@ type AgentFleetAgentDiagnostic struct {
RowKey string `json:"rowKey"`
ID string `json:"id"`
AgentID string `json:"agentId,omitempty"`
ConnectionID string `json:"connectionId,omitempty"`
Name string `json:"name"`
Hostname string `json:"hostname,omitempty"`
Platform string `json:"platform,omitempty"`
OSName string `json:"osName,omitempty"`
OSVersion string `json:"osVersion,omitempty"`
KernelVersion string `json:"kernelVersion,omitempty"`
Architecture string `json:"architecture,omitempty"`
MachineIDFingerprint string `json:"machineIdFingerprint,omitempty"`
ReportIP string `json:"reportIp,omitempty"`
InterfaceAddresses []string `json:"interfaceAddresses,omitempty"`
Types []string `json:"types"`
Status string `json:"status"`
RawStatus string `json:"rawStatus,omitempty"`
@@ -50,10 +81,31 @@ type AgentFleetAgentDiagnostic struct {
ProfileName string `json:"profileName,omitempty"`
ProfileVersion int `json:"profileVersion,omitempty"`
DeployedProfileVersion int `json:"deployedProfileVersion,omitempty"`
AgentUpdate *AgentFleetDiagnosticUpdate `json:"agentUpdate,omitempty"`
AgentModules []AgentFleetDiagnosticModule `json:"agentModules,omitempty"`
Reasons []AgentFleetDiagnosticReason `json:"reasons"`
RepairActions []AgentFleetDiagnosticRepair `json:"repairActions,omitempty"`
}
type AgentFleetDiagnosticUpdate struct {
State string `json:"state"`
AutoUpdate bool `json:"autoUpdate"`
UpdatedFrom string `json:"updatedFrom,omitempty"`
AvailableVersion string `json:"availableVersion,omitempty"`
LastCheckedAt *time.Time `json:"lastCheckedAt,omitempty"`
LastAttemptAt *time.Time `json:"lastAttemptAt,omitempty"`
LastSuccessAt *time.Time `json:"lastSuccessAt,omitempty"`
LastError string `json:"lastError,omitempty"`
}
type AgentFleetDiagnosticModule struct {
Name string `json:"name"`
Enabled bool `json:"enabled"`
State string `json:"state"`
LastError string `json:"lastError,omitempty"`
UpdatedAt time.Time `json:"updatedAt"`
}
type AgentFleetDiagnosticReason struct {
Code string `json:"code"`
Severity string `json:"severity"`
@@ -66,6 +118,8 @@ type AgentFleetDiagnosticRepair struct {
Label string `json:"label"`
Description string `json:"description"`
Supported bool `json:"supported"`
Mode string `json:"mode"`
Platform string `json:"platform,omitempty"`
Scope string `json:"scope,omitempty"`
}
@@ -88,17 +142,26 @@ type agentFleetSubject struct {
removedAt time.Time
}
// GetAgentFleetDiagnostics returns a read-only fleet health view derived from
// reported agent state, server version, and existing profile deployment state.
// GetAgentFleetDiagnostics preserves the original call contract for callers
// where the server version is also the agent update target.
func (m *Monitor) GetAgentFleetDiagnostics(serverVersion string, now time.Time) AgentFleetDiagnostics {
return m.GetAgentFleetDiagnosticsForTarget(serverVersion, serverVersion, now)
}
// GetAgentFleetDiagnosticsForTarget returns a read-only fleet health view
// derived from reported state, the canonical agent update target, and existing
// profile deployment state.
func (m *Monitor) GetAgentFleetDiagnosticsForTarget(serverVersion, agentUpdateTargetVersion string, now time.Time) AgentFleetDiagnostics {
if now.IsZero() {
now = time.Now().UTC()
}
now = now.UTC()
out := AgentFleetDiagnostics{
GeneratedAt: now.UnixMilli(),
ServerVersion: strings.TrimSpace(serverVersion),
SchemaVersion: AgentFleetDiagnosticsSchemaVersion,
GeneratedAt: now.UnixMilli(),
ServerVersion: strings.TrimSpace(serverVersion),
AgentUpdateTargetVersion: strings.TrimSpace(agentUpdateTargetVersion),
}
if m == nil || m.state == nil {
return out
@@ -112,7 +175,7 @@ func (m *Monitor) GetAgentFleetDiagnostics(serverVersion string, now time.Time)
subjects := buildAgentFleetSubjects(state)
for i := range subjects {
diagnostic := diagnoseAgentFleetSubject(subjects[i], state, out.ServerVersion, now, profileByID, assignmentByAgent, deploymentByAgentProfile)
diagnostic := diagnoseAgentFleetSubject(subjects[i], state, out.AgentUpdateTargetVersion, now, profileByID, assignmentByAgent, deploymentByAgentProfile)
out.Agents = append(out.Agents, diagnostic)
}
@@ -290,16 +353,28 @@ func diagnoseAgentFleetSubject(
assignmentByAgent map[string]models.AgentProfileAssignment,
deploymentByAgentProfile map[string]models.ProfileDeploymentStatus,
) AgentFleetAgentDiagnostic {
identity := agentFleetIdentityForSubject(subject)
result := AgentFleetAgentDiagnostic{
RowKey: subject.rowKey,
ID: subject.id,
AgentID: subject.agentID,
Name: firstNonEmpty(subject.name, subject.hostname, subject.id),
Hostname: subject.hostname,
Types: sortedAgentTypes(subject.types),
RawStatus: subject.rawStatus,
IntervalSeconds: subject.intervalSeconds,
Version: subject.version,
RowKey: subject.rowKey,
ID: subject.id,
AgentID: subject.agentID,
ConnectionID: identity.connectionID,
Name: firstNonEmpty(subject.name, subject.hostname, subject.id),
Hostname: subject.hostname,
Platform: identity.platform,
OSName: identity.osName,
OSVersion: identity.osVersion,
KernelVersion: identity.kernelVersion,
Architecture: identity.architecture,
MachineIDFingerprint: identity.machineIDFingerprint,
ReportIP: identity.reportIP,
InterfaceAddresses: identity.interfaceAddresses,
Types: sortedAgentTypes(subject.types),
RawStatus: subject.rawStatus,
IntervalSeconds: subject.intervalSeconds,
Version: subject.version,
AgentUpdate: agentFleetUpdateForSubject(subject),
AgentModules: agentFleetModulesForSubject(subject),
}
if !subject.lastSeen.IsZero() {
result.LastSeen = subject.lastSeen.UnixMilli()
@@ -319,10 +394,11 @@ func diagnoseAgentFleetSubject(
},
})
result.RepairActions = append(result.RepairActions, AgentFleetDiagnosticRepair{
Code: "allow_reenroll",
Code: AgentFleetActionAllowReenroll,
Label: "Allow re-enroll",
Description: "Uses the existing allow re-enroll action for removed agents.",
Supported: true,
Mode: AgentFleetRepairModeHandoff,
Scope: "settings:write",
})
return result
@@ -330,6 +406,8 @@ func diagnoseAgentFleetSubject(
result.Reasons = append(result.Reasons, diagnoseAgentConnectivity(subject, now)...)
result.Reasons = append(result.Reasons, diagnoseAgentVersion(subject, serverVersion)...)
result.Reasons = append(result.Reasons, diagnoseAgentUpdate(subject, serverVersion)...)
result.Reasons = append(result.Reasons, diagnoseAgentModules(subject)...)
result.Reasons = append(result.Reasons, diagnoseAgentIdentitySplit(subject, state)...)
assignment, hasAssignment := findAgentAssignment(subject, assignmentByAgent)
@@ -359,11 +437,28 @@ func diagnoseAgentFleetSubject(
for _, reason := range result.Reasons {
if reason.Code == "agent_version_stale" {
platform, supported := safeAgentUpdatePlatform(subject)
if !supported {
result.Reasons = append(result.Reasons, AgentFleetDiagnosticReason{
Code: AgentFleetReasonUpdatePlatformUnknown,
Severity: AgentFleetStatusWarning,
Message: "Pulse cannot safely choose an agent installer command because the reported platform is missing or unsupported.",
})
} else if platform == platformsupport.RuntimePlatformFreeBSD {
supported = false
result.Reasons = append(result.Reasons, AgentFleetDiagnosticReason{
Code: AgentFleetReasonUpdateStateUnverified,
Severity: AgentFleetStatusWarning,
Message: "Pulse cannot verify the saved FreeBSD or pfSense installer state required for a safe in-place update.",
})
}
result.RepairActions = append(result.RepairActions, AgentFleetDiagnosticRepair{
Code: "copy_upgrade_command",
Code: AgentFleetActionCopyUpgradeCommand,
Label: "Copy upgrade command",
Description: "Uses the existing installer command from Settings -> Agents; no remote command is queued.",
Supported: true,
Supported: supported,
Mode: AgentFleetRepairModeHandoff,
Platform: platform,
Scope: "local_admin_shell",
})
break
@@ -384,25 +479,21 @@ func diagnoseAgentConnectivity(subject agentFleetSubject, now time.Time) []Agent
if interval <= 0 {
interval = 30
}
staleAfter := time.Duration(interval*5) * time.Second
if staleAfter < 5*time.Minute {
staleAfter = 5 * time.Minute
}
if subject.lastSeen.IsZero() {
staleThreshold := fleethealth.AgentStaleThreshold(subject.intervalSeconds)
switch fleethealth.DeriveAgentLiveness(subject.lastSeen, now, subject.intervalSeconds) {
case fleethealth.AgentLivenessPending:
return append(reasons, AgentFleetDiagnosticReason{
Code: "agent_never_reported",
Severity: AgentFleetStatusCritical,
Message: "This agent has no last-seen timestamp, so Pulse cannot confirm it is reporting.",
})
}
age := now.Sub(subject.lastSeen)
if age > staleAfter {
case fleethealth.AgentLivenessStale:
age := now.Sub(subject.lastSeen)
reasons = append(reasons, AgentFleetDiagnosticReason{
Code: "agent_disconnected",
Severity: AgentFleetStatusCritical,
Message: fmt.Sprintf("No report has arrived for %s; this is beyond the %s stale threshold for a %ds reporting interval.", roundDuration(age), roundDuration(staleAfter), interval),
Message: fmt.Sprintf("No report has arrived for %s; this is beyond the canonical %s agent stale threshold.", roundDuration(age), roundDuration(staleThreshold)),
Evidence: []string{
"Last seen: " + subject.lastSeen.UTC().Format(time.RFC3339),
fmt.Sprintf("Expected report interval: %ds", interval),
@@ -421,7 +512,7 @@ func diagnoseAgentConnectivity(subject agentFleetSubject, now time.Time) []Agent
return reasons
}
func diagnoseAgentVersion(subject agentFleetSubject, serverVersion string) []AgentFleetDiagnosticReason {
func diagnoseAgentVersion(subject agentFleetSubject, targetVersion string) []AgentFleetDiagnosticReason {
if subject.removed {
return nil
}
@@ -434,32 +525,30 @@ func diagnoseAgentVersion(subject agentFleetSubject, serverVersion string) []Age
}}
}
serverVersion = strings.TrimSpace(serverVersion)
if serverVersion == "" || strings.EqualFold(serverVersion, "dev") {
targetVersion = strings.TrimSpace(targetVersion)
if targetVersion == "" || strings.EqualFold(targetVersion, "dev") {
return nil
}
serverParsed, err := updates.ParseVersion(serverVersion)
if err != nil {
if _, err := updates.ParseVersion(targetVersion); err != nil {
return nil
}
agentParsed, err := updates.ParseVersion(agentVersion)
if err != nil {
if _, err := updates.ParseVersion(agentVersion); err != nil {
return []AgentFleetDiagnosticReason{{
Code: "agent_version_unparseable",
Severity: AgentFleetStatusWarning,
Message: fmt.Sprintf("The agent reported version %q, which cannot be compared with server version %q.", agentVersion, serverVersion),
Message: fmt.Sprintf("The agent reported version %q, which cannot be compared with update target %q.", agentVersion, targetVersion),
}}
}
if serverParsed.IsNewerThan(agentParsed) {
if fleethealth.DeriveAgentVersionDrift(agentVersion, targetVersion) == fleethealth.AgentVersionBehind {
return []AgentFleetDiagnosticReason{{
Code: "agent_version_stale",
Severity: AgentFleetStatusWarning,
Message: fmt.Sprintf("Agent version %s is older than the Pulse server version %s.", agentVersion, serverVersion),
Message: fmt.Sprintf("Agent version %s is older than the agent update target %s.", agentVersion, targetVersion),
Evidence: []string{
"Agent version: " + agentVersion,
"Server version: " + serverVersion,
"Agent update target: " + targetVersion,
},
}}
}
@@ -467,6 +556,303 @@ func diagnoseAgentVersion(subject agentFleetSubject, serverVersion string) []Age
return nil
}
func diagnoseAgentUpdate(subject agentFleetSubject, targetVersion string) []AgentFleetDiagnosticReason {
if subject.host == nil || subject.host.AgentUpdate == nil {
return nil
}
update := subject.host.AgentUpdate
state := strings.ToLower(strings.TrimSpace(update.State))
evidence := updateStatusEvidence(update)
switch state {
case "", "idle", "checking", "update-available", "updating":
return nil
case "disabled":
if fleethealth.DeriveAgentVersionDrift(subject.version, targetVersion) != fleethealth.AgentVersionBehind {
return nil
}
return []AgentFleetDiagnosticReason{{
Code: AgentFleetReasonUpdateDisabled,
Severity: AgentFleetStatusWarning,
Message: "The agent is behind the update target and its automatic updater is disabled.",
Evidence: evidence,
}}
case "error":
return []AgentFleetDiagnosticReason{{
Code: AgentFleetReasonUpdateFailed,
Severity: AgentFleetStatusWarning,
Message: "The agent's most recent self-update check or attempt failed.",
Evidence: evidence,
}}
default:
return []AgentFleetDiagnosticReason{{
Code: AgentFleetReasonUpdateStateUnknown,
Severity: AgentFleetStatusWarning,
Message: fmt.Sprintf("The agent reported an unrecognized updater state %q.", update.State),
Evidence: evidence,
}}
}
}
func diagnoseAgentModules(subject agentFleetSubject) []AgentFleetDiagnosticReason {
if subject.host == nil {
return nil
}
reasons := make([]AgentFleetDiagnosticReason, 0)
for _, module := range subject.host.AgentModules {
if !module.Enabled {
continue
}
state := strings.ToLower(strings.TrimSpace(module.State))
if state == "running" {
continue
}
severity := AgentFleetStatusWarning
code := AgentFleetReasonModuleDegraded
message := fmt.Sprintf("Enabled agent module %q is not running.", strings.TrimSpace(module.Name))
if state == "error" || state == "failed" {
severity = AgentFleetStatusCritical
code = AgentFleetReasonModuleFailed
message = fmt.Sprintf("Enabled agent module %q failed.", strings.TrimSpace(module.Name))
}
reasons = append(reasons, AgentFleetDiagnosticReason{
Code: code,
Severity: severity,
Message: message,
Evidence: moduleStatusEvidence(module),
})
}
return reasons
}
type agentFleetIdentityEvidence struct {
connectionID string
platform string
osName string
osVersion string
kernelVersion string
architecture string
machineIDFingerprint string
reportIP string
interfaceAddresses []string
}
func agentFleetIdentityForSubject(subject agentFleetSubject) agentFleetIdentityEvidence {
identity := agentFleetIdentityEvidence{}
var machineID string
var interfaces []models.HostNetworkInterface
if subject.host != nil {
host := subject.host
identity.connectionID = fleethealth.AgentConnectionID(host.ID)
identity.platform, _ = safeAgentUpdatePlatform(subject)
if identity.platform == "" {
identity.platform = platformsupport.NormalizeAgentReportedPlatform(host.Platform)
}
identity.osName = strings.TrimSpace(host.OSName)
identity.osVersion = strings.TrimSpace(host.OSVersion)
identity.kernelVersion = strings.TrimSpace(host.KernelVersion)
identity.architecture = strings.TrimSpace(host.Architecture)
identity.reportIP = safeDiagnosticIPAddress(host.ReportIP)
machineID = host.MachineID
interfaces = append(interfaces, host.NetworkInterfaces...)
}
if subject.docker != nil {
docker := subject.docker
if identity.platform == "" {
identity.platform, _ = safeAgentUpdatePlatform(subject)
if identity.platform == "" {
identity.platform = platformsupport.NormalizeAgentReportedPlatform(docker.OS)
}
}
identity.osName = firstNonEmpty(identity.osName, docker.OS)
identity.kernelVersion = firstNonEmpty(identity.kernelVersion, docker.KernelVersion)
identity.architecture = firstNonEmpty(identity.architecture, docker.Architecture)
machineID = firstNonEmpty(machineID, docker.MachineID)
interfaces = append(interfaces, docker.NetworkInterfaces...)
}
identity.machineIDFingerprint = machineIDFingerprint(machineID)
identity.interfaceAddresses = safeDiagnosticInterfaceAddresses(interfaces)
return identity
}
func agentFleetUpdateForSubject(subject agentFleetSubject) *AgentFleetDiagnosticUpdate {
if subject.host == nil || subject.host.AgentUpdate == nil {
return nil
}
update := subject.host.AgentUpdate
return &AgentFleetDiagnosticUpdate{
State: strings.TrimSpace(update.State),
AutoUpdate: update.AutoUpdate,
UpdatedFrom: strings.TrimSpace(update.UpdatedFrom),
AvailableVersion: strings.TrimSpace(update.AvailableVersion),
LastCheckedAt: cloneFleetDiagnosticTime(update.LastCheckedAt),
LastAttemptAt: cloneFleetDiagnosticTime(update.LastAttemptAt),
LastSuccessAt: cloneFleetDiagnosticTime(update.LastSuccessAt),
LastError: safeDiagnosticError(update.LastError),
}
}
func agentFleetModulesForSubject(subject agentFleetSubject) []AgentFleetDiagnosticModule {
if subject.host == nil || len(subject.host.AgentModules) == 0 {
return nil
}
modules := make([]AgentFleetDiagnosticModule, 0, len(subject.host.AgentModules))
for _, module := range subject.host.AgentModules {
modules = append(modules, AgentFleetDiagnosticModule{
Name: strings.TrimSpace(module.Name),
Enabled: module.Enabled,
State: strings.TrimSpace(module.State),
LastError: safeDiagnosticError(module.LastError),
UpdatedAt: module.UpdatedAt.UTC(),
})
}
sort.Slice(modules, func(i, j int) bool {
return strings.ToLower(modules[i].Name) < strings.ToLower(modules[j].Name)
})
return modules
}
func safeAgentUpdatePlatform(subject agentFleetSubject) (string, bool) {
values := make([]string, 0, 3)
if subject.host != nil {
values = append(values, subject.host.Platform, subject.host.OSName)
}
if subject.docker != nil {
values = append(values, subject.docker.OS)
}
for _, value := range values {
appliance := strings.ToLower(strings.TrimSpace(value))
if strings.Contains(appliance, "pfsense") || strings.Contains(appliance, "opnsense") {
return platformsupport.RuntimePlatformFreeBSD, true
}
normalized := platformsupport.NormalizeAgentReportedPlatform(value)
switch normalized {
case platformsupport.RuntimePlatformWindows,
platformsupport.RuntimePlatformMacOS,
platformsupport.RuntimePlatformFreeBSD,
platformsupport.RuntimePlatformLinux:
return normalized, true
}
if knownLinuxAgentPlatform(normalized) {
return platformsupport.RuntimePlatformLinux, true
}
}
return "", false
}
func knownLinuxAgentPlatform(value string) bool {
value = strings.ToLower(strings.TrimSpace(value))
for _, token := range []string{
"almalinux", "alpine", "amazon linux", "arch", "centos", "debian",
"fedora", "gentoo", "linux", "nixos", "opensuse", "oracle linux",
"proxmox", "qnap", "raspbian", "red hat", "rhel", "rocky", "suse",
"synology", "ubuntu", "unraid",
} {
if value == token || strings.Contains(value, token+" ") || strings.Contains(value, token+"-") {
return true
}
}
return false
}
func machineIDFingerprint(machineID string) string {
machineID = strings.TrimSpace(machineID)
if machineID == "" {
return ""
}
digest := sha256.Sum256([]byte(machineID))
return "sha256:" + hex.EncodeToString(digest[:8])
}
func safeDiagnosticInterfaceAddresses(interfaces []models.HostNetworkInterface) []string {
const maxAddresses = 32
set := make(map[string]struct{})
for _, iface := range interfaces {
for _, value := range iface.Addresses {
if normalized := safeDiagnosticIPAddress(value); normalized != "" {
set[normalized] = struct{}{}
}
}
}
addresses := make([]string, 0, len(set))
for value := range set {
addresses = append(addresses, value)
}
sort.Strings(addresses)
if len(addresses) > maxAddresses {
addresses = addresses[:maxAddresses]
}
return addresses
}
func safeDiagnosticIPAddress(value string) string {
value = strings.TrimSpace(value)
if value == "" {
return ""
}
if address, err := netip.ParseAddr(value); err == nil {
return address.String()
}
if prefix, err := netip.ParsePrefix(value); err == nil {
return prefix.String()
}
return ""
}
func updateStatusEvidence(update *models.AgentUpdateStatus) []string {
if update == nil {
return nil
}
evidence := nonEmptyStrings("Updater state: " + strings.TrimSpace(update.State))
if update.LastCheckedAt != nil {
evidence = append(evidence, "Last checked: "+update.LastCheckedAt.UTC().Format(time.RFC3339))
}
if update.LastAttemptAt != nil {
evidence = append(evidence, "Last attempted: "+update.LastAttemptAt.UTC().Format(time.RFC3339))
}
if lastError := safeDiagnosticError(update.LastError); lastError != "" {
evidence = append(evidence, "Last error: "+lastError)
}
return evidence
}
func moduleStatusEvidence(module models.AgentModuleStatus) []string {
evidence := nonEmptyStrings(
"Module: "+strings.TrimSpace(module.Name),
"Module state: "+strings.TrimSpace(module.State),
)
if !module.UpdatedAt.IsZero() {
evidence = append(evidence, "Updated at: "+module.UpdatedAt.UTC().Format(time.RFC3339))
}
if lastError := safeDiagnosticError(module.LastError); lastError != "" {
evidence = append(evidence, "Last error: "+lastError)
}
return evidence
}
func safeDiagnosticError(value string) string {
const maxErrorLength = 512
value = strings.TrimSpace(unifiedresources.RedactAuditText(value))
runes := []rune(value)
if len(runes) > maxErrorLength {
value = string(runes[:maxErrorLength]) + "..."
}
return value
}
func cloneFleetDiagnosticTime(value *time.Time) *time.Time {
if value == nil || value.IsZero() {
return nil
}
cloned := value.UTC()
return &cloned
}
func diagnoseAgentIdentitySplit(subject agentFleetSubject, state models.StateSnapshot) []AgentFleetDiagnosticReason {
if subject.hostname == "" || subject.removed {
return nil
@@ -618,7 +1004,7 @@ func identitySplitReason(peerType, peerID, peerAgentID, peerTokenID string) Agen
evidence = append(evidence, "Peer agent ID: "+peerAgentID)
}
if peerTokenID != "" {
evidence = append(evidence, "Peer token ID: "+peerTokenID)
evidence = append(evidence, "Peer shares the reporting token binding")
}
return AgentFleetDiagnosticReason{
Code: "agent_identity_split",
@@ -1,11 +1,14 @@
package monitoring
import (
"reflect"
"strings"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/config"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/platformsupport"
)
func TestAgentFleetDiagnosticsDetectsStaleAgentVersion(t *testing.T) {
@@ -15,6 +18,7 @@ func TestAgentFleetDiagnosticsDetectsStaleAgentVersion(t *testing.T) {
ID: "agent-1",
Hostname: "pve-1",
DisplayName: "PVE 1",
Platform: "linux",
Status: "online",
LastSeen: now.Add(-30 * time.Second),
IntervalSeconds: 30,
@@ -33,6 +37,145 @@ func TestAgentFleetDiagnosticsDetectsStaleAgentVersion(t *testing.T) {
}
}
func TestAgentFleetDiagnosticsSurfacesReportedUpdateModuleAndIdentityEvidence(t *testing.T) {
now := time.Date(2026, 7, 13, 12, 0, 0, 0, time.UTC)
checkedAt := now.Add(-time.Minute)
monitor := newAgentFleetDoctorTestMonitor(t)
monitor.state.UpsertHost(models.Host{
ID: "agent-identity",
Hostname: "windows-node",
DisplayName: "Windows Node",
Platform: "Microsoft Windows 11 Pro",
OSName: "Windows 11",
OSVersion: "24H2",
KernelVersion: "10.0.26100",
Architecture: "amd64",
MachineID: "raw-machine-id-must-not-leak",
ReportIP: "192.0.2.10",
Status: "online",
LastSeen: now.Add(-30 * time.Second),
IntervalSeconds: 30,
AgentVersion: "6.2.0",
NetworkInterfaces: []models.HostNetworkInterface{{
Name: "Ethernet",
Addresses: []string{"192.0.2.10/24", "not-an-ip"},
}},
AgentUpdate: &models.AgentUpdateStatus{
State: "error",
AutoUpdate: true,
LastCheckedAt: &checkedAt,
LastError: "download failed token=must-not-leak",
},
AgentModules: []models.AgentModuleStatus{{
Name: "docker",
Enabled: true,
State: "failed",
LastError: "socket unavailable password=must-not-leak",
UpdatedAt: checkedAt,
}},
})
before := monitor.GetState()
diagnostics := monitor.GetAgentFleetDiagnosticsForTarget("6.2.0-pro", "6.2.0", now)
agent := requireAgentDiagnostic(t, diagnostics, "agent-agent-identity")
requireReasonCode(t, agent, AgentFleetReasonUpdateFailed)
requireReasonCode(t, agent, AgentFleetReasonModuleFailed)
if diagnostics.ServerVersion != "6.2.0-pro" || diagnostics.AgentUpdateTargetVersion != "6.2.0" {
t.Fatalf("version identities = server %q target %q", diagnostics.ServerVersion, diagnostics.AgentUpdateTargetVersion)
}
if agent.ConnectionID != "agent:agent-identity" || agent.Platform != "windows" || agent.Architecture != "amd64" {
t.Fatalf("canonical identity = %+v", agent)
}
if agent.MachineIDFingerprint == "" || strings.Contains(agent.MachineIDFingerprint, "raw-machine-id") {
t.Fatalf("unsafe machine identity fingerprint %q", agent.MachineIDFingerprint)
}
if agent.ReportIP != "192.0.2.10" || !reflect.DeepEqual(agent.InterfaceAddresses, []string{"192.0.2.10/24"}) {
t.Fatalf("safe IP evidence = report %q interfaces %#v", agent.ReportIP, agent.InterfaceAddresses)
}
if agent.AgentUpdate == nil || agent.AgentUpdate.LastCheckedAt == nil || strings.Contains(agent.AgentUpdate.LastError, "must-not-leak") {
t.Fatalf("update evidence was missing or unsafe: %+v", agent.AgentUpdate)
}
if len(agent.AgentModules) != 1 || strings.Contains(agent.AgentModules[0].LastError, "must-not-leak") {
t.Fatalf("module evidence was missing or unsafe: %+v", agent.AgentModules)
}
if after := monitor.GetState(); !reflect.DeepEqual(before, after) {
t.Fatal("fleet diagnostics mutated monitor state")
}
}
func TestAgentFleetDiagnosticsUpdaterReasonCodes(t *testing.T) {
now := time.Date(2026, 7, 13, 12, 0, 0, 0, time.UTC)
tests := []struct {
name string
version string
updaterState string
wantReason string
}{
{name: "disabled while behind", version: "6.1.0", updaterState: "disabled", wantReason: AgentFleetReasonUpdateDisabled},
{name: "failed", version: "6.2.0", updaterState: "error", wantReason: AgentFleetReasonUpdateFailed},
{name: "unknown state", version: "6.2.0", updaterState: "paused-by-policy", wantReason: AgentFleetReasonUpdateStateUnknown},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
monitor := newAgentFleetDoctorTestMonitor(t)
monitor.state.UpsertHost(models.Host{
ID: "agent-1",
Hostname: "node-1",
Platform: "linux",
Status: "online",
LastSeen: now,
AgentVersion: test.version,
AgentUpdate: &models.AgentUpdateStatus{State: test.updaterState},
})
agent := requireAgentDiagnostic(t, monitor.GetAgentFleetDiagnostics("6.2.0", now), "agent-agent-1")
requireReasonCode(t, agent, test.wantReason)
})
}
}
func TestAgentFleetDiagnosticsDoesNotOfferUpgradeCommandForUnknownPlatform(t *testing.T) {
now := time.Date(2026, 7, 13, 12, 0, 0, 0, time.UTC)
monitor := newAgentFleetDoctorTestMonitor(t)
monitor.state.UpsertHost(models.Host{
ID: "agent-unknown-platform",
Hostname: "unknown-node",
Platform: "plan9",
Status: "online",
LastSeen: now,
AgentVersion: "6.1.0",
})
agent := requireAgentDiagnostic(t, monitor.GetAgentFleetDiagnostics("6.2.0", now), "agent-agent-unknown-platform")
requireReasonCode(t, agent, AgentFleetReasonUpdatePlatformUnknown)
repair := requireRepairCode(t, agent, AgentFleetActionCopyUpgradeCommand)
if repair.Supported || repair.Platform != "" || repair.Mode != AgentFleetRepairModeHandoff {
t.Fatalf("unsafe platform repair = %+v", repair)
}
}
func TestAgentFleetDiagnosticsDoesNotOfferUpgradeCommandForUnverifiedFreeBSDState(t *testing.T) {
now := time.Date(2026, 7, 13, 12, 0, 0, 0, time.UTC)
monitor := newAgentFleetDoctorTestMonitor(t)
monitor.state.UpsertHost(models.Host{
ID: "agent-pfsense",
Hostname: "firewall",
Platform: "pfSense",
Status: "online",
LastSeen: now,
AgentVersion: "6.1.0",
})
agent := requireAgentDiagnostic(t, monitor.GetAgentFleetDiagnostics("6.2.0", now), "agent-agent-pfsense")
requireReasonCode(t, agent, AgentFleetReasonUpdateStateUnverified)
repair := requireRepairCode(t, agent, AgentFleetActionCopyUpgradeCommand)
if repair.Supported || repair.Platform != platformsupport.RuntimePlatformFreeBSD {
t.Fatalf("unverified FreeBSD repair = %+v", repair)
}
}
func TestAgentFleetDiagnosticsDetectsMissingDockerTelemetryFromProfile(t *testing.T) {
now := time.Date(2026, 6, 29, 12, 0, 0, 0, time.UTC)
monitor := newAgentFleetDoctorTestMonitor(t)
@@ -188,6 +331,17 @@ func hasSupportedRepair(agent AgentFleetAgentDiagnostic, code string) bool {
return false
}
func requireRepairCode(t *testing.T, agent AgentFleetAgentDiagnostic, code string) AgentFleetDiagnosticRepair {
t.Helper()
for _, repair := range agent.RepairActions {
if repair.Code == code {
return repair
}
}
t.Fatalf("repair %q not found in %#v", code, agent.RepairActions)
return AgentFleetDiagnosticRepair{}
}
func containsString(values []string, want string) bool {
for _, value := range values {
if value == want {
@@ -1,6 +1,7 @@
package monitoring
import (
"strings"
"testing"
"time"
)
@@ -19,8 +20,9 @@ import (
// Tests are prefixed with BranchCov so `-run BranchCov` selects them.
// identitySplitReason always emits the two base evidence lines ("Peer type:"
// and "Peer ID:") and conditionally appends agent/token lines only when those
// values are non-empty. The returned Code/Severity/Message are invariant.
// and "Peer ID:") and conditionally appends agent/bounded token-binding lines
// only when those values are non-empty. Raw token IDs are never evidence. The
// returned Code/Severity/Message are invariant.
func TestBranchCovIdentitySplitReason(t *testing.T) {
t.Parallel()
@@ -78,8 +80,9 @@ func TestBranchCovIdentitySplitReason(t *testing.T) {
t.Fatalf("Evidence[1] = %q, want %q", got, "Peer ID: "+tc.peerID)
}
// When the optional IDs are supplied they must be appended, in
// order, after the base lines.
// When the optional identities are supplied they must be appended,
// in order, after the base lines. Token binding stays descriptive
// without disclosing the token identifier.
offset := 2
if tc.peerAgentID != "" {
if got := reason.Evidence[offset]; got != "Peer agent ID: "+tc.peerAgentID {
@@ -88,8 +91,11 @@ func TestBranchCovIdentitySplitReason(t *testing.T) {
offset++
}
if tc.peerTokenID != "" {
if got := reason.Evidence[offset]; got != "Peer token ID: "+tc.peerTokenID {
t.Fatalf("token evidence = %q, want %q", got, "Peer token ID: "+tc.peerTokenID)
if got := reason.Evidence[offset]; got != "Peer shares the reporting token binding" {
t.Fatalf("token evidence = %q, want bounded token-binding description", got)
}
if strings.Contains(reason.Evidence[offset], tc.peerTokenID) {
t.Fatalf("token evidence disclosed raw token ID %q", tc.peerTokenID)
}
}
})
@@ -232,6 +232,7 @@ class CanonicalCompletionGuardTest(unittest.TestCase):
"mock-runtime-fixtures",
"pbs-protection-evidence-runtime",
"diskinventory-collection-trust",
"agent-fleet-diagnostics-runtime",
"monitoring-runtime",
],
)
@@ -64,6 +64,12 @@ class RegistryAuditTest(unittest.TestCase):
("agent-lifecycle", "unified-agent-installer-runtime"): {
"scripts/installtests/agent_state_dir_lifecycle_test.go",
},
("agent-lifecycle", "agent-doctor-settings-surface"): {
"frontend-modern/src/components/Settings/__tests__/InfrastructureSourceManager.test.tsx",
"frontend-modern/src/components/Settings/__tests__/InfrastructureWorkspace.test.tsx",
"frontend-modern/src/components/Settings/__tests__/infrastructureAgentDoctorModel.test.ts",
"frontend-modern/src/components/Settings/__tests__/infrastructureWorkspaceModel.test.ts",
},
("deployment-installability", "shell-installer-runtime"): {
"scripts/installtests/agent_state_dir_lifecycle_test.go",
},
@@ -84,6 +90,18 @@ class RegistryAuditTest(unittest.TestCase):
("api-contracts", "backend-payload-contracts"): {
"internal/api/ai_handlers_patrol_actions_additional_test.go",
},
("api-contracts", "agent-doctor-api-surface"): {
"frontend-modern/src/api/__tests__/agentDiagnostics.test.ts",
"frontend-modern/src/components/Settings/__tests__/infrastructureAgentDoctorModel.test.ts",
"internal/api/agent_fleet_doctor_test.go",
"internal/api/connections_aggregator_test.go",
"internal/api/contract_test.go",
"internal/api/update_readiness_test.go",
},
("frontend-primitives", "agent-doctor-settings-framing"): {
"frontend-modern/src/components/Settings/__tests__/DiagnosticsResultsPanel.test.tsx",
"frontend-modern/src/components/Settings/__tests__/settingsHeaderMeta.branchcov0713.test.ts",
},
("monitoring", "host-agent-ingest-runtime"): {
"internal/monitoring/issue1595_collection_trust_test.go",
},
@@ -94,6 +112,13 @@ class RegistryAuditTest(unittest.TestCase):
"internal/monitoring/issue1595_collection_trust_test.go",
"internal/monitoring/monitor_alert_override_migration_test.go",
},
("monitoring", "agent-fleet-diagnostics-runtime"): {
"internal/api/agent_fleet_doctor_test.go",
"internal/api/connections_aggregator_test.go",
"internal/fleethealth/agent_test.go",
"internal/monitoring/agent_fleet_doctor_test.go",
"internal/monitoring/monitoring_fleet_doctor_helpers_branchcov0716_test.go",
},
("monitoring", "diskinventory-collection-trust"): {
"internal/hostagent/issue1595_sas_collection_test.go",
"internal/monitoring/issue1595_collection_trust_test.go",
@@ -2981,8 +2981,8 @@ class SubsystemLookupTest(unittest.TestCase):
{
"heading": "## Shared Boundaries",
"path": "internal/api/access_control_handlers.go",
"line": 1247,
"heading_line": 157,
"line": 1248,
"heading_line": 158,
}
],
)