diff --git a/docs/API.md b/docs/API.md index fbd28358c..404d3a109 100644 --- a/docs/API.md +++ b/docs/API.md @@ -1154,6 +1154,10 @@ See [UNIFIED_AGENT.md](UNIFIED_AGENT.md) for installation instructions. `GET /api/agent/version` Returns the current server version for agent update checks. +### Agent Fleet Diagnostics +`GET /api/agents/diagnostics` (admin, `settings:read`) +Returns read-only fleet triage for reported host, Docker / Podman, and Kubernetes agents, including liveness, version drift, profile deployment drift, identity-split evidence, and supported repair handoff hints. It does not enqueue remote actions. + ### Unified Agent Installer Script `GET /install.sh` Serves the universal `install.sh` used to install `pulse-agent` on target machines. diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md index 4ce529aa1..cd7f3dd5b 100644 --- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md +++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md @@ -230,6 +230,13 @@ operator to mint a fresh install token or exposing agent IDs in the copied command. Windows stale-agent update commands remain on the existing token-gated install transport until the Windows installer owns an equivalent saved-state update mode. +Agent Fleet Doctor diagnostics extend that same read-only lifecycle triage +surface: `GET /api/agents/diagnostics` may explain stale versions, missing +reports, profile deployment drift, expected Docker/Kubernetes telemetry gaps, +identity splits, and removed-agent blocks, and may advertise existing repair +handoffs such as copy-upgrade-command or allow-reenroll. It must not perform +the repair, create an action plan, or replace the canonical `/api/connections` +fleet projection used by Infrastructure. Agent lifecycle and fleet-operation surfaces may consume `POST /api/actions/plan` for resource capability planning, but the action plan diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md index 2fc9c38de..b34b6dfc4 100644 --- a/docs/release-control/v6/internal/subsystems/api-contracts.md +++ b/docs/release-control/v6/internal/subsystems/api-contracts.md @@ -132,6 +132,7 @@ product API routes free of maintainer commercial analytics. 73a. `internal/agentcapabilities/` 73b. `pkg/extensions/ai_autofix.go` 83. `scripts/generate-types.go` + 83a. `internal/api/agent_fleet_doctor.go` ## Shared Boundaries @@ -4896,6 +4897,12 @@ reason, and the row must not be considered converged. Top-level `remoteControl` may stay as compact presentation compatibility, but it must not overstate desired server policy as applied agent runtime truth or collapse desired/applied disagreement into one enabled/disabled fact. +The adjacent Agent Fleet Doctor endpoint, `GET /api/agents/diagnostics`, is a +read-only admin `settings:read` API for deeper fleet triage. It may summarize +liveness, version drift, profile deployment drift, missing expected telemetry, +identity splits, removed-agent blocks, and supported repair handoff hints, but +it must not mutate configuration, enqueue remote commands, or become the +canonical `/api/connections` fleet row source. That same shared infrastructure-settings boundary also owns install-profile semantics surfaced by `frontend-modern/src/components/Settings/infrastructureOperationsModel.tsx`: diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md index e0366e12f..25d451e7f 100644 --- a/docs/release-control/v6/internal/subsystems/monitoring.md +++ b/docs/release-control/v6/internal/subsystems/monitoring.md @@ -301,6 +301,13 @@ truth for live infrastructure data. fields locally. Monitoring must preserve those objects as native cluster inventory instead of flattening them into pods, deployments, or generic networking, storage, configuration, or controller rows. + Agent Fleet Doctor diagnostics must derive from the current monitoring + `StateSnapshot`, agent-profile assignments, and profile deployment + acknowledgements only. `internal/monitoring/agent_fleet_doctor.go` may + explain liveness, version drift, identity splits, expected telemetry gaps, + and profile drift, but it must remain read-only and must not become a + separate collector, repair executor, or replacement for the canonical + `/api/connections` fleet projection. 19. Add or change unified-resource alert synchronization through `internal/monitoring/monitor_alert_sync.go` and the alerts subsystem contract together. Monitoring may pass the current unified-resource snapshot diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md index 86ad7975f..4a986385e 100644 --- a/docs/release-control/v6/internal/subsystems/storage-recovery.md +++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md @@ -964,6 +964,11 @@ recovery scope, or a storage/recovery-owned secret source. protected system or a storage-local duplicate host. 22. Keep backend-native platform actions on the adjacent AI/runtime and platform contracts. When `internal/api/` wires native TrueNAS app control for Assistant, storage and recovery may consume the refreshed recovery points afterward, but they must not grow a parallel recovery-local action transport or action-specific payload shape. 23. Keep backend-native platform diagnostics on the adjacent AI/runtime and platform contracts. When `internal/api/` wires native TrueNAS app log reads for Assistant, storage and recovery may use those diagnostics during investigation, but they must not grow a parallel recovery-local log transport or diagnostic payload shape. + The same adjacent-boundary rule applies to `GET /api/agents/diagnostics`: + storage and recovery may read Agent Fleet Doctor evidence as operational + context for stale agents, version drift, profile drift, or identity split + investigation, but must not treat that endpoint as a storage/recovery + health source, repair API, or recovery-local fleet payload shape. 24. Keep backend-native platform configuration reads on the adjacent AI/runtime and platform contracts. When `internal/api/` wires native TrueNAS app config for Assistant, storage and recovery may use that runtime shape during investigation, but they must not grow a parallel recovery-local config transport or provider-shaped configuration payload. 25. Keep provider-backed poll cadence and settings-runtime health on the adjacent platform-connections contract. When shared `internal/api/` and poller wiring expose TrueNAS last-sync status, failure summaries, discovered contribution counts, manual saved-test status refresh, or platform handoff links in settings, storage and recovery may consume the resulting datasets, apps, disks, and recovery artifacts but must not redefine those settings-runtime health semantics or connection-level handoffs in storage/recovery-local transport or page flows. 26. Keep recovery filter/query state on the shared route-state parsing contract without restoring standalone recovery navigation. When platform pages or other embedded owners expose TrueNAS recovery context, they may reuse the canonical recovery query vocabulary with owned `platform` and `node` fields, but they must land inside an owning platform/runtime route instead of inventing drawer-local recovery URLs, treating PBS services as the only recovery path, or sending operators to the retired Recovery aggregate route. diff --git a/internal/api/agent_fleet_doctor.go b/internal/api/agent_fleet_doctor.go new file mode 100644 index 000000000..8cde25cd3 --- /dev/null +++ b/internal/api/agent_fleet_doctor.go @@ -0,0 +1,33 @@ +package api + +import ( + "encoding/json" + "net/http" + "time" + + "github.com/rcourtman/pulse-go-rewrite/internal/updates" + "github.com/rs/zerolog/log" +) + +func (r *Router) handleAgentFleetDiagnostics(w http.ResponseWriter, req *http.Request) { + if req.Method != http.MethodGet { + writeErrorResponse(w, http.StatusMethodNotAllowed, "method_not_allowed", "Only GET is allowed", nil) + return + } + + monitor, err := r.getMonitor(req) + if err != nil || monitor == nil { + writeErrorResponse(w, http.StatusInternalServerError, "monitor_unavailable", "Monitor not available", nil) + return + } + + serverVersion := "dev" + if versionInfo, err := updates.GetCurrentVersion(); err == nil && versionInfo != nil { + serverVersion = versionInfo.Version + } + + w.Header().Set("Content-Type", "application/json") + if err := json.NewEncoder(w).Encode(monitor.GetAgentFleetDiagnostics(serverVersion, time.Now().UTC())); err != nil { + log.Error().Err(err).Msg("Failed to serialize agent fleet diagnostics") + } +} diff --git a/internal/api/agent_fleet_doctor_test.go b/internal/api/agent_fleet_doctor_test.go new file mode 100644 index 000000000..5ee68d887 --- /dev/null +++ b/internal/api/agent_fleet_doctor_test.go @@ -0,0 +1,51 @@ +package api + +import ( + "encoding/json" + "net/http" + "net/http/httptest" + "testing" + "time" + + "github.com/rcourtman/pulse-go-rewrite/internal/config" + "github.com/rcourtman/pulse-go-rewrite/internal/models" + "github.com/rcourtman/pulse-go-rewrite/internal/monitoring" +) + +func TestHandleAgentFleetDiagnosticsReturnsFleetPayload(t *testing.T) { + monitor, state, _ := newTestMonitor(t) + now := time.Now().UTC() + state.UpsertHost(models.Host{ + ID: "agent-1", + Hostname: "node-1", + DisplayName: "Node One", + Status: "online", + LastSeen: now.Add(-30 * time.Second), + IntervalSeconds: 30, + AgentVersion: "6.0.0", + }) + + router := &Router{config: &config.Config{}, monitor: monitor} + req := httptest.NewRequest(http.MethodGet, "/api/agents/diagnostics", nil) + rec := httptest.NewRecorder() + + router.handleAgentFleetDiagnostics(rec, req) + + if rec.Code != http.StatusOK { + t.Fatalf("status = %d, body=%s", rec.Code, rec.Body.String()) + } + + var payload monitoring.AgentFleetDiagnostics + if err := json.Unmarshal(rec.Body.Bytes(), &payload); err != nil { + t.Fatalf("decode diagnostics: %v", err) + } + if payload.Agents == nil { + t.Fatal("expected agents to be a non-nil array") + } + if payload.Summary.Total != 1 || len(payload.Agents) != 1 { + t.Fatalf("expected one agent diagnostic, summary=%+v agents=%+v", payload.Summary, payload.Agents) + } + if payload.Agents[0].Name != "Node One" { + t.Fatalf("agent name = %q, want Node One", payload.Agents[0].Name) + } +} diff --git a/internal/api/contract_test.go b/internal/api/contract_test.go index 35a423740..5432bc834 100644 --- a/internal/api/contract_test.go +++ b/internal/api/contract_test.go @@ -18499,6 +18499,44 @@ func TestContract_AgentFleetContextEndpointSurfacesStableShape(t *testing.T) { } } +func TestContract_AgentFleetDiagnosticsEndpointSurfacesStableShape(t *testing.T) { + handler, err := os.ReadFile("agent_fleet_doctor.go") + if err != nil { + t.Fatalf("read agent_fleet_doctor.go: %v", err) + } + router, err := os.ReadFile("router_routes_registration.go") + if err != nil { + t.Fatalf("read router_routes_registration.go: %v", err) + } + monitoringSource, err := os.ReadFile("../monitoring/agent_fleet_doctor.go") + if err != nil { + t.Fatalf("read monitoring agent_fleet_doctor.go: %v", err) + } + handlerSrc := string(handler) + routerSrc := string(router) + monitoringSrc := string(monitoringSource) + + if !strings.Contains(routerSrc, `"/api/agents/diagnostics"`) || + !strings.Contains(routerSrc, `RequireAdmin(r.config, RequireScope(config.ScopeSettingsRead, r.handleAgentFleetDiagnostics))`) { + t.Error("agent fleet diagnostics route must remain admin settings:read only") + } + if !strings.Contains(handlerSrc, "GetAgentFleetDiagnostics(serverVersion, time.Now().UTC())") { + t.Error("agent fleet diagnostics handler must delegate to the monitoring-owned read-only producer") + } + for _, required := range []string{ + "GeneratedAt int64 `json:\"generatedAt\"`", + "ServerVersion string `json:\"serverVersion,omitempty\"`", + "Summary AgentFleetDiagnosticSummary `json:\"summary\"`", + "Agents []AgentFleetAgentDiagnostic `json:\"agents\"`", + "Reasons []AgentFleetDiagnosticReason `json:\"reasons\"`", + "RepairActions []AgentFleetDiagnosticRepair `json:\"repairActions,omitempty\"`", + } { + if !strings.Contains(monitoringSrc, required) { + t.Errorf("agent fleet diagnostics payload missing stable field %q", required) + } + } +} + // TestContract_AgentOperationsLoopStatusEndpointSurfacesStableShape pins the // content-safe loop-status wire shape external agents use before choosing // fleet, resource, finding, or action tools. diff --git a/internal/api/route_inventory_test.go b/internal/api/route_inventory_test.go index 91d8d7a29..d1c531350 100644 --- a/internal/api/route_inventory_test.go +++ b/internal/api/route_inventory_test.go @@ -385,6 +385,7 @@ var allRouteAllowlist = []string{ "/api/agents/host/unlink", "/api/agents/agent/link", "/api/agents/host/link", + "/api/agents/diagnostics", "/api/agents/agent/", "/api/agents/host/", "/api/agents/docker/commands/", diff --git a/internal/api/router_routes_registration.go b/internal/api/router_routes_registration.go index ee8cf3559..2e01eaf14 100644 --- a/internal/api/router_routes_registration.go +++ b/internal/api/router_routes_registration.go @@ -67,6 +67,7 @@ func (r *Router) registerConfigSystemRoutes(updateHandlers *UpdateHandlers) { r.mux.HandleFunc("/api/agents/host/unlink", wrapLegacyHostAlias("/api/agents/host/unlink", RequireAdmin(r.config, RequireScope(config.ScopeSettingsWrite, r.unifiedAgentHandlers.HandleUnlink)))) r.mux.HandleFunc("/api/agents/agent/link", RequireAdmin(r.config, RequireScope(config.ScopeSettingsWrite, r.unifiedAgentHandlers.HandleLink))) r.mux.HandleFunc("/api/agents/host/link", wrapLegacyHostAlias("/api/agents/host/link", RequireAdmin(r.config, RequireScope(config.ScopeSettingsWrite, r.unifiedAgentHandlers.HandleLink)))) + r.mux.HandleFunc("/api/agents/diagnostics", RequireAdmin(r.config, RequireScope(config.ScopeSettingsRead, r.handleAgentFleetDiagnostics))) // Unified Agent management routes - config endpoint is accessible by agents (GET) and admins (PATCH) unifiedAgentManagementCore := func(w http.ResponseWriter, req *http.Request) { // Route /api/agents/agent/{id}/config to HandleConfig diff --git a/internal/api/security_regression_test.go b/internal/api/security_regression_test.go index 06038ec71..b3b62787d 100644 --- a/internal/api/security_regression_test.go +++ b/internal/api/security_regression_test.go @@ -3513,6 +3513,24 @@ func TestDiagnosticsRequireSettingsReadScope(t *testing.T) { } } +func TestAgentFleetDiagnosticsRequireSettingsReadScope(t *testing.T) { + rawToken := "agent-fleet-diag-token-123.12345678" + record := newTokenRecord(t, rawToken, []string{config.ScopeMonitoringRead}, nil) + cfg := newTestConfigWithTokens(t, record) + router := NewRouter(cfg, nil, nil, nil, nil, "1.0.0") + + req := httptest.NewRequest(http.MethodGet, "/api/agents/diagnostics", nil) + req.Header.Set("X-API-Token", rawToken) + rec := httptest.NewRecorder() + router.Handler().ServeHTTP(rec, req) + if rec.Code != http.StatusForbidden { + t.Fatalf("expected 403 for missing settings:read scope, got %d", rec.Code) + } + if !strings.Contains(rec.Body.String(), config.ScopeSettingsRead) { + t.Fatalf("expected missing scope response to mention %q, got %q", config.ScopeSettingsRead, rec.Body.String()) + } +} + func TestDiagnosticsPrepareTokenRequiresSettingsWriteScope(t *testing.T) { rawToken := "diag-write-token-123.12345678" record := newTokenRecord(t, rawToken, []string{config.ScopeSettingsRead}, nil) diff --git a/internal/monitoring/agent_fleet_doctor.go b/internal/monitoring/agent_fleet_doctor.go new file mode 100644 index 000000000..b97ad009f --- /dev/null +++ b/internal/monitoring/agent_fleet_doctor.go @@ -0,0 +1,814 @@ +package monitoring + +import ( + "fmt" + "sort" + "strconv" + "strings" + "time" + + "github.com/rcourtman/pulse-go-rewrite/internal/models" + "github.com/rcourtman/pulse-go-rewrite/internal/updates" + "github.com/rs/zerolog/log" +) + +const ( + AgentFleetStatusHealthy = "healthy" + AgentFleetStatusWarning = "warning" + AgentFleetStatusCritical = "critical" + AgentFleetStatusRemoved = "removed" +) + +type AgentFleetDiagnostics struct { + GeneratedAt int64 `json:"generatedAt"` + ServerVersion string `json:"serverVersion,omitempty"` + Summary AgentFleetDiagnosticSummary `json:"summary"` + Agents []AgentFleetAgentDiagnostic `json:"agents"` +} + +type AgentFleetDiagnosticSummary struct { + Total int `json:"total"` + Healthy int `json:"healthy"` + Warning int `json:"warning"` + Critical int `json:"critical"` + Removed int `json:"removed"` +} + +type AgentFleetAgentDiagnostic struct { + RowKey string `json:"rowKey"` + ID string `json:"id"` + AgentID string `json:"agentId,omitempty"` + Name string `json:"name"` + Hostname string `json:"hostname,omitempty"` + Types []string `json:"types"` + Status string `json:"status"` + RawStatus string `json:"rawStatus,omitempty"` + LastSeen int64 `json:"lastSeen,omitempty"` + IntervalSeconds int `json:"intervalSeconds,omitempty"` + Version string `json:"version,omitempty"` + ProfileID string `json:"profileId,omitempty"` + ProfileName string `json:"profileName,omitempty"` + ProfileVersion int `json:"profileVersion,omitempty"` + DeployedProfileVersion int `json:"deployedProfileVersion,omitempty"` + Reasons []AgentFleetDiagnosticReason `json:"reasons"` + RepairActions []AgentFleetDiagnosticRepair `json:"repairActions,omitempty"` +} + +type AgentFleetDiagnosticReason struct { + Code string `json:"code"` + Severity string `json:"severity"` + Message string `json:"message"` + Evidence []string `json:"evidence,omitempty"` +} + +type AgentFleetDiagnosticRepair struct { + Code string `json:"code"` + Label string `json:"label"` + Description string `json:"description"` + Supported bool `json:"supported"` + Scope string `json:"scope,omitempty"` +} + +type agentFleetSubject struct { + rowKey string + id string + agentID string + name string + hostname string + types map[string]struct{} + rawStatus string + lastSeen time.Time + intervalSeconds int + version string + tokenID string + host *models.Host + docker *models.DockerHost + kubernetes *models.KubernetesCluster + removed bool + removedAt time.Time +} + +// GetAgentFleetDiagnostics returns a read-only fleet health view derived from +// reported agent state, server version, and existing profile deployment state. +func (m *Monitor) GetAgentFleetDiagnostics(serverVersion string, now time.Time) AgentFleetDiagnostics { + if now.IsZero() { + now = time.Now().UTC() + } + now = now.UTC() + + out := AgentFleetDiagnostics{ + GeneratedAt: now.UnixMilli(), + ServerVersion: strings.TrimSpace(serverVersion), + } + if m == nil || m.state == nil { + return out + } + + state := m.GetState() + profiles, assignments, deployments := m.agentFleetProfileState() + profileByID := mapProfilesByID(profiles) + assignmentByAgent := mapAssignmentsByAgent(assignments) + deploymentByAgentProfile := mapDeploymentsByAgentProfile(deployments) + subjects := buildAgentFleetSubjects(state) + + for i := range subjects { + diagnostic := diagnoseAgentFleetSubject(subjects[i], state, out.ServerVersion, now, profileByID, assignmentByAgent, deploymentByAgentProfile) + out.Agents = append(out.Agents, diagnostic) + } + + sort.Slice(out.Agents, func(i, j int) bool { + if out.Agents[i].Status != out.Agents[j].Status { + return agentFleetStatusRank(out.Agents[i].Status) > agentFleetStatusRank(out.Agents[j].Status) + } + return strings.ToLower(out.Agents[i].Name) < strings.ToLower(out.Agents[j].Name) + }) + + out.Summary.Total = len(out.Agents) + for _, agent := range out.Agents { + switch agent.Status { + case AgentFleetStatusCritical: + out.Summary.Critical++ + case AgentFleetStatusWarning: + out.Summary.Warning++ + case AgentFleetStatusRemoved: + out.Summary.Removed++ + default: + out.Summary.Healthy++ + } + } + + return out +} + +func (m *Monitor) agentFleetProfileState() ([]models.AgentProfile, []models.AgentProfileAssignment, []models.ProfileDeploymentStatus) { + if m == nil || m.persistence == nil { + return nil, nil, nil + } + + profiles, assignments := m.getAgentProfileCache() + deployments, err := m.persistence.LoadProfileDeploymentStatus() + if err != nil { + log.Warn().Err(err).Msg("Failed to load agent profile deployment status for fleet diagnostics") + } + return profiles, assignments, deployments +} + +func buildAgentFleetSubjects(state models.StateSnapshot) []agentFleetSubject { + subjectsByID := make(map[string]*agentFleetSubject) + order := make([]string, 0, len(state.Hosts)+len(state.DockerHosts)+len(state.KubernetesClusters)) + + getOrCreate := func(id string) *agentFleetSubject { + id = strings.TrimSpace(id) + if existing := subjectsByID[id]; existing != nil { + return existing + } + subject := &agentFleetSubject{ + rowKey: "agent-" + id, + id: id, + types: map[string]struct{}{}, + } + subjectsByID[id] = subject + order = append(order, id) + return subject + } + + for i := range state.Hosts { + host := state.Hosts[i] + if strings.TrimSpace(host.ID) == "" { + continue + } + subject := getOrCreate(host.ID) + subject.host = &host + subject.types["host"] = struct{}{} + subject.agentID = firstNonEmpty(subject.agentID, host.ID) + subject.name = firstNonEmpty(host.DisplayName, host.Hostname, host.ID) + subject.hostname = firstNonEmpty(host.Hostname, subject.hostname) + subject.rawStatus = firstNonEmpty(host.Status, subject.rawStatus) + subject.lastSeen = newestTime(subject.lastSeen, host.LastSeen) + subject.intervalSeconds = maxInt(subject.intervalSeconds, host.IntervalSeconds) + subject.version = firstNonEmpty(host.AgentVersion, subject.version) + subject.tokenID = firstNonEmpty(host.TokenID, subject.tokenID) + } + + for i := range state.DockerHosts { + docker := state.DockerHosts[i] + if strings.TrimSpace(docker.ID) == "" || docker.Hidden { + continue + } + subject := getOrCreate(docker.ID) + subject.docker = &docker + subject.types["docker"] = struct{}{} + subject.agentID = firstNonEmpty(subject.agentID, docker.AgentID, docker.ID) + subject.name = firstNonEmpty(docker.CustomDisplayName, docker.DisplayName, docker.Hostname, docker.ID, subject.name) + subject.hostname = firstNonEmpty(docker.Hostname, subject.hostname) + subject.rawStatus = firstNonEmpty(docker.Status, subject.rawStatus) + subject.lastSeen = newestTime(subject.lastSeen, docker.LastSeen) + subject.intervalSeconds = maxInt(subject.intervalSeconds, docker.IntervalSeconds) + subject.version = firstNonEmpty(subject.version, docker.AgentVersion, docker.DockerVersion) + subject.tokenID = firstNonEmpty(subject.tokenID, docker.TokenID) + } + + for i := range state.KubernetesClusters { + cluster := state.KubernetesClusters[i] + if strings.TrimSpace(cluster.ID) == "" || cluster.Hidden { + continue + } + id := "k8s:" + cluster.ID + subject := &agentFleetSubject{ + rowKey: "k8s-" + cluster.ID, + id: cluster.ID, + agentID: firstNonEmpty(cluster.AgentID, cluster.ID), + name: firstNonEmpty(cluster.CustomDisplayName, cluster.DisplayName, cluster.Name, cluster.ID), + types: map[string]struct{}{"kubernetes": {}}, + rawStatus: cluster.Status, + lastSeen: cluster.LastSeen, + intervalSeconds: cluster.IntervalSeconds, + version: firstNonEmpty(cluster.AgentVersion, cluster.Version), + tokenID: cluster.TokenID, + kubernetes: &cluster, + } + subjectsByID[id] = subject + order = append(order, id) + } + + for i := range state.RemovedDockerHosts { + removed := state.RemovedDockerHosts[i] + subjectsByID["removed-docker:"+removed.ID] = &agentFleetSubject{ + rowKey: "removed-docker-" + removed.ID, + id: removed.ID, + name: firstNonEmpty(removed.DisplayName, removed.Hostname, removed.ID), + hostname: removed.Hostname, + types: map[string]struct{}{"docker": {}}, + removed: true, + removedAt: removed.RemovedAt, + } + order = append(order, "removed-docker:"+removed.ID) + } + + for i := range state.RemovedHostAgents { + removed := state.RemovedHostAgents[i] + subjectsByID["removed-host:"+removed.ID] = &agentFleetSubject{ + rowKey: "removed-host-" + removed.ID, + id: removed.ID, + name: firstNonEmpty(removed.DisplayName, removed.Hostname, removed.ID), + hostname: removed.Hostname, + types: map[string]struct{}{"host": {}}, + removed: true, + removedAt: removed.RemovedAt, + } + order = append(order, "removed-host:"+removed.ID) + } + + for i := range state.RemovedKubernetesClusters { + removed := state.RemovedKubernetesClusters[i] + subjectsByID["removed-k8s:"+removed.ID] = &agentFleetSubject{ + rowKey: "removed-k8s-" + removed.ID, + id: removed.ID, + name: firstNonEmpty(removed.DisplayName, removed.Name, removed.ID), + types: map[string]struct{}{"kubernetes": {}}, + removed: true, + removedAt: removed.RemovedAt, + } + order = append(order, "removed-k8s:"+removed.ID) + } + + subjects := make([]agentFleetSubject, 0, len(order)) + for _, key := range order { + if subject := subjectsByID[key]; subject != nil { + subjects = append(subjects, *subject) + } + } + return subjects +} + +func diagnoseAgentFleetSubject( + subject agentFleetSubject, + state models.StateSnapshot, + serverVersion string, + now time.Time, + profileByID map[string]models.AgentProfile, + assignmentByAgent map[string]models.AgentProfileAssignment, + deploymentByAgentProfile map[string]models.ProfileDeploymentStatus, +) AgentFleetAgentDiagnostic { + result := AgentFleetAgentDiagnostic{ + RowKey: subject.rowKey, + ID: subject.id, + AgentID: subject.agentID, + Name: firstNonEmpty(subject.name, subject.hostname, subject.id), + Hostname: subject.hostname, + Types: sortedAgentTypes(subject.types), + RawStatus: subject.rawStatus, + IntervalSeconds: subject.intervalSeconds, + Version: subject.version, + } + if !subject.lastSeen.IsZero() { + result.LastSeen = subject.lastSeen.UnixMilli() + } + + if subject.removed { + result.Status = AgentFleetStatusRemoved + if !subject.removedAt.IsZero() { + result.LastSeen = subject.removedAt.UnixMilli() + } + result.Reasons = append(result.Reasons, AgentFleetDiagnosticReason{ + Code: "agent_removed_blocked", + Severity: AgentFleetStatusWarning, + Message: "This agent is intentionally removed and blocked from re-enrolling until an admin allows it.", + Evidence: []string{ + fmt.Sprintf("Removed at %s", subject.removedAt.UTC().Format(time.RFC3339)), + }, + }) + result.RepairActions = append(result.RepairActions, AgentFleetDiagnosticRepair{ + Code: "allow_reenroll", + Label: "Allow re-enroll", + Description: "Uses the existing allow re-enroll action for removed agents.", + Supported: true, + Scope: "settings:write", + }) + return result + } + + result.Reasons = append(result.Reasons, diagnoseAgentConnectivity(subject, now)...) + result.Reasons = append(result.Reasons, diagnoseAgentVersion(subject, serverVersion)...) + result.Reasons = append(result.Reasons, diagnoseAgentIdentitySplit(subject, state)...) + + assignment, hasAssignment := findAgentAssignment(subject, assignmentByAgent) + if hasAssignment { + result.ProfileID = assignment.ProfileID + profile, hasProfile := profileByID[assignment.ProfileID] + if hasProfile { + result.ProfileName = profile.Name + result.ProfileVersion = expectedProfileVersion(profile, assignment) + result.Reasons = append(result.Reasons, diagnoseProfileDeployment(subject, assignment, profile, deploymentByAgentProfile)...) + result.Reasons = append(result.Reasons, diagnoseProfileCapabilityDrift(subject, state, profile)...) + } else { + result.Reasons = append(result.Reasons, AgentFleetDiagnosticReason{ + Code: "profile_missing", + Severity: AgentFleetStatusWarning, + Message: "A profile is assigned to this agent, but that profile no longer exists.", + Evidence: []string{ + "Assigned profile ID: " + assignment.ProfileID, + }, + }) + } + + if deployment, ok := deploymentByAgentProfile[deploymentKey(assignment.AgentID, assignment.ProfileID)]; ok { + result.DeployedProfileVersion = deployment.DeployedVersion + } + } + + for _, reason := range result.Reasons { + if reason.Code == "agent_version_stale" { + result.RepairActions = append(result.RepairActions, AgentFleetDiagnosticRepair{ + Code: "copy_upgrade_command", + Label: "Copy upgrade command", + Description: "Uses the existing installer command from Settings -> Agents; no remote command is queued.", + Supported: true, + Scope: "local_admin_shell", + }) + break + } + } + + result.Status = diagnosticStatusFromReasons(result.Reasons) + return result +} + +func diagnoseAgentConnectivity(subject agentFleetSubject, now time.Time) []AgentFleetDiagnosticReason { + if subject.kubernetes == nil && subject.host == nil && subject.docker == nil { + return nil + } + + reasons := []AgentFleetDiagnosticReason{} + interval := subject.intervalSeconds + if interval <= 0 { + interval = 30 + } + staleAfter := time.Duration(interval*5) * time.Second + if staleAfter < 5*time.Minute { + staleAfter = 5 * time.Minute + } + + if subject.lastSeen.IsZero() { + return append(reasons, AgentFleetDiagnosticReason{ + Code: "agent_never_reported", + Severity: AgentFleetStatusCritical, + Message: "This agent has no last-seen timestamp, so Pulse cannot confirm it is reporting.", + }) + } + + age := now.Sub(subject.lastSeen) + if age > staleAfter { + reasons = append(reasons, AgentFleetDiagnosticReason{ + Code: "agent_disconnected", + Severity: AgentFleetStatusCritical, + Message: fmt.Sprintf("No report has arrived for %s; this is beyond the %s stale threshold for a %ds reporting interval.", roundDuration(age), roundDuration(staleAfter), interval), + Evidence: []string{ + "Last seen: " + subject.lastSeen.UTC().Format(time.RFC3339), + fmt.Sprintf("Expected report interval: %ds", interval), + }, + }) + } + + if subject.rawStatus != "" && !agentFleetStatusIsOnline(subject.rawStatus) { + reasons = append(reasons, AgentFleetDiagnosticReason{ + Code: "agent_status_not_online", + Severity: AgentFleetStatusWarning, + Message: fmt.Sprintf("The agent reports status %q instead of online/running/healthy.", subject.rawStatus), + }) + } + + return reasons +} + +func diagnoseAgentVersion(subject agentFleetSubject, serverVersion string) []AgentFleetDiagnosticReason { + if subject.removed { + return nil + } + agentVersion := strings.TrimSpace(subject.version) + if agentVersion == "" { + return []AgentFleetDiagnosticReason{{ + Code: "agent_version_missing", + Severity: AgentFleetStatusWarning, + Message: "The agent did not report a version, so Pulse cannot verify update health.", + }} + } + + serverVersion = strings.TrimSpace(serverVersion) + if serverVersion == "" || strings.EqualFold(serverVersion, "dev") { + return nil + } + + serverParsed, err := updates.ParseVersion(serverVersion) + if err != nil { + return nil + } + agentParsed, err := updates.ParseVersion(agentVersion) + if err != nil { + return []AgentFleetDiagnosticReason{{ + Code: "agent_version_unparseable", + Severity: AgentFleetStatusWarning, + Message: fmt.Sprintf("The agent reported version %q, which cannot be compared with server version %q.", agentVersion, serverVersion), + }} + } + + if serverParsed.IsNewerThan(agentParsed) { + return []AgentFleetDiagnosticReason{{ + Code: "agent_version_stale", + Severity: AgentFleetStatusWarning, + Message: fmt.Sprintf("Agent version %s is older than the Pulse server version %s.", agentVersion, serverVersion), + Evidence: []string{ + "Agent version: " + agentVersion, + "Server version: " + serverVersion, + }, + }} + } + + return nil +} + +func diagnoseAgentIdentitySplit(subject agentFleetSubject, state models.StateSnapshot) []AgentFleetDiagnosticReason { + if subject.hostname == "" || subject.removed { + return nil + } + + if subject.host != nil && subject.docker == nil { + if docker, ok := findLikelyDockerPeer(subject, state.DockerHosts); ok { + return []AgentFleetDiagnosticReason{identitySplitReason("Docker", docker.ID, docker.AgentID, docker.TokenID)} + } + } + if subject.docker != nil && subject.host == nil { + if host, ok := findLikelyHostPeer(subject, state.Hosts); ok { + return []AgentFleetDiagnosticReason{identitySplitReason("Host", host.ID, host.ID, host.TokenID)} + } + } + + return nil +} + +func diagnoseProfileDeployment( + subject agentFleetSubject, + assignment models.AgentProfileAssignment, + profile models.AgentProfile, + deploymentByAgentProfile map[string]models.ProfileDeploymentStatus, +) []AgentFleetDiagnosticReason { + expectedVersion := expectedProfileVersion(profile, assignment) + if expectedVersion <= 0 { + expectedVersion = profile.Version + } + + deployment, ok := deploymentByAgentProfile[deploymentKey(assignment.AgentID, assignment.ProfileID)] + if !ok { + return []AgentFleetDiagnosticReason{{ + Code: "profile_deployment_missing", + Severity: AgentFleetStatusWarning, + Message: "A profile is assigned, but no deployment acknowledgement exists for this agent.", + Evidence: []string{ + fmt.Sprintf("Assigned profile: %s", firstNonEmpty(profile.Name, profile.ID)), + fmt.Sprintf("Expected profile version: %d", expectedVersion), + }, + }} + } + + reasons := []AgentFleetDiagnosticReason{} + if deployment.ProfileID != assignment.ProfileID { + reasons = append(reasons, AgentFleetDiagnosticReason{ + Code: "profile_deployment_mismatch", + Severity: AgentFleetStatusWarning, + Message: "The agent acknowledged a different profile than the one currently assigned.", + Evidence: []string{ + "Assigned profile ID: " + assignment.ProfileID, + "Deployed profile ID: " + deployment.ProfileID, + }, + }) + } + if deployment.DeploymentStatus == "failed" { + reasons = append(reasons, AgentFleetDiagnosticReason{ + Code: "profile_deployment_failed", + Severity: AgentFleetStatusCritical, + Message: "The assigned profile failed to deploy to this agent.", + Evidence: nonEmptyStrings(deployment.ErrorMessage), + }) + } else if deployment.DeploymentStatus != "" && deployment.DeploymentStatus != "deployed" { + reasons = append(reasons, AgentFleetDiagnosticReason{ + Code: "profile_deployment_pending", + Severity: AgentFleetStatusWarning, + Message: fmt.Sprintf("The assigned profile is still marked %q for this agent.", deployment.DeploymentStatus), + }) + } + if expectedVersion > 0 && deployment.DeployedVersion > 0 && deployment.DeployedVersion < expectedVersion { + reasons = append(reasons, AgentFleetDiagnosticReason{ + Code: "profile_version_drift", + Severity: AgentFleetStatusWarning, + Message: fmt.Sprintf("The agent has profile version %d, but version %d is assigned.", deployment.DeployedVersion, expectedVersion), + Evidence: []string{ + fmt.Sprintf("Profile: %s", firstNonEmpty(profile.Name, profile.ID)), + fmt.Sprintf("Last deployed at: %s", deployment.LastDeployedAt.UTC().Format(time.RFC3339)), + }, + }) + } + + _ = subject + return reasons +} + +func diagnoseProfileCapabilityDrift(subject agentFleetSubject, state models.StateSnapshot, profile models.AgentProfile) []AgentFleetDiagnosticReason { + reasons := []AgentFleetDiagnosticReason{} + + if enabled, ok := configBool(profile.Config, "enable_docker"); ok && enabled { + if subject.docker == nil { + if _, split := findLikelyDockerPeer(subject, state.DockerHosts); split { + reasons = append(reasons, AgentFleetDiagnosticReason{ + Code: "docker_profile_identity_split", + Severity: AgentFleetStatusWarning, + Message: "The assigned profile enables Docker monitoring, but Docker telemetry is reporting under a separate agent identity.", + Evidence: []string{ + "Profile key enable_docker=true", + "Likely same host matched by hostname or token, not by agent ID", + }, + }) + } else { + reasons = append(reasons, AgentFleetDiagnosticReason{ + Code: "docker_expected_missing", + Severity: AgentFleetStatusCritical, + Message: "The assigned profile enables Docker monitoring, but no Docker host telemetry is reporting for this agent.", + Evidence: []string{ + "Profile key enable_docker=true", + "No matching Docker host record by agent ID, host ID, hostname, or token", + "Local causes can include missing Docker socket access, installing on the wrong host, or Docker mode being disabled", + }, + }) + } + } + } + + if enabled, ok := configBool(profile.Config, "enable_kubernetes"); ok && enabled && subject.kubernetes == nil && !hasType(subject.types, "kubernetes") { + reasons = append(reasons, AgentFleetDiagnosticReason{ + Code: "kubernetes_expected_missing", + Severity: AgentFleetStatusWarning, + Message: "The assigned profile enables Kubernetes monitoring, but no Kubernetes cluster telemetry is reporting for this agent.", + Evidence: []string{"Profile key enable_kubernetes=true"}, + }) + } + + if enabled, ok := configBool(profile.Config, "enable_proxmox"); ok && enabled && subject.host != nil && subject.host.LinkedNodeID == "" { + reasons = append(reasons, AgentFleetDiagnosticReason{ + Code: "proxmox_profile_unlinked", + Severity: AgentFleetStatusWarning, + Message: "The assigned profile enables Proxmox mode, but this host agent is not linked to a Proxmox node.", + Evidence: []string{"Profile key enable_proxmox=true"}, + }) + } + + if enabled, ok := configBool(profile.Config, "enable_host"); ok && !enabled && subject.host != nil { + reasons = append(reasons, AgentFleetDiagnosticReason{ + Code: "host_profile_not_applied", + Severity: AgentFleetStatusWarning, + Message: "The assigned profile disables host monitoring, but host telemetry is still reporting.", + Evidence: []string{"Profile key enable_host=false"}, + }) + } + + return reasons +} + +func identitySplitReason(peerType, peerID, peerAgentID, peerTokenID string) AgentFleetDiagnosticReason { + evidence := []string{"Peer type: " + peerType, "Peer ID: " + peerID} + if peerAgentID != "" { + evidence = append(evidence, "Peer agent ID: "+peerAgentID) + } + if peerTokenID != "" { + evidence = append(evidence, "Peer token ID: "+peerTokenID) + } + return AgentFleetDiagnosticReason{ + Code: "agent_identity_split", + Severity: AgentFleetStatusWarning, + Message: "Host and workload telemetry appear to belong to the same machine but are reporting as separate agent identities.", + Evidence: evidence, + } +} + +func findLikelyDockerPeer(subject agentFleetSubject, dockerHosts []models.DockerHost) (models.DockerHost, bool) { + for _, docker := range dockerHosts { + if docker.Hidden || docker.ID == subject.id { + continue + } + if sameAgentIdentity(subject, docker.ID, docker.AgentID, docker.Hostname, docker.TokenID) { + return docker, true + } + } + return models.DockerHost{}, false +} + +func findLikelyHostPeer(subject agentFleetSubject, hosts []models.Host) (models.Host, bool) { + for _, host := range hosts { + if host.ID == subject.id { + continue + } + if sameAgentIdentity(subject, host.ID, host.ID, host.Hostname, host.TokenID) { + return host, true + } + } + return models.Host{}, false +} + +func sameAgentIdentity(subject agentFleetSubject, id, agentID, hostname, tokenID string) bool { + if subject.agentID != "" && (subject.agentID == agentID || subject.agentID == id) { + return true + } + if subject.tokenID != "" && tokenID != "" && subject.tokenID == tokenID { + return true + } + if subject.hostname != "" && strings.EqualFold(subject.hostname, hostname) { + return true + } + return false +} + +func mapProfilesByID(profiles []models.AgentProfile) map[string]models.AgentProfile { + result := make(map[string]models.AgentProfile, len(profiles)) + for _, profile := range profiles { + result[profile.ID] = profile + } + return result +} + +func mapAssignmentsByAgent(assignments []models.AgentProfileAssignment) map[string]models.AgentProfileAssignment { + result := make(map[string]models.AgentProfileAssignment, len(assignments)) + for _, assignment := range assignments { + result[assignment.AgentID] = assignment + } + return result +} + +func mapDeploymentsByAgentProfile(deployments []models.ProfileDeploymentStatus) map[string]models.ProfileDeploymentStatus { + result := make(map[string]models.ProfileDeploymentStatus, len(deployments)) + for _, deployment := range deployments { + result[deploymentKey(deployment.AgentID, deployment.ProfileID)] = deployment + } + return result +} + +func findAgentAssignment(subject agentFleetSubject, assignmentByAgent map[string]models.AgentProfileAssignment) (models.AgentProfileAssignment, bool) { + for _, id := range uniqueNonEmptyStrings(subject.agentID, subject.id) { + if assignment, ok := assignmentByAgent[id]; ok { + return assignment, true + } + } + return models.AgentProfileAssignment{}, false +} + +func deploymentKey(agentID, profileID string) string { + return strings.TrimSpace(agentID) + "\x00" + strings.TrimSpace(profileID) +} + +func expectedProfileVersion(profile models.AgentProfile, assignment models.AgentProfileAssignment) int { + if assignment.ProfileVersion > 0 { + return assignment.ProfileVersion + } + return profile.Version +} + +func configBool(config models.AgentConfigMap, key string) (bool, bool) { + raw, ok := config[key] + if !ok { + return false, false + } + switch value := raw.(type) { + case bool: + return value, true + case string: + parsed, err := strconv.ParseBool(strings.TrimSpace(value)) + return parsed, err == nil + default: + return false, false + } +} + +func sortedAgentTypes(types map[string]struct{}) []string { + order := []string{"host", "docker", "kubernetes"} + out := make([]string, 0, len(types)) + for _, candidate := range order { + if _, ok := types[candidate]; ok { + out = append(out, candidate) + } + } + return out +} + +func diagnosticStatusFromReasons(reasons []AgentFleetDiagnosticReason) string { + status := AgentFleetStatusHealthy + for _, reason := range reasons { + if agentFleetStatusRank(reason.Severity) > agentFleetStatusRank(status) { + status = reason.Severity + } + } + return status +} + +func agentFleetStatusRank(status string) int { + switch status { + case AgentFleetStatusCritical: + return 3 + case AgentFleetStatusWarning: + return 2 + case AgentFleetStatusRemoved: + return 1 + default: + return 0 + } +} + +func agentFleetStatusIsOnline(status string) bool { + switch strings.ToLower(strings.TrimSpace(status)) { + case "online", "running", "healthy": + return true + default: + return false + } +} + +func hasType(types map[string]struct{}, kind string) bool { + _, ok := types[kind] + return ok +} + +func firstNonEmpty(values ...string) string { + for _, value := range values { + if trimmed := strings.TrimSpace(value); trimmed != "" { + return trimmed + } + } + return "" +} + +func newestTime(a, b time.Time) time.Time { + if a.IsZero() || b.After(a) { + return b + } + return a +} + +func maxInt(a, b int) int { + if b > a { + return b + } + return a +} + +func nonEmptyStrings(values ...string) []string { + out := make([]string, 0, len(values)) + for _, value := range values { + if trimmed := strings.TrimSpace(value); trimmed != "" { + out = append(out, trimmed) + } + } + return out +} + +func roundDuration(duration time.Duration) string { + if duration < time.Second { + return duration.String() + } + return duration.Round(time.Second).String() +} diff --git a/internal/monitoring/agent_fleet_doctor_test.go b/internal/monitoring/agent_fleet_doctor_test.go new file mode 100644 index 000000000..f9abc4e0c --- /dev/null +++ b/internal/monitoring/agent_fleet_doctor_test.go @@ -0,0 +1,198 @@ +package monitoring + +import ( + "testing" + "time" + + "github.com/rcourtman/pulse-go-rewrite/internal/config" + "github.com/rcourtman/pulse-go-rewrite/internal/models" +) + +func TestAgentFleetDiagnosticsDetectsStaleAgentVersion(t *testing.T) { + now := time.Date(2026, 6, 29, 12, 0, 0, 0, time.UTC) + monitor := newAgentFleetDoctorTestMonitor(t) + monitor.state.UpsertHost(models.Host{ + ID: "agent-1", + Hostname: "pve-1", + DisplayName: "PVE 1", + Status: "online", + LastSeen: now.Add(-30 * time.Second), + IntervalSeconds: 30, + AgentVersion: "6.1.0", + }) + + diagnostics := monitor.GetAgentFleetDiagnostics("6.2.0", now) + agent := requireAgentDiagnostic(t, diagnostics, "agent-agent-1") + + requireReasonCode(t, agent, "agent_version_stale") + if agent.Status != AgentFleetStatusWarning { + t.Fatalf("status = %q, want %q", agent.Status, AgentFleetStatusWarning) + } + if !hasSupportedRepair(agent, "copy_upgrade_command") { + t.Fatalf("expected stale version diagnostic to expose the supported upgrade command action: %#v", agent.RepairActions) + } +} + +func TestAgentFleetDiagnosticsDetectsMissingDockerTelemetryFromProfile(t *testing.T) { + now := time.Date(2026, 6, 29, 12, 0, 0, 0, time.UTC) + monitor := newAgentFleetDoctorTestMonitor(t) + monitor.state.UpsertHost(models.Host{ + ID: "agent-1", + Hostname: "docker-node", + DisplayName: "Docker Node", + Status: "online", + LastSeen: now.Add(-30 * time.Second), + IntervalSeconds: 30, + AgentVersion: "6.2.0", + }) + saveAgentFleetProfileState(t, monitor.persistence, + []models.AgentProfile{{ + ID: "profile-docker", + Name: "Docker profile", + Version: 2, + Config: models.AgentConfigMap{ + "enable_docker": true, + }, + }}, + []models.AgentProfileAssignment{{ + AgentID: "agent-1", + ProfileID: "profile-docker", + ProfileVersion: 2, + UpdatedAt: now, + }}, + []models.ProfileDeploymentStatus{{ + AgentID: "agent-1", + ProfileID: "profile-docker", + AssignedVersion: 2, + DeployedVersion: 2, + DeploymentStatus: "deployed", + LastDeployedAt: now, + }}, + ) + + diagnostics := monitor.GetAgentFleetDiagnostics("6.2.0", now) + agent := requireAgentDiagnostic(t, diagnostics, "agent-agent-1") + reason := requireReasonCode(t, agent, "docker_expected_missing") + + if agent.Status != AgentFleetStatusCritical { + t.Fatalf("status = %q, want %q", agent.Status, AgentFleetStatusCritical) + } + if !containsString(reason.Evidence, "Local causes can include missing Docker socket access, installing on the wrong host, or Docker mode being disabled") { + t.Fatalf("missing Docker evidence should name unsupported local causes, got %#v", reason.Evidence) + } +} + +func TestAgentFleetDiagnosticsDetectsProfileVersionDrift(t *testing.T) { + now := time.Date(2026, 6, 29, 12, 0, 0, 0, time.UTC) + monitor := newAgentFleetDoctorTestMonitor(t) + monitor.state.UpsertHost(models.Host{ + ID: "agent-1", + Hostname: "profile-node", + DisplayName: "Profile Node", + Status: "online", + LastSeen: now.Add(-30 * time.Second), + IntervalSeconds: 30, + AgentVersion: "6.2.0", + }) + saveAgentFleetProfileState(t, monitor.persistence, + []models.AgentProfile{{ + ID: "profile-current", + Name: "Current profile", + Version: 4, + Config: models.AgentConfigMap{}, + }}, + []models.AgentProfileAssignment{{ + AgentID: "agent-1", + ProfileID: "profile-current", + ProfileVersion: 4, + UpdatedAt: now, + }}, + []models.ProfileDeploymentStatus{{ + AgentID: "agent-1", + ProfileID: "profile-current", + AssignedVersion: 4, + DeployedVersion: 2, + DeploymentStatus: "deployed", + LastDeployedAt: now.Add(-10 * time.Minute), + }}, + ) + + diagnostics := monitor.GetAgentFleetDiagnostics("6.2.0", now) + agent := requireAgentDiagnostic(t, diagnostics, "agent-agent-1") + reason := requireReasonCode(t, agent, "profile_version_drift") + + if agent.ProfileVersion != 4 || agent.DeployedProfileVersion != 2 { + t.Fatalf("profile versions = assigned %d deployed %d, want assigned 4 deployed 2", agent.ProfileVersion, agent.DeployedProfileVersion) + } + if reason.Message != "The agent has profile version 2, but version 4 is assigned." { + t.Fatalf("reason message = %q", reason.Message) + } +} + +func newAgentFleetDoctorTestMonitor(t *testing.T) *Monitor { + t.Helper() + return &Monitor{ + state: models.NewState(), + persistence: config.NewConfigPersistence(t.TempDir()), + config: &config.Config{}, + } +} + +func saveAgentFleetProfileState( + t *testing.T, + persistence *config.ConfigPersistence, + profiles []models.AgentProfile, + assignments []models.AgentProfileAssignment, + deployments []models.ProfileDeploymentStatus, +) { + t.Helper() + if err := persistence.SaveAgentProfiles(profiles); err != nil { + t.Fatalf("SaveAgentProfiles: %v", err) + } + if err := persistence.SaveAgentProfileAssignments(assignments); err != nil { + t.Fatalf("SaveAgentProfileAssignments: %v", err) + } + if err := persistence.SaveProfileDeploymentStatus(deployments); err != nil { + t.Fatalf("SaveProfileDeploymentStatus: %v", err) + } +} + +func requireAgentDiagnostic(t *testing.T, diagnostics AgentFleetDiagnostics, rowKey string) AgentFleetAgentDiagnostic { + t.Helper() + for _, agent := range diagnostics.Agents { + if agent.RowKey == rowKey { + return agent + } + } + t.Fatalf("diagnostic row %q not found in %#v", rowKey, diagnostics.Agents) + return AgentFleetAgentDiagnostic{} +} + +func requireReasonCode(t *testing.T, agent AgentFleetAgentDiagnostic, code string) AgentFleetDiagnosticReason { + t.Helper() + for _, reason := range agent.Reasons { + if reason.Code == code { + return reason + } + } + t.Fatalf("reason %q not found in %#v", code, agent.Reasons) + return AgentFleetDiagnosticReason{} +} + +func hasSupportedRepair(agent AgentFleetAgentDiagnostic, code string) bool { + for _, repair := range agent.RepairActions { + if repair.Code == code && repair.Supported { + return true + } + } + return false +} + +func containsString(values []string, want string) bool { + for _, value := range values { + if value == want { + return true + } + } + return false +} diff --git a/internal/monitoring/canonical_guardrails_test.go b/internal/monitoring/canonical_guardrails_test.go index d143fd8bf..a7114b645 100644 --- a/internal/monitoring/canonical_guardrails_test.go +++ b/internal/monitoring/canonical_guardrails_test.go @@ -2389,3 +2389,33 @@ func TestForEachMonitorVisitsAllTenantMonitors(t *testing.T) { nilMTM.ForEachMonitor(func(*Monitor) { t.Fatal("nil receiver must not invoke callback") }) mtm.ForEachMonitor(nil) } + +func TestAgentFleetDoctorStaysReadOnlyProjection(t *testing.T) { + source, err := os.ReadFile("agent_fleet_doctor.go") + if err != nil { + t.Fatalf("read agent_fleet_doctor.go: %v", err) + } + src := string(source) + for _, required := range []string{ + "GetAgentFleetDiagnostics", + "buildAgentFleetSubjects(state)", + "agentFleetProfileState()", + "LoadProfileDeploymentStatus()", + } { + if !strings.Contains(src, required) { + t.Fatalf("agent fleet doctor must keep read-only projection primitive %q", required) + } + } + for _, forbidden := range []string{ + "SaveAgentProfiles(", + "SaveAgentProfileAssignments(", + "SaveProfileDeploymentStatus(", + "UpdateHostAgentConfig(", + "HandleCommandAck", + "Execute", + } { + if strings.Contains(src, forbidden) { + t.Fatalf("agent fleet doctor must stay read-only; found forbidden mutation/execution primitive %q", forbidden) + } + } +} diff --git a/scripts/release_control/subsystem_lookup_test.py b/scripts/release_control/subsystem_lookup_test.py index f075f013c..878b19c2c 100644 --- a/scripts/release_control/subsystem_lookup_test.py +++ b/scripts/release_control/subsystem_lookup_test.py @@ -2916,8 +2916,8 @@ class SubsystemLookupTest(unittest.TestCase): { "heading": "## Shared Boundaries", "path": "internal/api/access_control_handlers.go", - "line": 1087, - "heading_line": 136, + "line": 1088, + "heading_line": 137, } ], )