Add agent fleet diagnostics endpoint

This commit is contained in:
rcourtman
2026-06-29 18:03:46 +01:00
parent 73208a2863
commit 6b7d0b45f9
15 changed files with 1216 additions and 2 deletions
+4
View File
@@ -1154,6 +1154,10 @@ See [UNIFIED_AGENT.md](UNIFIED_AGENT.md) for installation instructions.
`GET /api/agent/version`
Returns the current server version for agent update checks.
### Agent Fleet Diagnostics
`GET /api/agents/diagnostics` (admin, `settings:read`)
Returns read-only fleet triage for reported host, Docker / Podman, and Kubernetes agents, including liveness, version drift, profile deployment drift, identity-split evidence, and supported repair handoff hints. It does not enqueue remote actions.
### Unified Agent Installer Script
`GET /install.sh`
Serves the universal `install.sh` used to install `pulse-agent` on target machines.
@@ -230,6 +230,13 @@ operator to mint a fresh install token or exposing agent IDs in the copied
command. Windows stale-agent update commands remain on the existing token-gated
install transport until the Windows installer owns an equivalent saved-state
update mode.
Agent Fleet Doctor diagnostics extend that same read-only lifecycle triage
surface: `GET /api/agents/diagnostics` may explain stale versions, missing
reports, profile deployment drift, expected Docker/Kubernetes telemetry gaps,
identity splits, and removed-agent blocks, and may advertise existing repair
handoffs such as copy-upgrade-command or allow-reenroll. It must not perform
the repair, create an action plan, or replace the canonical `/api/connections`
fleet projection used by Infrastructure.
Agent lifecycle and fleet-operation surfaces may consume
`POST /api/actions/plan` for resource capability planning, but the action plan
@@ -132,6 +132,7 @@ product API routes free of maintainer commercial analytics.
73a. `internal/agentcapabilities/`
73b. `pkg/extensions/ai_autofix.go`
83. `scripts/generate-types.go`
83a. `internal/api/agent_fleet_doctor.go`
## Shared Boundaries
@@ -4896,6 +4897,12 @@ reason, and the row must not be considered converged. Top-level
`remoteControl` may stay as compact presentation compatibility, but it must
not overstate desired server policy as applied agent runtime truth or collapse
desired/applied disagreement into one enabled/disabled fact.
The adjacent Agent Fleet Doctor endpoint, `GET /api/agents/diagnostics`, is a
read-only admin `settings:read` API for deeper fleet triage. It may summarize
liveness, version drift, profile deployment drift, missing expected telemetry,
identity splits, removed-agent blocks, and supported repair handoff hints, but
it must not mutate configuration, enqueue remote commands, or become the
canonical `/api/connections` fleet row source.
That same shared infrastructure-settings boundary also owns install-profile
semantics surfaced by
`frontend-modern/src/components/Settings/infrastructureOperationsModel.tsx`:
@@ -301,6 +301,13 @@ truth for live infrastructure data.
fields locally. Monitoring must preserve those objects as native cluster
inventory instead of flattening them into pods, deployments, or generic
networking, storage, configuration, or controller rows.
Agent Fleet Doctor diagnostics must derive from the current monitoring
`StateSnapshot`, agent-profile assignments, and profile deployment
acknowledgements only. `internal/monitoring/agent_fleet_doctor.go` may
explain liveness, version drift, identity splits, expected telemetry gaps,
and profile drift, but it must remain read-only and must not become a
separate collector, repair executor, or replacement for the canonical
`/api/connections` fleet projection.
19. Add or change unified-resource alert synchronization through
`internal/monitoring/monitor_alert_sync.go` and the alerts subsystem
contract together. Monitoring may pass the current unified-resource snapshot
@@ -964,6 +964,11 @@ recovery scope, or a storage/recovery-owned secret source.
protected system or a storage-local duplicate host.
22. Keep backend-native platform actions on the adjacent AI/runtime and platform contracts. When `internal/api/` wires native TrueNAS app control for Assistant, storage and recovery may consume the refreshed recovery points afterward, but they must not grow a parallel recovery-local action transport or action-specific payload shape.
23. Keep backend-native platform diagnostics on the adjacent AI/runtime and platform contracts. When `internal/api/` wires native TrueNAS app log reads for Assistant, storage and recovery may use those diagnostics during investigation, but they must not grow a parallel recovery-local log transport or diagnostic payload shape.
The same adjacent-boundary rule applies to `GET /api/agents/diagnostics`:
storage and recovery may read Agent Fleet Doctor evidence as operational
context for stale agents, version drift, profile drift, or identity split
investigation, but must not treat that endpoint as a storage/recovery
health source, repair API, or recovery-local fleet payload shape.
24. Keep backend-native platform configuration reads on the adjacent AI/runtime and platform contracts. When `internal/api/` wires native TrueNAS app config for Assistant, storage and recovery may use that runtime shape during investigation, but they must not grow a parallel recovery-local config transport or provider-shaped configuration payload.
25. Keep provider-backed poll cadence and settings-runtime health on the adjacent platform-connections contract. When shared `internal/api/` and poller wiring expose TrueNAS last-sync status, failure summaries, discovered contribution counts, manual saved-test status refresh, or platform handoff links in settings, storage and recovery may consume the resulting datasets, apps, disks, and recovery artifacts but must not redefine those settings-runtime health semantics or connection-level handoffs in storage/recovery-local transport or page flows.
26. Keep recovery filter/query state on the shared route-state parsing contract without restoring standalone recovery navigation. When platform pages or other embedded owners expose TrueNAS recovery context, they may reuse the canonical recovery query vocabulary with owned `platform` and `node` fields, but they must land inside an owning platform/runtime route instead of inventing drawer-local recovery URLs, treating PBS services as the only recovery path, or sending operators to the retired Recovery aggregate route.
+33
View File
@@ -0,0 +1,33 @@
package api
import (
"encoding/json"
"net/http"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/updates"
"github.com/rs/zerolog/log"
)
func (r *Router) handleAgentFleetDiagnostics(w http.ResponseWriter, req *http.Request) {
if req.Method != http.MethodGet {
writeErrorResponse(w, http.StatusMethodNotAllowed, "method_not_allowed", "Only GET is allowed", nil)
return
}
monitor, err := r.getMonitor(req)
if err != nil || monitor == nil {
writeErrorResponse(w, http.StatusInternalServerError, "monitor_unavailable", "Monitor not available", nil)
return
}
serverVersion := "dev"
if versionInfo, err := updates.GetCurrentVersion(); err == nil && versionInfo != nil {
serverVersion = versionInfo.Version
}
w.Header().Set("Content-Type", "application/json")
if err := json.NewEncoder(w).Encode(monitor.GetAgentFleetDiagnostics(serverVersion, time.Now().UTC())); err != nil {
log.Error().Err(err).Msg("Failed to serialize agent fleet diagnostics")
}
}
+51
View File
@@ -0,0 +1,51 @@
package api
import (
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/config"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/monitoring"
)
func TestHandleAgentFleetDiagnosticsReturnsFleetPayload(t *testing.T) {
monitor, state, _ := newTestMonitor(t)
now := time.Now().UTC()
state.UpsertHost(models.Host{
ID: "agent-1",
Hostname: "node-1",
DisplayName: "Node One",
Status: "online",
LastSeen: now.Add(-30 * time.Second),
IntervalSeconds: 30,
AgentVersion: "6.0.0",
})
router := &Router{config: &config.Config{}, monitor: monitor}
req := httptest.NewRequest(http.MethodGet, "/api/agents/diagnostics", nil)
rec := httptest.NewRecorder()
router.handleAgentFleetDiagnostics(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, body=%s", rec.Code, rec.Body.String())
}
var payload monitoring.AgentFleetDiagnostics
if err := json.Unmarshal(rec.Body.Bytes(), &payload); err != nil {
t.Fatalf("decode diagnostics: %v", err)
}
if payload.Agents == nil {
t.Fatal("expected agents to be a non-nil array")
}
if payload.Summary.Total != 1 || len(payload.Agents) != 1 {
t.Fatalf("expected one agent diagnostic, summary=%+v agents=%+v", payload.Summary, payload.Agents)
}
if payload.Agents[0].Name != "Node One" {
t.Fatalf("agent name = %q, want Node One", payload.Agents[0].Name)
}
}
+38
View File
@@ -18499,6 +18499,44 @@ func TestContract_AgentFleetContextEndpointSurfacesStableShape(t *testing.T) {
}
}
func TestContract_AgentFleetDiagnosticsEndpointSurfacesStableShape(t *testing.T) {
handler, err := os.ReadFile("agent_fleet_doctor.go")
if err != nil {
t.Fatalf("read agent_fleet_doctor.go: %v", err)
}
router, err := os.ReadFile("router_routes_registration.go")
if err != nil {
t.Fatalf("read router_routes_registration.go: %v", err)
}
monitoringSource, err := os.ReadFile("../monitoring/agent_fleet_doctor.go")
if err != nil {
t.Fatalf("read monitoring agent_fleet_doctor.go: %v", err)
}
handlerSrc := string(handler)
routerSrc := string(router)
monitoringSrc := string(monitoringSource)
if !strings.Contains(routerSrc, `"/api/agents/diagnostics"`) ||
!strings.Contains(routerSrc, `RequireAdmin(r.config, RequireScope(config.ScopeSettingsRead, r.handleAgentFleetDiagnostics))`) {
t.Error("agent fleet diagnostics route must remain admin settings:read only")
}
if !strings.Contains(handlerSrc, "GetAgentFleetDiagnostics(serverVersion, time.Now().UTC())") {
t.Error("agent fleet diagnostics handler must delegate to the monitoring-owned read-only producer")
}
for _, required := range []string{
"GeneratedAt int64 `json:\"generatedAt\"`",
"ServerVersion string `json:\"serverVersion,omitempty\"`",
"Summary AgentFleetDiagnosticSummary `json:\"summary\"`",
"Agents []AgentFleetAgentDiagnostic `json:\"agents\"`",
"Reasons []AgentFleetDiagnosticReason `json:\"reasons\"`",
"RepairActions []AgentFleetDiagnosticRepair `json:\"repairActions,omitempty\"`",
} {
if !strings.Contains(monitoringSrc, required) {
t.Errorf("agent fleet diagnostics payload missing stable field %q", required)
}
}
}
// TestContract_AgentOperationsLoopStatusEndpointSurfacesStableShape pins the
// content-safe loop-status wire shape external agents use before choosing
// fleet, resource, finding, or action tools.
+1
View File
@@ -385,6 +385,7 @@ var allRouteAllowlist = []string{
"/api/agents/host/unlink",
"/api/agents/agent/link",
"/api/agents/host/link",
"/api/agents/diagnostics",
"/api/agents/agent/",
"/api/agents/host/",
"/api/agents/docker/commands/",
@@ -67,6 +67,7 @@ func (r *Router) registerConfigSystemRoutes(updateHandlers *UpdateHandlers) {
r.mux.HandleFunc("/api/agents/host/unlink", wrapLegacyHostAlias("/api/agents/host/unlink", RequireAdmin(r.config, RequireScope(config.ScopeSettingsWrite, r.unifiedAgentHandlers.HandleUnlink))))
r.mux.HandleFunc("/api/agents/agent/link", RequireAdmin(r.config, RequireScope(config.ScopeSettingsWrite, r.unifiedAgentHandlers.HandleLink)))
r.mux.HandleFunc("/api/agents/host/link", wrapLegacyHostAlias("/api/agents/host/link", RequireAdmin(r.config, RequireScope(config.ScopeSettingsWrite, r.unifiedAgentHandlers.HandleLink))))
r.mux.HandleFunc("/api/agents/diagnostics", RequireAdmin(r.config, RequireScope(config.ScopeSettingsRead, r.handleAgentFleetDiagnostics)))
// Unified Agent management routes - config endpoint is accessible by agents (GET) and admins (PATCH)
unifiedAgentManagementCore := func(w http.ResponseWriter, req *http.Request) {
// Route /api/agents/agent/{id}/config to HandleConfig
+18
View File
@@ -3513,6 +3513,24 @@ func TestDiagnosticsRequireSettingsReadScope(t *testing.T) {
}
}
func TestAgentFleetDiagnosticsRequireSettingsReadScope(t *testing.T) {
rawToken := "agent-fleet-diag-token-123.12345678"
record := newTokenRecord(t, rawToken, []string{config.ScopeMonitoringRead}, nil)
cfg := newTestConfigWithTokens(t, record)
router := NewRouter(cfg, nil, nil, nil, nil, "1.0.0")
req := httptest.NewRequest(http.MethodGet, "/api/agents/diagnostics", nil)
req.Header.Set("X-API-Token", rawToken)
rec := httptest.NewRecorder()
router.Handler().ServeHTTP(rec, req)
if rec.Code != http.StatusForbidden {
t.Fatalf("expected 403 for missing settings:read scope, got %d", rec.Code)
}
if !strings.Contains(rec.Body.String(), config.ScopeSettingsRead) {
t.Fatalf("expected missing scope response to mention %q, got %q", config.ScopeSettingsRead, rec.Body.String())
}
}
func TestDiagnosticsPrepareTokenRequiresSettingsWriteScope(t *testing.T) {
rawToken := "diag-write-token-123.12345678"
record := newTokenRecord(t, rawToken, []string{config.ScopeSettingsRead}, nil)
+814
View File
@@ -0,0 +1,814 @@
package monitoring
import (
"fmt"
"sort"
"strconv"
"strings"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/updates"
"github.com/rs/zerolog/log"
)
const (
AgentFleetStatusHealthy = "healthy"
AgentFleetStatusWarning = "warning"
AgentFleetStatusCritical = "critical"
AgentFleetStatusRemoved = "removed"
)
type AgentFleetDiagnostics struct {
GeneratedAt int64 `json:"generatedAt"`
ServerVersion string `json:"serverVersion,omitempty"`
Summary AgentFleetDiagnosticSummary `json:"summary"`
Agents []AgentFleetAgentDiagnostic `json:"agents"`
}
type AgentFleetDiagnosticSummary struct {
Total int `json:"total"`
Healthy int `json:"healthy"`
Warning int `json:"warning"`
Critical int `json:"critical"`
Removed int `json:"removed"`
}
type AgentFleetAgentDiagnostic struct {
RowKey string `json:"rowKey"`
ID string `json:"id"`
AgentID string `json:"agentId,omitempty"`
Name string `json:"name"`
Hostname string `json:"hostname,omitempty"`
Types []string `json:"types"`
Status string `json:"status"`
RawStatus string `json:"rawStatus,omitempty"`
LastSeen int64 `json:"lastSeen,omitempty"`
IntervalSeconds int `json:"intervalSeconds,omitempty"`
Version string `json:"version,omitempty"`
ProfileID string `json:"profileId,omitempty"`
ProfileName string `json:"profileName,omitempty"`
ProfileVersion int `json:"profileVersion,omitempty"`
DeployedProfileVersion int `json:"deployedProfileVersion,omitempty"`
Reasons []AgentFleetDiagnosticReason `json:"reasons"`
RepairActions []AgentFleetDiagnosticRepair `json:"repairActions,omitempty"`
}
type AgentFleetDiagnosticReason struct {
Code string `json:"code"`
Severity string `json:"severity"`
Message string `json:"message"`
Evidence []string `json:"evidence,omitempty"`
}
type AgentFleetDiagnosticRepair struct {
Code string `json:"code"`
Label string `json:"label"`
Description string `json:"description"`
Supported bool `json:"supported"`
Scope string `json:"scope,omitempty"`
}
type agentFleetSubject struct {
rowKey string
id string
agentID string
name string
hostname string
types map[string]struct{}
rawStatus string
lastSeen time.Time
intervalSeconds int
version string
tokenID string
host *models.Host
docker *models.DockerHost
kubernetes *models.KubernetesCluster
removed bool
removedAt time.Time
}
// GetAgentFleetDiagnostics returns a read-only fleet health view derived from
// reported agent state, server version, and existing profile deployment state.
func (m *Monitor) GetAgentFleetDiagnostics(serverVersion string, now time.Time) AgentFleetDiagnostics {
if now.IsZero() {
now = time.Now().UTC()
}
now = now.UTC()
out := AgentFleetDiagnostics{
GeneratedAt: now.UnixMilli(),
ServerVersion: strings.TrimSpace(serverVersion),
}
if m == nil || m.state == nil {
return out
}
state := m.GetState()
profiles, assignments, deployments := m.agentFleetProfileState()
profileByID := mapProfilesByID(profiles)
assignmentByAgent := mapAssignmentsByAgent(assignments)
deploymentByAgentProfile := mapDeploymentsByAgentProfile(deployments)
subjects := buildAgentFleetSubjects(state)
for i := range subjects {
diagnostic := diagnoseAgentFleetSubject(subjects[i], state, out.ServerVersion, now, profileByID, assignmentByAgent, deploymentByAgentProfile)
out.Agents = append(out.Agents, diagnostic)
}
sort.Slice(out.Agents, func(i, j int) bool {
if out.Agents[i].Status != out.Agents[j].Status {
return agentFleetStatusRank(out.Agents[i].Status) > agentFleetStatusRank(out.Agents[j].Status)
}
return strings.ToLower(out.Agents[i].Name) < strings.ToLower(out.Agents[j].Name)
})
out.Summary.Total = len(out.Agents)
for _, agent := range out.Agents {
switch agent.Status {
case AgentFleetStatusCritical:
out.Summary.Critical++
case AgentFleetStatusWarning:
out.Summary.Warning++
case AgentFleetStatusRemoved:
out.Summary.Removed++
default:
out.Summary.Healthy++
}
}
return out
}
func (m *Monitor) agentFleetProfileState() ([]models.AgentProfile, []models.AgentProfileAssignment, []models.ProfileDeploymentStatus) {
if m == nil || m.persistence == nil {
return nil, nil, nil
}
profiles, assignments := m.getAgentProfileCache()
deployments, err := m.persistence.LoadProfileDeploymentStatus()
if err != nil {
log.Warn().Err(err).Msg("Failed to load agent profile deployment status for fleet diagnostics")
}
return profiles, assignments, deployments
}
func buildAgentFleetSubjects(state models.StateSnapshot) []agentFleetSubject {
subjectsByID := make(map[string]*agentFleetSubject)
order := make([]string, 0, len(state.Hosts)+len(state.DockerHosts)+len(state.KubernetesClusters))
getOrCreate := func(id string) *agentFleetSubject {
id = strings.TrimSpace(id)
if existing := subjectsByID[id]; existing != nil {
return existing
}
subject := &agentFleetSubject{
rowKey: "agent-" + id,
id: id,
types: map[string]struct{}{},
}
subjectsByID[id] = subject
order = append(order, id)
return subject
}
for i := range state.Hosts {
host := state.Hosts[i]
if strings.TrimSpace(host.ID) == "" {
continue
}
subject := getOrCreate(host.ID)
subject.host = &host
subject.types["host"] = struct{}{}
subject.agentID = firstNonEmpty(subject.agentID, host.ID)
subject.name = firstNonEmpty(host.DisplayName, host.Hostname, host.ID)
subject.hostname = firstNonEmpty(host.Hostname, subject.hostname)
subject.rawStatus = firstNonEmpty(host.Status, subject.rawStatus)
subject.lastSeen = newestTime(subject.lastSeen, host.LastSeen)
subject.intervalSeconds = maxInt(subject.intervalSeconds, host.IntervalSeconds)
subject.version = firstNonEmpty(host.AgentVersion, subject.version)
subject.tokenID = firstNonEmpty(host.TokenID, subject.tokenID)
}
for i := range state.DockerHosts {
docker := state.DockerHosts[i]
if strings.TrimSpace(docker.ID) == "" || docker.Hidden {
continue
}
subject := getOrCreate(docker.ID)
subject.docker = &docker
subject.types["docker"] = struct{}{}
subject.agentID = firstNonEmpty(subject.agentID, docker.AgentID, docker.ID)
subject.name = firstNonEmpty(docker.CustomDisplayName, docker.DisplayName, docker.Hostname, docker.ID, subject.name)
subject.hostname = firstNonEmpty(docker.Hostname, subject.hostname)
subject.rawStatus = firstNonEmpty(docker.Status, subject.rawStatus)
subject.lastSeen = newestTime(subject.lastSeen, docker.LastSeen)
subject.intervalSeconds = maxInt(subject.intervalSeconds, docker.IntervalSeconds)
subject.version = firstNonEmpty(subject.version, docker.AgentVersion, docker.DockerVersion)
subject.tokenID = firstNonEmpty(subject.tokenID, docker.TokenID)
}
for i := range state.KubernetesClusters {
cluster := state.KubernetesClusters[i]
if strings.TrimSpace(cluster.ID) == "" || cluster.Hidden {
continue
}
id := "k8s:" + cluster.ID
subject := &agentFleetSubject{
rowKey: "k8s-" + cluster.ID,
id: cluster.ID,
agentID: firstNonEmpty(cluster.AgentID, cluster.ID),
name: firstNonEmpty(cluster.CustomDisplayName, cluster.DisplayName, cluster.Name, cluster.ID),
types: map[string]struct{}{"kubernetes": {}},
rawStatus: cluster.Status,
lastSeen: cluster.LastSeen,
intervalSeconds: cluster.IntervalSeconds,
version: firstNonEmpty(cluster.AgentVersion, cluster.Version),
tokenID: cluster.TokenID,
kubernetes: &cluster,
}
subjectsByID[id] = subject
order = append(order, id)
}
for i := range state.RemovedDockerHosts {
removed := state.RemovedDockerHosts[i]
subjectsByID["removed-docker:"+removed.ID] = &agentFleetSubject{
rowKey: "removed-docker-" + removed.ID,
id: removed.ID,
name: firstNonEmpty(removed.DisplayName, removed.Hostname, removed.ID),
hostname: removed.Hostname,
types: map[string]struct{}{"docker": {}},
removed: true,
removedAt: removed.RemovedAt,
}
order = append(order, "removed-docker:"+removed.ID)
}
for i := range state.RemovedHostAgents {
removed := state.RemovedHostAgents[i]
subjectsByID["removed-host:"+removed.ID] = &agentFleetSubject{
rowKey: "removed-host-" + removed.ID,
id: removed.ID,
name: firstNonEmpty(removed.DisplayName, removed.Hostname, removed.ID),
hostname: removed.Hostname,
types: map[string]struct{}{"host": {}},
removed: true,
removedAt: removed.RemovedAt,
}
order = append(order, "removed-host:"+removed.ID)
}
for i := range state.RemovedKubernetesClusters {
removed := state.RemovedKubernetesClusters[i]
subjectsByID["removed-k8s:"+removed.ID] = &agentFleetSubject{
rowKey: "removed-k8s-" + removed.ID,
id: removed.ID,
name: firstNonEmpty(removed.DisplayName, removed.Name, removed.ID),
types: map[string]struct{}{"kubernetes": {}},
removed: true,
removedAt: removed.RemovedAt,
}
order = append(order, "removed-k8s:"+removed.ID)
}
subjects := make([]agentFleetSubject, 0, len(order))
for _, key := range order {
if subject := subjectsByID[key]; subject != nil {
subjects = append(subjects, *subject)
}
}
return subjects
}
func diagnoseAgentFleetSubject(
subject agentFleetSubject,
state models.StateSnapshot,
serverVersion string,
now time.Time,
profileByID map[string]models.AgentProfile,
assignmentByAgent map[string]models.AgentProfileAssignment,
deploymentByAgentProfile map[string]models.ProfileDeploymentStatus,
) AgentFleetAgentDiagnostic {
result := AgentFleetAgentDiagnostic{
RowKey: subject.rowKey,
ID: subject.id,
AgentID: subject.agentID,
Name: firstNonEmpty(subject.name, subject.hostname, subject.id),
Hostname: subject.hostname,
Types: sortedAgentTypes(subject.types),
RawStatus: subject.rawStatus,
IntervalSeconds: subject.intervalSeconds,
Version: subject.version,
}
if !subject.lastSeen.IsZero() {
result.LastSeen = subject.lastSeen.UnixMilli()
}
if subject.removed {
result.Status = AgentFleetStatusRemoved
if !subject.removedAt.IsZero() {
result.LastSeen = subject.removedAt.UnixMilli()
}
result.Reasons = append(result.Reasons, AgentFleetDiagnosticReason{
Code: "agent_removed_blocked",
Severity: AgentFleetStatusWarning,
Message: "This agent is intentionally removed and blocked from re-enrolling until an admin allows it.",
Evidence: []string{
fmt.Sprintf("Removed at %s", subject.removedAt.UTC().Format(time.RFC3339)),
},
})
result.RepairActions = append(result.RepairActions, AgentFleetDiagnosticRepair{
Code: "allow_reenroll",
Label: "Allow re-enroll",
Description: "Uses the existing allow re-enroll action for removed agents.",
Supported: true,
Scope: "settings:write",
})
return result
}
result.Reasons = append(result.Reasons, diagnoseAgentConnectivity(subject, now)...)
result.Reasons = append(result.Reasons, diagnoseAgentVersion(subject, serverVersion)...)
result.Reasons = append(result.Reasons, diagnoseAgentIdentitySplit(subject, state)...)
assignment, hasAssignment := findAgentAssignment(subject, assignmentByAgent)
if hasAssignment {
result.ProfileID = assignment.ProfileID
profile, hasProfile := profileByID[assignment.ProfileID]
if hasProfile {
result.ProfileName = profile.Name
result.ProfileVersion = expectedProfileVersion(profile, assignment)
result.Reasons = append(result.Reasons, diagnoseProfileDeployment(subject, assignment, profile, deploymentByAgentProfile)...)
result.Reasons = append(result.Reasons, diagnoseProfileCapabilityDrift(subject, state, profile)...)
} else {
result.Reasons = append(result.Reasons, AgentFleetDiagnosticReason{
Code: "profile_missing",
Severity: AgentFleetStatusWarning,
Message: "A profile is assigned to this agent, but that profile no longer exists.",
Evidence: []string{
"Assigned profile ID: " + assignment.ProfileID,
},
})
}
if deployment, ok := deploymentByAgentProfile[deploymentKey(assignment.AgentID, assignment.ProfileID)]; ok {
result.DeployedProfileVersion = deployment.DeployedVersion
}
}
for _, reason := range result.Reasons {
if reason.Code == "agent_version_stale" {
result.RepairActions = append(result.RepairActions, AgentFleetDiagnosticRepair{
Code: "copy_upgrade_command",
Label: "Copy upgrade command",
Description: "Uses the existing installer command from Settings -> Agents; no remote command is queued.",
Supported: true,
Scope: "local_admin_shell",
})
break
}
}
result.Status = diagnosticStatusFromReasons(result.Reasons)
return result
}
func diagnoseAgentConnectivity(subject agentFleetSubject, now time.Time) []AgentFleetDiagnosticReason {
if subject.kubernetes == nil && subject.host == nil && subject.docker == nil {
return nil
}
reasons := []AgentFleetDiagnosticReason{}
interval := subject.intervalSeconds
if interval <= 0 {
interval = 30
}
staleAfter := time.Duration(interval*5) * time.Second
if staleAfter < 5*time.Minute {
staleAfter = 5 * time.Minute
}
if subject.lastSeen.IsZero() {
return append(reasons, AgentFleetDiagnosticReason{
Code: "agent_never_reported",
Severity: AgentFleetStatusCritical,
Message: "This agent has no last-seen timestamp, so Pulse cannot confirm it is reporting.",
})
}
age := now.Sub(subject.lastSeen)
if age > staleAfter {
reasons = append(reasons, AgentFleetDiagnosticReason{
Code: "agent_disconnected",
Severity: AgentFleetStatusCritical,
Message: fmt.Sprintf("No report has arrived for %s; this is beyond the %s stale threshold for a %ds reporting interval.", roundDuration(age), roundDuration(staleAfter), interval),
Evidence: []string{
"Last seen: " + subject.lastSeen.UTC().Format(time.RFC3339),
fmt.Sprintf("Expected report interval: %ds", interval),
},
})
}
if subject.rawStatus != "" && !agentFleetStatusIsOnline(subject.rawStatus) {
reasons = append(reasons, AgentFleetDiagnosticReason{
Code: "agent_status_not_online",
Severity: AgentFleetStatusWarning,
Message: fmt.Sprintf("The agent reports status %q instead of online/running/healthy.", subject.rawStatus),
})
}
return reasons
}
func diagnoseAgentVersion(subject agentFleetSubject, serverVersion string) []AgentFleetDiagnosticReason {
if subject.removed {
return nil
}
agentVersion := strings.TrimSpace(subject.version)
if agentVersion == "" {
return []AgentFleetDiagnosticReason{{
Code: "agent_version_missing",
Severity: AgentFleetStatusWarning,
Message: "The agent did not report a version, so Pulse cannot verify update health.",
}}
}
serverVersion = strings.TrimSpace(serverVersion)
if serverVersion == "" || strings.EqualFold(serverVersion, "dev") {
return nil
}
serverParsed, err := updates.ParseVersion(serverVersion)
if err != nil {
return nil
}
agentParsed, err := updates.ParseVersion(agentVersion)
if err != nil {
return []AgentFleetDiagnosticReason{{
Code: "agent_version_unparseable",
Severity: AgentFleetStatusWarning,
Message: fmt.Sprintf("The agent reported version %q, which cannot be compared with server version %q.", agentVersion, serverVersion),
}}
}
if serverParsed.IsNewerThan(agentParsed) {
return []AgentFleetDiagnosticReason{{
Code: "agent_version_stale",
Severity: AgentFleetStatusWarning,
Message: fmt.Sprintf("Agent version %s is older than the Pulse server version %s.", agentVersion, serverVersion),
Evidence: []string{
"Agent version: " + agentVersion,
"Server version: " + serverVersion,
},
}}
}
return nil
}
func diagnoseAgentIdentitySplit(subject agentFleetSubject, state models.StateSnapshot) []AgentFleetDiagnosticReason {
if subject.hostname == "" || subject.removed {
return nil
}
if subject.host != nil && subject.docker == nil {
if docker, ok := findLikelyDockerPeer(subject, state.DockerHosts); ok {
return []AgentFleetDiagnosticReason{identitySplitReason("Docker", docker.ID, docker.AgentID, docker.TokenID)}
}
}
if subject.docker != nil && subject.host == nil {
if host, ok := findLikelyHostPeer(subject, state.Hosts); ok {
return []AgentFleetDiagnosticReason{identitySplitReason("Host", host.ID, host.ID, host.TokenID)}
}
}
return nil
}
func diagnoseProfileDeployment(
subject agentFleetSubject,
assignment models.AgentProfileAssignment,
profile models.AgentProfile,
deploymentByAgentProfile map[string]models.ProfileDeploymentStatus,
) []AgentFleetDiagnosticReason {
expectedVersion := expectedProfileVersion(profile, assignment)
if expectedVersion <= 0 {
expectedVersion = profile.Version
}
deployment, ok := deploymentByAgentProfile[deploymentKey(assignment.AgentID, assignment.ProfileID)]
if !ok {
return []AgentFleetDiagnosticReason{{
Code: "profile_deployment_missing",
Severity: AgentFleetStatusWarning,
Message: "A profile is assigned, but no deployment acknowledgement exists for this agent.",
Evidence: []string{
fmt.Sprintf("Assigned profile: %s", firstNonEmpty(profile.Name, profile.ID)),
fmt.Sprintf("Expected profile version: %d", expectedVersion),
},
}}
}
reasons := []AgentFleetDiagnosticReason{}
if deployment.ProfileID != assignment.ProfileID {
reasons = append(reasons, AgentFleetDiagnosticReason{
Code: "profile_deployment_mismatch",
Severity: AgentFleetStatusWarning,
Message: "The agent acknowledged a different profile than the one currently assigned.",
Evidence: []string{
"Assigned profile ID: " + assignment.ProfileID,
"Deployed profile ID: " + deployment.ProfileID,
},
})
}
if deployment.DeploymentStatus == "failed" {
reasons = append(reasons, AgentFleetDiagnosticReason{
Code: "profile_deployment_failed",
Severity: AgentFleetStatusCritical,
Message: "The assigned profile failed to deploy to this agent.",
Evidence: nonEmptyStrings(deployment.ErrorMessage),
})
} else if deployment.DeploymentStatus != "" && deployment.DeploymentStatus != "deployed" {
reasons = append(reasons, AgentFleetDiagnosticReason{
Code: "profile_deployment_pending",
Severity: AgentFleetStatusWarning,
Message: fmt.Sprintf("The assigned profile is still marked %q for this agent.", deployment.DeploymentStatus),
})
}
if expectedVersion > 0 && deployment.DeployedVersion > 0 && deployment.DeployedVersion < expectedVersion {
reasons = append(reasons, AgentFleetDiagnosticReason{
Code: "profile_version_drift",
Severity: AgentFleetStatusWarning,
Message: fmt.Sprintf("The agent has profile version %d, but version %d is assigned.", deployment.DeployedVersion, expectedVersion),
Evidence: []string{
fmt.Sprintf("Profile: %s", firstNonEmpty(profile.Name, profile.ID)),
fmt.Sprintf("Last deployed at: %s", deployment.LastDeployedAt.UTC().Format(time.RFC3339)),
},
})
}
_ = subject
return reasons
}
func diagnoseProfileCapabilityDrift(subject agentFleetSubject, state models.StateSnapshot, profile models.AgentProfile) []AgentFleetDiagnosticReason {
reasons := []AgentFleetDiagnosticReason{}
if enabled, ok := configBool(profile.Config, "enable_docker"); ok && enabled {
if subject.docker == nil {
if _, split := findLikelyDockerPeer(subject, state.DockerHosts); split {
reasons = append(reasons, AgentFleetDiagnosticReason{
Code: "docker_profile_identity_split",
Severity: AgentFleetStatusWarning,
Message: "The assigned profile enables Docker monitoring, but Docker telemetry is reporting under a separate agent identity.",
Evidence: []string{
"Profile key enable_docker=true",
"Likely same host matched by hostname or token, not by agent ID",
},
})
} else {
reasons = append(reasons, AgentFleetDiagnosticReason{
Code: "docker_expected_missing",
Severity: AgentFleetStatusCritical,
Message: "The assigned profile enables Docker monitoring, but no Docker host telemetry is reporting for this agent.",
Evidence: []string{
"Profile key enable_docker=true",
"No matching Docker host record by agent ID, host ID, hostname, or token",
"Local causes can include missing Docker socket access, installing on the wrong host, or Docker mode being disabled",
},
})
}
}
}
if enabled, ok := configBool(profile.Config, "enable_kubernetes"); ok && enabled && subject.kubernetes == nil && !hasType(subject.types, "kubernetes") {
reasons = append(reasons, AgentFleetDiagnosticReason{
Code: "kubernetes_expected_missing",
Severity: AgentFleetStatusWarning,
Message: "The assigned profile enables Kubernetes monitoring, but no Kubernetes cluster telemetry is reporting for this agent.",
Evidence: []string{"Profile key enable_kubernetes=true"},
})
}
if enabled, ok := configBool(profile.Config, "enable_proxmox"); ok && enabled && subject.host != nil && subject.host.LinkedNodeID == "" {
reasons = append(reasons, AgentFleetDiagnosticReason{
Code: "proxmox_profile_unlinked",
Severity: AgentFleetStatusWarning,
Message: "The assigned profile enables Proxmox mode, but this host agent is not linked to a Proxmox node.",
Evidence: []string{"Profile key enable_proxmox=true"},
})
}
if enabled, ok := configBool(profile.Config, "enable_host"); ok && !enabled && subject.host != nil {
reasons = append(reasons, AgentFleetDiagnosticReason{
Code: "host_profile_not_applied",
Severity: AgentFleetStatusWarning,
Message: "The assigned profile disables host monitoring, but host telemetry is still reporting.",
Evidence: []string{"Profile key enable_host=false"},
})
}
return reasons
}
func identitySplitReason(peerType, peerID, peerAgentID, peerTokenID string) AgentFleetDiagnosticReason {
evidence := []string{"Peer type: " + peerType, "Peer ID: " + peerID}
if peerAgentID != "" {
evidence = append(evidence, "Peer agent ID: "+peerAgentID)
}
if peerTokenID != "" {
evidence = append(evidence, "Peer token ID: "+peerTokenID)
}
return AgentFleetDiagnosticReason{
Code: "agent_identity_split",
Severity: AgentFleetStatusWarning,
Message: "Host and workload telemetry appear to belong to the same machine but are reporting as separate agent identities.",
Evidence: evidence,
}
}
func findLikelyDockerPeer(subject agentFleetSubject, dockerHosts []models.DockerHost) (models.DockerHost, bool) {
for _, docker := range dockerHosts {
if docker.Hidden || docker.ID == subject.id {
continue
}
if sameAgentIdentity(subject, docker.ID, docker.AgentID, docker.Hostname, docker.TokenID) {
return docker, true
}
}
return models.DockerHost{}, false
}
func findLikelyHostPeer(subject agentFleetSubject, hosts []models.Host) (models.Host, bool) {
for _, host := range hosts {
if host.ID == subject.id {
continue
}
if sameAgentIdentity(subject, host.ID, host.ID, host.Hostname, host.TokenID) {
return host, true
}
}
return models.Host{}, false
}
func sameAgentIdentity(subject agentFleetSubject, id, agentID, hostname, tokenID string) bool {
if subject.agentID != "" && (subject.agentID == agentID || subject.agentID == id) {
return true
}
if subject.tokenID != "" && tokenID != "" && subject.tokenID == tokenID {
return true
}
if subject.hostname != "" && strings.EqualFold(subject.hostname, hostname) {
return true
}
return false
}
func mapProfilesByID(profiles []models.AgentProfile) map[string]models.AgentProfile {
result := make(map[string]models.AgentProfile, len(profiles))
for _, profile := range profiles {
result[profile.ID] = profile
}
return result
}
func mapAssignmentsByAgent(assignments []models.AgentProfileAssignment) map[string]models.AgentProfileAssignment {
result := make(map[string]models.AgentProfileAssignment, len(assignments))
for _, assignment := range assignments {
result[assignment.AgentID] = assignment
}
return result
}
func mapDeploymentsByAgentProfile(deployments []models.ProfileDeploymentStatus) map[string]models.ProfileDeploymentStatus {
result := make(map[string]models.ProfileDeploymentStatus, len(deployments))
for _, deployment := range deployments {
result[deploymentKey(deployment.AgentID, deployment.ProfileID)] = deployment
}
return result
}
func findAgentAssignment(subject agentFleetSubject, assignmentByAgent map[string]models.AgentProfileAssignment) (models.AgentProfileAssignment, bool) {
for _, id := range uniqueNonEmptyStrings(subject.agentID, subject.id) {
if assignment, ok := assignmentByAgent[id]; ok {
return assignment, true
}
}
return models.AgentProfileAssignment{}, false
}
func deploymentKey(agentID, profileID string) string {
return strings.TrimSpace(agentID) + "\x00" + strings.TrimSpace(profileID)
}
func expectedProfileVersion(profile models.AgentProfile, assignment models.AgentProfileAssignment) int {
if assignment.ProfileVersion > 0 {
return assignment.ProfileVersion
}
return profile.Version
}
func configBool(config models.AgentConfigMap, key string) (bool, bool) {
raw, ok := config[key]
if !ok {
return false, false
}
switch value := raw.(type) {
case bool:
return value, true
case string:
parsed, err := strconv.ParseBool(strings.TrimSpace(value))
return parsed, err == nil
default:
return false, false
}
}
func sortedAgentTypes(types map[string]struct{}) []string {
order := []string{"host", "docker", "kubernetes"}
out := make([]string, 0, len(types))
for _, candidate := range order {
if _, ok := types[candidate]; ok {
out = append(out, candidate)
}
}
return out
}
func diagnosticStatusFromReasons(reasons []AgentFleetDiagnosticReason) string {
status := AgentFleetStatusHealthy
for _, reason := range reasons {
if agentFleetStatusRank(reason.Severity) > agentFleetStatusRank(status) {
status = reason.Severity
}
}
return status
}
func agentFleetStatusRank(status string) int {
switch status {
case AgentFleetStatusCritical:
return 3
case AgentFleetStatusWarning:
return 2
case AgentFleetStatusRemoved:
return 1
default:
return 0
}
}
func agentFleetStatusIsOnline(status string) bool {
switch strings.ToLower(strings.TrimSpace(status)) {
case "online", "running", "healthy":
return true
default:
return false
}
}
func hasType(types map[string]struct{}, kind string) bool {
_, ok := types[kind]
return ok
}
func firstNonEmpty(values ...string) string {
for _, value := range values {
if trimmed := strings.TrimSpace(value); trimmed != "" {
return trimmed
}
}
return ""
}
func newestTime(a, b time.Time) time.Time {
if a.IsZero() || b.After(a) {
return b
}
return a
}
func maxInt(a, b int) int {
if b > a {
return b
}
return a
}
func nonEmptyStrings(values ...string) []string {
out := make([]string, 0, len(values))
for _, value := range values {
if trimmed := strings.TrimSpace(value); trimmed != "" {
out = append(out, trimmed)
}
}
return out
}
func roundDuration(duration time.Duration) string {
if duration < time.Second {
return duration.String()
}
return duration.Round(time.Second).String()
}
@@ -0,0 +1,198 @@
package monitoring
import (
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/config"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
)
func TestAgentFleetDiagnosticsDetectsStaleAgentVersion(t *testing.T) {
now := time.Date(2026, 6, 29, 12, 0, 0, 0, time.UTC)
monitor := newAgentFleetDoctorTestMonitor(t)
monitor.state.UpsertHost(models.Host{
ID: "agent-1",
Hostname: "pve-1",
DisplayName: "PVE 1",
Status: "online",
LastSeen: now.Add(-30 * time.Second),
IntervalSeconds: 30,
AgentVersion: "6.1.0",
})
diagnostics := monitor.GetAgentFleetDiagnostics("6.2.0", now)
agent := requireAgentDiagnostic(t, diagnostics, "agent-agent-1")
requireReasonCode(t, agent, "agent_version_stale")
if agent.Status != AgentFleetStatusWarning {
t.Fatalf("status = %q, want %q", agent.Status, AgentFleetStatusWarning)
}
if !hasSupportedRepair(agent, "copy_upgrade_command") {
t.Fatalf("expected stale version diagnostic to expose the supported upgrade command action: %#v", agent.RepairActions)
}
}
func TestAgentFleetDiagnosticsDetectsMissingDockerTelemetryFromProfile(t *testing.T) {
now := time.Date(2026, 6, 29, 12, 0, 0, 0, time.UTC)
monitor := newAgentFleetDoctorTestMonitor(t)
monitor.state.UpsertHost(models.Host{
ID: "agent-1",
Hostname: "docker-node",
DisplayName: "Docker Node",
Status: "online",
LastSeen: now.Add(-30 * time.Second),
IntervalSeconds: 30,
AgentVersion: "6.2.0",
})
saveAgentFleetProfileState(t, monitor.persistence,
[]models.AgentProfile{{
ID: "profile-docker",
Name: "Docker profile",
Version: 2,
Config: models.AgentConfigMap{
"enable_docker": true,
},
}},
[]models.AgentProfileAssignment{{
AgentID: "agent-1",
ProfileID: "profile-docker",
ProfileVersion: 2,
UpdatedAt: now,
}},
[]models.ProfileDeploymentStatus{{
AgentID: "agent-1",
ProfileID: "profile-docker",
AssignedVersion: 2,
DeployedVersion: 2,
DeploymentStatus: "deployed",
LastDeployedAt: now,
}},
)
diagnostics := monitor.GetAgentFleetDiagnostics("6.2.0", now)
agent := requireAgentDiagnostic(t, diagnostics, "agent-agent-1")
reason := requireReasonCode(t, agent, "docker_expected_missing")
if agent.Status != AgentFleetStatusCritical {
t.Fatalf("status = %q, want %q", agent.Status, AgentFleetStatusCritical)
}
if !containsString(reason.Evidence, "Local causes can include missing Docker socket access, installing on the wrong host, or Docker mode being disabled") {
t.Fatalf("missing Docker evidence should name unsupported local causes, got %#v", reason.Evidence)
}
}
func TestAgentFleetDiagnosticsDetectsProfileVersionDrift(t *testing.T) {
now := time.Date(2026, 6, 29, 12, 0, 0, 0, time.UTC)
monitor := newAgentFleetDoctorTestMonitor(t)
monitor.state.UpsertHost(models.Host{
ID: "agent-1",
Hostname: "profile-node",
DisplayName: "Profile Node",
Status: "online",
LastSeen: now.Add(-30 * time.Second),
IntervalSeconds: 30,
AgentVersion: "6.2.0",
})
saveAgentFleetProfileState(t, monitor.persistence,
[]models.AgentProfile{{
ID: "profile-current",
Name: "Current profile",
Version: 4,
Config: models.AgentConfigMap{},
}},
[]models.AgentProfileAssignment{{
AgentID: "agent-1",
ProfileID: "profile-current",
ProfileVersion: 4,
UpdatedAt: now,
}},
[]models.ProfileDeploymentStatus{{
AgentID: "agent-1",
ProfileID: "profile-current",
AssignedVersion: 4,
DeployedVersion: 2,
DeploymentStatus: "deployed",
LastDeployedAt: now.Add(-10 * time.Minute),
}},
)
diagnostics := monitor.GetAgentFleetDiagnostics("6.2.0", now)
agent := requireAgentDiagnostic(t, diagnostics, "agent-agent-1")
reason := requireReasonCode(t, agent, "profile_version_drift")
if agent.ProfileVersion != 4 || agent.DeployedProfileVersion != 2 {
t.Fatalf("profile versions = assigned %d deployed %d, want assigned 4 deployed 2", agent.ProfileVersion, agent.DeployedProfileVersion)
}
if reason.Message != "The agent has profile version 2, but version 4 is assigned." {
t.Fatalf("reason message = %q", reason.Message)
}
}
func newAgentFleetDoctorTestMonitor(t *testing.T) *Monitor {
t.Helper()
return &Monitor{
state: models.NewState(),
persistence: config.NewConfigPersistence(t.TempDir()),
config: &config.Config{},
}
}
func saveAgentFleetProfileState(
t *testing.T,
persistence *config.ConfigPersistence,
profiles []models.AgentProfile,
assignments []models.AgentProfileAssignment,
deployments []models.ProfileDeploymentStatus,
) {
t.Helper()
if err := persistence.SaveAgentProfiles(profiles); err != nil {
t.Fatalf("SaveAgentProfiles: %v", err)
}
if err := persistence.SaveAgentProfileAssignments(assignments); err != nil {
t.Fatalf("SaveAgentProfileAssignments: %v", err)
}
if err := persistence.SaveProfileDeploymentStatus(deployments); err != nil {
t.Fatalf("SaveProfileDeploymentStatus: %v", err)
}
}
func requireAgentDiagnostic(t *testing.T, diagnostics AgentFleetDiagnostics, rowKey string) AgentFleetAgentDiagnostic {
t.Helper()
for _, agent := range diagnostics.Agents {
if agent.RowKey == rowKey {
return agent
}
}
t.Fatalf("diagnostic row %q not found in %#v", rowKey, diagnostics.Agents)
return AgentFleetAgentDiagnostic{}
}
func requireReasonCode(t *testing.T, agent AgentFleetAgentDiagnostic, code string) AgentFleetDiagnosticReason {
t.Helper()
for _, reason := range agent.Reasons {
if reason.Code == code {
return reason
}
}
t.Fatalf("reason %q not found in %#v", code, agent.Reasons)
return AgentFleetDiagnosticReason{}
}
func hasSupportedRepair(agent AgentFleetAgentDiagnostic, code string) bool {
for _, repair := range agent.RepairActions {
if repair.Code == code && repair.Supported {
return true
}
}
return false
}
func containsString(values []string, want string) bool {
for _, value := range values {
if value == want {
return true
}
}
return false
}
@@ -2389,3 +2389,33 @@ func TestForEachMonitorVisitsAllTenantMonitors(t *testing.T) {
nilMTM.ForEachMonitor(func(*Monitor) { t.Fatal("nil receiver must not invoke callback") })
mtm.ForEachMonitor(nil)
}
func TestAgentFleetDoctorStaysReadOnlyProjection(t *testing.T) {
source, err := os.ReadFile("agent_fleet_doctor.go")
if err != nil {
t.Fatalf("read agent_fleet_doctor.go: %v", err)
}
src := string(source)
for _, required := range []string{
"GetAgentFleetDiagnostics",
"buildAgentFleetSubjects(state)",
"agentFleetProfileState()",
"LoadProfileDeploymentStatus()",
} {
if !strings.Contains(src, required) {
t.Fatalf("agent fleet doctor must keep read-only projection primitive %q", required)
}
}
for _, forbidden := range []string{
"SaveAgentProfiles(",
"SaveAgentProfileAssignments(",
"SaveProfileDeploymentStatus(",
"UpdateHostAgentConfig(",
"HandleCommandAck",
"Execute",
} {
if strings.Contains(src, forbidden) {
t.Fatalf("agent fleet doctor must stay read-only; found forbidden mutation/execution primitive %q", forbidden)
}
}
}
@@ -2916,8 +2916,8 @@ class SubsystemLookupTest(unittest.TestCase):
{
"heading": "## Shared Boundaries",
"path": "internal/api/access_control_handlers.go",
"line": 1087,
"heading_line": 136,
"line": 1088,
"heading_line": 137,
}
],
)