Resolve Patrol finding references before reading operator state

Patrol's finding operator-state provider read the store under the
finding's exact reference first and resolved it only on a miss. Once an
operator link folds an agent into a VM, a finding kept under the agent's
ID still found the agent's own row, even a default-valued one, and missed
the maintenance window or monitoring mode the operator then set on the
merged VM. The resources API and alert intent resolve first, so Patrol
disagreed with both.

The provider now resolves every reference through the organization
monitor's current read state (the published registry with saved hosts
overlaid, or the mock estate), which is the listing the resources API
seeds from, and reads only an unresolved reference as given. The old
miss path built a resources-API registry, which clones the unified
listing, for every finding without a row; the read state answers from a
registry it keeps.

One window still differs: a saved agent linked into a live VM that has
not reported since a restart stays its own row in the read state, so its
findings read the agent's own state there, as alert intent does.
This commit is contained in:
rcourtman
2026-10-08 03:57:15 +01:00
parent 30ab8fa31f
commit b60f441e22
8 changed files with 335 additions and 74 deletions
@@ -4348,6 +4348,21 @@ findings handed to the orchestrator, so investigation reasoning sees
the same lock-against-remediation flag that the action broker
enforces downstream — no possible drift between "what Patrol
proposes" and "what the broker accepts."
That builder resolves a finding's resource reference through the
organization monitor's current read state before it reads operator
state. A finding still keyed by an agent's canonical ID after an
operator linked the agent into its guest therefore reads the merged
row's state while the read state folds the agent, as the resources API
does. A finding naming a saved host that has not reported since a
restart by its agent reference reads the row stored under that host's
canonical ID, because the read state overlays host continuity as the
resources API listing does; a saved host joined to its guest by a link
stays its own row there, so its findings read the agent's own state
until it reports again. The
resolution only reads identity. It does not treat the folded agent as
removed, offline or unenrolled, does not turn a continuity-only host
into a live sighting, and grants no heartbeat, command, install or
repair authority.
The same router wiring owns the alert-bridge patrol-trigger callback. It now
receives the full alert payload as a struct and consults the operator's
@@ -5564,16 +5564,45 @@ the projection to the Finding before calling the orchestrator. This
is the one in-process write path that decides what the orchestrator
sees about operator commitments; all consumers (in-process Patrol,
external Claude Code, and MCP-speaking clients) read the same enriched shape.
The provider closure must tolerate the findings runtime's mixed
resource key spaces: unified-derived findings reference the hashed
canonical resource ID, but Patrol guest inventory rows reference the
node-scoped Proxmox source ID (`instance:node:vmid`). Operator state
is keyed by canonical ID only, so a reference that misses the store
directly resolves through the registry (`GetByReference`, which also
covers retired canonical-ID eras and node-scoped guest references
after #1669) before the provider reports no state — without that
hop, maintenance windows and intentionally-offline intent silently
never reach guest findings. Pinned by
The provider (`Router.patrolResourceOperatorStateProvider`) must
tolerate the findings runtime's mixed resource key spaces:
unified-derived findings reference the hashed canonical resource ID,
Patrol guest inventory rows reference the node-scoped Proxmox source
ID (`instance:node:vmid`), and a finding raised before an operator
link folded its resource into another keeps the folded canonical ID,
which Patrol re-projects as stored. Operator state is keyed by
canonical ID only, so every reference resolves before the store read,
through the identity-only `ResolveCanonicalResourceID` of the
organization monitor's current read state
(`GetUnifiedReadStateOrSnapshot`: the published registry with saved
hosts that have not reported since a restart overlaid, or the mock
estate). That read state is the listing the resources API seeds its
registry from, and resolution covers retired canonical-ID eras,
node-scoped guest references after #1669 and link-folded IDs. Only a
reference it cannot place is read as given. Resolving first is what
keeps Patrol on the row the resources API writes for the same
reference: an exact read first would let a row left under a folded
ID, even a default-valued one, hide the maintenance window the
operator then set on the merged resource. Three cases still resolve
differently in the resources API, whose registry re-ingests that
listing: references to records it replays from supplemental providers
on top of the listing; manual links in mock mode, whose view applies
none; and a saved host that has not reported since a restart but is
linked into a live resource. The read state keeps that host as its own
offline row, because continuity records never join links, while the
resources API's ingest re-applies the link and folds it, so Patrol
reads the saved agent's own row there, as alert intent does for that
agent's canonical ID, and state set only on the link primary does not
reach the agent's findings until it reports again. The read
state answers from a registry it keeps (a continuity overlay is shared
and reused for up to two seconds while the published registry and the
saved hosts are unchanged), so findings do not each clone the unified
listing the way a resources-API registry build does; only a monitor
left without a resource store, which the router never does, builds a
snapshot registry per call. Pinned by
`TestPatrolFindingOperatorStateFollowsLinkFoldedReference` and
`TestPatrolFindingOperatorStateResolvesContinuityBackedHost` in
`internal/api/ai_handlers_more_test.go` and
`TestContract_FindingsResourceOperatorStateProviderIsWired` in
`internal/api/contract_test.go`.
@@ -2845,6 +2845,11 @@ the tenant resource store. Runtime reconciliation may visit every live monitor,
but each alert manager resolves policy through its own tenant-scoped store, so a
matching provider ID in another organization cannot import the mutation. The
existing route scopes and authenticated actor attribution remain unchanged.
Patrol's finding operator-state provider follows the same rule: it resolves a
finding's reference through the read state of that organization's own monitor
(`resolveMonitorForOrg`) and then reads that organization's store, so a
source-native or link-folded reference never resolves through another tenant's
registry.
### Secret-bearing configuration transfer fails closed before data access
@@ -3531,6 +3531,13 @@ in-memory provider already wired against the durable
commitments survive across restarts on the investigation read path
the same way they do on the suppression read path — both flow
through the same provider over the same durable table.
That provider resolves a finding's resource reference through the
organization monitor's current read state before it reads the table,
so a reference that read state folds into another resource through an
operator link reads the merged row. This changes which existing row is read,
not what is stored: no new persistence, migration or recovery path
opens, and a row left under a folded ID is neither rewritten nor
removed.
The agent SSE stream at `/api/agent/events` is in-memory and
stateless. No persistence; each connection starts fresh from the
@@ -6686,10 +6686,16 @@ separate.
`ResolveReferenceID` and `GetByReference` follow the fold as well, as the
folded agent's alert reference (`agent:<host ID>`) already did through the
merged row's canonical aliases. Alert intent, operator-state reads and
writes through the resources API and resource lookups by the folded ID
(`GET /api/resources/{id}`) therefore reach the merged row. Patrol's finding auto-acknowledgement
still reads a row stored under the exact reference first, so an
operator-state row kept under the folded ID applies there. History does
writes through the resources API, Patrol's finding operator-state
projection and resource lookups by the folded ID
(`GET /api/resources/{id}`) therefore reach the merged row. Patrol
resolves a finding's reference through the monitor's current read
state before it reads, so a finding raised before the link reads the
merged row's maintenance and monitoring mode, not an operator-state
row left under the folded ID, whenever that read state folds the
agent. A saved agent that has not reported since a restart is not
folded there (continuity records never join links), so its findings
read its own row until it reports again. History does
not follow the fold: a history binding persists, and
`expandHistoryAliases` walks bindings in both directions, so binding a
folded ID would join the two journals for good. Folded IDs are never
@@ -6707,6 +6713,9 @@ separate.
`TestResourceUnlinkSplitsManuallyLinkedPair` pins the API lookup: the
folded agent's ID answers with the VM while linked and with the agent's own
row once unlinked.
`internal/api/ai_handlers_more_test.go`
(`TestPatrolFindingOperatorStateFollowsLinkFoldedReference`) pins the
Patrol read against the resources API's write.
`registry_merge_policy_test.go` pins the record links in both directions
through two rebuilds and the live supplemental refresh, and checks that a
+193
View File
@@ -24,6 +24,7 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/monitoring"
"github.com/rcourtman/pulse-go-rewrite/internal/servicediscovery"
unifiedresources "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"
internalauth "github.com/rcourtman/pulse-go-rewrite/pkg/auth"
)
@@ -1101,3 +1102,195 @@ func TestDemoModeRouterRejectsAssistantSessionMutationsOverSafeMethods(t *testin
t.Fatalf("demo GET messages status = %d, want %d: %s", rec.Code, http.StatusOK, rec.Body.String())
}
}
// countingUnifiedSeedProvider serves a unified listing to the resources API
// and counts the listing clones taken from it.
type countingUnifiedSeedProvider struct {
listing func() ([]unifiedresources.Resource, time.Time)
clones int
}
func (p *countingUnifiedSeedProvider) ReadSnapshot() models.StateSnapshot {
return models.StateSnapshot{}
}
func (p *countingUnifiedSeedProvider) UnifiedResourceSnapshot() ([]unifiedresources.Resource, time.Time) {
p.clones++
return p.listing()
}
// putOperatorStateThroughAPI writes operator state the way the resource
// editor does and returns the canonical ID the resources API stored it under.
func putOperatorStateThroughAPI(t *testing.T, handlers *ResourceHandlers, ref string, body map[string]any) string {
t.Helper()
payload, _ := json.Marshal(body)
rec := httptest.NewRecorder()
handlers.HandleResourceOperatorState(rec, httptest.NewRequest(http.MethodPut, "/api/resources/"+ref+"/operator-state", strings.NewReader(string(payload))))
if rec.Code != http.StatusOK {
t.Fatalf("PUT %s operator state: %d %s", ref, rec.Code, rec.Body.String())
}
var written resourceOperatorStateAPI
if err := json.Unmarshal(rec.Body.Bytes(), &written); err != nil {
t.Fatalf("decode PUT %s: %v", ref, err)
}
return written.CanonicalID
}
func activeMaintenanceBody(now time.Time, reason string) map[string]any {
return map[string]any{
"monitoringMode": "normal",
"lifecycleState": "active",
"maintenanceStartAt": now.Add(-time.Minute).Format(time.RFC3339),
"maintenanceEndAt": now.Add(time.Hour).Format(time.RFC3339),
"maintenanceReason": reason,
}
}
// A Patrol finding raised before an operator linked its agent into a VM keeps
// the agent's canonical ID, and Patrol re-projects it with that reference. Its
// operator state must be the merged VM's, which the resources API writes and
// alert intent reads for the same reference, even though the agent kept a row
// of its own from before the link. Resolving finding references must not clone
// the unified listing.
func TestPatrolFindingOperatorStateFollowsLinkFoldedReference(t *testing.T) {
now := time.Now().UTC()
handlers := NewResourceHandlers(&config.Config{DataPath: t.TempDir()})
t.Cleanup(func() { _ = handlers.CloseStores() })
store, err := handlers.getStore("default")
if err != nil {
t.Fatalf("resource store: %v", err)
}
adapter := unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(store))
monitor := &monitoring.Monitor{}
monitor.SetResourceStore(adapter)
freshness := now
seed := &countingUnifiedSeedProvider{listing: func() ([]unifiedresources.Resource, time.Time) {
return adapter.GetAll(), freshness
}}
handlers.SetStateProvider(seed)
router := &Router{monitor: monitor, resourceHandlers: handlers}
publish := func(at time.Time) {
freshness = at
adapter.PopulateSnapshotAndSupplemental(models.StateSnapshot{
Hosts: []models.Host{{ID: "host-app-guest", Hostname: "app-guest", MachineID: "machine-app-guest", Status: "online", LastSeen: now}},
LastUpdate: at,
}, map[unifiedresources.DataSource][]unifiedresources.IngestRecord{
unifiedresources.SourceVMware: {{
SourceID: "vc-1:vm:vm-42",
Resource: unifiedresources.Resource{
Type: unifiedresources.ResourceTypeVM,
Technology: "vmware",
Name: "app-guest",
Status: unifiedresources.StatusOnline,
LastSeen: now,
VMware: &unifiedresources.VMwareData{ConnectionID: "vc-1", ManagedObjectID: "vm-42", EntityType: "vm"},
},
Identity: unifiedresources.ResourceIdentity{Hostnames: []string{"app-guest"}},
}},
})
}
publish(now)
agentID := unifiedresources.MachineIdentityCanonicalID(unifiedresources.ResourceTypeAgent, "machine-app-guest")
vmID := ""
for _, resource := range adapter.GetAll() {
if resource.VMware != nil {
vmID = resource.ID
}
}
if vmID == "" {
t.Fatal("published estate has no vSphere VM")
}
// The agent's own row predates the link and holds only defaults.
if got := putOperatorStateThroughAPI(t, handlers, agentID, map[string]any{"monitoringMode": "normal", "lifecycleState": "active"}); got != agentID {
t.Fatalf("pre-link write landed on %q, want the agent %q", got, agentID)
}
if err := store.AddLink(unifiedresources.ResourceLink{ResourceA: vmID, ResourceB: agentID, PrimaryID: vmID}); err != nil {
t.Fatalf("add link: %v", err)
}
// Identity pins persist on the first rebuild; damage from them shows on the second.
publish(now.Add(time.Second))
publish(now.Add(2 * time.Second))
if resolved, ok := adapter.ResolveCanonicalResourceID(agentID); !ok || resolved != vmID {
t.Fatalf("published registry resolves folded agent %s to %q (ok=%v), want VM %s", agentID, resolved, ok, vmID)
}
// The operator puts the merged row into maintenance through the folded ID.
if got := putOperatorStateThroughAPI(t, handlers, agentID, activeMaintenanceBody(now, "moving hosts")); got != vmID {
t.Fatalf("maintenance written by folded ID landed on %q, want merged VM %q", got, vmID)
}
// A reference no registry resolves keeps its own row.
if got := putOperatorStateThroughAPI(t, handlers, "agent:retired-host", map[string]any{"monitoringMode": "muted", "lifecycleState": "active"}); got != "agent:retired-host" {
t.Fatalf("unresolved write landed on %q", got)
}
seed.clones = 0
provider := router.patrolResourceOperatorStateProvider("default")
for _, ref := range []string{agentID, vmID, "vc-1:vm:vm-42"} {
projection, ok := provider.OperatorStateProjection(ref, now)
if !ok || projection.MaintenanceWindow == nil || projection.MaintenanceWindow.Reason != "moving hosts" {
t.Fatalf("finding on %s projected %+v (ok=%v), want merged VM %s's maintenance window", ref, projection, ok, vmID)
}
}
projection, ok := provider.OperatorStateProjection("agent:retired-host", now)
if !ok || projection.MonitoringMode != "muted" {
t.Fatalf("unresolved reference projected %+v (ok=%v), want its own muted row", projection, ok)
}
if seed.clones != 0 {
t.Fatalf("finding projection cloned the unified listing %d times; it must resolve through the monitor's read state", seed.clones)
}
}
// After a restart, a host that has not reported again is listed only through
// saved-host continuity. A Patrol finding naming it by its agent reference
// must read the operator state the resources API wrote for that reference,
// still without cloning the unified listing per finding.
func TestPatrolFindingOperatorStateResolvesContinuityBackedHost(t *testing.T) {
now := time.Now().UTC()
monitorCfg := &config.Config{DataPath: t.TempDir()}
seedMonitor, err := monitoring.New(monitorCfg)
if err != nil {
t.Fatalf("monitoring.New seed monitor: %v", err)
}
t.Cleanup(seedMonitor.Stop)
if _, err := seedMonitor.ApplyHostReport(agentshost.Report{
Agent: agentshost.AgentInfo{ID: "agent-1", Version: "6.0.0-rc.1", IntervalSeconds: 30},
Host: agentshost.HostInfo{ID: "machine-1", MachineID: "machine-1", Hostname: "host-1.local", Platform: "linux"},
Timestamp: now,
}, &config.APITokenRecord{ID: "token-1", Name: "Token One"}); err != nil {
t.Fatalf("ApplyHostReport seed continuity: %v", err)
}
monitor, err := monitoring.New(monitorCfg)
if err != nil {
t.Fatalf("monitoring.New restarted monitor: %v", err)
}
t.Cleanup(monitor.Stop)
handlers := NewResourceHandlers(&config.Config{DataPath: t.TempDir()})
t.Cleanup(func() { _ = handlers.CloseStores() })
store, err := handlers.getStore("default")
if err != nil {
t.Fatalf("resource store: %v", err)
}
monitor.SetResourceStore(unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(store)))
seed := &countingUnifiedSeedProvider{listing: monitor.UnifiedResourceSnapshot}
handlers.SetStateProvider(seed)
router := &Router{monitor: monitor, resourceHandlers: handlers}
const ref = "agent:machine-1"
canonicalID := putOperatorStateThroughAPI(t, handlers, ref, activeMaintenanceBody(now, "disk swap"))
if canonicalID == ref {
t.Fatalf("resources API did not resolve continuity host reference %s", ref)
}
seed.clones = 0
projection, ok := router.patrolResourceOperatorStateProvider("default").OperatorStateProjection(ref, now)
if !ok || projection.MaintenanceWindow == nil || projection.MaintenanceWindow.Reason != "disk swap" {
t.Fatalf("finding on %s projected %+v (ok=%v), want the maintenance window the API stored on %s", ref, projection, ok, canonicalID)
}
if seed.clones != 0 {
t.Fatalf("finding projection cloned the unified listing %d times; it must resolve through the monitor's read state", seed.clones)
}
}
+2 -2
View File
@@ -22956,8 +22956,8 @@ func TestContract_FindingsResourceOperatorStateProviderIsWired(t *testing.T) {
if !strings.Contains(src, "GetResourceOperatorState(canonicalID)") {
t.Error("router.go must read state from the unified store's canonical accessor")
}
if !strings.Contains(src, "GetByReference(resourceRef)") {
t.Error("router.go must resolve non-canonical finding resource references (Patrol guest rows carry node-scoped source IDs) through the registry before reporting no operator state")
if !strings.Contains(src, "resolver.ResolveCanonicalResourceID(resourceRef)") {
t.Error("router.go must resolve finding resource references (Patrol guest rows carry node-scoped source IDs; retained findings keep link-folded IDs) through the monitor's published resolver before reading operator state")
}
if !strings.Contains(src, "IntentionallyOffline: state.IntentionallyOffline") {
t.Error("router.go must propagate state.IntentionallyOffline through the projection so the findings runtime sees it")
+61 -58
View File
@@ -2213,6 +2213,66 @@ func unifiedFindingFromAI(f *ai.Finding) *unified.UnifiedFinding {
}
}
// canonicalResourceIDResolver is the identity-only reference resolver a
// monitor's unified read state exposes.
type canonicalResourceIDResolver interface {
ResolveCanonicalResourceID(ref string) (string, bool)
}
// patrolResourceOperatorStateProvider projects durable operator state onto the
// resource a Patrol finding names. Findings carry whatever resource ID their
// producer used: unified-derived findings hold the canonical hash, Patrol
// guest inventory rows hold the node-scoped Proxmox source ID, and a finding
// raised before an operator link folded its resource into another keeps the
// folded ID. Operator state is keyed by canonical ID, so the reference
// resolves before any read, as the resources API resolves it; reading the
// exact reference first would let a row left under a folded ID hide the merged
// resource's maintenance window. The resolver is the monitor's current read
// state (its published registry with saved hosts that have not reported since
// a restart overlaid, or the mock estate), the listing the resources API seeds
// from. It answers from a registry it keeps (a continuity overlay is shared
// and reused for up to two seconds while nothing it depends on changes), so
// findings do not each clone the unified listing. Only a reference it cannot
// resolve is read as given.
func (r *Router) patrolResourceOperatorStateProvider(orgID string) ai.ResourceOperatorStateProvider {
return ai.ResourceOperatorStateProviderFunc(
func(resourceRef string, now time.Time) (ai.ResourceOperatorStateProjection, bool) {
orgStore, lookupErr := r.resourceHandlers.getStore(orgID)
if lookupErr != nil {
return ai.ResourceOperatorStateProjection{}, false
}
canonicalID := resourceRef
if resolver, ok := r.resolveMonitorForOrg(orgID).GetUnifiedReadStateOrSnapshot().(canonicalResourceIDResolver); ok {
if resolvedID, resolved := resolver.ResolveCanonicalResourceID(resourceRef); resolved {
canonicalID = resolvedID
}
}
state, found, fetchErr := orgStore.GetResourceOperatorState(canonicalID)
if fetchErr != nil || !found {
return ai.ResourceOperatorStateProjection{}, false
}
projection := ai.ResourceOperatorStateProjection{
IntentionallyOffline: state.IntentionallyOffline,
MonitoringMode: string(state.MonitoringMode),
LifecycleState: string(state.LifecycleState),
NeverAutoRemediate: state.NeverAutoRemediate,
Criticality: string(state.Criticality),
}
if state.LifecycleState == unifiedresources.LifecycleStateRetired {
projection.NeverAutoRemediate = true
}
if state.IsInMaintenanceAt(now) {
projection.MaintenanceWindow = &ai.ResourceOperatorStateMaintenanceWindow{
StartAt: *state.MaintenanceStartAt,
EndAt: *state.MaintenanceEndAt,
Reason: state.MaintenanceReason,
}
}
return projection, true
},
)
}
func (r *Router) startPatrolForContext(ctx context.Context, orgID string) bool {
if r == nil || r.aiSettingsHandler == nil {
return false
@@ -2480,64 +2540,7 @@ func (r *Router) startPatrolForContext(ctx context.Context, orgID string) bool {
// signals later does not multiply round-trips per finding.
// Keeps `internal/ai` clear of an
// `internal/unifiedresources` import.
patrol.GetFindings().SetResourceOperatorStateProvider(
ai.ResourceOperatorStateProviderFunc(
func(resourceRef string, now time.Time) (ai.ResourceOperatorStateProjection, bool) {
orgStore, lookupErr := r.resourceHandlers.getStore(orgID)
if lookupErr != nil {
return ai.ResourceOperatorStateProjection{}, false
}
// Findings carry whatever resource ID their
// producer used: unified-derived findings hold
// the canonical hash, but Patrol guest inventory
// rows hold the node-scoped Proxmox source ID.
// Operator state is keyed by canonical ID only,
// so a reference that misses directly resolves
// through the registry before giving up —
// otherwise maintenance windows and
// intentionally-offline intent silently never
// reach guest findings.
canonicalID := resourceRef
state, found, fetchErr := orgStore.GetResourceOperatorState(canonicalID)
if fetchErr != nil {
return ai.ResourceOperatorStateProjection{}, false
}
if !found {
registry, registryErr := r.resourceHandlers.buildRegistry(orgID)
if registryErr != nil || registry == nil {
return ai.ResourceOperatorStateProjection{}, false
}
_, resolvedID, resolved := registry.GetByReference(resourceRef)
if !resolved || resolvedID == canonicalID {
return ai.ResourceOperatorStateProjection{}, false
}
canonicalID = resolvedID
state, found, fetchErr = orgStore.GetResourceOperatorState(canonicalID)
}
if fetchErr != nil || !found {
return ai.ResourceOperatorStateProjection{}, false
}
projection := ai.ResourceOperatorStateProjection{
IntentionallyOffline: state.IntentionallyOffline,
MonitoringMode: string(state.MonitoringMode),
LifecycleState: string(state.LifecycleState),
NeverAutoRemediate: state.NeverAutoRemediate,
Criticality: string(state.Criticality),
}
if state.LifecycleState == unifiedresources.LifecycleStateRetired {
projection.NeverAutoRemediate = true
}
if state.IsInMaintenanceAt(now) {
projection.MaintenanceWindow = &ai.ResourceOperatorStateMaintenanceWindow{
StartAt: *state.MaintenanceStartAt,
EndAt: *state.MaintenanceEndAt,
Reason: state.MaintenanceReason,
}
}
return projection, true
},
),
)
patrol.GetFindings().SetResourceOperatorStateProvider(r.patrolResourceOperatorStateProvider(orgID))
// Wire push notifications: patrol findings → relay client (best-effort)
patrol.SetPushNotifyCallback(func(n relay.PushNotificationPayload) {