diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md index d798a3318..3f9518191 100644 --- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md +++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md @@ -917,6 +917,15 @@ profile and assignment columns, but embedded table framing must route through ## Current State +The router projection-builder (`internal/api/router.go`) that wires +the operator-state provider into the findings runtime now also +populates `NeverAutoRemediate` on the projection. The investigation +runtime reads the same projection to attach `OperatorContext` to +findings handed to the orchestrator, so investigation reasoning sees +the same lock-against-remediation flag that the action broker +enforces downstream — no possible drift between "what Patrol +proposes" and "what the broker accepts." + `/api/agent/events` is the SSE stream agents subscribe to for real-time notifications (`finding.created` events plus a 15-second heartbeat keepalive). The broadcaster drops events for slow diff --git a/docs/release-control/v6/internal/subsystems/ai-runtime.md b/docs/release-control/v6/internal/subsystems/ai-runtime.md index 2f4651070..825a4871e 100644 --- a/docs/release-control/v6/internal/subsystems/ai-runtime.md +++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md @@ -591,6 +591,29 @@ call (active maintenance window plus the `IntentionallyOffline` flag) so adding new signals later does not multiply round-trips per finding. +The investigation runtime now hands the orchestrator a Finding +pre-enriched with operator-set state and operational memory. +`MaybeInvestigateFinding` (in `internal/ai/patrol_findings.go`) +calls `f.ToCoreFinding()` then attaches a +`FindingOperatorContext` from the in-memory operator-state +projection (intentionally offline, never auto-remediate, active +maintenance window) and a `FindingOperationalMemory` projection +(regression count, previous resolved fix summary, times raised) +populated from fields the internal `Finding` already carries. The +orchestrator (in pulse-pro) consumes these fields when reasoning +about the next move — it does not need a separate read to get +the situated picture, and it can avoid proposing fixes the +operator has locked the resource against. + +`ResourceOperatorStateProjection` carries `NeverAutoRemediate` +alongside `IntentionallyOffline` and `MaintenanceWindow` so the +investigation read path and the suppression read path share a +single projection. The findings store exposes the projection via +`OperatorStateProjectionFor`; the suppression hot path keeps its +existing internal access. Both paths see the same operator-state +facts so investigation reasoning and suppression behavior cannot +drift against each other. + A cross-slice consequence worth pinning: operator-state-suppressed findings (auto-dismissed with `DismissedReason="expected_behavior"` and `operator_state_cause` metadata) are also ineligible for diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md index ec44190c3..77e1f7d5c 100644 --- a/docs/release-control/v6/internal/subsystems/api-contracts.md +++ b/docs/release-control/v6/internal/subsystems/api-contracts.md @@ -1343,6 +1343,17 @@ the canonical monitored-system blocked payload. ## Current State +`aicontracts.Finding` (the shape Patrol hands the investigation +orchestrator) carries optional `OperatorContext` and +`OperationalMemory` projections. The router-side wire-up populates +the operator-state projection with `NeverAutoRemediate` alongside +`IntentionallyOffline` and the maintenance-window block, and the +investigation runtime in `internal/ai/patrol_findings.go` attaches +the projection to the Finding before calling the orchestrator. This +is the one in-process write path that decides what the orchestrator +sees about operator commitments; all consumers (in-process Patrol, +external Claude Code, future MCP) read the same enriched shape. + `/api/agent/events` is the agent SSE stream — the substrate piece that closes the agent-paradigm triangle (discovery + bundled reads + push notifications). Agents subscribe once and receive real-time diff --git a/docs/release-control/v6/internal/subsystems/patrol-intelligence.md b/docs/release-control/v6/internal/subsystems/patrol-intelligence.md index b8f20c081..ab9f41bd2 100644 --- a/docs/release-control/v6/internal/subsystems/patrol-intelligence.md +++ b/docs/release-control/v6/internal/subsystems/patrol-intelligence.md @@ -1198,6 +1198,21 @@ the other. The section uses `createNonSuspendingQuery` rather than `createResource` so the drawer's parent Suspense boundary does not flicker the page-level fallback while operator state is in flight. +The investigation runtime hands the orchestrator a finding +pre-enriched with the operator-set state and operational memory it +needs to reason from. The `aicontracts.Finding` shape carries an +optional `OperatorContext` (intentionally offline, never +auto-remediate, maintenance window) and an `OperationalMemory` +projection (regression count, previous resolved fix summary, times +raised); `MaybeInvestigateFinding` populates both from the +in-process findings store and the operator-state provider before +calling the orchestrator. This is the data path that converts +"Pulse holds privileged context" into "Pulse uses privileged +context" — the orchestrator (in pulse-pro) reads the fields when +formatting its system prompt, so investigations on locked or +under-maintenance resources reason from the operator's +commitments instead of contradicting them. + The findings store also consumes per-resource operator-set state via the narrow `ResourceOperatorStateProvider` interface installed by `SetResourceOperatorStateProvider`. The API layer wires an adapter diff --git a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md index fcc706b53..736d5f41a 100644 --- a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md +++ b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md @@ -506,6 +506,15 @@ shell clickable behind another overlay. ## Current State +The investigation enrichment path in `MaybeInvestigateFinding` +adds at most one operator-state projection lookup per investigation +(in-memory via the existing provider) and zero additional reads — +`OperationalMemory` is built from fields the internal Finding +already carries. Investigation enrichment does not multiply +per-finding cost as the operator-state surface grows; the projection +serves both suppression and investigation reads from a single +shape. + The agent SSE stream at `/api/agent/events` keeps publishers non-blocking by design: the broadcaster's per-subscriber buffer is bounded (64 events) and a full buffer drops the event for that diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md index 1922c9c5c..37e5412ef 100644 --- a/docs/release-control/v6/internal/subsystems/storage-recovery.md +++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md @@ -947,6 +947,13 @@ bypass the API fail-closed execution gate. ## Current State +The investigation enrichment path reads operator-state from the +in-memory provider already wired against the durable +`resource_operator_state` SQLite table, so an operator's +commitments survive across restarts on the investigation read path +the same way they do on the suppression read path — both flow +through the same provider over the same durable table. + The agent SSE stream at `/api/agent/events` is in-memory and stateless. No persistence; each connection starts fresh from the moment of subscribe. Agents that need to catch up across reconnects diff --git a/internal/ai/findings.go b/internal/ai/findings.go index c91b9eb01..069a75462 100644 --- a/internal/ai/findings.go +++ b/internal/ai/findings.go @@ -567,8 +567,15 @@ func (f *Finding) CanRetryInvestigation() bool { // ToCoreFinding converts to the shared InvestigationFinding type used by the // investigation orchestrator. Centralised here so a field added to // InvestigationFinding causes a compile error in exactly one place. +// +// The OperationalMemory projection is populated unconditionally from +// fields the internal Finding already carries (regression count, +// previous resolved fix). OperatorContext is NOT populated here +// because that data lives in the unified-resources store, which this +// package does not import. Callers that have access to the operator +// state attach it via WithOperatorContext after conversion. func (f *Finding) ToCoreFinding() *InvestigationFinding { - return &InvestigationFinding{ + core := &InvestigationFinding{ ID: f.ID, Key: f.Key, Severity: string(f.Severity), @@ -586,6 +593,23 @@ func (f *Finding) ToCoreFinding() *InvestigationFinding { LastInvestigatedAt: f.LastInvestigatedAt, InvestigationAttempts: f.InvestigationAttempts, } + + // Populate operational memory from fields the internal Finding + // already accumulates. Only attach when there is something + // meaningful to say — a fresh finding (no regressions, no prior + // fix, no raises beyond 1) leaves the field nil so the + // orchestrator can treat absence as "no prior history" rather + // than parsing a zero-valued struct. + if f.RegressionCount > 0 || f.PreviousResolvedFixSummary != "" || f.TimesRaised > 1 { + core.OperationalMemory = &aicontracts.FindingOperationalMemory{ + RegressionCount: f.RegressionCount, + LastRegressionAt: f.LastRegressionAt, + PreviousResolvedFixSummary: f.PreviousResolvedFixSummary, + TimesRaised: f.TimesRaised, + } + } + + return core } // Getter methods for aicontracts.OrchestratorAIFinding interface @@ -692,6 +716,14 @@ type ResourceOperatorStateMaintenanceWindow struct { type ResourceOperatorStateProjection struct { MaintenanceWindow *ResourceOperatorStateMaintenanceWindow IntentionallyOffline bool + // NeverAutoRemediate is the operator's "do not act on this + // resource" flag. The findings store does not consume it during + // suppression (that's the action broker's concern), but the + // investigation runtime needs it so the orchestrator can avoid + // proposing fixes that the broker would refuse. Carrying it on + // the same projection means the suppression and investigation + // hot paths share a single read. + NeverAutoRemediate bool } // ResourceOperatorStateProvider is the narrow interface the findings @@ -740,6 +772,23 @@ func (s *FindingsStore) SetResourceOperatorStateProvider(p ResourceOperatorState s.resourceOperatorStateProvider = p } +// OperatorStateProjectionFor exposes the operator-state projection for +// a resource so callers outside the suppression hot path (e.g. the +// patrol investigation runtime, when it's about to hand a finding to +// the orchestrator) can attach the same per-resource intent the +// store uses internally. Returns false when no provider is wired or +// when the provider reports no state for the resource — callers +// branch on the bool rather than parsing zero-valued projections. +func (s *FindingsStore) OperatorStateProjectionFor(resourceID string, now time.Time) (ResourceOperatorStateProjection, bool) { + s.mu.RLock() + provider := s.resourceOperatorStateProvider + s.mu.RUnlock() + if provider == nil || resourceID == "" { + return ResourceOperatorStateProjection{}, false + } + return provider.OperatorStateProjection(resourceID, now) +} + // SetPersistence sets the persistence layer and loads existing findings func (s *FindingsStore) SetPersistence(p FindingsPersistence) error { s.mu.Lock() diff --git a/internal/ai/findings_test.go b/internal/ai/findings_test.go index f66140109..33fc13d9f 100644 --- a/internal/ai/findings_test.go +++ b/internal/ai/findings_test.go @@ -2109,3 +2109,116 @@ func TestFindingsStore_OperatorStateSuppressedFindingsDoNotTriggerInvestigation( }) } } + +func TestToCoreFinding_AttachesOperationalMemoryWhenHistoryExists(t *testing.T) { + // The orchestrator (in pulse-pro) reads OperationalMemory to know + // "what we already know" about this finding — regression count, + // previous fix that worked, total times raised. Without it, the + // LLM proposing a fix has no operational context and may suggest + // something that has already been tried and failed. Pin the + // projection so the in-repo data path stays wired. + regressionTime := time.Date(2026, 5, 1, 10, 0, 0, 0, time.UTC) + f := &Finding{ + ID: "f-history", + ResourceID: "vm:101", + Severity: FindingSeverityWarning, + Title: "Disk pressure", + RegressionCount: 3, + LastRegressionAt: ®ressionTime, + PreviousResolvedFixSummary: "Free 200GB before backup window", + TimesRaised: 7, + } + + core := f.ToCoreFinding() + if core.OperationalMemory == nil { + t.Fatal("OperationalMemory must be attached when the finding has prior history") + } + if core.OperationalMemory.RegressionCount != 3 { + t.Errorf("regression count: got %d want 3", core.OperationalMemory.RegressionCount) + } + if core.OperationalMemory.PreviousResolvedFixSummary != "Free 200GB before backup window" { + t.Errorf("previous fix summary must round-trip; got %q", core.OperationalMemory.PreviousResolvedFixSummary) + } + if core.OperationalMemory.TimesRaised != 7 { + t.Errorf("times raised: got %d want 7", core.OperationalMemory.TimesRaised) + } + if core.OperationalMemory.LastRegressionAt == nil || !core.OperationalMemory.LastRegressionAt.Equal(regressionTime) { + t.Errorf("last regression timestamp must round-trip; got %v", core.OperationalMemory.LastRegressionAt) + } +} + +func TestToCoreFinding_LeavesOperationalMemoryNilForFreshFindings(t *testing.T) { + // A fresh finding (first detection, no regressions, no prior fix) + // must NOT carry an OperationalMemory — the orchestrator's + // reasoning should treat absence as "no prior history" rather + // than parsing a zero-valued struct. + f := &Finding{ + ID: "f-fresh", + ResourceID: "vm:102", + Severity: FindingSeverityWarning, + Title: "CPU saturated", + TimesRaised: 1, // first raise — not regression history + } + + core := f.ToCoreFinding() + if core.OperationalMemory != nil { + t.Errorf("OperationalMemory must be nil for fresh findings; got %+v", core.OperationalMemory) + } +} + +func TestFindingsStore_OperatorStateProjectionFor_ReturnsFalseWithoutProvider(t *testing.T) { + // Default store with no provider wired must report false so + // callers can branch on absence rather than parsing zero-valued + // projections. + store := NewFindingsStore() + _, ok := store.OperatorStateProjectionFor("vm:101", time.Now()) + if ok { + t.Error("expected false when no provider is wired") + } +} + +func TestFindingsStore_OperatorStateProjectionFor_ProxiesProvider(t *testing.T) { + // When a provider is wired, OperatorStateProjectionFor must + // surface the same projection the suppression hot path sees. + // The investigation runtime relies on this to attach + // OperatorContext to findings handed to the orchestrator — + // drift between this read path and the suppression read path + // would mean the orchestrator reasons against different facts + // than the suppression branch. + store := NewFindingsStore() + expected := ResourceOperatorStateProjection{ + IntentionallyOffline: true, + NeverAutoRemediate: true, + } + store.SetResourceOperatorStateProvider( + ResourceOperatorStateProviderFunc(func(_ string, _ time.Time) (ResourceOperatorStateProjection, bool) { + return expected, true + }), + ) + + got, ok := store.OperatorStateProjectionFor("vm:locked", time.Now()) + if !ok { + t.Fatal("expected projection from wired provider") + } + if got.IntentionallyOffline != expected.IntentionallyOffline { + t.Errorf("IntentionallyOffline must round-trip; got %v want %v", got.IntentionallyOffline, expected.IntentionallyOffline) + } + if got.NeverAutoRemediate != expected.NeverAutoRemediate { + t.Errorf("NeverAutoRemediate must round-trip — investigation runtime reads this projection too; got %v want %v", got.NeverAutoRemediate, expected.NeverAutoRemediate) + } +} + +func TestFindingsStore_OperatorStateProjectionFor_RejectsEmptyResourceID(t *testing.T) { + // Defensive: empty resource ID short-circuits to false rather + // than calling the provider with a useless query. + store := NewFindingsStore() + store.SetResourceOperatorStateProvider( + ResourceOperatorStateProviderFunc(func(_ string, _ time.Time) (ResourceOperatorStateProjection, bool) { + t.Fatal("provider must not be called with empty resource id") + return ResourceOperatorStateProjection{}, false + }), + ) + if _, ok := store.OperatorStateProjectionFor("", time.Now()); ok { + t.Error("expected false on empty resource id") + } +} diff --git a/internal/ai/patrol_findings.go b/internal/ai/patrol_findings.go index 9dcb6bb09..43a66dae3 100644 --- a/internal/ai/patrol_findings.go +++ b/internal/ai/patrol_findings.go @@ -1216,6 +1216,42 @@ func (p *PatrolService) MaybeInvestigateFinding(f *Finding) { // Convert Finding to shared finding type for the investigation orchestrator invFinding := f.ToCoreFinding() + // Attach the operator-set state for the finding's resource so the + // orchestrator's reasoning can incorporate the operator's + // commitments (intentionally offline, never auto-remediate, active + // maintenance window). Without this, Patrol can propose fixes that + // contradict the operator's intent — the action broker refuses + // them downstream (slice 33's `resource_remediation_locked:`), but + // the proposal shouldn't have happened in the first place. The + // projection is also the data path "Pulse uses the privileged + // context" that defines the product's differentiation. + if p.findings != nil { + now := time.Now() + if projection, ok := p.findings.OperatorStateProjectionFor(f.ResourceID, now); ok { + ctx := &aicontracts.FindingOperatorContext{ + IntentionallyOffline: projection.IntentionallyOffline, + NeverAutoRemediate: projection.NeverAutoRemediate, + } + if window := projection.MaintenanceWindow; window != nil { + ctx.MaintenanceWindowActive = !now.Before(window.StartAt) && now.Before(window.EndAt) + start := window.StartAt + end := window.EndAt + ctx.MaintenanceStartAt = &start + ctx.MaintenanceEndAt = &end + ctx.MaintenanceReason = window.Reason + } + // Attach only when something meaningful is set — + // otherwise leave the field nil so the orchestrator can + // branch on absence rather than on zero values. + if ctx.IntentionallyOffline || + ctx.NeverAutoRemediate || + ctx.MaintenanceStartAt != nil || + ctx.MaintenanceWindowActive { + invFinding.OperatorContext = ctx + } + } + } + // Trigger investigation in background with a timeout to prevent indefinite runs. // Track with WaitGroup so graceful shutdown can wait for completion. p.investigationWg.Add(1) diff --git a/internal/api/contract_test.go b/internal/api/contract_test.go index c7a77f489..306061ab5 100644 --- a/internal/api/contract_test.go +++ b/internal/api/contract_test.go @@ -13233,6 +13233,24 @@ func assertJSONSnapshot(t *testing.T, got []byte, want string) { } } +// TestContract_InvestigationProjectionCarriesNeverAutoRemediate pins +// the cross-path agreement: the projection-builder closure in +// router.go must populate NeverAutoRemediate on the projection so +// the investigation read path sees the same lock-against-remediation +// flag the action broker enforces. Without this, Patrol could +// propose fixes on locked resources that the broker would refuse — +// the fix proposal shouldn't have happened in the first place. +func TestContract_InvestigationProjectionCarriesNeverAutoRemediate(t *testing.T) { + source, err := os.ReadFile("router.go") + if err != nil { + t.Fatalf("read router.go: %v", err) + } + src := string(source) + if !strings.Contains(src, "NeverAutoRemediate: state.NeverAutoRemediate,") { + t.Error("router.go must populate NeverAutoRemediate on the projection so the investigation read path sees the operator's lock — drift here means Patrol can propose fixes the broker refuses") + } +} + // TestContract_AgentEventsStreamPublishesOnFindingCreated pins the // agent SSE stream's most consequential producer hook: the // findings-runtime callback in router.go must publish a diff --git a/internal/api/router.go b/internal/api/router.go index a84b556bd..840126a09 100644 --- a/internal/api/router.go +++ b/internal/api/router.go @@ -1842,6 +1842,7 @@ func (r *Router) startPatrolForContext(ctx context.Context, orgID string) bool { } projection := ai.ResourceOperatorStateProjection{ IntentionallyOffline: state.IntentionallyOffline, + NeverAutoRemediate: state.NeverAutoRemediate, } if state.IsInMaintenanceAt(now) { projection.MaintenanceWindow = &ai.ResourceOperatorStateMaintenanceWindow{ diff --git a/pkg/aicontracts/investigation.go b/pkg/aicontracts/investigation.go index df622dbf4..8bb970640 100644 --- a/pkg/aicontracts/investigation.go +++ b/pkg/aicontracts/investigation.go @@ -75,6 +75,62 @@ type Finding struct { InvestigationOutcome string `json:"investigation_outcome,omitempty"` LastInvestigatedAt *time.Time `json:"last_investigated_at,omitempty"` InvestigationAttempts int `json:"investigation_attempts"` + + // OperatorContext carries the operator's per-resource intent the + // investigation runtime should respect when reasoning about this + // finding. nil when no operator-set state has been recorded for + // the resource. The orchestrator (in pulse-pro) consumes this to + // avoid proposing fixes that contradict operator commitments — + // e.g. if NeverAutoRemediate is set, propose investigation-only + // outputs instead of remediation actions; if a maintenance + // window is active, frame the finding as "expected during + // maintenance" rather than urgent. + OperatorContext *FindingOperatorContext `json:"operator_context,omitempty"` + + // OperationalMemory carries the regression and fix history Pulse + // already accumulated for this finding's identity. It exists to + // give the orchestrator's reasoning a "what we already know" block + // without it having to query the findings store separately. nil + // when there is no prior history (fresh finding). + OperationalMemory *FindingOperationalMemory `json:"operational_memory,omitempty"` +} + +// FindingOperatorContext is the orchestrator-facing projection of +// operator-set state for the finding's resource. Mirrors the shape of +// `internal/api/agent_resource_context.go`'s +// AgentResourceOperatorState so external agents and the in-process +// orchestrator see the same field names. +type FindingOperatorContext struct { + IntentionallyOffline bool `json:"intentionally_offline"` + NeverAutoRemediate bool `json:"never_auto_remediate"` + MaintenanceStartAt *time.Time `json:"maintenance_start_at,omitempty"` + MaintenanceEndAt *time.Time `json:"maintenance_end_at,omitempty"` + MaintenanceReason string `json:"maintenance_reason,omitempty"` + Criticality string `json:"criticality,omitempty"` + Note string `json:"note,omitempty"` + MaintenanceWindowActive bool `json:"maintenance_window_active"` +} + +// FindingOperationalMemory bundles the regression-and-fix history +// the orchestrator should reason from when proposing the next move. +// All fields zero/empty mean "no operational memory" — the +// orchestrator should treat the finding as fresh. +type FindingOperationalMemory struct { + // RegressionCount is the number of times this finding has been + // resolved and re-detected. >0 means "this is not a one-off." + RegressionCount int `json:"regression_count"` + // LastRegressionAt is the most recent regression timestamp. + LastRegressionAt *time.Time `json:"last_regression_at,omitempty"` + // PreviousResolvedFixSummary is the description of the proposed + // fix that resolved the finding the last time it was active. + // Captured at regression time from the prior investigation + // record. Operator memory: "what worked last time." + PreviousResolvedFixSummary string `json:"previous_resolved_fix_summary,omitempty"` + // TimesRaised is the total raise count across all detections. + // Distinct from RegressionCount because it includes + // re-detections while still active (not just regressions + // after resolution). + TimesRaised int `json:"times_raised"` } // ---------------------------------------------------------------------------