Run validated local Patrol observers

This commit is contained in:
rcourtman
2026-08-14 00:45:22 +01:00
parent 71a0fa37a1
commit ff15c91274
10 changed files with 764 additions and 30 deletions
@@ -6970,3 +6970,30 @@ Governed Patrol fixes are delivered through the approved-action pipeline and
are already counted there (`pulse_intelligence_approved_action_*`), which is the
signal to read for remediation activity. Existing persisted history files that
still contain `auto_fix_count` load cleanly; the unknown key is ignored.
### Model-authored observers have a core-owned local execution floor
`internal/ai/patrol_observer_runtime.go` owns the first executable observer ABI
for retained Patrol objectives. The model may author the predicate, but it
cannot author executable code or advance lifecycle authority. Core strictly
decodes `pulse-resource-state/v1`, requires explicit canonical resource scope,
an interval trigger, an empty external-requirements object, one canonical
`status` predicate, a 10-to-300-second local sample interval, and a bounded
consecutive-failure window. Unknown fields and unsupported runtimes, triggers,
requirements, paths, operators, or values fail closed and persist a safe
machine reason in the `rejected` state while coverage remains uncovered.
Accepted observers transition through proposed, validated, and installed under
core authority. They evaluate Pulse's canonical `ReadState` status locally,
falling back to the existing Patrol snapshot only where canonical read state is
unavailable. Health leasing is persisted without changing the operator
objective revision, with all leases due in one sweep committed in one atomic
encrypted-document transaction. A failure transition queues one scoped
`objective_evidence` check through the existing TriggerManager, so the model
wakes for reasoning only after local evidence crosses the declared window and
does not become the polling loop. The observer remains read-only and cannot
enter infrastructure mutation; any later response still uses the canonical
capability, autonomy, approval, dispatch, audit, and verification kernel.
Changing the retained brief, optional context, or resource scope disables the
installed observer and clears its lease; coverage cannot survive a semantic
change to the intent it was designed to protect.
@@ -128,9 +128,10 @@ with bounded provenance and rollback; app-named integration code is an
optimization only when repeated evidence justifies productizing it, never a
prerequisite for a new objective.
The canonical retained-intent and proposal slices are implemented by
The canonical retained-intent, proposal, and first local execution slices are implemented by
`internal/ai/patrol_objectives.go`, `internal/ai/patrol_observer_builder.go`,
`internal/ai/tools/tools_patrol.go`, and `internal/api/patrol_objectives.go`.
`internal/ai/patrol_observer_runtime.go`, `internal/ai/tools/tools_patrol.go`, and
`internal/api/patrol_objectives.go`.
They provide encrypted retained intent, optimistic revisions, resource scoping,
model seed projection, the core-owned observer state machine, and the
Patrol-detection-only `patrol_propose_observer` builder. The model supplies a
@@ -139,10 +140,34 @@ declared requirements, and one trigger kind; core supplies identity, version,
read-only posture, digest, encrypted persistence, and the `proposed` state. The
artifact is excluded from public objective reads and later prompt seeds. The
public API cannot attach an observer or author coverage, and the model-facing
tool cannot validate, install, execute, lease, or advance its proposal. Sandbox
validation, installation, event delivery, health leasing, and redesign
execution remain extensions of this lifecycle; until they record a healthy
installed observer, the saved objective truthfully remains uncovered.
tool cannot validate, install, execute, lease, or advance its proposal.
Core currently validates and installs one generic declarative ABI,
`pulse-resource-state/v1`, for an objective with explicit canonical resource
scope. It compares the canonical resource `status` with a bounded `equals` or
`not_equals` predicate at a 10-to-300-second local interval, requires an empty
external-requirements object, rejects unknown JSON fields, arbitrary code,
network, filesystem, and secret requirements, and wakes the model only after a
bounded consecutive-failure window. The installer is sandboxed by construction:
it registers typed data, not a process or script. Its runtime persists a
renewable health lease without incrementing the operator objective revision,
batches every due lease in a sweep into one encrypted persistence transaction,
and queues one scoped `objective_evidence` Patrol check on the transition into
failure rather than repeatedly polling a model. Observer health and observed
outcome remain separate: a healthy observer may report that its objective is
currently breached. Editing the retained brief, optional context, or resource
scope disables the existing observer and clears its lease, because an artifact
validated against old intent cannot remain proof of coverage for new intent.
Unsupported trigger or probe designs transition to `rejected` with an explicit
machine validation reason and uncovered coverage. A rejected or degraded
version may be replaced by a new model proposal, while core may reconsider an
unchanged rejected artifact after a runtime upgrade. Richer event, webhook, log,
file, socket, API, network, filesystem, secret-backed, and non-status signal
runtimes remain extensions of the same lifecycle. They must add a constrained
capability and validator rather than executing proposal JSON generically; until
such a runtime records a healthy installed observer, the saved objective
truthfully remains uncovered.
Desktop Autopilot activation consumes the server-owned acknowledgement
contract through `frontend-modern/src/api/patrol.ts` and
+2
View File
@@ -469,6 +469,7 @@ type PatrolService struct {
config PatrolConfig
findings *FindingsStore
objectiveStore *PatrolObjectiveStore
observerRuntime *patrolObserverRuntime
knowledgeStore *knowledge.Store // For per-resource notes in patrol context
discoveryStore *servicediscovery.Store // For AI-discovered infrastructure context
guestProber GuestProber // For pre-patrol guest reachability checks
@@ -733,5 +734,6 @@ func NewPatrolService(aiService *Service, stateProvider StateProvider) *PatrolSe
p.aptWorkflowWatcher = newAPTWorkflowWatcher()
p.proxmoxGuestLifecycleWatcher = newProxmoxGuestLifecycleWatcher(p.loadProxmoxGuestActionAudits)
p.pdmAlertBridge = newPDMAlertBridge(newPDMAlertSourceFromEnv())
p.observerRuntime = newPatrolObserverRuntime()
return p
}
+1 -1
View File
@@ -1322,7 +1322,7 @@ A direct provider-reported failed health check, failed backup, or broken replica
**Step 3 Report or assess findings.** Report new confirmed issues with patrol_report_finding. Every report call must independently include all required arguments: ` + strings.Join(tools.PatrolReportFindingRequiredArguments(), ", ") + `. This also applies when reporting several findings in parallel; do not omit a field because it is shared with another call. Call patrol_get_findings exactly once near the beginning of the run and reuse that result; do not call it again before the final summary. For every active finding it returned, call patrol_assess_finding exactly once with present, resolved, or uncertain and current evidence. Do not silently skip a known finding: omission is not evidence that it cleared. patrol_resolve_finding remains available for compatibility, but patrol_assess_finding is the complete existing-finding verdict.
**Operator objectives.** Objectives are retained outcomes, not scripts. When an active objective is explicitly marked observer_missing, use current estate context to call patrol_propose_observer once with the smallest useful read-only local observer design. Prefer event-driven evidence and use interval only for a bounded local probe, never repeated model polling. Do not re-propose an observer already marked proposed, validated, installed, or degraded unless the current evidence explicitly requires a new design. A successful proposal remains uncovered until core validates, sandboxes, installs, and leases it; never describe proposal creation as monitoring being active.
**Operator objectives.** Objectives are retained outcomes, not scripts. When an active objective is explicitly marked observer_missing, use current estate context to call patrol_propose_observer once with the smallest useful read-only local observer design. Use the generic pulse-resource-state/v1 interval ABI when canonical resource status truthfully measures the outcome; that interval runs locally and never polls the model. Prefer event-driven evidence for richer designs. Do not re-propose an observer already marked proposed, validated, installed, or degraded unless the current evidence explicitly requires a new design. A successful proposal remains uncovered until core validates, installs, evaluates, and leases it; never describe proposal creation as monitoring being active.
The snapshot eliminates routine data gathering. When a notable signal needs current or historical confirmation, gather enough evidence to distinguish real problems from noise before reporting it.
+85 -9
View File
@@ -63,6 +63,7 @@ type PatrolObserverState string
const (
PatrolObserverProposed PatrolObserverState = "proposed"
PatrolObserverRejected PatrolObserverState = "rejected"
PatrolObserverValidated PatrolObserverState = "validated"
PatrolObserverInstalled PatrolObserverState = "installed"
PatrolObserverDegraded PatrolObserverState = "degraded"
@@ -99,7 +100,7 @@ type PatrolObserverRecord struct {
UpdatedAt time.Time `json:"updated_at"`
// Artifact is encrypted at rest with the objective document and is never
// projected through objectiveForRead. It is model-authored proposal input,
// not trusted executable code; only the future core validator/installer may
// not trusted executable code; only the core validator/installer may
// consume it.
Artifact *PatrolObserverArtifact `json:"artifact,omitempty"`
}
@@ -128,6 +129,14 @@ type ProposePatrolObserverInput struct {
Actor string
}
type patrolObserverHealthUpdate struct {
ObjectiveID string
ObserverID string
Version uint64
ValidUntil time.Time
EvidenceAt time.Time
}
type PatrolObjectiveCoverage struct {
State PatrolObjectiveCoverageState `json:"state"`
ReasonCode string `json:"reason_code"`
@@ -357,6 +366,7 @@ func (s *PatrolObjectiveStore) Update(id string, input UpdatePatrolObjectiveInpu
return PatrolObjective{}, ErrPatrolObjectiveConflict
}
updated := clonePatrolObjective(current)
intentChanged := input.Brief != nil || input.OptionalContext != nil || input.ResourceIDs != nil
if input.Brief != nil {
brief, err := normalizePatrolObjectiveText(*input.Brief, MaxPatrolObjectiveBriefBytes, false)
if err != nil {
@@ -387,6 +397,15 @@ func (s *PatrolObjectiveStore) Update(id string, input UpdatePatrolObjectiveInpu
}
updated.Status = *input.Status
}
// An observer is evidence for the exact retained intent it was designed
// against. Editing that intent invalidates the lease fail-closed; Patrol may
// propose a new version after reasoning over the revised brief and scope.
if intentChanged && updated.Observer != nil && updated.Observer.State != PatrolObserverDisabled {
updated.Observer.State = PatrolObserverDisabled
updated.Observer.ValidUntil = nil
updated.Observer.FailureCode = ""
updated.Observer.UpdatedAt = now
}
updated.UpdatedBy = normalizePatrolObjectiveActor(input.Actor)
updated.UpdatedAt = now
updated.Revision++
@@ -449,7 +468,7 @@ func (s *PatrolObjectiveStore) ProposeObserver(id string, input ProposePatrolObs
if current.Status != PatrolObjectiveActive {
return PatrolObjective{}, fmt.Errorf("%w: observer proposals require an active objective", ErrPatrolObjectiveInvalid)
}
if current.Observer != nil && current.Observer.State != PatrolObserverDisabled {
if current.Observer != nil && current.Observer.State != PatrolObserverDisabled && current.Observer.State != PatrolObserverRejected && current.Observer.State != PatrolObserverDegraded {
return PatrolObjective{}, fmt.Errorf("%w: an existing observer cannot be displaced by a proposal", ErrPatrolObjectiveInvalid)
}
@@ -519,8 +538,8 @@ func (s *PatrolObjectiveStore) GetObserverArtifact(id string) (PatrolObserverArt
}
// RecordObserver persists a core-owned observer lifecycle transition. Public
// objective clients cannot call this method through HTTP. A future constrained
// monitor builder and installer must use it only after validating the observer
// objective clients cannot call this method through HTTP. The constrained
// monitor builder and installer use it only after validating the observer
// artifact, declared read-only posture, triggers, and health lease.
func (s *PatrolObjectiveStore) RecordObserver(id string, expectedRevision uint64, observer PatrolObserverRecord, actor string, now time.Time) (PatrolObjective, error) {
if s == nil {
@@ -566,6 +585,55 @@ func (s *PatrolObjectiveStore) RecordObserver(id string, expectedRevision uint64
return objectiveForRead(updated, now), nil
}
// RefreshObserverHealthBatch renews installed observer leases in one atomic
// persistence transaction without changing operator-authored revisions.
// Runtime heartbeats must not create optimistic-concurrency conflicts or one
// encrypted document rewrite per observer.
func (s *PatrolObjectiveStore) RefreshObserverHealthBatch(updates []patrolObserverHealthUpdate, now time.Time) (int, error) {
if s == nil {
return 0, fmt.Errorf("%w: store unavailable", ErrPatrolObjectiveInvalid)
}
now = normalizePatrolObjectiveTime(now)
if len(updates) == 0 {
return 0, nil
}
s.mu.Lock()
defer s.mu.Unlock()
next := clonePatrolObjectiveMap(s.objectives)
refreshed := 0
seen := make(map[string]struct{}, len(updates))
for _, update := range updates {
objectiveID := strings.TrimSpace(update.ObjectiveID)
observerID := strings.TrimSpace(update.ObserverID)
validUntil := update.ValidUntil.UTC()
evidenceAt := update.EvidenceAt.UTC()
if objectiveID == "" || observerID == "" || update.Version == 0 || !validUntil.After(now) || evidenceAt.After(now.Add(time.Minute)) {
return 0, fmt.Errorf("%w: invalid observer health lease", ErrPatrolObjectiveInvalid)
}
if _, duplicate := seen[objectiveID]; duplicate {
return 0, fmt.Errorf("%w: duplicate observer health lease", ErrPatrolObjectiveInvalid)
}
seen[objectiveID] = struct{}{}
current, ok := next[objectiveID]
if !ok || current.Status != PatrolObjectiveActive || current.Observer == nil || current.Observer.ID != observerID || current.Observer.Version != update.Version || current.Observer.State != PatrolObserverInstalled {
continue
}
current.Observer.ValidUntil = &validUntil
current.Observer.LastEvidenceAt = &evidenceAt
current.Observer.UpdatedAt = now
refreshed++
}
if refreshed == 0 {
return 0, nil
}
if err := s.persistLocked(next); err != nil {
return 0, err
}
s.objectives = next
return refreshed, nil
}
func (s *PatrolObjectiveStore) persistLocked(objectives map[string]*PatrolObjective) error {
if s.filePath == "" {
return nil
@@ -664,6 +732,13 @@ func derivePatrolObjectiveCoverage(objective *PatrolObjective, now time.Time) Pa
coverage.State = PatrolObjectiveUncovered
coverage.ReasonCode = "observer_proposed"
coverage.Summary = "An observer has been proposed but not validated or installed."
case PatrolObserverRejected:
coverage.State = PatrolObjectiveUncovered
coverage.ReasonCode = observer.FailureCode
if coverage.ReasonCode == "" {
coverage.ReasonCode = "observer_rejected"
}
coverage.Summary = "The proposed observer could not be validated for the local runtime."
case PatrolObserverValidated:
coverage.State = PatrolObjectiveUncovered
coverage.ReasonCode = "observer_not_installed"
@@ -832,10 +907,10 @@ func normalizePatrolObserver(observer PatrolObserverRecord, now time.Time) (Patr
if observer.FailureCode != "" && !isPatrolMachineCode(observer.FailureCode) {
return PatrolObserverRecord{}, fmt.Errorf("%w: invalid observer failure code", ErrPatrolObjectiveInvalid)
}
if observer.State == PatrolObserverDegraded && observer.FailureCode == "" {
return PatrolObserverRecord{}, fmt.Errorf("%w: degraded observer requires a failure code", ErrPatrolObjectiveInvalid)
if (observer.State == PatrolObserverDegraded || observer.State == PatrolObserverRejected) && observer.FailureCode == "" {
return PatrolObserverRecord{}, fmt.Errorf("%w: rejected or degraded observer requires a failure code", ErrPatrolObjectiveInvalid)
}
if observer.State != PatrolObserverDegraded {
if observer.State != PatrolObserverDegraded && observer.State != PatrolObserverRejected {
observer.FailureCode = ""
}
observer.ValidUntil = clonePatrolTime(observer.ValidUntil)
@@ -876,7 +951,8 @@ func validatePatrolObserverTransition(current *PatrolObserverRecord, next Patrol
return fmt.Errorf("%w: observer identity and artifact are immutable within a version", ErrPatrolObjectiveInvalid)
}
allowed := map[PatrolObserverState]map[PatrolObserverState]bool{
PatrolObserverProposed: {PatrolObserverProposed: true, PatrolObserverValidated: true, PatrolObserverDisabled: true},
PatrolObserverProposed: {PatrolObserverProposed: true, PatrolObserverRejected: true, PatrolObserverValidated: true, PatrolObserverDisabled: true},
PatrolObserverRejected: {PatrolObserverRejected: true, PatrolObserverValidated: true, PatrolObserverDisabled: true},
PatrolObserverValidated: {PatrolObserverValidated: true, PatrolObserverInstalled: true, PatrolObserverDegraded: true, PatrolObserverDisabled: true},
PatrolObserverInstalled: {PatrolObserverInstalled: true, PatrolObserverDegraded: true, PatrolObserverDisabled: true},
PatrolObserverDegraded: {PatrolObserverDegraded: true, PatrolObserverInstalled: true, PatrolObserverDisabled: true},
@@ -919,7 +995,7 @@ func isPatrolObjectiveStatus(status PatrolObjectiveStatus) bool {
func isPatrolObserverState(state PatrolObserverState) bool {
switch state {
case PatrolObserverProposed, PatrolObserverValidated, PatrolObserverInstalled, PatrolObserverDegraded, PatrolObserverDisabled:
case PatrolObserverProposed, PatrolObserverRejected, PatrolObserverValidated, PatrolObserverInstalled, PatrolObserverDegraded, PatrolObserverDisabled:
return true
default:
return false
+411
View File
@@ -0,0 +1,411 @@
package ai
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"sort"
"strings"
"sync"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/rs/zerolog/log"
)
const (
patrolObserverRuntimeFormat = "pulse-resource-state/v1"
patrolObserverSweepInterval = 5 * time.Second
patrolObserverMinSampleInterval = 10 * time.Second
patrolObserverMaxSampleInterval = 5 * time.Minute
patrolObserverMinLease = 2 * time.Minute
patrolObserverMaxLease = 30 * time.Minute
patrolObserverMaxConsecutiveFailures = 10
)
type patrolResourceStateProbe struct {
Runtime string `json:"runtime"`
Path string `json:"path"`
Operator string `json:"operator"`
Value string `json:"value"`
SampleIntervalSeconds int `json:"sample_interval_seconds"`
WakeAfterConsecutiveFailures int `json:"wake_after_consecutive_failures"`
}
type patrolObserverValidationError struct{ code string }
func (e *patrolObserverValidationError) Error() string { return e.code }
type patrolObserverExecution struct {
nextDue time.Time
consecutiveFailures int
wakeEmitted bool
}
type patrolObserverRuntime struct {
mu sync.Mutex
executions map[string]patrolObserverExecution
}
func newPatrolObserverRuntime() *patrolObserverRuntime {
return &patrolObserverRuntime{executions: make(map[string]patrolObserverExecution)}
}
func (p *PatrolService) objectiveObserverLoop(ctx context.Context) {
ticker := time.NewTicker(patrolObserverSweepInterval)
defer ticker.Stop()
p.processObjectiveObservers(time.Now().UTC())
for {
select {
case <-ctx.Done():
return
case <-p.stopCh:
return
case now := <-ticker.C:
p.processObjectiveObservers(now.UTC())
}
}
}
func (p *PatrolService) processObjectiveObservers(now time.Time) {
if p == nil {
return
}
p.mu.RLock()
store := p.objectiveStore
runtime := p.observerRuntime
p.mu.RUnlock()
if store == nil || runtime == nil {
return
}
activeExecutions := make(map[string]struct{})
healthUpdates := make([]patrolObserverHealthUpdate, 0)
for _, objective := range store.List(false, now) {
if objective.Status != PatrolObjectiveActive || objective.Observer == nil {
continue
}
activeExecutions[fmt.Sprintf("%s/%d", objective.Observer.ID, objective.Observer.Version)] = struct{}{}
if update := p.reconcileObjectiveObserver(store, runtime, objective, now); update != nil {
healthUpdates = append(healthUpdates, *update)
}
}
runtime.retain(activeExecutions)
if _, err := store.RefreshObserverHealthBatch(healthUpdates, now); err != nil {
log.Warn().Err(err).Int("lease_count", len(healthUpdates)).Msg("Patrol observer health lease batch failed")
}
}
func (r *patrolObserverRuntime) retain(active map[string]struct{}) {
r.mu.Lock()
defer r.mu.Unlock()
for key := range r.executions {
if _, ok := active[key]; !ok {
delete(r.executions, key)
}
}
}
func (p *PatrolService) reconcileObjectiveObserver(store *PatrolObjectiveStore, runtime *patrolObserverRuntime, objective PatrolObjective, now time.Time) *patrolObserverHealthUpdate {
artifact, ok := store.GetObserverArtifact(objective.ID)
if !ok {
p.recordObserverValidationFailure(store, objective, "observer_artifact_missing", now)
return nil
}
probe, validationErr := validatePatrolObserverArtifact(objective, artifact)
if validationErr != nil {
p.recordObserverValidationFailure(store, objective, validationErr.code, now)
return nil
}
for objective.Observer != nil && (objective.Observer.State == PatrolObserverProposed || objective.Observer.State == PatrolObserverRejected || objective.Observer.State == PatrolObserverValidated || objective.Observer.State == PatrolObserverDegraded) {
next := *clonePatrolObserver(objective.Observer)
next.Artifact = &artifact
switch objective.Observer.State {
case PatrolObserverProposed:
next.State = PatrolObserverValidated
next.FailureCode = ""
case PatrolObserverRejected:
next.State = PatrolObserverValidated
next.FailureCode = ""
case PatrolObserverValidated:
next.State = PatrolObserverInstalled
next.ValidUntil = nil
next.LastEvidenceAt = nil
case PatrolObserverDegraded:
next.State = PatrolObserverInstalled
next.FailureCode = ""
next.ValidUntil = nil
next.LastEvidenceAt = nil
}
updated, err := store.RecordObserver(objective.ID, objective.Revision, next, "patrol:observer-runtime", now)
if err != nil {
if !errors.Is(err, ErrPatrolObjectiveConflict) {
log.Warn().Err(err).Str("objective_id", objective.ID).Msg("Patrol observer lifecycle transition failed")
}
return nil
}
objective = updated
}
if objective.Observer == nil || (objective.Observer.State != PatrolObserverInstalled && objective.Observer.State != PatrolObserverDegraded) {
return nil
}
return p.evaluateObjectiveObserver(runtime, objective, probe, now)
}
func (p *PatrolService) recordObserverValidationFailure(store *PatrolObjectiveStore, objective PatrolObjective, code string, now time.Time) {
if objective.Observer == nil || objective.Observer.State == PatrolObserverDisabled || objective.Observer.FailureCode == code {
return
}
artifact, _ := store.GetObserverArtifact(objective.ID)
next := *clonePatrolObserver(objective.Observer)
next.FailureCode = code
if next.State == PatrolObserverProposed || next.State == PatrolObserverRejected {
next.State = PatrolObserverRejected
} else {
next.State = PatrolObserverDegraded
next.ValidUntil = nil
}
if artifact.Format != "" {
next.Artifact = &artifact
}
if _, err := store.RecordObserver(objective.ID, objective.Revision, next, "patrol:observer-validator", now); err != nil && !errors.Is(err, ErrPatrolObjectiveConflict) {
log.Warn().Err(err).Str("objective_id", objective.ID).Str("failure_code", code).Msg("Patrol observer validation failure could not be recorded")
}
}
func validatePatrolObserverArtifact(objective PatrolObjective, artifact PatrolObserverArtifact) (patrolResourceStateProbe, *patrolObserverValidationError) {
fail := func(code string) (patrolResourceStateProbe, *patrolObserverValidationError) {
return patrolResourceStateProbe{}, &patrolObserverValidationError{code: code}
}
if objective.Observer == nil || !objective.Observer.ReadOnly {
return fail("observer_not_read_only")
}
if len(objective.Scope.ResourceIDs) == 0 {
return fail("observer_scope_required")
}
if len(objective.Observer.TriggerKinds) != 1 || objective.Observer.TriggerKinds[0] != PatrolObserverTriggerInterval {
return fail("observer_trigger_unsupported")
}
var requirements map[string]json.RawMessage
if err := decodeStrictJSONObject(artifact.Requirements, &requirements); err != nil {
return fail("observer_requirements_invalid")
}
if len(requirements) != 0 {
return fail("observer_requirements_unsupported")
}
var probe patrolResourceStateProbe
if err := decodeStrictJSONObject(artifact.Probe, &probe); err != nil {
return fail("observer_probe_invalid")
}
if probe.Runtime != patrolObserverRuntimeFormat {
return fail("observer_runtime_unsupported")
}
if probe.Path != "status" {
return fail("observer_path_unsupported")
}
if probe.Operator != "equals" && probe.Operator != "not_equals" {
return fail("observer_operator_unsupported")
}
switch unifiedresources.ResourceStatus(strings.ToLower(strings.TrimSpace(probe.Value))) {
case unifiedresources.StatusOnline, unifiedresources.StatusOffline, unifiedresources.StatusWarning, unifiedresources.StatusUnknown:
probe.Value = strings.ToLower(strings.TrimSpace(probe.Value))
default:
return fail("observer_value_unsupported")
}
interval := time.Duration(probe.SampleIntervalSeconds) * time.Second
if interval < patrolObserverMinSampleInterval || interval > patrolObserverMaxSampleInterval {
return fail("observer_interval_out_of_bounds")
}
if probe.WakeAfterConsecutiveFailures < 1 || probe.WakeAfterConsecutiveFailures > patrolObserverMaxConsecutiveFailures {
return fail("observer_failure_window_out_of_bounds")
}
return probe, nil
}
func decodeStrictJSONObject(data []byte, target interface{}) error {
decoder := json.NewDecoder(bytes.NewReader(data))
decoder.DisallowUnknownFields()
if err := decoder.Decode(target); err != nil {
return err
}
if err := decoder.Decode(&struct{}{}); err != io.EOF {
return fmt.Errorf("expected one JSON object")
}
return nil
}
func (p *PatrolService) evaluateObjectiveObserver(runtime *patrolObserverRuntime, objective PatrolObjective, probe patrolResourceStateProbe, now time.Time) *patrolObserverHealthUpdate {
observer := objective.Observer
key := fmt.Sprintf("%s/%d", observer.ID, observer.Version)
runtime.mu.Lock()
execution := runtime.executions[key]
if !execution.nextDue.IsZero() && now.Before(execution.nextDue) {
runtime.mu.Unlock()
return nil
}
interval := time.Duration(probe.SampleIntervalSeconds) * time.Second
execution.nextDue = now.Add(interval)
runtime.mu.Unlock()
statuses := patrolRuntimeCanonicalStatuses(p.currentPatrolRuntimeState())
failing := make([]string, 0)
for _, resourceID := range objective.Scope.ResourceIDs {
status, exists := statuses[canonicalPatrolScopeToken(resourceID)]
matched := exists && string(status) == probe.Value
if probe.Operator == "not_equals" {
matched = exists && string(status) != probe.Value
}
if !matched {
failing = append(failing, resourceID)
}
}
runtime.mu.Lock()
execution = runtime.executions[key]
if len(failing) == 0 {
execution.consecutiveFailures = 0
execution.wakeEmitted = false
} else {
execution.consecutiveFailures++
}
shouldWake := len(failing) > 0 && execution.consecutiveFailures >= probe.WakeAfterConsecutiveFailures && !execution.wakeEmitted
runtime.executions[key] = execution
runtime.mu.Unlock()
leaseDuration := 3 * interval
if leaseDuration < patrolObserverMinLease {
leaseDuration = patrolObserverMinLease
}
if leaseDuration > patrolObserverMaxLease {
leaseDuration = patrolObserverMaxLease
}
refreshLease := observer.ValidUntil == nil || observer.ValidUntil.Sub(now) <= leaseDuration/2
var healthUpdate *patrolObserverHealthUpdate
if refreshLease {
healthUpdate = &patrolObserverHealthUpdate{
ObjectiveID: objective.ID,
ObserverID: observer.ID,
Version: observer.Version,
ValidUntil: now.Add(leaseDuration),
EvidenceAt: now,
}
}
if !shouldWake {
return healthUpdate
}
sort.Strings(failing)
scope := PatrolScope{
ResourceIDs: failing,
Depth: PatrolDepthQuick,
Reason: TriggerReasonObjectiveEvidence,
Priority: triggerPriorityObjective,
Context: fmt.Sprintf("Local observer %s for objective %s detected %d of %d scoped resources outside its canonical status predicate.", observer.ID, objective.ID, len(failing), len(objective.Scope.ResourceIDs)),
}
p.mu.RLock()
tm := p.triggerManager
p.mu.RUnlock()
accepted := tm != nil && tm.TriggerPatrol(scope)
if accepted {
runtime.mu.Lock()
execution = runtime.executions[key]
execution.wakeEmitted = true
runtime.executions[key] = execution
runtime.mu.Unlock()
log.Info().Str("objective_id", objective.ID).Int("affected_resources", len(failing)).Msg("Patrol objective observer queued an evidence-triggered check")
}
return healthUpdate
}
func patrolRuntimeCanonicalStatuses(state patrolRuntimeState) map[string]unifiedresources.ResourceStatus {
result := make(map[string]unifiedresources.ResourceStatus)
add := func(status unifiedresources.ResourceStatus, ids ...string) {
for _, id := range ids {
if token := canonicalPatrolScopeToken(id); token != "" {
result[token] = normalizePatrolResourceStatus(string(status))
}
}
}
if rs := state.readState; rs != nil {
for _, v := range rs.VMs() {
add(v.Status(), v.ID(), v.SourceID())
}
for _, v := range rs.Containers() {
add(v.Status(), v.ID(), v.SourceID())
}
for _, v := range rs.Nodes() {
add(v.Status(), v.ID(), v.SourceID())
}
for _, v := range rs.Hosts() {
add(v.Status(), v.ID())
}
for _, v := range rs.DockerHosts() {
add(v.Status(), v.ID(), v.HostSourceID())
}
for _, v := range rs.DockerContainers() {
add(v.Status(), v.ID(), v.ContainerID())
}
for _, v := range rs.StoragePools() {
add(v.Status(), v.ID(), v.SourceID())
}
for _, v := range rs.PhysicalDisks() {
add(v.Status(), v.ID())
}
for _, v := range rs.PBSInstances() {
add(v.Status(), v.ID())
}
for _, v := range rs.PMGInstances() {
add(v.Status(), v.ID())
}
for _, v := range rs.K8sClusters() {
add(v.Status(), v.ID())
}
return result
}
for _, v := range state.VMs {
add(normalizePatrolResourceStatus(v.Status), v.ID)
}
for _, v := range state.Containers {
add(normalizePatrolResourceStatus(v.Status), v.ID)
}
for _, v := range state.Nodes {
add(normalizePatrolResourceStatus(v.Status), v.ID)
}
for _, v := range state.Hosts {
add(normalizePatrolResourceStatus(v.Status), v.ID)
}
for _, v := range state.DockerHosts {
add(normalizePatrolResourceStatus(v.Status), v.ID)
for _, container := range v.Containers {
add(normalizePatrolResourceStatus(container.State), container.ID)
}
}
for _, v := range state.Storage {
add(normalizePatrolResourceStatus(v.Status), v.ID)
}
for _, v := range state.PBSInstances {
add(normalizePatrolResourceStatus(v.Status), v.ID)
}
for _, v := range state.PMGInstances {
add(normalizePatrolResourceStatus(v.Status), v.ID)
}
for _, v := range state.KubernetesClusters {
add(normalizePatrolResourceStatus(v.Status), v.ID)
}
return result
}
func normalizePatrolResourceStatus(value string) unifiedresources.ResourceStatus {
switch strings.ToLower(strings.TrimSpace(value)) {
case "online", "running", "active", "healthy", "up", "available", "ready":
return unifiedresources.StatusOnline
case "offline", "stopped", "inactive", "failed", "error", "down", "unavailable":
return unifiedresources.StatusOffline
case "warning", "degraded", "unhealthy":
return unifiedresources.StatusWarning
default:
return unifiedresources.StatusUnknown
}
}
+188
View File
@@ -0,0 +1,188 @@
package ai
import (
"strings"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
)
func createInstallablePatrolObserver(t *testing.T, store *PatrolObjectiveStore, now time.Time, resourceIDs ...string) PatrolObjective {
t.Helper()
objective, err := store.Create(CreatePatrolObjectiveInput{
Brief: "Keep scoped resources online",
ResourceIDs: resourceIDs,
}, now)
if err != nil {
t.Fatalf("create objective: %v", err)
}
objective, err = store.ProposeObserver(objective.ID, ProposePatrolObserverInput{
ExpectedRevision: objective.Revision,
Interpretation: "Every scoped canonical resource remains online.",
TriggerKinds: []PatrolObserverTriggerKind{PatrolObserverTriggerInterval},
ProbeJSON: `{"runtime":"pulse-resource-state/v1","path":"status","operator":"equals","value":"online","sample_interval_seconds":10,"wake_after_consecutive_failures":2}`,
WakeEvidence: "A scoped resource is not online for two consecutive samples.",
RequirementsJSON: `{}`,
Actor: "patrol:model",
}, now)
if err != nil {
t.Fatalf("propose observer: %v", err)
}
return objective
}
func TestPatrolObserverRuntimeInstallsEvaluatesAndLeasesWithoutObjectiveRevisionChurn(t *testing.T) {
now := time.Date(2026, 8, 14, 1, 0, 0, 0, time.UTC)
store := NewInMemoryPatrolObjectiveStore()
objective := createInstallablePatrolObserver(t, store, now, "node-1")
proposalRevision := objective.Revision
patrol := NewPatrolService(nil, mockPatrolStateProvider{state: models.StateSnapshot{
Nodes: []models.Node{{ID: "node-1", Name: "node-1", Status: "online"}},
}})
patrol.SetObjectiveStore(store)
patrol.processObjectiveObservers(now.Add(time.Second))
got, ok := store.Get(objective.ID, now.Add(time.Second))
if !ok || got.Observer == nil {
t.Fatalf("installed objective missing: %+v, found=%v", got, ok)
}
if got.Observer.State != PatrolObserverInstalled || got.Coverage.State != PatrolObjectiveCovered || got.Coverage.ReasonCode != "observer_healthy" {
t.Fatalf("installed observer = %+v, coverage = %+v", got.Observer, got.Coverage)
}
if got.Observer.ValidUntil == nil || got.Observer.LastEvidenceAt == nil {
t.Fatalf("observer lease was not persisted: %+v", got.Observer)
}
if got.Revision != proposalRevision+2 {
t.Fatalf("objective revision = %d, want proposal revision + validation/install only (%d)", got.Revision, proposalRevision+2)
}
patrol.processObjectiveObservers(now.Add(2 * time.Minute))
refreshed, _ := store.Get(objective.ID, now.Add(2*time.Minute))
if refreshed.Revision != got.Revision {
t.Fatalf("health heartbeat changed objective revision from %d to %d", got.Revision, refreshed.Revision)
}
if !refreshed.Observer.ValidUntil.After(*got.Observer.ValidUntil) {
t.Fatalf("health lease did not advance: old=%v new=%v", got.Observer.ValidUntil, refreshed.Observer.ValidUntil)
}
}
func TestPatrolObserverRuntimeRecordsExplicitUnsupportedValidationReason(t *testing.T) {
now := time.Date(2026, 8, 14, 1, 0, 0, 0, time.UTC)
store := NewInMemoryPatrolObjectiveStore()
objective, err := store.Create(CreatePatrolObjectiveInput{Brief: "Keep playback smooth", ResourceIDs: []string{"jellyfin"}}, now)
if err != nil {
t.Fatalf("create objective: %v", err)
}
objective, err = store.ProposeObserver(objective.ID, ProposePatrolObserverInput{
ExpectedRevision: objective.Revision,
Interpretation: "Observe playback buffering events.",
TriggerKinds: []PatrolObserverTriggerKind{PatrolObserverTriggerAPI},
ProbeJSON: `{"runtime":"jellyfin-api/v1","event":"playback"}`,
WakeEvidence: "Playback buffers.",
RequirementsJSON: `{"network":["jellyfin"]}`,
}, now)
if err != nil {
t.Fatalf("propose observer: %v", err)
}
patrol := NewPatrolService(nil, nil)
patrol.SetObjectiveStore(store)
patrol.processObjectiveObservers(now.Add(time.Second))
got, _ := store.Get(objective.ID, now.Add(time.Second))
if got.Observer == nil || got.Observer.State != PatrolObserverRejected {
t.Fatalf("unsupported proposal state = %+v", got.Observer)
}
if got.Observer.FailureCode != "observer_trigger_unsupported" || got.Coverage.ReasonCode != "observer_trigger_unsupported" {
t.Fatalf("validation failure was not projected explicitly: observer=%+v coverage=%+v", got.Observer, got.Coverage)
}
if got.Coverage.State != PatrolObjectiveUncovered || !strings.Contains(got.Coverage.Summary, "could not be validated") {
t.Fatalf("unsupported proposal coverage = %+v", got.Coverage)
}
revised, err := store.ProposeObserver(objective.ID, ProposePatrolObserverInput{
ExpectedRevision: got.Revision,
Interpretation: "Keep the canonical Jellyfin resource online.",
TriggerKinds: []PatrolObserverTriggerKind{PatrolObserverTriggerInterval},
ProbeJSON: `{"runtime":"pulse-resource-state/v1","path":"status","operator":"equals","value":"online","sample_interval_seconds":30,"wake_after_consecutive_failures":2}`,
WakeEvidence: "The Jellyfin resource is not online twice.",
RequirementsJSON: `{}`,
}, now.Add(2*time.Second))
if err != nil {
t.Fatalf("replace rejected observer proposal: %v", err)
}
if revised.Observer == nil || revised.Observer.Version != got.Observer.Version+1 || revised.Observer.State != PatrolObserverProposed {
t.Fatalf("replacement proposal = %+v", revised.Observer)
}
}
func TestPatrolObserverRuntimeWakesModelOnlyAfterLocalFailureWindow(t *testing.T) {
now := time.Date(2026, 8, 14, 1, 0, 0, 0, time.UTC)
store := NewInMemoryPatrolObjectiveStore()
objective := createInstallablePatrolObserver(t, store, now, "camera-1")
patrol := NewPatrolService(nil, mockPatrolStateProvider{state: models.StateSnapshot{
Hosts: []models.Host{{ID: "camera-1", Hostname: "camera-1", Status: "offline"}},
}})
patrol.SetObjectiveStore(store)
tm := NewTriggerManager(TriggerManagerConfig{MaxPendingTriggers: 10})
patrol.SetTriggerManager(tm)
patrol.processObjectiveObservers(now.Add(time.Second))
if got := tm.GetPendingCount(); got != 0 {
t.Fatalf("observer woke Patrol after first failed local sample; pending=%d", got)
}
patrol.processObjectiveObservers(now.Add(11 * time.Second))
if got := tm.GetPendingCount(); got != 1 {
t.Fatalf("observer did not queue one Patrol wake after failure window; pending=%d", got)
}
patrol.processObjectiveObservers(now.Add(21 * time.Second))
if got := tm.GetPendingCount(); got != 1 {
t.Fatalf("observer repeated wake while failure remained active; pending=%d", got)
}
got, _ := store.Get(objective.ID, now.Add(21*time.Second))
if got.Coverage.State != PatrolObjectiveCovered {
t.Fatalf("runtime health and objective outcome were conflated: %+v", got.Coverage)
}
}
func TestValidatePatrolObserverArtifactRejectsUnknownExecutableFields(t *testing.T) {
now := time.Now().UTC()
store := NewInMemoryPatrolObjectiveStore()
objective := createInstallablePatrolObserver(t, store, now, "node-1")
artifact, _ := store.GetObserverArtifact(objective.ID)
artifact.Probe = []byte(`{"runtime":"pulse-resource-state/v1","path":"status","operator":"equals","value":"online","sample_interval_seconds":30,"wake_after_consecutive_failures":2,"command":"rm -rf /"}`)
_, err := validatePatrolObserverArtifact(objective, artifact)
if err == nil || err.code != "observer_probe_invalid" {
t.Fatalf("unknown executable field validation error = %v", err)
}
}
func TestPatrolObjectiveIntentEditInvalidatesInstalledObserver(t *testing.T) {
now := time.Date(2026, 8, 14, 1, 0, 0, 0, time.UTC)
store := NewInMemoryPatrolObjectiveStore()
objective := createInstallablePatrolObserver(t, store, now, "node-1")
patrol := NewPatrolService(nil, mockPatrolStateProvider{state: models.StateSnapshot{
Nodes: []models.Node{{ID: "node-1", Status: "online"}},
}})
patrol.SetObjectiveStore(store)
patrol.processObjectiveObservers(now.Add(time.Second))
installed, _ := store.Get(objective.ID, now.Add(time.Second))
if installed.Coverage.State != PatrolObjectiveCovered {
t.Fatalf("observer was not installed before edit: %+v", installed.Coverage)
}
revisedBrief := "Keep this node online and its workload responsive"
updated, err := store.Update(objective.ID, UpdatePatrolObjectiveInput{
ExpectedRevision: installed.Revision,
Brief: &revisedBrief,
Actor: "operator",
}, now.Add(2*time.Second))
if err != nil {
t.Fatalf("update objective intent: %v", err)
}
if updated.Observer == nil || updated.Observer.State != PatrolObserverDisabled || updated.Observer.ValidUntil != nil {
t.Fatalf("intent edit retained stale observer authority: %+v", updated.Observer)
}
if updated.Coverage.State != PatrolObjectiveUncovered || updated.Coverage.ReasonCode != "observer_disabled" {
t.Fatalf("intent edit coverage = %+v", updated.Coverage)
}
}
+1
View File
@@ -50,6 +50,7 @@ func (p *PatrolService) Start(ctx context.Context) {
go p.patrolLoop(ctx)
go p.remindSweepLoop(ctx)
go p.objectiveObserverLoop(ctx)
}
// Stop stops the patrol service. It signals the patrol loop to exit, then
+13 -11
View File
@@ -44,6 +44,7 @@ func BackgroundAutomationEventTriggerBlock() PatrolEventTriggerBlock {
const (
triggerPriorityManual = 100 // User-initiated patrols run first
triggerPriorityAlertFired = 80 // New alerts are high priority
triggerPriorityObjective = 70 // Retained-objective evidence needs prompt investigation
triggerPriorityAnomaly = 60 // Anomaly detection is medium-high
triggerPriorityAlertCleared = 40 // Cleared alerts are lower priority
triggerPriorityUserAction = 30 // User finding actions
@@ -51,16 +52,17 @@ const (
)
const (
TriggerReasonScheduled TriggerReason = "scheduled" // Regular interval trigger
TriggerReasonManual TriggerReason = "manual" // User-initiated patrol
TriggerReasonAlertFired TriggerReason = "alert_fired" // New alert triggered
TriggerReasonAlertCleared TriggerReason = "alert_cleared" // Alert was resolved
TriggerReasonAlertFlapping TriggerReason = "alert_flapping" // Alert flapping detected and suppressed
TriggerReasonAnomalyDetected TriggerReason = "anomaly" // Baseline breach detected
TriggerReasonUserAction TriggerReason = "user_action" // User dismissed/snoozed finding
TriggerReasonConfigChanged TriggerReason = "config_changed" // System configuration changed
TriggerReasonStartup TriggerReason = "startup" // Service startup
TriggerReasonVerification TriggerReason = "verification" // Post-fix verification
TriggerReasonScheduled TriggerReason = "scheduled" // Regular interval trigger
TriggerReasonManual TriggerReason = "manual" // User-initiated patrol
TriggerReasonAlertFired TriggerReason = "alert_fired" // New alert triggered
TriggerReasonAlertCleared TriggerReason = "alert_cleared" // Alert was resolved
TriggerReasonAlertFlapping TriggerReason = "alert_flapping" // Alert flapping detected and suppressed
TriggerReasonAnomalyDetected TriggerReason = "anomaly" // Baseline breach detected
TriggerReasonUserAction TriggerReason = "user_action" // User dismissed/snoozed finding
TriggerReasonConfigChanged TriggerReason = "config_changed" // System configuration changed
TriggerReasonStartup TriggerReason = "startup" // Service startup
TriggerReasonVerification TriggerReason = "verification" // Post-fix verification
TriggerReasonObjectiveEvidence TriggerReason = "objective_evidence" // Local observer detected an objective breach
)
// PatrolScope defines the scope of a patrol run
@@ -423,7 +425,7 @@ func (tm *TriggerManager) SetEventTriggersEnabled(enabled bool) {
// (alerts firing/clearing, anomaly detection) as opposed to user-initiated or scheduled triggers.
func isEventDrivenTrigger(reason TriggerReason) bool {
switch reason {
case TriggerReasonAlertFired, TriggerReasonAlertCleared, TriggerReasonAlertFlapping, TriggerReasonAnomalyDetected:
case TriggerReasonAlertFired, TriggerReasonAlertCleared, TriggerReasonAlertFlapping, TriggerReasonAnomalyDetected, TriggerReasonObjectiveEvidence:
return true
default:
return false
+5 -3
View File
@@ -229,7 +229,9 @@ Returns a list of active findings with their IDs, severity, resource, and title.
Name: agentcapabilities.PatrolProposeObserverToolName,
Description: `Propose a durable read-only observer for an active operator objective that is currently missing coverage.
Use this only when the objective context says observer_missing, or when current evidence clearly requires a new observer version. Translate the operator's outcome into the smallest useful local observer without hard-coding an application into Pulse. The probe_json and requirements_json fields must each be one bounded JSON object. Describe what a future constrained runtime should observe; do not include mutation commands, credentials, or secret values.
Use this only when the objective context says observer_missing, or when current evidence clearly requires a new observer version. Translate the operator's outcome into the smallest useful local observer without hard-coding an application into Pulse. The probe_json and requirements_json fields must each be one bounded JSON object. Do not include mutation commands, credentials, or secret values.
Core can currently install one generic local ABI for objectives scoped to canonical Pulse resources: trigger_kind must be interval, requirements_json must be {}, and probe_json must be exactly {"runtime":"pulse-resource-state/v1","path":"status","operator":"equals","value":"online","sample_interval_seconds":30,"wake_after_consecutive_failures":2}. The operator may be equals or not_equals; value may be online, offline, warning, or unknown; interval is 10-300 seconds and the failure window is 1-10 samples. Use this ABI when canonical resource status is a truthful interpretation. If the outcome needs an app API, event, log, file, socket, network, filesystem, secret, or richer signal, describe that honest proposal instead; core will retain it with an explicit unsupported validation reason rather than pretending it is active.
This tool records only a versioned proposed artifact. It does not validate, install, execute, or claim coverage. Core owns the observer ID, version, SHA-256 digest, read-only posture, sandboxing, installation, health lease, and any later transition.
@@ -256,7 +258,7 @@ Returns the proposed observer identity and the truthful uncovered coverage reaso
},
"probe_json": {
Type: "string",
Description: "One JSON object describing the read-only probe, signal extraction, and health/failure outputs. It is proposal material, not executable authority.",
Description: "One JSON object describing the read-only probe. Use the documented pulse-resource-state/v1 ABI exactly when canonical resource status is sufficient; otherwise provide an honest bounded proposal for future capability validation.",
},
"wake_evidence": {
Type: "string",
@@ -264,7 +266,7 @@ Returns the proposed observer identity and the truthful uncovered coverage reaso
},
"requirements_json": {
Type: "string",
Description: "One JSON object declaring network, filesystem, secret-reference, runtime, timeout, and resource-budget requirements. Use empty arrays/objects when none; never include secret values.",
Description: "One JSON object declaring external requirements. Use {} for the installable pulse-resource-state/v1 ABI. Never include secret values.",
},
},
Required: []string{"objective_id", "expected_revision", "interpretation", "trigger_kind", "probe_json", "wake_evidence", "requirements_json"},