mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-09-10 02:25:56 +00:00
Run validated local Patrol observers
This commit is contained in:
@@ -6970,3 +6970,30 @@ Governed Patrol fixes are delivered through the approved-action pipeline and
|
||||
are already counted there (`pulse_intelligence_approved_action_*`), which is the
|
||||
signal to read for remediation activity. Existing persisted history files that
|
||||
still contain `auto_fix_count` load cleanly; the unknown key is ignored.
|
||||
|
||||
### Model-authored observers have a core-owned local execution floor
|
||||
|
||||
`internal/ai/patrol_observer_runtime.go` owns the first executable observer ABI
|
||||
for retained Patrol objectives. The model may author the predicate, but it
|
||||
cannot author executable code or advance lifecycle authority. Core strictly
|
||||
decodes `pulse-resource-state/v1`, requires explicit canonical resource scope,
|
||||
an interval trigger, an empty external-requirements object, one canonical
|
||||
`status` predicate, a 10-to-300-second local sample interval, and a bounded
|
||||
consecutive-failure window. Unknown fields and unsupported runtimes, triggers,
|
||||
requirements, paths, operators, or values fail closed and persist a safe
|
||||
machine reason in the `rejected` state while coverage remains uncovered.
|
||||
|
||||
Accepted observers transition through proposed, validated, and installed under
|
||||
core authority. They evaluate Pulse's canonical `ReadState` status locally,
|
||||
falling back to the existing Patrol snapshot only where canonical read state is
|
||||
unavailable. Health leasing is persisted without changing the operator
|
||||
objective revision, with all leases due in one sweep committed in one atomic
|
||||
encrypted-document transaction. A failure transition queues one scoped
|
||||
`objective_evidence` check through the existing TriggerManager, so the model
|
||||
wakes for reasoning only after local evidence crosses the declared window and
|
||||
does not become the polling loop. The observer remains read-only and cannot
|
||||
enter infrastructure mutation; any later response still uses the canonical
|
||||
capability, autonomy, approval, dispatch, audit, and verification kernel.
|
||||
Changing the retained brief, optional context, or resource scope disables the
|
||||
installed observer and clears its lease; coverage cannot survive a semantic
|
||||
change to the intent it was designed to protect.
|
||||
|
||||
@@ -128,9 +128,10 @@ with bounded provenance and rollback; app-named integration code is an
|
||||
optimization only when repeated evidence justifies productizing it, never a
|
||||
prerequisite for a new objective.
|
||||
|
||||
The canonical retained-intent and proposal slices are implemented by
|
||||
The canonical retained-intent, proposal, and first local execution slices are implemented by
|
||||
`internal/ai/patrol_objectives.go`, `internal/ai/patrol_observer_builder.go`,
|
||||
`internal/ai/tools/tools_patrol.go`, and `internal/api/patrol_objectives.go`.
|
||||
`internal/ai/patrol_observer_runtime.go`, `internal/ai/tools/tools_patrol.go`, and
|
||||
`internal/api/patrol_objectives.go`.
|
||||
They provide encrypted retained intent, optimistic revisions, resource scoping,
|
||||
model seed projection, the core-owned observer state machine, and the
|
||||
Patrol-detection-only `patrol_propose_observer` builder. The model supplies a
|
||||
@@ -139,10 +140,34 @@ declared requirements, and one trigger kind; core supplies identity, version,
|
||||
read-only posture, digest, encrypted persistence, and the `proposed` state. The
|
||||
artifact is excluded from public objective reads and later prompt seeds. The
|
||||
public API cannot attach an observer or author coverage, and the model-facing
|
||||
tool cannot validate, install, execute, lease, or advance its proposal. Sandbox
|
||||
validation, installation, event delivery, health leasing, and redesign
|
||||
execution remain extensions of this lifecycle; until they record a healthy
|
||||
installed observer, the saved objective truthfully remains uncovered.
|
||||
tool cannot validate, install, execute, lease, or advance its proposal.
|
||||
|
||||
Core currently validates and installs one generic declarative ABI,
|
||||
`pulse-resource-state/v1`, for an objective with explicit canonical resource
|
||||
scope. It compares the canonical resource `status` with a bounded `equals` or
|
||||
`not_equals` predicate at a 10-to-300-second local interval, requires an empty
|
||||
external-requirements object, rejects unknown JSON fields, arbitrary code,
|
||||
network, filesystem, and secret requirements, and wakes the model only after a
|
||||
bounded consecutive-failure window. The installer is sandboxed by construction:
|
||||
it registers typed data, not a process or script. Its runtime persists a
|
||||
renewable health lease without incrementing the operator objective revision,
|
||||
batches every due lease in a sweep into one encrypted persistence transaction,
|
||||
and queues one scoped `objective_evidence` Patrol check on the transition into
|
||||
failure rather than repeatedly polling a model. Observer health and observed
|
||||
outcome remain separate: a healthy observer may report that its objective is
|
||||
currently breached. Editing the retained brief, optional context, or resource
|
||||
scope disables the existing observer and clears its lease, because an artifact
|
||||
validated against old intent cannot remain proof of coverage for new intent.
|
||||
|
||||
Unsupported trigger or probe designs transition to `rejected` with an explicit
|
||||
machine validation reason and uncovered coverage. A rejected or degraded
|
||||
version may be replaced by a new model proposal, while core may reconsider an
|
||||
unchanged rejected artifact after a runtime upgrade. Richer event, webhook, log,
|
||||
file, socket, API, network, filesystem, secret-backed, and non-status signal
|
||||
runtimes remain extensions of the same lifecycle. They must add a constrained
|
||||
capability and validator rather than executing proposal JSON generically; until
|
||||
such a runtime records a healthy installed observer, the saved objective
|
||||
truthfully remains uncovered.
|
||||
|
||||
Desktop Autopilot activation consumes the server-owned acknowledgement
|
||||
contract through `frontend-modern/src/api/patrol.ts` and
|
||||
|
||||
@@ -469,6 +469,7 @@ type PatrolService struct {
|
||||
config PatrolConfig
|
||||
findings *FindingsStore
|
||||
objectiveStore *PatrolObjectiveStore
|
||||
observerRuntime *patrolObserverRuntime
|
||||
knowledgeStore *knowledge.Store // For per-resource notes in patrol context
|
||||
discoveryStore *servicediscovery.Store // For AI-discovered infrastructure context
|
||||
guestProber GuestProber // For pre-patrol guest reachability checks
|
||||
@@ -733,5 +734,6 @@ func NewPatrolService(aiService *Service, stateProvider StateProvider) *PatrolSe
|
||||
p.aptWorkflowWatcher = newAPTWorkflowWatcher()
|
||||
p.proxmoxGuestLifecycleWatcher = newProxmoxGuestLifecycleWatcher(p.loadProxmoxGuestActionAudits)
|
||||
p.pdmAlertBridge = newPDMAlertBridge(newPDMAlertSourceFromEnv())
|
||||
p.observerRuntime = newPatrolObserverRuntime()
|
||||
return p
|
||||
}
|
||||
|
||||
@@ -1322,7 +1322,7 @@ A direct provider-reported failed health check, failed backup, or broken replica
|
||||
|
||||
**Step 3 — Report or assess findings.** Report new confirmed issues with patrol_report_finding. Every report call must independently include all required arguments: ` + strings.Join(tools.PatrolReportFindingRequiredArguments(), ", ") + `. This also applies when reporting several findings in parallel; do not omit a field because it is shared with another call. Call patrol_get_findings exactly once near the beginning of the run and reuse that result; do not call it again before the final summary. For every active finding it returned, call patrol_assess_finding exactly once with present, resolved, or uncertain and current evidence. Do not silently skip a known finding: omission is not evidence that it cleared. patrol_resolve_finding remains available for compatibility, but patrol_assess_finding is the complete existing-finding verdict.
|
||||
|
||||
**Operator objectives.** Objectives are retained outcomes, not scripts. When an active objective is explicitly marked observer_missing, use current estate context to call patrol_propose_observer once with the smallest useful read-only local observer design. Prefer event-driven evidence and use interval only for a bounded local probe, never repeated model polling. Do not re-propose an observer already marked proposed, validated, installed, or degraded unless the current evidence explicitly requires a new design. A successful proposal remains uncovered until core validates, sandboxes, installs, and leases it; never describe proposal creation as monitoring being active.
|
||||
**Operator objectives.** Objectives are retained outcomes, not scripts. When an active objective is explicitly marked observer_missing, use current estate context to call patrol_propose_observer once with the smallest useful read-only local observer design. Use the generic pulse-resource-state/v1 interval ABI when canonical resource status truthfully measures the outcome; that interval runs locally and never polls the model. Prefer event-driven evidence for richer designs. Do not re-propose an observer already marked proposed, validated, installed, or degraded unless the current evidence explicitly requires a new design. A successful proposal remains uncovered until core validates, installs, evaluates, and leases it; never describe proposal creation as monitoring being active.
|
||||
|
||||
The snapshot eliminates routine data gathering. When a notable signal needs current or historical confirmation, gather enough evidence to distinguish real problems from noise before reporting it.
|
||||
|
||||
|
||||
@@ -63,6 +63,7 @@ type PatrolObserverState string
|
||||
|
||||
const (
|
||||
PatrolObserverProposed PatrolObserverState = "proposed"
|
||||
PatrolObserverRejected PatrolObserverState = "rejected"
|
||||
PatrolObserverValidated PatrolObserverState = "validated"
|
||||
PatrolObserverInstalled PatrolObserverState = "installed"
|
||||
PatrolObserverDegraded PatrolObserverState = "degraded"
|
||||
@@ -99,7 +100,7 @@ type PatrolObserverRecord struct {
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
// Artifact is encrypted at rest with the objective document and is never
|
||||
// projected through objectiveForRead. It is model-authored proposal input,
|
||||
// not trusted executable code; only the future core validator/installer may
|
||||
// not trusted executable code; only the core validator/installer may
|
||||
// consume it.
|
||||
Artifact *PatrolObserverArtifact `json:"artifact,omitempty"`
|
||||
}
|
||||
@@ -128,6 +129,14 @@ type ProposePatrolObserverInput struct {
|
||||
Actor string
|
||||
}
|
||||
|
||||
type patrolObserverHealthUpdate struct {
|
||||
ObjectiveID string
|
||||
ObserverID string
|
||||
Version uint64
|
||||
ValidUntil time.Time
|
||||
EvidenceAt time.Time
|
||||
}
|
||||
|
||||
type PatrolObjectiveCoverage struct {
|
||||
State PatrolObjectiveCoverageState `json:"state"`
|
||||
ReasonCode string `json:"reason_code"`
|
||||
@@ -357,6 +366,7 @@ func (s *PatrolObjectiveStore) Update(id string, input UpdatePatrolObjectiveInpu
|
||||
return PatrolObjective{}, ErrPatrolObjectiveConflict
|
||||
}
|
||||
updated := clonePatrolObjective(current)
|
||||
intentChanged := input.Brief != nil || input.OptionalContext != nil || input.ResourceIDs != nil
|
||||
if input.Brief != nil {
|
||||
brief, err := normalizePatrolObjectiveText(*input.Brief, MaxPatrolObjectiveBriefBytes, false)
|
||||
if err != nil {
|
||||
@@ -387,6 +397,15 @@ func (s *PatrolObjectiveStore) Update(id string, input UpdatePatrolObjectiveInpu
|
||||
}
|
||||
updated.Status = *input.Status
|
||||
}
|
||||
// An observer is evidence for the exact retained intent it was designed
|
||||
// against. Editing that intent invalidates the lease fail-closed; Patrol may
|
||||
// propose a new version after reasoning over the revised brief and scope.
|
||||
if intentChanged && updated.Observer != nil && updated.Observer.State != PatrolObserverDisabled {
|
||||
updated.Observer.State = PatrolObserverDisabled
|
||||
updated.Observer.ValidUntil = nil
|
||||
updated.Observer.FailureCode = ""
|
||||
updated.Observer.UpdatedAt = now
|
||||
}
|
||||
updated.UpdatedBy = normalizePatrolObjectiveActor(input.Actor)
|
||||
updated.UpdatedAt = now
|
||||
updated.Revision++
|
||||
@@ -449,7 +468,7 @@ func (s *PatrolObjectiveStore) ProposeObserver(id string, input ProposePatrolObs
|
||||
if current.Status != PatrolObjectiveActive {
|
||||
return PatrolObjective{}, fmt.Errorf("%w: observer proposals require an active objective", ErrPatrolObjectiveInvalid)
|
||||
}
|
||||
if current.Observer != nil && current.Observer.State != PatrolObserverDisabled {
|
||||
if current.Observer != nil && current.Observer.State != PatrolObserverDisabled && current.Observer.State != PatrolObserverRejected && current.Observer.State != PatrolObserverDegraded {
|
||||
return PatrolObjective{}, fmt.Errorf("%w: an existing observer cannot be displaced by a proposal", ErrPatrolObjectiveInvalid)
|
||||
}
|
||||
|
||||
@@ -519,8 +538,8 @@ func (s *PatrolObjectiveStore) GetObserverArtifact(id string) (PatrolObserverArt
|
||||
}
|
||||
|
||||
// RecordObserver persists a core-owned observer lifecycle transition. Public
|
||||
// objective clients cannot call this method through HTTP. A future constrained
|
||||
// monitor builder and installer must use it only after validating the observer
|
||||
// objective clients cannot call this method through HTTP. The constrained
|
||||
// monitor builder and installer use it only after validating the observer
|
||||
// artifact, declared read-only posture, triggers, and health lease.
|
||||
func (s *PatrolObjectiveStore) RecordObserver(id string, expectedRevision uint64, observer PatrolObserverRecord, actor string, now time.Time) (PatrolObjective, error) {
|
||||
if s == nil {
|
||||
@@ -566,6 +585,55 @@ func (s *PatrolObjectiveStore) RecordObserver(id string, expectedRevision uint64
|
||||
return objectiveForRead(updated, now), nil
|
||||
}
|
||||
|
||||
// RefreshObserverHealthBatch renews installed observer leases in one atomic
|
||||
// persistence transaction without changing operator-authored revisions.
|
||||
// Runtime heartbeats must not create optimistic-concurrency conflicts or one
|
||||
// encrypted document rewrite per observer.
|
||||
func (s *PatrolObjectiveStore) RefreshObserverHealthBatch(updates []patrolObserverHealthUpdate, now time.Time) (int, error) {
|
||||
if s == nil {
|
||||
return 0, fmt.Errorf("%w: store unavailable", ErrPatrolObjectiveInvalid)
|
||||
}
|
||||
now = normalizePatrolObjectiveTime(now)
|
||||
if len(updates) == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
next := clonePatrolObjectiveMap(s.objectives)
|
||||
refreshed := 0
|
||||
seen := make(map[string]struct{}, len(updates))
|
||||
for _, update := range updates {
|
||||
objectiveID := strings.TrimSpace(update.ObjectiveID)
|
||||
observerID := strings.TrimSpace(update.ObserverID)
|
||||
validUntil := update.ValidUntil.UTC()
|
||||
evidenceAt := update.EvidenceAt.UTC()
|
||||
if objectiveID == "" || observerID == "" || update.Version == 0 || !validUntil.After(now) || evidenceAt.After(now.Add(time.Minute)) {
|
||||
return 0, fmt.Errorf("%w: invalid observer health lease", ErrPatrolObjectiveInvalid)
|
||||
}
|
||||
if _, duplicate := seen[objectiveID]; duplicate {
|
||||
return 0, fmt.Errorf("%w: duplicate observer health lease", ErrPatrolObjectiveInvalid)
|
||||
}
|
||||
seen[objectiveID] = struct{}{}
|
||||
current, ok := next[objectiveID]
|
||||
if !ok || current.Status != PatrolObjectiveActive || current.Observer == nil || current.Observer.ID != observerID || current.Observer.Version != update.Version || current.Observer.State != PatrolObserverInstalled {
|
||||
continue
|
||||
}
|
||||
current.Observer.ValidUntil = &validUntil
|
||||
current.Observer.LastEvidenceAt = &evidenceAt
|
||||
current.Observer.UpdatedAt = now
|
||||
refreshed++
|
||||
}
|
||||
if refreshed == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
if err := s.persistLocked(next); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
s.objectives = next
|
||||
return refreshed, nil
|
||||
}
|
||||
|
||||
func (s *PatrolObjectiveStore) persistLocked(objectives map[string]*PatrolObjective) error {
|
||||
if s.filePath == "" {
|
||||
return nil
|
||||
@@ -664,6 +732,13 @@ func derivePatrolObjectiveCoverage(objective *PatrolObjective, now time.Time) Pa
|
||||
coverage.State = PatrolObjectiveUncovered
|
||||
coverage.ReasonCode = "observer_proposed"
|
||||
coverage.Summary = "An observer has been proposed but not validated or installed."
|
||||
case PatrolObserverRejected:
|
||||
coverage.State = PatrolObjectiveUncovered
|
||||
coverage.ReasonCode = observer.FailureCode
|
||||
if coverage.ReasonCode == "" {
|
||||
coverage.ReasonCode = "observer_rejected"
|
||||
}
|
||||
coverage.Summary = "The proposed observer could not be validated for the local runtime."
|
||||
case PatrolObserverValidated:
|
||||
coverage.State = PatrolObjectiveUncovered
|
||||
coverage.ReasonCode = "observer_not_installed"
|
||||
@@ -832,10 +907,10 @@ func normalizePatrolObserver(observer PatrolObserverRecord, now time.Time) (Patr
|
||||
if observer.FailureCode != "" && !isPatrolMachineCode(observer.FailureCode) {
|
||||
return PatrolObserverRecord{}, fmt.Errorf("%w: invalid observer failure code", ErrPatrolObjectiveInvalid)
|
||||
}
|
||||
if observer.State == PatrolObserverDegraded && observer.FailureCode == "" {
|
||||
return PatrolObserverRecord{}, fmt.Errorf("%w: degraded observer requires a failure code", ErrPatrolObjectiveInvalid)
|
||||
if (observer.State == PatrolObserverDegraded || observer.State == PatrolObserverRejected) && observer.FailureCode == "" {
|
||||
return PatrolObserverRecord{}, fmt.Errorf("%w: rejected or degraded observer requires a failure code", ErrPatrolObjectiveInvalid)
|
||||
}
|
||||
if observer.State != PatrolObserverDegraded {
|
||||
if observer.State != PatrolObserverDegraded && observer.State != PatrolObserverRejected {
|
||||
observer.FailureCode = ""
|
||||
}
|
||||
observer.ValidUntil = clonePatrolTime(observer.ValidUntil)
|
||||
@@ -876,7 +951,8 @@ func validatePatrolObserverTransition(current *PatrolObserverRecord, next Patrol
|
||||
return fmt.Errorf("%w: observer identity and artifact are immutable within a version", ErrPatrolObjectiveInvalid)
|
||||
}
|
||||
allowed := map[PatrolObserverState]map[PatrolObserverState]bool{
|
||||
PatrolObserverProposed: {PatrolObserverProposed: true, PatrolObserverValidated: true, PatrolObserverDisabled: true},
|
||||
PatrolObserverProposed: {PatrolObserverProposed: true, PatrolObserverRejected: true, PatrolObserverValidated: true, PatrolObserverDisabled: true},
|
||||
PatrolObserverRejected: {PatrolObserverRejected: true, PatrolObserverValidated: true, PatrolObserverDisabled: true},
|
||||
PatrolObserverValidated: {PatrolObserverValidated: true, PatrolObserverInstalled: true, PatrolObserverDegraded: true, PatrolObserverDisabled: true},
|
||||
PatrolObserverInstalled: {PatrolObserverInstalled: true, PatrolObserverDegraded: true, PatrolObserverDisabled: true},
|
||||
PatrolObserverDegraded: {PatrolObserverDegraded: true, PatrolObserverInstalled: true, PatrolObserverDisabled: true},
|
||||
@@ -919,7 +995,7 @@ func isPatrolObjectiveStatus(status PatrolObjectiveStatus) bool {
|
||||
|
||||
func isPatrolObserverState(state PatrolObserverState) bool {
|
||||
switch state {
|
||||
case PatrolObserverProposed, PatrolObserverValidated, PatrolObserverInstalled, PatrolObserverDegraded, PatrolObserverDisabled:
|
||||
case PatrolObserverProposed, PatrolObserverRejected, PatrolObserverValidated, PatrolObserverInstalled, PatrolObserverDegraded, PatrolObserverDisabled:
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
|
||||
@@ -0,0 +1,411 @@
|
||||
package ai
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
"github.com/rs/zerolog/log"
|
||||
)
|
||||
|
||||
const (
|
||||
patrolObserverRuntimeFormat = "pulse-resource-state/v1"
|
||||
patrolObserverSweepInterval = 5 * time.Second
|
||||
patrolObserverMinSampleInterval = 10 * time.Second
|
||||
patrolObserverMaxSampleInterval = 5 * time.Minute
|
||||
patrolObserverMinLease = 2 * time.Minute
|
||||
patrolObserverMaxLease = 30 * time.Minute
|
||||
patrolObserverMaxConsecutiveFailures = 10
|
||||
)
|
||||
|
||||
type patrolResourceStateProbe struct {
|
||||
Runtime string `json:"runtime"`
|
||||
Path string `json:"path"`
|
||||
Operator string `json:"operator"`
|
||||
Value string `json:"value"`
|
||||
SampleIntervalSeconds int `json:"sample_interval_seconds"`
|
||||
WakeAfterConsecutiveFailures int `json:"wake_after_consecutive_failures"`
|
||||
}
|
||||
|
||||
type patrolObserverValidationError struct{ code string }
|
||||
|
||||
func (e *patrolObserverValidationError) Error() string { return e.code }
|
||||
|
||||
type patrolObserverExecution struct {
|
||||
nextDue time.Time
|
||||
consecutiveFailures int
|
||||
wakeEmitted bool
|
||||
}
|
||||
|
||||
type patrolObserverRuntime struct {
|
||||
mu sync.Mutex
|
||||
executions map[string]patrolObserverExecution
|
||||
}
|
||||
|
||||
func newPatrolObserverRuntime() *patrolObserverRuntime {
|
||||
return &patrolObserverRuntime{executions: make(map[string]patrolObserverExecution)}
|
||||
}
|
||||
|
||||
func (p *PatrolService) objectiveObserverLoop(ctx context.Context) {
|
||||
ticker := time.NewTicker(patrolObserverSweepInterval)
|
||||
defer ticker.Stop()
|
||||
p.processObjectiveObservers(time.Now().UTC())
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-p.stopCh:
|
||||
return
|
||||
case now := <-ticker.C:
|
||||
p.processObjectiveObservers(now.UTC())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (p *PatrolService) processObjectiveObservers(now time.Time) {
|
||||
if p == nil {
|
||||
return
|
||||
}
|
||||
p.mu.RLock()
|
||||
store := p.objectiveStore
|
||||
runtime := p.observerRuntime
|
||||
p.mu.RUnlock()
|
||||
if store == nil || runtime == nil {
|
||||
return
|
||||
}
|
||||
activeExecutions := make(map[string]struct{})
|
||||
healthUpdates := make([]patrolObserverHealthUpdate, 0)
|
||||
for _, objective := range store.List(false, now) {
|
||||
if objective.Status != PatrolObjectiveActive || objective.Observer == nil {
|
||||
continue
|
||||
}
|
||||
activeExecutions[fmt.Sprintf("%s/%d", objective.Observer.ID, objective.Observer.Version)] = struct{}{}
|
||||
if update := p.reconcileObjectiveObserver(store, runtime, objective, now); update != nil {
|
||||
healthUpdates = append(healthUpdates, *update)
|
||||
}
|
||||
}
|
||||
runtime.retain(activeExecutions)
|
||||
if _, err := store.RefreshObserverHealthBatch(healthUpdates, now); err != nil {
|
||||
log.Warn().Err(err).Int("lease_count", len(healthUpdates)).Msg("Patrol observer health lease batch failed")
|
||||
}
|
||||
}
|
||||
|
||||
func (r *patrolObserverRuntime) retain(active map[string]struct{}) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
for key := range r.executions {
|
||||
if _, ok := active[key]; !ok {
|
||||
delete(r.executions, key)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (p *PatrolService) reconcileObjectiveObserver(store *PatrolObjectiveStore, runtime *patrolObserverRuntime, objective PatrolObjective, now time.Time) *patrolObserverHealthUpdate {
|
||||
artifact, ok := store.GetObserverArtifact(objective.ID)
|
||||
if !ok {
|
||||
p.recordObserverValidationFailure(store, objective, "observer_artifact_missing", now)
|
||||
return nil
|
||||
}
|
||||
probe, validationErr := validatePatrolObserverArtifact(objective, artifact)
|
||||
if validationErr != nil {
|
||||
p.recordObserverValidationFailure(store, objective, validationErr.code, now)
|
||||
return nil
|
||||
}
|
||||
|
||||
for objective.Observer != nil && (objective.Observer.State == PatrolObserverProposed || objective.Observer.State == PatrolObserverRejected || objective.Observer.State == PatrolObserverValidated || objective.Observer.State == PatrolObserverDegraded) {
|
||||
next := *clonePatrolObserver(objective.Observer)
|
||||
next.Artifact = &artifact
|
||||
switch objective.Observer.State {
|
||||
case PatrolObserverProposed:
|
||||
next.State = PatrolObserverValidated
|
||||
next.FailureCode = ""
|
||||
case PatrolObserverRejected:
|
||||
next.State = PatrolObserverValidated
|
||||
next.FailureCode = ""
|
||||
case PatrolObserverValidated:
|
||||
next.State = PatrolObserverInstalled
|
||||
next.ValidUntil = nil
|
||||
next.LastEvidenceAt = nil
|
||||
case PatrolObserverDegraded:
|
||||
next.State = PatrolObserverInstalled
|
||||
next.FailureCode = ""
|
||||
next.ValidUntil = nil
|
||||
next.LastEvidenceAt = nil
|
||||
}
|
||||
updated, err := store.RecordObserver(objective.ID, objective.Revision, next, "patrol:observer-runtime", now)
|
||||
if err != nil {
|
||||
if !errors.Is(err, ErrPatrolObjectiveConflict) {
|
||||
log.Warn().Err(err).Str("objective_id", objective.ID).Msg("Patrol observer lifecycle transition failed")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
objective = updated
|
||||
}
|
||||
if objective.Observer == nil || (objective.Observer.State != PatrolObserverInstalled && objective.Observer.State != PatrolObserverDegraded) {
|
||||
return nil
|
||||
}
|
||||
return p.evaluateObjectiveObserver(runtime, objective, probe, now)
|
||||
}
|
||||
|
||||
func (p *PatrolService) recordObserverValidationFailure(store *PatrolObjectiveStore, objective PatrolObjective, code string, now time.Time) {
|
||||
if objective.Observer == nil || objective.Observer.State == PatrolObserverDisabled || objective.Observer.FailureCode == code {
|
||||
return
|
||||
}
|
||||
artifact, _ := store.GetObserverArtifact(objective.ID)
|
||||
next := *clonePatrolObserver(objective.Observer)
|
||||
next.FailureCode = code
|
||||
if next.State == PatrolObserverProposed || next.State == PatrolObserverRejected {
|
||||
next.State = PatrolObserverRejected
|
||||
} else {
|
||||
next.State = PatrolObserverDegraded
|
||||
next.ValidUntil = nil
|
||||
}
|
||||
if artifact.Format != "" {
|
||||
next.Artifact = &artifact
|
||||
}
|
||||
if _, err := store.RecordObserver(objective.ID, objective.Revision, next, "patrol:observer-validator", now); err != nil && !errors.Is(err, ErrPatrolObjectiveConflict) {
|
||||
log.Warn().Err(err).Str("objective_id", objective.ID).Str("failure_code", code).Msg("Patrol observer validation failure could not be recorded")
|
||||
}
|
||||
}
|
||||
|
||||
func validatePatrolObserverArtifact(objective PatrolObjective, artifact PatrolObserverArtifact) (patrolResourceStateProbe, *patrolObserverValidationError) {
|
||||
fail := func(code string) (patrolResourceStateProbe, *patrolObserverValidationError) {
|
||||
return patrolResourceStateProbe{}, &patrolObserverValidationError{code: code}
|
||||
}
|
||||
if objective.Observer == nil || !objective.Observer.ReadOnly {
|
||||
return fail("observer_not_read_only")
|
||||
}
|
||||
if len(objective.Scope.ResourceIDs) == 0 {
|
||||
return fail("observer_scope_required")
|
||||
}
|
||||
if len(objective.Observer.TriggerKinds) != 1 || objective.Observer.TriggerKinds[0] != PatrolObserverTriggerInterval {
|
||||
return fail("observer_trigger_unsupported")
|
||||
}
|
||||
var requirements map[string]json.RawMessage
|
||||
if err := decodeStrictJSONObject(artifact.Requirements, &requirements); err != nil {
|
||||
return fail("observer_requirements_invalid")
|
||||
}
|
||||
if len(requirements) != 0 {
|
||||
return fail("observer_requirements_unsupported")
|
||||
}
|
||||
var probe patrolResourceStateProbe
|
||||
if err := decodeStrictJSONObject(artifact.Probe, &probe); err != nil {
|
||||
return fail("observer_probe_invalid")
|
||||
}
|
||||
if probe.Runtime != patrolObserverRuntimeFormat {
|
||||
return fail("observer_runtime_unsupported")
|
||||
}
|
||||
if probe.Path != "status" {
|
||||
return fail("observer_path_unsupported")
|
||||
}
|
||||
if probe.Operator != "equals" && probe.Operator != "not_equals" {
|
||||
return fail("observer_operator_unsupported")
|
||||
}
|
||||
switch unifiedresources.ResourceStatus(strings.ToLower(strings.TrimSpace(probe.Value))) {
|
||||
case unifiedresources.StatusOnline, unifiedresources.StatusOffline, unifiedresources.StatusWarning, unifiedresources.StatusUnknown:
|
||||
probe.Value = strings.ToLower(strings.TrimSpace(probe.Value))
|
||||
default:
|
||||
return fail("observer_value_unsupported")
|
||||
}
|
||||
interval := time.Duration(probe.SampleIntervalSeconds) * time.Second
|
||||
if interval < patrolObserverMinSampleInterval || interval > patrolObserverMaxSampleInterval {
|
||||
return fail("observer_interval_out_of_bounds")
|
||||
}
|
||||
if probe.WakeAfterConsecutiveFailures < 1 || probe.WakeAfterConsecutiveFailures > patrolObserverMaxConsecutiveFailures {
|
||||
return fail("observer_failure_window_out_of_bounds")
|
||||
}
|
||||
return probe, nil
|
||||
}
|
||||
|
||||
func decodeStrictJSONObject(data []byte, target interface{}) error {
|
||||
decoder := json.NewDecoder(bytes.NewReader(data))
|
||||
decoder.DisallowUnknownFields()
|
||||
if err := decoder.Decode(target); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := decoder.Decode(&struct{}{}); err != io.EOF {
|
||||
return fmt.Errorf("expected one JSON object")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (p *PatrolService) evaluateObjectiveObserver(runtime *patrolObserverRuntime, objective PatrolObjective, probe patrolResourceStateProbe, now time.Time) *patrolObserverHealthUpdate {
|
||||
observer := objective.Observer
|
||||
key := fmt.Sprintf("%s/%d", observer.ID, observer.Version)
|
||||
runtime.mu.Lock()
|
||||
execution := runtime.executions[key]
|
||||
if !execution.nextDue.IsZero() && now.Before(execution.nextDue) {
|
||||
runtime.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
interval := time.Duration(probe.SampleIntervalSeconds) * time.Second
|
||||
execution.nextDue = now.Add(interval)
|
||||
runtime.mu.Unlock()
|
||||
|
||||
statuses := patrolRuntimeCanonicalStatuses(p.currentPatrolRuntimeState())
|
||||
failing := make([]string, 0)
|
||||
for _, resourceID := range objective.Scope.ResourceIDs {
|
||||
status, exists := statuses[canonicalPatrolScopeToken(resourceID)]
|
||||
matched := exists && string(status) == probe.Value
|
||||
if probe.Operator == "not_equals" {
|
||||
matched = exists && string(status) != probe.Value
|
||||
}
|
||||
if !matched {
|
||||
failing = append(failing, resourceID)
|
||||
}
|
||||
}
|
||||
|
||||
runtime.mu.Lock()
|
||||
execution = runtime.executions[key]
|
||||
if len(failing) == 0 {
|
||||
execution.consecutiveFailures = 0
|
||||
execution.wakeEmitted = false
|
||||
} else {
|
||||
execution.consecutiveFailures++
|
||||
}
|
||||
shouldWake := len(failing) > 0 && execution.consecutiveFailures >= probe.WakeAfterConsecutiveFailures && !execution.wakeEmitted
|
||||
runtime.executions[key] = execution
|
||||
runtime.mu.Unlock()
|
||||
|
||||
leaseDuration := 3 * interval
|
||||
if leaseDuration < patrolObserverMinLease {
|
||||
leaseDuration = patrolObserverMinLease
|
||||
}
|
||||
if leaseDuration > patrolObserverMaxLease {
|
||||
leaseDuration = patrolObserverMaxLease
|
||||
}
|
||||
refreshLease := observer.ValidUntil == nil || observer.ValidUntil.Sub(now) <= leaseDuration/2
|
||||
var healthUpdate *patrolObserverHealthUpdate
|
||||
if refreshLease {
|
||||
healthUpdate = &patrolObserverHealthUpdate{
|
||||
ObjectiveID: objective.ID,
|
||||
ObserverID: observer.ID,
|
||||
Version: observer.Version,
|
||||
ValidUntil: now.Add(leaseDuration),
|
||||
EvidenceAt: now,
|
||||
}
|
||||
}
|
||||
if !shouldWake {
|
||||
return healthUpdate
|
||||
}
|
||||
sort.Strings(failing)
|
||||
scope := PatrolScope{
|
||||
ResourceIDs: failing,
|
||||
Depth: PatrolDepthQuick,
|
||||
Reason: TriggerReasonObjectiveEvidence,
|
||||
Priority: triggerPriorityObjective,
|
||||
Context: fmt.Sprintf("Local observer %s for objective %s detected %d of %d scoped resources outside its canonical status predicate.", observer.ID, objective.ID, len(failing), len(objective.Scope.ResourceIDs)),
|
||||
}
|
||||
p.mu.RLock()
|
||||
tm := p.triggerManager
|
||||
p.mu.RUnlock()
|
||||
accepted := tm != nil && tm.TriggerPatrol(scope)
|
||||
if accepted {
|
||||
runtime.mu.Lock()
|
||||
execution = runtime.executions[key]
|
||||
execution.wakeEmitted = true
|
||||
runtime.executions[key] = execution
|
||||
runtime.mu.Unlock()
|
||||
log.Info().Str("objective_id", objective.ID).Int("affected_resources", len(failing)).Msg("Patrol objective observer queued an evidence-triggered check")
|
||||
}
|
||||
return healthUpdate
|
||||
}
|
||||
|
||||
func patrolRuntimeCanonicalStatuses(state patrolRuntimeState) map[string]unifiedresources.ResourceStatus {
|
||||
result := make(map[string]unifiedresources.ResourceStatus)
|
||||
add := func(status unifiedresources.ResourceStatus, ids ...string) {
|
||||
for _, id := range ids {
|
||||
if token := canonicalPatrolScopeToken(id); token != "" {
|
||||
result[token] = normalizePatrolResourceStatus(string(status))
|
||||
}
|
||||
}
|
||||
}
|
||||
if rs := state.readState; rs != nil {
|
||||
for _, v := range rs.VMs() {
|
||||
add(v.Status(), v.ID(), v.SourceID())
|
||||
}
|
||||
for _, v := range rs.Containers() {
|
||||
add(v.Status(), v.ID(), v.SourceID())
|
||||
}
|
||||
for _, v := range rs.Nodes() {
|
||||
add(v.Status(), v.ID(), v.SourceID())
|
||||
}
|
||||
for _, v := range rs.Hosts() {
|
||||
add(v.Status(), v.ID())
|
||||
}
|
||||
for _, v := range rs.DockerHosts() {
|
||||
add(v.Status(), v.ID(), v.HostSourceID())
|
||||
}
|
||||
for _, v := range rs.DockerContainers() {
|
||||
add(v.Status(), v.ID(), v.ContainerID())
|
||||
}
|
||||
for _, v := range rs.StoragePools() {
|
||||
add(v.Status(), v.ID(), v.SourceID())
|
||||
}
|
||||
for _, v := range rs.PhysicalDisks() {
|
||||
add(v.Status(), v.ID())
|
||||
}
|
||||
for _, v := range rs.PBSInstances() {
|
||||
add(v.Status(), v.ID())
|
||||
}
|
||||
for _, v := range rs.PMGInstances() {
|
||||
add(v.Status(), v.ID())
|
||||
}
|
||||
for _, v := range rs.K8sClusters() {
|
||||
add(v.Status(), v.ID())
|
||||
}
|
||||
return result
|
||||
}
|
||||
for _, v := range state.VMs {
|
||||
add(normalizePatrolResourceStatus(v.Status), v.ID)
|
||||
}
|
||||
for _, v := range state.Containers {
|
||||
add(normalizePatrolResourceStatus(v.Status), v.ID)
|
||||
}
|
||||
for _, v := range state.Nodes {
|
||||
add(normalizePatrolResourceStatus(v.Status), v.ID)
|
||||
}
|
||||
for _, v := range state.Hosts {
|
||||
add(normalizePatrolResourceStatus(v.Status), v.ID)
|
||||
}
|
||||
for _, v := range state.DockerHosts {
|
||||
add(normalizePatrolResourceStatus(v.Status), v.ID)
|
||||
for _, container := range v.Containers {
|
||||
add(normalizePatrolResourceStatus(container.State), container.ID)
|
||||
}
|
||||
}
|
||||
for _, v := range state.Storage {
|
||||
add(normalizePatrolResourceStatus(v.Status), v.ID)
|
||||
}
|
||||
for _, v := range state.PBSInstances {
|
||||
add(normalizePatrolResourceStatus(v.Status), v.ID)
|
||||
}
|
||||
for _, v := range state.PMGInstances {
|
||||
add(normalizePatrolResourceStatus(v.Status), v.ID)
|
||||
}
|
||||
for _, v := range state.KubernetesClusters {
|
||||
add(normalizePatrolResourceStatus(v.Status), v.ID)
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
func normalizePatrolResourceStatus(value string) unifiedresources.ResourceStatus {
|
||||
switch strings.ToLower(strings.TrimSpace(value)) {
|
||||
case "online", "running", "active", "healthy", "up", "available", "ready":
|
||||
return unifiedresources.StatusOnline
|
||||
case "offline", "stopped", "inactive", "failed", "error", "down", "unavailable":
|
||||
return unifiedresources.StatusOffline
|
||||
case "warning", "degraded", "unhealthy":
|
||||
return unifiedresources.StatusWarning
|
||||
default:
|
||||
return unifiedresources.StatusUnknown
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,188 @@
|
||||
package ai
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
)
|
||||
|
||||
func createInstallablePatrolObserver(t *testing.T, store *PatrolObjectiveStore, now time.Time, resourceIDs ...string) PatrolObjective {
|
||||
t.Helper()
|
||||
objective, err := store.Create(CreatePatrolObjectiveInput{
|
||||
Brief: "Keep scoped resources online",
|
||||
ResourceIDs: resourceIDs,
|
||||
}, now)
|
||||
if err != nil {
|
||||
t.Fatalf("create objective: %v", err)
|
||||
}
|
||||
objective, err = store.ProposeObserver(objective.ID, ProposePatrolObserverInput{
|
||||
ExpectedRevision: objective.Revision,
|
||||
Interpretation: "Every scoped canonical resource remains online.",
|
||||
TriggerKinds: []PatrolObserverTriggerKind{PatrolObserverTriggerInterval},
|
||||
ProbeJSON: `{"runtime":"pulse-resource-state/v1","path":"status","operator":"equals","value":"online","sample_interval_seconds":10,"wake_after_consecutive_failures":2}`,
|
||||
WakeEvidence: "A scoped resource is not online for two consecutive samples.",
|
||||
RequirementsJSON: `{}`,
|
||||
Actor: "patrol:model",
|
||||
}, now)
|
||||
if err != nil {
|
||||
t.Fatalf("propose observer: %v", err)
|
||||
}
|
||||
return objective
|
||||
}
|
||||
|
||||
func TestPatrolObserverRuntimeInstallsEvaluatesAndLeasesWithoutObjectiveRevisionChurn(t *testing.T) {
|
||||
now := time.Date(2026, 8, 14, 1, 0, 0, 0, time.UTC)
|
||||
store := NewInMemoryPatrolObjectiveStore()
|
||||
objective := createInstallablePatrolObserver(t, store, now, "node-1")
|
||||
proposalRevision := objective.Revision
|
||||
patrol := NewPatrolService(nil, mockPatrolStateProvider{state: models.StateSnapshot{
|
||||
Nodes: []models.Node{{ID: "node-1", Name: "node-1", Status: "online"}},
|
||||
}})
|
||||
patrol.SetObjectiveStore(store)
|
||||
|
||||
patrol.processObjectiveObservers(now.Add(time.Second))
|
||||
got, ok := store.Get(objective.ID, now.Add(time.Second))
|
||||
if !ok || got.Observer == nil {
|
||||
t.Fatalf("installed objective missing: %+v, found=%v", got, ok)
|
||||
}
|
||||
if got.Observer.State != PatrolObserverInstalled || got.Coverage.State != PatrolObjectiveCovered || got.Coverage.ReasonCode != "observer_healthy" {
|
||||
t.Fatalf("installed observer = %+v, coverage = %+v", got.Observer, got.Coverage)
|
||||
}
|
||||
if got.Observer.ValidUntil == nil || got.Observer.LastEvidenceAt == nil {
|
||||
t.Fatalf("observer lease was not persisted: %+v", got.Observer)
|
||||
}
|
||||
if got.Revision != proposalRevision+2 {
|
||||
t.Fatalf("objective revision = %d, want proposal revision + validation/install only (%d)", got.Revision, proposalRevision+2)
|
||||
}
|
||||
|
||||
patrol.processObjectiveObservers(now.Add(2 * time.Minute))
|
||||
refreshed, _ := store.Get(objective.ID, now.Add(2*time.Minute))
|
||||
if refreshed.Revision != got.Revision {
|
||||
t.Fatalf("health heartbeat changed objective revision from %d to %d", got.Revision, refreshed.Revision)
|
||||
}
|
||||
if !refreshed.Observer.ValidUntil.After(*got.Observer.ValidUntil) {
|
||||
t.Fatalf("health lease did not advance: old=%v new=%v", got.Observer.ValidUntil, refreshed.Observer.ValidUntil)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPatrolObserverRuntimeRecordsExplicitUnsupportedValidationReason(t *testing.T) {
|
||||
now := time.Date(2026, 8, 14, 1, 0, 0, 0, time.UTC)
|
||||
store := NewInMemoryPatrolObjectiveStore()
|
||||
objective, err := store.Create(CreatePatrolObjectiveInput{Brief: "Keep playback smooth", ResourceIDs: []string{"jellyfin"}}, now)
|
||||
if err != nil {
|
||||
t.Fatalf("create objective: %v", err)
|
||||
}
|
||||
objective, err = store.ProposeObserver(objective.ID, ProposePatrolObserverInput{
|
||||
ExpectedRevision: objective.Revision,
|
||||
Interpretation: "Observe playback buffering events.",
|
||||
TriggerKinds: []PatrolObserverTriggerKind{PatrolObserverTriggerAPI},
|
||||
ProbeJSON: `{"runtime":"jellyfin-api/v1","event":"playback"}`,
|
||||
WakeEvidence: "Playback buffers.",
|
||||
RequirementsJSON: `{"network":["jellyfin"]}`,
|
||||
}, now)
|
||||
if err != nil {
|
||||
t.Fatalf("propose observer: %v", err)
|
||||
}
|
||||
patrol := NewPatrolService(nil, nil)
|
||||
patrol.SetObjectiveStore(store)
|
||||
patrol.processObjectiveObservers(now.Add(time.Second))
|
||||
|
||||
got, _ := store.Get(objective.ID, now.Add(time.Second))
|
||||
if got.Observer == nil || got.Observer.State != PatrolObserverRejected {
|
||||
t.Fatalf("unsupported proposal state = %+v", got.Observer)
|
||||
}
|
||||
if got.Observer.FailureCode != "observer_trigger_unsupported" || got.Coverage.ReasonCode != "observer_trigger_unsupported" {
|
||||
t.Fatalf("validation failure was not projected explicitly: observer=%+v coverage=%+v", got.Observer, got.Coverage)
|
||||
}
|
||||
if got.Coverage.State != PatrolObjectiveUncovered || !strings.Contains(got.Coverage.Summary, "could not be validated") {
|
||||
t.Fatalf("unsupported proposal coverage = %+v", got.Coverage)
|
||||
}
|
||||
revised, err := store.ProposeObserver(objective.ID, ProposePatrolObserverInput{
|
||||
ExpectedRevision: got.Revision,
|
||||
Interpretation: "Keep the canonical Jellyfin resource online.",
|
||||
TriggerKinds: []PatrolObserverTriggerKind{PatrolObserverTriggerInterval},
|
||||
ProbeJSON: `{"runtime":"pulse-resource-state/v1","path":"status","operator":"equals","value":"online","sample_interval_seconds":30,"wake_after_consecutive_failures":2}`,
|
||||
WakeEvidence: "The Jellyfin resource is not online twice.",
|
||||
RequirementsJSON: `{}`,
|
||||
}, now.Add(2*time.Second))
|
||||
if err != nil {
|
||||
t.Fatalf("replace rejected observer proposal: %v", err)
|
||||
}
|
||||
if revised.Observer == nil || revised.Observer.Version != got.Observer.Version+1 || revised.Observer.State != PatrolObserverProposed {
|
||||
t.Fatalf("replacement proposal = %+v", revised.Observer)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPatrolObserverRuntimeWakesModelOnlyAfterLocalFailureWindow(t *testing.T) {
|
||||
now := time.Date(2026, 8, 14, 1, 0, 0, 0, time.UTC)
|
||||
store := NewInMemoryPatrolObjectiveStore()
|
||||
objective := createInstallablePatrolObserver(t, store, now, "camera-1")
|
||||
patrol := NewPatrolService(nil, mockPatrolStateProvider{state: models.StateSnapshot{
|
||||
Hosts: []models.Host{{ID: "camera-1", Hostname: "camera-1", Status: "offline"}},
|
||||
}})
|
||||
patrol.SetObjectiveStore(store)
|
||||
tm := NewTriggerManager(TriggerManagerConfig{MaxPendingTriggers: 10})
|
||||
patrol.SetTriggerManager(tm)
|
||||
|
||||
patrol.processObjectiveObservers(now.Add(time.Second))
|
||||
if got := tm.GetPendingCount(); got != 0 {
|
||||
t.Fatalf("observer woke Patrol after first failed local sample; pending=%d", got)
|
||||
}
|
||||
patrol.processObjectiveObservers(now.Add(11 * time.Second))
|
||||
if got := tm.GetPendingCount(); got != 1 {
|
||||
t.Fatalf("observer did not queue one Patrol wake after failure window; pending=%d", got)
|
||||
}
|
||||
patrol.processObjectiveObservers(now.Add(21 * time.Second))
|
||||
if got := tm.GetPendingCount(); got != 1 {
|
||||
t.Fatalf("observer repeated wake while failure remained active; pending=%d", got)
|
||||
}
|
||||
got, _ := store.Get(objective.ID, now.Add(21*time.Second))
|
||||
if got.Coverage.State != PatrolObjectiveCovered {
|
||||
t.Fatalf("runtime health and objective outcome were conflated: %+v", got.Coverage)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidatePatrolObserverArtifactRejectsUnknownExecutableFields(t *testing.T) {
|
||||
now := time.Now().UTC()
|
||||
store := NewInMemoryPatrolObjectiveStore()
|
||||
objective := createInstallablePatrolObserver(t, store, now, "node-1")
|
||||
artifact, _ := store.GetObserverArtifact(objective.ID)
|
||||
artifact.Probe = []byte(`{"runtime":"pulse-resource-state/v1","path":"status","operator":"equals","value":"online","sample_interval_seconds":30,"wake_after_consecutive_failures":2,"command":"rm -rf /"}`)
|
||||
_, err := validatePatrolObserverArtifact(objective, artifact)
|
||||
if err == nil || err.code != "observer_probe_invalid" {
|
||||
t.Fatalf("unknown executable field validation error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPatrolObjectiveIntentEditInvalidatesInstalledObserver(t *testing.T) {
|
||||
now := time.Date(2026, 8, 14, 1, 0, 0, 0, time.UTC)
|
||||
store := NewInMemoryPatrolObjectiveStore()
|
||||
objective := createInstallablePatrolObserver(t, store, now, "node-1")
|
||||
patrol := NewPatrolService(nil, mockPatrolStateProvider{state: models.StateSnapshot{
|
||||
Nodes: []models.Node{{ID: "node-1", Status: "online"}},
|
||||
}})
|
||||
patrol.SetObjectiveStore(store)
|
||||
patrol.processObjectiveObservers(now.Add(time.Second))
|
||||
installed, _ := store.Get(objective.ID, now.Add(time.Second))
|
||||
if installed.Coverage.State != PatrolObjectiveCovered {
|
||||
t.Fatalf("observer was not installed before edit: %+v", installed.Coverage)
|
||||
}
|
||||
|
||||
revisedBrief := "Keep this node online and its workload responsive"
|
||||
updated, err := store.Update(objective.ID, UpdatePatrolObjectiveInput{
|
||||
ExpectedRevision: installed.Revision,
|
||||
Brief: &revisedBrief,
|
||||
Actor: "operator",
|
||||
}, now.Add(2*time.Second))
|
||||
if err != nil {
|
||||
t.Fatalf("update objective intent: %v", err)
|
||||
}
|
||||
if updated.Observer == nil || updated.Observer.State != PatrolObserverDisabled || updated.Observer.ValidUntil != nil {
|
||||
t.Fatalf("intent edit retained stale observer authority: %+v", updated.Observer)
|
||||
}
|
||||
if updated.Coverage.State != PatrolObjectiveUncovered || updated.Coverage.ReasonCode != "observer_disabled" {
|
||||
t.Fatalf("intent edit coverage = %+v", updated.Coverage)
|
||||
}
|
||||
}
|
||||
@@ -50,6 +50,7 @@ func (p *PatrolService) Start(ctx context.Context) {
|
||||
|
||||
go p.patrolLoop(ctx)
|
||||
go p.remindSweepLoop(ctx)
|
||||
go p.objectiveObserverLoop(ctx)
|
||||
}
|
||||
|
||||
// Stop stops the patrol service. It signals the patrol loop to exit, then
|
||||
|
||||
@@ -44,6 +44,7 @@ func BackgroundAutomationEventTriggerBlock() PatrolEventTriggerBlock {
|
||||
const (
|
||||
triggerPriorityManual = 100 // User-initiated patrols run first
|
||||
triggerPriorityAlertFired = 80 // New alerts are high priority
|
||||
triggerPriorityObjective = 70 // Retained-objective evidence needs prompt investigation
|
||||
triggerPriorityAnomaly = 60 // Anomaly detection is medium-high
|
||||
triggerPriorityAlertCleared = 40 // Cleared alerts are lower priority
|
||||
triggerPriorityUserAction = 30 // User finding actions
|
||||
@@ -51,16 +52,17 @@ const (
|
||||
)
|
||||
|
||||
const (
|
||||
TriggerReasonScheduled TriggerReason = "scheduled" // Regular interval trigger
|
||||
TriggerReasonManual TriggerReason = "manual" // User-initiated patrol
|
||||
TriggerReasonAlertFired TriggerReason = "alert_fired" // New alert triggered
|
||||
TriggerReasonAlertCleared TriggerReason = "alert_cleared" // Alert was resolved
|
||||
TriggerReasonAlertFlapping TriggerReason = "alert_flapping" // Alert flapping detected and suppressed
|
||||
TriggerReasonAnomalyDetected TriggerReason = "anomaly" // Baseline breach detected
|
||||
TriggerReasonUserAction TriggerReason = "user_action" // User dismissed/snoozed finding
|
||||
TriggerReasonConfigChanged TriggerReason = "config_changed" // System configuration changed
|
||||
TriggerReasonStartup TriggerReason = "startup" // Service startup
|
||||
TriggerReasonVerification TriggerReason = "verification" // Post-fix verification
|
||||
TriggerReasonScheduled TriggerReason = "scheduled" // Regular interval trigger
|
||||
TriggerReasonManual TriggerReason = "manual" // User-initiated patrol
|
||||
TriggerReasonAlertFired TriggerReason = "alert_fired" // New alert triggered
|
||||
TriggerReasonAlertCleared TriggerReason = "alert_cleared" // Alert was resolved
|
||||
TriggerReasonAlertFlapping TriggerReason = "alert_flapping" // Alert flapping detected and suppressed
|
||||
TriggerReasonAnomalyDetected TriggerReason = "anomaly" // Baseline breach detected
|
||||
TriggerReasonUserAction TriggerReason = "user_action" // User dismissed/snoozed finding
|
||||
TriggerReasonConfigChanged TriggerReason = "config_changed" // System configuration changed
|
||||
TriggerReasonStartup TriggerReason = "startup" // Service startup
|
||||
TriggerReasonVerification TriggerReason = "verification" // Post-fix verification
|
||||
TriggerReasonObjectiveEvidence TriggerReason = "objective_evidence" // Local observer detected an objective breach
|
||||
)
|
||||
|
||||
// PatrolScope defines the scope of a patrol run
|
||||
@@ -423,7 +425,7 @@ func (tm *TriggerManager) SetEventTriggersEnabled(enabled bool) {
|
||||
// (alerts firing/clearing, anomaly detection) as opposed to user-initiated or scheduled triggers.
|
||||
func isEventDrivenTrigger(reason TriggerReason) bool {
|
||||
switch reason {
|
||||
case TriggerReasonAlertFired, TriggerReasonAlertCleared, TriggerReasonAlertFlapping, TriggerReasonAnomalyDetected:
|
||||
case TriggerReasonAlertFired, TriggerReasonAlertCleared, TriggerReasonAlertFlapping, TriggerReasonAnomalyDetected, TriggerReasonObjectiveEvidence:
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
|
||||
@@ -229,7 +229,9 @@ Returns a list of active findings with their IDs, severity, resource, and title.
|
||||
Name: agentcapabilities.PatrolProposeObserverToolName,
|
||||
Description: `Propose a durable read-only observer for an active operator objective that is currently missing coverage.
|
||||
|
||||
Use this only when the objective context says observer_missing, or when current evidence clearly requires a new observer version. Translate the operator's outcome into the smallest useful local observer without hard-coding an application into Pulse. The probe_json and requirements_json fields must each be one bounded JSON object. Describe what a future constrained runtime should observe; do not include mutation commands, credentials, or secret values.
|
||||
Use this only when the objective context says observer_missing, or when current evidence clearly requires a new observer version. Translate the operator's outcome into the smallest useful local observer without hard-coding an application into Pulse. The probe_json and requirements_json fields must each be one bounded JSON object. Do not include mutation commands, credentials, or secret values.
|
||||
|
||||
Core can currently install one generic local ABI for objectives scoped to canonical Pulse resources: trigger_kind must be interval, requirements_json must be {}, and probe_json must be exactly {"runtime":"pulse-resource-state/v1","path":"status","operator":"equals","value":"online","sample_interval_seconds":30,"wake_after_consecutive_failures":2}. The operator may be equals or not_equals; value may be online, offline, warning, or unknown; interval is 10-300 seconds and the failure window is 1-10 samples. Use this ABI when canonical resource status is a truthful interpretation. If the outcome needs an app API, event, log, file, socket, network, filesystem, secret, or richer signal, describe that honest proposal instead; core will retain it with an explicit unsupported validation reason rather than pretending it is active.
|
||||
|
||||
This tool records only a versioned proposed artifact. It does not validate, install, execute, or claim coverage. Core owns the observer ID, version, SHA-256 digest, read-only posture, sandboxing, installation, health lease, and any later transition.
|
||||
|
||||
@@ -256,7 +258,7 @@ Returns the proposed observer identity and the truthful uncovered coverage reaso
|
||||
},
|
||||
"probe_json": {
|
||||
Type: "string",
|
||||
Description: "One JSON object describing the read-only probe, signal extraction, and health/failure outputs. It is proposal material, not executable authority.",
|
||||
Description: "One JSON object describing the read-only probe. Use the documented pulse-resource-state/v1 ABI exactly when canonical resource status is sufficient; otherwise provide an honest bounded proposal for future capability validation.",
|
||||
},
|
||||
"wake_evidence": {
|
||||
Type: "string",
|
||||
@@ -264,7 +266,7 @@ Returns the proposed observer identity and the truthful uncovered coverage reaso
|
||||
},
|
||||
"requirements_json": {
|
||||
Type: "string",
|
||||
Description: "One JSON object declaring network, filesystem, secret-reference, runtime, timeout, and resource-budget requirements. Use empty arrays/objects when none; never include secret values.",
|
||||
Description: "One JSON object declaring external requirements. Use {} for the installable pulse-resource-state/v1 ABI. Never include secret values.",
|
||||
},
|
||||
},
|
||||
Required: []string{"objective_id", "expected_revision", "interpretation", "trigger_kind", "probe_json", "wake_evidence", "requirements_json"},
|
||||
|
||||
Reference in New Issue
Block a user