32 KiB
Pulse v6 Operational Trust Implementation Spec
Last updated: 2026-07-19 Status: ACTIVE Primary governance surface:
status.json.candidate_lanes.protection-posture-attention-queue
Resolved related governed surface:
internal/records/operational-trust-availability-resource-facet-2026-07-19.md
Intent
This specification turns Pulse's product direction into one executable contract.
Pulse must become the trusted operations layer for mixed private infrastructure. It must tell an operator:
- what needs attention
- which resource is affected
- why Pulse believes it
- what the likely impact is
- what protection or recovery evidence exists
- what safe next action is available
- whether that action worked
The implementation loop is:
observe -> identify -> detect -> relate -> explain -> act -> verify
This is a monitor-first product. Platform pages and the dashboard remain the normal observation surfaces. Patrol is the cross-platform attention queue and explain-act-verify workbench. Assistant is contextual help for selected resources, evidence, findings, and actions. It is not the primary navigation model.
The work is complete only when the runtime, APIs, product surfaces, migrations, tests, live-browser journeys, and governance evidence satisfy every applicable criterion in this document. A narrow detector, a polished card, or a passing unit test is not completion.
Canonical Root-Fix Decision
The canonical fix is a shared operational-trust model, not UI composition over unrelated alert, notification, recovery, and Patrol payloads.
The implementation must establish typed shared contracts for:
- resource identity and relationships
- evidence provenance, freshness, completeness, and confidence
- operational lifecycle state
- protection posture
- attention items
- governed actions and verification
Platform adapters may emit provider-specific evidence, but they must not own parallel lifecycle, posture, or action semantics. Frontend code may present the shared contracts, but it must not infer canonical trust state from labels, message text, or loose metadata maps.
Product Jobs
The least-expert plausible operator should be able to complete these jobs in their own vocabulary:
- "Show me what needs my attention."
- "Tell me what is affected and how serious it is."
- "Tell me whether the data is current and where it came from."
- "Tell me whether this workload has a usable recent backup."
- "Give me the safest next step."
- "Let me approve or run a supported action."
- "Show me whether the problem is actually fixed."
- "When everything is healthy, say so plainly and get out of the way."
The default experience must not require the operator to understand internal terms such as detector, evidence envelope, rollup, canonical identity, projection, or verification intent.
Governing Product Rules
1. Evidence before assertion
Every user-visible operational assertion must link to typed evidence. Evidence
must include source, observation time, ingestion time, subject identity, and
completeness. When freshness or completeness is insufficient, the product says
unknown or stale; it does not silently downgrade those states to healthy.
2. One lifecycle truth
The active count, navigation badge, alert list, Patrol queue, notification history, and resource detail must derive from the same canonical lifecycle truth. They may show different filtered views, but they must not disagree about whether the same operational item is open, acknowledged, suppressed, stale, or resolved.
3. Identity before correlation
Pulse must resolve observations to canonical resources before it correlates impact, recovery evidence, availability, or actions. Explicit links win. Provider-scoped stable identities are next. Unambiguous secondary identity is allowed only with auditable match evidence. Ambiguous matches remain separate and visible as unresolved.
4. Protection evidence, not a backup engine
Pulse reports protection and recovery evidence from connected providers. It does not claim to execute, own, or guarantee backup and restore workflows that belong to PBS, Proxmox, TrueNAS, Kubernetes tooling, or another native control plane.
5. Monitor first, attention second
Platform surfaces answer "what is happening here?" Patrol answers "what needs me across everything?" The same attention item may be entered from either surface without becoming two separate records.
6. Actions are declared capabilities
An action exists only when the target resource declares the capability and the policy engine can determine its eligibility. Unknown, stale, incomplete, or ambiguous evidence must fail closed for mutating actions.
7. Verification is part of the action
An execution result is not enough. Every mutating action must define a
postcondition and a bounded verification policy. The UI distinguishes
executed from verified.
8. Calm is a valid state
When nothing needs attention, Patrol shows a short calm-day state with the last successful evaluation time and any stale or disconnected coverage caveat. It must not invent proof strips, scores, or decorative activity to make an empty queue look busy.
Explicit Non-Goals
This specification does not authorize:
- a general backup scheduler, retention manager, or restore engine
- broad autonomous remediation
- arbitrary shell or vendor API execution
- a chat-first home screen
- replacement of native vendor control planes
- a universal recoverability score that hides missing evidence
- automatic merging of ambiguous resources
- a second alert engine inside Patrol
- availability checks as duplicate resources when a canonical target is known
- platform-specific lifecycle states in primary APIs
- hiding stale, partial-permission, or collection-error states
- MSP multi-tenancy, fleet-wide delegation, or policy inheritance beyond existing governed enterprise boundaries
Canonical Domain Model
The implementation may refine names to match existing packages, but it must preserve the semantics and typed boundaries below.
1. Evidence envelope
Every detector input and every trust-bearing output must be traceable to an evidence envelope.
EvidenceEnvelope
id stable within its source and observation
source provider and collector identity
subject canonical resource id or unresolved provider ref
observedAt time represented by the source
ingestedAt time Pulse accepted the observation
validUntil optional source-specific freshness boundary
completeness complete | partial | unavailable
confidence confirmed | inferred | unknown
reason typed reason for partial, unavailable, or inferred
permissions sufficient | partial | denied | unknown
payloadRef bounded reference to provider detail
correlation optional auditable identity match record
Required behavior:
- freshness is evaluated from
observedAtand source policy, never only from frontend wall-clock assumptions - partial permission is different from no data
- provider failure is different from an observed healthy value
- inferred identity records the matched fields and rule
- raw provider payloads stay bounded and policy-shaped
- evidence needed for an open lifecycle item remains available for the lifecycle retention window
2. Resource identity and relationships
The unified resource registry remains the owner of canonical resource identity. It must support auditable relationships required by operational reasoning:
ResourceRelationship
id
fromResourceId
toResourceId
kind runs-on | hosted-by | depends-on | stores-on |
protected-by | checks | member-of | exposes
source
observedAt
confidence
evidenceId
Identity resolution order:
- explicit canonical resource id
- existing canonical provider link
- stable provider-scoped identity
- unambiguous normalized secondary identity
- unresolved provider reference
Secondary identity correlation must:
- normalize case, address form, and hostname form consistently
- reject zero-match and multi-match results
- record the rule and matching values
- avoid per-row network or database fetches
- preserve provider scope so bare vendor ids are never workspace-global
3. Operational lifecycle
The existing canonical alert lifecycle is the starting point. The completed contract must represent:
OperationalState
observing
open
acknowledged
suppressed
resolving
resolved
stale
unknown
These are operational states, not severity levels.
Every lifecycle record must have:
OperationalRecord
id
canonicalSpecId
subjectResourceId
state
severity
firstObservedAt
lastObservedAt
stateChangedAt
resolvedAt
acknowledgement
suppression
evidenceIds
causeKey
relatedResourceIds
impactSummary
recommendedNextStep
Required transition rules:
observing -> openonly through a canonical detector decision- acknowledgement never means resolution
- suppression never rewrites the underlying detector state
- an absent observation cannot resolve an item when collection is stale, partial, failed, or permission-denied
- resolution records the decisive recovery evidence
- recurrence after resolution either reopens the same stable cause or creates a linked recurrence according to the detector's canonical identity policy
- transitions are idempotent
- concurrent reads and writes are race-safe
- retention preserves a usable timeline after resolution
4. Notification linkage
Notification delivery is a consequence of lifecycle transitions, not a second source of operational truth.
Each queued delivery and audit record must carry:
NotificationLink
notificationId
operationalRecordId
transitionId
lifecycleState
causeKey
destinationId
deliveryState
attemptedAt
completedAt
Required behavior:
- retries preserve one stable event id
- grouped notifications preserve every included operational and transition id
- resolve notifications link to the resolving transition
- cancelled, failed, and dead-letter delivery remain inspectable
- navigation counts never derive from queue status
- disabling notification delivery does not disable detection or erase open operational records
- configuration surfaces clearly distinguish detection, in-product alerts, and external notification delivery
5. Protection posture
Protection posture is a typed, per-subject read model derived from recovery points and provider job evidence.
ProtectionPosture
subjectResourceId
state protected | attention | unprotected | unknown
lastAttemptAt
lastSuccessfulPointAt
lastVerifiedAt
freshness current | stale | unknown
verification verified | unverified | stale | unknown
coverage complete | partial | none | unknown
providerStates[]
repositoryResourceIds[]
evidenceIds[]
explanation
evaluatedAt
providerStates[] must preserve provider-specific uncertainty:
ProtectionProviderState
provider
source
scope
jobState
historyCompleteness
permissions
lastAttemptAt
lastSuccessAt
lastVerifiedAt
evidenceIds[]
Derivation rules:
protectedrequires a successful recovery point inside the configured freshness window and no known provider failure that invalidates the claimattentionmeans evidence exists but is stale, failing, unverified where verification is expected, or incomplete in a way the operator can addressunprotectedrequires sufficient evidence to affirm that no qualifying protection existsunknownis mandatory when identity, permission, history, or collection completeness cannot support a stronger claim- snapshots and backups preserve their canonical distinction
- snapshot presence alone never implies independent recovery
- provider job success without a subject-linked recovery point may be useful evidence, but it must not silently become subject-level protection
- the posture explanation names the strongest evidence and the limiting uncertainty in plain language
The existing recovery-point and protection-rollup models must be migrated or adapted into this contract without breaking supported clients.
6. Attention item
Patrol consumes a shared attention read model. It does not inspect arbitrary metadata to invent a queue.
AttentionItem
id
operationalRecordId
subjectResourceId
title
plainLanguageSummary
severity
state
firstObservedAt
lastObservedAt
evidenceFreshness
evidenceCompleteness
impact
protectionPosture
relatedResources[]
recommendedNextStep
availableActions[]
verificationState
Queue rules:
- default ordering is actionable severity, blast radius, protection concern, evidence freshness, then age
- stale or unknown evidence is visible but never presented as a confirmed failure
- duplicate symptoms sharing a stable cause may group under one item, while every member remains inspectable
- acknowledged and suppressed items are filterable and retain their true lifecycle state
- resolved items leave the active queue and remain in recent history
- the same item id deep-links from navigation, platform rows, resource detail, notifications, and Patrol
7. Availability facet
Availability is evidence about a resource, not a parallel inventory taxonomy.
Required ingest behavior:
- attach by explicit canonical resource link first
- otherwise attach by one unambiguous normalized address or hostname match
- retain a standalone
network-endpointonly for genuinely unowned targets - preserve ambiguous targets as unresolved rather than guessing
- emit a
checksrelationship and evidence envelope - use the unified-resource hot path with no per-row fetch
Required presentation:
- show a compact Availability facet on the owning platform resource row and resource detail
- show protocol identity, target, latest result, latency when relevant, freshness, and last observation
- route failures into the canonical lifecycle and attention model
- avoid duplicating the target on a generic Machines page once attached
This portion remains linked to the separately governed
availability-as-resource-facet candidate and must be claimed in sequence
before its runtime mutation begins.
8. Governed action and verification
Available actions are declared by resource capability and evaluated against policy, evidence, and current lifecycle state.
ActionOffer
actionId
targetResourceId
kind
mode plan | dry-run | execute
risk read-only | low | elevated
approval not-required | required | granted | denied
eligibility eligible | ineligible | unknown
reasons[]
evidenceIds[]
expectedPostcondition
verificationPolicy
ActionRun
runId
actionId
operationalRecordId
actor
requestedAt
approvedAt
startedAt
completedAt
executionState
result
auditRef
verificationState pending | verified | failed | inconclusive | timed-out
verificationEvidenceIds[]
Required behavior:
- planning and dry-run are preferred where the provider supports them
- elevated actions require explicit approval
- mutating actions fail closed on stale, incomplete, unknown, or ambiguous target evidence unless a narrower canonical policy explicitly allows them
- action parameters are typed and bounded
- idempotency keys prevent duplicate execution
- execution records actor, target, parameters, policy result, provider result, and timestamps
- verification uses fresh post-action evidence and a declared postcondition
- verification failure or timeout keeps the attention item open
- the first shipped mutating actions must be narrow, reversible where possible, and already supported by the canonical action framework
Runtime Ownership
The intended ownership boundaries are:
internal/unifiedresources: canonical identity and relationshipsinternal/alerts: detector decisions and operational lifecycleinternal/notifications: transition-linked delivery and auditinternal/recovery: recovery points, provider evidence, and protection posture derivationinternal/monitoring: observation collection and availability evidenceinternal/aior the existing Patrol application owner: attention projection, explanation, and supported action proposalsinternal/api: versioned transport and authorization onlyfrontend-modern: typed presentation and interaction only
If existing ownership differs, the implementation must update subsystem contracts before moving logic. It must not create circular dependencies or place canonical derivation in API handlers.
API Contract
Exact route names may follow existing conventions, but the completed API must provide these capabilities through versioned typed payloads.
Operational records
- list active records with filters and stable pagination
- get one record with transition timeline and evidence references
- acknowledge and unacknowledge
- suppress with bounded scope and expiry
- list recent resolved records
- expose one canonical active-count summary used by all badges
Evidence
- get bounded evidence detail authorized for the current operator
- show source, freshness, completeness, confidence, and reason
- redact sensitive provider detail by policy
- return typed unavailable responses when retained detail has expired
Protection
- get posture for one canonical resource
- batch posture for bounded resource tables
- list attention postures
- drill into contributing recovery points and provider evidence
- expose evaluation policy and timestamps
Patrol
- list active attention items
- get item detail
- list recent resolved history
- expose calm-day evaluation and coverage state
- deep-link to the owning resource and source platform
Actions
- list offers for an attention item or resource
- plan or dry-run
- request and record approval
- execute with idempotency
- retrieve execution and verification status
- cancel only where the provider contract supports safe cancellation
Compatibility rules
- existing supported alert and recovery routes remain compatible during migration
- compatibility adapters are read-side boundaries, not parallel truth
- new typed fields are additive before deprecated fields are removed
- deprecations include usage evidence, migration tests, and a governed removal decision
- table APIs support bounded batch reads; the frontend must not issue a request per row
Product Surface Contract
1. Navigation and dashboard
- one active attention count comes from the canonical read model
- label and tooltip distinguish attention items from notification delivery
- the count never remains non-zero beside an empty active queue
- the dashboard shows concise actionable summaries and links into Patrol or the affected resource
- monitor pages remain the normal authenticated launch path
2. Platform resource rows
- health, active attention, protection posture, and availability appear only when relevant to that resource
- compact indicators show state and freshness without widening tables into consoles
- row expansion or a drawer holds provider evidence and forensic detail
- every indicator has a clear operator action or is demoted from the default row
- canonical table alignment helpers remain the sole alignment owner
3. Resource detail
The default detail answers, in order:
- what resource is this
- what is its current state
- what needs attention
- what recently changed
- what it depends on and what depends on it
- what protection evidence exists
- what safe actions are available
Metrics and provider detail remain available without displacing those answers.
4. Patrol active queue
The initial view contains:
- a plain heading and accurate active count
- concise filters for open, acknowledged, suppressed, stale/unknown, and recent resolved
- an ordered list of attention items
- resource identity, impact, evidence state, protection context, and next step
- a clear calm-day state when the list is empty
It must not contain:
- a generic object browser
- vanity trust scores
- a large Assistant prompt before the queue
- stale findings presented as live
- read-only replicas of native vendor consoles
5. Attention detail
The detail or drawer contains:
- plain-language finding
- affected and related resources
- timeline from first observation through current state
- evidence source, age, completeness, confidence, and limiting caveats
- protection posture with provider drill-down
- recommended next step
- eligible actions, approval state, and risk
- execution and verification results
- links to the owning platform or native tool when Pulse cannot act
6. Assistant
Assistant receives selected typed context. It may:
- explain evidence
- summarize impact and relationships
- describe a proposed action or verification result
- help the operator navigate to supporting detail
Assistant may not:
- become the source of lifecycle or protection truth
- infer an action capability absent from the canonical resource
- bypass approval or evidence eligibility
- hide uncertainty
Security, Privacy, and Trust
- evidence access follows existing tenant, role, and sensitivity boundaries
- secret values and full credential-bearing payloads are never stored as evidence
- action authorization is checked at offer time and again at execution time
- approvals bind actor, exact target, action kind, parameters, and expiry
- evidence and action audit records use stable ids without leaking sensitive provider identifiers into public logs
- API errors do not reveal unavailable resources across authorization boundaries
- notification destination configuration remains protected
- Assistant receives policy-shaped summaries and references, not unrestricted raw provider payloads
Performance and Retention
- active-count and Patrol list reads must not scan unbounded history
- platform tables use bounded batch posture and availability data
- no per-row fetches
- lifecycle writes and notification enqueue remain idempotent under retry
- relationship and evidence indexes support subject, time, source, and active state reads
- retained evidence is sufficient to explain open records and recent resolution
- high-cardinality recovery points remain outside unified-resource inventory
- retention expiry preserves a typed summary and explicitly reports expired detail
- load proofs cover realistic resource, alert, recovery-point, and queue cardinalities
Telemetry and Product Health
The implementation must expose operational measurements for:
- observation-to-open latency
- open-to-notification-enqueue latency
- notification delivery success, retries, and dead letters
- stale or incomplete evidence counts by source
- unresolved or ambiguous identity correlations
- protection posture distribution and evaluation failures
- attention queue age and acknowledgement time
- action offer, approval, execution, and verification outcomes
- active-count/read-model mismatch detection
- calm-day evaluation age
Telemetry must not use high-cardinality raw resource ids as metric labels.
Migration and Rollout
Phase 0: contract and contradiction containment
- finalize typed contracts and subsystem ownership
- inventory every active-count and alert-enabled interpretation
- make detection, in-product visibility, and external delivery separate
- add read-model consistency assertions
- preserve supported payload compatibility
Phase 1: lifecycle and evidence foundation
- land evidence envelopes
- make lifecycle transition ids durable and idempotent
- link notification audit to transitions
- surface freshness, completeness, and source
- migrate loose trust-bearing metadata into typed fields
Phase 2: protection posture
- build canonical posture derivation over recovery points and provider job evidence
- add bounded subject and table APIs
- distinguish protected, attention, unprotected, and unknown
- prove PBS first without baking PBS semantics into the shared model
- add provider adapters only with explicit evidence-quality mappings
Phase 3: Patrol attention workbench
- project lifecycle records into typed attention items
- unify active counts and deep links
- deliver active, acknowledged, suppressed, stale/unknown, resolved, and calm states
- show evidence, impact, relationships, and protection in detail
- demote Assistant to selected context
Phase 4: availability attachment
- claim the
availability-as-resource-facetgoverned surface - implement explicit and unambiguous correlation
- attach the facet and relationship
- remove duplicate primary presentation for attached checks
- route failures through canonical lifecycle and Patrol
Phase 5: governed actions and verification
- select the smallest useful existing declared capabilities
- implement offer eligibility and evidence gating
- prove approval, idempotent execution, audit, and postcondition verification
- ship no action whose failed verification cannot be represented honestly
Phase 6: hardening and rollout completion
- compatibility and migration proofs
- concurrency, failure, retention, and load proofs
- accessibility and responsive proofs
- production-observed telemetry where required by governance
- documentation and upgrade notes
- remove superseded compatibility paths only after governed evidence
Each phase must leave one coherent runtime path. Feature flags may control exposure, but they must not create two writable sources of truth.
Required Proof Matrix
Unit and contract proof
- every lifecycle transition, including recurrence and stale collection
- evidence freshness, partial permission, provider failure, and expiry
- identity resolution, ambiguous rejection, and provider scoping
- protection posture truth table
- notification grouping, retry, resolve linkage, cancellation, and dead letter
- attention projection and stable ordering
- action eligibility, approval binding, idempotency, and verification
- JSON compatibility for supported clients
Integration proof
- observation opens one operational record and one attention item
- repeated observation does not duplicate either
- acknowledgement is reflected everywhere without resolving
- suppressed items leave the default queue but remain inspectable
- source outage produces stale or unknown, not false resolution
- recovery evidence changes protection posture and the related attention item
- notification audit links to the exact transition
- an action executes once and only closes after verified recovery evidence
- an attached availability check does not create a duplicate resource
- authorization and tenant boundaries hold across evidence and actions
Browser proof
Playwright inspection is mandatory for:
- monitor-first launch
- platform row with healthy, attention, stale, unknown, protected, unprotected, and unavailable-evidence states as applicable
- Patrol with active work
- Patrol calm day
- acknowledged and suppressed filters
- stale collection without false reassurance
- evidence and protection drill-down
- action approval, execution, verification success, and verification failure
- navigation count consistency
- keyboard navigation, focus restoration, screen-reader names, narrow viewport, and reduced-motion behavior
Proof must exercise expansions, drawers, empty states, and deepest supported detail. Source inspection and top-level screenshots are insufficient.
Failure proof
- collector disconnected
- provider partial permission
- provider rate limit or timeout
- notification destination failure and dead letter
- ambiguous identity
- missing recovery history
- stale successful backup
- action provider timeout
- successful provider response with failed postcondition
- restart during lifecycle transition or queued delivery
Performance proof
- bounded active-count latency
- bounded Patrol pagination
- bounded table batch enrichment
- no per-row network pattern
- stable memory and database behavior under retained evidence and recovery history
User-Lens Acceptance
Before each customer-facing phase is accepted:
- state the operator's job in their words
- exercise the live surface to deepest state
- count navigation depth and visible elements before the answer
- identify the action supported by each default-visible element
- replace or explain unfamiliar vocabulary
- compare the design with relevant GitHub issue evidence
- classify each element as keep, demote to expansion, or cut
- record whether the surface is
prototypeorproduct
No phase closes while its main surface remains prototype-grade.
Completion Criteria
This specification is fully implemented only when all of the following are true:
- every required canonical domain contract exists in code and subsystem governance
- alert, navigation, Patrol, resource, and notification views share one lifecycle truth
- stale, partial, denied, failed, ambiguous, and unknown evidence cannot masquerade as healthy or resolved
- protection posture is provider-aware, subject-linked, explainable, and available without per-row fetches
- Patrol is a functional attention queue with active-work and calm-day product states
- availability attaches to known resources and participates in the same evidence and lifecycle model
- at least one useful narrow mutating action completes the plan/approve/execute/verify loop through the canonical action framework
- every required proof in this document passes or is explicitly marked not applicable with a canonical justification
- live Playwright inspection passes the product quality and user-lens gates
- supported API and persisted-data migrations are proven
- subsystem contracts, status evidence, readiness assertions, release gates, upgrade documentation, and operator documentation reflect runtime truth
- no temporary compatibility path remains in the primary runtime
- all implementation slices are committed with scoped proof and no unrelated workspace changes
- the governing candidate lanes and coverage gaps are resolved or normalized to an explicit remaining decision owned outside this specification
Passing one phase does not satisfy the overall goal. The goal closes only after the full completion criteria pass.
Governance Mapping
Primary lane and subsystem ownership:
- L6: alerts, lifecycle, notifications, API coherence, Patrol projection
- L8: platform and resource-facing product surfaces
- L13: monitoring, recovery evidence, unified resources, availability
- L20: action capability, approval, audit, and verification
- L22: product trust, Assistant boundaries, policy-shaped context
- L23: proactive operations and Patrol workbench
Primary candidate:
protection-posture-attention-queue
Resolved sequential related candidate:
availability-as-resource-facet, closed byinternal/records/operational-trust-availability-resource-facet-2026-07-19.md
The primary candidate's subsystem mapping must include the canonical owners that this implementation touches, including alerts, notifications, storage and recovery, Patrol intelligence, monitoring, unified resources, API contracts, frontend primitives, action governance, and policy-shaped Assistant context.
Direction Captured
This specification records a release_gate:
Pulse v6 is not operationally trustworthy until every customer-facing active state, evidence explanation, protection assertion, notification consequence, and governed action resolves through one contradiction-free canonical lifecycle and evidence model.
It also records a readiness_assertion:
A resource may be called protected, healthy, resolved, or action-eligible only when current, sufficiently complete, subject-linked evidence supports that assertion; otherwise Pulse must display the limiting stale, partial, ambiguous, failed, or unknown state.