Projection replay walked the entire alert event log on every boot, twice
(monitor start and the resource-store boundary), synchronously on the
serving path, and every replayed event queued a full incident-store JSON
rewrite. A 64k-event log made startup exceed the dev supervisor's
health-kill window, so the backend restarted forever and never served.
The event log now owns per-consumer projection watermarks in
alert_store_meta, WalkOldest takes an AfterID cursor, and the single
catch-up pass runs in the background at the canonical resource-store
boundary with periodic watermark checkpoints. The watermark only
advances when both the incident store and the canonical change recorder
are attached, so partial-surface passes repair without marking events
applied, and lowering the watermark forces a full repair replay.
Incident-store saves now coalesce: a burst of mutations queues one
whole-store serialization instead of one per event, live paths included.
Registers the alert-lifecycle-replay-startup-scalability coverage gap
and amends the alerts and monitoring subsystem contracts accordingly.
The public demo runs PULSE_MOCK_UPDATE_INTERVAL=60s, which stretches the
10-cohort supplemental rotation to 600s while the unified registry still
marks TrueNAS/VMware/availability sources stale after 120s. Every row those
sources own spent ~80% of each rotation flagged as a degraded stale source,
so the TrueNAS Storage tab read Healthy 0 / Attention 21 (and the vSphere
page all-Attention) despite healthy fixture data.
Rebase provider-backed fixtures every tick once the rotation would outlive
the 120s freshness budget (at slow ticks that is cheaper per wall-clock
second than the default 2s config's once-per-20s cadence), and derive mock
supplemental stale thresholds from the actual refresh cadence the same way
PVE/PBS/PMG thresholds already derive from their polling intervals. Split
the supplemental uptime step so per-tick refreshes do not advance PBS
uptime by the rotation-scaled step.
Removing a Docker host blocks its reports, but the report path only ever
consulted the in-memory block and offered no way for a legitimate
re-enroll to clear it: reinstalling the agent with a newly generated
token kept being rejected until a server restart happened to wipe the
in-memory map, which read as the Docker tab staying missing until the
Pulse server was rebooted. The restart also silently lifted deliberate
blocks, because the persisted entry was never consulted on report.
Docker reports now follow the host-agent rule from #1581: the block
check consults both the in-memory map and the persisted store, and a
token minted after the removal is explicit re-enroll intent that clears
the block from both. A still-running old agent presenting its
pre-removal token stays blocked, including across restarts.
Refs #1728
Wire outbound alert callbacks before a monitor is published so immediately reconnecting agents cannot create active warnings in the constructor-to-Start gap. Add a pre-Start custom-sensor regression and bind the ordering in monitoring and agent lifecycle contracts.
Change-source: pulse-maintainer
Name every outbound alert-quality field in both operator-facing privacy documents, lock the current schema value in verification, and align the alert-manager adoption test with the telemetry contract.
Change-source: pulse-maintainer
Use unique provider-observed node interface addresses to restore automatic
host-agent linking when standalone sites share a short hostname but use
distinct FQDN endpoints. Keep reused addresses ambiguous and fail closed.
Refs #1753
Contract-Neutral: narrows internal agent-to-node identity reconciliation without changing API or extension shapes
Serialize monitoring-owned host, Docker, and Kubernetes token cleanup with API-token persistence. Restore the complete live inventory when a reduced inventory cannot commit so restart never silently reverses an apparent revocation, and govern the boundary with forced-failure proof.
Change-source: pulse-maintainer
Keep in-memory observations authoritative over duplicate persisted history, add direct provider and cache reset coverage, and register the rolling-window proof files with canonical governance.\n\nChange-source: pulse-maintainer
Contract-Neutral: monitor construction only moves alerts-owned durable-store initialization before alert workers; monitoring and host-agent behavior are unchanged
Sort backup-server and datastore rows deterministically so provider refresh order cannot reset expanded detail state. Exclude optical devices from PVE, legacy agent, and canonical physical-disk projections.
Fixes#1785Fixes#1786
Contract-Neutral: Corrects refresh ordering and non-disk filtering without changing public APIs or subsystem contracts.
Change-source: pulse-maintainer
Allow one retiring-agent health window after a trusted replacement install token is minted, while preserving clone-safe forks for ambiguous, conflicting, or longer-lived identities. Retire superseded token bindings and document the remaining migration gap for identities already forked.
Change-source: pulse-maintainer
Phase 1 capstone of docs/ALERT_ENGINE_EVOLUTION.md. The deterministic
reducer now runs continuously inside the live manager against the same
production observations: the canonical lifecycle path (connectivity,
powered-state, discrete-state kinds) via a deferred hook that replays
each evaluation — including the resolved intent context — through the
reducer; the poll-driven recovery paths (PBS/PMG/storage, node,
connection-degraded); and manual acknowledge/unacknowledge/clear. The
feed seeds from active canonical alerts at enable so restarts do not
read as mass divergence.
Every state disagreement is counted (Manager.ShadowDivergences) and
recorded in the alert event log as a shadow_divergence event with both
engines' states, rate-limited to one report per key per ten minutes.
After each divergence the reducer resyncs to the manager, so one
divergence yields one event — including divergences caused by manager
mutations the feed does not observe. Appends never block evaluation and
a disabled feed is a nil-check no-op.
This converts the parity harnesses' test-time guarantee into an
always-on invariant; the production divergence rate becomes the
go/no-go evidence for each Phase 2 family cutover. The full
activation/ack/recovery/re-fire cycle runs divergence-free in tests.
Bind monitor-side event-log bootstrap to its canonical boundary and exercise the new event infrastructure through the registry-recognized alerts, API, monitoring, and agent-lifecycle proof files.
Change-source: pulse-maintainer
Adds internal/alerts/eventlog: a SQLite-backed, additive event log that
records lifecycle transitions (resolved, acknowledged, unacknowledged,
escalated, flapping detected) and notification decisions (dispatched,
deferred by quiet hours, suppressed — with the mechanism that held
them). Appends never block alert evaluation: a full buffer drops the
event and counts the drop; a store that fails to open degrades to
recording nothing. 90-day retention, hourly prune.
The manager emits at the existing funnels only — dispatchAlert and the
safe-call resolve/ack/escalate seams — so no lifecycle behavior
changes. Lifecycle "fired" is deliberately not recorded yet: the
active-alert store funnel also runs on persisted restore, so firing
waits for the explicit activation seam in a later phase. The monitoring
bootstrap enables the log per manager; ephemeral managers and tests
record nothing unless they opt in.
GET /api/alerts/events (monitoring:read) reads the log with
alertIdentifier/type/since/limit filters, newest first.
Phase 0 of docs/ALERT_ENGINE_EVOLUTION.md (coverage gap
alert-engine-suppression-observability).
Contract-Neutral: Freezes mock sampler cadence only inside the existing cache regression and updates release-preflight assertions to recognize current conditional runner routing and shard-degradation behavior; runtime and public contracts are unchanged.
Change-source: pulse-maintainer
Third increment of the resource-payload-static-metadata gap, the
snapshot-side statics. Measurement first: the mock estate ships only 5
distinct non-default policy postures across 723 resources and 56
distinct templated AI-safe summary strings across the same rows
(~150KB of inline duplication per snapshot), and every canonical alias
list carries the row's own wire id verbatim.
Non-default policies and AI-safe summaries now dedupe through the same
content-addressed catalog contract as capabilities: state-level
policyCatalog and aiSafeSummaryCatalog hold the distinct blobs,
resources carry policyRef / aiSafeSummaryRef, and ingestion expands
refs back to the inline shape (default posture synthesis unchanged for
ref-less rows). The broadcast conversion returns one catalogs struct so
the next deduped field is a loop entry, and catalog changes ride the
same delta frame as the refs that need them. Alias lists stop shipping
the row's own wire id — every identity consumer unions resource.id
beside the alias spread — while the primary id stays so aliases remain
the complete live vocabulary a host is reachable by, per the
established slimming invariant.
Verified against a 1857-resource estate: zero inline policies or
summaries on the wire (743 refs resolving into 1KB + 7KB catalogs),
zero alias lists carrying the wire id, catalog-expanded postures and
summaries reaching the store byte-equivalently, and the estate
rendering clean on desktop and the 36-row phone window.
Remaining gap scope: canonicalIdentity alias/superseded history is
genuine per-resource identity data (~600KB per snapshot) with no safe
derivation shortcut, and the proxmox-facet/platformData mirror
duplication (~200KB) is deliberate canonical-merge fabric; both stay
recorded under resource-payload-static-metadata for a future owner.
rcourtman