Commit Graph

4197 Commits

Author SHA1 Message Date
rcourtman 6eee5a1641 chore(alerts): delete the legacy transition-state maps
The reducer core became the authoritative transition state for every
per-observation family in the Phase 2 cutovers; the manager's tracking
maps (offlineConfirmations, offlineRecoveryConfirmations,
nodeOfflineCount, connectionDegradedCount, dockerOfflineCount,
dockerStateConfirm, pendingAlerts) had been reduced to write-only
mirrors. This deletes them, per the plan's retirement list
(docs/ALERT_ENGINE_EVOLUTION.md).

Hygiene the maps' cleanup loops used to provide moves into the core:

- reducer.PruneStalePending reaps pending runs whose resource stopped
  being observed (Cleanup at 10 minutes, cleanupStaleMaps at the stale
  threshold) — previously the pendingAlerts age sweep.
- Docker container cleanup drops core pending runs for containers no
  longer in the seen set (reducer.PendingResourceIDs +
  DropPendingForResource) — previously the dockerStateConfirm loops.
- HandleHostOnline / HandleDockerHostOnline apply a healthy
  observation to the core so an in-flight offline confirmation run
  ends — previously a map delete.

Two real gaps surfaced by the test conversion, fixed at the root:

- Config-change auto-resolution removed alerts without mirroring the
  forget into the core, leaving the incident firing after a policy
  disabled it.
- Intent pending state created by evaluateIntentNoLock carried no
  ResourceID/ResourceType/TrackingKey, so per-resource clears
  (guest suppression) could not match it.

Guest node-move migration no longer re-keys pending runs: a move
restarts an in-flight pending run (firing continuity still comes from
alert adoption). Deliberate simplification, noted in the helper.

Tests convert their map seeds and asserts to the core seams
(testCoreConfirmations / testCoreRecoveryCount / testCoreHasIncident /
testCoreIsPending, direct ApplyDiscrete seeding); assertions that only
tested the deleted maps' bookkeeping are removed.
2026-08-27 01:00:12 +01:00
rcourtman 5e588c0be1 feat(alerts): cut the stateful family over to the reducer core
Phase 2 continued. evaluateCanonicalStatefulAlert (health assessments,
posture thresholds, change thresholds — ZFS pool state, backup and
snapshot age, docker update-delay states) no longer reconstructs
previous state from the pending-since maps: the reducer core owns the
transition via the shared Match predicate, with the legacy maps kept as
read-only mirrors. The family keeps its occurrence-stamping semantics
(observation time or the caller's override such as a backup timestamp)
and its refire-without-new-history behavior, now driven by the core's
refire events.

Manual clears now record a resolved occurrence in the core (Forget
takes the clear time), matching the manager's recently-resolved
behavior, so a quick re-fire after a manual clear reactivates the same
occurrence instead of duplicating history — with the ack entering its
retention window. statefulPreviousState survives only until the last
mirror deletion.
2026-08-27 00:31:27 +01:00
pulse-triage[bot] df709db7a5 docs(alerts): govern reducer runtime cutover
Record the reducer as transition truth for canonical lifecycle and shared metric thresholds, including intent grace and activation-event ownership. Add a focused proof that metric intent remains pending until activation and emits one fired event.

Change-source: pulse-maintainer
2026-08-27 00:29:46 +01:00
rcourtman 110cbb5773 feat(alerts): cut the canonical metric-threshold family over to the reducer core
Phase 2 continued. evaluateCanonicalMetricAlert — the main metric path
behind CheckUnifiedResource — no longer reconstructs previous state and
re-evaluates: the reducer core owns the transition, keyed by the
canonical metric spec ID. The spec's explicit critical threshold and
recovery latch map onto the rule (Critical pointer; CriticalDisabled
when the spec omits escalation at high triggers, where the legacy
derived cap would incorrectly escalate); the hysteresis-latched hold
keeps its no-refresh behavior; explicit metric intent policies gate in
the reducer with monotonic-tick grace, and the legacy delay applies
otherwise. New occurrences take start time and ack restoration from the
core and record fired events. Both metric wrappers now observe on the
manager clock seam (policyNow), keeping simulated-clock tests and
production behavior on one path.

The host-dedup tests read the core's pending incidents instead of the
retired pendingAlerts map.
2026-08-27 00:27:01 +01:00
rcourtman b1b35bf093 feat(alerts): cut the metric-threshold family over to the reducer core, emit lifecycle fired events
Phase 2 continued. checkMetric (guest per-disk usage, the legacy metric
path) no longer runs its imperative pending/hysteresis logic: the
reducer core owns the transition, keyed by the canonical metric spec ID,
with the metric name kept for percentage classification. Explicit
metric intent policies gate activation in the reducer (replacing the
legacy time-threshold branch, with monotonic-tick grace), the
minimum-delta spam guard remains a manager-side pre-check, and
re-notify/cooldown/critical-escalation logic drives off the core's
severity. Existing alerts unknown to the core are adopted as firing.

With activation now explicit in both families, the event log's deferred
lifecycle events land: fired and refired are recorded at the reducer
core's activation events, so persisted restores can never be
misreported as firings — closing the last deferred piece of Phase 0.

Tests that backdated the legacy pendingAlerts map use the core's
ShiftPending seam instead. Full alerts tree and monitoring suites
green.
2026-08-27 00:21:47 +01:00
rcourtman 6a04dcb84f feat(alerts): cut the canonical lifecycle family over to the reducer core
Phase 2 of docs/ALERT_ENGINE_EVOLUTION.md, first and largest cutover:
the deterministic reducer is now the AUTHORITATIVE transition state for
the entire match-spec family (connectivity, powered-state, discrete
state, provider incidents, health assessments, service gaps) and the
poll-driven recovery paths.

evaluateCanonicalLifecycleAlert no longer reconstructs previous state
from the confirmation-count maps and re-evaluates: it derives the
observation via the exported spec Match predicate, applies it to the
manager-owned reducer core (with the resolved intent context and the
monotonic intent tick), and translates the core's incidents and events
into the existing side effects — alert objects, dispatch, history,
recently-resolved, callbacks — synthesizing the evaluator-shaped result
its callers consume. The recovery paths (PBS/PMG/storage, node,
connection-degraded) feed healthy observations to the same core, whose
recovery gate replaces confirmOfflineRecovery. Guest powered-off clears
and suppressions clear the core run. User acks, unacks, and manual
clears mirror into the core; restore seeds it; an existing alert the
core does not know about is adopted as firing, exactly as the old
engine treated any active alert as previously firing.

Wall-clock immunity for intent grace is preserved deterministically:
the monotonic runtime tick is now an explicit reducer signal input
(with a validity flag — zero is a legitimate process-start reading), so
suspend and NTP jumps neither fire nor starve gated activations.

The legacy count maps are maintained as read-only mirrors of the core
during the transition and are no longer consulted by any engine logic;
tests that pre-seeded them now drive real observations, and tests that
pinned the reconstruct-from-maps workarounds are replaced by pins of
the core-owned behavior. lifecyclePreviousState and lifecycleFirstMatched
are deleted — the defect class they patched cannot exist when the run
state is owned rather than reconstructed.

All five parity harnesses, the shadow feed, and the full
characterization suite pass against the cutover engine.
2026-08-27 00:15:58 +01:00
rcourtman ee35d178b5 feat(alerts): backup-offline deferral in the reducer intent gate, completing Phase 1 characterization
The last uncharacterized behavior: while Pulse has fresh evidence a
Proxmox backup caused an offline state, activation defers — bounded by
the max-deferral cap on total condition-active time — and after the
backup ends the grace extends to the backup's end plus the post-grace,
still capped. The discrete activation path is unified so the gate always
operates on a tracked pending incident, and the shadow feed now models
the deferral independently from the manager's decision instead of
echoing its hold as operator suppression.

Unit tests cover deferral, post-grace release, the never-ending-backup
cap, and post-grace capping; parity runs the real
LoadIntentPolicies/IntentBackup composition with simulated clocks.
Fourth consecutive parity slice with no manager defect. Every
discrete-family and metric-family behavior is now pinned.
2026-08-26 23:54:12 +01:00
rcourtman bfc1a81867 feat(alerts): shadow-mode reducer feed with always-on divergence telemetry
Phase 1 capstone of docs/ALERT_ENGINE_EVOLUTION.md. The deterministic
reducer now runs continuously inside the live manager against the same
production observations: the canonical lifecycle path (connectivity,
powered-state, discrete-state kinds) via a deferred hook that replays
each evaluation — including the resolved intent context — through the
reducer; the poll-driven recovery paths (PBS/PMG/storage, node,
connection-degraded); and manual acknowledge/unacknowledge/clear. The
feed seeds from active canonical alerts at enable so restarts do not
read as mass divergence.

Every state disagreement is counted (Manager.ShadowDivergences) and
recorded in the alert event log as a shadow_divergence event with both
engines' states, rate-limited to one report per key per ten minutes.
After each divergence the reducer resyncs to the manager, so one
divergence yields one event — including divergences caused by manager
mutations the feed does not observe. Appends never block evaluation and
a disabled feed is a nil-check no-op.

This converts the parity harnesses' test-time guarantee into an
always-on invariant; the production divergence rate becomes the
go/no-go evidence for each Phase 2 family cutover. The full
activation/ack/recovery/re-fire cycle runs divergence-free in tests.
2026-08-26 23:49:12 +01:00
rcourtman 9ddfc69ec7 feat(alerts): intent gate in the reducer with parity harness
Phase 1 slice 6 of docs/ALERT_ENGINE_EVOLUTION.md.
DiscreteRule.Intent characterizes the manager's intent gate as composed
by evaluateCanonicalLifecycleAlert: operator suppression (expected
offline, muted, retired, active maintenance windows) and explicit grace
policies hold activation only — confirmations keep counting, the
incident stays pending, and release activates with the run's first
active observation as the start; an already-firing incident is never
suppressed. Grace accrues concurrently with operator suppression. The
manager measures grace on monotonic process runtime; the reducer uses
the signal clock, coincident under continuous operation.

reducer_parity_intent_test.go drives the real composition with the
manager's m.now and m.intentClock seams on the simulated clock, a
scenario-controlled operator resolver, and policies loaded through
LoadIntentPolicies. Third consecutive parity slice with no manager
defect found; the operator scenarios' StartTime parity exercises the
lifecycleFirstMatched preservation in composition. Deferred: the
backup-offline deferral sub-policy.
2026-08-26 23:29:40 +01:00
rcourtman f02474e838 feat(alerts): acknowledge lifecycle in the reducer with parity harness
Phase 1 slice 5 of docs/ALERT_ENGINE_EVOLUTION.md. State.Acknowledge /
Unacknowledge characterize the manager's ack semantics: an ack marks the
firing incident and a canonical record, survives per-tick rebuilds and
short resolve/re-fire cycles (restored on re-activation), is cleared by
unacknowledge, and expires after an hour of inactivity. The manager's
restore path has no age check — expiry comes from cleanup pruning the
inactive record — so the reducer enforces AckRetention deterministically
at restore time, on the signal clock. Restoration wires into both
families: checkMetric shares preserveAlertState.

reducer_parity_ack_test.go drives AcknowledgeAlert/UnacknowledgeAlert
and observations through both engines; the expiry scenario backdates the
manager's records and runs the real Cleanup pass. Second consecutive
parity slice with no manager defect found.
2026-08-26 23:24:09 +01:00
rcourtman 3d48649867 feat(alerts): re-fire start restoration in the reducer with parity harness
Phase 1 slice 4 of docs/ALERT_ENGINE_EVOLUTION.md. The reducer now keeps
a resolved-occurrence ledger: an activation inside RefireRetention (5
minutes, on the signal clock) consumes the record, restores the original
occurrence's StartedAt, and emits EventRefired — mirroring
consumeRecentlyResolvedForRefireWithPrimaryLock, where a re-fire within
the recently-resolved window reactivates the same occurrence without a
new history entry. Outside the window the re-fire is a fresh occurrence
with a fresh start.

reducer_parity_refire_test.go anchors the simulated epoch at wall time
and backdates recentlyResolved timestamps per step so the manager's
wall-clock retention check follows simulated time; StartTime is asserted
exactly on both engines. First parity slice with no manager defect
found. The wall-vs-evidence clock mix in the manager's retention check
is recorded as a deliberately deferred nuance.
2026-08-26 23:19:04 +01:00
rcourtman 07aa2116d1 feat(alerts): recovery-confirmation gate in the reducer with composed parity harness
Phase 1 slice 3 of docs/ALERT_ENGINE_EVOLUTION.md.
DiscreteRule.RecoveryConfirmations characterizes the poll-driven offline
composition: a firing incident resolves only after N consecutive
non-matching observations (default 3, storage 2), any matching
observation resets the run, pending still clears on a single
non-matching observation, and disable bypasses the gate — mirroring
clearResourceOfflineAlert + confirmOfflineRecoveryNoLock.

reducer_parity_recovery_test.go drives the manager through the exact
production composition (offline poll: reset recovery counter + evaluate
connectivity spec; healthy poll: clearResourceOfflineAlert) and diffs
the reducer after every step, including first-activation StartTime. The
harness caught the stale first-matched backdating fixed in the previous
commit.
2026-08-26 23:13:11 +01:00
rcourtman 44b85f754b fix(alerts): re-stamp lifecycle first-matched time when a new confirmation run starts
The slice-2 first-matched preservation kept an entry as long as one
existed, but several callers reset the confirmation-count maps directly
without the evaluator path — clearResourceOfflineAlert among them — so
a stale entry from a prior run backdated the next run's alert to the
previous run's first observation. Stamp the first-matched time whenever
the pre-evaluation count is zero, making stale entries harmless at
every reset site. Found by the recovery-gate parity harness.
2026-08-26 23:13:11 +01:00
pulse-triage[bot] de489abc65 Repair confirmation reducer governance proof
Record discrete confirmation and first-match timing ownership in the alerts contract, move the manager regression into the recognized incident proof, and restore status revision alignment with the stable source of truth.

Change-source: pulse-maintainer
2026-08-26 23:07:52 +01:00
rcourtman bb5fddf463 feat(alerts): confirmation-family reducer with discrete parity harness
Phase 1 slice 2 of docs/ALERT_ENGINE_EVOLUTION.md. ApplyDiscrete
characterizes the canonical lifecycle path's match-spec semantics for
connectivity / powered-state / discrete-state kinds: N consecutive
matching observations activate with StartTime at the first match, one
non-matching observation clears at this layer, severity follows the
spec while firing, and a disabled spec resolves. The incident sub-key
generalizes from metric name to state key.

reducer_parity_discrete_test.go diffs the reducer against
evaluateCanonicalLifecycleAlert after every step with fully simulated
time on both sides, including alert StartTime on first activations. The
harness again caught a real defect on first run — the confirmation
start-time understatement fixed in the previous commit. Recovery
confirmations and re-fire start restoration are documented as later
slices.
2026-08-26 23:04:35 +01:00
rcourtman 7c256cd04c fix(alerts): stamp confirmation-based lifecycle alerts at the first matched observation
The confirmation maps persist only counts, so lifecyclePreviousState
reconstructed pending runs dated at the current observation and a
confirmation-based alert (node/PBS/PMG offline, discrete states) stamped
StartTime at the final confirming poll — understating outage start by
the whole confirmation window. The manager had already fixed this class
once for unified incidents (unifiedIncidentFirstSeen) but not for the
generic canonical lifecycle path.

Preserve the first matched observation per tracking key
(lifecycleFirstMatched), consume it when reconstructing pending state,
clear it with the confirmation run, and clean leaked entries alongside
the other tracking maps. Found by the Phase 1 discrete-family parity
harness on its first run.
2026-08-26 23:04:35 +01:00
pulse-triage[bot] b1867a1de2 Repair metric reducer governance proof
Record canonical metric resolution and reducer parity ownership in the alerts contract, move the stale-resolution regression into the recognized shared proof, and restore deterministic status evidence ordering.

Change-source: pulse-maintainer
2026-08-26 23:00:46 +01:00
rcourtman 7c212b906e feat(alerts): deterministic metric-transition reducer with manager parity harness
Phase 1 of docs/ALERT_ENGINE_EVOLUTION.md. internal/alerts/reducer is a
pure transition core for the metric-threshold family — hysteresis
trigger/clear, sustained-for delay with dip reset, warning/critical
severity derivation with the percentage 99-cap — characterized from
Manager.checkMetric, with time entering only through the signal's
ObservedAt so every sequence is deterministic and replayable.

reducer_parity_test.go drives the live manager and the reducer through
identical observation sequences (simulated time via the established
pending-backdate trick) and fails on any divergence after every step,
with the manager as the reference. The harness proved itself on its
first run by catching the stale-resolve defect fixed in the previous
commit.

Registers coverage gap alert-lifecycle-contract-coverage. Remaining
slices: offline/confirmation families, ack lifecycle, intent
interaction, shadow-mode runtime feed.
2026-08-26 22:53:26 +01:00
rcourtman 20ea1ea4d3 fix(alerts): resolve canonically keyed metric alerts instead of leaving them stale
checkMetric stores canonical-identity alerts under the canonical state
key, but its hysteresis resolution removed by the legacy
<resourceID>-<metric> ID, which is never registered as an alias. The
removal silently no-oped: a resolved notification went out and a
recently-resolved entry was created while the alert stayed active and
re-resolved on every subsequent poll. Guest per-disk usage alerts were
the remaining production caller of this path — user-visible as the
stale-alert class (#1580).

Remove by the alert's actual storage key. Found by the Phase 1 reducer
parity harness on its first run.
2026-08-26 22:53:26 +01:00
pulse-triage[bot] d15181e960 Govern API token deletion durability
Record durable persistence as the token-revocation commit boundary across API, security, agent-lifecycle, and storage contracts. Route the exact-removal and rollback proofs through the canonical token lifecycle suite.

Change-source: pulse-maintainer
2026-08-26 22:11:28 +01:00
pulse-triage[bot] 9f971b4727 Keep API token deletion atomic
Roll back the in-memory token inventory and return an error when durable persistence fails. Cover exact deletion from a multi-token inventory and the persistence-failure rollback path.

Change-source: pulse-maintainer
2026-08-26 22:01:56 +01:00
pulse-triage[bot] 18bf3e1a8d Complete alert event log governance proof
Bind monitor-side event-log bootstrap to its canonical boundary and exercise the new event infrastructure through the registry-recognized alerts, API, monitoring, and agent-lifecycle proof files.

Change-source: pulse-maintainer
2026-08-26 21:32:10 +01:00
pulse-triage[bot] e88583866a Repair alert delivery governance proof
Bind the delivery diagnosis, active-card presentation, and additive event log to their canonical contracts and recognized API/frontend tests. Sync the shipped API reference, record current browser evidence, and restore sorted truthful control-plane status evidence.

Change-source: pulse-maintainer
2026-08-26 21:29:03 +01:00
rcourtman 9572a113ae feat(alerts): append-only alert event log with suppression reasons
Adds internal/alerts/eventlog: a SQLite-backed, additive event log that
records lifecycle transitions (resolved, acknowledged, unacknowledged,
escalated, flapping detected) and notification decisions (dispatched,
deferred by quiet hours, suppressed — with the mechanism that held
them). Appends never block alert evaluation: a full buffer drops the
event and counts the drop; a store that fails to open degrades to
recording nothing. 90-day retention, hourly prune.

The manager emits at the existing funnels only — dispatchAlert and the
safe-call resolve/ack/escalate seams — so no lifecycle behavior
changes. Lifecycle "fired" is deliberately not recorded yet: the
active-alert store funnel also runs on persisted restore, so firing
waits for the explicit activation seam in a later phase. The monitoring
bootstrap enables the log per manager; ephemeral managers and tests
record nothing unless they opt in.

GET /api/alerts/events (monitoring:read) reads the log with
alertIdentifier/type/since/limit filters, newest first.

Phase 0 of docs/ALERT_ENGINE_EVOLUTION.md (coverage gap
alert-engine-suppression-observability).
2026-08-26 21:15:37 +01:00
rcourtman 515aa319e8 feat(alerts): bulk delivery diagnosis for all active alerts
GET /api/alerts/delivery-diagnosis without alertIdentifier now returns
the diagnosis array for every active alert in one manager pass, so list
surfaces do not need a request per alert. Extracts the per-alert
diagnosis into a locked helper shared by both paths; single-alert
behavior is unchanged.

First slice of coverage gap alert-engine-suppression-observability
(docs/ALERT_ENGINE_EVOLUTION.md Phase 0).
2026-08-26 21:05:37 +01:00
pulse-triage[bot] 8fb7b37641 Stabilize release regression tests
Contract-Neutral: Freezes mock sampler cadence only inside the existing cache regression and updates release-preflight assertions to recognize current conditional runner routing and shard-degradation behavior; runtime and public contracts are unchanged.

Change-source: pulse-maintainer
2026-08-26 20:22:20 +01:00
pulse-triage[bot] a4a26fe842 Preserve availability result delivery order
Change-source: pulse-maintainer
2026-08-26 19:32:17 +01:00
rcourtman 4bca38395a Harden custom sensor test fixture permissions 2026-08-26 19:25:56 +01:00
rcourtman 9e1b8bfa1a Dedupe policy postures, AI-safe summaries, and self-alias ids in the stream
Third increment of the resource-payload-static-metadata gap, the
snapshot-side statics. Measurement first: the mock estate ships only 5
distinct non-default policy postures across 723 resources and 56
distinct templated AI-safe summary strings across the same rows
(~150KB of inline duplication per snapshot), and every canonical alias
list carries the row's own wire id verbatim.

Non-default policies and AI-safe summaries now dedupe through the same
content-addressed catalog contract as capabilities: state-level
policyCatalog and aiSafeSummaryCatalog hold the distinct blobs,
resources carry policyRef / aiSafeSummaryRef, and ingestion expands
refs back to the inline shape (default posture synthesis unchanged for
ref-less rows). The broadcast conversion returns one catalogs struct so
the next deduped field is a loop entry, and catalog changes ride the
same delta frame as the refs that need them. Alias lists stop shipping
the row's own wire id — every identity consumer unions resource.id
beside the alias spread — while the primary id stays so aliases remain
the complete live vocabulary a host is reachable by, per the
established slimming invariant.

Verified against a 1857-resource estate: zero inline policies or
summaries on the wire (743 refs resolving into 1KB + 7KB catalogs),
zero alias lists carrying the wire id, catalog-expanded postures and
summaries reaching the store byte-equivalently, and the estate
rendering clean on desktop and the 36-row phone window.

Remaining gap scope: canonicalIdentity alias/superseded history is
genuine per-resource identity data (~600KB per snapshot) with no safe
derivation shortcut, and the proxmox-facet/platformData mirror
duplication (~200KB) is deliberate canonical-merge fabric; both stay
recorded under resource-payload-static-metadata for a future owner.
2026-08-26 18:46:12 +01:00
rcourtman cba1e2abeb Reconcile delivery health and Proxmox Assistant controls 2026-08-26 18:33:41 +01:00
rcourtman 0fece386c4 Ship active alerts as per-client keyed deltas
Second increment of the resource-payload-static-metadata gap: the full
activeAlerts array (~37-75KB at mock-estate churn) re-shipped whenever
any single alert changed. Alerts now ride the generalized keyed delta
transport: the per-field keying and transition handling in the state
delta engine is a declarative keyedDeltaFields list shared with the
connected-infrastructure projection, and activeAlertsDelta carries
id-keyed merge patches, removed ids, and order. Measured on the mock
estate the per-tick alert payload drops to 184-1981 bytes.

One boundary held deliberately fixed: alert application timing does not
change. The client applies alert deltas to an isolated baseline and
feeds the exact same applyActiveAlerts path immediately on arrival,
input-active or not — alert lifecycle truth never waits for the
operator-input idle gate (alerts subsystem boundary, documented in
6ecdc81d1), and the pending-acknowledgment safeguards see byte-
equivalent records. Alerts handed to the store are cloned so it never
adopts baseline-owned objects. Deltas without a baseline are ignored
and heal on the next full payload; unkeyable entries fall back to
whole-field diffing.

Remaining gap scope unchanged: snapshot-side static metadata
(canonicalIdentity alias history ~609KB per snapshot, and the partially
regressed aiSafeSummary/policy omissions) stays open under
resource-payload-static-metadata.
2026-08-26 18:23:50 +01:00
rcourtman f3d3434652 Ship the infrastructure projection as per-client keyed deltas
First increment of the resource-payload-static-metadata gap: the
connected-infrastructure reporting projection was re-shipped whole
(~55-60KB at 50-node scale) on every realtime broadcast even though
measurement showed the only field that ever changes tick-to-tick is
lastSeen, and the client deep-reconciled all 69 rich items per tick —
the dominant reactive cost inside every websocket message after the
resource merge itself.

The projection now rides the same per-client keyed delta engine as
resources. buildClientStateSnapshot keys it by id beside the resource
map, and buildClientStateDelta emits connectedInfrastructureDelta with
id-keyed JSON merge patches, removed ids, and order through one shared
keyed-array builder. Measured on the mock estate the per-tick payload
drops from ~55-60KB to 0.7-4KB of timestamp patches. Entries that
cannot key by id fall the whole field back to plain whole-payload
diffing, and keyed/unkeyed baseline transitions re-ship the full array
once, so the keyed path can never corrupt the projection.

The client mirrors the resource baseline pattern: full payloads
establish an isolated raw baseline, deltas apply as merge patches
through the now-generic keyed apply helper, and the store sync hands
reconcile reference-stable objects for untouched items so a tick
reconciles only what changed instead of deep-walking the projection.
Deltas without a baseline are ignored and heal on the next full
payload; the operator-input deferral gate carries the pending changed
ids across a gesture and flushes them in the same coalesced batch.

The remaining gap scope (snapshot-side static metadata such as
canonicalIdentity alias history, ~609KB per snapshot) stays open under
resource-payload-static-metadata.
2026-08-26 18:09:34 +01:00
pulse-triage[bot] 8d573cd0ac Make formatting-only governance deterministic in CI
Change-source: pulse-maintainer
2026-08-26 16:03:27 +01:00
pulse-triage[bot] 75b6655d6e Restore Proxmox node network details
Refs #1716

Change-source: pulse-maintainer
2026-08-26 15:17:12 +01:00
rcourtman 74d0895e6e Fix mock storage history continuity
Contract-Neutral: Storage history behavior is documented in the staged API and storage contracts and covered by staged focused regressions; shared router references do not change agent-lifecycle, performance, or security contracts, and the proof allowlists do not yet name the focused test files.
2026-08-26 13:54:55 +01:00
rcourtman 5d46e7d0e7 Complete mock drawer history coverage
Contract-Neutral: restores existing metricsTarget identity and complete mock history fallback contracts without changing their public shape
2026-08-26 12:11:10 +01:00
pulse-triage[bot] 18b74c0adb Expose stable webhook message keys
Add a language-neutral condition key and explicit resource context to generic alert payloads so receivers can translate notifications without parsing English text.

Refs #1779

Change-source: pulse-maintainer
2026-08-26 12:04:43 +01:00
rcourtman 1943a1e97f Fix workload drawer history metrics targeting
Contract-Neutral: restores existing metricsTarget identity and mock history fallback contracts without changing their public shape
2026-08-26 11:17:41 +01:00
rcourtman dc1ddaa24e Fix stale backup alert attribution
Change-source: pulse-maintainer
2026-08-26 04:33:47 +01:00
rcourtman 8f7d5c2e98 Fix API-only Proxmox node history
Change-source: pulse-maintainer
2026-08-26 00:27:58 +01:00
rcourtman b34a3d8164 Fix stable demo runtime profile convergence 2026-08-26 00:20:50 +01:00
rcourtman e497ff987c Fix mock disk health presentation
Contract-Neutral: The storage wearout sentinel contract is unchanged; mock fixtures now encode unreported endurance as the existing canonical -1 value and carry canonical mock proof.
2026-08-25 21:34:41 +01:00
rcourtman e70fd875cc Stop stale container health alerts after exit
Contract-Neutral: The release-note line documents an alert evaluator fix and does not change release promotion or artifact dispatch.

Change-source: pulse-maintainer
2026-08-25 20:44:11 +01:00
rcourtman b8603b76b6 Preserve disabled backup collection scope
Contract-Neutral: The release-note line records a persistence bug fix; it does not change release promotion or artifact dispatch contracts.
2026-08-25 20:27:49 +01:00
rcourtman f675b5a4d8 Fix false self-container update reports 2026-08-25 18:37:57 +01:00
rcourtman f18f15bf7b Release seed-pinned metrics history backing arrays on trim
The retention and max-points trims re-sliced the series front
(metrics[i:]), which keeps the entire backing array reachable while
hiding the trimmed prefix from cap(). After the dense mock trends seed
(~2,800 points/series) aged out of its 6h window, every series held a
few hundred live points while pinning its seed-sized array - roughly
700MB across the demo estate's ~8,000 series. HeapSys drifted over the
droplet's memory.high, kernel reclaim throttling starved the accept
loop, and demo.pulserelay.pro served 502s (2026-08-25 outage; verified
live via pprof: appendMetric-attributed live heap grew +117MB in six
minutes after restart while goroutines stayed at 80).

Trims now compact in place so cap() reflects the true array size, and
releaseTrimmedCapacity copies the window into a right-sized array when
it occupies under a quarter of the backing array. A healthy sliding
window sits between 1x and 2x capacity and never pays the copy.

Regression tests pin both paths: a live append after an aged-out seed
backfill, and Cleanup on a mostly-expired series, including an aliasing
assertion that catches the cap()-blind front re-slice.
2026-08-25 16:55:20 +01:00
rcourtman 6e5571997e fix: show RAID controller member disks in Proxmox storage 2026-08-25 15:19:28 +01:00
rcourtman d5440ff43e Slim the resource stream and catch up realtime gaps incrementally
The 2026-08-25 audit follow-up profiled the remaining frontend costs on
the 50-node mock rig. Entering a tab more than one websocket tick behind
the shared cache fell off the incremental path onto a full-estate
deep-unwrap merge — a 2.1s unwrap dominated warm Alerts entry at 4x
throttle — and the client-facing resource stream re-shipped ~3KB of
mostly-static metadata per resource on every snapshot, REST recovery,
and reconnect (governed gap resource-payload-static-metadata).

The connection store now keeps a bounded per-tick changed-id history and
serves unions from it, so an instance that mounts or resumes a few
revisions behind catches up with a delta merge over the union instead of
remerging the estate. Uncovered gaps and full-snapshot commits still
fall back to keyed full reconciliation.

Broadcast payloads dedupe the estate's distinct capability blobs into a
content-addressed state-level capabilityCatalog referenced per resource
by capabilitiesRef; default-posture resources (internal sensitivity,
cloud-summary routing, no redactions) omit policy and aiSafeSummary; and
canonicalIdentity.aliases no longer duplicates supersededIds. Ingestion
reverses all three: refs expand through the catalog as per-row clones
(reconcile mutates adopted objects in place), missing policies
synthesize the default posture so a policy:null transition patch cannot
strand a stale governed policy, and identity alias resolution consults
supersededIds explicitly. Slimming edits only the per-broadcast copy,
never stored monitor state, and the delta engine's generic top-level
field diff carries catalog changes in the same frame as the first
resource referencing a new entry.

Rig A/B on the pinned 50-node mock (same VM as the morning baseline,
single runs, +-30% variance; RandomMetrics mutates nearly every row per
tick, so this estate is the worst case for the catch-up union): mobile
4x Alerts entry settles 3.9s -> 2.0s with long tasks 5.5s -> 3.3s,
idle-30s long tasks drop 12.6s -> 10.8s mobile and 3.4s -> 2.8s
desktop, remaining steps flat within variance. Sparse-delta estates
gain more, because their catch-up unions stay small.

/api/state on the pinned mock estate drops 4.75MB -> 4.09MB (-13.9%);
the capability catalog is 7 entries totalling 2.5KB where 946 resources
previously inlined the blobs, and 806 default-posture resources shed
policy and aiSafeSummary. Browser-verified against a live build of this
tree: byte-identical UI outcomes vs the unmodified baseline across
overview/storage/docker/alerts probes at both viewports, with docker
container lifecycle actions enabled through catalog-expanded
capabilities. pulse-mobile and pulse-enterprise do not read the slimmed
fields; the AI runtime consumes the internal model, which is unchanged.

Contracts for monitoring, api-contracts, unified-resources,
performance-and-scalability, alerts, agent-lifecycle, and
storage-recovery are amended with the slimmed wire shape, the ingestion
synthesis obligations, the catch-up history, and superseded-id
resolution.
2026-08-25 12:24:32 +01:00
rcourtman d3bfb3bb7b Tolerate heartbeat pings in websocket initial state test
TestWebSocketSendsInitialState asserted initialState arrives as the
second message, but the hub interleaves 30s heartbeat pings with the
connect sequence, so a slow state build on a loaded runner put a ping
there first and reddened main twice in three runs. Wait for each
expected type the way TestWebsocketPayloadContractShape already does.
The delivery and payload assertions are unchanged.
2026-08-25 07:19:24 +01:00
rcourtman 2ca84cddd7 Prevent connection alerts from bypassing offline policy 2026-08-24 22:45:59 +01:00