Phase 3 of docs/ALERT_ENGINE_EVOLUTION.md: the effective alert policy for
a resource — type default thresholds, the type's DisableAll switches,
custom rules, the per-resource override — is now answered by one ordered
fold (effectiveAlertPolicyNoLock in alert_policy.go), translated from the
persisted AlertConfig. The config keeps its shape; the engine stops
reading it piecemeal.
Before this, every check path read its own DisableAll* boolean and picked
its own override lookup, and the scattered reads drifted (#1738 was an
override lookup that existed on some paths and not others;
connection.go hand-rolled its own type-to-switch mapping). Now:
- resolveResourceThresholds and getGuestThresholds delegate to the fold.
- All 40+ DisableAll* reads across the check paths, the config-change
reconciliation, and the connection detector go through
alertPolicyTypeSwitchesNoLock — the single place those booleans are
read on behalf of evaluation.
Characterization first: alert_policy_test.go pins the fold against the
legacy resolution paths (per-type defaults, overrides, storage aliases,
guest custom rules by priority, every type's switch pair) before any
call site moved.
The reducer core became the authoritative transition state for every
per-observation family in the Phase 2 cutovers; the manager's tracking
maps (offlineConfirmations, offlineRecoveryConfirmations,
nodeOfflineCount, connectionDegradedCount, dockerOfflineCount,
dockerStateConfirm, pendingAlerts) had been reduced to write-only
mirrors. This deletes them, per the plan's retirement list
(docs/ALERT_ENGINE_EVOLUTION.md).
Hygiene the maps' cleanup loops used to provide moves into the core:
- reducer.PruneStalePending reaps pending runs whose resource stopped
being observed (Cleanup at 10 minutes, cleanupStaleMaps at the stale
threshold) — previously the pendingAlerts age sweep.
- Docker container cleanup drops core pending runs for containers no
longer in the seen set (reducer.PendingResourceIDs +
DropPendingForResource) — previously the dockerStateConfirm loops.
- HandleHostOnline / HandleDockerHostOnline apply a healthy
observation to the core so an in-flight offline confirmation run
ends — previously a map delete.
Two real gaps surfaced by the test conversion, fixed at the root:
- Config-change auto-resolution removed alerts without mirroring the
forget into the core, leaving the incident firing after a policy
disabled it.
- Intent pending state created by evaluateIntentNoLock carried no
ResourceID/ResourceType/TrackingKey, so per-resource clears
(guest suppression) could not match it.
Guest node-move migration no longer re-keys pending runs: a move
restarts an in-flight pending run (firing continuity still comes from
alert adoption). Deliberate simplification, noted in the helper.
Tests convert their map seeds and asserts to the core seams
(testCoreConfirmations / testCoreRecoveryCount / testCoreHasIncident /
testCoreIsPending, direct ApplyDiscrete seeding); assertions that only
tested the deleted maps' bookkeeping are removed.
Phase 2 of docs/ALERT_ENGINE_EVOLUTION.md, first and largest cutover:
the deterministic reducer is now the AUTHORITATIVE transition state for
the entire match-spec family (connectivity, powered-state, discrete
state, provider incidents, health assessments, service gaps) and the
poll-driven recovery paths.
evaluateCanonicalLifecycleAlert no longer reconstructs previous state
from the confirmation-count maps and re-evaluates: it derives the
observation via the exported spec Match predicate, applies it to the
manager-owned reducer core (with the resolved intent context and the
monotonic intent tick), and translates the core's incidents and events
into the existing side effects — alert objects, dispatch, history,
recently-resolved, callbacks — synthesizing the evaluator-shaped result
its callers consume. The recovery paths (PBS/PMG/storage, node,
connection-degraded) feed healthy observations to the same core, whose
recovery gate replaces confirmOfflineRecovery. Guest powered-off clears
and suppressions clear the core run. User acks, unacks, and manual
clears mirror into the core; restore seeds it; an existing alert the
core does not know about is adopted as firing, exactly as the old
engine treated any active alert as previously firing.
Wall-clock immunity for intent grace is preserved deterministically:
the monotonic runtime tick is now an explicit reducer signal input
(with a validity flag — zero is a legitimate process-start reading), so
suspend and NTP jumps neither fire nor starve gated activations.
The legacy count maps are maintained as read-only mirrors of the core
during the transition and are no longer consulted by any engine logic;
tests that pre-seeded them now drive real observations, and tests that
pinned the reconstruct-from-maps workarounds are replaced by pins of
the core-owned behavior. lifecyclePreviousState and lifecycleFirstMatched
are deleted — the defect class they patched cannot exist when the run
state is owned rather than reconstructed.
All five parity harnesses, the shadow feed, and the full
characterization suite pass against the cutover engine.
Phase 1 capstone of docs/ALERT_ENGINE_EVOLUTION.md. The deterministic
reducer now runs continuously inside the live manager against the same
production observations: the canonical lifecycle path (connectivity,
powered-state, discrete-state kinds) via a deferred hook that replays
each evaluation — including the resolved intent context — through the
reducer; the poll-driven recovery paths (PBS/PMG/storage, node,
connection-degraded); and manual acknowledge/unacknowledge/clear. The
feed seeds from active canonical alerts at enable so restarts do not
read as mass divergence.
Every state disagreement is counted (Manager.ShadowDivergences) and
recorded in the alert event log as a shadow_divergence event with both
engines' states, rate-limited to one report per key per ten minutes.
After each divergence the reducer resyncs to the manager, so one
divergence yields one event — including divergences caused by manager
mutations the feed does not observe. Appends never block evaluation and
a disabled feed is a nil-check no-op.
This converts the parity harnesses' test-time guarantee into an
always-on invariant; the production divergence rate becomes the
go/no-go evidence for each Phase 2 family cutover. The full
activation/ack/recovery/re-fire cycle runs divergence-free in tests.
Move Proxmox node metric, temperature, offline lifecycle, host-agent deduplication, and node display-name cache support into internal/alerts/node.go.
Keep the Manager API unchanged while recording the node checker owner in the alerts subsystem contract and adding a focused display-name cache key characterization.
Proof: go test ./internal/alerts/...
Pulse Test