Three rules chose a node's single CPU temperature. Node history took
CPUMax first, agent history took cpu_package first, and the node drawer
recomputed the maximum of package, CPUMax and every core in the
browser. On an agent-fed node CPUMax is built from cores alone and can
sit below the package, and elsewhere a core can run above the package,
so the drawer's CPU temperature headline could disagree with the line
drawn beneath it and with the History tab. Node alerts and the CPU
low/record tracking used yet another package-first rule.
models.PrimaryCPUTemperatureCelsius is now the one rule: the hottest
usable package, CPU-maximum or core reading, with a sensor-map form for
agents. Node and agent history, the router's live fallbacks, the
unified resource temperature, node temperature alerts and the CPU
low/record tracking all call it. The drawer reads the resource value
through a new Node.cpuTemperature field instead of recomputing it, and
getCpuTemperature is removed.
Mock mode wrote a second package reading under a cpu.package key into
every agent host, including node-linked ones, from a series unrelated
to the node's static reading. A node and its linked agent now carry one
reading, sampled from the agent's series under the agent's real keys,
and both histories seed from that series, so the current reading
continues the line.
Adapt remaining PR2599/2586 behavior to current source: evidence-bound ownership, report-ordered link outcomes, immediate configuration revocation, direct/inherited override precedence and canonical/shadow pending/intent reset. Close node-side alerts as transfer to a real per-metric owner rather than observed recovery. Preserve volatile metric status, child identity and observation-gap interruption.
Actual entry-point controls cover overlap, unknown values, saved direct filesystem disable, missing thresholds and real monitor lifecycle/History transfers. Preserve per-type default normalization: this is not a new Off setting. PR2567 is already merged and held SMART collection/native work is not revived.
Source-Provenance: additive adaptation of PR2599 at 765ac23046 over bd3432ab22
Change-source: pulse-maintainer
The broader alert race suite identified that usage-only dispatch treated the separately evaluated node temperature as missing. Declare that separate owner, keep its existing continuous healthy and missing-read semantics, and pin the real CheckNode path.
Contract-Neutral: Restore the existing typed node temperature timing and ownership boundary; the private dispatch scope and contract clarification introduce no public API, canonical schema or intended policy delta.
Change-source: pulse-maintainer
Drop pending activation and reset recovery timing across retained guest readings, rejected unified values and unverified rolling windows. Keep firing alerts and independent current metrics untouched, and clear interrupted intent checkpoints so a restart cannot accrue unknown time.
Change-source: pulse-maintainer
A review of the previous commit found three ways the node and its
linked agent could still disagree about who alerts.
Memory ownership required a known reading on every report. When an
agent's memory reading went missing while its alert was open, the agent
kept that alert (a missing reading is not a recovery) but stopped
owning memory, so the node raised a second alert. It resolved again as
soon as the reading returned, with history and notifications for each
flip. Ownership now follows the agent's configured thresholds, not one
report's data.
With several agents linked to one node, only the lowest agent ID's
coverage counted. If that agent had CPU off and another linked agent
evaluated CPU, the node alerted on CPU as well. A metric is now owned
when any linked agent evaluates it.
Disk ownership needed any evaluated filesystem, but a linked node's
disk metric is the agent's summary filesystem (root when reported, see
models.SummaryDisk). With root disabled by a per-disk override and
/data still alerting, the node gave up its root disk alert and a full
root filesystem raised nothing. The agent now owns the node's disk
metric only while it evaluates that same filesystem.
CheckHost registered a host agent's node link before its disabled
returns. With agent alerts switched off, or a host override disabling
the agent, the node still released its CPU, memory and disk alerts to an
agent that evaluated nothing, so nothing alerted on that machine's
usage. The same happened per metric: an agent CPU threshold set to off
with the node CPU threshold on left the machine with no CPU alert.
The link now records which usage metrics the agent actually evaluated
on its last report: a live CPU threshold, a live memory threshold with
a known reading, and at least one filesystem with a live disk
threshold. CheckHost registers it after evaluation and removes it on
the disabled returns. CheckNode releases only the metrics the agent
owns and evaluates the rest itself, so deduplication can only remove a
duplicate, never the last alert for a machine.
The link also carries the agent's display name, which the handover
notification work builds on.
A review of the node temperature fix found three more ways node
alerts went wrong.
Node deduplication matched a host agent's hostname against node names
across every PVE instance. With two instances that each have a node
called pve and an agent on only one of them, both nodes released their
CPU, memory and disk alerts, so the second machine went unmonitored. An
agent reporting an FQDN never matched its node, so that machine alerted
twice, and an operator unlinking an agent did not hand the alerts back.
Dedup now keys on the agent's LinkedNodeID, which the monitor already
sets automatically or from an operator link. On pulse-prod the delly2
agent is linked to homelab-delly2, the node alert ID it should dedup.
Saving the alert config resolved node temperature alerts that were
still over their trigger. Reevaluation only treated an alert as a node
alert when its Instance equalled its node name, but current node alerts
keep the PVE instance name there, so they fell through to guest
thresholds, which have no temperature threshold. Metadata resourceType
"node" now decides, and the Instance heuristic only covers legacy alerts.
A missing temperature reading no longer resolves an open alert, but a
pending run still spanned the gap: one hot sample after ten minutes
without readings fired at once, and a recovery run could complete the
same way. The reducer gains InterruptMetricRun, which drops a pending
run or restarts a recovery run without resolving anything, and the node
temperature check calls it when the reading is missing.
Installing a Pulse agent on a Proxmox node silently ended CPU
temperature alerting for it. CheckNode skipped every node metric,
temperature included, once a host agent with the node's hostname
registered, but CheckHost evaluates only CPU, memory and disk usage and
SMART disk temperatures, never CPU temperature. On pulse-prod the last
node temperature alert fired on 5 October; delly2 has since read 90°C
and minipc 78-87°C with no alert.
Temperature now stays with the node whether or not an agent is
registered. The node poll already merges the agent's sensors into
node.Temperature, so this gives one temperature alert per machine.
The dedup skip path also never cleared node alerts it stopped
evaluating, so a node alert open when the agent registered stayed
frozen until the agent went offline. pulse-prod shows this today: the
delly2 node memory alert, open since 29 September, beside the agent's
own memory alert. CheckNode now releases node CPU, memory and disk
alerts through the disabled-threshold path, which drops the pending run
and resolves the open alert, leaving the agent as the only owner.
A missing CPU reading also fed 0°C into the evaluator, which could
resolve an open temperature alert without any evidence that the node had
cooled. With a live trigger the check now skips the evaluation instead,
while a disabled threshold still clears.
Phase 3 of docs/ALERT_ENGINE_EVOLUTION.md: the effective alert policy for
a resource — type default thresholds, the type's DisableAll switches,
custom rules, the per-resource override — is now answered by one ordered
fold (effectiveAlertPolicyNoLock in alert_policy.go), translated from the
persisted AlertConfig. The config keeps its shape; the engine stops
reading it piecemeal.
Before this, every check path read its own DisableAll* boolean and picked
its own override lookup, and the scattered reads drifted (#1738 was an
override lookup that existed on some paths and not others;
connection.go hand-rolled its own type-to-switch mapping). Now:
- resolveResourceThresholds and getGuestThresholds delegate to the fold.
- All 40+ DisableAll* reads across the check paths, the config-change
reconciliation, and the connection detector go through
alertPolicyTypeSwitchesNoLock — the single place those booleans are
read on behalf of evaluation.
Characterization first: alert_policy_test.go pins the fold against the
legacy resolution paths (per-type defaults, overrides, storage aliases,
guest custom rules by priority, every type's switch pair) before any
call site moved.
The reducer core became the authoritative transition state for every
per-observation family in the Phase 2 cutovers; the manager's tracking
maps (offlineConfirmations, offlineRecoveryConfirmations,
nodeOfflineCount, connectionDegradedCount, dockerOfflineCount,
dockerStateConfirm, pendingAlerts) had been reduced to write-only
mirrors. This deletes them, per the plan's retirement list
(docs/ALERT_ENGINE_EVOLUTION.md).
Hygiene the maps' cleanup loops used to provide moves into the core:
- reducer.PruneStalePending reaps pending runs whose resource stopped
being observed (Cleanup at 10 minutes, cleanupStaleMaps at the stale
threshold) — previously the pendingAlerts age sweep.
- Docker container cleanup drops core pending runs for containers no
longer in the seen set (reducer.PendingResourceIDs +
DropPendingForResource) — previously the dockerStateConfirm loops.
- HandleHostOnline / HandleDockerHostOnline apply a healthy
observation to the core so an in-flight offline confirmation run
ends — previously a map delete.
Two real gaps surfaced by the test conversion, fixed at the root:
- Config-change auto-resolution removed alerts without mirroring the
forget into the core, leaving the incident firing after a policy
disabled it.
- Intent pending state created by evaluateIntentNoLock carried no
ResourceID/ResourceType/TrackingKey, so per-resource clears
(guest suppression) could not match it.
Guest node-move migration no longer re-keys pending runs: a move
restarts an in-flight pending run (firing continuity still comes from
alert adoption). Deliberate simplification, noted in the helper.
Tests convert their map seeds and asserts to the core seams
(testCoreConfirmations / testCoreRecoveryCount / testCoreHasIncident /
testCoreIsPending, direct ApplyDiscrete seeding); assertions that only
tested the deleted maps' bookkeeping are removed.
Phase 2 of docs/ALERT_ENGINE_EVOLUTION.md, first and largest cutover:
the deterministic reducer is now the AUTHORITATIVE transition state for
the entire match-spec family (connectivity, powered-state, discrete
state, provider incidents, health assessments, service gaps) and the
poll-driven recovery paths.
evaluateCanonicalLifecycleAlert no longer reconstructs previous state
from the confirmation-count maps and re-evaluates: it derives the
observation via the exported spec Match predicate, applies it to the
manager-owned reducer core (with the resolved intent context and the
monotonic intent tick), and translates the core's incidents and events
into the existing side effects — alert objects, dispatch, history,
recently-resolved, callbacks — synthesizing the evaluator-shaped result
its callers consume. The recovery paths (PBS/PMG/storage, node,
connection-degraded) feed healthy observations to the same core, whose
recovery gate replaces confirmOfflineRecovery. Guest powered-off clears
and suppressions clear the core run. User acks, unacks, and manual
clears mirror into the core; restore seeds it; an existing alert the
core does not know about is adopted as firing, exactly as the old
engine treated any active alert as previously firing.
Wall-clock immunity for intent grace is preserved deterministically:
the monotonic runtime tick is now an explicit reducer signal input
(with a validity flag — zero is a legitimate process-start reading), so
suspend and NTP jumps neither fire nor starve gated activations.
The legacy count maps are maintained as read-only mirrors of the core
during the transition and are no longer consulted by any engine logic;
tests that pre-seeded them now drive real observations, and tests that
pinned the reconstruct-from-maps workarounds are replaced by pins of
the core-owned behavior. lifecyclePreviousState and lifecycleFirstMatched
are deleted — the defect class they patched cannot exist when the run
state is owned rather than reconstructed.
All five parity harnesses, the shadow feed, and the full
characterization suite pass against the cutover engine.
Phase 1 capstone of docs/ALERT_ENGINE_EVOLUTION.md. The deterministic
reducer now runs continuously inside the live manager against the same
production observations: the canonical lifecycle path (connectivity,
powered-state, discrete-state kinds) via a deferred hook that replays
each evaluation — including the resolved intent context — through the
reducer; the poll-driven recovery paths (PBS/PMG/storage, node,
connection-degraded); and manual acknowledge/unacknowledge/clear. The
feed seeds from active canonical alerts at enable so restarts do not
read as mass divergence.
Every state disagreement is counted (Manager.ShadowDivergences) and
recorded in the alert event log as a shadow_divergence event with both
engines' states, rate-limited to one report per key per ten minutes.
After each divergence the reducer resyncs to the manager, so one
divergence yields one event — including divergences caused by manager
mutations the feed does not observe. Appends never block evaluation and
a disabled feed is a nil-check no-op.
This converts the parity harnesses' test-time guarantee into an
always-on invariant; the production divergence rate becomes the
go/no-go evidence for each Phase 2 family cutover. The full
activation/ack/recovery/re-fire cycle runs divergence-free in tests.
Move Proxmox node metric, temperature, offline lifecycle, host-agent deduplication, and node display-name cache support into internal/alerts/node.go.
Keep the Manager API unchanged while recording the node checker owner in the alerts subsystem contract and adding a focused display-name cache key characterization.
Proof: go test ./internal/alerts/...
rcourtman