Expose host disk SMART health, counter, endurance, spare, and CRC-growth rules in alert settings while preserving existing defaults and supporting explicit disable values.
Change-source: pulse-maintainer
Record the unique PBS-to-Agent presentation boundary and the desktop and phone browser proof for Agent-backed host History.
Change-source: pulse-maintainer
Sort backup-server and datastore rows deterministically so provider refresh order cannot reset expanded detail state. Exclude optical devices from PVE, legacy agent, and canonical physical-disk projections.
Fixes#1785Fixes#1786
Contract-Neutral: Corrects refresh ordering and non-disk filtering without changing public APIs or subsystem contracts.
Change-source: pulse-maintainer
Record event-log history ownership, migration and tombstone semantics in the canonical alerts contract. Register the schema, migration and projection proofs so future runtime changes satisfy the explicit alerts verification policy.
Change-source: pulse-maintainer
The unified-incidents reconciler and system alerts dispatched without a
fired event, so the event log's lifecycle timeline — now the history
authority — began at their resolution. Both families record TypeFired
with a snapshot at activation, completing the log for every alert
family.
The retirement scoped by docs/ALERT_ENGINE_EVOLUTION.md: with an event
log enabled, GetAlertHistory and GetAlertHistorySince serve the
projection from the log's lifecycle snapshots; the JSON snapshot file
stops being written and is renamed to *.imported after a one-time
migration. The #1693 class — fire/resolve churn rewriting the JSON file
and deleting history — cannot recur against an append-only store.
Migration: on the first EnableEventLog with a JSON history file
present, every entry is written synchronously into the log as a
history_imported event (ImportEvents bypasses the droppable append
buffer — losing an entry to a full buffer would silently lose user
data), then the files retire to *.imported as backups. The file's
presence is the migration marker, so the import is idempotent and a
failed import leaves the JSON authoritative for the next attempt.
History retention (30 days) sits inside log retention (90 days), so no
entry can be pruned out from under the migration.
Clearing history keeps the log append-only: the clear is a
history_cleared tombstone and the projection ignores lifecycle events
that precede it. One deliberate difference from the JSON behavior:
still-active alerts reappear in history immediately after a clear —
they are current state, not cleared history.
The in-memory history manager remains as the fallback read model for
managers without an event log, and the parity suite now reads it
directly so it keeps characterizing the projection instead of comparing
the projection with itself.
The step before the event log can retire the JSON snapshot history file
(docs/ALERT_ENGINE_EVOLUTION.md): lifecycle events — fired, refired,
resolved, acknowledged, unacknowledged, escalated — now carry a full
alert snapshot in the log (new nullable snapshot column, upgraded in
place on pre-snapshot databases), and AlertHistoryFromEvents projects
the history list from those snapshots alone: one row per occurrence,
refires folded in by their restored start time, live active alerts
overlaid so open rows show current state.
The JSON-backed history manager stays authoritative and characterizes
the projection: history_projection_parity_test.go holds the two equal
across active, resolved, acknowledged, multi-resource, and
second-occurrence runs. Cutover follows once an import path carries
existing JSON history into the log.
Parity surfaced one real defect, fixed at the root: the JSON history
never propagated acknowledgement onto its rows, so the History view
showed an acknowledged alert as unacknowledged for its entire active
life. The live alert is authoritative; the row now follows it.
The lifecycle contract suite from docs/ALERT_ENGINE_EVOLUTION.md — the
audit's end-to-end assertions over externally observable alert behavior,
each one a recurring regression class from the issue record:
- A config save that changes nothing emits nothing (#1682: config saves
dispatched resolved notifications for still-active alerts), and one
that genuinely resolves an alert resolves it exactly once, with no
fire event and no residue in the reducer core.
- An acknowledged alert does not escalate, resumes escalating when
unacknowledged, and its recovery notification is suppressed.
- N grouped alerts render N on every delivery surface (#1683: built-in
service payloads silently dropped N−1 of N): grouped email HTML and
text, apprise, the generic webhook payload, a custom template, and
every built-in service template — discord, slack, telegram, teams,
teams-adaptive, pagerduty, pushover, gotify, ntfy, mattermost —
driven through the real grouped send path against a capture server.
- Cancelling an alert still inside the grouping window reports the
firing as never delivered, the answer the monitor uses to hold the
recovery notification (#1553: resolves sent for suppressed firings);
a delivered firing is never misreported as undelivered.
The engine contracts drive real metric evaluations through the public
read model (GetActiveAlerts / AcknowledgeAlert by exported ID) so they
gate the path users actually exercise.
Allow one retiring-agent health window after a trusted replacement install token is minted, while preserving clone-safe forks for ambiguous, conflicting, or longer-lived identities. Retire superseded token bindings and document the remaining migration gap for identities already forked.
Change-source: pulse-maintainer
Join Proxmox config paths with Linux semantics across native test platforms, add the regression proof, and update the canonical agent lifecycle contract for the new namespace-statfs collector.
Change-source: pulse-maintainer
Include the host-agent filesystem collection correction that landed after the initial rc.6 preparation, and keep the release metadata proof aligned with the new main head.
Change-source: pulse-maintainer
Pause the automatic mock metric ticker while the test asserts same-version snapshot identity, then restore it during cleanup. This prevents the full race suite from invalidating the fixture between the two reads it is comparing.
Change-source: pulse-maintainer
pct df takes the container config lock and costs over a second per
guest, and every per-container query shared one 10-second budget with
pct list, so hosts with more than a handful of running containers only
ever reported filesystem usage for the first few (signal: killed on
the rest).
Resolve per-mount usage with statfs through /proc/<pid>/root instead:
read the config-declared mounts from /etc/pve/lxc, resolve the init
pid via lxc-info, and drop config mounts whose device identity matches
their parent directory rather than inventing the parent's numbers.
pct df remains as a per-container fallback for least-privilege
installs that cannot traverse /proc/<pid>/root, each container now
carries its own deadline inside a bounded collection window, and
exhausting that window logs how many containers were skipped instead
of failing silently at debug level.
Refs #1477
Package the standalone PBS detail restoration, SMART CRC growth alerts, consolidated alert policy, bounded event queries, and Go 1.26.7 toolchain for the next prerelease.
Change-source: pulse-maintainer
Use the current supported Go 1.26 security patch across source, release, dev, and production container build surfaces. Keep the immutable official builder pin and governance proofs aligned with the exact toolchain.
Change-source: pulse-maintainer
Advance local, release, and container build surfaces together so reachable standard-library advisories cannot re-enter shipped binaries. Pin the official amd64 builder manifest and govern the updated floor with installability and dev-runtime proofs.
Change-source: pulse-maintainer
Phase 3 of docs/ALERT_ENGINE_EVOLUTION.md: the effective alert policy for
a resource — type default thresholds, the type's DisableAll switches,
custom rules, the per-resource override — is now answered by one ordered
fold (effectiveAlertPolicyNoLock in alert_policy.go), translated from the
persisted AlertConfig. The config keeps its shape; the engine stops
reading it piecemeal.
Before this, every check path read its own DisableAll* boolean and picked
its own override lookup, and the scattered reads drifted (#1738 was an
override lookup that existed on some paths and not others;
connection.go hand-rolled its own type-to-switch mapping). Now:
- resolveResourceThresholds and getGuestThresholds delegate to the fold.
- All 40+ DisableAll* reads across the check paths, the config-change
reconciliation, and the connection detector go through
alertPolicyTypeSwitchesNoLock — the single place those booleans are
read on behalf of evaluation.
Characterization first: alert_policy_test.go pins the fold against the
legacy resolution paths (per-type defaults, overrides, storage aliases,
guest custom rules by priority, every type's switch pair) before any
call site moved.
The reducer core became the authoritative transition state for every
per-observation family in the Phase 2 cutovers; the manager's tracking
maps (offlineConfirmations, offlineRecoveryConfirmations,
nodeOfflineCount, connectionDegradedCount, dockerOfflineCount,
dockerStateConfirm, pendingAlerts) had been reduced to write-only
mirrors. This deletes them, per the plan's retirement list
(docs/ALERT_ENGINE_EVOLUTION.md).
Hygiene the maps' cleanup loops used to provide moves into the core:
- reducer.PruneStalePending reaps pending runs whose resource stopped
being observed (Cleanup at 10 minutes, cleanupStaleMaps at the stale
threshold) — previously the pendingAlerts age sweep.
- Docker container cleanup drops core pending runs for containers no
longer in the seen set (reducer.PendingResourceIDs +
DropPendingForResource) — previously the dockerStateConfirm loops.
- HandleHostOnline / HandleDockerHostOnline apply a healthy
observation to the core so an in-flight offline confirmation run
ends — previously a map delete.
Two real gaps surfaced by the test conversion, fixed at the root:
- Config-change auto-resolution removed alerts without mirroring the
forget into the core, leaving the incident firing after a policy
disabled it.
- Intent pending state created by evaluateIntentNoLock carried no
ResourceID/ResourceType/TrackingKey, so per-resource clears
(guest suppression) could not match it.
Guest node-move migration no longer re-keys pending runs: a move
restarts an in-flight pending run (firing continuity still comes from
alert adoption). Deliberate simplification, noted in the helper.
Tests convert their map seeds and asserts to the core seams
(testCoreConfirmations / testCoreRecoveryCount / testCoreHasIncident /
testCoreIsPending, direct ApplyDiscrete seeding); assertions that only
tested the deleted maps' bookkeeping are removed.
Package the atomic API-token deletion fix, alert delivery evidence, reducer-backed lifecycle cutover, separated agent install tokens, and filesystem history feedback for the next release candidate.
Change-source: pulse-maintainer
Document reducer ownership for stateful alert transitions, timestamping, manual clears, acknowledgement retention, and refire history. Register the purpose-built stateful regression suite as canonical runtime and runtime-support proof.
Change-source: pulse-maintainer
Phase 2 continued. evaluateCanonicalStatefulAlert (health assessments,
posture thresholds, change thresholds — ZFS pool state, backup and
snapshot age, docker update-delay states) no longer reconstructs
previous state from the pending-since maps: the reducer core owns the
transition via the shared Match predicate, with the legacy maps kept as
read-only mirrors. The family keeps its occurrence-stamping semantics
(observation time or the caller's override such as a backup timestamp)
and its refire-without-new-history behavior, now driven by the core's
refire events.
Manual clears now record a resolved occurrence in the core (Forget
takes the clear time), matching the manager's recently-resolved
behavior, so a quick re-fire after a manual clear reactivates the same
occurrence instead of duplicating history — with the ack entering its
retention window. statefulPreviousState survives only until the last
mirror deletion.
Record the reducer as transition truth for canonical lifecycle and shared metric thresholds, including intent grace and activation-event ownership. Add a focused proof that metric intent remains pending until activation and emits one fired event.
Change-source: pulse-maintainer
Phase 2 continued. evaluateCanonicalMetricAlert — the main metric path
behind CheckUnifiedResource — no longer reconstructs previous state and
re-evaluates: the reducer core owns the transition, keyed by the
canonical metric spec ID. The spec's explicit critical threshold and
recovery latch map onto the rule (Critical pointer; CriticalDisabled
when the spec omits escalation at high triggers, where the legacy
derived cap would incorrectly escalate); the hysteresis-latched hold
keeps its no-refresh behavior; explicit metric intent policies gate in
the reducer with monotonic-tick grace, and the legacy delay applies
otherwise. New occurrences take start time and ack restoration from the
core and record fired events. Both metric wrappers now observe on the
manager clock seam (policyNow), keeping simulated-clock tests and
production behavior on one path.
The host-dedup tests read the core's pending incidents instead of the
retired pendingAlerts map.
Phase 2 continued. checkMetric (guest per-disk usage, the legacy metric
path) no longer runs its imperative pending/hysteresis logic: the
reducer core owns the transition, keyed by the canonical metric spec ID,
with the metric name kept for percentage classification. Explicit
metric intent policies gate activation in the reducer (replacing the
legacy time-threshold branch, with monotonic-tick grace), the
minimum-delta spam guard remains a manager-side pre-check, and
re-notify/cooldown/critical-escalation logic drives off the core's
severity. Existing alerts unknown to the core are adopted as firing.
With activation now explicit in both families, the event log's deferred
lifecycle events land: fired and refired are recorded at the reducer
core's activation events, so persisted restores can never be
misreported as firings — closing the last deferred piece of Phase 0.
Tests that backdated the legacy pendingAlerts map use the core's
ShiftPending seam instead. Full alerts tree and monitoring suites
green.
Phase 2 of docs/ALERT_ENGINE_EVOLUTION.md, first and largest cutover:
the deterministic reducer is now the AUTHORITATIVE transition state for
the entire match-spec family (connectivity, powered-state, discrete
state, provider incidents, health assessments, service gaps) and the
poll-driven recovery paths.
evaluateCanonicalLifecycleAlert no longer reconstructs previous state
from the confirmation-count maps and re-evaluates: it derives the
observation via the exported spec Match predicate, applies it to the
manager-owned reducer core (with the resolved intent context and the
monotonic intent tick), and translates the core's incidents and events
into the existing side effects — alert objects, dispatch, history,
recently-resolved, callbacks — synthesizing the evaluator-shaped result
its callers consume. The recovery paths (PBS/PMG/storage, node,
connection-degraded) feed healthy observations to the same core, whose
recovery gate replaces confirmOfflineRecovery. Guest powered-off clears
and suppressions clear the core run. User acks, unacks, and manual
clears mirror into the core; restore seeds it; an existing alert the
core does not know about is adopted as firing, exactly as the old
engine treated any active alert as previously firing.
Wall-clock immunity for intent grace is preserved deterministically:
the monotonic runtime tick is now an explicit reducer signal input
(with a validity flag — zero is a legitimate process-start reading), so
suspend and NTP jumps neither fire nor starve gated activations.
The legacy count maps are maintained as read-only mirrors of the core
during the transition and are no longer consulted by any engine logic;
tests that pre-seeded them now drive real observations, and tests that
pinned the reconstruct-from-maps workarounds are replaced by pins of
the core-owned behavior. lifecyclePreviousState and lifecycleFirstMatched
are deleted — the defect class they patched cannot exist when the run
state is owned rather than reconstructed.
All five parity harnesses, the shadow feed, and the full
characterization suite pass against the cutover engine.
Open the existing token reveal dialog after manual agent-token creation and provide a responsive Show token only action so Docker and Compose users can copy PULSE_TOKEN without extracting it from a command.
Also normalize two pre-existing frontend formatting drifts required by the full formatting gate.
Refs #1775
Contract-Neutral: Token-only reveal reuses the existing security token result and dialog without changing API, persistence, or install command contracts.
Change-source: pulse-maintainer
The last uncharacterized behavior: while Pulse has fresh evidence a
Proxmox backup caused an offline state, activation defers — bounded by
the max-deferral cap on total condition-active time — and after the
backup ends the grace extends to the backup's end plus the post-grace,
still capped. The discrete activation path is unified so the gate always
operates on a tracked pending incident, and the shadow feed now models
the deferral independently from the manager's decision instead of
echoing its hold as operator suppression.
Unit tests cover deferral, post-grace release, the never-ending-backup
cap, and post-grace capping; parity runs the real
LoadIntentPolicies/IntentBackup composition with simulated clocks.
Fourth consecutive parity slice with no manager defect. Every
discrete-family and metric-family behavior is now pinned.
Phase 1 capstone of docs/ALERT_ENGINE_EVOLUTION.md. The deterministic
reducer now runs continuously inside the live manager against the same
production observations: the canonical lifecycle path (connectivity,
powered-state, discrete-state kinds) via a deferred hook that replays
each evaluation — including the resolved intent context — through the
reducer; the poll-driven recovery paths (PBS/PMG/storage, node,
connection-degraded); and manual acknowledge/unacknowledge/clear. The
feed seeds from active canonical alerts at enable so restarts do not
read as mass divergence.
Every state disagreement is counted (Manager.ShadowDivergences) and
recorded in the alert event log as a shadow_divergence event with both
engines' states, rate-limited to one report per key per ten minutes.
After each divergence the reducer resyncs to the manager, so one
divergence yields one event — including divergences caused by manager
mutations the feed does not observe. Appends never block evaluation and
a disabled feed is a nil-check no-op.
This converts the parity harnesses' test-time guarantee into an
always-on invariant; the production divergence rate becomes the
go/no-go evidence for each Phase 2 family cutover. The full
activation/ack/recovery/re-fire cycle runs divergence-free in tests.
The input-layer direction stands; the transition-core and suppression
freezes do not (docs/ALERT_ENGINE_EVOLUTION.md). The stable-behaviors
list is a characterization inventory pinned by the reducer parity
harnesses, not a freeze — so a future session cannot mistake the frozen
scoping for current direction.
Document activation-only intent gating and register the reducer and manager-parity proofs required by canonical governance.
Change-source: pulse-maintainer
pulse-triage[bot]