Commit Graph

4249 Commits

Author SHA1 Message Date
pulse-triage[bot] 0c2d3f6d80 Keep agent install token issuance atomic
Restore the complete prior API-token inventory when an agent install credential cannot be persisted, so failed issuance neither admits an undisclosed secret nor evicts an older valid token. Prove the package and API failure boundaries and govern the shared lifecycle contract.

Change-source: pulse-maintainer
2026-08-28 07:33:22 +01:00
pulse-triage[bot] 73be3df3c5 Keep API token creation atomic
Restore the complete pre-creation token inventory when persistence fails so sorting cannot evict an older valid token or leave an unreturned credential active. Govern and test the durable creation boundary across API, security, agent lifecycle, and storage.

Change-source: pulse-maintainer
2026-08-28 07:08:08 +01:00
Pulse Test 674ca18968 fix(unifiedresources): merge PVE and agent rows for the same RAID array volume
The PVE disks/list poll and the agent SMART report both observe a RAID
controller's exported volume, but PVE surfaces its NAA identifier as a
bare-hex serial while smartctl reports the same value as a naa.-prefixed
WWN with no serial. The linked-disk join compared serial to serial and
WWN to WWN verbatim, so the volume rendered twice (/dev/sda and sda)
with independent metric histories.

Hardware identity comparison now normalizes reporter framing (naa., eui.,
wwn-, 0x prefixes, case) and folds serial and WWN together before
comparing. Values are never truncated: sibling volumes on one controller
share their leading WWN bytes, and a truncated udev ID_WWN must stay
unequal to a full identifier. Placeholder serials no longer count as an
identity match.

Refs #1720
2026-08-28 06:20:10 +01:00
pulse-triage[bot] f5f854afa2 Complete Unraid monitoring contract 2026-08-28 01:32:29 +01:00
pulse-triage[bot] cb27b8eb24 Fix Unraid empty slot health reporting 2026-08-28 01:23:12 +01:00
Pulse Test 5ee1cdd002 Make informational alerts a first-class severity 2026-08-28 00:45:20 +01:00
Pulse Test 413288d049 Keep offline mock hosts on the host alert lifecycle 2026-08-28 00:07:17 +01:00
Pulse Test 1a8d4aff8e Keep mock alert history behind the observation clock 2026-08-27 23:43:36 +01:00
Pulse Test ff6c6a7fe7 Make mock alert history lifecycle-consistent 2026-08-27 23:36:55 +01:00
pulse-triage[bot] c8781b993e Repair latest main test gates 2026-08-27 23:27:29 +01:00
Pulse Test e0a089baab Coalesce repeated alert delivery holds 2026-08-27 23:21:06 +01:00
pulse-triage[bot] b8d3a768bd Keep API token watcher mutations ordered
Serialize API-token disk reload snapshots with live token mutations so a stale watcher read cannot overwrite a newer rename or revoke. Add deterministic rename and revoke race coverage and record the governed concurrency boundary.

Change-source: pulse-maintainer
2026-08-27 22:44:46 +01:00
rcourtman 8ade613dfd Add complete mock alert timelines 2026-08-27 22:04:26 +01:00
pulse-triage[bot] 386041e46a Expose workload CPU window inheritance
Add the canonical workload fallback to rolling CPU policy controls and show each profile's effective parent duration. Refresh stale alert and Ceph guardrail expectations found during full-suite validation.

Change-source: pulse-maintainer
2026-08-27 22:03:23 +01:00
rcourtman 1878245cdb Make alert incident timelines complete 2026-08-27 21:42:25 +01:00
pulse-triage[bot] fec18f343f Preserve fresh rolling-window metric authority
Keep in-memory observations authoritative over duplicate persisted history, add direct provider and cache reset coverage, and register the rolling-window proof files with canonical governance.\n\nChange-source: pulse-maintainer
2026-08-27 21:22:12 +01:00
rcourtman 4b9abd4eca Add rolling-window metric alert evaluation 2026-08-27 21:11:44 +01:00
rcourtman a9076de15a fix(patrol): require Docker health recovery 2026-08-27 20:43:54 +01:00
rcourtman b0c1eca4cd Add predictive storage capacity alerts 2026-08-27 20:41:08 +01:00
pulse-triage[bot] f9340e3334 Complete Docker action verification proof
Change-source: pulse-maintainer
2026-08-27 20:19:17 +01:00
rcourtman 6163a2a564 feat(patrol): verify Docker actions independently 2026-08-27 20:14:38 +01:00
pulse-triage[bot] 7e1dbd91eb Bound alert escalation schedule delays
Change-source: pulse-maintainer
2026-08-27 20:12:36 +01:00
rcourtman e0baecc418 Make alert escalations destination-specific and repeatable 2026-08-27 19:51:12 +01:00
pulse-triage[bot] 5f6bfefa56 Make notification destination updates durable 2026-08-27 19:46:40 +01:00
rcourtman 1164d37302 feat(patrol): reconcile actionable findings on activation 2026-08-27 19:44:02 +01:00
rcourtman 8df9db7ab1 feat(alerts): add destination severity routing 2026-08-27 19:13:32 +01:00
pulse-triage[bot] e1d05e7b12 Make deploy enrollment credentials atomic 2026-08-27 18:49:41 +01:00
rcourtman 3d4c43162a feat(alerts): add recurring scoped maintenance 2026-08-27 18:20:03 +01:00
rcourtman ff507a6e44 feat(alerts): add per-alert snooze 2026-08-27 17:20:32 +01:00
pulse-triage[bot] 735acbb05a fix(dockeragent): bound image storage computation
Keep live image identity fresh without recomputing shared layer sizes on every report. Reuse the throttled storage snapshot and qualify the reported Synology inventory shape.

Change-source: pulse-maintainer
2026-08-27 16:33:14 +01:00
rcourtman f4e1e47e74 feat(alerts): add external dead-man monitoring 2026-08-27 16:27:46 +01:00
pulse-triage[bot] 30b4fbd95e test(notifications): isolate terminal recovery paths 2026-08-27 16:08:53 +01:00
pulse-triage[bot] 947a26b72f fix(alerts): stop mutating resolved snapshots
Contract-Neutral: resolved-alert alias lookup no longer mutates shared alert snapshots; canonical identity and public alert behavior are unchanged
2026-08-27 15:56:23 +01:00
rcourtman f6773f262a fix(alerts): make active restart state durable
Contract-Neutral: monitor construction only moves alerts-owned durable-store initialization before alert workers; monitoring and host-agent behavior are unchanged
2026-08-27 15:21:18 +01:00
rcourtman b6babd0b4b fix(alerts): make incident timelines lifecycle-driven
Contract-Neutral: monitor startup wiring changes only alert lifecycle projection behavior; agent-lifecycle deletion and reenrollment contracts are unchanged
2026-08-27 14:45:36 +01:00
pulse-triage[bot] 9622f89fe6 feat(alerts): configure SMART disk thresholds
Expose host disk SMART health, counter, endurance, spare, and CRC-growth rules in alert settings while preserving existing defaults and supporting explicit disable values.

Change-source: pulse-maintainer
2026-08-27 13:20:36 +01:00
rcourtman 73499d7161 feat(alerts): migrate persisted alert identities 2026-08-27 12:04:49 +01:00
pulse-triage[bot] 4f91232cc6 fix(proxmox): stabilize storage inventory refreshes
Sort backup-server and datastore rows deterministically so provider refresh order cannot reset expanded detail state. Exclude optical devices from PVE, legacy agent, and canonical physical-disk projections.

Fixes #1785
Fixes #1786

Contract-Neutral: Corrects refresh ordering and non-disk filtering without changing public APIs or subsystem contracts.
Change-source: pulse-maintainer
2026-08-27 11:05:14 +01:00
rcourtman 96add5d57a fix(alerts): harden event history recovery 2026-08-27 10:05:37 +01:00
rcourtman d6ff4151d1 test(alerts): widen history projection parity to every rendered field 2026-08-27 09:34:27 +01:00
rcourtman fbbdb929dc feat(alerts): record fired events for reconciler and system alerts
The unified-incidents reconciler and system alerts dispatched without a
fired event, so the event log's lifecycle timeline — now the history
authority — began at their resolution. Both families record TypeFired
with a snapshot at activation, completing the log for every alert
family.
2026-08-27 09:23:22 +01:00
rcourtman 2dc2b72759 feat(alerts): make the event log the alert history authority
The retirement scoped by docs/ALERT_ENGINE_EVOLUTION.md: with an event
log enabled, GetAlertHistory and GetAlertHistorySince serve the
projection from the log's lifecycle snapshots; the JSON snapshot file
stops being written and is renamed to *.imported after a one-time
migration. The #1693 class — fire/resolve churn rewriting the JSON file
and deleting history — cannot recur against an append-only store.

Migration: on the first EnableEventLog with a JSON history file
present, every entry is written synchronously into the log as a
history_imported event (ImportEvents bypasses the droppable append
buffer — losing an entry to a full buffer would silently lose user
data), then the files retire to *.imported as backups. The file's
presence is the migration marker, so the import is idempotent and a
failed import leaves the JSON authoritative for the next attempt.
History retention (30 days) sits inside log retention (90 days), so no
entry can be pruned out from under the migration.

Clearing history keeps the log append-only: the clear is a
history_cleared tombstone and the projection ignores lifecycle events
that precede it. One deliberate difference from the JSON behavior:
still-active alerts reappear in history immediately after a clear —
they are current state, not cleared history.

The in-memory history manager remains as the fallback read model for
managers without an event log, and the parity suite now reads it
directly so it keeps characterizing the projection instead of comparing
the projection with itself.
2026-08-27 09:22:05 +01:00
rcourtman 871f2348bd feat(alerts): project alert history from the event log, in parity
The step before the event log can retire the JSON snapshot history file
(docs/ALERT_ENGINE_EVOLUTION.md): lifecycle events — fired, refired,
resolved, acknowledged, unacknowledged, escalated — now carry a full
alert snapshot in the log (new nullable snapshot column, upgraded in
place on pre-snapshot databases), and AlertHistoryFromEvents projects
the history list from those snapshots alone: one row per occurrence,
refires folded in by their restored start time, live active alerts
overlaid so open rows show current state.

The JSON-backed history manager stays authoritative and characterizes
the projection: history_projection_parity_test.go holds the two equal
across active, resolved, acknowledged, multi-resource, and
second-occurrence runs. Cutover follows once an import path carries
existing JSON history into the log.

Parity surfaced one real defect, fixed at the root: the JSON history
never propagated acknowledgement onto its rows, so the History view
showed an acknowledged alert as unacknowledged for its entire active
life. The live alert is authoritative; the row now follows it.
2026-08-27 09:15:38 +01:00
rcourtman 3049da030e test(alerts): pin the lifecycle contracts as a release gate
The lifecycle contract suite from docs/ALERT_ENGINE_EVOLUTION.md — the
audit's end-to-end assertions over externally observable alert behavior,
each one a recurring regression class from the issue record:

- A config save that changes nothing emits nothing (#1682: config saves
  dispatched resolved notifications for still-active alerts), and one
  that genuinely resolves an alert resolves it exactly once, with no
  fire event and no residue in the reducer core.
- An acknowledged alert does not escalate, resumes escalating when
  unacknowledged, and its recovery notification is suppressed.
- N grouped alerts render N on every delivery surface (#1683: built-in
  service payloads silently dropped N−1 of N): grouped email HTML and
  text, apprise, the generic webhook payload, a custom template, and
  every built-in service template — discord, slack, telegram, teams,
  teams-adaptive, pagerduty, pushover, gotify, ntfy, mattermost —
  driven through the real grouped send path against a capture server.
- Cancelling an alert still inside the grouping window reports the
  firing as never delivered, the answer the monitor uses to hold the
  recovery notification (#1553: resolves sent for suppressed firings);
  a delivered firing is never misreported as undelivered.

The engine contracts drive real metric evaluations through the public
read model (GetActiveAlerts / AcknowledgeAlert by exported ID) so they
gate the path users actually exercise.
2026-08-27 09:09:56 +01:00
pulse-triage[bot] 0fd9171ff7 Prevent host identity forks during install handoff
Allow one retiring-agent health window after a trusted replacement install token is minted, while preserving clone-safe forks for ambiguous, conflicting, or longer-lived identities. Retire superseded token bindings and document the remaining migration gap for identities already forked.

Change-source: pulse-maintainer
2026-08-27 08:47:42 +01:00
pulse-triage[bot] 8fde82b8a2 Keep LXC filesystem paths native-proof
Join Proxmox config paths with Linux semantics across native test platforms, add the regression proof, and update the canonical agent lifecycle contract for the new namespace-statfs collector.

Change-source: pulse-maintainer
2026-08-27 06:51:36 +01:00
pulse-triage[bot] 7348c17905 Stabilize mock snapshot memoization proof
Pause the automatic mock metric ticker while the test asserts same-version snapshot identity, then restore it during cleanup. This prevents the full race suite from invalidating the fixture between the two reads it is comparing.

Change-source: pulse-maintainer
2026-08-27 06:44:56 +01:00
rcourtman 8318c74687 fix(hostagent): stop pct df from starving LXC filesystem collection
pct df takes the container config lock and costs over a second per
guest, and every per-container query shared one 10-second budget with
pct list, so hosts with more than a handful of running containers only
ever reported filesystem usage for the first few (signal: killed on
the rest).

Resolve per-mount usage with statfs through /proc/<pid>/root instead:
read the config-declared mounts from /etc/pve/lxc, resolve the init
pid via lxc-info, and drop config mounts whose device identity matches
their parent directory rather than inventing the parent's numbers.
pct df remains as a per-container fallback for least-privilege
installs that cannot traverse /proc/<pid>/root, each container now
carries its own deadline inside a bounded collection window, and
exhausting that window logs how many containers were skipped instead
of failing silently at debug level.

Refs #1477
2026-08-27 06:22:10 +01:00
pulse-triage[bot] 1dc5ee4942 docs(alerts): govern SMART counter growth
Change-source: pulse-maintainer
2026-08-27 03:30:44 +01:00
pulse-triage[bot] 02cfd4f7b3 feat(alerts): detect SMART CRC counter growth
Change-source: pulse-maintainer
2026-08-27 03:26:21 +01:00