fix(monitor): collapse repeated host-metric alerts into per-window summary (F-11) (#1175)

* fix(monitor): collapse repeated host-metric alerts into per-window summary (F-11)

A host metric over threshold previously dispatched one notification every 5
minutes for the duration of the breach, producing 7+ identical messages
in 35 minutes and spamming Discord/Slack routes. Replace the hardcoded
5-minute cooldown for CPU/RAM/disk with a per-metric suppression window
(default 60 min, configurable via host_alert_suppression_mins). The first
breach fires immediately; subsequent cycles within the window are silently
counted; the next dispatch after the window elapses carries a summary
suffix listing how many cycles were suppressed and when the breach first
crossed threshold. Recovery clears the counter so re-breach fires fresh.

The pattern mirrors PolicyEnforcement.notifyTrivyMissingOnce: module-scope
Map, in-memory only, in-cycle dedup, with a test-reset helper. The
existing system_state row keeps post-restart re-fires bounded.

Janitor and per-stack alert rules are unchanged; they already have
adequate cadence and per-rule cooldown respectively.

* fix(ci): restore backend and frontend checks

* fix(e2e): remove create button timing race

* fix(e2e): harden create double-click test

* fix(monitor): clear persisted F-11 timestamp on recovery + clamp suppression window

Independent audit on the previous commit surfaced two issues.

1. clearHostMetricSuppression early-returned on missing in-memory state,
   leaving a stale system_state.last_host_*_alert_ts row alive after a
   process restart. Scenario: breach fires + persists timestamp, process
   restarts, metric recovers before another evaluate cycle re-seeds the
   in-memory Map, recovery cleanup early-returns. Next re-breach inside
   the original window hits the restart-survivability branch and is
   silently suppressed instead of firing fresh. Fix: read persisted
   state in clearHostMetricSuppression and reset to '0' independently
   of in-memory presence. The read-before-write also skips redundant
   writes when the row is already cleared.

2. host_alert_suppression_mins is validated by zod on the bulk PATCH
   path but the single-key POST /api/settings path accepts allowlisted
   keys without re-validation. A 999999999-minute value would silence
   host alerts for centuries. Add MAX_HOST_ALERT_SUPPRESSION_MIN = 1440
   mirroring the zod max, and clamp via Math.min in evaluateGlobalSettings.

Two new vitest cases (restart-then-recovery-then-rebreach; the 1440
clamp) confirmed failing before the fix, passing after. The existing
"metric drop" case updated to use a mock-backed persistence pattern
consistent with the new restart-scenario tests. 73/73 monitor-service
tests green; full backend suite 2507/2510 (same pre-existing Windows
EBUSY flake on filesystem-backup.test.ts as baseline).
This commit is contained in:
Anso
2026-05-23 06:28:01 -04:00
committed by GitHub
parent e46f6980f8
commit fcff8e9047
8 changed files with 560 additions and 30 deletions
+5 -1
View File
@@ -282,7 +282,11 @@ Real-time on the Docker event stream:
### Host thresholds
`warning`/`monitor_alert` for host CPU, RAM, and disk when the configured threshold is exceeded. 5-minute cooldown per signal. Example: `Host CPU utilization is critically high: 92% (Threshold: 90%)`. Evaluated on the 30-second monitor tick.
`warning`/`monitor_alert` for host CPU, RAM, and disk when the configured threshold is exceeded. Evaluated on the 30-second monitor tick. Example: `Host CPU utilization is critically high: 92% (Threshold: 90%)`.
Each metric (CPU, RAM, disk) carries its own suppression window. The first time a metric crosses its threshold, one notification fires immediately. While the metric stays over threshold within the window, additional cycles are silently counted but not dispatched. The next dispatch after the window elapses carries a summary suffix: `Suppressed 119 alerts in the last 60m; first over threshold at 14:23 UTC.` When the metric drops below threshold, the counter resets so the next breach fires fresh.
The suppression window defaults to 60 minutes and is configured per node in **Settings → System → Host thresholds → Alert suppression**.
### Docker janitor