Files
sencho/docs/features
Anso fcff8e9047 fix(monitor): collapse repeated host-metric alerts into per-window summary (F-11) (#1175)
* fix(monitor): collapse repeated host-metric alerts into per-window summary (F-11)

A host metric over threshold previously dispatched one notification every 5
minutes for the duration of the breach, producing 7+ identical messages
in 35 minutes and spamming Discord/Slack routes. Replace the hardcoded
5-minute cooldown for CPU/RAM/disk with a per-metric suppression window
(default 60 min, configurable via host_alert_suppression_mins). The first
breach fires immediately; subsequent cycles within the window are silently
counted; the next dispatch after the window elapses carries a summary
suffix listing how many cycles were suppressed and when the breach first
crossed threshold. Recovery clears the counter so re-breach fires fresh.

The pattern mirrors PolicyEnforcement.notifyTrivyMissingOnce: module-scope
Map, in-memory only, in-cycle dedup, with a test-reset helper. The
existing system_state row keeps post-restart re-fires bounded.

Janitor and per-stack alert rules are unchanged; they already have
adequate cadence and per-rule cooldown respectively.

* fix(ci): restore backend and frontend checks

* fix(e2e): remove create button timing race

* fix(e2e): harden create double-click test

* fix(monitor): clear persisted F-11 timestamp on recovery + clamp suppression window

Independent audit on the previous commit surfaced two issues.

1. clearHostMetricSuppression early-returned on missing in-memory state,
   leaving a stale system_state.last_host_*_alert_ts row alive after a
   process restart. Scenario: breach fires + persists timestamp, process
   restarts, metric recovers before another evaluate cycle re-seeds the
   in-memory Map, recovery cleanup early-returns. Next re-breach inside
   the original window hits the restart-survivability branch and is
   silently suppressed instead of firing fresh. Fix: read persisted
   state in clearHostMetricSuppression and reset to '0' independently
   of in-memory presence. The read-before-write also skips redundant
   writes when the row is already cleared.

2. host_alert_suppression_mins is validated by zod on the bulk PATCH
   path but the single-key POST /api/settings path accepts allowlisted
   keys without re-validation. A 999999999-minute value would silence
   host alerts for centuries. Add MAX_HOST_ALERT_SUPPRESSION_MIN = 1440
   mirroring the zod max, and clamp via Math.min in evaluateGlobalSettings.

Two new vitest cases (restart-then-recovery-then-rebreach; the 1440
clamp) confirmed failing before the fix, passing after. The existing
"metric drop" case updated to use a mock-backed persistence pattern
consistent with the new restart-scenario tests. 73/73 monitor-service
tests green; full backend suite 2507/2510 (same pre-existing Windows
EBUSY flake on filesystem-backup.test.ts as baseline).
2026-05-23 06:28:01 -04:00
..
2026-05-07 23:44:13 -04:00
2026-05-20 08:43:18 -04:00