mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-11 03:06:54 +00:00
fix(monitor): collapse repeated host-metric alerts into per-window summary (F-11) (#1175)
* fix(monitor): collapse repeated host-metric alerts into per-window summary (F-11) A host metric over threshold previously dispatched one notification every 5 minutes for the duration of the breach, producing 7+ identical messages in 35 minutes and spamming Discord/Slack routes. Replace the hardcoded 5-minute cooldown for CPU/RAM/disk with a per-metric suppression window (default 60 min, configurable via host_alert_suppression_mins). The first breach fires immediately; subsequent cycles within the window are silently counted; the next dispatch after the window elapses carries a summary suffix listing how many cycles were suppressed and when the breach first crossed threshold. Recovery clears the counter so re-breach fires fresh. The pattern mirrors PolicyEnforcement.notifyTrivyMissingOnce: module-scope Map, in-memory only, in-cycle dedup, with a test-reset helper. The existing system_state row keeps post-restart re-fires bounded. Janitor and per-stack alert rules are unchanged; they already have adequate cadence and per-rule cooldown respectively. * fix(ci): restore backend and frontend checks * fix(e2e): remove create button timing race * fix(e2e): harden create double-click test * fix(monitor): clear persisted F-11 timestamp on recovery + clamp suppression window Independent audit on the previous commit surfaced two issues. 1. clearHostMetricSuppression early-returned on missing in-memory state, leaving a stale system_state.last_host_*_alert_ts row alive after a process restart. Scenario: breach fires + persists timestamp, process restarts, metric recovers before another evaluate cycle re-seeds the in-memory Map, recovery cleanup early-returns. Next re-breach inside the original window hits the restart-survivability branch and is silently suppressed instead of firing fresh. Fix: read persisted state in clearHostMetricSuppression and reset to '0' independently of in-memory presence. The read-before-write also skips redundant writes when the row is already cleared. 2. host_alert_suppression_mins is validated by zod on the bulk PATCH path but the single-key POST /api/settings path accepts allowlisted keys without re-validation. A 999999999-minute value would silence host alerts for centuries. Add MAX_HOST_ALERT_SUPPRESSION_MIN = 1440 mirroring the zod max, and clamp via Math.min in evaluateGlobalSettings. Two new vitest cases (restart-then-recovery-then-rebreach; the 1440 clamp) confirmed failing before the fix, passing after. The existing "metric drop" case updated to use a mock-backed persistence pattern consistent with the new restart-scenario tests. 73/73 monitor-service tests green; full backend suite 2507/2510 (same pre-existing Windows EBUSY flake on filesystem-backup.test.ts as baseline).
This commit is contained in:
Anso
@@ -58,6 +58,31 @@ const HOST_ALERT_KEYS = {
|
||||
janitor: 'last_janitor_alert_timestamp',
|
||||
} as const;
|
||||
|
||||
type HostMetricKey = 'cpu' | 'ram' | 'disk';
|
||||
|
||||
interface HostAlertSuppressionState {
|
||||
firstFiredAt: number;
|
||||
lastFiredAt: number;
|
||||
suppressedCount: number;
|
||||
}
|
||||
|
||||
// F-11 dedup for host-metric notifications (CPU, RAM, disk). The previous
|
||||
// 5-minute cooldown produced 7+ identical messages over a 35-minute window
|
||||
// when a host stayed over threshold, flooding Discord/Slack routes. State is
|
||||
// per-metric and module-scope so a singleton-replacement (only happens in
|
||||
// tests) does not lose tracking. Cleared on process restart; the persisted
|
||||
// system_state row keeps post-restart re-fires bounded.
|
||||
const hostAlertSuppressionState = new Map<HostMetricKey, HostAlertSuppressionState>();
|
||||
const DEFAULT_HOST_ALERT_SUPPRESSION_MIN = 60;
|
||||
// Mirror the zod schema upper bound in `routes/settings.ts` so the single-key
|
||||
// POST path (which lacks zod re-validation) cannot push the consumer past
|
||||
// the validated range.
|
||||
const MAX_HOST_ALERT_SUPPRESSION_MIN = 1440;
|
||||
|
||||
export function _resetHostAlertSuppressionStateForTests(): void {
|
||||
hostAlertSuppressionState.clear();
|
||||
}
|
||||
|
||||
const STATS_TIMEOUT_MS = 10_000;
|
||||
const FLOAT_EQ_EPSILON = 0.01;
|
||||
|
||||
@@ -226,7 +251,17 @@ export class MonitorService {
|
||||
}
|
||||
|
||||
private async evaluateGlobalSettings(settings: Record<string, string>) {
|
||||
const HOST_ALERT_COOLDOWN_MS = 5 * 60 * 1000; // 5 minutes between repeat alerts
|
||||
// F-11 suppression window. Default 60 min; configurable via global
|
||||
// settings. NaN / non-positive values fall back to the default so a
|
||||
// misconfigured row never disables the dedup entirely. Cap at 24h
|
||||
// to defend against the single-key POST /api/settings path (which
|
||||
// stores allowlisted keys without zod re-validation): an accidental
|
||||
// 999999999-minute value must not silence host alerts for centuries.
|
||||
const suppressionMinRaw = parseInt(settings['host_alert_suppression_mins'] || '', 10);
|
||||
const suppressionMin = isNaN(suppressionMinRaw) || suppressionMinRaw <= 0
|
||||
? DEFAULT_HOST_ALERT_SUPPRESSION_MIN
|
||||
: Math.min(suppressionMinRaw, MAX_HOST_ALERT_SUPPRESSION_MIN);
|
||||
const suppressionMs = suppressionMin * 60 * 1000;
|
||||
|
||||
// 1. Host Limits — fetch CPU, RAM, disk concurrently
|
||||
try {
|
||||
@@ -239,23 +274,29 @@ export class MonitorService {
|
||||
const cpuUsage = currentLoad.currentLoad;
|
||||
const cpuLimit = parseFloat(settings['host_cpu_limit']);
|
||||
if (!isNaN(cpuLimit) && cpuLimit > 0 && cpuUsage > cpuLimit) {
|
||||
await this.dispatchWithCooldown(HOST_ALERT_KEYS.cpu, HOST_ALERT_COOLDOWN_MS, 'warning', 'monitor_alert',
|
||||
await this.dispatchHostMetricAlert('cpu', 'warning', suppressionMs,
|
||||
`Host CPU utilization is critically high: ${cpuUsage.toFixed(1)}% (Threshold: ${cpuLimit}%)`);
|
||||
} else {
|
||||
this.clearHostMetricSuppression('cpu');
|
||||
}
|
||||
|
||||
const ramUsage = (mem.used / mem.total) * 100;
|
||||
const ramLimit = parseFloat(settings['host_ram_limit']);
|
||||
if (!isNaN(ramLimit) && ramLimit > 0 && ramUsage > ramLimit) {
|
||||
await this.dispatchWithCooldown(HOST_ALERT_KEYS.ram, HOST_ALERT_COOLDOWN_MS, 'warning', 'monitor_alert',
|
||||
await this.dispatchHostMetricAlert('ram', 'warning', suppressionMs,
|
||||
`Host Memory utilization is critically high: ${ramUsage.toFixed(1)}% (Threshold: ${ramLimit}%)`);
|
||||
} else {
|
||||
this.clearHostMetricSuppression('ram');
|
||||
}
|
||||
|
||||
const mainDisk = fsSize.find(fs => fs.mount === '/' || fs.mount === 'C:') || fsSize[0];
|
||||
if (mainDisk) {
|
||||
const diskLimit = parseFloat(settings['host_disk_limit']);
|
||||
if (!isNaN(diskLimit) && diskLimit > 0 && mainDisk.use > diskLimit) {
|
||||
await this.dispatchWithCooldown(HOST_ALERT_KEYS.disk, HOST_ALERT_COOLDOWN_MS, 'warning', 'monitor_alert',
|
||||
await this.dispatchHostMetricAlert('disk', 'warning', suppressionMs,
|
||||
`Host Disk space utilization is critically high: ${mainDisk.use.toFixed(1)}% (Threshold: ${diskLimit}%)`);
|
||||
} else {
|
||||
this.clearHostMetricSuppression('disk');
|
||||
}
|
||||
}
|
||||
} catch (e) {
|
||||
@@ -687,6 +728,104 @@ export class MonitorService {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* F-11 host-metric dispatch with per-key suppression window + count
|
||||
* summary. Replaces the old 5-minute hardcoded cooldown for CPU/RAM/disk.
|
||||
*
|
||||
* Behaviour:
|
||||
* - First breach (no in-memory state, no recent persisted timestamp) →
|
||||
* dispatch immediately and seed in-memory state.
|
||||
* - Restart on a still-breaching host (no in-memory state, but persisted
|
||||
* timestamp falls inside the window) → silently re-seed from the
|
||||
* persisted timestamp so post-restart cycles do not re-fire.
|
||||
* - Breach within an open suppression window → silently increment
|
||||
* suppressedCount; no dispatch, no broadcast.
|
||||
* - Breach after window elapses → fire one follow-up with the suffix
|
||||
* `Suppressed N alerts in the last Xm; first over threshold at HH:MM UTC.`
|
||||
*
|
||||
* Recovery is handled by the caller via clearHostMetricSuppression(): when
|
||||
* the metric drops below threshold, in-memory state and the persisted
|
||||
* timestamp are cleared so the next breach fires fresh.
|
||||
*/
|
||||
private async dispatchHostMetricAlert(
|
||||
key: HostMetricKey,
|
||||
severity: 'info' | 'warning' | 'error',
|
||||
suppressionMs: number,
|
||||
baseMessage: string,
|
||||
): Promise<void> {
|
||||
const db = DatabaseService.getInstance();
|
||||
const stateKey = HOST_ALERT_KEYS[key];
|
||||
const now = Date.now();
|
||||
const state = hostAlertSuppressionState.get(key);
|
||||
|
||||
if (!state) {
|
||||
const persistedLast = parseInt(db.getSystemState(stateKey) || '0', 10);
|
||||
if (persistedLast > 0 && now - persistedLast < suppressionMs) {
|
||||
// Post-restart on a still-breaching host: respect the
|
||||
// persisted cooldown so we do not re-fire immediately. Seed
|
||||
// in-memory state from the persisted timestamp; count this
|
||||
// cycle as suppressed for an honest follow-up summary.
|
||||
hostAlertSuppressionState.set(key, {
|
||||
firstFiredAt: persistedLast,
|
||||
lastFiredAt: persistedLast,
|
||||
suppressedCount: 1,
|
||||
});
|
||||
return;
|
||||
}
|
||||
// First-ever fire (fresh process, or post-recovery re-breach).
|
||||
await NotificationService.getInstance().dispatchAlert(severity, 'monitor_alert', baseMessage);
|
||||
hostAlertSuppressionState.set(key, {
|
||||
firstFiredAt: now,
|
||||
lastFiredAt: now,
|
||||
suppressedCount: 0,
|
||||
});
|
||||
db.setSystemState(stateKey, now.toString());
|
||||
return;
|
||||
}
|
||||
|
||||
if (now - state.lastFiredAt < suppressionMs) {
|
||||
state.suppressedCount += 1;
|
||||
return;
|
||||
}
|
||||
|
||||
const windowMin = Math.max(1, Math.round((now - state.lastFiredAt) / 60000));
|
||||
const firstAt = new Date(state.firstFiredAt).toISOString().slice(11, 16);
|
||||
const plural = state.suppressedCount === 1 ? '' : 's';
|
||||
const suffix = ` Suppressed ${state.suppressedCount} alert${plural} in the last ${windowMin}m; first over threshold at ${firstAt} UTC.`;
|
||||
await NotificationService.getInstance().dispatchAlert(severity, 'monitor_alert', baseMessage + suffix);
|
||||
state.lastFiredAt = now;
|
||||
state.suppressedCount = 0;
|
||||
db.setSystemState(stateKey, now.toString());
|
||||
}
|
||||
|
||||
/**
|
||||
* Clear in-memory + persisted suppression state for a host metric. Called
|
||||
* when the metric drops below threshold so the next breach fires fresh
|
||||
* (no "Suppressed N" suffix). Resetting the persisted row to '0' is
|
||||
* essential: a stale persisted timestamp inside the window would silently
|
||||
* absorb the next breach via the restart-survivability path.
|
||||
*
|
||||
* The persisted reset must run independently of in-memory state because
|
||||
* the two halves can drift after a process restart: the in-memory Map is
|
||||
* empty on a fresh process, but the system_state row from the previous
|
||||
* process still points at the old fire timestamp. If recovery happens
|
||||
* before another evaluate cycle re-seeds in-memory state, an early-return
|
||||
* on the in-memory check alone would leave the persisted row alive, and
|
||||
* the next re-breach within the original window would be silently
|
||||
* suppressed instead of firing fresh (Codex audit on PR #1175).
|
||||
*/
|
||||
private clearHostMetricSuppression(key: HostMetricKey): void {
|
||||
const stateKey = HOST_ALERT_KEYS[key];
|
||||
const db = DatabaseService.getInstance();
|
||||
const hadInMemory = hostAlertSuppressionState.delete(key);
|
||||
const persisted = db.getSystemState(stateKey);
|
||||
const needsPersistedReset = persisted !== null && persisted !== '0';
|
||||
if (!hadInMemory && !needsPersistedReset) return;
|
||||
if (needsPersistedReset) {
|
||||
db.setSystemState(stateKey, '0');
|
||||
}
|
||||
}
|
||||
|
||||
private calculateCpuPercent(stats: DockerContainerStats): number {
|
||||
let cpuPercent = 0.0;
|
||||
if (!stats?.cpu_stats?.cpu_usage || !stats?.precpu_stats?.cpu_usage) return 0.0;
|
||||
|
||||
Reference in New Issue
Block a user