From 1c4241ad0df62faaa5de88947d5e80bb2e2ff064 Mon Sep 17 00:00:00 2001 From: rcourtman Date: Wed, 26 Aug 2026 20:54:14 +0100 Subject: [PATCH] docs: scope alert engine evolution and register suppression-observability gap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The March canonical migration froze the transition core and suppression path; the post-March regression record (#1682, #1683, #1553, #1693) shows those layers are where the recurring lifecycle bug classes live. ALERT_ENGINE_EVOLUTION.md extends the migration end-state one layer deeper — additive event log (Phase 0), shadow reducer (Phase 1), family-by-family cutover (Phase 2), declarative rules (Phase 3) — using the same strangler mechanism. Registers the Phase 0 work as coverage gap alert-engine-suppression-observability with its record. --- docs/ALERT_ENGINE_EVOLUTION.md | 93 +++++++++++++++++++ ...ne-suppression-observability-2026-08-26.md | 41 ++++++++ docs/release-control/v6/internal/status.json | 47 +++++++++- 3 files changed, 179 insertions(+), 2 deletions(-) create mode 100644 docs/ALERT_ENGINE_EVOLUTION.md create mode 100644 docs/release-control/v6/internal/records/alert-engine-suppression-observability-2026-08-26.md diff --git a/docs/ALERT_ENGINE_EVOLUTION.md b/docs/ALERT_ENGINE_EVOLUTION.md new file mode 100644 index 000000000..6149b0493 --- /dev/null +++ b/docs/ALERT_ENGINE_EVOLUTION.md @@ -0,0 +1,93 @@ +# Alert Engine Evolution + +Status: Active +Date: 2026-08-26 +Scope: `pulse` only +Predecessor: `docs/CANONICAL_ALERT_ENGINE_MIGRATION_2026-03-10.md` + +## Purpose + +The March 2026 canonical migration correctly rejected a full alert-engine +rewrite and began moving alert *inputs* onto canonical identity +(`internal/unifiedresources`) and specs (`internal/alerts/specs`). It +deliberately froze two layers: transition state stays in the manager's +tracking maps, and notification-suppression rules stay where they are. + +The post-March issue record shows the recurring regression classes live in +exactly those frozen layers: config saves emitting resolved notifications +(#1682), grouping silently dropping N−1 of N alerts (#1683), resolves sent +for suppressed firings (#1553), self-resolving fire/resolve churn deleting +history (#1693), chronic re-fires on terminal container state (#1724). +Cleaner inputs feeding the same fragile core does not retire these classes. + +This document extends the migration's end-state one layer deeper. The +mechanism stays the same — strangler migration inside one engine, no +big-bang rewrite — applied to the transition core and the suppression path. + +## Target model (delta over the March doc) + +The March doc's layers 1 (unified resource identity), 2 (canonical specs), +and 5 (notification fan-out boundary in `monitor_alerts.go`) stand +unchanged. Two layers are added or replaced: + +### Event log (new, Phase 0) + +An append-only alert event log, SQLite-backed, owned by `internal/alerts`. +Every lifecycle transition (pending, firing, resolved, acknowledged, +escalated) and every notification decision — including suppressions, with +the mechanism and reason that held them — is an event. History, the +delivery log, frequency analytics, and `AlertDeliveryDiagnosis` become +projections of one log instead of separately maintained structures. +Transparency stops being a feature and becomes a property of the +architecture. + +Phase 0 is strictly additive: the existing manager writes events alongside +its current behavior. No lifecycle semantics change. + +### Incident reducer (replaces the frozen transition core, Phases 1–2) + +A pure transition function — `next(state, signal, rules, clock) → +(state', events)` — replaces the manager's string-keyed tracking maps. +Identity is computed in exactly one place (canonical resource ID + spec +ID); lifecycle is an explicit state machine; acknowledgement, snooze, and +flapping are typed fields on one incident record. Because the reducer is +pure, the lifecycle contract suite (config save ⇒ zero events; suppressed +firing ⇒ no resolve; N grouped ⇒ N rendered) runs exhaustively and gates +releases. + +The March doc's "stable behaviors that must not change" list is the seed +of the reducer's characterization suite: ack preservation, cooldown +re-notify, escalation carry-over, and confirmation semantics are pinned by +parity tests against the live manager before any cutover, not rewritten +from memory. + +## Phases + +- **Phase 0 — event log, additive.** Event store + manager wiring for + transition and suppression events + API + UI surfacing of the existing + delivery diagnosis and held-notification rows. Zero lifecycle risk. + Registered as coverage gap `alert-engine-suppression-observability`. +- **Phase 1 — reducer in shadow mode.** Built beside the manager, fed the + same inputs, outputs diffed continuously (the + `unified_eval_parity_test.go` pattern). No user-visible change. +- **Phase 2 — family-by-family cutover.** Worst identity offenders first + (Docker, PBS/storage). Each family cutover deletes its tracking maps and + the imperative bulk of its `Check*` path. Alert IDs and override keys + stay stable until an explicit config migration. +- **Phase 3 — declarative rule model.** Scope selector + condition + + policy replaces the per-platform config blocks and `DisableAll*` + booleans, with a translator from the existing `AlertConfig` so persisted + user configs keep working. UI migrates tab by tab. + +## What is kept + +The delivery pipeline (`internal/notifications` queue, DLQ, receipts, +templates, delivery health), the operator-intent fabric, unified +resources, the specs/evaluator layer, the UI shell, and the +`monitor_alerts.go` fan-out boundary — which becomes an event consumer. + +## What is retired (by the end of Phase 2–3) + +The manager's per-family tracking maps, the imperative per-platform +check bodies, the scattered suppression checks, and — once the event log +is authoritative — the JSON snapshot history file. diff --git a/docs/release-control/v6/internal/records/alert-engine-suppression-observability-2026-08-26.md b/docs/release-control/v6/internal/records/alert-engine-suppression-observability-2026-08-26.md new file mode 100644 index 000000000..b29ded8e2 --- /dev/null +++ b/docs/release-control/v6/internal/records/alert-engine-suppression-observability-2026-08-26.md @@ -0,0 +1,41 @@ +# Alert engine suppression observability and lifecycle trust + +Recorded: 2026-08-26 +Origin: full alerting audit (interactive session, 2026-08-26) covering +`internal/alerts`, `internal/notifications`, the Alerts UI, and ~60 +alert-related GitHub issues. + +## Gap + +Three related defects in the alerting trust surface: + +1. **Suppression decisions are unobservable.** Eight-plus noise mechanisms + (cooldown, max-per-hour, suppression window, minimum delta, flapping, + grouping, quiet hours, activation state) can hold or drop a notification, + and no surface records which one did, when, or why. The delivery log + records attempts only. Users experiencing held notifications read the + silence as breakage (issues #1159, #1444, #937, #980 are this class). + +2. **The delivery-diagnosis endpoint has no consumer.** + `GET /api/alerts/delivery-diagnosis` projects exactly why a given active + alert would or would not notify (`AlertDeliveryDiagnosis` in + `internal/alerts/notification_policy.go`), and no frontend code calls it. + The most common alerting trust question — "why didn't I get notified?" — + is answerable by the backend and invisible to users. + +3. **Alert lifecycle transitions have no durable event trail.** History + stores merged alert snapshots, not transitions; several lifecycle + regressions shipped undetected because the end-to-end contract + (config save ⇒ zero notifications #1682; N grouped alerts ⇒ N rendered + lines #1683; suppressed firing ⇒ no resolve #1553) is asserted nowhere + durable and observable. + +## Proposed resolution + +Phase 0 of `docs/ALERT_ENGINE_EVOLUTION.md`: an additive append-only alert +event log (SQLite, alongside existing behavior — no lifecycle changes), +manager wiring that records transition and suppression events with reasons, +an API to read them, and UI surfacing of the delivery diagnosis plus +held-notification rows. Later phases (shadow reducer, family-by-family +cutover, rule model) are scoped in the same doc and are not part of this +gap's resolution. diff --git a/docs/release-control/v6/internal/status.json b/docs/release-control/v6/internal/status.json index 893ed9aa9..f2c8cca39 100644 --- a/docs/release-control/v6/internal/status.json +++ b/docs/release-control/v6/internal/status.json @@ -1,6 +1,6 @@ { "version": "6.0", - "updated_at": "2026-08-08", + "updated_at": "2026-08-26", "scope": { "active_repos": [ "pulse", @@ -9758,10 +9758,53 @@ "kind": "file" } ] + }, + { + "id": "alert-engine-suppression-observability", + "summary": "Alert notification suppression decisions (cooldown, quiet hours, flapping, grouping, rate limit, activation state) are unobservable: the delivery log records attempts only, the delivery-diagnosis endpoint has no frontend consumer, and lifecycle transitions have no durable event trail, so held notifications read as breakage and lifecycle regressions (#1682, #1683, #1553, #1693) ship without an end-to-end contract gate. Resolution is Phase 0 of docs/ALERT_ENGINE_EVOLUTION.md: an additive SQLite alert event log recording transitions and suppressions with reasons, plus UI surfacing of the existing delivery diagnosis.", + "owner": "project-owner", + "status": "triaged", + "recorded_at": "2026-08-26", + "lane_ids": [ + "L6", + "L13" + ], + "subsystem_ids": [ + "alerts", + "notifications" + ], + "proposed_resolution": "lane-expansion", + "coverage_impact": 5, + "evidence": [ + { + "repo": "pulse", + "path": "docs/release-control/v6/internal/records/alert-engine-suppression-observability-2026-08-26.md", + "kind": "file" + }, + { + "repo": "pulse", + "path": "docs/ALERT_ENGINE_EVOLUTION.md", + "kind": "file" + } + ] } ], "candidate_lanes": [], - "work_claims": [], + "work_claims": [ + { + "id": "claude-alert-eventlog-coverage-gap-alert-engine-suppression-observability", + "agent_id": "claude-alert-eventlog", + "summary": "Phase 0 of ALERT_ENGINE_EVOLUTION: additive alert event log (transitions + suppression reasons), delivery-diagnosis UI surfacing", + "target_id": "v6-product-lane-expansion", + "claimed_at": "2026-08-26T19:54:04Z", + "heartbeat_at": "2026-08-26T19:54:04Z", + "expires_at": "2026-08-27T03:54:04Z", + "work_item": { + "kind": "coverage-gap", + "id": "alert-engine-suppression-observability" + } + } + ], "open_decisions": [], "source_of_truth_file": "docs/release-control/v6/internal/SOURCE_OF_TRUTH.md", "resolved_decisions": [