mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-09-10 02:25:56 +00:00
docs: scope alert engine evolution and register suppression-observability gap
The March canonical migration froze the transition core and suppression path; the post-March regression record (#1682, #1683, #1553, #1693) shows those layers are where the recurring lifecycle bug classes live. ALERT_ENGINE_EVOLUTION.md extends the migration end-state one layer deeper — additive event log (Phase 0), shadow reducer (Phase 1), family-by-family cutover (Phase 2), declarative rules (Phase 3) — using the same strangler mechanism. Registers the Phase 0 work as coverage gap alert-engine-suppression-observability with its record.
This commit is contained in:
rcourtman
@@ -0,0 +1,93 @@
|
||||
# Alert Engine Evolution
|
||||
|
||||
Status: Active
|
||||
Date: 2026-08-26
|
||||
Scope: `pulse` only
|
||||
Predecessor: `docs/CANONICAL_ALERT_ENGINE_MIGRATION_2026-03-10.md`
|
||||
|
||||
## Purpose
|
||||
|
||||
The March 2026 canonical migration correctly rejected a full alert-engine
|
||||
rewrite and began moving alert *inputs* onto canonical identity
|
||||
(`internal/unifiedresources`) and specs (`internal/alerts/specs`). It
|
||||
deliberately froze two layers: transition state stays in the manager's
|
||||
tracking maps, and notification-suppression rules stay where they are.
|
||||
|
||||
The post-March issue record shows the recurring regression classes live in
|
||||
exactly those frozen layers: config saves emitting resolved notifications
|
||||
(#1682), grouping silently dropping N−1 of N alerts (#1683), resolves sent
|
||||
for suppressed firings (#1553), self-resolving fire/resolve churn deleting
|
||||
history (#1693), chronic re-fires on terminal container state (#1724).
|
||||
Cleaner inputs feeding the same fragile core does not retire these classes.
|
||||
|
||||
This document extends the migration's end-state one layer deeper. The
|
||||
mechanism stays the same — strangler migration inside one engine, no
|
||||
big-bang rewrite — applied to the transition core and the suppression path.
|
||||
|
||||
## Target model (delta over the March doc)
|
||||
|
||||
The March doc's layers 1 (unified resource identity), 2 (canonical specs),
|
||||
and 5 (notification fan-out boundary in `monitor_alerts.go`) stand
|
||||
unchanged. Two layers are added or replaced:
|
||||
|
||||
### Event log (new, Phase 0)
|
||||
|
||||
An append-only alert event log, SQLite-backed, owned by `internal/alerts`.
|
||||
Every lifecycle transition (pending, firing, resolved, acknowledged,
|
||||
escalated) and every notification decision — including suppressions, with
|
||||
the mechanism and reason that held them — is an event. History, the
|
||||
delivery log, frequency analytics, and `AlertDeliveryDiagnosis` become
|
||||
projections of one log instead of separately maintained structures.
|
||||
Transparency stops being a feature and becomes a property of the
|
||||
architecture.
|
||||
|
||||
Phase 0 is strictly additive: the existing manager writes events alongside
|
||||
its current behavior. No lifecycle semantics change.
|
||||
|
||||
### Incident reducer (replaces the frozen transition core, Phases 1–2)
|
||||
|
||||
A pure transition function — `next(state, signal, rules, clock) →
|
||||
(state', events)` — replaces the manager's string-keyed tracking maps.
|
||||
Identity is computed in exactly one place (canonical resource ID + spec
|
||||
ID); lifecycle is an explicit state machine; acknowledgement, snooze, and
|
||||
flapping are typed fields on one incident record. Because the reducer is
|
||||
pure, the lifecycle contract suite (config save ⇒ zero events; suppressed
|
||||
firing ⇒ no resolve; N grouped ⇒ N rendered) runs exhaustively and gates
|
||||
releases.
|
||||
|
||||
The March doc's "stable behaviors that must not change" list is the seed
|
||||
of the reducer's characterization suite: ack preservation, cooldown
|
||||
re-notify, escalation carry-over, and confirmation semantics are pinned by
|
||||
parity tests against the live manager before any cutover, not rewritten
|
||||
from memory.
|
||||
|
||||
## Phases
|
||||
|
||||
- **Phase 0 — event log, additive.** Event store + manager wiring for
|
||||
transition and suppression events + API + UI surfacing of the existing
|
||||
delivery diagnosis and held-notification rows. Zero lifecycle risk.
|
||||
Registered as coverage gap `alert-engine-suppression-observability`.
|
||||
- **Phase 1 — reducer in shadow mode.** Built beside the manager, fed the
|
||||
same inputs, outputs diffed continuously (the
|
||||
`unified_eval_parity_test.go` pattern). No user-visible change.
|
||||
- **Phase 2 — family-by-family cutover.** Worst identity offenders first
|
||||
(Docker, PBS/storage). Each family cutover deletes its tracking maps and
|
||||
the imperative bulk of its `Check*` path. Alert IDs and override keys
|
||||
stay stable until an explicit config migration.
|
||||
- **Phase 3 — declarative rule model.** Scope selector + condition +
|
||||
policy replaces the per-platform config blocks and `DisableAll*`
|
||||
booleans, with a translator from the existing `AlertConfig` so persisted
|
||||
user configs keep working. UI migrates tab by tab.
|
||||
|
||||
## What is kept
|
||||
|
||||
The delivery pipeline (`internal/notifications` queue, DLQ, receipts,
|
||||
templates, delivery health), the operator-intent fabric, unified
|
||||
resources, the specs/evaluator layer, the UI shell, and the
|
||||
`monitor_alerts.go` fan-out boundary — which becomes an event consumer.
|
||||
|
||||
## What is retired (by the end of Phase 2–3)
|
||||
|
||||
The manager's per-family tracking maps, the imperative per-platform
|
||||
check bodies, the scattered suppression checks, and — once the event log
|
||||
is authoritative — the JSON snapshot history file.
|
||||
+41
@@ -0,0 +1,41 @@
|
||||
# Alert engine suppression observability and lifecycle trust
|
||||
|
||||
Recorded: 2026-08-26
|
||||
Origin: full alerting audit (interactive session, 2026-08-26) covering
|
||||
`internal/alerts`, `internal/notifications`, the Alerts UI, and ~60
|
||||
alert-related GitHub issues.
|
||||
|
||||
## Gap
|
||||
|
||||
Three related defects in the alerting trust surface:
|
||||
|
||||
1. **Suppression decisions are unobservable.** Eight-plus noise mechanisms
|
||||
(cooldown, max-per-hour, suppression window, minimum delta, flapping,
|
||||
grouping, quiet hours, activation state) can hold or drop a notification,
|
||||
and no surface records which one did, when, or why. The delivery log
|
||||
records attempts only. Users experiencing held notifications read the
|
||||
silence as breakage (issues #1159, #1444, #937, #980 are this class).
|
||||
|
||||
2. **The delivery-diagnosis endpoint has no consumer.**
|
||||
`GET /api/alerts/delivery-diagnosis` projects exactly why a given active
|
||||
alert would or would not notify (`AlertDeliveryDiagnosis` in
|
||||
`internal/alerts/notification_policy.go`), and no frontend code calls it.
|
||||
The most common alerting trust question — "why didn't I get notified?" —
|
||||
is answerable by the backend and invisible to users.
|
||||
|
||||
3. **Alert lifecycle transitions have no durable event trail.** History
|
||||
stores merged alert snapshots, not transitions; several lifecycle
|
||||
regressions shipped undetected because the end-to-end contract
|
||||
(config save ⇒ zero notifications #1682; N grouped alerts ⇒ N rendered
|
||||
lines #1683; suppressed firing ⇒ no resolve #1553) is asserted nowhere
|
||||
durable and observable.
|
||||
|
||||
## Proposed resolution
|
||||
|
||||
Phase 0 of `docs/ALERT_ENGINE_EVOLUTION.md`: an additive append-only alert
|
||||
event log (SQLite, alongside existing behavior — no lifecycle changes),
|
||||
manager wiring that records transition and suppression events with reasons,
|
||||
an API to read them, and UI surfacing of the delivery diagnosis plus
|
||||
held-notification rows. Later phases (shadow reducer, family-by-family
|
||||
cutover, rule model) are scoped in the same doc and are not part of this
|
||||
gap's resolution.
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"version": "6.0",
|
||||
"updated_at": "2026-08-08",
|
||||
"updated_at": "2026-08-26",
|
||||
"scope": {
|
||||
"active_repos": [
|
||||
"pulse",
|
||||
@@ -9758,10 +9758,53 @@
|
||||
"kind": "file"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "alert-engine-suppression-observability",
|
||||
"summary": "Alert notification suppression decisions (cooldown, quiet hours, flapping, grouping, rate limit, activation state) are unobservable: the delivery log records attempts only, the delivery-diagnosis endpoint has no frontend consumer, and lifecycle transitions have no durable event trail, so held notifications read as breakage and lifecycle regressions (#1682, #1683, #1553, #1693) ship without an end-to-end contract gate. Resolution is Phase 0 of docs/ALERT_ENGINE_EVOLUTION.md: an additive SQLite alert event log recording transitions and suppressions with reasons, plus UI surfacing of the existing delivery diagnosis.",
|
||||
"owner": "project-owner",
|
||||
"status": "triaged",
|
||||
"recorded_at": "2026-08-26",
|
||||
"lane_ids": [
|
||||
"L6",
|
||||
"L13"
|
||||
],
|
||||
"subsystem_ids": [
|
||||
"alerts",
|
||||
"notifications"
|
||||
],
|
||||
"proposed_resolution": "lane-expansion",
|
||||
"coverage_impact": 5,
|
||||
"evidence": [
|
||||
{
|
||||
"repo": "pulse",
|
||||
"path": "docs/release-control/v6/internal/records/alert-engine-suppression-observability-2026-08-26.md",
|
||||
"kind": "file"
|
||||
},
|
||||
{
|
||||
"repo": "pulse",
|
||||
"path": "docs/ALERT_ENGINE_EVOLUTION.md",
|
||||
"kind": "file"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"candidate_lanes": [],
|
||||
"work_claims": [],
|
||||
"work_claims": [
|
||||
{
|
||||
"id": "claude-alert-eventlog-coverage-gap-alert-engine-suppression-observability",
|
||||
"agent_id": "claude-alert-eventlog",
|
||||
"summary": "Phase 0 of ALERT_ENGINE_EVOLUTION: additive alert event log (transitions + suppression reasons), delivery-diagnosis UI surfacing",
|
||||
"target_id": "v6-product-lane-expansion",
|
||||
"claimed_at": "2026-08-26T19:54:04Z",
|
||||
"heartbeat_at": "2026-08-26T19:54:04Z",
|
||||
"expires_at": "2026-08-27T03:54:04Z",
|
||||
"work_item": {
|
||||
"kind": "coverage-gap",
|
||||
"id": "alert-engine-suppression-observability"
|
||||
}
|
||||
}
|
||||
],
|
||||
"open_decisions": [],
|
||||
"source_of_truth_file": "docs/release-control/v6/internal/SOURCE_OF_TRUTH.md",
|
||||
"resolved_decisions": [
|
||||
|
||||
Reference in New Issue
Block a user