Pulse could tell that its own delivery had stopped and had no way to say so.
Queue health was computed for one API endpoint and rendered on the alerts
surfaces, and that was the whole escalation path: a destination that stops
delivering cannot announce itself through a notification, so an operator who
does not open the alerts page learns nothing.
Telemetry: 78 installs retry every notification to the maxAttempts ceiling and
deliver almost nothing, 155,967 attempts against 221 deliveries in a week. 50
have delivered nothing in 30 days and 28 more delivered normally at some point
and silently stopped. The dominant terminal failure classes are authentication
and configuration, so most are fixable in minutes once someone knows.
Move the queue-health rule into internal/notifications as ClassifyQueueHealth
and DeliveryHealth, next to the queue that produces the counts, and delegate
the API handler to it so the endpoint and the new evaluator cannot drift into
two rules. A queue that cannot be read reports unavailable rather than healthy,
because silence is the failure mode being guarded against.
Monitoring evaluates delivery health on the poll ticker and raises or clears
the notification-delivery system alert, which puts it in the alert list and the
navigation badge. Reading queue health costs a SQLite query and the poll
cadence can be seconds, so the evaluation is throttled to five minutes.
RaiseSystemAlert is idempotent for an unchanged condition, so the timer neither
re-notifies nor accumulates alerts.
The alert is a warning rather than critical. It is real, but a new
critical-by-default alert appearing on upgrade across the affected installs is
a bigger change than this should make on its own.
The new Monitor field is deliberately named to fit inside the existing struct
alignment column. A longer name makes gofmt re-pad the whole block, which
breaks the canonical guardrail tests that pin those field declarations
verbatim.
Proof: delivery_health_test.go pins that in-flight work and retries stay
healthy while retained terminal failures do not, and that an unreadable queue
is never reported healthy. system_alerts_test.go pins the message for each
outcome including singular and plural, that it points at the destinations
surface, and that the throttle honours its interval. internal/monitoring,
internal/notifications and internal/alerts suites all pass.
Contract-Neutral: Staged contract deltas cover every boundary this change actually moves: notifications.md gains delivery_health.go plus the ClassifyQueueHealth rule, and monitoring.md gains system_alerts.go plus the system-alert evaluation extension point. The residual demands are inapplicable. agent-lifecycle is demanded only because monitor.go is one of its canonical files, and its verification artifact is host-agent deletion and re-enrollment proof, which this change does not touch: the diff to monitor.go is one struct field and one call in the poll loop. api-contracts and storage-recovery are demanded only because internal/api/notifications.go falls under a broad internal/api reference, and that file's change is a pure delegation of an internal helper to notifications.ClassifyQueueHealth with a byte-identical HTTP response and no route or payload delta.
rcourtman
1a14212785