Files
pulse/internal
rcourtman e9069aed8c Carry a typed reason when an alert closes without recovering (#2580)
When a Pulse agent is linked to a Proxmox node, the node releases its
CPU, memory and disk alerts to the agent, and any node alert still open
closes. The close must reach integrations, so a PagerDuty or webhook
incident opened from the node alert does not stay open forever. But it
went out as a recovery: "pve1 on pve1 is now healthy" for a node still
at 95% memory, backed by synthesized "recovery projection" evidence.
#1682 is the same complaint from another trigger.

This adds a typed `AlertResolution` (first reason: `moved_to_agent`,
with the agent's alert resource ID and name) and carries it from the
release path to every resolved consumer:

- `releaseCanonicalMetricAlert` is the disabled-threshold release pulled
out of `evaluateCanonicalMetricAlert`. It takes a resolution, and
`clearAlertWithResolution` stamps a copy on the alert before it
resolves.
- The resolution rides the Alert JSON into the resolved callback, the
delivery queue, the history row and the event log (reason code plus
summary).
- The synthesized closing evidence and the resolve transition say what
happened instead of claiming a recovery projection. The cause stays
`recovery_evidence`, the only cause operational trust lets enter
`resolved`.
- Notifications still send the close. Messages say "Memory alert moved
to pve1 (Host Agent). This is not a recovery: check the agent for the
current reading." in webhooks, `MOVED:` on a single ntfy push, and
"Pulse alert moved" on email and Apprise. When any close in a delivery
was not a recovery, `WebhookPayloadData.NotRecovered` drops recovery
visuals: Discord's green embed turns neutral blue, both Teams cards lose
recovery green, Mattermost's check emoji becomes an arrow, and ntfy uses
the arrow tag for a single alert or a batch. Titles still say Resolved.
PagerDuty still gets `resolve`.

Every caller passes a nil resolution today, so ordinary recoveries and
the disabled-threshold path are unchanged. The node release adopts the
reason in a follow-up, once the host-agent node link records which
metrics the agent evaluates (#2577 lane).

Tests: `TestReleasedMetricAlertCarriesResolutionInsteadOfRecovery` and
`TestResolvedHistoryRowKeepsResolutionWithoutEventLog`
(operational_contract_test.go); `resolved_handover_test.go` drives the
firing, its receipt and the queued close through the normal ntfy entry
points, and renders Discord, both Teams cards, Mattermost, PagerDuty, a
grouped Slack list, a grouped ntfy batch and email/Apprise content, with
isolated managers. The full alerts, notifications and monitoring suites
pass locally.
2026-10-06 18:57:09 +01:00
..