Prove resolved alert lock discipline

Route canonical cooldown refires through one resolved-state critical section and document the m.mu to resolvedMutex invariant. Exercise production refire, recovery, broadcast, alias repair, cleanup, and shutdown paths under the race detector while preserving cooldown history and start-time semantics.
This commit is contained in:
rcourtman
2026-07-23 21:56:38 +01:00
parent 5772589363
commit b14cb376bc
7 changed files with 316 additions and 78 deletions
@@ -298,6 +298,16 @@ unbounded history store. `recentlyResolved` must prune expired entries and cap
the newest retained entries on insert as well as during cleanup, so monitor
sync and websocket state snapshots remain bounded; durable resolved-alert
history belongs in the alert history store, not in this live transition cache.
`recentlyResolved` and `resolvedAlias` have one lock owner:
`Manager.resolvedMutex`. Every access holds that mutex, including alias-repair
lookups, which require the write lock. When an operation needs both manager
state and resolved state, the only permitted nested order is
`Manager.mu` then `Manager.resolvedMutex`; no path may acquire `Manager.mu`
while holding `Manager.resolvedMutex`. Resolved critical sections are limited
to map access and must not dispatch, persist history, invoke callbacks, or
perform notification work. Canonical lifecycle and stateful cooldown refires
consume resolved state through the shared lock-order-aware helper, preserve the
original alert `StartTime`, and keep the five-minute refire/history semantics.
The browser thresholds surface is also platform-shaped: Proxmox, Docker,
Kubernetes, TrueNAS, vSphere, PBS, PMG, and Systems. It must use the shared
FilterBar chip and "+ Filter" pattern for resource filtering, and alert tables
@@ -2319,6 +2319,7 @@
"internal/alerts/history_test.go",
"internal/alerts/intent_policy_test.go",
"internal/alerts/operational_contract_test.go",
"internal/alerts/resolved_lock_discipline_test.go",
"internal/alerts/synology_test.go",
"internal/alerts/threshold_resolution_shared_test.go",
"internal/alerts/update_alerts_test.go",