Revalidate queued alert delivery against current quiet hours

Keep continuous and late replay held without spending a provider attempt. Split mixed batches atomically so eligible alerts retain admitted destinations, occurrence links and retry budgets, and bind saved monitor policy before activating persisted work. Add local receipt, cancellation, rollback, reconstruction and race regression coverage with the owning contracts and verification routes.

Change-source: pulse-maintainer
This commit is contained in:
pulse-triage[bot]
2026-10-01 23:05:07 +01:00
parent 680ac756e6
commit 5213b794b2
13 changed files with 1009 additions and 14 deletions
@@ -794,6 +794,14 @@ installer download and the agent's subsequent Pulse TLS connection.
## Shared Boundaries
The shared monitor constructor also composes notification bootstrap: it loads
saved alert policy and destinations, binds the alert owner's immutable
quiet-hours policy provider, then activates persistent queue delivery.
This ordering is notifications/alerts-owned and must not change agent enrollment
or transport. `internal/monitoring/monitor_notification_startup_test.go` checks
the actual constructor and reconstruction, including holding due persisted
notifications under a saved continuous schedule without provider attempts.
- Local administrator setup synchronises the router authorizer before returning a browser session, so API Access remains available to manage agent credentials. This changes neither agent token scopes nor command-policy intent, and must not grant an unrelated identity access to credential management.
### Notification recovery reload ownership
@@ -1988,14 +1988,25 @@ quiet-hours policy; nonexistent clock minutes do not shift the configured
start or end. Initial queued replay uses the first real minute outside the
current non-full-day window, including an unselected day reached at midnight.
This does not reinterpret the day toggles as the previous evening's ownership.
Full-day windows retain the existing daily replay boundary; this clock repair
is not current-policy revalidation of already queued work or external receipt.
Full-day windows retain the existing daily revalidation boundary; it is not an
invented open minute or permission to send during a continuous schedule.
`TestQuietHoursDaylightSavingClockAndReplay` in
`internal/alerts/quiet_hours_test.go` checks London, New York and Lord Howe
(one-hour and half-hour changes), inclusive boundaries and exact UTC replay.
`TestQuietHoursOvernightSelectedCalendarDays` checks overnight day selection.
Quiet-hours location fallback is read-only; configuration updates own the
location cache, including when public suppression helpers run concurrently.
`Manager.QuietHoursNotificationPolicy` exposes an immutable read-only snapshot
to the notification owner before it takes queue or per-alert delivery locks.
Initial admission and queued replay share civil-clock/category semantics,
including repeated/skipped minutes and inclusive ends. The queue revalidates
every due attempt against current policy rather than trusting old replay
metadata. Policy snapshots neither mutate alert lifecycle/acknowledgement nor
select destinations; cancellation and actual delivery remain queue-owned.
`TestQuietHoursDeliverySnapshotIsCurrentIndependentAndReadOnly` and
`TestQuietHoursDeliverySnapshotConcurrentConfigUpdates` in
`internal/alerts/quiet_hours_test.go` cover snapshot isolation and concurrent
configuration reads. This establishes source scheduling, not external receipt.
Quiet-hours suppression also applies to alert delivery lifecycle, not only the
initial raised notification. Resolved notifications must not fan out when the
alert was never notified or was already acknowledged, and monitoring-driven
@@ -1799,6 +1799,19 @@ service-history reads plus denial/recovery without fabricated samples.
## Current State
### Saved quiet-hours policy before queue activation
`Monitor.New` binds `alerts.Manager.QuietHoursNotificationPolicy` to the
notification manager after saved policy/destinations load and before
`StartQueueProcessing`. The notification owner revalidates due stored work and
owns all partitioning, delivery and cancellation; monitoring neither duplicates
the schedule nor changes destination selection or operational lifecycle.
`TestNewRevalidatesPersistedQuietHoursBeforeDelivery` in
`internal/monitoring/monitor_notification_startup_test.go` reconstructs the
actual monitor twice with due persisted work under a full-day saved schedule,
checking pending state and no local delivery/audit. This is source bootstrap
proof, not installed restart or provider acceptance.
TrueNAS REST alert arguments may be object, scalar, array, null or absent.
Non-object arguments must not abort snapshot collection or discard alerts.
Only typed object fields supply disk identity and SMART counters; neither scalar
@@ -73,6 +73,7 @@ or displayed a notification.
10. `internal/notifications/delivery_health.go`
11. `internal/notifications/deadman_config.go`
12. `internal/notifications/failure_class.go`
13. `internal/notifications/quiet_hours_queue.go`
## Shared Boundaries
@@ -141,6 +142,44 @@ stable opaque routing identities and must not expose credentials.
## Current State
### Current-policy quiet-hours replay
The persistent queue checks every due provider attempt against an immutable
snapshot from `alerts.Manager.QuietHoursNotificationPolicy`, installed by
`NotificationManager.SetQuietHoursPolicyProvider` before saved work is activated
in the monitor constructor. The snapshot is obtained before queue/database or
per-alert delivery locks; its evaluator does not call back into the alert owner.
Stored replay timestamps are wake conditions, not evidence that the current
schedule permits delivery. Full-day schedules recheck at the retained daily
boundary, and late replay into another quiet interval remains held.
`quiet_hours_queue.go` reloads the pending row after acquiring delivery gates,
preserving cancellations that rewrote a grouped row after worker discovery.
An all-held batch moves only its wake time. A mixed batch atomically retains the
held original and creates one ready child with the admitted destination bytes,
occurrence/transition links, creation time and remaining attempt budget. Child
IDs bind the parent and exact ready payload; held work does not gain an attempt,
audit entry, firing receipt or false success. Prior attempt audits remain on the
original batch, while actual child attempts carry their own linked receipts.
Already-due recovery groups cannot reopen their grouping window while held.
Partition persistence or ambiguous linkage failure leaves the original intact
and makes no provider attempt. Due-time refresh prevents stale worker snapshots
from repeatedly postponing or claiming a now-held row. Actual retries retain
the existing claim/cancellation gates and provider enabled checks.
This does not accelerate a future stored deadline after a schedule edit; the
current rule is applied at the next admitted wake. It does not change initial
target selection, mutable recovery tag admission, acknowledgement semantics,
operator retry budgets or at-least-once delivery after an interrupted send.
`quiet_hours_queue_test.go` proves continuous/late/DST replay holds, mixed
immediate/held delivery, exact admitted destination and operational linkage,
budget retention, transaction rollback, durable reopen, cancellation and stale
concurrent snapshots, grouped recoveries and all six provider/event families.
`internal/monitoring/monitor_notification_startup_test.go` exercises the actual
constructor with due persisted work across two startups. These are local source
and HTTP-receipt proofs, not installed provider or natural-day acceptance.
### Persistent resolved-alert grouping
`SendResolvedAlert` applies the configured grouping window to recoveries as
@@ -1623,6 +1623,7 @@
"internal/monitoring/docker_metric_presence_test.go",
"internal/monitoring/monitor_host_agent_removal_lifecycle_test.go",
"internal/monitoring/monitor_host_agents_test.go",
"internal/monitoring/monitor_notification_startup_test.go",
"internal/monitoring/physical_disk_roundtrip_test.go",
"scripts/installtests/agent_state_dir_lifecycle_test.go",
"scripts/installtests/install_ps1_test.go",
@@ -6565,6 +6566,7 @@
"internal/monitoring/monitor_docker_test.go",
"internal/monitoring/monitor_host_agent_removal_lifecycle_test.go",
"internal/monitoring/monitor_host_agents_test.go",
"internal/monitoring/monitor_notification_startup_test.go",
"internal/monitoring/monitor_mock_alerts_test.go",
"internal/monitoring/monitor_pve_cluster_refresh_test.go",
"internal/monitoring/monitor_pve_guest_lxc_test.go",