Keep continuous and late replay held without spending a provider attempt. Split mixed batches atomically so eligible alerts retain admitted destinations, occurrence links and retry budgets, and bind saved monitor policy before activating persisted work. Add local receipt, cancellation, rollback, reconstruction and race regression coverage with the owning contracts and verification routes.
Change-source: pulse-maintainer
A persisted notification that was already due at startup could be terminally cancelled as 'delivery disabled'. NewNotificationManagerWithDataDir woke the queue worker during construction, before the owner had applied saved webhook/email/Apprise configuration, so the still-empty destination list looked like a removed destination.
Install the queue processor without waking it. The worker ticker and later enqueues start delivery once configuration is present. This removes the order/state-dependent failure in TestResolvedGroupingOrdinaryRestart and drops a persisted grouped recovery in production.
Also wait past the queue poll interval in the grouping contract test, add a regression test for the startup ordering, and update the notifications subsystem contract's processor-attachment semantics.
Change-source: pulse-maintainer
Prevent simultaneous recoveries from bypassing the configured grouping window. Preserve destination receipts and pending restart recovery, and keep PagerDuty incident keys separate. Include the notification contract and regression coverage with the runtime change.
Change-source: pulse-maintainer
Distinguish policy skips from provider success so suppressed jobs do not create false sent rows or successful audit entries. Reconcile cancelled queue health after releasing alert gates, preserve real attempt history, and cover all three providers for firing/recovery and global/destination disablement.
Change-source: pulse-maintainer
Resolution only cancelled pending/sending rows, so retained failures could resurrect healthy incidents after operator retry and restart. Suppress terminal firing entries too, preserve unrelated grouped alerts and recovery jobs, and reconcile queue health after releasing locks. Reject stale per-item retries of cancelled or delivered rows. Regression tests reproduce both bypasses and retain failed-attempt history; focused race tests pass.
Change-source: pulse-maintainer
72 installs delivered no notification at all in the week to 2026-09-03 and
between them burned 196,562 attempts for 62,618 dead letters. Their
categorised failures are 79% authentication (11,693), configuration
(9,554) and rejected (5,954) against 930 connectivity: deterministic
verdicts about the request, not conditions that clear. The queue retried
each of them the full ladder anyway, because the retry decision never
consulted the failure class.
Those three classes now dead-letter on the attempt that produced them.
The same payload, sent again to the same destination with the same
credentials, gets the same answer; spending two more attempts on it only
delays the dead letter the operator needs to act on. Connectivity, rate
limiting, server errors and unclassified failures keep the full ladder.
TLS deliberately stays on the retrying side. A handshake can fail
transiently during a rotation, and one wasted ladder is cheaper than
dropping a recoverable notification.
This is the same call webhook delivery already made for HTTP 4xx in
isRetryableWebhookError, now generalised to every destination type and
owned in one place. It was not safely expressible before the class became
sender-declared rather than guessed from error prose.
Nothing is lost by giving up sooner: RetryTerminalFailures still returns
retained terminal failures to the queue with a fresh budget once the
operator fixes the credentials or the configuration, and dead-letter rows
now record the failure class and whether the cause was an exhausted
ladder or a non-retryable class.
Note for telemetry reads: notification_attempts_7d will fall sharply in
the blackout cohort while notification_failures_7d is unchanged, because
the same terminal failures now cost one attempt instead of three. That is
the intended effect and not a drop in notification volume.
Terminal notification failures are 36% of resolved delivery outcomes
fleet-wide (126,337 dead-lettered against 224,692 delivered in the week
to 2026-09-03, over 6,668 clean installs), and the category breakdown
could not say why: unknown was the modal bucket at 31,218.
The class was being derived by substring-matching the Go error message.
That fails in two ways. Any failure whose text carries none of the ~50
recognised tokens falls through to unknown, which is most of what SMTP
produces: net/smtp reports the server's verdict as a reply code, and only
535 was ever matched, so a 550 relay refusal and a 451 temporary failure
both recorded as unknown. Worse, the text being matched includes the
destination's own response body, so a third party can choose the reason
code Pulse records and shows the operator - a 500 whose body contains
"rate limit" was recorded as rate_limited rather than server_error.
Senders now declare the class where they already know it, and the
classifier reads Go's own error types before it reads any prose:
*textproto.Error for SMTP reply codes, x509 and tls for certificate
failures, net.DNSError and timeouts for connectivity. HTTP status codes
set the class at the five sites that build a status error, so the
response body is preserved for the operator's audit row but can no longer
influence the classification. Prose matching remains only as the last
resort for paths that declare nothing.
SMTP 5xx is deliberately not mapped the way HTTP 5xx is: a 550 is the
destination refusing the message, not the destination breaking, so only
the transient 4xx replies count as server_error.
Registers the wider finding as a coverage gap. The 36% is concentration,
not breadth - 72 installs that delivered nothing at all in seven days
account for half of all terminal failures, and Pulse neither backs off
nor tells those operators the destination has never once succeeded.
The default attempt budget was three, and with the 1s/2s/4s backoff a
destination that was unreachable for about ten seconds had its
notifications dead-lettered permanently. A webhook receiver rebooting
alongside the infrastructure it monitors is routine, not terminal.
Eight attempts under the same doubling schedule span roughly three
minutes before dead-lettering. Dead-letter semantics are unchanged.
Refs #1721
Contract-Neutral: Refs #1721: raises the default delivery attempt budget only; dead-letter semantics, retention, and outcome vocabulary unchanged, notifications.md pins no attempt count
The delivery health verdict said when something was wrong; nothing showed
what actually fired and where it went. Expose the queue's retained
per-attempt audit rows as GET /api/notifications/delivery-log (newest
first, retention-labeled, webhook secrets redacted from error text) and
render them as a Recent delivery activity card on the alert destinations
tab, with outcome badges, destination names, and failure classes. Audit
rows now persist the normalized destination identity in a destination_id
column; older rows fall back to their operational links.
Test sends bypass both the queue and the activation gate, which is exactly
how installs came to believe delivery worked while every real alert was
suppressed (483 installs in the 08-18 telemetry read). Successful test
responses now carry deliveryPaused: true whenever the manager is gated
off, and the destinations UI warns instead of celebrating.
An alert that resolved while its firing notification was still in the
grouping window or waiting in the persistent queue (alert delay pushes
activation close to resolution; quiet-hours replay defers delivery)
produced a recovery-only notification: CancelAlert dropped the queued
firing, but LastNotified had been set optimistically at dispatch, so
the resolved-notification gate believed the firing had been sent.
CancelAlert now reports whether it cancelled a firing notification that
had not been delivered (grouping window entries and pending queue rows;
mid-send rows are excluded because their delivery may still complete),
and handleAlertResolved suppresses the recovery in that case. This also
covers the quiet-hours replay bypass: a recovery only follows a deferred
firing if the replay was actually delivered.
Addresses #1553
Security fixes:
- Auto-register now requires settings:write scope for API tokens
- X-Forwarded-For in auto-register only trusted from verified proxies
- Public URL capture requires authentication (no loopback bypass)
- Lockout reset now uses RequireAdmin for session users
Reliability fixes:
- Docker stop command expiration clears PendingUninstall flag
- Cancelled notifications get completed_at set and are cleaned up
- Move all SQLite pragmas from db.Exec() to DSN parameters so every
connection the pool creates gets busy_timeout and other settings.
Previously only the first connection had these applied.
- Set MaxOpenConns(1) on audit, RBAC, and notification databases
(metrics already had this). Fixes potential for multiple connections
where new ones lack busy_timeout.
- Increase busy_timeout from 5s to 30s across all databases to
tolerate disk I/O pressure during backup windows.
- Fix nested query deadlocks in GetRoles(), GetUserAssignments(), and
CancelByAlertIDs() that would deadlock with MaxOpenConns(1).
- Fix circuit breaker retryInterval not resetting on recovery, which
caused the next trip to start at 5-minute backoff instead of 5s.
Related to #1156
Switch from mattn/go-sqlite3 (CGO) to modernc.org/sqlite (pure Go)
for auth, audit, and notification queue storage. This enables SQLite
functionality on arm64 Docker images which are built with CGO_ENABLED=0.
Related to #1140
Fixed deadlock where CancelByAlertIDs held nq.mu.Lock() and then called
UpdateStatus() which also tried to acquire the same lock. Now uses
direct SQL while holding the lock.
Tests added for CancelByAlertIDs:
- No matching notifications (notification stays pending)
- Matching notification cancelled
- Multiple alerts with partial match (any match cancels)
Coverage: CancelByAlertIDs 65.7% -> 81.1%
Three categories of fixes:
1. Goroutine leak causing 10-minute timeout:
- Add defer mon.notificationMgr.Stop() in monitor_memory_test.go
- Background goroutines from notification manager weren't being stopped
2. Database NULL column scanning errors:
- Change LastError from string to *string in queue.go
- Change PayloadBytes from int to *int in queue.go
- SQL NULL values require pointer types in Go
3. SSRF protection blocking test servers:
- Check allowlist for localhost before rejecting in notifications.go
- Set PULSE_DATA_DIR to temp directory in tests
- Add defer nm.Stop() calls to prevent goroutine leaks
Fixes for preflight test failures in workflow run 19280879903.
Queue cancellation mechanism:
- Add CancelByAlertIDs method to mark queued notifications as cancelled when alerts resolve
- Update CancelAlert to cancel queued notifications containing resolved alert IDs
- Skip cancelled notifications in queue processor
- Prevents resolved alerts from triggering notifications after they clear
Atomic DB operations:
- Add IncrementAttemptAndSetStatus to atomically update attempt counter and status
- Replace separate IncrementAttempt + UpdateStatus calls with single atomic operation
- Prevents orphaned queue entries when crashes occur between operations
- Eliminates race condition where rows get stuck in "pending" or "sending" status
These fixes ensure queued notifications are properly cancelled when alerts resolve
and prevent database inconsistencies during crash scenarios.
Critical fixes (P0):
- Fix cooldown timing: Mark cooldown only after successful delivery, not before enqueue
- Add os.MkdirAll to queue initialization to prevent silent failures on fresh installs
- Add DNS re-validation at webhook send time to prevent DNS rebinding SSRF attacks
- Add SSRF validation for Apprise HTTP URLs
- Remove secret logging (bot tokens, routing keys) from debug logs
- Implement lastNotified cleanup to prevent unbounded memory growth
- Use shared HTTP client for webhooks to enable TLS connection reuse
- Add fallback to direct sending when queue enqueue fails
- Make queue worker concurrent (5 workers with semaphore) to prevent head-of-line blocking
- Fix webhook rate limiter race condition with separate mutex
- Fix email manager thread safety with mutex on rate limiter
- Fix grouping timer leak by adding stopCleanup signal
- Fix webhook 429 double sleep (use Retry-After OR backoff, not both)
Frontend improvements:
- Add queue/DLQ management API methods (getQueueStats, getDLQ, retryDLQItem, deleteDLQItem)
- Add getNotificationHealth and getWebhookHistory endpoints
- Add Apprise test support to NotificationTestRequest type
Related to notification system audit
Remove 4 LLM-generated internal development docs that don't belong in the repository:
- MIGRATION_SCAFFOLDING.md
- NOTIFICATION_AUDIT.md
- NOTIFICATION_QUICK_REFERENCE.md
- NOTIFICATION_SYSTEM_MAP.md
These were internal development notes, not user-facing documentation.
pulse-triage[bot]