I-005: notification retry loop + dead-letter queue

Critical alerts can no longer be silently dropped by a transient
notifier failure. Failed notification attempts now ride an exponential
backoff retry loop, with a 5-attempt budget before promotion to the
dead-letter queue for operator intervention.

Schema (migration 000016, idempotent):
- retry_count INTEGER NOT NULL DEFAULT 0
- next_retry_at TIMESTAMPTZ
- last_error TEXT
- idx_notification_events_retry_sweep partial index
  (next_retry_at) WHERE status='failed' AND next_retry_at IS NOT NULL
  Dead rows clear next_retry_at so the index stops matching them.

Service contract:
- NotificationService.RetryFailedNotifications drives 2^n-minute
  exponential backoff capped at 1h (notifRetryBackoffCap) with
  5-attempt budget (notifRetryMaxAttempts).
- Exhaustion (RetryCount >= notifRetryMaxAttempts-1) promotes to
  status='dead' via MarkAsDead.
- Non-terminal failures record via RecordFailedAttempt.
- Success path promotes to 'sent' without touching retry_count
  (audit preserves "delivered on attempt N").
- Missing-notifier branch defensively promotes to 'sent' to avoid
  wedging a row on a deleted channel.
- RequeueNotification operator escape hatch atomically resets
  retry_count -> 0, next_retry_at -> NULL, last_error -> NULL,
  status -> pending via notifRepo.Requeue.

Scheduler:
- New always-on notificationRetryLoop wired into the base loop set at
  CERTCTL_NOTIFICATION_RETRY_INTERVAL (default 2m).
- sync/atomic.Bool idempotency guard.
- sync.WaitGroup shutdown drain via WaitForCompletion.

StatsService:
- SetNotifRepo setter pattern preserves 9 pre-existing
  NewStatsService call sites (main.go + stats_test.go + 8 digest
  tests) without touching the constructor signature.
- DashboardSummary.NotificationsDead populated via
  notifRepo.CountByStatus(ctx, "dead") — nil-safe when unwired
  (reports zero on systems without a notification repository).
- CountByStatus error is non-fatal (dashboard summary is
  best-effort for this field).
- Prometheus certctl_notification_dead_total counter emitted from
  the same snapshot.

Handler:
- New POST /api/v1/notifications/{id}/requeue endpoint.
- dead status surfaces to MCP + CLI.

Frontend:
- NotificationsPage gains two-tab toolbar ("All" / "Dead letter")
  with queryKey: ['notifications', activeTab] so switching tabs
  doesn't serve stale data until the 30s refetch.
- Dead rows surface "Retry {n}/5" + truncated last_error with
  full-text title tooltip.
- Requeue mutation wrapped as
    mutationFn: (id: string) => requeueNotification(id)
  to prevent react-query v5's positional context argument from
  leaking into the API client — pinned against future refactors
  by strict-match toHaveBeenCalledWith('notif-dead-001') in
  NotificationsPage.test.tsx:181.

Closes I-005.
This commit is contained in:
Shankar
2026-04-19 15:17:27 +00:00
parent 6836286c37
commit 15daf008aa
33 changed files with 3758 additions and 228 deletions
+64
View File
@@ -2037,6 +2037,16 @@ paths:
parameters:
- $ref: "#/components/parameters/page"
- $ref: "#/components/parameters/per_page"
- name: status
in: query
required: false
description: |
Filter by lifecycle status. I-005: `dead` powers the Dead letter
tab on the GUI; empty/omitted returns the default all-statuses
listing to preserve pre-I-005 behavior.
schema:
type: string
enum: [pending, sent, failed, dead, read]
responses:
"200":
description: Paginated list of notifications
@@ -2094,6 +2104,36 @@ paths:
"500":
$ref: "#/components/responses/InternalError"
/api/v1/notifications/{id}/requeue:
post:
tags: [Notifications]
summary: Requeue a dead notification
description: |
I-005: flip a notification from the `dead` dead-letter queue back to
`pending` so the retry sweep (default 2 minutes) picks it up on its
next tick. Used by operators after fixing the underlying delivery
failure (SMTP config, webhook endpoint, etc.). Clears `next_retry_at`
and resets the `retry_count` budget; `last_error` is preserved for
audit continuity.
operationId: requeueNotification
parameters:
- $ref: "#/components/parameters/resourceId"
responses:
"200":
description: Requeued
content:
application/json:
schema:
$ref: "#/components/schemas/StatusResponse"
"400":
$ref: "#/components/responses/BadRequest"
"404":
$ref: "#/components/responses/NotFound"
"405":
description: Method not allowed (POST only)
"500":
$ref: "#/components/responses/InternalError"
# ─── Stats ───────────────────────────────────────────────────────────
/api/v1/stats/summary:
get:
@@ -3905,8 +3945,32 @@ components:
format: date-time
status:
type: string
enum: [pending, sent, failed, dead, read]
description: |
Notification lifecycle status. I-005 adds `dead` for notifications
that exhausted their 5-attempt retry budget and were moved to the
dead-letter queue; operators triage these in the GUI's Dead letter
tab and use POST /notifications/{id}/requeue to resurrect them.
error:
type: string
retry_count:
type: integer
description: |
Number of delivery attempts made. I-005 retry-sweep field; caps
at max_attempts=5 before the notification transitions to `dead`.
next_retry_at:
type: string
format: date-time
description: |
When the next retry attempt is scheduled. I-005 retry-sweep field;
null for `sent`, `dead`, and `read` statuses. Backoff follows
`min(2^retry_count * 1m, 1h)`.
last_error:
type: string
description: |
Most recent transient delivery error (SMTP failure, webhook 5xx,
etc.). I-005 retry-sweep field; surfaced on the Dead letter tab
so operators can triage without chasing server logs.
created_at:
type: string
format: date-time