I-005: notification retry loop + dead-letter queue

Critical alerts can no longer be silently dropped by a transient
notifier failure. Failed notification attempts now ride an exponential
backoff retry loop, with a 5-attempt budget before promotion to the
dead-letter queue for operator intervention.

Schema (migration 000016, idempotent):
- retry_count INTEGER NOT NULL DEFAULT 0
- next_retry_at TIMESTAMPTZ
- last_error TEXT
- idx_notification_events_retry_sweep partial index
  (next_retry_at) WHERE status='failed' AND next_retry_at IS NOT NULL
  Dead rows clear next_retry_at so the index stops matching them.

Service contract:
- NotificationService.RetryFailedNotifications drives 2^n-minute
  exponential backoff capped at 1h (notifRetryBackoffCap) with
  5-attempt budget (notifRetryMaxAttempts).
- Exhaustion (RetryCount >= notifRetryMaxAttempts-1) promotes to
  status='dead' via MarkAsDead.
- Non-terminal failures record via RecordFailedAttempt.
- Success path promotes to 'sent' without touching retry_count
  (audit preserves "delivered on attempt N").
- Missing-notifier branch defensively promotes to 'sent' to avoid
  wedging a row on a deleted channel.
- RequeueNotification operator escape hatch atomically resets
  retry_count -> 0, next_retry_at -> NULL, last_error -> NULL,
  status -> pending via notifRepo.Requeue.

Scheduler:
- New always-on notificationRetryLoop wired into the base loop set at
  CERTCTL_NOTIFICATION_RETRY_INTERVAL (default 2m).
- sync/atomic.Bool idempotency guard.
- sync.WaitGroup shutdown drain via WaitForCompletion.

StatsService:
- SetNotifRepo setter pattern preserves 9 pre-existing
  NewStatsService call sites (main.go + stats_test.go + 8 digest
  tests) without touching the constructor signature.
- DashboardSummary.NotificationsDead populated via
  notifRepo.CountByStatus(ctx, "dead") — nil-safe when unwired
  (reports zero on systems without a notification repository).
- CountByStatus error is non-fatal (dashboard summary is
  best-effort for this field).
- Prometheus certctl_notification_dead_total counter emitted from
  the same snapshot.

Handler:
- New POST /api/v1/notifications/{id}/requeue endpoint.
- dead status surfaces to MCP + CLI.

Frontend:
- NotificationsPage gains two-tab toolbar ("All" / "Dead letter")
  with queryKey: ['notifications', activeTab] so switching tabs
  doesn't serve stale data until the 30s refetch.
- Dead rows surface "Retry {n}/5" + truncated last_error with
  full-text title tooltip.
- Requeue mutation wrapped as
    mutationFn: (id: string) => requeueNotification(id)
  to prevent react-query v5's positional context argument from
  leaking into the API client — pinned against future refactors
  by strict-match toHaveBeenCalledWith('notif-dead-001') in
  NotificationsPage.test.tsx:181.

Closes I-005.

web/src/api/client.ts

@@ -301,6 +301,19 @@ export const getNotification = (id: string) =>
 export const markNotificationRead = (id: string) =>
   fetchJSON<{ message: string }>(`${BASE}/notifications/${id}/read`, { method: 'POST' });
 
+/**
+ * I-005: requeue a dead notification back to the retry queue. Flips status
+ * 'dead' → 'pending' and clears next_retry_at so the retry sweep picks it up
+ * on its next tick (default 2 minutes, CERTCTL_NOTIFICATION_RETRY_INTERVAL).
+ * Used by the Dead letter tab's "Requeue" button after an operator fixes the
+ * underlying delivery failure (SMTP config, webhook endpoint, etc.). The
+ * handler returns a StatusResponse ({ status: "requeued" }) — the frontend
+ * only needs to know the call succeeded so the mutation can invalidate the
+ * notifications query.
+ */
+export const requeueNotification = (id: string) =>
+  fetchJSON<{ status: string }>(`${BASE}/notifications/${id}/requeue`, { method: 'POST' });
+
 // Audit
 export const getAuditEvents = (params: Record<string, string> = {}) => {
   const qs = new URLSearchParams({ page: '1', per_page: '200', ...params }).toString();

web/src/api/types.ts

@@ -126,15 +126,47 @@ export interface Job {
   verification_error?: string;
 }
 
+/**
+ * Notification mirrors internal/domain/notification.go#NotificationEvent.
+ *
+ * I-005 (Notification Retry + Dead-letter Queue) widens the shape with three
+ * audit fields:
+ *
+ *   - retry_count   — number of delivery attempts already consumed (0..5). The
+ *                     5-cap is enforced server-side by NotificationsMaxAttempts.
+ *   - next_retry_at — RFC3339 timestamp the retry sweep will next consider this
+ *                     notification. Null for sent/dead/read and between sweeps
+ *                     for pending rows; the sweep populates it on each failure
+ *                     using min(2^retry_count * 1m, 1h).
+ *   - last_error    — most recent transient delivery failure. Preserved across
+ *                     requeue so Dead letter triage shows *why* the row died
+ *                     without chasing server logs.
+ *
+ * `sent_at` and `error` are the pre-I-005 audit fields on the backend struct.
+ * `subject` is a historical frontend-only field the backend never emits; it's
+ * kept optional so legacy fixtures and the pendingNotif test mock still type
+ * correctly without forcing a rewrite of every existing consumer.
+ *
+ * Status values follow the backend NotificationStatus constants:
+ *   pending · sent · failed · dead · read
+ * The existing list view tolerates the legacy title-cased "Pending" alias at
+ * render time (NotificationRow) so upgraded clients talking to older servers
+ * don't regress — see `isUnread` logic in NotificationsPage.tsx.
+ */
 export interface Notification {
   id: string;
   type: string;
   channel: string;
   recipient: string;
-  subject: string;
+  subject?: string;
   message: string;
   status: string;
-  certificate_id: string;
+  certificate_id?: string;
+  sent_at?: string | null;
+  error?: string | null;
+  retry_count?: number;
+  next_retry_at?: string | null;
+  last_error?: string | null;
   created_at: string;
 }