I-005: notification retry loop + dead-letter queue

Critical alerts can no longer be silently dropped by a transient
notifier failure. Failed notification attempts now ride an exponential
backoff retry loop, with a 5-attempt budget before promotion to the
dead-letter queue for operator intervention.

Schema (migration 000016, idempotent):
- retry_count INTEGER NOT NULL DEFAULT 0
- next_retry_at TIMESTAMPTZ
- last_error TEXT
- idx_notification_events_retry_sweep partial index
  (next_retry_at) WHERE status='failed' AND next_retry_at IS NOT NULL
  Dead rows clear next_retry_at so the index stops matching them.

Service contract:
- NotificationService.RetryFailedNotifications drives 2^n-minute
  exponential backoff capped at 1h (notifRetryBackoffCap) with
  5-attempt budget (notifRetryMaxAttempts).
- Exhaustion (RetryCount >= notifRetryMaxAttempts-1) promotes to
  status='dead' via MarkAsDead.
- Non-terminal failures record via RecordFailedAttempt.
- Success path promotes to 'sent' without touching retry_count
  (audit preserves "delivered on attempt N").
- Missing-notifier branch defensively promotes to 'sent' to avoid
  wedging a row on a deleted channel.
- RequeueNotification operator escape hatch atomically resets
  retry_count -> 0, next_retry_at -> NULL, last_error -> NULL,
  status -> pending via notifRepo.Requeue.

Scheduler:
- New always-on notificationRetryLoop wired into the base loop set at
  CERTCTL_NOTIFICATION_RETRY_INTERVAL (default 2m).
- sync/atomic.Bool idempotency guard.
- sync.WaitGroup shutdown drain via WaitForCompletion.

StatsService:
- SetNotifRepo setter pattern preserves 9 pre-existing
  NewStatsService call sites (main.go + stats_test.go + 8 digest
  tests) without touching the constructor signature.
- DashboardSummary.NotificationsDead populated via
  notifRepo.CountByStatus(ctx, "dead") — nil-safe when unwired
  (reports zero on systems without a notification repository).
- CountByStatus error is non-fatal (dashboard summary is
  best-effort for this field).
- Prometheus certctl_notification_dead_total counter emitted from
  the same snapshot.

Handler:
- New POST /api/v1/notifications/{id}/requeue endpoint.
- dead status surfaces to MCP + CLI.

Frontend:
- NotificationsPage gains two-tab toolbar ("All" / "Dead letter")
  with queryKey: ['notifications', activeTab] so switching tabs
  doesn't serve stale data until the 30s refetch.
- Dead rows surface "Retry {n}/5" + truncated last_error with
  full-text title tooltip.
- Requeue mutation wrapped as
    mutationFn: (id: string) => requeueNotification(id)
  to prevent react-query v5's positional context argument from
  leaking into the API client — pinned against future refactors
  by strict-match toHaveBeenCalledWith('notif-dead-001') in
  NotificationsPage.test.tsx:181.

Closes I-005.
This commit is contained in:
shankar0123
2026-04-19 15:17:27 +00:00
parent 707d8de4fb
commit 675b87ba63
33 changed files with 3758 additions and 228 deletions
+26 -22
View File
@@ -45,28 +45,28 @@ func (r *Router) RegisterFunc(pattern string, handler func(http.ResponseWriter,
// HandlerRegistry groups all API handler dependencies for router registration.
type HandlerRegistry struct {
Certificates handler.CertificateHandler
Issuers handler.IssuerHandler
Targets handler.TargetHandler
Agents handler.AgentHandler
Jobs handler.JobHandler
Policies handler.PolicyHandler
Profiles handler.ProfileHandler
Teams handler.TeamHandler
Owners handler.OwnerHandler
AgentGroups handler.AgentGroupHandler
Audit handler.AuditHandler
Notifications handler.NotificationHandler
Stats handler.StatsHandler
Metrics handler.MetricsHandler
Health handler.HealthHandler
Discovery handler.DiscoveryHandler
NetworkScan handler.NetworkScanHandler
Verification handler.VerificationHandler
Export handler.ExportHandler
Digest handler.DigestHandler
HealthChecks *handler.HealthCheckHandler
BulkRevocation handler.BulkRevocationHandler
Certificates handler.CertificateHandler
Issuers handler.IssuerHandler
Targets handler.TargetHandler
Agents handler.AgentHandler
Jobs handler.JobHandler
Policies handler.PolicyHandler
Profiles handler.ProfileHandler
Teams handler.TeamHandler
Owners handler.OwnerHandler
AgentGroups handler.AgentGroupHandler
Audit handler.AuditHandler
Notifications handler.NotificationHandler
Stats handler.StatsHandler
Metrics handler.MetricsHandler
Health handler.HealthHandler
Discovery handler.DiscoveryHandler
NetworkScan handler.NetworkScanHandler
Verification handler.VerificationHandler
Export handler.ExportHandler
Digest handler.DigestHandler
HealthChecks *handler.HealthCheckHandler
BulkRevocation handler.BulkRevocationHandler
}
// RegisterHandlers sets up all API routes with their handlers.
@@ -204,6 +204,10 @@ func (r *Router) RegisterHandlers(reg HandlerRegistry) {
r.Register("GET /api/v1/notifications", http.HandlerFunc(reg.Notifications.ListNotifications))
r.Register("GET /api/v1/notifications/{id}", http.HandlerFunc(reg.Notifications.GetNotification))
r.Register("POST /api/v1/notifications/{id}/read", http.HandlerFunc(reg.Notifications.MarkAsRead))
// I-005: requeue a dead notification back to pending so the retry sweep
// picks it up again. Go 1.22 ServeMux resolves the literal /requeue segment
// before falling back to the {id} path-variable route above.
r.Register("POST /api/v1/notifications/{id}/requeue", http.HandlerFunc(reg.Notifications.RequeueNotification))
// Stats routes: /api/v1/stats
r.Register("GET /api/v1/stats/summary", http.HandlerFunc(reg.Stats.GetDashboardSummary))