mirror of
https://github.com/shankar0123/certctl.git
synced 2026-06-11 09:29:03 +00:00
I-005: notification retry loop + dead-letter queue
Critical alerts can no longer be silently dropped by a transient
notifier failure. Failed notification attempts now ride an exponential
backoff retry loop, with a 5-attempt budget before promotion to the
dead-letter queue for operator intervention.
Schema (migration 000016, idempotent):
- retry_count INTEGER NOT NULL DEFAULT 0
- next_retry_at TIMESTAMPTZ
- last_error TEXT
- idx_notification_events_retry_sweep partial index
(next_retry_at) WHERE status='failed' AND next_retry_at IS NOT NULL
Dead rows clear next_retry_at so the index stops matching them.
Service contract:
- NotificationService.RetryFailedNotifications drives 2^n-minute
exponential backoff capped at 1h (notifRetryBackoffCap) with
5-attempt budget (notifRetryMaxAttempts).
- Exhaustion (RetryCount >= notifRetryMaxAttempts-1) promotes to
status='dead' via MarkAsDead.
- Non-terminal failures record via RecordFailedAttempt.
- Success path promotes to 'sent' without touching retry_count
(audit preserves "delivered on attempt N").
- Missing-notifier branch defensively promotes to 'sent' to avoid
wedging a row on a deleted channel.
- RequeueNotification operator escape hatch atomically resets
retry_count -> 0, next_retry_at -> NULL, last_error -> NULL,
status -> pending via notifRepo.Requeue.
Scheduler:
- New always-on notificationRetryLoop wired into the base loop set at
CERTCTL_NOTIFICATION_RETRY_INTERVAL (default 2m).
- sync/atomic.Bool idempotency guard.
- sync.WaitGroup shutdown drain via WaitForCompletion.
StatsService:
- SetNotifRepo setter pattern preserves 9 pre-existing
NewStatsService call sites (main.go + stats_test.go + 8 digest
tests) without touching the constructor signature.
- DashboardSummary.NotificationsDead populated via
notifRepo.CountByStatus(ctx, "dead") — nil-safe when unwired
(reports zero on systems without a notification repository).
- CountByStatus error is non-fatal (dashboard summary is
best-effort for this field).
- Prometheus certctl_notification_dead_total counter emitted from
the same snapshot.
Handler:
- New POST /api/v1/notifications/{id}/requeue endpoint.
- dead status surfaces to MCP + CLI.
Frontend:
- NotificationsPage gains two-tab toolbar ("All" / "Dead letter")
with queryKey: ['notifications', activeTab] so switching tabs
doesn't serve stale data until the 30s refetch.
- Dead rows surface "Retry {n}/5" + truncated last_error with
full-text title tooltip.
- Requeue mutation wrapped as
mutationFn: (id: string) => requeueNotification(id)
to prevent react-query v5's positional context argument from
leaking into the API client — pinned against future refactors
by strict-match toHaveBeenCalledWith('notif-dead-001') in
NotificationsPage.test.tsx:181.
Closes I-005.
This commit is contained in:
@@ -285,6 +285,12 @@ type AuditRepository interface {
|
||||
}
|
||||
|
||||
// NotificationRepository defines operations for managing notifications.
|
||||
//
|
||||
// I-005 extends the interface with four retry/DLQ methods. The retry scheduler
|
||||
// loop calls ListRetryEligible on every tick to pull overdue failed rows, then
|
||||
// either RecordFailedAttempt (still-retrying) or MarkAsDead (exhausted). The
|
||||
// operator-facing dead-letter tab calls Requeue to move a row from 'dead' (or
|
||||
// 'failed') back to 'pending' so ProcessPendingNotifications picks it up again.
|
||||
type NotificationRepository interface {
|
||||
// Create stores a new notification.
|
||||
Create(ctx context.Context, notif *domain.NotificationEvent) error
|
||||
@@ -292,6 +298,44 @@ type NotificationRepository interface {
|
||||
List(ctx context.Context, filter *NotificationFilter) ([]*domain.NotificationEvent, error)
|
||||
// UpdateStatus updates a notification's delivery status.
|
||||
UpdateStatus(ctx context.Context, id string, status string, sentAt time.Time) error
|
||||
// ListRetryEligible returns failed notification rows whose next_retry_at
|
||||
// is <= now AND retry_count < maxAttempts, ordered by next_retry_at ASC
|
||||
// (oldest overdue first — same fairness as I-001's RetryFailedJobs). The
|
||||
// WHERE clause mirrors the partial retry-sweep index predicate from
|
||||
// migration 000016 so the planner uses it. A limit<=0 is normalised to
|
||||
// a sane default in the repo implementation to avoid accidental unbounded
|
||||
// sweeps. I-005 coverage-gap closure.
|
||||
ListRetryEligible(ctx context.Context, now time.Time, maxAttempts, limit int) ([]*domain.NotificationEvent, error)
|
||||
// RecordFailedAttempt is called by the retry sweep after a notifier.Send
|
||||
// transient failure. The UPDATE increments retry_count by exactly 1,
|
||||
// overwrites last_error, overwrites next_retry_at, and KEEPS status='failed'
|
||||
// so the row remains a candidate for ListRetryEligible on the next sweep.
|
||||
// Returns "not found" when no row matches the id (mirrors UpdateStatus).
|
||||
// I-005 coverage-gap closure.
|
||||
RecordFailedAttempt(ctx context.Context, id string, lastError string, nextRetryAt time.Time) error
|
||||
// MarkAsDead performs the DLQ transition when retry_count reaches
|
||||
// max_attempts. Flips status='dead', clears next_retry_at so the partial
|
||||
// retry-sweep index drops the row, writes the final last_error, and
|
||||
// PRESERVES retry_count as historical evidence of how many attempts were
|
||||
// burned. Returns "not found" when no row matches.
|
||||
// I-005 coverage-gap closure.
|
||||
MarkAsDead(ctx context.Context, id string, lastError string) error
|
||||
// Requeue is the operator "try again" action from the UI's Dead letter
|
||||
// tab. Flips status='pending' (so ProcessPendingNotifications picks it
|
||||
// up), resets retry_count to 0 (otherwise the operator's first retry
|
||||
// would already be at hour-long waits), clears next_retry_at, and clears
|
||||
// last_error. Valid from both 'dead' and 'failed'. Returns "not found"
|
||||
// when no row matches. I-005 coverage-gap closure.
|
||||
Requeue(ctx context.Context, id string) error
|
||||
// CountByStatus returns the number of notification_events rows whose
|
||||
// status column matches the given string exactly. Used by StatsService
|
||||
// to populate DashboardSummary.NotificationsDead which in turn drives
|
||||
// the Prometheus counter certctl_notification_dead_total (I-005 Phase 2
|
||||
// observability gate). A dedicated SQL COUNT(*) is used instead of
|
||||
// List(filter{Status: ...}) because List silently resets PerPage>500 to
|
||||
// 50 — a latent scale bug for any status-filtered count. I-005
|
||||
// coverage-gap closure.
|
||||
CountByStatus(ctx context.Context, status string) (int64, error)
|
||||
}
|
||||
|
||||
// TeamRepository defines operations for managing teams.
|
||||
|
||||
Reference in New Issue
Block a user