mirror of
https://github.com/shankar0123/certctl.git
synced 2026-06-07 12:31:29 +00:00
I-005: notification retry loop + dead-letter queue
Critical alerts can no longer be silently dropped by a transient
notifier failure. Failed notification attempts now ride an exponential
backoff retry loop, with a 5-attempt budget before promotion to the
dead-letter queue for operator intervention.
Schema (migration 000016, idempotent):
- retry_count INTEGER NOT NULL DEFAULT 0
- next_retry_at TIMESTAMPTZ
- last_error TEXT
- idx_notification_events_retry_sweep partial index
(next_retry_at) WHERE status='failed' AND next_retry_at IS NOT NULL
Dead rows clear next_retry_at so the index stops matching them.
Service contract:
- NotificationService.RetryFailedNotifications drives 2^n-minute
exponential backoff capped at 1h (notifRetryBackoffCap) with
5-attempt budget (notifRetryMaxAttempts).
- Exhaustion (RetryCount >= notifRetryMaxAttempts-1) promotes to
status='dead' via MarkAsDead.
- Non-terminal failures record via RecordFailedAttempt.
- Success path promotes to 'sent' without touching retry_count
(audit preserves "delivered on attempt N").
- Missing-notifier branch defensively promotes to 'sent' to avoid
wedging a row on a deleted channel.
- RequeueNotification operator escape hatch atomically resets
retry_count -> 0, next_retry_at -> NULL, last_error -> NULL,
status -> pending via notifRepo.Requeue.
Scheduler:
- New always-on notificationRetryLoop wired into the base loop set at
CERTCTL_NOTIFICATION_RETRY_INTERVAL (default 2m).
- sync/atomic.Bool idempotency guard.
- sync.WaitGroup shutdown drain via WaitForCompletion.
StatsService:
- SetNotifRepo setter pattern preserves 9 pre-existing
NewStatsService call sites (main.go + stats_test.go + 8 digest
tests) without touching the constructor signature.
- DashboardSummary.NotificationsDead populated via
notifRepo.CountByStatus(ctx, "dead") — nil-safe when unwired
(reports zero on systems without a notification repository).
- CountByStatus error is non-fatal (dashboard summary is
best-effort for this field).
- Prometheus certctl_notification_dead_total counter emitted from
the same snapshot.
Handler:
- New POST /api/v1/notifications/{id}/requeue endpoint.
- dead status surfaces to MCP + CLI.
Frontend:
- NotificationsPage gains two-tab toolbar ("All" / "Dead letter")
with queryKey: ['notifications', activeTab] so switching tabs
doesn't serve stale data until the 30s refetch.
- Dead rows surface "Retry {n}/5" + truncated last_error with
full-text title tooltip.
- Requeue mutation wrapped as
mutationFn: (id: string) => requeueNotification(id)
to prevent react-query v5's positional context argument from
leaking into the API client — pinned against future refactors
by strict-match toHaveBeenCalledWith('notif-dead-001') in
NotificationsPage.test.tsx:181.
Closes I-005.
This commit is contained in:
@@ -2037,6 +2037,16 @@ paths:
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/page"
|
||||
- $ref: "#/components/parameters/per_page"
|
||||
- name: status
|
||||
in: query
|
||||
required: false
|
||||
description: |
|
||||
Filter by lifecycle status. I-005: `dead` powers the Dead letter
|
||||
tab on the GUI; empty/omitted returns the default all-statuses
|
||||
listing to preserve pre-I-005 behavior.
|
||||
schema:
|
||||
type: string
|
||||
enum: [pending, sent, failed, dead, read]
|
||||
responses:
|
||||
"200":
|
||||
description: Paginated list of notifications
|
||||
@@ -2094,6 +2104,36 @@ paths:
|
||||
"500":
|
||||
$ref: "#/components/responses/InternalError"
|
||||
|
||||
/api/v1/notifications/{id}/requeue:
|
||||
post:
|
||||
tags: [Notifications]
|
||||
summary: Requeue a dead notification
|
||||
description: |
|
||||
I-005: flip a notification from the `dead` dead-letter queue back to
|
||||
`pending` so the retry sweep (default 2 minutes) picks it up on its
|
||||
next tick. Used by operators after fixing the underlying delivery
|
||||
failure (SMTP config, webhook endpoint, etc.). Clears `next_retry_at`
|
||||
and resets the `retry_count` budget; `last_error` is preserved for
|
||||
audit continuity.
|
||||
operationId: requeueNotification
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/resourceId"
|
||||
responses:
|
||||
"200":
|
||||
description: Requeued
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/StatusResponse"
|
||||
"400":
|
||||
$ref: "#/components/responses/BadRequest"
|
||||
"404":
|
||||
$ref: "#/components/responses/NotFound"
|
||||
"405":
|
||||
description: Method not allowed (POST only)
|
||||
"500":
|
||||
$ref: "#/components/responses/InternalError"
|
||||
|
||||
# ─── Stats ───────────────────────────────────────────────────────────
|
||||
/api/v1/stats/summary:
|
||||
get:
|
||||
@@ -3905,8 +3945,32 @@ components:
|
||||
format: date-time
|
||||
status:
|
||||
type: string
|
||||
enum: [pending, sent, failed, dead, read]
|
||||
description: |
|
||||
Notification lifecycle status. I-005 adds `dead` for notifications
|
||||
that exhausted their 5-attempt retry budget and were moved to the
|
||||
dead-letter queue; operators triage these in the GUI's Dead letter
|
||||
tab and use POST /notifications/{id}/requeue to resurrect them.
|
||||
error:
|
||||
type: string
|
||||
retry_count:
|
||||
type: integer
|
||||
description: |
|
||||
Number of delivery attempts made. I-005 retry-sweep field; caps
|
||||
at max_attempts=5 before the notification transitions to `dead`.
|
||||
next_retry_at:
|
||||
type: string
|
||||
format: date-time
|
||||
description: |
|
||||
When the next retry attempt is scheduled. I-005 retry-sweep field;
|
||||
null for `sent`, `dead`, and `read` statuses. Backoff follows
|
||||
`min(2^retry_count * 1m, 1h)`.
|
||||
last_error:
|
||||
type: string
|
||||
description: |
|
||||
Most recent transient delivery error (SMTP failure, webhook 5xx,
|
||||
etc.). I-005 retry-sweep field; surfaced on the Dead letter tab
|
||||
so operators can triage without chasing server logs.
|
||||
created_at:
|
||||
type: string
|
||||
format: date-time
|
||||
|
||||
Reference in New Issue
Block a user