mirror of
https://github.com/shankar0123/certctl.git
synced 2026-06-07 17:41:29 +00:00
shankar0123
675b87ba63
Critical alerts can no longer be silently dropped by a transient
notifier failure. Failed notification attempts now ride an exponential
backoff retry loop, with a 5-attempt budget before promotion to the
dead-letter queue for operator intervention.
Schema (migration 000016, idempotent):
- retry_count INTEGER NOT NULL DEFAULT 0
- next_retry_at TIMESTAMPTZ
- last_error TEXT
- idx_notification_events_retry_sweep partial index
(next_retry_at) WHERE status='failed' AND next_retry_at IS NOT NULL
Dead rows clear next_retry_at so the index stops matching them.
Service contract:
- NotificationService.RetryFailedNotifications drives 2^n-minute
exponential backoff capped at 1h (notifRetryBackoffCap) with
5-attempt budget (notifRetryMaxAttempts).
- Exhaustion (RetryCount >= notifRetryMaxAttempts-1) promotes to
status='dead' via MarkAsDead.
- Non-terminal failures record via RecordFailedAttempt.
- Success path promotes to 'sent' without touching retry_count
(audit preserves "delivered on attempt N").
- Missing-notifier branch defensively promotes to 'sent' to avoid
wedging a row on a deleted channel.
- RequeueNotification operator escape hatch atomically resets
retry_count -> 0, next_retry_at -> NULL, last_error -> NULL,
status -> pending via notifRepo.Requeue.
Scheduler:
- New always-on notificationRetryLoop wired into the base loop set at
CERTCTL_NOTIFICATION_RETRY_INTERVAL (default 2m).
- sync/atomic.Bool idempotency guard.
- sync.WaitGroup shutdown drain via WaitForCompletion.
StatsService:
- SetNotifRepo setter pattern preserves 9 pre-existing
NewStatsService call sites (main.go + stats_test.go + 8 digest
tests) without touching the constructor signature.
- DashboardSummary.NotificationsDead populated via
notifRepo.CountByStatus(ctx, "dead") — nil-safe when unwired
(reports zero on systems without a notification repository).
- CountByStatus error is non-fatal (dashboard summary is
best-effort for this field).
- Prometheus certctl_notification_dead_total counter emitted from
the same snapshot.
Handler:
- New POST /api/v1/notifications/{id}/requeue endpoint.
- dead status surfaces to MCP + CLI.
Frontend:
- NotificationsPage gains two-tab toolbar ("All" / "Dead letter")
with queryKey: ['notifications', activeTab] so switching tabs
doesn't serve stale data until the 30s refetch.
- Dead rows surface "Retry {n}/5" + truncated last_error with
full-text title tooltip.
- Requeue mutation wrapped as
mutationFn: (id: string) => requeueNotification(id)
to prevent react-query v5's positional context argument from
leaking into the API client — pinned against future refactors
by strict-match toHaveBeenCalledWith('notif-dead-001') in
NotificationsPage.test.tsx:181.
Closes I-005.
82 lines
3.7 KiB
Go
82 lines
3.7 KiB
Go
package domain
|
|
|
|
import (
|
|
"time"
|
|
)
|
|
|
|
// NotificationEvent records a notification sent to users about certificate events.
|
|
//
|
|
// I-005 extends the event with a retry counter, a nullable next-retry timestamp
|
|
// that drives the retry-sweep partial index, and a nullable last-error string
|
|
// preserving the most recent transient failure so operators triaging the dead
|
|
// letter queue can see *why* a notification died without chasing server logs.
|
|
// Status stays a plain `string` (not retyped to NotificationStatus) because the
|
|
// repo layer materialises it directly from PostgreSQL's VARCHAR column and the
|
|
// service layer compares against the NotificationStatus* constants via
|
|
// `string(...)` casts at call sites — see service.RetryFailedNotifications.
|
|
type NotificationEvent struct {
|
|
ID string `json:"id"`
|
|
Type NotificationType `json:"type"`
|
|
CertificateID *string `json:"certificate_id,omitempty"`
|
|
Channel NotificationChannel `json:"channel"`
|
|
Recipient string `json:"recipient"`
|
|
Message string `json:"message"`
|
|
SentAt *time.Time `json:"sent_at,omitempty"`
|
|
Status string `json:"status"`
|
|
Error *string `json:"error,omitempty"`
|
|
RetryCount int `json:"retry_count"`
|
|
NextRetryAt *time.Time `json:"next_retry_at,omitempty"`
|
|
LastError *string `json:"last_error,omitempty"`
|
|
CreatedAt time.Time `json:"created_at"`
|
|
}
|
|
|
|
// NotificationStatus is the typed string alias for the lifecycle status of a
|
|
// NotificationEvent. It mirrors the VARCHAR(50) column on notification_events
|
|
// and the status values used by the I-005 retry/DLQ machinery.
|
|
//
|
|
// Status transitions:
|
|
//
|
|
// pending → sent (delivery succeeded)
|
|
// pending → failed → pending (transient failure, re-armed by retry sweep)
|
|
// pending → failed → dead (retry_count reached max_attempts; DLQ)
|
|
// pending → read (operator acknowledged, no delivery needed)
|
|
//
|
|
// Values are lowercase to match the pre-I-005 on-wire representation used by
|
|
// existing UpdateStatus calls and the seed_demo.sql fixtures; retyping
|
|
// NotificationEvent.Status to NotificationStatus would be a breaking DB scan
|
|
// change, so the type is kept additive and consumed via `string(const)` casts.
|
|
type NotificationStatus string
|
|
|
|
const (
|
|
NotificationStatusPending NotificationStatus = "pending"
|
|
NotificationStatusSent NotificationStatus = "sent"
|
|
NotificationStatusFailed NotificationStatus = "failed"
|
|
NotificationStatusDead NotificationStatus = "dead"
|
|
NotificationStatusRead NotificationStatus = "read"
|
|
)
|
|
|
|
// NotificationType represents the event that triggered a notification.
|
|
type NotificationType string
|
|
|
|
const (
|
|
NotificationTypeExpirationWarning NotificationType = "ExpirationWarning"
|
|
NotificationTypeRenewalSuccess NotificationType = "RenewalSuccess"
|
|
NotificationTypeRenewalFailure NotificationType = "RenewalFailure"
|
|
NotificationTypeDeploymentSuccess NotificationType = "DeploymentSuccess"
|
|
NotificationTypeDeploymentFailure NotificationType = "DeploymentFailure"
|
|
NotificationTypePolicyViolation NotificationType = "PolicyViolation"
|
|
NotificationTypeRevocation NotificationType = "Revocation"
|
|
)
|
|
|
|
// NotificationChannel represents the communication medium for a notification.
|
|
type NotificationChannel string
|
|
|
|
const (
|
|
NotificationChannelEmail NotificationChannel = "Email"
|
|
NotificationChannelWebhook NotificationChannel = "Webhook"
|
|
NotificationChannelSlack NotificationChannel = "Slack"
|
|
NotificationChannelTeams NotificationChannel = "Teams"
|
|
NotificationChannelPagerDuty NotificationChannel = "PagerDuty"
|
|
NotificationChannelOpsGenie NotificationChannel = "OpsGenie"
|
|
)
|