mirror of
https://github.com/shankar0123/certctl.git
synced 2026-06-12 16:08:51 +00:00
I-005: notification retry loop + dead-letter queue
Critical alerts can no longer be silently dropped by a transient
notifier failure. Failed notification attempts now ride an exponential
backoff retry loop, with a 5-attempt budget before promotion to the
dead-letter queue for operator intervention.
Schema (migration 000016, idempotent):
- retry_count INTEGER NOT NULL DEFAULT 0
- next_retry_at TIMESTAMPTZ
- last_error TEXT
- idx_notification_events_retry_sweep partial index
(next_retry_at) WHERE status='failed' AND next_retry_at IS NOT NULL
Dead rows clear next_retry_at so the index stops matching them.
Service contract:
- NotificationService.RetryFailedNotifications drives 2^n-minute
exponential backoff capped at 1h (notifRetryBackoffCap) with
5-attempt budget (notifRetryMaxAttempts).
- Exhaustion (RetryCount >= notifRetryMaxAttempts-1) promotes to
status='dead' via MarkAsDead.
- Non-terminal failures record via RecordFailedAttempt.
- Success path promotes to 'sent' without touching retry_count
(audit preserves "delivered on attempt N").
- Missing-notifier branch defensively promotes to 'sent' to avoid
wedging a row on a deleted channel.
- RequeueNotification operator escape hatch atomically resets
retry_count -> 0, next_retry_at -> NULL, last_error -> NULL,
status -> pending via notifRepo.Requeue.
Scheduler:
- New always-on notificationRetryLoop wired into the base loop set at
CERTCTL_NOTIFICATION_RETRY_INTERVAL (default 2m).
- sync/atomic.Bool idempotency guard.
- sync.WaitGroup shutdown drain via WaitForCompletion.
StatsService:
- SetNotifRepo setter pattern preserves 9 pre-existing
NewStatsService call sites (main.go + stats_test.go + 8 digest
tests) without touching the constructor signature.
- DashboardSummary.NotificationsDead populated via
notifRepo.CountByStatus(ctx, "dead") — nil-safe when unwired
(reports zero on systems without a notification repository).
- CountByStatus error is non-fatal (dashboard summary is
best-effort for this field).
- Prometheus certctl_notification_dead_total counter emitted from
the same snapshot.
Handler:
- New POST /api/v1/notifications/{id}/requeue endpoint.
- dead status surfaces to MCP + CLI.
Frontend:
- NotificationsPage gains two-tab toolbar ("All" / "Dead letter")
with queryKey: ['notifications', activeTab] so switching tabs
doesn't serve stale data until the 30s refetch.
- Dead rows surface "Retry {n}/5" + truncated last_error with
full-text title tooltip.
- Requeue mutation wrapped as
mutationFn: (id: string) => requeueNotification(id)
to prevent react-query v5's positional context argument from
leaking into the API client — pinned against future refactors
by strict-match toHaveBeenCalledWith('notif-dead-001') in
NotificationsPage.test.tsx:181.
Closes I-005.
This commit is contained in:
+46
-11
@@ -15,6 +15,12 @@ type StatsService struct {
|
||||
certRepo repository.CertificateRepository
|
||||
jobRepo repository.JobRepository
|
||||
agentRepo repository.AgentRepository
|
||||
// notifRepo is injected post-construction via SetNotifRepo so that
|
||||
// NewStatsService's nine call sites (main.go + stats_test.go + 8 digest
|
||||
// tests) keep their existing signatures. When nil, the dead-letter count
|
||||
// falls through to zero — see GetDashboardSummary. I-005 coverage-gap
|
||||
// closure.
|
||||
notifRepo repository.NotificationRepository
|
||||
}
|
||||
|
||||
// NewStatsService creates a new stats service.
|
||||
@@ -30,19 +36,35 @@ func NewStatsService(
|
||||
}
|
||||
}
|
||||
|
||||
// SetNotifRepo injects the notification repository used to populate
|
||||
// DashboardSummary.NotificationsDead. Setter pattern (matching the
|
||||
// certificateService.SetTargetRepo / SetProfileRepo / SetDigestService
|
||||
// precedent) keeps the NewStatsService signature stable across its
|
||||
// pre-existing call sites. I-005 coverage-gap closure.
|
||||
func (s *StatsService) SetNotifRepo(notifRepo repository.NotificationRepository) {
|
||||
s.notifRepo = notifRepo
|
||||
}
|
||||
|
||||
// DashboardSummary represents a high-level summary of system state.
|
||||
type DashboardSummary struct {
|
||||
TotalCertificates int64 `json:"total_certificates"`
|
||||
ExpiringCertificates int64 `json:"expiring_certificates"`
|
||||
ExpiredCertificates int64 `json:"expired_certificates"`
|
||||
RevokedCertificates int64 `json:"revoked_certificates"`
|
||||
ActiveAgents int64 `json:"active_agents"`
|
||||
OfflineAgents int64 `json:"offline_agents"`
|
||||
TotalAgents int64 `json:"total_agents"`
|
||||
PendingJobs int64 `json:"pending_jobs"`
|
||||
FailedJobs int64 `json:"failed_jobs"`
|
||||
CompleteJobs int64 `json:"complete_jobs"`
|
||||
CompletedAt time.Time `json:"completed_at"`
|
||||
TotalCertificates int64 `json:"total_certificates"`
|
||||
ExpiringCertificates int64 `json:"expiring_certificates"`
|
||||
ExpiredCertificates int64 `json:"expired_certificates"`
|
||||
RevokedCertificates int64 `json:"revoked_certificates"`
|
||||
ActiveAgents int64 `json:"active_agents"`
|
||||
OfflineAgents int64 `json:"offline_agents"`
|
||||
TotalAgents int64 `json:"total_agents"`
|
||||
PendingJobs int64 `json:"pending_jobs"`
|
||||
FailedJobs int64 `json:"failed_jobs"`
|
||||
CompleteJobs int64 `json:"complete_jobs"`
|
||||
// NotificationsDead is the number of notification_events rows currently
|
||||
// in the terminal "dead" status (I-005 dead-letter queue). Exposed here
|
||||
// so the metrics handler can derive the Prometheus counter
|
||||
// certctl_notification_dead_total from the same snapshot used by the
|
||||
// dashboard. DB-COUNT rather than in-memory — notifications can grow
|
||||
// without bound, and filter-based List() is PerPage-capped to 50.
|
||||
NotificationsDead int64 `json:"notifications_dead"`
|
||||
CompletedAt time.Time `json:"completed_at"`
|
||||
}
|
||||
|
||||
// GetDashboardSummary returns a summary of key metrics.
|
||||
@@ -106,6 +128,19 @@ func (s *StatsService) GetDashboardSummary(ctx context.Context) (interface{}, er
|
||||
}
|
||||
}
|
||||
|
||||
// I-005: dead-letter count for certctl_notification_dead_total. nil-safe
|
||||
// so the nine existing NewStatsService call sites that haven't yet been
|
||||
// updated to call SetNotifRepo keep working — they'll simply report
|
||||
// NotificationsDead=0, which is the correct value on a system without a
|
||||
// notification repository wired in. A CountByStatus error is non-fatal:
|
||||
// the dashboard summary is best-effort for this field.
|
||||
if s.notifRepo != nil {
|
||||
deadCount, err := s.notifRepo.CountByStatus(ctx, string(domain.NotificationStatusDead))
|
||||
if err == nil {
|
||||
summary.NotificationsDead = deadCount
|
||||
}
|
||||
}
|
||||
|
||||
return summary, nil
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user