mirror of
https://github.com/shankar0123/certctl.git
synced 2026-06-07 19:31:31 +00:00
shankar0123
675b87ba63
Critical alerts can no longer be silently dropped by a transient
notifier failure. Failed notification attempts now ride an exponential
backoff retry loop, with a 5-attempt budget before promotion to the
dead-letter queue for operator intervention.
Schema (migration 000016, idempotent):
- retry_count INTEGER NOT NULL DEFAULT 0
- next_retry_at TIMESTAMPTZ
- last_error TEXT
- idx_notification_events_retry_sweep partial index
(next_retry_at) WHERE status='failed' AND next_retry_at IS NOT NULL
Dead rows clear next_retry_at so the index stops matching them.
Service contract:
- NotificationService.RetryFailedNotifications drives 2^n-minute
exponential backoff capped at 1h (notifRetryBackoffCap) with
5-attempt budget (notifRetryMaxAttempts).
- Exhaustion (RetryCount >= notifRetryMaxAttempts-1) promotes to
status='dead' via MarkAsDead.
- Non-terminal failures record via RecordFailedAttempt.
- Success path promotes to 'sent' without touching retry_count
(audit preserves "delivered on attempt N").
- Missing-notifier branch defensively promotes to 'sent' to avoid
wedging a row on a deleted channel.
- RequeueNotification operator escape hatch atomically resets
retry_count -> 0, next_retry_at -> NULL, last_error -> NULL,
status -> pending via notifRepo.Requeue.
Scheduler:
- New always-on notificationRetryLoop wired into the base loop set at
CERTCTL_NOTIFICATION_RETRY_INTERVAL (default 2m).
- sync/atomic.Bool idempotency guard.
- sync.WaitGroup shutdown drain via WaitForCompletion.
StatsService:
- SetNotifRepo setter pattern preserves 9 pre-existing
NewStatsService call sites (main.go + stats_test.go + 8 digest
tests) without touching the constructor signature.
- DashboardSummary.NotificationsDead populated via
notifRepo.CountByStatus(ctx, "dead") — nil-safe when unwired
(reports zero on systems without a notification repository).
- CountByStatus error is non-fatal (dashboard summary is
best-effort for this field).
- Prometheus certctl_notification_dead_total counter emitted from
the same snapshot.
Handler:
- New POST /api/v1/notifications/{id}/requeue endpoint.
- dead status surfaces to MCP + CLI.
Frontend:
- NotificationsPage gains two-tab toolbar ("All" / "Dead letter")
with queryKey: ['notifications', activeTab] so switching tabs
doesn't serve stale data until the 30s refetch.
- Dead rows surface "Retry {n}/5" + truncated last_error with
full-text title tooltip.
- Requeue mutation wrapped as
mutationFn: (id: string) => requeueNotification(id)
to prevent react-query v5's positional context argument from
leaking into the API client — pinned against future refactors
by strict-match toHaveBeenCalledWith('notif-dead-001') in
NotificationsPage.test.tsx:181.
Closes I-005.
85 lines
1.0 KiB
Plaintext
85 lines
1.0 KiB
Plaintext
# Binaries for programs and plugins
|
|
*.exe
|
|
*.exe~
|
|
*.dll
|
|
*.so
|
|
*.so.*
|
|
*.dylib
|
|
bin/
|
|
|
|
# Frontend
|
|
web/node_modules/
|
|
web/dist/
|
|
|
|
# Test binary, built with `go test -c`
|
|
*.test
|
|
|
|
# Output of the go coverage tool
|
|
*.out
|
|
coverage.out
|
|
coverage.html
|
|
|
|
# Go workspace file
|
|
go.work
|
|
|
|
# Dependency directories
|
|
vendor/
|
|
|
|
# Environment variables
|
|
.env
|
|
.env.local
|
|
.env.*.local
|
|
|
|
# IDE
|
|
.idea/
|
|
.vscode/
|
|
*.swp
|
|
*.swo
|
|
*~
|
|
.DS_Store
|
|
*.iml
|
|
|
|
# Temporary files
|
|
tmp/
|
|
temp/
|
|
*.log
|
|
*.bak
|
|
|
|
# Private keys (agent-generated, never commit)
|
|
cmd/agent/*.key
|
|
cmd/agent/*.pem
|
|
|
|
# Database
|
|
*.db
|
|
*.sqlite3
|
|
|
|
# Allow migration SQL files (don't ignore *.sql globally)
|
|
# SQL files in migrations/ are tracked
|
|
|
|
# Build artifacts
|
|
certctl-server
|
|
certctl-agent
|
|
certctl-cli
|
|
/server
|
|
/agent
|
|
/cli
|
|
|
|
# Private strategy docs
|
|
strategy.md
|
|
SECURITY_REMEDIATION.md
|
|
|
|
# OS
|
|
.DS_Store
|
|
Thumbs.db
|
|
mcp-server
|
|
|
|
# Local Go build/module caches (session-scoped, never committed)
|
|
/.gocache/
|
|
/.gomodcache/
|
|
/.gopath/
|
|
/.gomodcache-gopath/
|
|
|
|
# Design scratch files (session-scoped)
|
|
/.i004-design.md
|
|
/.i005-design.md
|