Critical alerts can no longer be silently dropped by a transient
notifier failure. Failed notification attempts now ride an exponential
backoff retry loop, with a 5-attempt budget before promotion to the
dead-letter queue for operator intervention.
Schema (migration 000016, idempotent):
- retry_count INTEGER NOT NULL DEFAULT 0
- next_retry_at TIMESTAMPTZ
- last_error TEXT
- idx_notification_events_retry_sweep partial index
(next_retry_at) WHERE status='failed' AND next_retry_at IS NOT NULL
Dead rows clear next_retry_at so the index stops matching them.
Service contract:
- NotificationService.RetryFailedNotifications drives 2^n-minute
exponential backoff capped at 1h (notifRetryBackoffCap) with
5-attempt budget (notifRetryMaxAttempts).
- Exhaustion (RetryCount >= notifRetryMaxAttempts-1) promotes to
status='dead' via MarkAsDead.
- Non-terminal failures record via RecordFailedAttempt.
- Success path promotes to 'sent' without touching retry_count
(audit preserves "delivered on attempt N").
- Missing-notifier branch defensively promotes to 'sent' to avoid
wedging a row on a deleted channel.
- RequeueNotification operator escape hatch atomically resets
retry_count -> 0, next_retry_at -> NULL, last_error -> NULL,
status -> pending via notifRepo.Requeue.
Scheduler:
- New always-on notificationRetryLoop wired into the base loop set at
CERTCTL_NOTIFICATION_RETRY_INTERVAL (default 2m).
- sync/atomic.Bool idempotency guard.
- sync.WaitGroup shutdown drain via WaitForCompletion.
StatsService:
- SetNotifRepo setter pattern preserves 9 pre-existing
NewStatsService call sites (main.go + stats_test.go + 8 digest
tests) without touching the constructor signature.
- DashboardSummary.NotificationsDead populated via
notifRepo.CountByStatus(ctx, "dead") — nil-safe when unwired
(reports zero on systems without a notification repository).
- CountByStatus error is non-fatal (dashboard summary is
best-effort for this field).
- Prometheus certctl_notification_dead_total counter emitted from
the same snapshot.
Handler:
- New POST /api/v1/notifications/{id}/requeue endpoint.
- dead status surfaces to MCP + CLI.
Frontend:
- NotificationsPage gains two-tab toolbar ("All" / "Dead letter")
with queryKey: ['notifications', activeTab] so switching tabs
doesn't serve stale data until the 30s refetch.
- Dead rows surface "Retry {n}/5" + truncated last_error with
full-text title tooltip.
- Requeue mutation wrapped as
mutationFn: (id: string) => requeueNotification(id)
to prevent react-query v5's positional context argument from
leaking into the API client — pinned against future refactors
by strict-match toHaveBeenCalledWith('notif-dead-001') in
NotificationsPage.test.tsx:181.
Closes I-005.
Complete frontend visual redesign using certctl logo color palette:
- Deep teal sidebar (#0c2e25) with prominent centered logo (64px in white pill)
- Light content area (#f0f4f8) with white cards and visible borders
- Brand colors from logo: teal (#2ea88f), blue (#3b7dd8), orange (#e8873a), green (#4ebe6e)
- Inter + JetBrains Mono typography, colored stat card top borders
- All 17 pages + 7 components updated (25 files, ~700 lines changed)
- 15 new dashboard screenshots replacing old dark theme screenshots
- Prometheus metrics e2e test added, integration test mock fixes
- Docs updated: architecture.md theme description, testing-guide.md DNS-PERSIST-01 coverage
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire all remaining dashboard views to real API: agent detail page
with heartbeat status and capabilities, audit trail with time range/
actor/resource filters, notifications with grouped-by-cert view and
read/unread state, policies with severity summary bar, new issuers
and targets list views. Add GitHub Actions CI with parallel Go and
Frontend jobs. Update Makefile with test-cover and frontend-build
targets.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Backend hardening:
- Fix 6 nginx.go non-constant format string build errors
- Add validation.go with hostname, PEM, and enum validators
- Apply input validation to all POST/PUT handlers (certificates,
agents, CSR, policies, teams, owners, targets, issuers)
- Fix unchecked JSON decode in TriggerDeployment handler
Frontend (Vite + React + TypeScript):
- Migrate from single-file SPA to proper build pipeline
- 7 pages: Dashboard, Certificates (list+detail), Agents, Jobs,
Notifications, Policies, Audit Trail
- TanStack Query for server state with auto-refetch intervals
- Certificate detail with version history and renewal trigger
- Job cancellation, status/type filtering, expiry countdowns
- Reusable components: DataTable, StatusBadge, ErrorState, PageHeader
- Dark theme with Tailwind CSS, sidebar nav via React Router
Server integration:
- Go server serves web/dist/ (Vite output) with SPA fallback
- Falls back to web/index.html for legacy mode
- .gitignore updated for web/node_modules/ and web/dist/
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Shankar