mirror of
https://github.com/shankar0123/certctl.git
synced 2026-06-07 12:41:30 +00:00
109f32ff41
Closes Rank 4 of the 2026-05-03 Infisical deep-research deliverable
(see cowork/infisical-deep-research-results.md Part 5). Pre-fix,
RenewalService.CheckExpiringCertificates already ran daily,
RenewalPolicy.AlertThresholdsDays drove per-cert thresholds, and
NotificationService.SendThresholdAlert deduped per (cert, threshold)
— but the channel was hardcoded to Email
(internal/service/notification.go:118 pre-fix). Operators who
configured PagerDuty / Slack / Teams / OpsGenie via
CERTCTL_PAGERDUTY_ROUTING_KEY etc. got nothing at any threshold
unless SMTP was also wired. Their first signal of an expired cert
was a 3 AM outage.
This commit lands the routing matrix on top of the existing
infrastructure:
1. RenewalPolicy gains AlertChannels (per-tier channel list) +
AlertSeverityMap (per-threshold tier assignment) +
EffectiveAlertChannels / EffectiveAlertSeverity accessors.
Default*() helpers preserve the back-compat Email-only
behaviour for operators who haven't touched their policies
post-upgrade. Migration 000026 adds the JSONB columns
idempotently.
2. NotificationService.SendThresholdAlertOnChannel — the new
per-channel dispatch helper. Old SendThresholdAlert stays as
an Email-only alias so non-policy callers (admin "send test
alert" surfaces) keep working byte-for-byte.
3. NotificationService.HasThresholdNotificationOnChannel — per-
(cert, threshold, channel) deduplication so a transient
PagerDuty 5xx today does NOT suppress today's Slack alert and
tomorrow's PagerDuty retry will still fire.
4. RenewalService.sendThresholdAlerts walks the resolved channel
set per threshold tier, fans out to every configured channel,
handles per-channel failures independently, defensively drops
off-enum channels with an audit row trail, and records a per-
channel audit event with metadata.channel + metadata.severity_tier.
5. service.ExpiryAlertMetrics — atomic counter table mirrored on
the VaultRenewalMetrics shape from the 2026-05-03 audit fix #5
(commit 0792271). Three labels: channel × threshold × result
(success / failure / deduped). Cardinality bound: 6 × 4 × 3 =
72 series for the standard 4-threshold matrix.
6. handler.MetricsHandler.SetExpiryAlerts wires the Prometheus
exposer for certctl_expiry_alerts_total{channel,threshold,result}.
Pre-sorted snapshot for byte-stable emission.
7. cmd/server/main.go threads ONE service.ExpiryAlertMetrics
instance through both the recording side (notificationService.
SetExpiryAlertMetrics) and the exposing side
(metricsHandler.SetExpiryAlerts).
Dispatch flow (post-fix, per renewal-loop tick):
cert ages past T-30 → daily renewal-loop fires
→ policy lookup
→ for each crossed threshold:
- resolve severity tier (informational/
warning/critical) via AlertSeverityMap
- look up channel set in AlertChannels[tier]
- for each channel: dedup → SendThresholdAlertOnChannel
→ notifierRegistry[channel] → audit row →
Prometheus counter increment
Tests (internal/service/renewal_expiry_alerts_test.go):
TestExpiryAlerts_DefaultMatrix_EmailOnly
TestExpiryAlerts_PerTierFanOut
TestExpiryAlerts_PerChannelDedup
TestExpiryAlerts_OneChannelFails_OthersStillFire
TestExpiryAlerts_OffEnumChannelDropped
TestExpiryAlerts_MetricCounterIncrements
TestExpiryAlerts_NilPolicy_FallsToDefault
TestExpiryAlerts_OperatorOptOutOfTier
The PerTierFanOut test wires 6 mock notifiers, drives a cert at 0
days through the canonical 4 thresholds with the matrix
{informational:[Slack], warning:[Slack,Email],
critical:[PagerDuty,OpsGenie,Email]}, and asserts the exact
recipient counts: Slack=3, Email=3, PagerDuty=1, OpsGenie=1, no
Teams, no Webhook. The OneChannelFails test pins that PagerDuty
returning a 503 does NOT skip Slack/Email at the same threshold.
Drive-by fix (internal/service/testutil_test.go): the existing
mockNotifRepo.List ignored its filter and returned all rows, which
let legacy tests pass on dedup-via-substring even though the
postgres repo actually applied the filter. Updated the mock to
honour CertificateID / Type / Status / Channel / MessageLike
filters in the same shape as the postgres implementation
(internal/repository/postgres/notification.go). All pre-existing
service tests still pass — the legacy test suite happened to be
robust to the mock filter doing nothing.
Documentation:
- docs/connectors.md Notifier section gains "Routing expiry
alerts across channels" — operator-facing, JSON example,
procurement playbook ("How do I make sure PagerDuty pages on
the T-1 alert?"), debug recipe via SQL on audit_events +
notification_events + Prometheus.
- docs/runbook-expiry-alerts.md — sysadmin-grade flowchart,
per-policy channel-matrix configuration recipes, "did the on-
call team get paged?" SQL queries, cardinality budget, V3-Pro
forward path.
- cowork/WORKSPACE-ROADMAP.md gains "Multi-channel expiry
alerts: per-owner routing" V3-Pro entry under Adapter
hardening.
Out of scope (intentional, flagged in V3-Pro forward path):
- Per-owner / per-team / per-tenant channel routing (matrix is
per-policy today, not per-owner).
- Calendar-aware suppression (no T-30 alerts on weekends).
- Escalation chains (T-1 unanswered for 30m → escalate).
- Per-channel rate limiting (downstream of I-005 retry+DLQ).
CHANGELOG.md is intentionally not hand-edited per CHANGELOG.md
itself ("no longer maintains a hand-edited per-version changelog;
per-release notes are auto-generated from commit messages between
consecutive tags").
Verified locally:
- gofmt clean.
- go vet ./internal/domain/... ./internal/service/...
./internal/api/handler/... ./cmd/server/... clean.
(./internal/repository/postgres/... vet failed on transitive
testcontainers/docker module download — sandbox disk pressure,
not a code issue; postgres-repo build succeeds and tests pass.)
- go test -short -count=1 ./internal/domain/...
./internal/service/... ./internal/api/handler/... green.
- go test -race -count=10 -run 'TestExpiryAlerts'
./internal/service/... green (per-channel dedup race-free).
Reference: cowork/infisical-deep-research-results.md Part 5 Rank 4.
Acquisition prompt: cowork/rank-4-multichannel-expiry-alerts-prompt.md.
162 lines
5.4 KiB
Go
162 lines
5.4 KiB
Go
package service
|
||
|
||
import (
|
||
"sort"
|
||
"sync"
|
||
"sync/atomic"
|
||
)
|
||
|
||
// ExpiryAlertMetrics is a thread-safe counter table for the per-policy
|
||
// multi-channel expiry-alert dispatch path. Rank 4 of the 2026-05-03
|
||
// Infisical deep-research deliverable
|
||
// (cowork/infisical-deep-research-results.md Part 5). Closes the
|
||
// procurement-checklist gap where a customer who configured PagerDuty
|
||
// for cert-expiry pages got silent nothing — ExpirationWarning shipped
|
||
// only to Email pre-fix.
|
||
//
|
||
// Dimensions:
|
||
//
|
||
// channel — closed-enum NotificationChannel value (Email, Slack,
|
||
// Teams, PagerDuty, OpsGenie, Webhook). Off-enum
|
||
// channels are silently dropped at the dispatch site
|
||
// BEFORE this counter sees them, so cardinality stays
|
||
// bounded.
|
||
// threshold — int days-until-expiry the alert fired for (e.g. 30,
|
||
// 14, 7, 0). Custom-thresholds policies can grow this
|
||
// dimension; production deploys with the standard 4
|
||
// thresholds give 4 distinct values.
|
||
// result — closed enum:
|
||
// "success" — the channel's notifier accepted the
|
||
// send. (Underlying delivery may still
|
||
// fail if e.g. SMTP queue is broken;
|
||
// those failures surface via the
|
||
// existing I-005 retry/DLQ machinery.)
|
||
// "failure" — the channel's notifier returned an
|
||
// error, OR the notification row failed
|
||
// to persist. Operators alert on
|
||
// sustained {result="failure"} > 0.
|
||
// "deduped" — a prior (cert, threshold, channel)
|
||
// notification was already in
|
||
// persistence; today's loop skipped the
|
||
// send. Useful for detecting
|
||
// "everything is healthy and steady-
|
||
// state" — high deduped counts mean
|
||
// the daily loop is doing its job.
|
||
//
|
||
// Cardinality bound: 6 channels × 4 thresholds × 3 results = 72 series.
|
||
// A custom-thresholds policy can grow this; bound is operator-controlled.
|
||
//
|
||
// Wiring: cmd/server/main.go constructs ONE instance of
|
||
// *ExpiryAlertMetrics, calls notificationService.SetExpiryAlertMetrics
|
||
// to register the recording side, AND
|
||
// metricsHandler.SetExpiryAlerts to register the exposing side.
|
||
// Mirror of the VaultRenewalMetrics shape from the 2026-05-03
|
||
// audit fix #5 (commit `ceca364`) for operator-symmetry — same
|
||
// snapshot interface, same atomic-counters-under-RW-mutex pattern.
|
||
type ExpiryAlertMetrics struct {
|
||
mu sync.RWMutex
|
||
counters map[expiryAlertKey]*atomic.Uint64
|
||
}
|
||
|
||
type expiryAlertKey struct {
|
||
Channel string
|
||
Threshold int
|
||
Result string
|
||
}
|
||
|
||
// NewExpiryAlertMetrics constructs a fresh ExpiryAlertMetrics with all
|
||
// counters at zero. Pass to NotificationService.SetExpiryAlertMetrics
|
||
// (recording side) and MetricsHandler.SetExpiryAlerts (exposing side).
|
||
func NewExpiryAlertMetrics() *ExpiryAlertMetrics {
|
||
return &ExpiryAlertMetrics{
|
||
counters: make(map[expiryAlertKey]*atomic.Uint64),
|
||
}
|
||
}
|
||
|
||
// RecordExpiryAlert bumps the (channel, threshold, result) counter.
|
||
// Implements service.ExpiryAlertRecorder (from notification.go) so
|
||
// NotificationService can call this on every dispatch outcome without
|
||
// importing the metrics package.
|
||
//
|
||
// Off-enum result values silently no-op (closed-enum discipline; we
|
||
// don't dynamic-cardinality-grow the Prometheus exposition on a
|
||
// caller typo).
|
||
func (m *ExpiryAlertMetrics) RecordExpiryAlert(channel string, threshold int, result string) {
|
||
if m == nil {
|
||
return
|
||
}
|
||
switch result {
|
||
case "success", "failure", "deduped":
|
||
// ok
|
||
default:
|
||
return
|
||
}
|
||
|
||
key := expiryAlertKey{Channel: channel, Threshold: threshold, Result: result}
|
||
|
||
m.mu.RLock()
|
||
c, ok := m.counters[key]
|
||
m.mu.RUnlock()
|
||
if ok {
|
||
c.Add(1)
|
||
return
|
||
}
|
||
|
||
m.mu.Lock()
|
||
if c, ok := m.counters[key]; ok {
|
||
// Lost the race; another goroutine inserted while we were
|
||
// upgrading the lock.
|
||
m.mu.Unlock()
|
||
c.Add(1)
|
||
return
|
||
}
|
||
c = &atomic.Uint64{}
|
||
c.Add(1)
|
||
m.counters[key] = c
|
||
m.mu.Unlock()
|
||
}
|
||
|
||
// ExpiryAlertSnapshotEntry is one row in the snapshot result. The
|
||
// Prometheus exposer iterates these to produce the
|
||
// certctl_expiry_alerts_total{channel, threshold, result} series.
|
||
type ExpiryAlertSnapshotEntry struct {
|
||
Channel string
|
||
Threshold int
|
||
Result string
|
||
Count uint64
|
||
}
|
||
|
||
// SnapshotExpiryAlerts returns a point-in-time read of every
|
||
// (channel, threshold, result) counter. The slice is sorted by
|
||
// (channel, threshold, result) so the Prometheus exposition is
|
||
// stable across requests.
|
||
//
|
||
// Implements handler.ExpiryAlertSnapshotter for the metrics emitter.
|
||
func (m *ExpiryAlertMetrics) SnapshotExpiryAlerts() []ExpiryAlertSnapshotEntry {
|
||
if m == nil {
|
||
return nil
|
||
}
|
||
m.mu.RLock()
|
||
defer m.mu.RUnlock()
|
||
|
||
out := make([]ExpiryAlertSnapshotEntry, 0, len(m.counters))
|
||
for k, v := range m.counters {
|
||
out = append(out, ExpiryAlertSnapshotEntry{
|
||
Channel: k.Channel,
|
||
Threshold: k.Threshold,
|
||
Result: k.Result,
|
||
Count: v.Load(),
|
||
})
|
||
}
|
||
sort.Slice(out, func(i, j int) bool {
|
||
if out[i].Channel != out[j].Channel {
|
||
return out[i].Channel < out[j].Channel
|
||
}
|
||
if out[i].Threshold != out[j].Threshold {
|
||
return out[i].Threshold < out[j].Threshold
|
||
}
|
||
return out[i].Result < out[j].Result
|
||
})
|
||
return out
|
||
}
|