Files
certctl/internal/repository/postgres/renewal_policy.go
T
shankar0123 6af95ccf5f notifications: per-policy multi-channel expiry-alert routing
Closes Rank 4 of the 2026-05-03 Infisical deep-research deliverable
(see cowork/infisical-deep-research-results.md Part 5). Pre-fix,
RenewalService.CheckExpiringCertificates already ran daily,
RenewalPolicy.AlertThresholdsDays drove per-cert thresholds, and
NotificationService.SendThresholdAlert deduped per (cert, threshold)
— but the channel was hardcoded to Email
(internal/service/notification.go:118 pre-fix). Operators who
configured PagerDuty / Slack / Teams / OpsGenie via
CERTCTL_PAGERDUTY_ROUTING_KEY etc. got nothing at any threshold
unless SMTP was also wired. Their first signal of an expired cert
was a 3 AM outage.

This commit lands the routing matrix on top of the existing
infrastructure:

  1. RenewalPolicy gains AlertChannels (per-tier channel list) +
     AlertSeverityMap (per-threshold tier assignment) +
     EffectiveAlertChannels / EffectiveAlertSeverity accessors.
     Default*() helpers preserve the back-compat Email-only
     behaviour for operators who haven't touched their policies
     post-upgrade. Migration 000026 adds the JSONB columns
     idempotently.
  2. NotificationService.SendThresholdAlertOnChannel — the new
     per-channel dispatch helper. Old SendThresholdAlert stays as
     an Email-only alias so non-policy callers (admin "send test
     alert" surfaces) keep working byte-for-byte.
  3. NotificationService.HasThresholdNotificationOnChannel — per-
     (cert, threshold, channel) deduplication so a transient
     PagerDuty 5xx today does NOT suppress today's Slack alert and
     tomorrow's PagerDuty retry will still fire.
  4. RenewalService.sendThresholdAlerts walks the resolved channel
     set per threshold tier, fans out to every configured channel,
     handles per-channel failures independently, defensively drops
     off-enum channels with an audit row trail, and records a per-
     channel audit event with metadata.channel + metadata.severity_tier.
  5. service.ExpiryAlertMetrics — atomic counter table mirrored on
     the VaultRenewalMetrics shape from the 2026-05-03 audit fix #5
     (commit ceca364). Three labels: channel × threshold × result
     (success / failure / deduped). Cardinality bound: 6 × 4 × 3 =
     72 series for the standard 4-threshold matrix.
  6. handler.MetricsHandler.SetExpiryAlerts wires the Prometheus
     exposer for certctl_expiry_alerts_total{channel,threshold,result}.
     Pre-sorted snapshot for byte-stable emission.
  7. cmd/server/main.go threads ONE service.ExpiryAlertMetrics
     instance through both the recording side (notificationService.
     SetExpiryAlertMetrics) and the exposing side
     (metricsHandler.SetExpiryAlerts).

Dispatch flow (post-fix, per renewal-loop tick):

  cert ages past T-30  → daily renewal-loop fires
                       → policy lookup
                       → for each crossed threshold:
                           - resolve severity tier (informational/
                             warning/critical) via AlertSeverityMap
                           - look up channel set in AlertChannels[tier]
                           - for each channel: dedup → SendThresholdAlertOnChannel
                             → notifierRegistry[channel] → audit row →
                             Prometheus counter increment

Tests (internal/service/renewal_expiry_alerts_test.go):

  TestExpiryAlerts_DefaultMatrix_EmailOnly
  TestExpiryAlerts_PerTierFanOut
  TestExpiryAlerts_PerChannelDedup
  TestExpiryAlerts_OneChannelFails_OthersStillFire
  TestExpiryAlerts_OffEnumChannelDropped
  TestExpiryAlerts_MetricCounterIncrements
  TestExpiryAlerts_NilPolicy_FallsToDefault
  TestExpiryAlerts_OperatorOptOutOfTier

The PerTierFanOut test wires 6 mock notifiers, drives a cert at 0
days through the canonical 4 thresholds with the matrix
{informational:[Slack], warning:[Slack,Email],
critical:[PagerDuty,OpsGenie,Email]}, and asserts the exact
recipient counts: Slack=3, Email=3, PagerDuty=1, OpsGenie=1, no
Teams, no Webhook. The OneChannelFails test pins that PagerDuty
returning a 503 does NOT skip Slack/Email at the same threshold.

Drive-by fix (internal/service/testutil_test.go): the existing
mockNotifRepo.List ignored its filter and returned all rows, which
let legacy tests pass on dedup-via-substring even though the
postgres repo actually applied the filter. Updated the mock to
honour CertificateID / Type / Status / Channel / MessageLike
filters in the same shape as the postgres implementation
(internal/repository/postgres/notification.go). All pre-existing
service tests still pass — the legacy test suite happened to be
robust to the mock filter doing nothing.

Documentation:
  - docs/connectors.md Notifier section gains "Routing expiry
    alerts across channels" — operator-facing, JSON example,
    procurement playbook ("How do I make sure PagerDuty pages on
    the T-1 alert?"), debug recipe via SQL on audit_events +
    notification_events + Prometheus.
  - docs/runbook-expiry-alerts.md — sysadmin-grade flowchart,
    per-policy channel-matrix configuration recipes, "did the on-
    call team get paged?" SQL queries, cardinality budget, V3-Pro
    forward path.
  - cowork/WORKSPACE-ROADMAP.md gains "Multi-channel expiry
    alerts: per-owner routing" V3-Pro entry under Adapter
    hardening.

Out of scope (intentional, flagged in V3-Pro forward path):
  - Per-owner / per-team / per-tenant channel routing (matrix is
    per-policy today, not per-owner).
  - Calendar-aware suppression (no T-30 alerts on weekends).
  - Escalation chains (T-1 unanswered for 30m → escalate).
  - Per-channel rate limiting (downstream of I-005 retry+DLQ).

CHANGELOG.md is intentionally not hand-edited per CHANGELOG.md
itself ("no longer maintains a hand-edited per-version changelog;
per-release notes are auto-generated from commit messages between
consecutive tags").

Verified locally:
- gofmt clean.
- go vet ./internal/domain/... ./internal/service/...
  ./internal/api/handler/... ./cmd/server/...  clean.
  (./internal/repository/postgres/... vet failed on transitive
  testcontainers/docker module download — sandbox disk pressure,
  not a code issue; postgres-repo build succeeds and tests pass.)
- go test -short -count=1 ./internal/domain/...
  ./internal/service/... ./internal/api/handler/...  green.
- go test -race -count=10 -run 'TestExpiryAlerts'
  ./internal/service/...  green (per-channel dedup race-free).

Reference: cowork/infisical-deep-research-results.md Part 5 Rank 4.
Acquisition prompt: cowork/rank-4-multichannel-expiry-alerts-prompt.md.
2026-05-03 22:12:32 +00:00

382 lines
13 KiB
Go

package postgres
import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"regexp"
"strings"
"github.com/lib/pq"
"github.com/shankar0123/certctl/internal/domain"
"github.com/shankar0123/certctl/internal/repository"
)
// RenewalPolicyRepository implements repository.RenewalPolicyRepository.
type RenewalPolicyRepository struct {
db *sql.DB
}
// NewRenewalPolicyRepository creates a new RenewalPolicyRepository.
func NewRenewalPolicyRepository(db *sql.DB) *RenewalPolicyRepository {
return &RenewalPolicyRepository{db: db}
}
// SELECT column order is the shared contract between scanRenewalPolicy and
// every SELECT/RETURNING in this file. Keep them in lockstep; if you add a
// new column, add it to all SELECTs, all scan calls, and scanRenewalPolicy.
//
// Note: certificate_profile_id and agent_group_id live on renewal_policies
// (migrations 000003 and 000004) but are deliberately NOT read here — that
// pre-existing drift is out of G-1's minimum-viable-delta and is tracked in
// the design doc §8. Introducing them would change struct shapes / JSON tags
// and require domain-layer churn we're not taking on in this change.
//
// alert_channels / alert_severity_map (migration 000026) ARE read here —
// they're the per-policy channel matrix that drives multi-channel expiry
// alert routing (Rank 4 of the 2026-05-03 Infisical deep-research
// deliverable). Both default to '{}' at the DB level; scanRenewalPolicy
// unmarshals an empty map into nil so domain.EffectiveAlertChannels /
// EffectiveAlertSeverityMap fall through to the back-compat defaults.
const renewalPolicyColumns = `
id, name, renewal_window_days, auto_renew, max_retries,
retry_interval_seconds, alert_thresholds_days,
alert_channels, alert_severity_map,
created_at, updated_at
`
// scanRenewalPolicy decodes one renewal_policies row from a Row or Rows
// scanner, unmarshaling alert_thresholds_days JSONB into the domain slice.
// Malformed JSONB silently falls back to DefaultAlertThresholds() — same
// behavior as the pre-G-1 code so we don't change observable semantics.
//
// alert_channels + alert_severity_map (migration 000026) follow the same
// "malformed → fall through to default" rule. The default-fallthrough
// happens at read time in domain.EffectiveAlertChannels /
// EffectiveAlertSeverity, so populating these fields with nil on parse
// failure is the correct shape — the runtime still gets the back-compat
// Email-only matrix.
func scanRenewalPolicy(scanner interface {
Scan(dest ...any) error
}) (*domain.RenewalPolicy, error) {
var policy domain.RenewalPolicy
var thresholdsJSON []byte
var channelsJSON []byte
var severityJSON []byte
if err := scanner.Scan(
&policy.ID, &policy.Name, &policy.RenewalWindowDays, &policy.AutoRenew,
&policy.MaxRetries, &policy.RetryInterval, &thresholdsJSON,
&channelsJSON, &severityJSON,
&policy.CreatedAt, &policy.UpdatedAt,
); err != nil {
return nil, err
}
if len(thresholdsJSON) > 0 {
if err := json.Unmarshal(thresholdsJSON, &policy.AlertThresholdsDays); err != nil {
policy.AlertThresholdsDays = domain.DefaultAlertThresholds()
}
}
if len(channelsJSON) > 0 && string(channelsJSON) != "{}" {
if err := json.Unmarshal(channelsJSON, &policy.AlertChannels); err != nil {
policy.AlertChannels = nil // EffectiveAlertChannels falls through to default
}
}
if len(severityJSON) > 0 && string(severityJSON) != "{}" {
// JSONB stores int keys as string; unmarshal via a string-keyed map
// then convert. JSON does not support non-string object keys, so
// the wire representation is e.g. {"30":"informational"}.
stringKeyed := map[string]string{}
if err := json.Unmarshal(severityJSON, &stringKeyed); err == nil {
converted := make(map[int]string, len(stringKeyed))
for k, v := range stringKeyed {
var threshold int
if _, scanErr := fmt.Sscanf(k, "%d", &threshold); scanErr == nil {
converted[threshold] = v
}
}
policy.AlertSeverityMap = converted
}
}
return &policy, nil
}
// marshalSeverityMap converts the domain's int-keyed map into the
// string-keyed form Postgres JSONB stores. Mirror of the inverse
// conversion in scanRenewalPolicy. Returns "{}" for nil/empty maps so
// the DB never sees null where NOT NULL is required.
func marshalSeverityMap(m map[int]string) ([]byte, error) {
if len(m) == 0 {
return []byte("{}"), nil
}
stringKeyed := make(map[string]string, len(m))
for k, v := range m {
stringKeyed[fmt.Sprintf("%d", k)] = v
}
return json.Marshal(stringKeyed)
}
// marshalAlertChannels marshals the channel matrix as JSONB. nil/empty
// returns "{}" so the DB NOT NULL constraint is satisfied.
func marshalAlertChannels(m map[string][]string) ([]byte, error) {
if len(m) == 0 {
return []byte("{}"), nil
}
return json.Marshal(m)
}
// Get retrieves a renewal policy by ID.
func (r *RenewalPolicyRepository) Get(ctx context.Context, id string) (*domain.RenewalPolicy, error) {
row := r.db.QueryRowContext(ctx, `SELECT `+renewalPolicyColumns+` FROM renewal_policies WHERE id = $1`, id)
policy, err := scanRenewalPolicy(row)
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
return nil, fmt.Errorf("renewal policy not found: %w", repository.ErrNotFound)
}
return nil, fmt.Errorf("failed to query renewal policy: %w", err)
}
return policy, nil
}
// List returns all renewal policies, ordered by name (matches the index on
// renewal_policies.name from migration 000001 so ORDER BY is index-served).
func (r *RenewalPolicyRepository) List(ctx context.Context) ([]*domain.RenewalPolicy, error) {
rows, err := r.db.QueryContext(ctx, `SELECT `+renewalPolicyColumns+` FROM renewal_policies ORDER BY name`)
if err != nil {
return nil, fmt.Errorf("failed to query renewal policies: %w", err)
}
defer rows.Close()
var policies []*domain.RenewalPolicy
for rows.Next() {
policy, err := scanRenewalPolicy(rows)
if err != nil {
return nil, fmt.Errorf("failed to scan renewal policy: %w", err)
}
policies = append(policies, policy)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("error iterating renewal policy rows: %w", err)
}
return policies, nil
}
// slugRegex matches non-alphanumeric characters that slugifyPolicyName strips.
var slugRegex = regexp.MustCompile(`[^a-z0-9-]+`)
// slugifyPolicyName produces `rp-<slug>` for an auto-generated policy ID.
// Slug: lowercase, spaces→hyphens, non-alphanumeric stripped, trimmed to 64
// chars. Matches the existing seed convention (rp-default, rp-standard,
// rp-urgent). Collision resolution is handled by Create's retry loop.
func slugifyPolicyName(name string) string {
slug := strings.ToLower(strings.TrimSpace(name))
slug = strings.ReplaceAll(slug, " ", "-")
slug = slugRegex.ReplaceAllString(slug, "")
slug = strings.Trim(slug, "-")
if slug == "" {
slug = "policy"
}
if len(slug) > 64 {
slug = slug[:64]
}
return "rp-" + slug
}
// isUniqueViolation reports whether err is a PostgreSQL 23505 unique_violation.
// Used by Create/Update to translate name-collision errors onto the typed
// ErrRenewalPolicyDuplicateName sentinel.
func isUniqueViolation(err error) bool {
var pqErr *pq.Error
return errors.As(err, &pqErr) && pqErr.Code == "23505"
}
// isForeignKeyViolation reports whether err is a PostgreSQL 23503
// foreign_key_violation. Used by Delete to translate ON DELETE RESTRICT
// failures onto the typed ErrRenewalPolicyInUse sentinel.
func isForeignKeyViolation(err error) bool {
var pqErr *pq.Error
return errors.As(err, &pqErr) && pqErr.Code == "23503"
}
// Create inserts a new renewal policy. If policy.ID is empty, auto-generates
// `rp-<slug(name)>` with -2/-3/... suffixes on collision (up to 10 attempts).
// Returns ErrRenewalPolicyDuplicateName on pg 23505 (name collision).
//
// alert_thresholds_days is marshaled to JSONB here rather than relying on the
// DB default because the service layer already applies DefaultAlertThresholds
// for empty input — the DB default is a safety net, not the primary path.
func (r *RenewalPolicyRepository) Create(ctx context.Context, policy *domain.RenewalPolicy) error {
if policy == nil {
return errors.New("renewal policy is nil")
}
thresholdsJSON, err := json.Marshal(policy.AlertThresholdsDays)
if err != nil {
return fmt.Errorf("failed to marshal alert thresholds: %w", err)
}
channelsJSON, err := marshalAlertChannels(policy.AlertChannels)
if err != nil {
return fmt.Errorf("failed to marshal alert channels: %w", err)
}
severityJSON, err := marshalSeverityMap(policy.AlertSeverityMap)
if err != nil {
return fmt.Errorf("failed to marshal alert severity map: %w", err)
}
// ID auto-generation with collision retry. We attempt up to 10 suffix
// variants (rp-foo, rp-foo-2, ..., rp-foo-10) before giving up — the
// 23505 error the caller gets back past that point is on Name (their
// job to fix) rather than on a slug-collision we swallowed.
baseID := policy.ID
if baseID == "" {
baseID = slugifyPolicyName(policy.Name)
}
insertSQL := `
INSERT INTO renewal_policies (
id, name, renewal_window_days, auto_renew, max_retries,
retry_interval_seconds, alert_thresholds_days,
alert_channels, alert_severity_map,
created_at, updated_at
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, NOW(), NOW())
RETURNING ` + renewalPolicyColumns
maxAttempts := 10
if policy.ID != "" {
// Caller supplied a specific ID — no collision-retry, just one shot.
maxAttempts = 1
}
for attempt := 1; attempt <= maxAttempts; attempt++ {
candidateID := baseID
if attempt > 1 {
candidateID = fmt.Sprintf("%s-%d", baseID, attempt)
}
row := r.db.QueryRowContext(ctx, insertSQL,
candidateID, policy.Name, policy.RenewalWindowDays, policy.AutoRenew,
policy.MaxRetries, policy.RetryInterval, thresholdsJSON,
channelsJSON, severityJSON,
)
inserted, scanErr := scanRenewalPolicy(row)
if scanErr == nil {
*policy = *inserted
return nil
}
if isUniqueViolation(scanErr) {
// Determine which unique constraint — if it's the name UNIQUE
// we can't recover (caller has to pick a new name); if it's the
// primary-key slug collision we loop to the next suffix.
var pqErr *pq.Error
errors.As(scanErr, &pqErr)
// Postgres reports the constraint name in pqErr.Constraint;
// renewal_policies_name_key is the name UNIQUE, renewal_policies_pkey
// is the PK. Name collision is terminal, PK collision is retryable.
if pqErr.Constraint != "" && !strings.Contains(pqErr.Constraint, "pkey") {
return repository.ErrRenewalPolicyDuplicateName
}
// PK collision — try next suffix.
continue
}
return fmt.Errorf("failed to insert renewal policy: %w", scanErr)
}
// Exhausted retry budget on PK collisions — surface as duplicate so the
// caller at least gets a 409 rather than a mysterious 500.
return repository.ErrRenewalPolicyDuplicateName
}
// Update modifies an existing renewal policy by ID. Returns an error wrapping
// sql.ErrNoRows when id is unknown (detected by RETURNING returning zero rows),
// or ErrRenewalPolicyDuplicateName on pg 23505 (name collision with another row).
func (r *RenewalPolicyRepository) Update(ctx context.Context, id string, policy *domain.RenewalPolicy) error {
if policy == nil {
return errors.New("renewal policy is nil")
}
thresholdsJSON, err := json.Marshal(policy.AlertThresholdsDays)
if err != nil {
return fmt.Errorf("failed to marshal alert thresholds: %w", err)
}
channelsJSON, err := marshalAlertChannels(policy.AlertChannels)
if err != nil {
return fmt.Errorf("failed to marshal alert channels: %w", err)
}
severityJSON, err := marshalSeverityMap(policy.AlertSeverityMap)
if err != nil {
return fmt.Errorf("failed to marshal alert severity map: %w", err)
}
row := r.db.QueryRowContext(ctx, `
UPDATE renewal_policies SET
name = $2,
renewal_window_days = $3,
auto_renew = $4,
max_retries = $5,
retry_interval_seconds = $6,
alert_thresholds_days = $7,
alert_channels = $8,
alert_severity_map = $9,
updated_at = NOW()
WHERE id = $1
RETURNING `+renewalPolicyColumns,
id, policy.Name, policy.RenewalWindowDays, policy.AutoRenew,
policy.MaxRetries, policy.RetryInterval, thresholdsJSON,
channelsJSON, severityJSON,
)
updated, err := scanRenewalPolicy(row)
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
return fmt.Errorf("renewal policy not found: %w", repository.ErrNotFound)
}
if isUniqueViolation(err) {
return repository.ErrRenewalPolicyDuplicateName
}
return fmt.Errorf("failed to update renewal policy: %w", err)
}
*policy = *updated
return nil
}
// Delete removes a renewal policy by ID. Returns ErrRenewalPolicyInUse when
// the policy is still referenced by rows in managed_certificates (pg 23503
// foreign_key_violation against the ON DELETE RESTRICT FK from
// managed_certificates.renewal_policy_id). Returns an error wrapping
// sql.ErrNoRows when id is unknown.
func (r *RenewalPolicyRepository) Delete(ctx context.Context, id string) error {
result, err := r.db.ExecContext(ctx, `DELETE FROM renewal_policies WHERE id = $1`, id)
if err != nil {
if isForeignKeyViolation(err) {
return repository.ErrRenewalPolicyInUse
}
return fmt.Errorf("failed to delete renewal policy: %w", err)
}
rows, err := result.RowsAffected()
if err != nil {
return fmt.Errorf("failed to read RowsAffected for delete: %w", err)
}
if rows == 0 {
return fmt.Errorf("renewal policy not found: %w", repository.ErrNotFound)
}
return nil
}