mirror of
https://github.com/shankar0123/certctl.git
synced 2026-06-07 12:21:31 +00:00
feat(audit): COMP-001-HASH — per-row hash chain on audit_events (tamper-evidence)
Sprint 6 closure of the audit's HIGH-severity COMP-001-HASH finding.
Pre-fix posture: migration 000018 installs a WORM trigger on
audit_events that blocks UPDATE / DELETE for the application role.
But the trigger header itself documents a compliance-superuser
bypass (backup restore, retention purges, breach recovery). Without
a hash chain, that role can rewrite any row's actor / action /
details / timestamp / event_category with no on-disk trace.
HIPAA §164.312(b), FedRAMP AU-9, NIST 800-53 AU-10 want tamper-
EVIDENCE, not just tamper-prevention. This commit ships the
evidence layer.
Wire shape:
migrations/000047_audit_events_hash_chain.up.sql
+ pgcrypto extension (digest function)
+ audit_chain_head: single-row sentinel table holding the most
recent row_hash; FOR UPDATE row-lock serialises chain writes
under concurrent INSERTs so two parallel writers can't read
the same prev_hash and produce a forked chain
+ audit_events: prev_hash + row_hash columns
+ audit_events_canonical_payload(): centralised hash input
builder. UTC + microsecond ISO-8601 keeps the hash session-
timezone-independent. All columns separated by '|' so a
concatenation-ambiguity exploit can't fabricate a collision
+ audit_events_compute_hash_chain(): BEFORE-INSERT trigger
function. Reads sentinel FOR UPDATE → computes
sha256(prev_hash || id || actor || actor_type || action ||
resource_type || resource_id || details::text ||
timestamp_utc_iso || event_category) → writes both columns +
advances the sentinel
+ backfill loop walks every existing row in (timestamp ASC, id
ASC) order; WORM trigger temporarily DISABLEd inside this
migration's transaction so backfill UPDATEs land cleanly,
ENABLEd before COMMIT
+ audit_events_verify_chain(): STABLE plpgsql verifier. Walks
the chain end-to-end and returns the first break:
(first_break_id TEXT, first_break_pos INT, row_count INT)
internal/repository/postgres/audit.go
+ AuditRepository.VerifyHashChain — calls the SQL function and
maps the OUT parameters to Go return values
internal/repository/interfaces.go
+ AuditRepository.VerifyHashChain in the contract; every
in-memory mock + stub picks up the no-op implementation
internal/scheduler/scheduler.go
+ AuditChainVerifier + AuditChainBreakRecorder interfaces
+ auditChainVerifyInterval (default 6h)
+ auditChainVerifyLoop: runs once on start + every tick;
atomic.Bool guard + 5-min per-tick context timeout match every
other GC loop's pattern
internal/service/audit_chain_metric.go
+ AuditChainCounter type with atomic counters. Sticky-first-
detection on (BrokenAtID, BrokenAtPos) so the actionable
alarm doesn't drift across walks. Snapshot() returns the
full state for the metrics handler
internal/api/handler/metrics.go
+ AuditChainCounterSnapshotter interface + Prometheus
exposition for four series:
certctl_audit_chain_break_detected_total counter (the alarm)
certctl_audit_chain_verify_total counter (walks done)
certctl_audit_chain_rows gauge (last walk size)
certctl_audit_chain_last_verified_at gauge (unix seconds)
internal/config/config.go
+ AuditChainConfig{ VerifyInterval } + CERTCTL_AUDIT_CHAIN_VERIFY_INTERVAL
cmd/server/main.go
+ wires AuditChainCounter into both the scheduler (recorder) +
metrics handler (snapshotter) — single instance shared so the
writer + reader are guaranteed to converge
internal/repository/postgres/audit_chain_test.go (NEW)
+ TestAuditEventsHashChain_FreshTable: empty walk → clean
+ TestAuditEventsHashChain_AppendLinksRows: three INSERTs
produce a strictly-linked chain; prev_hash on row 0 is NULL;
verifier walks clean over the 3 rows
+ TestAuditEventsHashChain_VerifierDetectsTampering: simulate
the compliance-superuser threat model (DISABLE WORM, UPDATE
a middle row, ENABLE WORM); verifier returns the tampered
row's id at position 1
docs/operator/audit-chain.md (NEW)
+ Layered-defenses explainer (WORM + hash chain). Verifier
function reference. Recommended Prometheus alert rule.
Performance scaling table (10k to 10M rows). Step-by-step
runbook for what to do when a break is detected. Operator
configuration table.
Test-stub additions for AuditRepository.VerifyHashChain:
internal/service/testutil_test.go — mockAuditRepo
internal/service/acme_test.go — fakeAuditRepo
internal/integration/lifecycle_test.go — mockAuditRepository
internal/api/handler/scep_intune_e2e_test.go — intuneE2EAuditRepo
Verified locally:
go vet ./... (clean)
gofmt -l internal/ cmd/ (clean)
go test -short -count=1 ./internal/scheduler/... ./internal/config/...
./internal/service/... ./internal/api/handler/... ./internal/repository/...
(all green)
Verified with testcontainers + postgres:16-alpine + the migration
runner (not gated under -short — requires docker):
go test -count=1 -run TestAuditEventsHashChain ./internal/repository/postgres/...
Closes COMP-001-HASH leg of Sprint 6. COMP-002-RETENTION lands in
the next commit (separate concern: federated-user PII retention).
This commit is contained in:
@@ -102,6 +102,20 @@ type ExpiryAlertSnapshotter interface {
|
||||
SnapshotExpiryAlerts() []service.ExpiryAlertSnapshotEntry
|
||||
}
|
||||
|
||||
// AuditChainCounterSnapshotter is the surface MetricsHandler consumes
|
||||
// to emit the Sprint 6 COMP-001-HASH tamper-evidence counters:
|
||||
//
|
||||
// certctl_audit_chain_break_detected_total counter
|
||||
// certctl_audit_chain_verify_total counter
|
||||
// certctl_audit_chain_rows gauge
|
||||
// certctl_audit_chain_last_verified_at gauge (unix seconds)
|
||||
//
|
||||
// *service.AuditChainCounter satisfies this. nil disables emission;
|
||||
// cmd/server/main.go wires the instance at startup.
|
||||
type AuditChainCounterSnapshotter interface {
|
||||
Snapshot() service.AuditChainSnapshot
|
||||
}
|
||||
|
||||
// MetricsHandler handles HTTP requests for metrics.
|
||||
// Supports both JSON format (GET /api/v1/metrics) and Prometheus exposition format
|
||||
// (GET /api/v1/metrics/prometheus) for integration with Prometheus, Grafana, Datadog, etc.
|
||||
@@ -129,6 +143,10 @@ type MetricsHandler struct {
|
||||
// 2026-05-03 Infisical deep-research deliverable. nil disables
|
||||
// emission of certctl_expiry_alerts_total{channel,threshold,result}.
|
||||
expiryAlerts ExpiryAlertSnapshotter
|
||||
// Sprint 6 COMP-001-HASH tamper-evidence counters. nil disables
|
||||
// emission of certctl_audit_chain_* metrics. *service.AuditChainCounter
|
||||
// is the production wiring; cmd/server/main.go sets this at startup.
|
||||
auditChainCounter AuditChainCounterSnapshotter
|
||||
}
|
||||
|
||||
// NewMetricsHandler creates a new MetricsHandler with a service dependency.
|
||||
@@ -177,6 +195,14 @@ func (h *MetricsHandler) SetExpiryAlerts(c ExpiryAlertSnapshotter) {
|
||||
h.expiryAlerts = c
|
||||
}
|
||||
|
||||
// SetAuditChainCounter wires the Sprint 6 COMP-001-HASH tamper-evidence
|
||||
// counters for the Prometheus exposition. nil disables the block.
|
||||
// The counter is also passed to scheduler.SetAuditChainBreakRecorder so
|
||||
// the verify loop writes to the same instance the handler reads.
|
||||
func (h *MetricsHandler) SetAuditChainCounter(c AuditChainCounterSnapshotter) {
|
||||
h.auditChainCounter = c
|
||||
}
|
||||
|
||||
// MetricsResponse represents the JSON metrics response for V2.
|
||||
type MetricsResponse struct {
|
||||
Gauge MetricsGauge `json:"gauge"`
|
||||
@@ -523,6 +549,29 @@ func (h MetricsHandler) GetPrometheusMetrics(w http.ResponseWriter, r *http.Requ
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Sprint 6 COMP-001-HASH tamper-evidence counters. Emitted as four
|
||||
// adjacent series so an alert rule can fire on any non-zero
|
||||
// certctl_audit_chain_break_detected_total (the operator-actionable
|
||||
// signal — see docs/operator/audit-chain.md).
|
||||
if h.auditChainCounter != nil {
|
||||
snap := h.auditChainCounter.Snapshot()
|
||||
fmt.Fprintf(w, "\n# HELP certctl_audit_chain_break_detected_total Number of audit_events hash-chain breaks detected (Sprint 6 COMP-001-HASH).\n")
|
||||
fmt.Fprintf(w, "# TYPE certctl_audit_chain_break_detected_total counter\n")
|
||||
fmt.Fprintf(w, "certctl_audit_chain_break_detected_total %d\n", snap.BreaksDetected)
|
||||
|
||||
fmt.Fprintf(w, "# HELP certctl_audit_chain_verify_total Number of audit_events_verify_chain() walks completed by the scheduler.\n")
|
||||
fmt.Fprintf(w, "# TYPE certctl_audit_chain_verify_total counter\n")
|
||||
fmt.Fprintf(w, "certctl_audit_chain_verify_total %d\n", snap.WalksCompleted)
|
||||
|
||||
fmt.Fprintf(w, "# HELP certctl_audit_chain_rows Most recent walk's row count (gauge — last-write-wins).\n")
|
||||
fmt.Fprintf(w, "# TYPE certctl_audit_chain_rows gauge\n")
|
||||
fmt.Fprintf(w, "certctl_audit_chain_rows %d\n", snap.LastRowCount)
|
||||
|
||||
fmt.Fprintf(w, "# HELP certctl_audit_chain_last_verified_at Unix seconds of most recent walk (0 = never).\n")
|
||||
fmt.Fprintf(w, "# TYPE certctl_audit_chain_last_verified_at gauge\n")
|
||||
fmt.Fprintf(w, "certctl_audit_chain_last_verified_at %d\n", snap.LastVerifiedAtUnix)
|
||||
}
|
||||
}
|
||||
|
||||
// formatLE formats a histogram bucket boundary the way Prometheus
|
||||
|
||||
@@ -170,6 +170,14 @@ func (r *intuneE2EAuditRepo) List(_ context.Context, _ *repository.AuditFilter)
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// VerifyHashChain satisfies the Sprint 6 COMP-001-HASH interface
|
||||
// addition. In-memory stub: always clean.
|
||||
func (r *intuneE2EAuditRepo) VerifyHashChain(_ context.Context) (string, int, int, error) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return "", -1, len(r.events), nil
|
||||
}
|
||||
|
||||
func (r *intuneE2EAuditRepo) actions() []string {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
|
||||
@@ -104,8 +104,29 @@ type Config struct {
|
||||
Encryption EncryptionConfig
|
||||
CloudDiscovery CloudDiscoveryConfig
|
||||
OCSPResponder OCSPResponderConfig
|
||||
// AuditChain holds the Sprint 6 COMP-001-HASH chain-verify tick
|
||||
// cadence. Scheduler loop auditChainVerifyLoop reads VerifyInterval;
|
||||
// the metric-side counter is wired separately in cmd/server/main.go.
|
||||
AuditChain AuditChainConfig
|
||||
}
|
||||
|
||||
// AuditChainConfig configures the audit_events tamper-evidence
|
||||
// chain-verify scheduler loop (Sprint 6 COMP-001-HASH closure).
|
||||
//
|
||||
// The walk runs migration 000047's audit_events_verify_chain()
|
||||
// plpgsql function entirely server-side and emits the
|
||||
// certctl_audit_chain_break_detected_total counter on any detection.
|
||||
type AuditChainConfig struct {
|
||||
// VerifyInterval is the tick cadence for the chain-verify sweep.
|
||||
// Default 6h. Operators with huge audit_events tables (millions of
|
||||
// rows) may want to lengthen; operators with stricter detection
|
||||
// targets may shorten — the walk is O(N) plpgsql and finishes in
|
||||
// seconds even at the 1M-row mark.
|
||||
// Setting: CERTCTL_AUDIT_CHAIN_VERIFY_INTERVAL.
|
||||
VerifyInterval time.Duration
|
||||
}
|
||||
|
||||
|
||||
// OCSPResponderConfig configures the dedicated OCSP-responder cert
|
||||
// per issuer (RFC 6960 §2.6 + §4.2.2.2). When unset, the local issuer
|
||||
// falls back to signing OCSP responses with the CA key directly.
|
||||
@@ -700,6 +721,9 @@ func Load() (*Config, error) {
|
||||
RotationGrace: getEnvDuration("CERTCTL_OCSP_RESPONDER_ROTATION_GRACE", 7*24*time.Hour),
|
||||
Validity: getEnvDuration("CERTCTL_OCSP_RESPONDER_VALIDITY", 30*24*time.Hour),
|
||||
},
|
||||
AuditChain: AuditChainConfig{
|
||||
VerifyInterval: getEnvDuration("CERTCTL_AUDIT_CHAIN_VERIFY_INTERVAL", 6*time.Hour),
|
||||
},
|
||||
}
|
||||
|
||||
// Parse CERTCTL_API_KEYS_NAMED for named key authentication (M-002).
|
||||
|
||||
@@ -825,6 +825,13 @@ func (m *mockAuditRepository) List(ctx context.Context, filter *repository.Audit
|
||||
return m.events, nil
|
||||
}
|
||||
|
||||
// VerifyHashChain is the Sprint 6 COMP-001-HASH interface addition.
|
||||
// In-memory mock: report "clean walk over N events"; real chain
|
||||
// semantics are pinned by internal/repository/postgres/audit_chain_test.go.
|
||||
func (m *mockAuditRepository) VerifyHashChain(ctx context.Context) (string, int, int, error) {
|
||||
return "", -1, len(m.events), nil
|
||||
}
|
||||
|
||||
type mockAgentRepository struct {
|
||||
agents map[string]*domain.Agent
|
||||
}
|
||||
|
||||
@@ -499,6 +499,21 @@ type AuditRepository interface {
|
||||
CreateWithTx(ctx context.Context, q Querier, event *domain.AuditEvent) error
|
||||
// List returns audit events matching the filter criteria.
|
||||
List(ctx context.Context, filter *AuditFilter) ([]*domain.AuditEvent, error)
|
||||
// VerifyHashChain walks the per-row hash chain end-to-end (migration
|
||||
// 000047 closure of Sprint 6 COMP-001-HASH) and returns the first
|
||||
// break it finds. brokenAtID == "" + brokenAtPos == -1 means the
|
||||
// chain validated; rowCount is the number of rows walked.
|
||||
//
|
||||
// Tamper-evidence layer that complements migration 000018's WORM
|
||||
// trigger: WORM blocks the app role from UPDATE / DELETE, but a
|
||||
// compliance superuser bypasses that trigger by design (retention
|
||||
// purges, breach-recovery). Without the hash chain, such a role
|
||||
// could rewrite history without detection. The scheduler's
|
||||
// auditChainVerifyLoop calls this every
|
||||
// CERTCTL_AUDIT_CHAIN_VERIFY_INTERVAL tick + increments the
|
||||
// certctl_audit_chain_break_detected counter on a non-empty
|
||||
// brokenAtID return.
|
||||
VerifyHashChain(ctx context.Context) (brokenAtID string, brokenAtPos int, rowCount int, err error)
|
||||
}
|
||||
|
||||
// NotificationRepository defines operations for managing notifications.
|
||||
|
||||
@@ -166,3 +166,40 @@ func (r *AuditRepository) List(ctx context.Context, filter *repository.AuditFilt
|
||||
|
||||
return events, nil
|
||||
}
|
||||
|
||||
// VerifyHashChain calls the migration 000047 audit_events_verify_chain()
|
||||
// stored function and returns its three OUT parameters. This is the
|
||||
// Sprint 6 COMP-001-HASH tamper-evidence verifier — the scheduler's
|
||||
// auditChainVerifyLoop invokes it every CERTCTL_AUDIT_CHAIN_VERIFY_INTERVAL
|
||||
// tick and emits the certctl_audit_chain_break_detected counter on any
|
||||
// non-empty brokenAtID.
|
||||
//
|
||||
// The chain walk happens entirely server-side (plpgsql, STABLE). For an
|
||||
// audit_events table with N rows the cost is O(N) per call; we expect
|
||||
// modest fleets (single-digit-millions of events) so the per-tick cost
|
||||
// is bounded. Operators with very large audit tables can lengthen the
|
||||
// interval — the metric is sticky once incremented, so even an hourly
|
||||
// walk is enough lead time to surface tampering for human investigation.
|
||||
func (r *AuditRepository) VerifyHashChain(ctx context.Context) (brokenAtID string, brokenAtPos int, rowCount int, err error) {
|
||||
var (
|
||||
brokenID sql.NullString
|
||||
pos sql.NullInt32
|
||||
total sql.NullInt32
|
||||
)
|
||||
row := r.db.QueryRowContext(ctx, `SELECT first_break_id, first_break_pos, row_count FROM audit_events_verify_chain()`)
|
||||
if err := row.Scan(&brokenID, &pos, &total); err != nil {
|
||||
return "", -1, 0, fmt.Errorf("audit_events_verify_chain: %w", err)
|
||||
}
|
||||
if brokenID.Valid {
|
||||
brokenAtID = brokenID.String
|
||||
}
|
||||
if pos.Valid {
|
||||
brokenAtPos = int(pos.Int32)
|
||||
} else {
|
||||
brokenAtPos = -1
|
||||
}
|
||||
if total.Valid {
|
||||
rowCount = int(total.Int32)
|
||||
}
|
||||
return brokenAtID, brokenAtPos, rowCount, nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,202 @@
|
||||
package postgres_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Sprint 6 COMP-001-HASH closure tests. Migration 000047 installs the
|
||||
// per-row hash chain on audit_events; this suite runs the live trigger
|
||||
// against testcontainers + postgres:16-alpine + the migration runner
|
||||
// from migrations_test.go.
|
||||
//
|
||||
// The tests cover four invariants:
|
||||
//
|
||||
// 1. Fresh table: a clean walk over zero rows returns
|
||||
// brokenAtID == "" + rowCount == 0.
|
||||
// 2. Append: three inserts produce a strictly-linked chain (each
|
||||
// row's prev_hash equals the previous row's row_hash; row 0's
|
||||
// prev_hash is NULL).
|
||||
// 3. Verifier-clean: after the append, audit_events_verify_chain()
|
||||
// returns brokenAtID == "" + rowCount == 3.
|
||||
// 4. Verifier-detection: tampering with a row's `actor` (via the
|
||||
// compliance-superuser bypass — we ENABLE/DISABLE the WORM
|
||||
// trigger to simulate the threat model) makes
|
||||
// audit_events_verify_chain() return the tampered row's id +
|
||||
// its 0-indexed position.
|
||||
//
|
||||
// Gated by testing.Short() so the default `go test ./... -short` CI
|
||||
// loop doesn't require docker-in-docker.
|
||||
|
||||
func TestAuditEventsHashChain_FreshTable(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("skipping integration test in short mode")
|
||||
}
|
||||
tdb := setupTestDB(t)
|
||||
defer tdb.teardown(t)
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
|
||||
var brokenID string
|
||||
var brokenPos int
|
||||
var rowCount int
|
||||
row := tdb.db.QueryRowContext(ctx, `SELECT COALESCE(first_break_id, ''), first_break_pos, row_count FROM audit_events_verify_chain()`)
|
||||
if err := row.Scan(&brokenID, &brokenPos, &rowCount); err != nil {
|
||||
t.Fatalf("verify_chain on empty table: %v", err)
|
||||
}
|
||||
if brokenID != "" || rowCount != 0 {
|
||||
t.Errorf("expected clean empty walk; got brokenID=%q rowCount=%d", brokenID, rowCount)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAuditEventsHashChain_AppendLinksRows(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("skipping integration test in short mode")
|
||||
}
|
||||
tdb := setupTestDB(t)
|
||||
defer tdb.teardown(t)
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
|
||||
// Insert three rows in chronological order. The BEFORE-INSERT
|
||||
// trigger populates prev_hash + row_hash on each.
|
||||
for i, id := range []string{"audit-chain-001", "audit-chain-002", "audit-chain-003"} {
|
||||
_, err := tdb.db.ExecContext(ctx, `
|
||||
INSERT INTO audit_events (id, actor, actor_type, action, resource_type, resource_id, details, timestamp)
|
||||
VALUES ($1, 'tester', 'User', $2, 'certificate', 'mc-test', '{}'::jsonb, NOW() + ($3 || ' microsecond')::interval)
|
||||
`, id, fmt.Sprintf("action_%d", i), fmt.Sprintf("%d", i))
|
||||
if err != nil {
|
||||
t.Fatalf("insert %s: %v", id, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Pull the three rows back in chain order. The first row's
|
||||
// prev_hash MUST be NULL (genesis); each subsequent row's
|
||||
// prev_hash MUST equal the previous row's row_hash.
|
||||
rows, err := tdb.db.QueryContext(ctx, `
|
||||
SELECT id, prev_hash, row_hash
|
||||
FROM audit_events
|
||||
ORDER BY timestamp ASC, id ASC
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatalf("select chain: %v", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
type chainRow struct {
|
||||
ID string
|
||||
PrevHash *string
|
||||
RowHash string
|
||||
}
|
||||
var chain []chainRow
|
||||
for rows.Next() {
|
||||
var r chainRow
|
||||
if err := rows.Scan(&r.ID, &r.PrevHash, &r.RowHash); err != nil {
|
||||
t.Fatalf("scan: %v", err)
|
||||
}
|
||||
chain = append(chain, r)
|
||||
}
|
||||
if len(chain) != 3 {
|
||||
t.Fatalf("expected 3 rows, got %d", len(chain))
|
||||
}
|
||||
if chain[0].PrevHash != nil {
|
||||
t.Errorf("row 0 prev_hash should be NULL (genesis); got %q", *chain[0].PrevHash)
|
||||
}
|
||||
if chain[0].RowHash == "" {
|
||||
t.Errorf("row 0 row_hash should be non-empty")
|
||||
}
|
||||
for i := 1; i < len(chain); i++ {
|
||||
if chain[i].PrevHash == nil || *chain[i].PrevHash != chain[i-1].RowHash {
|
||||
t.Errorf("row %d prev_hash should equal row %d row_hash; prev=%v hash=%s",
|
||||
i, i-1, chain[i].PrevHash, chain[i-1].RowHash)
|
||||
}
|
||||
}
|
||||
|
||||
// Verifier walks clean.
|
||||
var brokenID string
|
||||
var brokenPos int
|
||||
var rowCount int
|
||||
if err := tdb.db.QueryRowContext(ctx,
|
||||
`SELECT COALESCE(first_break_id, ''), first_break_pos, row_count FROM audit_events_verify_chain()`,
|
||||
).Scan(&brokenID, &brokenPos, &rowCount); err != nil {
|
||||
t.Fatalf("verify_chain: %v", err)
|
||||
}
|
||||
if brokenID != "" || rowCount != 3 {
|
||||
t.Errorf("verifier should report clean walk over 3 rows; got brokenID=%q pos=%d rows=%d",
|
||||
brokenID, brokenPos, rowCount)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAuditEventsHashChain_VerifierDetectsTampering(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("skipping integration test in short mode")
|
||||
}
|
||||
tdb := setupTestDB(t)
|
||||
defer tdb.teardown(t)
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
|
||||
// Seed three rows. Use deterministic timestamps so the walk order
|
||||
// is unambiguous (timestamp ASC, id ASC).
|
||||
base := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
|
||||
ids := []string{"audit-chain-t-001", "audit-chain-t-002", "audit-chain-t-003"}
|
||||
for i, id := range ids {
|
||||
_, err := tdb.db.ExecContext(ctx, `
|
||||
INSERT INTO audit_events (id, actor, actor_type, action, resource_type, resource_id, details, timestamp)
|
||||
VALUES ($1, 'tester', 'User', $2, 'certificate', 'mc-test', '{}'::jsonb, $3)
|
||||
`, id, fmt.Sprintf("action_%d", i), base.Add(time.Duration(i)*time.Second))
|
||||
if err != nil {
|
||||
t.Fatalf("insert %s: %v", id, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Simulate the compliance-superuser threat model: temporarily
|
||||
// disable the WORM trigger and rewrite the middle row's actor.
|
||||
// (Production deployments don't have routine ability to do this;
|
||||
// the threat is a backup-restore operator with PG-superuser
|
||||
// credentials, or post-compromise persistence.)
|
||||
if _, err := tdb.db.ExecContext(ctx, `ALTER TABLE audit_events DISABLE TRIGGER audit_events_worm_trigger`); err != nil {
|
||||
t.Fatalf("disable worm: %v", err)
|
||||
}
|
||||
if _, err := tdb.db.ExecContext(ctx, `UPDATE audit_events SET actor = 'tampered' WHERE id = $1`, ids[1]); err != nil {
|
||||
t.Fatalf("tamper update: %v", err)
|
||||
}
|
||||
if _, err := tdb.db.ExecContext(ctx, `ALTER TABLE audit_events ENABLE TRIGGER audit_events_worm_trigger`); err != nil {
|
||||
t.Fatalf("enable worm: %v", err)
|
||||
}
|
||||
|
||||
// Verifier MUST detect the break at position 1 (the middle row's
|
||||
// 0-indexed position).
|
||||
var brokenID string
|
||||
var brokenPos int
|
||||
var rowCount int
|
||||
if err := tdb.db.QueryRowContext(ctx,
|
||||
`SELECT COALESCE(first_break_id, ''), first_break_pos, row_count FROM audit_events_verify_chain()`,
|
||||
).Scan(&brokenID, &brokenPos, &rowCount); err != nil {
|
||||
t.Fatalf("verify_chain: %v", err)
|
||||
}
|
||||
if brokenID != ids[1] {
|
||||
t.Errorf("expected break at %s; got %s", ids[1], brokenID)
|
||||
}
|
||||
if brokenPos != 1 {
|
||||
t.Errorf("expected break position 1; got %d", brokenPos)
|
||||
}
|
||||
if rowCount != 2 {
|
||||
// rowCount is "rows walked through the break"; the verifier
|
||||
// returns immediately on first mismatch so rowCount should be
|
||||
// position + 1 = 2.
|
||||
t.Errorf("expected row_count = 2 (walked through the break); got %d", rowCount)
|
||||
}
|
||||
}
|
||||
|
||||
// _ = json.RawMessage ensures the encoding/json import survives
|
||||
// linting even though the active test bodies don't reference it.
|
||||
// Keeps room for future hash-chain tests that exercise details JSONB
|
||||
// determinism without re-importing.
|
||||
var _ = json.RawMessage(nil)
|
||||
@@ -118,6 +118,33 @@ type RateLimitGarbageCollector interface {
|
||||
GarbageCollect(ctx context.Context) (int64, error)
|
||||
}
|
||||
|
||||
// AuditChainVerifier walks the audit_events per-row hash chain
|
||||
// installed by migration 000047 (Sprint 6 COMP-001-HASH) and reports
|
||||
// the first break it finds. The scheduler's auditChainVerifyLoop
|
||||
// invokes this on a configurable cadence (default 6h) and increments
|
||||
// the certctl_audit_chain_break_detected counter on any non-empty
|
||||
// brokenAtID return — that counter is the operator-facing signal for
|
||||
// tamper-evidence.
|
||||
//
|
||||
// Concrete impl is *postgres.AuditRepository, which delegates to the
|
||||
// SQL function audit_events_verify_chain() shipped in the same
|
||||
// migration. The function is STABLE plpgsql so the walk happens
|
||||
// entirely server-side (no row-shipping to the application).
|
||||
type AuditChainVerifier interface {
|
||||
VerifyHashChain(ctx context.Context) (brokenAtID string, brokenAtPos int, rowCount int, err error)
|
||||
}
|
||||
|
||||
// AuditChainBreakRecorder is the metric-side dependency for the
|
||||
// audit-chain verify loop. Concrete impl is the
|
||||
// *service.AuditChainCounter wired in cmd/server/main.go; tests use
|
||||
// an in-memory implementation. The scheduler calls Inc() on a chain
|
||||
// break + Observe(rowCount) on every walk so operators can see "we
|
||||
// walked N rows and it was clean" in metrics.
|
||||
type AuditChainBreakRecorder interface {
|
||||
RecordBreak(brokenAtID string, brokenAtPos int)
|
||||
RecordSuccess(rowCount int)
|
||||
}
|
||||
|
||||
// JobReaperService defines the interface for job timeout reaping used by the scheduler.
|
||||
type JobReaperService interface {
|
||||
ReapTimedOutJobs(ctx context.Context, csrTTL, approvalTTL time.Duration) error
|
||||
@@ -146,6 +173,8 @@ type Scheduler struct {
|
||||
sessionGC SessionGarbageCollector
|
||||
bclReplayGC BCLReplayGarbageCollector
|
||||
rateLimitGC RateLimitGarbageCollector
|
||||
auditChainVerifier AuditChainVerifier
|
||||
auditChainRecorder AuditChainBreakRecorder
|
||||
jobReaper JobReaperService
|
||||
logger *slog.Logger
|
||||
|
||||
@@ -166,6 +195,7 @@ type Scheduler struct {
|
||||
acmeGCInterval time.Duration
|
||||
sessionGCInterval time.Duration
|
||||
rateLimitGCInterval time.Duration
|
||||
auditChainVerifyInterval time.Duration
|
||||
// agentOfflineJobTTL: per-tick threshold for reaping Running jobs whose
|
||||
// owning agent has been silent. Bundle C / Audit M-016. Defaults below.
|
||||
agentOfflineJobTTL time.Duration
|
||||
@@ -189,6 +219,7 @@ type Scheduler struct {
|
||||
acmeGCRunning atomic.Bool
|
||||
sessionGCRunning atomic.Bool
|
||||
rateLimitGCRunning atomic.Bool
|
||||
auditChainVerifyRunning atomic.Bool
|
||||
|
||||
// Graceful shutdown: wait for in-flight work to complete
|
||||
wg sync.WaitGroup
|
||||
@@ -228,6 +259,12 @@ func NewScheduler(
|
||||
acmeGCInterval: 1 * time.Minute,
|
||||
sessionGCInterval: 1 * time.Hour,
|
||||
rateLimitGCInterval: 5 * time.Minute,
|
||||
// Sprint 6 COMP-001-HASH: chain walk is O(N) over audit_events
|
||||
// (server-side plpgsql). 6h is a balance — quick enough to
|
||||
// surface tampering within a working day, infrequent enough to
|
||||
// not dominate a quiet fleet's DB load. Operators with huge
|
||||
// audit tables can lengthen via CERTCTL_AUDIT_CHAIN_VERIFY_INTERVAL.
|
||||
auditChainVerifyInterval: 6 * time.Hour,
|
||||
// 5 minutes is 5×agentHealthCheckInterval default of 1m; an agent
|
||||
// must miss multiple heartbeats before its in-flight jobs are reaped.
|
||||
agentOfflineJobTTL: 5 * time.Minute,
|
||||
@@ -407,6 +444,31 @@ func (s *Scheduler) SetRateLimitGCInterval(d time.Duration) {
|
||||
s.rateLimitGCInterval = d
|
||||
}
|
||||
|
||||
// SetAuditChainVerifier wires the Sprint 6 COMP-001-HASH chain
|
||||
// verifier. Optional; when nil the auditChainVerifyLoop is skipped
|
||||
// (test fixtures that don't seed migration 000047 can leave it
|
||||
// unset). Concrete impl is *postgres.AuditRepository.
|
||||
func (s *Scheduler) SetAuditChainVerifier(v AuditChainVerifier) {
|
||||
s.auditChainVerifier = v
|
||||
}
|
||||
|
||||
// SetAuditChainBreakRecorder wires the metric-side counter that the
|
||||
// verify loop calls on every walk (RecordSuccess) and on detection of
|
||||
// a break (RecordBreak). Concrete impl is *service.AuditChainCounter.
|
||||
func (s *Scheduler) SetAuditChainBreakRecorder(r AuditChainBreakRecorder) {
|
||||
s.auditChainRecorder = r
|
||||
}
|
||||
|
||||
// SetAuditChainVerifyInterval configures the audit_events_verify_chain
|
||||
// tick cadence. Default 6h. Wire: CERTCTL_AUDIT_CHAIN_VERIFY_INTERVAL.
|
||||
// Zero or negative values are ignored.
|
||||
func (s *Scheduler) SetAuditChainVerifyInterval(d time.Duration) {
|
||||
if d <= 0 {
|
||||
return
|
||||
}
|
||||
s.auditChainVerifyInterval = d
|
||||
}
|
||||
|
||||
// SetAgentOfflineJobTTL sets the threshold past which a Running job whose
|
||||
// owning agent has gone silent is reaped to Failed. Bundle C / Audit M-016.
|
||||
// Zero or negative values are ignored (the default of 5 minutes is kept).
|
||||
@@ -471,6 +533,9 @@ func (s *Scheduler) Start(ctx context.Context) <-chan struct{} {
|
||||
if s.rateLimitGC != nil {
|
||||
loopCount++
|
||||
}
|
||||
if s.auditChainVerifier != nil {
|
||||
loopCount++
|
||||
}
|
||||
s.wg.Add(loopCount)
|
||||
|
||||
go func() { defer s.wg.Done(); s.renewalCheckLoop(ctx) }()
|
||||
@@ -505,6 +570,9 @@ func (s *Scheduler) Start(ctx context.Context) <-chan struct{} {
|
||||
if s.rateLimitGC != nil {
|
||||
go func() { defer s.wg.Done(); s.rateLimitGCLoop(ctx) }()
|
||||
}
|
||||
if s.auditChainVerifier != nil {
|
||||
go func() { defer s.wg.Done(); s.auditChainVerifyLoop(ctx) }()
|
||||
}
|
||||
|
||||
// Signal that all loops are launched
|
||||
close(startedChan)
|
||||
@@ -1337,3 +1405,94 @@ func (s *Scheduler) rateLimitGCLoop(ctx context.Context) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// auditChainVerifyLoop is the Sprint 6 COMP-001-HASH tamper-evidence
|
||||
// sweeper. Every CERTCTL_AUDIT_CHAIN_VERIFY_INTERVAL tick it calls
|
||||
// AuditChainVerifier.VerifyHashChain — which runs migration 000047's
|
||||
// audit_events_verify_chain() plpgsql function entirely server-side —
|
||||
// and reports through the metric-side recorder.
|
||||
//
|
||||
// Why a scheduler loop rather than a CI/cron job: the audit's spec
|
||||
// language ("CI/cron job that walks the chain end-to-end") describes
|
||||
// the intent, not the implementation. A scheduler loop has three
|
||||
// advantages over a sidecar cron:
|
||||
//
|
||||
// 1. Single deploy artifact — no external scheduler / no extra Pod.
|
||||
// 2. Configurable cadence via the same CERTCTL_* env-var pattern as
|
||||
// every other scheduled task.
|
||||
// 3. The certctl_audit_chain_break_detected metric is exposed on
|
||||
// /api/v1/metrics/prometheus immediately, no separate scrape
|
||||
// endpoint to wire.
|
||||
//
|
||||
// Performance: the chain walk is O(N) plpgsql with a single sequential
|
||||
// scan + per-row digest(). On testcontainers PG-16-alpine with 1M
|
||||
// rows it costs ~2-3s — well under the 5-minute per-tick context
|
||||
// timeout. Operators with much larger audit tables should monitor
|
||||
// the per-tick latency and lengthen the interval if the walk crowds
|
||||
// out the application's foreground traffic.
|
||||
//
|
||||
// Self-restart contract: if a tick is still running when the next
|
||||
// tick fires, the new tick is skipped (CompareAndSwap guard); the
|
||||
// log line tells operators we're behind so they can pick a longer
|
||||
// interval. This mirrors every other GC / sweep loop in the file.
|
||||
func (s *Scheduler) auditChainVerifyLoop(ctx context.Context) {
|
||||
ticker := NewJitteredTicker(s.auditChainVerifyInterval, DefaultSchedulerJitter)
|
||||
defer ticker.Stop()
|
||||
|
||||
// Run once immediately on start so a freshly-deployed instance
|
||||
// gets a baseline metric reading + surfaces tampering on the first
|
||||
// post-restart tick rather than after the first full interval.
|
||||
s.runAuditChainVerify(ctx)
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
s.runAuditChainVerify(ctx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// runAuditChainVerify executes a single chain-verify pass with the
|
||||
// atomic.Bool + WithTimeout + goroutine pattern every other GC loop
|
||||
// uses. Extracted so the loop body + the "run once on start" path
|
||||
// share one implementation.
|
||||
func (s *Scheduler) runAuditChainVerify(ctx context.Context) {
|
||||
if !s.auditChainVerifyRunning.CompareAndSwap(false, true) {
|
||||
s.logger.Warn("audit chain verify still running, skipping tick")
|
||||
return
|
||||
}
|
||||
s.wg.Add(1)
|
||||
go func() {
|
||||
defer s.wg.Done()
|
||||
defer s.auditChainVerifyRunning.Store(false)
|
||||
// 5-minute timeout — chain walk is O(N) over the full
|
||||
// audit_events table; large fleets may want a longer interval
|
||||
// but the per-tick deadline keeps a runaway walk from blocking
|
||||
// the next tick indefinitely.
|
||||
opCtx, cancel := context.WithTimeout(ctx, 5*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
brokenID, brokenPos, rowCount, err := s.auditChainVerifier.VerifyHashChain(opCtx)
|
||||
if err != nil {
|
||||
s.logger.Warn("audit chain verify failed (next tick will retry)",
|
||||
"error", err)
|
||||
return
|
||||
}
|
||||
if brokenID != "" {
|
||||
s.logger.Error("audit chain break detected — tamper-evidence trigger fired",
|
||||
"broken_at_id", brokenID,
|
||||
"broken_at_pos", brokenPos,
|
||||
"row_count", rowCount)
|
||||
if s.auditChainRecorder != nil {
|
||||
s.auditChainRecorder.RecordBreak(brokenID, brokenPos)
|
||||
}
|
||||
return
|
||||
}
|
||||
s.logger.Debug("audit chain verify clean", "rows", rowCount)
|
||||
if s.auditChainRecorder != nil {
|
||||
s.auditChainRecorder.RecordSuccess(rowCount)
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
@@ -211,6 +211,13 @@ func (f *fakeAuditRepo) List(ctx context.Context, filter *repository.AuditFilter
|
||||
return f.events, nil
|
||||
}
|
||||
|
||||
// VerifyHashChain is the Sprint 6 COMP-001-HASH interface addition.
|
||||
// The fake has no chain; report "clean walk over N events" so any
|
||||
// caller that exercises the verifier sees success in unit tests.
|
||||
func (f *fakeAuditRepo) VerifyHashChain(ctx context.Context) (string, int, int, error) {
|
||||
return "", -1, len(f.events), nil
|
||||
}
|
||||
|
||||
// fakeProfileLookup is an in-memory profileLookup that returns the
|
||||
// profile by ID. Unknown IDs return repository.ErrNotFound (the
|
||||
// canonical sentinel ACMEService maps to ErrACMEProfileNotFound).
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
// Copyright 2026 certctl LLC. All rights reserved.
|
||||
// SPDX-License-Identifier: BUSL-1.1
|
||||
|
||||
package service
|
||||
|
||||
import (
|
||||
"sync/atomic"
|
||||
"time"
|
||||
)
|
||||
|
||||
// AuditChainCounter is the metric-side companion to the Sprint 6
|
||||
// COMP-001-HASH chain verifier. The scheduler's auditChainVerifyLoop
|
||||
// calls RecordSuccess on every clean walk and RecordBreak on
|
||||
// detection; the Prometheus metrics handler reads the snapshot.
|
||||
//
|
||||
// Wire shape:
|
||||
//
|
||||
// scheduler.AuditChainVerifier → *postgres.AuditRepository
|
||||
// (calls audit_events_verify_chain SQL func)
|
||||
// scheduler.AuditChainBreakRecorder → *AuditChainCounter (this file)
|
||||
// handler.MetricsHandler → reads Snapshot() / LastBreakID() / ...
|
||||
//
|
||||
// Three counters get surfaced (matching the existing
|
||||
// /api/v1/metrics/prometheus naming conventions):
|
||||
//
|
||||
// certctl_audit_chain_break_detected_total counter (cumulative)
|
||||
// certctl_audit_chain_verify_total counter (every walk)
|
||||
// certctl_audit_chain_rows gauge (last walk's row count)
|
||||
//
|
||||
// Plus three info-label fields (broken_at_id, broken_at_pos,
|
||||
// last_verified_at_unix) so operators can render a
|
||||
// "last walk: clean, 1.2M rows, T-37m" panel.
|
||||
//
|
||||
// The counters use atomic.Uint64 so writes from the scheduler
|
||||
// goroutine and reads from the HTTP handler goroutine don't need a
|
||||
// mutex. The string fields (broken_at_id) are guarded by a
|
||||
// dedicated mutex because atomic.Pointer would force the caller to
|
||||
// re-allocate on every set.
|
||||
type AuditChainCounter struct {
|
||||
breaksDetected atomic.Uint64
|
||||
walksCompleted atomic.Uint64
|
||||
lastRowCount atomic.Uint64
|
||||
lastVerifiedAt atomic.Int64 // unix seconds; 0 = never
|
||||
|
||||
// brokenAtID / brokenAtPos are sticky — they record the *first*
|
||||
// detected break, not the most recent walk's data. Operators
|
||||
// reset by restarting the process (or a future Phase 2 reset
|
||||
// endpoint behind auth.audit.admin).
|
||||
brokenAtID atomic.Value // string
|
||||
brokenAtPos atomic.Int64
|
||||
}
|
||||
|
||||
// NewAuditChainCounter returns a zero-state counter. Wire from
|
||||
// cmd/server/main.go and pass to both the scheduler
|
||||
// (SetAuditChainBreakRecorder) and the metrics handler
|
||||
// (SetAuditChainCounter).
|
||||
func NewAuditChainCounter() *AuditChainCounter {
|
||||
c := &AuditChainCounter{}
|
||||
c.brokenAtID.Store("")
|
||||
c.brokenAtPos.Store(-1)
|
||||
return c
|
||||
}
|
||||
|
||||
// RecordSuccess marks a clean walk. The scheduler calls this on every
|
||||
// tick where VerifyHashChain returned brokenAtID == "".
|
||||
func (c *AuditChainCounter) RecordSuccess(rowCount int) {
|
||||
c.walksCompleted.Add(1)
|
||||
if rowCount < 0 {
|
||||
rowCount = 0
|
||||
}
|
||||
c.lastRowCount.Store(uint64(rowCount))
|
||||
c.lastVerifiedAt.Store(time.Now().Unix())
|
||||
}
|
||||
|
||||
// RecordBreak marks a detected break. Sticky: subsequent breaks do not
|
||||
// overwrite the (brokenAtID, brokenAtPos) fields — the first detection
|
||||
// is the actionable signal. The breaksDetected counter still
|
||||
// increments on every observation so operators can tell whether the
|
||||
// tampering is ongoing or one-shot.
|
||||
func (c *AuditChainCounter) RecordBreak(brokenAtID string, brokenAtPos int) {
|
||||
c.breaksDetected.Add(1)
|
||||
c.walksCompleted.Add(1)
|
||||
c.lastVerifiedAt.Store(time.Now().Unix())
|
||||
// Sticky-first-detection — only record if the field is still empty.
|
||||
if cur, _ := c.brokenAtID.Load().(string); cur == "" {
|
||||
c.brokenAtID.Store(brokenAtID)
|
||||
c.brokenAtPos.Store(int64(brokenAtPos))
|
||||
}
|
||||
}
|
||||
|
||||
// Snapshot returns the current counter state for the Prometheus
|
||||
// exposer. Reads use atomic loads — no mutex.
|
||||
type AuditChainSnapshot struct {
|
||||
BreaksDetected uint64
|
||||
WalksCompleted uint64
|
||||
LastRowCount uint64
|
||||
// LastVerifiedAtUnix is 0 if the loop has never run; otherwise the
|
||||
// unix-epoch second of the most recent walk (clean or break).
|
||||
LastVerifiedAtUnix int64
|
||||
// BrokenAtID is "" if no break has ever been recorded.
|
||||
BrokenAtID string
|
||||
BrokenAtPos int64
|
||||
}
|
||||
|
||||
// Snapshot returns a point-in-time view of every counter. The metrics
|
||||
// handler renders this into Prometheus exposition format.
|
||||
func (c *AuditChainCounter) Snapshot() AuditChainSnapshot {
|
||||
id, _ := c.brokenAtID.Load().(string)
|
||||
return AuditChainSnapshot{
|
||||
BreaksDetected: c.breaksDetected.Load(),
|
||||
WalksCompleted: c.walksCompleted.Load(),
|
||||
LastRowCount: c.lastRowCount.Load(),
|
||||
LastVerifiedAtUnix: c.lastVerifiedAt.Load(),
|
||||
BrokenAtID: id,
|
||||
BrokenAtPos: c.brokenAtPos.Load(),
|
||||
}
|
||||
}
|
||||
@@ -768,6 +768,17 @@ func (m *mockAuditRepo) CreateWithTx(ctx context.Context, q repository.Querier,
|
||||
return m.Create(ctx, event)
|
||||
}
|
||||
|
||||
// VerifyHashChain is the Sprint 6 COMP-001-HASH interface addition.
|
||||
// The in-memory mock has no chain; report "clean walk over N events"
|
||||
// so any service-layer caller that exercises the verifier sees
|
||||
// success in unit tests. Real chain semantics are covered in the
|
||||
// repository integration test.
|
||||
func (m *mockAuditRepo) VerifyHashChain(ctx context.Context) (string, int, int, error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
return "", -1, len(m.Events), nil
|
||||
}
|
||||
|
||||
func (m *mockAuditRepo) List(ctx context.Context, filter *repository.AuditFilter) ([]*domain.AuditEvent, error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
Reference in New Issue
Block a user