fix(test-compose): set CERTCTL_AGENT_BOOTSTRAP_TOKEN placeholder (deploy-vendor-e2e job)

deploy-vendor-e2e was hidden behind the go-build-and-test failure; once that cleared (b1ca046), the vendor-e2e job actually booted certctl-test- server for the first time in a while and hit the Sprint 5 ACQ RED-003 fallout: Failed to load configuration: phase-2 SEC-H1 fail-closed guard: CERTCTL_AGENT_BOOTSTRAP_TOKEN is empty and CERTCTL_AGENT_BOOTSTRAP_TOKEN_DENY_EMPTY=true — refuse to start. The Sprint 5 RED-003 closure flipped DENY_EMPTY's default from false→true in production code, but the test compose stack never set a token. The fail-closed guard (internal/config/config.go:1054) refuses to start unless one of: - CERTCTL_AGENT_BOOTSTRAP_TOKEN is non-empty, OR - CERTCTL_DEMO_MODE_ACK=true (demo-mode override), OR - CERTCTL_AGENT_BOOTSTRAP_TOKEN_DENY_EMPTY=false (warn-mode escape hatch for v2.1.x→v2.2.x upgrade window) This is the e2e TEST stack with production-like auth posture (CERTCTL_AUTH_TYPE=api-key), not a demo stack. The right fix is the first option — set a deterministic placeholder token. Picking the warn-mode escape hatch would silently test the wrong posture; picking DEMO_MODE_ACK would also flip CERTCTL_AUTH_TYPE expectations. Also fixed deploy/ENVIRONMENTS.md: the entry still said 'default flip to true scheduled for v2.2.0', which became stale on 2026-05-16 when Sprint 5 ACQ RED-003 actually flipped it. Updated the default column from `false` to `true` and rewrote the description to reflect the current posture + the v2.1.x→v2.2.x warn-mode escape hatch. Verified locally: all 53 locally-runnable ci-guards still green (4 skipped: H-001-bare-from + H-002-bare-compose-image + digest-validity + no-precompiled-binary, all need docker-registry network). CI re-run on this commit should clear deploy-vendor-e2e's certctl-test-server dependency-failed-to-start step.
fix(deps): go mod tidy — drop unused google.golang.org/genproto bare module (CI go-mod-tidy gate)
2026-06-08 09:08:53 +00:00 · 2026-05-16 23:15:22 +00:00 · 2026-05-16 22:49:19 +00:00 · 2026-05-16 22:49:01 +00:00 · 2026-05-16 22:48:47 +00:00 · 2026-05-16 22:29:56 +00:00
991 changed files with 73795 additions and 11878 deletions
@@ -7,7 +7,7 @@
 # ==============================================================================
 POSTGRES_DB=certctl
 POSTGRES_USER=certctl
-POSTGRES_PASSWORD=change-me-in-production
+POSTGRES_PASSWORD=replace-with-openssl-rand-hex-32
 # ==============================================================================
 # Certctl Server
@@ -24,7 +24,7 @@ POSTGRES_PASSWORD=change-me-in-production
 # seeds pg_authid on first boot of an empty volume. See docs/quickstart.md
 # "Warning" callout and `internal/repository/postgres/db.go::wrapPingError`
 # for the SQLSTATE 28P01 diagnostic that fires when the two drift.
-CERTCTL_DATABASE_URL=postgres://certctl:change-me-in-production@postgres:5432/certctl?sslmode=disable
+CERTCTL_DATABASE_URL=postgres://certctl:replace-with-openssl-rand-hex-32@postgres:5432/certctl?sslmode=disable
 CERTCTL_SERVER_HOST=0.0.0.0
 CERTCTL_SERVER_PORT=8443
 CERTCTL_LOG_LEVEL=info
@@ -42,10 +42,27 @@ CERTCTL_LOG_FORMAT=json
 # option (no JWT middleware shipped - silent auth downgrade); see
 # docs/upgrade-to-v2-jwt-removal.md if you previously set
 # CERTCTL_AUTH_TYPE=jwt.
-CERTCTL_AUTH_TYPE=none
+#
-# Required when CERTCTL_AUTH_TYPE is "api-key".
+# Bundle 2 closure (2026-05-12): the docker-compose base file no longer
-# Generate with: openssl rand -base64 32
+# defaults to AUTH_TYPE=none. The base ships production-shaped; the demo
-# CERTCTL_AUTH_SECRET=change-me-in-production
+# overlay (deploy/docker-compose.demo.yml) flips this baseline into the
 # populated-dashboard demo path.
 CERTCTL_AUTH_TYPE=api-key
 # Required when CERTCTL_AUTH_TYPE is "api-key". Generate with:
 #   openssl rand -base64 32
 # The Bundle 2 fail-closed Validate() REFUSES TO START if this value
 # equals the placeholder string "change-me-in-production" outside of
 # demo mode (CERTCTL_DEMO_MODE_ACK=true).
 CERTCTL_AUTH_SECRET=replace-with-openssl-rand-base64-32
 # Bundle 2 closure: AES-256-GCM key for encrypting issuer/target config
 # secrets at rest. Required for any deployment that uses the dynamic
 # config GUI to store issuer credentials. Generate with:
 #   openssl rand -base64 32
 # Minimum 32 bytes. The Bundle 2 fail-closed Validate() REFUSES TO
 # START if this value equals the placeholder string
 # "change-me-32-char-encryption-key" outside of demo mode.
 CERTCTL_CONFIG_ENCRYPTION_KEY=replace-with-openssl-rand-base64-32
 # ==============================================================================
 # Certctl Agent
@@ -54,8 +71,14 @@ CERTCTL_AUTH_TYPE=none
 # startup. Use the docker-compose self-signed bootstrap CA bundle from
 # `deploy/test/certs/ca.crt` or supply your own via CERTCTL_SERVER_CA_BUNDLE_PATH.
 CERTCTL_SERVER_URL=https://localhost:8443
-CERTCTL_API_KEY=change-me-in-production
+# Matches one of the server's CERTCTL_AUTH_SECRET rotation values. The
 # placeholder is rejected outside demo mode (Bundle 2 fail-closed guard).
 CERTCTL_API_KEY=replace-with-openssl-rand-base64-32
 CERTCTL_AGENT_NAME=local-agent
 # Returned from `POST /api/v1/agents` during agent enrollment. The agent
 # fail-fasts at startup with "agent-id flag or CERTCTL_AGENT_ID env var
 # is required" if this is unset.
 # CERTCTL_AGENT_ID=agent-from-registration-response
 # ==============================================================================
 # Optional: Scheduler Tuning (defaults are usually fine)
@@ -0,0 +1,118 @@
 # Acquisition-audit DEPL-005 + DATA-012 closure (Sprint 4 ACQ,
 # 2026-05-16). Weekly backup-restore smoke test.
 #
 # Why
 # ===
 # The Helm CronJob at deploy/helm/certctl/templates/backup-cronjob.yaml
 # and the operator runbook at docs/operator/runbooks/postgres-backup.md
 # both document a pg_dump -Fc -based backup strategy, but the dump has
 # never been restored end-to-end under CI. A backup procedure that has
 # never been restore-tested is not a backup procedure. This workflow
 # adds the missing assertion.
 #
 # What
 # ====
 # Each Monday at 07:00 UTC (1h offset from loadtest.yml's 06:00 UTC
 # slot so they don't fight for runners), boot a real Postgres
 # 16-alpine container against the same digest pin as the production
 # deploy/docker-compose.yml, exercise the audit_events hash chain
 # with a small synthetic workload, pg_dump the database, drop the
 # schema, pg_restore, and assert the chain head + row count
 # round-trip byte-for-byte.
 #
 # The chain head round-trip property is the load-bearing assertion.
 # Migration 000047 hashes each audit_events row's canonical payload
 # with `to_char(timestamp AT TIME ZONE 'UTC',
 # 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"')`. Any TIMESTAMPTZ-precision loss
 # in the dump→restore path (a real concern across major Postgres
 # upgrades or with --format=plain) would corrupt the hash. The whole
 # point of testing instead of trusting docs is to PROVE the property
 # under a real workload.
 #
 # Workflow boundaries
 # ===================
 # - Does not exercise PITR / WAL archiving (DR runbook owns that).
 # - Does not exercise the Helm CronJob's S3 sink or scheduling
 #   (operator-side concern, not a property of the dump shape).
 # - Does not deploy or boot the certctl-server itself — the smoke
 #   harness talks to Postgres directly; we're testing the dump,
 #   not the server.
 name: backup-restore-smoke
 on:
  # Manual trigger from the Actions tab — useful before tagging a
  # release that touches the audit_events schema, or after a dep
  # bump that could affect canonical-payload formatting.
  workflow_dispatch:
  schedule:
    # Mondays at 07:00 UTC. Off-peak, off-set 1h from loadtest.yml
    # (06:00 UTC) so the two jobs don't fight for runners on the
    # GitHub-hosted ubuntu-latest pool.
    - cron: '0 7 * * 1'
 # Defense-in-depth: this job reads source and exercises a database;
 # it never needs write access to PRs, branches, releases, or
 # packages. Pin permissions to the minimum.
 permissions:
  contents: read
 jobs:
  backup-restore:
    name: pg_dump / pg_restore smoke
    runs-on: ubuntu-latest
    # 15-minute hard cap. The actual workload + dump + restore + verify
    # cycle runs in well under a minute on a warm runner; 15 minutes
    # absorbs cold image pulls, slow runner provisioning, and the
    # Postgres service-container readiness wait without letting a stuck
    # job consume the runner indefinitely.
    timeout-minutes: 15
    # Postgres service container. Pin to the same digest as
    # deploy/docker-compose.yml so the smoke runs against the exact
    # image the production deploy uses — a regression that surfaces
    # only on a specific Postgres minor bump shows up here on the
    # next image refresh in compose, not silently on a customer site.
    services:
      postgres:
        image: postgres:16-alpine@sha256:890480b08124ce7f79960a9bb16fe39729aa302bd384bfd7c408fee6c8f7adb7
        env:
          POSTGRES_DB: certctl
          POSTGRES_USER: certctl
          POSTGRES_PASSWORD: certctl
        ports:
          - 5432:5432
        # GitHub's services-container health check. The smoke shell
        # also waits for pg_isready as a belt-and-suspenders guard.
        options: >-
          --health-cmd "pg_isready -U certctl -d certctl"
          --health-interval 5s
          --health-timeout 3s
          --health-retries 10
    steps:
      - name: Checkout
        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5  # v4
      - name: Set up Go
        uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff  # v5
        with:
          go-version: '1.25.10'
          # Cache go-build + go-mod for the weekly run. Keep the
          # cache key bound to go.sum so a dep bump invalidates it.
          cache: true
      - name: Run backup-restore smoke
        env:
          PGHOST: 127.0.0.1
          PGPORT: '5432'
          PGUSER: certctl
          PGPASSWORD: certctl
          PGDATABASE: certctl
          # Insert enough rows to exercise the chain over a non-trivial
          # length. 24 ≫ 1 — large enough to surface ordering bugs,
          # small enough that the dump finishes in seconds.
          SMOKE_ROWS: '24'
        run: bash deploy/test/backup-restore-smoke.sh
@@ -14,12 +14,17 @@ jobs:
    name: Go Build & Test
    runs-on: ubuntu-latest
    steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5  # v4
      - name: Set up Go
-        uses: actions/setup-go@v5
+        uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff  # v5
        with:
          go-version: '1.25.10'
          # Phase 3 TEST-L1 closure (2026-05-13): enable Go's module +
          # build cache so re-runs hit the cache instead of recompiling
          # the world. setup-go v5 cache: true by default; making it
          # explicit so a future setup-go upgrade can't silently flip it.
          cache: true
      - name: Go Build
        run: |
@@ -103,11 +108,41 @@ jobs:
        run: staticcheck ./...
      - name: Race Detection
-        run: go test -race ./internal/service/... ./internal/api/handler/... ./internal/api/middleware/... ./internal/scheduler/... ./internal/connector/... ./internal/crypto/... ./internal/domain/... ./internal/validation/... ./internal/tlsprobe/... -count=1 -timeout 300s
+        # Phase 3 TEST-H1 closure (2026-05-13): the pre-Phase-3 invocation
        # listed 9 explicit package roots, excluding internal/auth/*,
        # internal/repository/*, internal/mcp, internal/scep, internal/pkcs7,
        # internal/api/router, internal/api/acme, internal/cli, internal/cms,
        # internal/config, internal/deploy, internal/integration,
        # internal/ratelimit, internal/secret, internal/trustanchor, plus
        # all of cmd/. Audit finding TEST-H1 flagged this as silent
        # race-detection drift — packages added after the original list
        # was authored were never covered.
        #
        # Post-Phase-3: ./... with -short. The 76 testing.Short() guards
        # already in the integration-test surface (testcontainers, live-DB,
        # multi-process) gate behind this flag, so race detection runs
        # across every package without dragging in long-running suites.
        # Timeout doubled from 300s to 600s because ./... is broader; the
        # broader scope is what makes race coverage trustworthy.
        run: go test -race -short ./... -count=1 -timeout 600s
      - name: Go Test with Coverage
        # internal/ciparity/... — post-v2.1.0 anti-rot item 2 surface-
        # parity tests; stdlib-only so they always pass in this job.
        run: |
-          go test ./internal/service/... ./internal/api/handler/... ./internal/api/middleware/... ./internal/api/router/... ./internal/auth/... ./internal/integration/... ./internal/connector/issuer/... ./internal/connector/target/... ./internal/connector/notifier/... ./internal/connector/discovery/... ./internal/crypto/... ./internal/mcp/... ./internal/cli/... ./internal/domain/... ./internal/validation/... ./internal/tlsprobe/... -count=1 -cover -coverprofile=coverage.out
+          go test ./internal/service/... ./internal/api/handler/... ./internal/api/middleware/... ./internal/api/router/... ./internal/auth/... ./internal/integration/... ./internal/connector/issuer/... ./internal/connector/target/... ./internal/connector/notifier/... ./internal/connector/discovery/... ./internal/crypto/... ./internal/mcp/... ./internal/cli/... ./internal/domain/... ./internal/validation/... ./internal/tlsprobe/... ./internal/ciparity/... -count=1 -cover -coverprofile=coverage.out
      - name: Multi-replica rate-limit integration test (Phase 13 Sprint 13.2/13.3 — ARCH-M1 closure proof)
        # The falsifiable proof that CERTCTL_RATE_LIMIT_BACKEND=postgres
        # enforces caps cluster-wide. testcontainers-go spins one
        # Postgres container; 3 *PostgresSlidingWindowLimiter instances
        # share it; 100 concurrent Allow("test-key") with cap=10 must
        # see exactly 10 succeed + 90 ErrRateLimited. Failure here =
        # the row-lock arbitration broke; ARCH-M1 closure is invalid.
        run: |
          go test -tags=integration -race -count=1 -timeout=300s \
              -run TestRateLimit_PostgresBackend_CapEnforcedAcrossReplicas \
              ./internal/integration/...
      - name: Check Coverage Thresholds
        # ci-pipeline-cleanup Phase 2: per-package floors moved to
@@ -118,7 +153,7 @@ jobs:
        run: bash scripts/check-coverage-thresholds.sh
      - name: Upload Coverage Report
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02  # v4
        with:
          name: go-coverage
          path: coverage.out
@@ -135,52 +170,6 @@ jobs:
          GITHUB_REPOSITORY: ${{ github.repository }}
        run: bash scripts/coverage-pr-comment.sh
      # Bundle P / Strengthening #6 — QA-doc seed-count drift guard. Forces
      # every PR that adds a seed row to migrations/seed_demo.sql to keep
      # docs/contributor/qa-test-suite.md::Seed Data Reference in sync.
      #
      # Phase 5 of the 2026-05-04 docs overhaul (commit c64777f) deleted
      # docs/testing-guide.md (its content dispersed across the new
      # audience-organized doc tree); the previous QA-doc Part-count drift
      # guard tracked Part counts between testing-guide.md and the old
      # qa-test-guide.md headline. With testing-guide.md gone, that guard's
      # premise is dead and it has been removed. The seed-count drift class
      # is still live: qa-test-suite.md::Seed Data Reference enumerates
      # certs/issuers and seed_demo.sql is the source of truth.
      - name: QA-doc seed-count drift guard
        run: |
          set -e
          DOC=docs/contributor/qa-test-suite.md
          # Seed-cert count: agnostic to documented header format. The current
          # documented count lives in `### Certificates (32 total in ...` —
          # extract the first integer in that header.
          DOC_CERTS=$(grep -oE '### Certificates \([0-9]+' "$DOC" | grep -oE '[0-9]+' | head -1)
          # Authoritative count: unique mc-* IDs in seed_demo.sql.
          SEED_CERTS=$(grep -oE 'mc-[a-z0-9_-]+' migrations/seed_demo.sql | sort -u | wc -l | tr -d ' ')
          if [ -z "$DOC_CERTS" ]; then
            echo "::warning::Could not extract documented cert count from $DOC."
            echo "  Skipping cert-count drift check (header format may have changed)."
          elif [ "$DOC_CERTS" != "$SEED_CERTS" ]; then
            echo "::error::DRIFT — $DOC says $DOC_CERTS certs; seed_demo.sql has $SEED_CERTS unique mc-* IDs."
            echo "  Update $DOC::Seed Data Reference to match."
            exit 1
          fi
          # Issuers: seed-table count vs doc claim.
          DOC_ISS=$(grep -oE '### Issuers \([0-9]+' "$DOC" | grep -oE '[0-9]+' | head -1)
          # Authoritative: unique iss-* IDs (close enough proxy; the issuers
          # table count IS the unique-ID count for this prefix).
          SEED_ISS=$(grep -oE 'iss-[a-z0-9_-]+' migrations/seed_demo.sql | sort -u | wc -l | tr -d ' ')
          if [ -z "$DOC_ISS" ]; then
            echo "::warning::Could not extract documented issuer count."
          elif [ "$DOC_ISS" != "$SEED_ISS" ] && [ "$((SEED_ISS - DOC_ISS))" -gt 5 ]; then
            # Allow up to 5pp slack — iss-* IDs appear in audit_events and
            # other reference tables that aren't issuer-table rows. Drift
            # only flags when the spread grows large.
            echo "::error::DRIFT — $DOC says $DOC_ISS issuers; seed_demo.sql has $SEED_ISS unique iss-* IDs (spread > 5)."
            exit 1
          fi
          echo "QA-doc seed-count drift guard: clean."
      # Bundle Q / I-001 closure — test-naming convention guard (informational).
      # The convention is `Test<Func>_<Scenario>_<ExpectedResult>`. This step
      # prints any non-conformant tests but does NOT fail the build until the
@@ -197,9 +186,17 @@ jobs:
      # internal scenarios expressed via `t.Run` subtests. Requiring the
      # underscore-Scenario-Result triple repo-wide would mean renaming
      # 167 legitimate tests for no observable behavior change. The
-      # Test<Func>_<Scenario>_<ExpectedResult> form remains documented as
+      # Test<Func>_<Scenario>_<ExpectedResult> form remains the
-      # the recommended pattern for parameterized scenarios in
+      # recommended pattern for parameterized scenarios, but is not gated.
-      # docs/contributor/qa-test-suite.md, but is not gated.
+      # Phase 4 DEPL-* prerequisite (2026-05-14): helm-templates-lint.sh
      # needs the `helm` CLI on PATH to run helm lint + helm template
      # against the chart. The official azure/setup-helm action installs
      # a SHA-pinned helm binary into the runner.
      - name: Install Helm (for helm-templates-lint guard)
        uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112  # v4.3.0
        with:
          version: v3.16.0
      - name: Regression guards (extracted to scripts/ci-guards/)
        # All named regression guards live at scripts/ci-guards/<id>.sh per
        # ci-pipeline-cleanup bundle Phase 1. Each guard is callable locally:
@@ -207,6 +204,7 @@ jobs:
        # Adding a new guard: drop a new <id>.sh; this loop auto-picks it up.
        # Contract: each guard MUST exit 0 on clean repo, non-zero with
        # ::error:: prefix on regression. See scripts/ci-guards/README.md.
        #
        run: |
          set -e
          fail=0
@@ -219,14 +217,225 @@ jobs:
          done
          exit $fail
  cross-platform-build:
    # Phase 3 TEST-H2 closure (2026-05-13): the pre-Phase-3 CI ran
    # exclusively on ubuntu-latest, leaving Windows-specific bugs
    # (path separators, file permissions, exec.Command semantics)
    # undetected. The agent + CLI binaries ship for Windows + macOS
    # users; this matrix asserts they at least BUILD on every OS we
    # claim to support.
    #
    # Build-only — no test run. Full test parity across OSes is a
    # larger investment (testcontainers is Linux-only on Windows CI
    # runners, file-permission tests differ, etc.). The build gate
    # is the minimum that catches the cross-platform regressions
    # we've seen in practice.
    name: Cross-platform build (ubuntu / windows / macos)
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5  # v4
      - name: Set up Go
        uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff  # v5
        with:
          go-version: '1.25.10'
          cache: true
      - name: Build server + agent + CLI + mcp-server
        run: |
          go build ./cmd/server
          go build ./cmd/agent
          go build ./cmd/cli
          go build ./cmd/mcp-server
  cold-db-compose-smoke:
    # Per post-v2.1.0 anti-rot item 6 (Auditable Codebase Bundle).
    #
    # Catches migration-on-cold-DB regressions: wipe the postgres
    # volume, bring the stack up cold, mint a day-0 admin, issue +
    # renew + revoke a test certificate, assert audit rows, tear down.
    # Targets the bug class that the warm-DB integration suite misses
    # (canonical case: 2026-05-09 migration 000045 broken INSERT,
    # fixed in commit 6444e13).
    name: Cold-DB compose smoke
    runs-on: ubuntu-latest
    needs: go-build-and-test
    steps:
      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5  # v4
      - name: Show Docker versions
        run: |
          docker --version
          docker compose version
      - name: Cold-DB compose smoke
        # The smoke deliberately focuses on the bug class that ONLY a
        # cold boot can catch: stack-startup correctness against a
        # blank database. It is intentionally NOT a functional API
        # walkthrough — the integration test suite under
        # 'Go Test with Coverage' already covers issue / renew /
        # revoke / audit-row plumbing against a warm DB.
        #
        # The bugs this gate is uniquely positioned to catch:
        #   - Missing required env vars that fail Config.Validate()
        #     at startup (e.g. CERTCTL_DEMO_MODE_ACK gap, 2026-05-12).
        #   - Non-idempotent migrations that crash on the second boot
        #     (e.g. migration 000043 CHECK constraint, 2026-05-12).
        #   - Documented manual flows that don't work end-to-end on
        #     a clean compose (e.g. CERTCTL_BOOTSTRAP_TOKEN
        #     interpolation gap, 2026-05-12).
        #
        # Bugs OUTSIDE the scope of this smoke (covered elsewhere):
        #   - API request/response contract changes (integration suite).
        #   - Cert lifecycle correctness (integration suite + handler
        #     tests).
        #   - Audit row plumbing (handler tests).
        #
        # 10-min wall-clock cap covers cold image pull + compose-up +
        # force-recreate + admin bootstrap + teardown. Increase only
        # if the underlying steps legitimately grow.
        #
        # The smoke is inlined here on purpose — it is NOT a script in
        # scripts/ci-guards/, because there is no value in a developer
        # running this locally. The whole point of the gate is that CI
        # owns the cold-DB state; the operator never has to remember to
        # run it.
        timeout-minutes: 10
        working-directory: deploy
        env:
          STARTUP_TIMEOUT_SECONDS: 300
        run: |
          set -e
          set -o pipefail
          SERVER_URL="https://localhost:8443"
          CACERT_PATH="${GITHUB_WORKSPACE}/deploy/test/certs/ca.crt"
          log() { echo "[cold-db-smoke] $*"; }
          wait_for_service_healthy() {
            local svc="$1" deadline=$(( $(date +%s) + STARTUP_TIMEOUT_SECONDS ))
            while [ "$(date +%s)" -lt "$deadline" ]; do
              local state
              state="$(docker compose ps --format json "$svc" 2>/dev/null | python3 -c '
          import json, sys
          try:
              line = sys.stdin.read().strip()
              if not line:
                  print("not-up"); sys.exit(0)
              rows = json.loads(line) if line.startswith("[") else [json.loads(l) for l in line.splitlines() if l.strip()]
              if not rows:
                  print("not-up")
              else:
                  print(rows[0].get("Health", rows[0].get("State", "?")))
          except Exception as e:
              print(f"err: {e}")
          ')"
              if [ "$state" = "healthy" ] || [ "$state" = "running" ]; then
                log "  $svc → $state"; return 0
              fi
              sleep 2
            done
            log "  $svc did NOT reach healthy within ${STARTUP_TIMEOUT_SECONDS}s (last: $state)"
            return 1
          }
          http_call() {
            local method="$1" path="$2" data="${3:-}"
            local args=(--silent --show-error --max-time 30 -X "$method" "$SERVER_URL$path")
            [ -f "$CACERT_PATH" ] && args+=(--cacert "$CACERT_PATH") || args+=(--insecure)
            [ -n "$data" ] && args+=(-H "Content-Type: application/json" -d "$data")
            curl "${args[@]}"
          }
          # Bundle 2 closure (2026-05-12): the base compose is now
          # production-shaped — auth=api-key + agent-keygen + fail-closed
          # placeholder guards. The cold-DB smoke layers in the demo
          # overlay so the boot path remains zero-config: the overlay
          # supplies AUTH_TYPE=none + DEMO_MODE_ACK=true + the matching
          # placeholder creds the fail-closed guards accept under
          # DEMO_MODE_ACK. The agent service in the overlay also
          # pre-seeds CERTCTL_AGENT_ID=agent-demo-1 so the bundled
          # agent doesn't restart-loop. The smoke's purpose (catch
          # migration-on-cold-DB regressions + verify bootstrap-token
          # endpoint mints a day-0 admin against a freshly migrated
          # schema) is orthogonal to whether the auth posture is
          # demo-mode or api-key, so the overlay is acceptable here.
          COMPOSE_FILES=(-f docker-compose.yml -f docker-compose.demo.yml)
          # Phase 2 SEC-H3 (2026-05-13): the demo overlay sets
          # CERTCTL_DEMO_MODE_ACK=true; the SEC-H3 fail-closed guard
          # requires a paired CERTCTL_DEMO_MODE_ACK_TS within the last
          # 24h (a static YAML value would rot). The overlay reads
          # ${CERTCTL_DEMO_MODE_ACK_TS:-} from the shell, so we mint a
          # fresh timestamp here and export it for every compose
          # invocation in this job (initial up-d AND the force-recreate
          # at step 4).
          export CERTCTL_DEMO_MODE_ACK_TS="$(date +%s)"
          log "1/4 down -v --remove-orphans"
          docker compose "${COMPOSE_FILES[@]}" down -v --remove-orphans 2>&1 | tail -3 || true
          log "2/4 up -d (cold boot)"
          docker compose "${COMPOSE_FILES[@]}" up -d 2>&1 | tail -3
          log "3/4 wait for healthchecks"
          wait_for_service_healthy postgres
          wait_for_service_healthy certctl-server
          wait_for_service_healthy certctl-agent || log "  (agent skipped)"
          log "4/4 minting day-0 admin (proves migration ladder + bootstrap path)"
          TOKEN="$(openssl rand -base64 32 | tr -d '\n')"
          {
            echo "CERTCTL_BOOTSTRAP_TOKEN=$TOKEN"
            # Re-emit the demo-mode ACK TS into the --env-file so the
            # force-recreate at step 4 inherits it. `--env-file` REPLACES
            # the shell-env source for variable interpolation on compose
            # operations that use it, so omitting this line would re-trip
            # the SEC-H3 guard.
            echo "CERTCTL_DEMO_MODE_ACK_TS=$CERTCTL_DEMO_MODE_ACK_TS"
          } > /tmp/_smoke.env
          docker compose "${COMPOSE_FILES[@]}" --env-file /tmp/_smoke.env up -d --force-recreate certctl-server 2>&1 | tail -2
          sleep 5
          wait_for_service_healthy certctl-server
          BODY="$(http_call POST /api/v1/auth/bootstrap "{\"token\":\"$TOKEN\",\"actor_name\":\"smoke-admin\"}")"
          KEY="$(echo "$BODY" | python3 -c 'import json,sys; print(json.load(sys.stdin)["key_value"])')"
          [ -n "$KEY" ] || { log "bootstrap failed: $BODY"; exit 1; }
          log "PASS — cold boot + force-recreate + admin bootstrap all green"
          log "tearing down"
          docker compose "${COMPOSE_FILES[@]}" down -v 2>&1 | tail -2
      - name: Dump compose logs on failure
        if: failure()
        working-directory: deploy
        run: |
          for svc in postgres certctl-server certctl-agent certctl-tls-init; do
            echo "==== $svc ===="
            docker compose -f docker-compose.yml -f docker-compose.demo.yml logs --no-color --tail 200 "$svc" || true
          done
  frontend-build:
    name: Frontend Build
    runs-on: ubuntu-latest
    steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5  # v4
        with:
          # ARCH-001-A closure (Sprint 5, 2026-05-16). The
          # openapi-version-tag-parity guard needs the v* tags to
          # be present locally so it can confirm openapi.yaml's
          # info.version matches the latest release. Without
          # fetch-tags, the guard falls back to the GitHub API —
          # works but adds a network round-trip per CI run.
          fetch-tags: true
          fetch-depth: 0
      - name: Set up Node.js
-        uses: actions/setup-node@v4
+        uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020  # v4
        with:
          node-version: '22'
@@ -234,6 +443,17 @@ jobs:
        working-directory: web
        run: npm ci
      - name: npm audit (production deps, high+critical)
        # Phase 1 TEST-L2 closure (2026-05-13):
        # Production frontend dependencies must not carry high or
        # critical CVEs. Dev-only deps (vitest, vite, eslint, etc.)
        # are excluded via --omit=dev since they never ship to
        # operators. If this gate fires, triage each finding via npm
        # overrides, dep upgrade, or a tracked --ignore with an issue
        # link. Do not mass-silence findings.
        working-directory: web
        run: npm audit --omit=dev --audit-level=high
      - name: TypeScript Check
        working-directory: web
        run: npx tsc --noEmit
@@ -246,6 +466,17 @@ jobs:
        working-directory: web
        run: npx vite build
      - name: Frontend bundle-size budget (size-limit)
        # Acquisition-audit SCALE-007 closure (Sprint 6 ACQ, 2026-05-16).
        # Per-chunk + per-tier budgets in web/.size-limit.json; brotli-
        # compressed sizes match real-world download cost. A regression
        # that bloats a chunk past its cap fails this step and forces
        # an explicit operator decision (fix vs raise cap with rationale).
        # The script wrapper at scripts/ci-guards/G-frontend-bundle-budget.sh
        # is the local-runnable counterpart; both invoke `npm run size`.
        working-directory: web
        run: npm run size
      - name: Regression guards (extracted to scripts/ci-guards/)
        # All named regression guards live at scripts/ci-guards/<id>.sh per
        # ci-pipeline-cleanup bundle Phase 1. Each guard is callable locally:
@@ -269,10 +500,10 @@ jobs:
    name: Helm Chart Validation
    runs-on: ubuntu-latest
    steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5  # v4
      - name: Install Helm
-        uses: azure/setup-helm@v4
+        uses: azure/setup-helm@1a275c3b69536ee54be43f2070a358922e12c8d4  # v4
        with:
          version: '3.13.0'
@@ -280,15 +511,25 @@ jobs:
      # configured. Every lint/template invocation below must pick exactly one
      # provisioning mode — see deploy/helm/certctl/templates/_helpers.tpl
      # (certctl.tls.required) and docs/operator/tls.md.
      #
      # Bundle 3 closure (2026-05-12, commit f1fa311): the chart now ALSO
      # fails render when (a) server.auth.type=api-key + apiKey empty, or
      # (b) postgresql.enabled=true + postgresql.auth.password empty.
      # Every positive render below MUST pass both secrets; inverse tests
      # at the bottom of this job pin the fail-fast guards in place.
      - name: Lint Helm Chart
        run: |
          helm lint deploy/helm/certctl/ \
-            --set server.tls.existingSecret=certctl-tls-ci
+            --set server.tls.existingSecret=certctl-tls-ci \
            --set server.auth.apiKey=ci-api-key-placeholder \
            --set postgresql.auth.password=ci-postgres-placeholder
      - name: Template Helm Chart (existingSecret mode)
        run: |
          helm template certctl deploy/helm/certctl/ \
            --set server.tls.existingSecret=certctl-tls-ci \
            --set server.auth.apiKey=ci-api-key-placeholder \
            --set postgresql.auth.password=ci-postgres-placeholder \
            > /dev/null
      - name: Template Helm Chart (cert-manager mode)
@@ -296,8 +537,30 @@ jobs:
          helm template certctl deploy/helm/certctl/ \
            --set server.tls.certManager.enabled=true \
            --set server.tls.certManager.issuerRef.name=letsencrypt-prod \
            --set server.auth.apiKey=ci-api-key-placeholder \
            --set postgresql.auth.password=ci-postgres-placeholder \
            > /dev/null
      - name: Template Helm Chart (external Postgres mode — Bundle 3 D2)
        run: |
          # Closes Bundle 3 D2: postgresql.enabled=false must (a) render
          # cleanly with externalDatabase.url and (b) emit ZERO postgres-*
          # templates. The render output is grep-checked below.
          out=$(helm template certctl deploy/helm/certctl/ \
            --set server.tls.existingSecret=certctl-tls-ci \
            --set postgresql.enabled=false \
            --set externalDatabase.url='postgres://u:p@db.example.com:5432/certctl?sslmode=require' \
            --set server.auth.apiKey=ci-api-key-placeholder)
          # Bundled-Postgres resources must not appear when postgresql.enabled=false.
          if echo "$out" | grep -qE "^kind: StatefulSet$"; then
            echo "::error::Bundle 3 D2 regression: postgres StatefulSet rendered with postgresql.enabled=false"
            exit 1
          fi
          if echo "$out" | grep -q "postgres-secret.yaml"; then
            echo "::error::Bundle 3 D2 regression: postgres-secret rendered with postgresql.enabled=false"
            exit 1
          fi
      - name: Template Helm Chart (guard fails without TLS)
        run: |
          # Inverse test: the chart MUST refuse to render when no TLS source is
@@ -308,6 +571,58 @@ jobs:
            exit 1
          fi
      - name: Template Helm Chart (guard fails — Bundle 3 D7 TLS both-set)
        run: |
          # Bundle 3 D7: setting BOTH existingSecret AND certManager.enabled
          # creates two conflicting TLS sources of truth. Chart must refuse.
          if helm template certctl deploy/helm/certctl/ \
                --set server.tls.existingSecret=ci \
                --set server.tls.certManager.enabled=true \
                --set server.tls.certManager.issuerRef.name=foo \
                --set server.auth.apiKey=k \
                --set postgresql.auth.password=p \
                > /dev/null 2>&1; then
            echo "::error::Bundle 3 D7 regression: chart rendered with BOTH TLS sources configured"
            exit 1
          fi
      - name: Template Helm Chart (guard fails — Bundle 3 D1 missing apiKey)
        run: |
          # Bundle 3 D1: missing server.auth.apiKey when auth.type=api-key
          # must fail at template time, not silently render an empty Secret.
          if helm template certctl deploy/helm/certctl/ \
                --set server.tls.existingSecret=ci \
                --set postgresql.auth.password=p \
                > /dev/null 2>&1; then
            echo "::error::Bundle 3 D1 regression: chart rendered with empty server.auth.apiKey"
            exit 1
          fi
      - name: Template Helm Chart (guard fails — Bundle 3 D1 missing pg password)
        run: |
          # Bundle 3 D1: missing postgresql.auth.password when postgresql.enabled=true
          # must fail at template time, not silently use a fallback default.
          if helm template certctl deploy/helm/certctl/ \
                --set server.tls.existingSecret=ci \
                --set server.auth.apiKey=k \
                > /dev/null 2>&1; then
            echo "::error::Bundle 3 D1 regression: chart rendered with empty postgresql.auth.password"
            exit 1
          fi
      - name: Template Helm Chart (guard fails — Bundle 3 D1 missing external DB URL)
        run: |
          # Bundle 3 D1: missing externalDatabase.url when postgresql.enabled=false
          # must fail at template time.
          if helm template certctl deploy/helm/certctl/ \
                --set server.tls.existingSecret=ci \
                --set postgresql.enabled=false \
                --set server.auth.apiKey=k \
                > /dev/null 2>&1; then
            echo "::error::Bundle 3 D1 regression: chart rendered with postgresql.enabled=false + empty externalDatabase.url"
            exit 1
          fi
  # =============================================================================
  # deploy-vendor-e2e — single-job (collapsed from 12-job matrix)
  # =============================================================================
@@ -338,10 +653,10 @@ jobs:
    needs: [go-build-and-test]
    timeout-minutes: 30
    steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd  # v5
      - name: Set up Go
-        uses: actions/setup-go@v5
+        uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff  # v5
        with:
          go-version: '1.25.10'
          cache: true
@@ -435,10 +750,10 @@ jobs:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
-      - uses: actions/checkout@v5
+      - uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd  # v5
      - name: Set up Go
-        uses: actions/setup-go@v5
+        uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff  # v5
        with:
          go-version: '1.25.10'
          cache: true
@@ -53,17 +53,17 @@ jobs:
    steps:
      - name: Checkout
-        uses: actions/checkout@v4
+        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5  # v4
      - name: Set up Go
        if: matrix.language == 'go'
-        uses: actions/setup-go@v5
+        uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff  # v5
        with:
          # Match ci.yml + release.yml + security-deep-scan.yml.
          go-version: '1.25.10'
      - name: Initialize CodeQL
-        uses: github/codeql-action/init@v3
+        uses: github/codeql-action/init@7fd177fa680c9881b53cdab4d346d32574c9f7f4  # v3
        with:
          languages: ${{ matrix.language }}
          # Use the security-and-quality query suite — security finds plus
@@ -72,10 +72,10 @@ jobs:
          queries: security-and-quality
      - name: Autobuild
-        uses: github/codeql-action/autobuild@v3
+        uses: github/codeql-action/autobuild@7fd177fa680c9881b53cdab4d346d32574c9f7f4  # v3
      - name: Perform CodeQL Analysis
-        uses: github/codeql-action/analyze@v3
+        uses: github/codeql-action/analyze@7fd177fa680c9881b53cdab4d346d32574c9f7f4  # v3
        with:
          category: "/language:${{ matrix.language }}"
          # SARIF upload is implicit (and is what populates the Security tab).
@@ -0,0 +1,112 @@
 # Phase 8 closure (TEST-H1 + TEST-H2): browser-driven E2E + visual
 # regression.
 #
 # TEST-003 closure (Sprint 5, 2026-05-16): the suite has accumulated
 # the empirical green-run evidence the Phase 8 prompt required. 14
 # consecutive green runs across 2026-05-14 to 2026-05-15 (sampled
 # via api.github.com/repos/certctl-io/certctl/actions/runs) during
 # heavy Sprint 1-4 frontend churn confirm stability. The job is
 # now part of the merge gate (continue-on-error: false below).
 #
 # Operator action still required AFTER this commit pushes:
 #   - Add this job's "id" to the branch-protection required-checks
 #     list at https://github.com/certctl-io/certctl/settings/branches.
 #     Without that, the workflow's failure-blocks-merge contract
 #     only fires on PRs whose author is configured to honour the
 #     status check; configured required-checks make it universal.
 #
 # Visual regression: the 04-visual-regression.spec.ts file uses
 # Playwright `toHaveScreenshot()`. First-run on a new branch
 # regenerates baselines via the `--update-snapshots` flag; the
 # operator commits the resulting PNG bytes to git. Subsequent runs
 # pixel-diff. The dispatch input below provides an explicit knob
 # for that initial baseline pass without needing to edit the
 # workflow file. See docs/operator/runbooks/e2e-snapshot-update.md
 # for the snapshot-bump workflow.
 name: Frontend E2E
 on:
  push:
    branches: [master]
    paths:
      - 'web/**'
      - '.github/workflows/e2e.yml'
  pull_request:
    paths:
      - 'web/**'
      - '.github/workflows/e2e.yml'
  workflow_dispatch:
    inputs:
      update_snapshots:
        description: 'Regenerate visual-regression baselines (use sparingly)'
        type: boolean
        default: false
 permissions:
  contents: read
 jobs:
  e2e:
    name: Playwright E2E + visual regression
    runs-on: ubuntu-latest
    # TEST-003 closure (Sprint 5, 2026-05-16): flipped from
    # continue-on-error: true after 14 consecutive green runs across
    # 2026-05-14 to 2026-05-15 confirmed stability. Failures here
    # now fail the workflow, which (combined with the branch
    # protection update the operator owns post-merge) blocks merge.
    continue-on-error: false
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5  # v4
      - name: Set up Node.js
        uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020  # v4
        with:
          node-version: '22'
      - name: Install Dependencies
        working-directory: web
        run: npm ci
      - name: Install Playwright browsers
        working-directory: web
        # --with-deps installs OS packages (libnss3, libatk1.0-0, etc.)
        # the chromium browser needs. Skipping this is the #1 source
        # of "tests pass locally but fail on CI" for new Playwright
        # users. The browser binary downloads to ~/.cache/ms-playwright;
        # the actions/setup-node cache key does NOT include it, so each
        # CI run re-downloads. Add an actions/cache step targeting
        # ~/.cache/ms-playwright keyed by the @playwright/test version
        # in package-lock.json once the suite is stable.
        run: npx playwright install --with-deps chromium
      - name: Run Playwright E2E + visual regression
        working-directory: web
        # The webServer block in playwright.config.ts boots `npm run dev`
        # automatically and waits for http://localhost:5173 to be
        # responsive before the first test fires. No separate "start
        # server" step needed.
        run: |
          if [[ "${{ github.event.inputs.update_snapshots }}" == "true" ]]; then
            echo "::warning::Regenerating visual-regression baselines"
            npx playwright test --update-snapshots
          else
            npx playwright test
          fi
      - name: Upload Playwright report on failure
        if: failure()
        uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882  # v4
        with:
          name: playwright-report
          path: web/playwright-report/
          retention-days: 7
      - name: Upload visual-regression diffs on failure
        if: failure()
        uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882  # v4
        with:
          name: visual-regression-diffs
          path: web/test-results/
          retention-days: 7
@@ -49,13 +49,13 @@ jobs:
    steps:
      - name: Checkout
-        uses: actions/checkout@v4
+        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5  # v4
      - name: Set up Docker Buildx
        # The compose stack builds the certctl image from the repo
        # root Dockerfile. Buildx gives the build a usable cache and
        # works with newer compose versions.
-        uses: docker/setup-buildx-action@v3
+        uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f  # v3
      - name: Run loadtest
        run: make loadtest
@@ -70,8 +70,70 @@ jobs:
        # authoritative machine-readable form; summary.txt is the
        # human-readable text the README baseline tracks.
        if: always()
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02  # v4
        with:
          name: k6-summary-${{ github.run_id }}
          path: deploy/test/loadtest/results/
          retention-days: 90
  # ---------------------------------------------------------------------------
  # Phase 8 SCALE-H2 — scale-tier scenarios. Three new k6 drivers:
  #   - bulk-renewal: 10K-cert seed + criteria-mode POST /bulk-renew
  #   - acme-burst:   200 concurrent VUs against directory/nonce/ARI
  #   - agent-storm:  5K-agent seed + 167 heartbeats/sec sustained
  #
  # Matrix dispatch so each scenario runs on its own runner and a
  # regression in one doesn't mask another. The matrix runs in parallel,
  # which keeps total wall time around the existing 25-minute cap rather
  # than ~70 minutes serialised. Each scenario brings up the full
  # loadtest compose stack independently — there's no shared state
  # between scenarios that would benefit from a single-runner serial
  # invocation.
  #
  # Cadence: same as the API + connector tier job above (workflow_dispatch
  # + Mondays 06:00 UTC). The scale scenarios DO produce useful per-PR
  # signal in theory, but the per-run cost (image build + 5min run × 3)
  # is too high to gate on every PR; weekly is the right trade-off.
  # ---------------------------------------------------------------------------
  k6-scale:
    name: k6 scale tier (${{ matrix.scenario }})
    runs-on: ubuntu-latest
    timeout-minutes: 25
    needs: k6
    strategy:
      # Parallel: a failure in one scenario shouldn't cancel the others.
      # Each scenario's threshold breach is independent diagnostic data.
      fail-fast: false
      matrix:
        scenario:
          - bulk-renewal
          - acme-burst
          - agent-storm
    steps:
      - name: Checkout
        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5  # v4
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f  # v3
      - name: Run scale loadtest (${{ matrix.scenario }})
        env:
          BUILDKIT_PROGRESS: plain
        run: |
          case "${{ matrix.scenario }}" in
            bulk-renewal) make loadtest-scale-bulk ;;
            acme-burst)   make loadtest-scale-acme ;;
            agent-storm)  make loadtest-scale-agent ;;
            *) echo "::error::unknown scenario ${{ matrix.scenario }}"; exit 1 ;;
          esac
      - name: Upload summary
        if: always()
        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02  # v4
        with:
          # Per-scenario artifact name so the three matrix runs don't
          # collide on upload.
          name: k6-scale-${{ matrix.scenario }}-${{ github.run_id }}
          path: deploy/test/loadtest/results/
          retention-days: 90
@@ -39,10 +39,10 @@ jobs:
        os: [linux, darwin]
        arch: [amd64, arm64]
    steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5  # v4
      - name: Set up Go
-        uses: actions/setup-go@v5
+        uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff  # v5
        with:
          go-version: ${{ env.GO_VERSION }}
@@ -123,7 +123,7 @@ jobs:
          cat "${OUTPUT_NAME}.sha256"
      - name: Upload build artefacts
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02  # v4
        with:
          name: binary-${{ steps.build.outputs.output_name }}
          path: |
@@ -151,7 +151,7 @@ jobs:
      hashes: ${{ steps.hashes.outputs.hashes }}
    steps:
      - name: Download binary artefacts
-        uses: actions/download-artifact@v4
+        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093  # v4
        with:
          pattern: binary-*
          path: artifacts
@@ -191,7 +191,7 @@ jobs:
            checksums.txt
      - name: Upload artefacts to GitHub Release
-        uses: softprops/action-gh-release@v2
+        uses: softprops/action-gh-release@3bb12739c298aeb8a4eeaf626c5b8d85266b0e65  # v2
        if: startsWith(github.ref, 'refs/tags/')
        with:
          files: |
@@ -212,11 +212,24 @@ jobs:
      actions: read
      id-token: write
      contents: write
-    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v2.1.0
+    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@f7dd8c54c2067bafc12ca7a55595d5ee9b75204a  # v2.1.0
    with:
      base64-subjects: "${{ needs.aggregate-checksums.outputs.hashes }}"
      upload-assets: true
      provenance-name: multiple.intoto.jsonl
      # Phase 1 RED-2 compat (2026-05-14): the SLSA reusable workflow's
      # default path downloads a pre-built generator binary from a
      # GitHub *release* of slsa-framework/slsa-github-generator —
      # releases are keyed by tag name (vX.Y.Z), and the workflow
      # rejects SHA-form refs with "Expected ref of the form
      # refs/tags/vX.Y.Z". Phase 1 RED-2 SHA-pinned every Actions
      # uses: line, so the default path errors out. Setting
      # compile-generator: true instead builds the generator from the
      # pinned-SHA source inside the workflow run — preserves
      # supply-chain integrity (SHA pin retained), adds ~1 min build
      # time. This is the SLSA project's documented escape hatch for
      # SHA-pinned reusable-workflow consumers.
      compile-generator: true
  # ----------------------------------------------------------------------
  # build-and-push-docker: push container images to GHCR with native
@@ -235,10 +248,10 @@ jobs:
      id-token: write  # Cosign keyless OIDC identity token
    steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5  # v4
      - name: Log in to GitHub Container Registry
-        uses: docker/login-action@v3
+        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9  # v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
@@ -249,14 +262,14 @@ jobs:
        run: echo "VERSION=${GITHUB_REF#refs/tags/}" >> "$GITHUB_OUTPUT"
      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v3
+        uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f  # v3
      - name: Install Cosign
        uses: sigstore/cosign-installer@cad07c2e89fa2edd6e2d7bab4c1aa38e53f76003  # v4.1.1
      - name: Build and push server image
        id: server-push
-        uses: docker/build-push-action@v6
+        uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8  # v6
        with:
          context: .
          file: ./Dockerfile
@@ -291,7 +304,7 @@ jobs:
      - name: Build and push agent image
        id: agent-push
-        uses: docker/build-push-action@v6
+        uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8  # v6
        with:
          context: .
          file: ./Dockerfile.agent
@@ -334,7 +347,7 @@ jobs:
      contents: write
    steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5  # v4
      - name: Extract version from tag
        id: version
@@ -351,7 +364,7 @@ jobs:
        # README is the source of truth for those, and inlining them in every
        # release page produces the kind of "every release looks identical"
        # noise that gives operators no signal about what actually changed.
-        uses: softprops/action-gh-release@v2
+        uses: softprops/action-gh-release@3bb12739c298aeb8a4eeaf626c5b8d85266b0e65  # v2
        with:
          # Pin the release title to the tag name. softprops/action-gh-release@v2
          # falls back to the most recent commit subject when `name:` is omitted,
@@ -36,9 +36,9 @@ jobs:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5  # v4
-      - uses: actions/setup-go@v5
+      - uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff  # v5
        with:
          go-version: '1.25'
@@ -48,15 +48,26 @@ jobs:
      # --- Static analysis (slow paths) ---
-      - name: gosec
+      - name: gosec (G201/G202/G304/G108 subset — Phase 3 TEST-M2 hard gate)
-        run: |
+        # Phase 3 TEST-M2 closure (2026-05-13): gosec promoted from
-          $(go env GOPATH)/bin/gosec -fmt sarif -out gosec.sarif ./... || true
+        # continue-on-error (advisory) to blocking on the 4 high-signal
-        continue-on-error: true
+        # rule subset that targets real prod-bug classes:
        #   G201 = SQL string formatting (SQL injection)
        #   G202 = SQL string concatenation (SQL injection)
        #   G304 = file-path traversal via tainted input
        #   G108 = profiling endpoint exposed
        # Other gosec rules (G1xx-G7xx broadly) remain in the SARIF
        # report but don't gate the build — they have higher false-
        # positive rates than these 4.
        run: $(go env GOPATH)/bin/gosec -fmt sarif -out gosec.sarif -include=G201,G202,G304,G108 ./...
-      - name: osv-scanner (multi-ecosystem CVE)
+      - name: osv-scanner (multi-ecosystem CVE — Phase 3 TEST-M2 hard gate)
-        run: |
+        # Phase 3 TEST-M2 closure (2026-05-13): osv-scanner promoted from
-          $(go env GOPATH)/bin/osv-scanner -r --format json --output osv-scanner.json . || true
+        # advisory to blocking. Complements govulncheck (already blocking
-        continue-on-error: true
+        # in ci.yml) by covering non-Go dependencies (npm under web/,
        # any docker base image deps). Findings fail the build; the
        # exact CVE list lands in osv-scanner.json as a receipt either way.
        run: $(go env GOPATH)/bin/osv-scanner -r --format json --output osv-scanner.json .
      # --- Race detector at -count=10 (D-002) ---
@@ -90,14 +101,39 @@ jobs:
        run: go install github.com/zimmski/go-mutesting/cmd/go-mutesting@latest
        continue-on-error: true
-      - name: go-mutesting (crypto cluster)
+      - name: go-mutesting (crypto cluster — Phase 3 TEST-M1 hard gate at 55%)
        # Phase 3 TEST-M1 closure (2026-05-13): go-mutesting promoted
        # from advisory (continue-on-error + per-package `|| true`) to
        # blocking with an explicit mutation-score floor of 55%.
        # Per-package summary lines emit `The mutation score is X.YZ`;
        # the awk filter extracts each, and the post-loop check fails
        # the step if any package drops below 0.55.
        #
        # Floor rationale: 55% is the starter ratio that catches major
        # regressions without rejecting the audit's "this is OK" steady
        # state. Raise quarterly as the test suite hardens; the floor
        # change ships in the same commit that adds the strengthening
        # tests so the ratchet is documented.
        run: |
          set -e
          : > go-mutesting.txt
          for pkg in ./internal/crypto/... ./internal/pkcs7/... ./internal/connector/issuer/local/...; do
            echo "=== $pkg ===" | tee -a go-mutesting.txt
-            $(go env GOPATH)/bin/go-mutesting "$pkg" 2>&1 | tee -a go-mutesting.txt || true
+            $(go env GOPATH)/bin/go-mutesting "$pkg" 2>&1 | tee -a go-mutesting.txt
          done
-        continue-on-error: true
+          # Extract every "The mutation score is X.YZ" line; fail on any
          # score below 0.55. The check works against floats via awk so
          # 0.55 is the literal threshold (not a percentage).
          floor=0.55
          fail=0
          while IFS= read -r score; do
            ok=$(awk -v s="$score" -v f="$floor" 'BEGIN{print (s>=f) ? 1 : 0}')
            if [ "$ok" -ne 1 ]; then
              echo "::error::mutation score $score below floor $floor"
              fail=1
            fi
          done < <(grep -oE "The mutation score is [0-9.]+" go-mutesting.txt | awk '{print $NF}')
          exit $fail
      # --- Container + supply chain (D-001 partial, D-006 partial) ---
@@ -105,11 +141,21 @@ jobs:
        run: docker build -t certctl:deep-scan .
        continue-on-error: true
-      - name: trivy image scan
+      - name: trivy image scan (HIGH+CRITICAL — Phase 3 TEST-M2 hard gate)
        # Phase 3 TEST-M2 closure (2026-05-13): trivy promoted from
        # advisory to blocking. --severity filter keeps the gate
        # noise-free (LOW + MEDIUM findings stay in the JSON receipt
        # but don't fail the build); --exit-code 1 makes HIGH+CRITICAL
        # findings the actual gate. Trivy is the third hard deep-scan
        # gate (alongside gosec + osv-scanner); ZAP / schemathesis /
        # nuclei / testssl stay advisory because their false-positive
        # rates on https://localhost:8443-targeted DAST runs are high.
        run: |
          docker run --rm -v "$PWD":/src aquasec/trivy:latest image \
-            --format json --output /src/trivy.json certctl:deep-scan || true
+            --format json --output /src/trivy.json \
-        continue-on-error: true
+            --severity HIGH,CRITICAL \
            --exit-code 1 \
            certctl:deep-scan
      - name: syft SBOM
        run: |
@@ -126,7 +172,7 @@ jobs:
        continue-on-error: true
      - name: ZAP baseline
-        uses: zaproxy/action-baseline@v0.10.0
+        uses: zaproxy/action-baseline@1e1871e84428617b969d4a1f981a8255630d54b0  # v0.10.0
        with:
          target: 'https://localhost:8443'
        continue-on-error: true
@@ -175,7 +221,7 @@ jobs:
      # --- Upload everything as artefacts ---
      - name: Upload deep-scan receipts
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02  # v4
        if: always()
        with:
          name: security-deep-scan-${{ github.run_id }}
@@ -10,6 +10,7 @@ bin/
 # Frontend
 web/node_modules/
 web/dist/
 web/.storybook-static/
 # Test binary, built with `go test -c`
 *.test
@@ -88,3 +89,17 @@ Thumbs.db
 # CERTCTL_TEST_CA_BUNDLE=./certs/ca.crt. Material is regenerated on every
 # `docker compose up` and never belongs in git.
 /deploy/test/certs/
 # Phase 1 RED-1 closure (2026-05-13): the f5-mock-icontrol Dockerfile
 # rebuilds from source via multi-stage build (deploy/test/f5-mock-icontrol/
 # Dockerfile line 13). The compiled ELF must not be tracked.
 deploy/test/f5-mock-icontrol/f5-mock-icontrol
 # Phase 0 closure (2026-05-13): cowork/ holds the operator's internal
 # legal / audit / strategy artifacts (counsel-signed AI-authorship
 # declaration, filter-repo callback, pre-rewrite bundle, audit HTML
 # scratch). It is private operator scratch space and must never
 # accidentally land in the public repo. See
 # docs/history-normalization.md for the public-facing description of
 # the Phase 0 git-history rewrite.
 cowork/
@@ -2,6 +2,73 @@
 ## Unreleased
 ### Breaking changes (scheduled for v2.2.0)
 - **SEC-H1 staged: `CERTCTL_AGENT_BOOTSTRAP_TOKEN_DENY_EMPTY` opt-in flag.**
  Phase 2 of the architecture diligence remediation (2026-05-13) introduces
  a new env var that, when set to `true`, makes the server refuse to start
  unless `CERTCTL_AGENT_BOOTSTRAP_TOKEN` is also set to a real value.
  Default in this release: `false` (preserves the v2.1.x warn-mode
  pass-through behavior for backward compatibility). Default flip to
  `true` is scheduled for v2.2.0 per `WORKSPACE-ROADMAP.md`.
  **Operator action before the v2.2.0 upgrade:** generate a real
  bootstrap token (`openssl rand -base64 32`) and set
  `CERTCTL_AGENT_BOOTSTRAP_TOKEN` in your env. When v2.2.0 ships, the
  deny-empty default flips to `true` and a missing or empty token will
  fail closed at boot. Operators with the token already set: no action
  required.
 - **SEC-M4: `CERTCTL_ACME_INSECURE` now requires explicit ACK.**
  Pre-Phase-2, `CERTCTL_ACME_INSECURE=true` produced only a boot-time
  WARN log. Post-Phase-2 (THIS release), the server refuses to start
  unless `CERTCTL_ACME_INSECURE_ACK=true` is set alongside it. ACME
  directory TLS verification is the load-bearing defense against a
  network attacker intercepting ACME enrollment; the existing flag was
  too easy to flip via a copy-pasted Pebble runbook.
  **Operator action:** if you intentionally run against a self-signed
  ACME server (Pebble, step-ca, internal dev), add
  `CERTCTL_ACME_INSECURE_ACK=true` to your env. Production deploys
  MUST never set either flag.
 - **SEC-H3: `CERTCTL_DEMO_MODE_ACK` is no longer sticky — 24h re-ack required.**
  Pre-Phase-2, setting `CERTCTL_DEMO_MODE_ACK=true` was sticky for the
  lifetime of the container. Post-Phase-2, operators must ALSO set
  `CERTCTL_DEMO_MODE_ACK_TS=$(date +%s)` to a unix epoch within the
  last 24h. The next container restart past 24h refuses to start
  unless a fresh TS is supplied. Catches the "forgotten demo deployment
  promoted to production" failure mode.
  **Operator action:** demo deploys must set `CERTCTL_DEMO_MODE_ACK_TS`
  at every `docker compose up`. The demo Compose helper script handles
  this automatically when wired; standalone demo deploys add it
  manually. Production deploys: this guard is irrelevant
  (`CERTCTL_DEMO_MODE_ACK` should not be set in production).
 ### Fixed
 - **GitHub #13 / Hotfix #19 — GUI "Something went wrong" after browser
  refresh on a real (non-demo) install.** Refresh-after-login wipes the
  in-memory `apiKey` (deliberate — the GUI never persists it to
  localStorage as a security posture). The next API call returns a
  bare 401 with no `WWW-Authenticate` header. Pre-Hotfix-19 the
  AuthProvider 401 handler only hard-navigated to `/login` when `cause`
  was a recognised OIDC session-expiry category (`idle_timeout` /
  `absolute_timeout` / `back_channel_revoked`); bare 401s
  (`cause === ''`) and `invalid_token` causes fell through to an
  in-place `AuthGate` state flip that unmounted `BrowserRouter` under
  an in-flight `<Link>`, triggering a `react-router-dom` invariant
  that surfaced via `ErrorBoundary` as the "Something went wrong"
  screen. **Fix:** every 401 now hard-navigates to `/login` regardless
  of cause; the cause-aware UX is preserved by forwarding
  `?session_expired=<cause>` only when cause is non-empty (bare 401s
  redirect to plain `/login`). Three-line change in
  `web/src/components/AuthProvider.tsx`; 4 regression tests added to
  `AuthProvider.test.tsx` (empty cause from `/targets`, `invalid_token`
  cause, `idle_timeout` cause, already-on-`/login` no-op guard).
  Closes #13.
 ### Security
 - **Alg-downgrade defense relaxed for Keycloak-shape IdPs (v2.1.0 pre-tag fix).**
@@ -2,9 +2,9 @@ Business Source License 1.1
 Parameters
-Licensor:             Shankar Kambam
+Licensor:             certctl LLC
 Licensed Work:        certctl
-                      The Licensed Work is © 2026 Shankar Kambam.
+                      The Licensed Work is © 2026 certctl LLC.
 Additional Use Grant: You may make use of the Licensed Work, including in
                      production for your internal business operations and
@@ -12,15 +12,23 @@ Additional Use Grant: You may make use of the Licensed Work, including in
                      your own customers, provided that you may not offer
                      the Licensed Work as a Commercial Certificate Service.
-                      A "Commercial Certificate Service" is a product or
+                      A "Commercial Certificate Service" is any product
-                      service whose principal value to a third party is the
+                      or service that provides third parties with access
                      to or control of any substantial set of the
                      certificate management functionality of the Licensed
                      Work — including but not limited to lifecycle
                      management, discovery, monitoring, alerting, renewal
-                      automation, deployment, and revocation — where the
+                      automation, deployment, revocation, certificate
-                      third party accesses or controls that functionality
+                      authority operation, certificate issuance,
-                      and compensation is received for that access or
+                      certificate signing, or any combination thereof —
-                      control.
+                      where compensation, in any form, is received in
                      connection with such access or control. This
                      restriction applies irrespective of whether such
                      functionality is the principal, ancillary,
                      supporting, or one of several values provided by the
                      product or service, and irrespective of whether the
                      Licensed Work is presented under its original name,
                      a modified name, or no name at all.
                      For the avoidance of doubt:
@@ -36,12 +44,17 @@ Additional Use Grant: You may make use of the Licensed Work, including in
                      (b) for the purposes of this Additional Use Grant,
                          "third party" excludes (i) your employees, (ii)
-                          your contractors acting on your behalf, and (iii)
+                          your contractors acting on your behalf, and
-                          your Affiliates. "Affiliate" means any entity
+                          (iii) your Affiliates. "Affiliate" means any
-                          that controls, is controlled by, or is under
+                          entity that (1) directly or indirectly controls
-                          common control with, you, where "control" means
+                          you, (2) is directly or indirectly controlled by
-                          ownership of more than fifty percent (50%) of
+                          you, or (3) is directly or indirectly under
-                          the voting interests of the entity;
+                          common control with you, where "control" means
                          either (A) ownership of more than fifty percent
                          (50%) of the voting interests of the entity, or
                          (B) the power to direct the management and
                          policies of the entity, whether through voting
                          securities, contract, or otherwise;
                      (c) the restriction on offering a Commercial
                          Certificate Service applies regardless of whether
@@ -67,16 +80,34 @@ works, redistribute, and make non-production use of the Licensed Work. The
 Licensor may make an Additional Use Grant, above, permitting limited production
 use.
-Effective on the Change Date, or the fourth anniversary of the first publicly
+Effective on the Change Date, the Licensor hereby grants you rights under
 available distribution of a specific version of the Licensed Work under this
 License, whichever comes first, the Licensor hereby grants you rights under
 the terms of the Change License, and the rights granted in the paragraph
 above terminate.
 If your use of the Licensed Work does not comply with the requirements
 currently in effect as described in this License, you must purchase a
 commercial license from the Licensor, its affiliated entities, or authorized
-resellers, or you must refrain from using the Licensed Work.
+resellers, or you must refrain from using the Licensed Work. Rights granted
 under any commercial license from the Licensor are personal to the licensee
 and may not be sublicensed, transferred, assigned, or resold to any third
 party without the Licensor's prior written consent. Any attempted sublicense,
 transfer, assignment, or resale in violation of this provision is void.
 Restricted Activities. Notwithstanding any other provision of this License,
 you may not:
  (i)   provide the Licensed Work or substantially similar functionality
        to third parties as a hosted, managed, embedded, bundled, or
        integrated service, except as expressly permitted in the
        Additional Use Grant;
  (ii)  move, change, disable, circumvent, or work around any license,
        security, attribution, audit-trail, or feature-gating
        functionality contained in the Licensed Work; or
  (iii) alter or remove any license, copyright, attribution, trademark,
        or other notice from the Licensed Work, its derivatives, or any
        substantial portion thereof.
 All copies of the original and modified Licensed Work, and derivative works
 of the Licensed Work, are subject to this License. This License applies
@@ -110,8 +141,12 @@ the Licensor or to any repository hosting the Licensed Work is provided at
 the submitter's sole risk, confers no rights or obligations on the
 Licensor, and is not incorporated into the Licensed Work.
-This License does not grant you any right in any trademark or logo of the
+Trademark and naming. This License does not grant you any right in any
-Licensor or its Affiliates.
+trademark, service mark, trade name, or logo of the Licensor or its
 Affiliates. Forks, derivative works, and modifications of the Licensed Work
 must not use the name "certctl," any name confusingly similar to "certctl,"
 or any Licensor trademark in their distributed form, marketing materials,
 package metadata, or service offerings.
 Governing law and venue. This License shall be governed by and construed in
 accordance with the laws of the State of Florida, USA, without giving
@@ -1,4 +1,4 @@
-.PHONY: help build run test lint verify verify-docs verify-deploy loadtest acme-cert-manager-test acme-rfc-conformance-test keycloak-integration-test okta-smoke-test benchmark-auth benchmark-auth-coldcache clean docker-up docker-down migrate-up migrate-down generate test-cover frontend-build qa-stats
+.PHONY: help build run test lint verify verify-deploy loadtest loadtest-scale loadtest-scale-bulk loadtest-scale-acme loadtest-scale-agent acme-cert-manager-test acme-rfc-conformance-test keycloak-integration-test okta-smoke-test benchmark-auth benchmark-auth-coldcache clean docker-up docker-down migrate-up migrate-down generate test-cover frontend-build e2e-test qa-stats
 # Default target - show help
 help:
@@ -16,7 +16,6 @@ help:
 	@echo "  make lint           Run linter (golangci-lint)"
 	@echo "  make fmt            Format code with gofmt"
 	@echo "  make verify         Pre-commit gate: fmt + vet + lint + test (CI-parity)"
 	@echo "  make verify-docs    Pre-tag gate:    QA-doc drift checks (operator-facing docs)"
 	@echo "  make verify-deploy  Pre-push gate:   digest validity + OpenAPI parity + docker build smoke"
 	@echo "  make loadtest       k6 throughput run against postgres + certctl (NOT in verify; manual + cron only)"
 	@echo ""
@@ -119,23 +118,6 @@ verify:
 	@echo ""
 	@echo "verify: PASS — safe to commit"
 # verify-docs: pre-tag gate. Runs the QA-doc seed-count drift guard
 # that ci-pipeline-cleanup Phase 11 / frozen decision 0.13 moved out
 # of CI (was per-push blocking; now operator-runs pre-tag). Protects
 # docs/contributor/qa-test-suite.md::Seed Data Reference from
 # drifting vs migrations/seed_demo.sql. Operator-facing docs only —
 # not product-affecting.
 #
 # The QA-doc Part-count drift guard retired in the 2026-05-04 docs
 # overhaul Phase 5 when docs/testing-guide.md was pruned (its content
 # dispersed across the audience-organized doc tree); the Part-count
 # class no longer exists outside the qa_test.go file itself.
 verify-docs:
 	@echo "==> QA-doc seed-count drift"
 	@bash scripts/qa-doc-seed-count.sh
 	@echo ""
 	@echo "verify-docs: PASS — safe to tag"
 # verify-deploy: optional pre-push gate. Runs the digest-validity check,
 # the OpenAPI ↔ handler parity check, and a Docker build smoke for the
 # production images (server + agent only — fast subset for local; CI
@@ -171,6 +153,49 @@ loadtest:
 	@echo "==> results landed in deploy/test/loadtest/results/"
 	@if [ -f deploy/test/loadtest/results/summary.txt ]; then cat deploy/test/loadtest/results/summary.txt; fi
 # Phase 8 SCALE-H2 — scale-tier load tests. Profile-gated in the
 # loadtest compose so the default `make loadtest` stays fast and
 # focused on the per-PR regression scope (API tier + connector tier).
 #
 # loadtest-scale-bulk runs the 10K-cert bulk-renew scenario.
 # loadtest-scale-acme runs the 200-VU ACME directory/nonce/ARI burst.
 # loadtest-scale-agent runs the 5K-agent heartbeat storm.
 #
 # Each target uses --exit-code-from <scenario-driver> so a threshold
 # breach surfaces as a non-zero make exit. The scale-seed init runs
 # once per invocation (idempotent via ON CONFLICT) so re-running a
 # target against the same compose stack is fine.
 loadtest-scale-bulk:
 	@echo "==> Phase 8 SCALE-H2: bulk-renewal scenario (10K cert fixture, ~6m)"
 	@cd deploy/test/loadtest && docker compose --profile scale up --build \
 	  --abort-on-container-exit --exit-code-from k6-scale-bulk
 	@echo ""
 	@echo "==> results: deploy/test/loadtest/results/summary-bulk-renewal.{json,txt}"
 	@if [ -f deploy/test/loadtest/results/summary-bulk-renewal.txt ]; then \
 	  cat deploy/test/loadtest/results/summary-bulk-renewal.txt; fi
 loadtest-scale-acme:
 	@echo "==> Phase 8 SCALE-H2: ACME enrollment burst (200 VU, ~6m)"
 	@cd deploy/test/loadtest && docker compose --profile scale up --build \
 	  --abort-on-container-exit --exit-code-from k6-scale-acme
 	@echo ""
 	@echo "==> results: deploy/test/loadtest/results/summary-acme-burst.{json,txt}"
 	@if [ -f deploy/test/loadtest/results/summary-acme-burst.txt ]; then \
 	  cat deploy/test/loadtest/results/summary-acme-burst.txt; fi
 loadtest-scale-agent:
 	@echo "==> Phase 8 SCALE-H2: agent heartbeat storm (5K agent fixture, ~6m)"
 	@cd deploy/test/loadtest && docker compose --profile scale up --build \
 	  --abort-on-container-exit --exit-code-from k6-scale-agent
 	@echo ""
 	@echo "==> results: deploy/test/loadtest/results/summary-agent-storm.{json,txt}"
 	@if [ -f deploy/test/loadtest/results/summary-agent-storm.txt ]; then \
 	  cat deploy/test/loadtest/results/summary-agent-storm.txt; fi
 # All three Phase 8 scenarios serially. Use the matrix in
 # .github/workflows/loadtest.yml for parallel CI runs.
 loadtest-scale: loadtest-scale-bulk loadtest-scale-acme loadtest-scale-agent
 # Auth Bundle 2 Phase 10 — Keycloak end-to-end OIDC integration test.
 # Boots a Keycloak container via testcontainers-go (quay.io/keycloak:25.0),
 # imports a canned realm with two groups + two users, and drives the
@@ -313,13 +338,23 @@ frontend-build:
 	cd web && npm ci && npx vite build
 	@echo "Frontend build complete"
-# QA Suite Stats — Bundle P / Strengthening #8.
+# Phase 3 TEST-M3 closure (2026-05-13): browser-driven E2E smoke
-# Single source-of-truth for every count claim in
+# target. The full 15-flow suite from web/src/__tests__/e2e/README.md
-# docs/contributor/qa-test-suite.md. The Strengthening #6 CI drift guards
+# ships in frontend-design-audit Phase 8; this target is the harness
-# (now scoped to the seed-count class only — the Part-count class retired
+# wiring that lets `make e2e-test` work today.
-# in the 2026-05-04 docs overhaul Phase 5 when testing-guide.md was
+#
-# pruned) consume the same numbers, eliminating the doc-drift class
+# First-time setup: `cd web && npm install && npx playwright install --with-deps chromium`.
-# structurally.
+# The webServer block in web/playwright.config.ts boots `npm run dev`
 # automatically; no separate `make docker-up` needed.
 e2e-test:
 	@echo "Running Playwright E2E (smoke + any *.spec.ts under web/src/__tests__/e2e/)..."
 	cd web && npx playwright test
 	@echo "E2E run complete"
 # qa-stats: snapshot of the test-suite size at the current commit.
 # Backend Go tests + subtests + fuzz targets + skipped sites, plus the
 # seed-data counts in migrations/seed_demo.sql. Useful before a release
 # to spot-check that no whole layer dropped off.
 qa-stats:
 	@echo "=== certctl QA Suite Stats ==="
 	@echo "Date: $$(date +%Y-%m-%d)"
@@ -0,0 +1,18 @@
 certctl
 Copyright 2026 certctl LLC.
 This product is distributed under the Business Source License 1.1.
 See LICENSE at the repository root for the full license text and
 the Additional Use Grant carve-outs.
 This product links third-party Go modules and JavaScript packages
 whose own license terms apply to those components. The full
 inventory of third-party dependencies and their respective licenses
 is enumerated in THIRD_PARTY_NOTICES.md at the repository root.
 Effective March 14, 2076, the BSL 1.1 license converts to the
 Apache License 2.0 per the Change Date in LICENSE.
 For inquiries about commercial licensing terms outside the
 Additional Use Grant — including the Commercial Certificate
 Service restriction — contact certctl@proton.me.
@@ -9,13 +9,17 @@
 [![GitHub Release](https://img.shields.io/github/v/release/certctl-io/certctl)](https://github.com/certctl-io/certctl/releases)
 [![GitHub Stars](https://img.shields.io/github/stars/certctl-io/certctl?style=flat&logo=github)](https://github.com/certctl-io/certctl/stargazers)
-certctl is a self-hosted platform that automates the entire TLS certificate lifecycle, from issuance through renewal to deployment, with zero human intervention. It works with any certificate authority, deploys to any server, and keeps private keys on your infrastructure where they belong. Free, source-available under BSL 1.1, covers the same lifecycle that enterprise platforms charge $100K+/year for.
+certctl is a self-hosted platform that automates the entire TLS certificate lifecycle, from issuance through renewal to deployment, with zero human intervention. Twelve native CA connectors plus an OpenSSL / shell-script adapter for custom CAs; fourteen production-ready native deployment-target connectors plus Kubernetes Secrets (preview) and a proxy-agent pattern for network appliances and agentless targets. In agent-mode (the default), private keys stay on the host they were generated on and never touch the control plane; a demo-only `CERTCTL_KEYGEN_MODE=server` flag mints keys server-side, refuses to start without an explicit `CERTCTL_DEMO_MODE_ACK=true` acknowledgement. Free, source-available under BSL 1.1, covers the same lifecycle that enterprise platforms charge $100K+/year for.
 The CA/Browser Forum's [Ballot SC-081v3](https://cabforum.org/2025/04/11/ballot-sc081v3-introduce-schedule-of-reducing-validity-and-data-reuse-periods/) caps public TLS certificates at **200 days by March 2026**, **100 days by 2027**, and **47 days by 2029**. At 47-day lifespans, a team managing 100 certificates is processing 7+ renewals per week, every week, forever. Manual workflows stop being a choice.
-> **Status: Early-access.** Production-quality core — Local CA, ACME, agent deployment, CRUD, audit, role-based authz (auditor split + day-0 bootstrap + four-eyes approval). Broader surface — intermediate CA hierarchy, ACME/SCEP/EST servers, network appliances — still maturing.
+> **Status: Early-access — actively looking for design partners.**
-> v2.1.0 ships federated identity in early-access: OIDC SSO across Keycloak, Authentik, Okta, Auth0, Entra ID, and Google Workspace; HMAC-signed server-side sessions with `__Host-` cookies and CSRF rotation; OIDC Back-Channel Logout; Argon2id break-glass admin. Lab and dev deployments encouraged; production welcomed with the understanding that customer-scale battle-testing is in progress — please [file issues](https://github.com/certctl-io/certctl/issues) on the federated-identity surface, where real-world IdP shapes surface fast.
+> The certificate lifecycle core is production-quality today: Local CA, ACME, agent deployment, audit, [role-based access control](docs/operator/rbac.md) with auditor split and four-eyes approval. v2.1.0 adds federated identity on top — [OIDC SSO](docs/operator/oidc-runbooks/index.md), server-side sessions, back-channel logout, and a break-glass admin path for SSO-outage recovery.
 > If your team runs PKI infrastructure that could use real automation, we'd love to have you on certctl. Lab and dev deployments are great. Production is welcome too — especially on the federated-identity surface, where real-world IdP shapes are exactly the exposure we can't manufacture in CI. Battle-testing certctl in your environment is genuinely valuable to us.
 > [File issues](https://github.com/certctl-io/certctl/issues) liberally. Every IdP quirk, every connector edge, every doc gap you hit — that's how the platform earns the right to drop the "early-access" label. The faster the loop, the faster everyone benefits.
 > **Actively maintained, shipping weekly.** [Open an issue](https://github.com/certctl-io/certctl/issues) if something breaks. CI runs the full test suite with race detection, static analysis, and vulnerability scanning on every commit.
@@ -31,7 +35,6 @@ The full audience-organized index lives at [`docs/README.md`](docs/README.md). T
 | Production operator | [Architecture](docs/reference/architecture.md) → [Security posture](docs/operator/security.md) → [Disaster recovery runbook](docs/operator/runbooks/disaster-recovery.md) |
 | PKI engineer | [ACME server](docs/reference/protocols/acme-server.md) → [SCEP server](docs/reference/protocols/scep-server.md) → [EST server](docs/reference/protocols/est.md) → [CA hierarchy](docs/reference/intermediate-ca-hierarchy.md) |
 | Migrating from another tool | [from certbot](docs/migration/from-certbot.md) / [from acme.sh](docs/migration/from-acmesh.md) / [cert-manager coexistence](docs/migration/cert-manager-coexistence.md) |
 | Contributor | [Architecture](docs/reference/architecture.md) → [Testing strategy](docs/contributor/testing-strategy.md) → [CI pipeline](docs/contributor/ci-pipeline.md) |
 For the connector reference (12 issuers, 15 targets, 6 notifiers) see [`docs/reference/connectors/index.md`](docs/reference/connectors/index.md).
@@ -61,7 +64,7 @@ Built for **platform engineering and DevOps teams** managing 10 to 500+ certific
 certctl handles the full certificate lifecycle in one self-hosted control plane:
 - **Issue and renew** from any CA. Let's Encrypt and any ACME provider, an embedded ACME server you can point cert-manager / certbot / lego at directly, a built-in local CA with sub-CA mode (chains under your enterprise root like ADCS), step-ca, Vault PKI, EJBCA, AWS ACM PCA, Google CAS, DigiCert, Sectigo, GlobalSign, Entrust, plus an OpenSSL / shell-script adapter for anything custom. Twelve native issuer connectors. See the [connector reference](docs/reference/connectors/index.md).
- **Deploy automatically** to NGINX, Apache, HAProxy, Caddy, Traefik, Envoy, IIS, Windows Cert Store, Java keystore, Kubernetes Secrets, AWS ACM, Azure Key Vault, SSH known-hosts, Postfix + Dovecot, F5 BIG-IP. Fifteen native target connectors. Every deploy goes through atomic-write + ownership-preservation + SHA-256 idempotency + per-target Prometheus counters + pre-deploy snapshot + on-failure rollback. See [`docs/reference/deployment-model.md`](docs/reference/deployment-model.md).
+- **Deploy automatically** to NGINX, Apache, HAProxy, Caddy, Traefik, Envoy, IIS, Windows Cert Store, Java keystore, AWS ACM, Azure Key Vault, SSH known-hosts, Postfix + Dovecot, F5 BIG-IP. **Fourteen production-ready native target connectors plus Kubernetes Secrets (preview).** File-based targets share an atomic-write + SHA-256 idempotency + on-failure rollback + per-target Prometheus counters primitive (the `deploy.Apply` path covers 12 of 13 file-based connectors). Cloud / API targets (AWS ACM, Azure Key Vault) use vendor-SDK semantics rather than the file primitive; F5 uses iControl REST transactions. The Kubernetes Secrets connector is shipped as preview because the production `client-go` integration is incomplete — see [`docs/reference/deployment-model.md`](docs/reference/deployment-model.md) for the per-target guarantee matrix. The reload / validate commands operators configure for shell-using targets (NGINX, Apache, HAProxy, Postfix, JavaKeystore, SSH) are validated server-side AND agent-side against shell-metacharacter injection before execution (see [`internal/connector/target/configcheck`](internal/connector/target/configcheck)).
 - **Run as an ACME server** so existing client tooling plugs in directly. RFC 8555 + RFC 9773 ARI, two per-profile auth modes (public-trust-style validation or trust_authenticated for internal PKI), doubly-signed key rollover, revoke-cert on both kid path and jwk path, per-account rate limiting. Cert-manager / certbot / lego all work pointed at it. See [`docs/reference/protocols/acme-server.md`](docs/reference/protocols/acme-server.md).
 - **Run as a SCEP server** for Microsoft Intune-managed phones, ChromeOS devices, network appliances. RFC 8894 native with full PKIMessage wire format, native Intune challenge dispatch with replay protection, per-profile dispatch with separate RA cert per profile. See [`docs/reference/protocols/scep-server.md`](docs/reference/protocols/scep-server.md).
 - **Run as an EST server** for HTTPS-based PKCS#10 enrollment. 802.1X / Wi-Fi authentication, IoT device enrollment, RFC 9266 channel binding. See [`docs/reference/protocols/est.md`](docs/reference/protocols/est.md).
@@ -72,11 +75,11 @@ certctl handles the full certificate lifecycle in one self-hosted control plane:
 - **Discover** existing certs across your fleet via filesystem scanning on agents, network TLS probing across CIDR ranges, and cloud secret manager imports (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager). Triage workflow for claim / dismiss / investigate.
 - **Revoke** with full RFC 5280 reason codes, DER CRL generation per issuer (scheduler-pre-generated and ETag-cached), and an embedded RFC 6960 OCSP responder with dedicated per-issuer responder certs. Single + bulk revocation. See [`docs/reference/protocols/crl-ocsp.md`](docs/reference/protocols/crl-ocsp.md).
 - **Alert** via Slack, Microsoft Teams, PagerDuty, OpsGenie, email, webhooks. Per-policy multi-channel routing matrix with severity tiers and fault-isolating per-channel dispatch. See [`docs/operator/runbooks/expiry-alerts.md`](docs/operator/runbooks/expiry-alerts.md).
- **Drive the platform from natural language** via the bundled MCP (Model Context Protocol) server. The full REST API is exposed as MCP tools — ask your AI client "show me all expiring certificates", "revoke the VPN cert, key compromised", or "what agents are offline?" and it translates to API calls. Stateless stdio-transport binary at `cmd/mcp-server/`; same auth as the REST API; no extra attack surface. See [`docs/reference/mcp.md`](docs/reference/mcp.md).
+- **Drive the platform from natural language** via the bundled MCP (Model Context Protocol) server. The bulk of the REST API surface is exposed as MCP tools — ask your AI client "show me all expiring certificates", "revoke the VPN cert, key compromised", or "what agents are offline?" and it translates to API calls. Stateless stdio-transport binary at `cmd/mcp-server/`; same auth as the REST API; no extra attack surface. MCP-vs-REST parity (162 tools covering 221 routes; the gap is a small allowlist of streaming + protocol-conformance endpoints that don't fit the request-response tool shape) is tracked in [`docs/reference/mcp-coverage.md`](docs/reference/mcp-coverage.md) with a CI guard that fails the build if a new REST route lands without either an MCP tool or an explicit allowlist entry. See [`docs/reference/mcp.md`](docs/reference/mcp.md).
 ## Architecture and security
-Go 1.25 control plane with handler → service → repository layering. PostgreSQL 16 backend with idempotent migrations. Pull-only deployment model — the server never initiates outbound connections. Agents poll for work and generate ECDSA P-256 keys locally so private keys never touch the control plane. For network appliances and agentless servers, a proxy agent in the same network zone handles deployment via the target's API (WinRM, iControl REST, SSH/SFTP). See the [Architecture Guide](docs/reference/architecture.md) for full system diagrams.
+Go 1.25 control plane with handler → service → repository layering. PostgreSQL 16 backend with idempotent migrations. Pull-only deployment model — the server never initiates outbound connections. **In agent-keygen mode (the production default), agents poll for work and generate ECDSA P-256 keys locally, so private keys never touch the control plane.** The opposite path (`CERTCTL_KEYGEN_MODE=server`) is demo-only and refuses to boot in production without an explicit `CERTCTL_DEMO_MODE_ACK=true` acknowledgement. For network appliances and agentless servers, a proxy agent in the same network zone handles deployment via the target's API (WinRM, iControl REST, SSH/SFTP). See the [Architecture Guide](docs/reference/architecture.md) for full system diagrams.
 Security: three authentication paths — API keys (SHA-256 hashed + constant-time compared), [OIDC SSO](docs/operator/oidc-runbooks/index.md) (Keycloak / Authentik / Okta / Auth0 / Entra ID / Google Workspace), and Argon2id [break-glass admin](docs/operator/security.md) for SSO-outage recovery. Successful OIDC login mints an HMAC-signed server-side session with `__Host-` cookies, CSRF rotation on every privileged write, and [RFC OIDC Back-Channel Logout](docs/reference/auth-standards-implemented.md) for IdP-driven session revoke. Role-based authorization on every gated handler with global / per-profile / per-issuer scope. Auditor split keeps regulator-class actors strictly read-only on the audit trail. Day-0 admin via a one-shot bootstrap token; granting or revoking roles requires the dedicated `auth.role.assign` permission. CORS deny-by-default. Shell injection prevention on all connector scripts. SSRF protection (reserved IP filtering) on the network scanner. Issuer + target + OIDC client_secret credentials encrypted at rest with AES-256-GCM. HTTPS-only control plane with TLS 1.3 pinned and a fail-closed startup gate that refuses to boot if the TLS bundle is unusable. Every API call recorded to an immutable audit trail with actor attribution, body hash, and latency tracking. CI runs race detection, static analysis, and vulnerability scanning on every commit. See [`docs/operator/security.md`](docs/operator/security.md) for the full posture and [`docs/operator/auth-threat-model.md`](docs/operator/auth-threat-model.md) for what's defended vs deferred.
@@ -84,15 +87,30 @@ Security: three authentication paths — API keys (SHA-256 hashed + constant-tim
 ### Docker Compose (recommended)
 **Demo path — zero config, populated dashboard:**
 ```bash
 git clone https://github.com/certctl-io/certctl.git
 cd certctl
-docker compose -f deploy/docker-compose.yml -f deploy/docker-compose.demo.yml up -d --build
+./deploy/demo-up.sh -d --build
 ```
-Wait ~30 seconds, then open **https://localhost:8443** in your browser. The shipped demo overlay seeds 180 days of realistic history across 13 issuers, 8 agents, managed + discovered certs, jobs, deploys, audit, and notification events. The `certctl-tls-init` init container self-signs an ECDSA-P256 cert on first boot — accept the browser warning for the demo, or feed the generated `ca.crt` to your client.
+Wait ~30 seconds, then open **https://localhost:8443** in your browser. The `demo-up.sh` wrapper exports a fresh `CERTCTL_DEMO_MODE_ACK_TS=$(date +%s)` and forwards the remaining args to `docker compose -f docker-compose.yml -f docker-compose.demo.yml up`. The timestamp export is required by the Phase 2 SEC-H3 fail-closed guard in `internal/config/config.go::Validate` — demo deploys must re-ACK every 24h so a forgotten demo container never silently ends up serving production traffic with `auth-type=none`. The bare `docker compose ... up` command without the timestamp refuses to boot; the wrapper script is the supported entry point.
-For a clean install without demo data, drop the `-f deploy/docker-compose.demo.yml` flag and run `docker compose -f deploy/docker-compose.yml up -d --build`. The four compose files (`docker-compose.yml` base, `docker-compose.demo.yml` overlay, `docker-compose.dev.yml` for PgAdmin + debug logging, `docker-compose.test.yml` for integration tests) are documented at [`deploy/ENVIRONMENTS.md`](deploy/ENVIRONMENTS.md).
+The demo overlay flips the base into demo-mode auth (every request served as the synthetic admin actor `actor-demo-anon` — the server emits a prominent ⚠ DEMO MODE banner at boot reminding you this posture is for evaluation only) and seeds 180 days of realistic history across 13 issuers, 8 agents, managed + discovered certs, jobs, deploys, audit, and notification events. The `certctl-tls-init` init container self-signs an ECDSA-P256 cert on first boot — accept the browser warning for the demo, or feed the generated `ca.crt` to your client.
 **Production path — `.env` required, fail-closed on placeholders:**
 ```bash
 cp .env.example deploy/.env       # or root .env if running outside compose
 "${EDITOR:-nano}" deploy/.env     # set POSTGRES_PASSWORD, CERTCTL_AUTH_SECRET,
                                   # CERTCTL_API_KEY, CERTCTL_CONFIG_ENCRYPTION_KEY,
                                   # CERTCTL_AGENT_ID — all via openssl rand
                                   # (replace nano with your preferred editor)
 docker compose -f deploy/docker-compose.yml up -d --build
 ```
 The base compose alone (no demo overlay) ships production-shaped: default `auth-type=api-key`, default `keygen-mode=agent`, no demo seed, no demo-mode synthetic admin. The fail-closed startup guards in `internal/config/config.go::Validate` refuse to boot when any of the change-me-... placeholder credentials reach config outside of demo mode (Bundle 2 closure, 2026-05-12). The four compose files (`docker-compose.yml` base, `docker-compose.demo.yml` overlay, `docker-compose.dev.yml` for PgAdmin + debug logging, `docker-compose.test.yml` for integration tests) are documented at [`deploy/ENVIRONMENTS.md`](deploy/ENVIRONMENTS.md).
 ```bash
 curl --cacert $(docker compose -f deploy/docker-compose.yml exec -T certctl-server cat /etc/certctl/tls/ca.crt) https://localhost:8443/health
@@ -112,12 +130,15 @@ Detects your OS and architecture, downloads the binary, configures systemd (Linu
 ### Helm chart (Kubernetes)
 ```bash
 # Required: TLS (pick one), server API key, and Postgres password.
 # The chart fail-fasts at template time if any required value is missing.
 helm install certctl deploy/helm/certctl/ \
-  --set server.auth.apiKey=your-api-key \
+  --set server.tls.existingSecret=<your-kubernetes.io/tls-secret-name> \
-  --set postgresql.password=your-db-password
+  --set server.auth.apiKey=$(openssl rand -base64 32) \
  --set postgresql.auth.password=$(openssl rand -base64 32)
 ```
-Production-ready chart with Server Deployment, PostgreSQL StatefulSet, Agent DaemonSet, health probes, security contexts (non-root, read-only rootfs), and optional Ingress. See [values.yaml](deploy/helm/certctl/values.yaml).
+Production-ready chart with Server Deployment, PostgreSQL StatefulSet (or external Postgres), Agent DaemonSet, health probes, container-scope security hardening (read-only rootfs, drop-all capabilities, non-root UID), optional PodDisruptionBudget, NetworkPolicy, Prometheus ServiceMonitor, and Ingress. See [values.yaml](deploy/helm/certctl/values.yaml) and the [external-Postgres example](deploy/helm/examples/values-external-db.yaml).
 ### Container images
@@ -156,8 +177,6 @@ make docker-up          # Start Docker Compose stack
 CI runs `go vet`, `go test -race`, `golangci-lint`, `govulncheck`, and per-package coverage thresholds (service 70%, handler 75%, crypto 88%, auth packages 85-95%) on every push. The thresholds-as-data file is `.github/coverage-thresholds.yml`; lowering a floor requires corresponding test work, not a config flip. Frontend CI runs TypeScript type checking, Vitest tests, and Vite production build.
 For the full contributor guide see [`docs/contributor/`](docs/contributor/) — testing strategy, test environment, CI pipeline, QA prerequisites.
 ## License
 Licensed under the [Business Source License 1.1](LICENSE). The source code is publicly available and free to use, modify, and self-host. The one restriction: you may not use certctl's certificate management functionality as part of a commercial certificate-management offering to third parties. See the LICENSE file for the full Additional Use Grant.
@@ -0,0 +1,161 @@
 # Third-Party Notices
 certctl is distributed under the Business Source License 1.1
 (see [LICENSE](LICENSE)). The binaries built from this source link
 third-party Go and JavaScript libraries listed below; certctl LLC
 acknowledges each library's authors and reproduces their copyright
 and license terms here in compliance with each library's license.
 Full license text for each library lives in that library's upstream
 repository. The license type is provided per-row; for the canonical
 notice, refer to the upstream source.
 - **Last reviewed:** 2026-05-13
 - **Holder:** certctl LLC
 - **License:** BSL 1.1 (Apache 2.0 effective March 14, 2076)
 ## Go Modules (binary-link dependencies)
 Generated by walking `go list -deps ./...` against the certctl
 server, agent, CLI, and MCP-server build paths. Excludes the Go
 standard library and the certctl-io/certctl module itself.
 **Count:** see commit; generate via `go list -deps -f '{{if .Module}}{{.Module.Path}} {{.Module.Version}}{{end}}' ./...`
 | Module | Version | License |
 |---|---|---|
 | `github.com/Azure/azure-sdk-for-go/sdk/azcore` | v1.20.0 | MIT |
 | `github.com/Azure/azure-sdk-for-go/sdk/azidentity` | v1.13.1 | MIT |
 | `github.com/Azure/azure-sdk-for-go/sdk/internal` | v1.11.2 | MIT |
 | `github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azcertificates` | v1.4.0 | MIT |
 | `github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/internal` | v1.2.0 | MIT |
 | `github.com/Azure/go-ntlmssp` | v0.1.1 | MIT |
 | `github.com/AzureAD/microsoft-authentication-library-for-go` | v1.6.0 | MIT |
 | `github.com/ChrisTrenkamp/goxpath` | v0.0.0-20210404020558-97928f7e12b6 | MIT |
 | `github.com/aws/aws-sdk-go-v2` | v1.41.7 | Apache-2.0 |
 | `github.com/aws/aws-sdk-go-v2/config` | v1.32.17 | Apache-2.0 |
 | `github.com/aws/aws-sdk-go-v2/credentials` | v1.19.16 | Apache-2.0 |
 | `github.com/aws/aws-sdk-go-v2/feature/ec2/imds` | v1.18.23 | Apache-2.0 |
 | `github.com/aws/aws-sdk-go-v2/internal/configsources` | v1.4.23 | Apache-2.0 |
 | `github.com/aws/aws-sdk-go-v2/internal/endpoints/v2` | v2.7.23 | Apache-2.0 |
 | `github.com/aws/aws-sdk-go-v2/internal/v4a` | v1.4.24 | Apache-2.0 |
 | `github.com/aws/aws-sdk-go-v2/service/acm` | v1.38.3 | Apache-2.0 |
 | `github.com/aws/aws-sdk-go-v2/service/acmpca` | v1.46.14 | Apache-2.0 |
 | `github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding` | v1.13.9 | Apache-2.0 |
 | `github.com/aws/aws-sdk-go-v2/service/internal/presigned-url` | v1.13.23 | Apache-2.0 |
 | `github.com/aws/aws-sdk-go-v2/service/signin` | v1.0.11 | Apache-2.0 |
 | `github.com/aws/aws-sdk-go-v2/service/sso` | v1.30.17 | Apache-2.0 |
 | `github.com/aws/aws-sdk-go-v2/service/ssooidc` | v1.35.21 | Apache-2.0 |
 | `github.com/aws/aws-sdk-go-v2/service/sts` | v1.42.1 | Apache-2.0 |
 | `github.com/aws/smithy-go` | v1.25.1 | Apache-2.0 |
 | `github.com/bodgit/ntlmssp` | v0.0.0-20240506230425-31973bb52d9b | BSD-2/3-Clause |
 | `github.com/bodgit/windows` | v1.0.1 | BSD-2/3-Clause |
 | `github.com/coreos/go-oidc/v3` | v3.18.0 | Apache-2.0 |
 | `github.com/go-jose/go-jose/v4` | v4.1.4 | Apache-2.0 |
 | `github.com/go-logr/logr` | v1.4.3 | Apache-2.0 |
 | `github.com/gofrs/uuid` | v4.4.0+incompatible | MIT |
 | `github.com/golang-jwt/jwt/v5` | v5.3.0 | MIT |
 | `github.com/google/jsonschema-go` | v0.4.2 | MIT |
 | `github.com/google/uuid` | v1.6.0 | BSD-2/3-Clause |
 | `github.com/hashicorp/go-cleanhttp` | v0.5.2 | MPL-2.0 |
 | `github.com/hashicorp/go-uuid` | v1.0.3 | MPL-2.0 |
 | `github.com/jcmturner/aescts/v2` | v2.0.0 | Apache-2.0 |
 | `github.com/jcmturner/dnsutils/v2` | v2.0.0 | Apache-2.0 |
 | `github.com/jcmturner/gofork` | v1.7.6 | BSD-2/3-Clause |
 | `github.com/jcmturner/goidentity/v6` | v6.0.1 | Apache-2.0 |
 | `github.com/jcmturner/gokrb5/v8` | v8.4.4 | Apache-2.0 |
 | `github.com/jcmturner/rpc/v2` | v2.0.3 | Apache-2.0 |
 | `github.com/kr/fs` | v0.1.0 | BSD-2/3-Clause |
 | `github.com/kylelemons/godebug` | v1.1.0 | Apache-2.0 |
 | `github.com/lib/pq` | v1.10.9 | MIT |
 | `github.com/masterzen/simplexml` | v0.0.0-20190410153822-31eea3082786 | Apache-2.0 |
 | `github.com/masterzen/winrm` | v0.0.0-20250927112105-5f8e6c707321 | Apache-2.0 |
 | `github.com/modelcontextprotocol/go-sdk` | v1.4.1 | Apache-2.0 |
 | `github.com/pkg/browser` | v0.0.0-20240102092130-5ac0b6a4141c | BSD-2/3-Clause |
 | `github.com/pkg/sftp` | v1.13.10 | BSD-2/3-Clause |
 | `github.com/segmentio/asm` | v1.1.3 | MIT |
 | `github.com/segmentio/encoding` | v0.5.4 | MIT |
 | `github.com/tidwall/transform` | v0.0.0-20201103190739-32f242e2dbde | ISC |
 | `github.com/yosida95/uritemplate/v3` | v3.0.2 | BSD-2/3-Clause |
 | `golang.org/x/crypto` | v0.50.0 | BSD-2/3-Clause |
 | `golang.org/x/net` | v0.53.0 | BSD-2/3-Clause |
 | `golang.org/x/oauth2` | v0.36.0 | BSD-2/3-Clause |
 | `golang.org/x/sync` | v0.20.0 | BSD-2/3-Clause |
 | `golang.org/x/sys` | v0.43.0 | BSD-2/3-Clause |
 | `golang.org/x/text` | v0.36.0 | BSD-2/3-Clause |
 | `software.sslmate.com/src/go-pkcs12` | v0.7.0 | BSD-2/3-Clause |
 ## JavaScript Packages (production transitive closure)
 Generated by walking the `dependencies` graph from `web/package.json`
 through `node_modules/`. Excludes devDependencies (Vitest, Playwright,
 Vite, etc.) since they don't ship in the distributed frontend bundle.
 | Package | Version | License |
 |---|---|---|
 | `@reduxjs/toolkit` | 2.11.2 | MIT |
 | `@remix-run/router` | 1.23.2 | MIT |
 | `@standard-schema/spec` | 1.1.0 | MIT |
 | `@standard-schema/utils` | 0.3.0 | MIT |
 | `@tanstack/query-core` | 5.90.20 | MIT |
 | `@tanstack/react-query` | 5.90.21 | MIT |
 | `@types/d3-array` | 3.2.2 | MIT |
 | `@types/d3-color` | 3.1.3 | MIT |
 | `@types/d3-ease` | 3.0.2 | MIT |
 | `@types/d3-interpolate` | 3.0.4 | MIT |
 | `@types/d3-path` | 3.1.1 | MIT |
 | `@types/d3-scale` | 4.0.9 | MIT |
 | `@types/d3-shape` | 3.1.8 | MIT |
 | `@types/d3-time` | 3.0.4 | MIT |
 | `@types/d3-timer` | 3.0.2 | MIT |
 | `@types/use-sync-external-store` | 0.0.6 | MIT |
 | `clsx` | 2.1.1 | MIT |
 | `d3-array` | 3.2.4 | ISC |
 | `d3-color` | 3.1.0 | ISC |
 | `d3-ease` | 3.0.1 | BSD-3-Clause |
 | `d3-format` | 3.1.2 | ISC |
 | `d3-interpolate` | 3.0.1 | ISC |
 | `d3-path` | 3.1.0 | ISC |
 | `d3-scale` | 4.0.2 | ISC |
 | `d3-shape` | 3.2.0 | ISC |
 | `d3-time` | 3.1.0 | ISC |
 | `d3-time-format` | 4.1.0 | ISC |
 | `d3-timer` | 3.0.1 | ISC |
 | `decimal.js-light` | 2.5.1 | MIT |
 | `es-toolkit` | 1.45.1 | MIT |
 | `eventemitter3` | 5.0.4 | MIT |
 | `immer` | 10.2.0 | MIT |
 | `internmap` | 2.0.3 | ISC |
 | `js-tokens` | 4.0.0 | MIT |
 | `loose-envify` | 1.4.0 | MIT |
 | `react` | 18.3.1 | MIT |
 | `react-dom` | 18.3.1 | MIT |
 | `react-redux` | 9.2.0 | MIT |
 | `react-router` | 6.30.3 | MIT |
 | `react-router-dom` | 6.30.3 | MIT |
 | `recharts` | 3.8.0 | MIT |
 | `redux` | 5.0.1 | MIT |
 | `redux-thunk` | 3.1.0 | MIT |
 | `reselect` | 5.1.1 | MIT |
 | `scheduler` | 0.23.2 | MIT |
 | `tiny-invariant` | 1.3.3 | MIT |
 | `use-sync-external-store` | 1.6.0 | MIT |
 | `victory-vendor` | 37.3.6 | MIT AND ISC |
 ## Test-fixture-only dependencies
 **Cisco libest.** The certctl integration test suite exercises the EST
 (RFC 7030) endpoints against Cisco's libest reference client. libest
 runs as a sidecar container (`certctl-test-libest`) only when the
 `est-e2e` Docker Compose profile is active — it is **not** vendored
 into the certctl source tree and **not** linked into any distributed
 release artifact (server, agent, CLI, MCP-server, container images,
 or release tarballs). For libest's own license terms, see
 <https://github.com/cisco/libest>.
 **f5-mock-icontrol.** The F5 deployment-target integration test
 ships a small Go program at `deploy/test/f5-mock-icontrol/main.go`
 under the same BSL 1.1 license as the rest of certctl. The compiled
 ELF was removed from the tracked tree in Phase 1 closure (commit
 eda3b48, 2026-05-13); it now rebuilds via the Dockerfile's
 multi-stage build on demand.
@@ -0,0 +1 @@
 0
@@ -1,30 +1,100 @@
 # Routes registered in internal/api/router/router.go that are intentionally
-# NOT in api/openapi.yaml. Each entry needs a one-line `why:` justification.
+# NOT in api/openapi.yaml. Each entry needs a one-line `why:` justification
 # AND a required `category:` field (added in Phase 13 Sprint 13.1,
 # 2026-05-14, architecture diligence audit ARCH-H1).
 #
 # Adding a new entry requires PR-time review.
 #
 # OpenAPI-shaped REST endpoints belong in api/openapi.yaml, NOT here.
-# This list is for protocol-shaped (SCEP wire endpoints) and operational
+# This list is for protocol-shaped (SCEP/ACME/EST wire endpoints) and
-# (health, metrics, pprof) routes only.
+# operational (health, metrics, pprof) routes only.
 #
 # Per ci-pipeline-cleanup bundle Phase 9 / frozen decision 0.11.
 #
 # ──────────────────────────────────────────────────────────────────────
 # The two-bucket contract (Phase 13 Sprint 13.1)
 # ──────────────────────────────────────────────────────────────────────
 #
 #   category: wire-protocol
 #     The route's wire shape is dictated by an IETF RFC (SCEP RFC 8894,
 #     ACME RFC 8555, ACME ARI RFC 9773, EST RFC 7030) or it's a
 #     sibling/shorthand variant of such a route (same wire semantics,
 #     different cosmetic path — e.g. trailing-slash forms, default-
 #     profile shorthands). Documenting these as REST operations in
 #     openapi.yaml would duplicate the RFC with no information gain;
 #     the canonical operator references live in docs/acme-server.md +
 #     docs/operator/scep.md + docs/operator/est.md. These entries
 #     NEVER burn down — they're protocol contracts, not gaps.
 #
 #   category: rest-deferred
 #     The route is REST-shaped (resource CRUD, JSON request/response,
 #     RBAC-gated) but its OpenAPI operation was deferred when the
 #     handler shipped. These MUST monotonically decrease to zero.
 #     Phase 13 Sprints 13.4-13.6 author the OpenAPI ops + delete the
 #     corresponding exception entries; the
 #     openapi-rest-deferred-monotonic.sh CI guard fails any PR that
 #     grows the rest-deferred bucket vs the checked-in baseline at
 #     api/openapi-handler-exceptions-baseline.txt.
 #
 # ──────────────────────────────────────────────────────────────────────
 # Phase 13 Sprint 13.1 categorization (2026-05-14)
 # ──────────────────────────────────────────────────────────────────────
 #
 # Current split, re-derived by the parity script's bucket-reporting
 # subcommand (post-Sprint-13.6 / 2026-05-14):
 #
 #   total entries:           36
 #   wire-protocol:           36
 #   rest-deferred:           0    ← THE FLOOR — ARCH-H1 substantive close
 #
 # Burn-down progress:
 #
 #   Sprint 13.4 SHIPPED — 28 - 13 = 15 (auth/sessions cluster 3 ops +
 #                               auth/oidc CRUD + JWKS + test + refresh
 #                               + group-mappings cluster, 10 ops)
 #   Sprint 13.5 SHIPPED — 15 -  8 =  7 (auth/breakglass admin 4 ops +
 #                               auth/users 3 ops + auth/runtime-config
 #                               1 op, 8 ops total)
 #   Sprint 13.6 SHIPPED —  7 -  7 =  0 (audit/export 1 op + demo-
 #                               residual/cleanup 1 op + auth/logout 1 op +
 #                               auth/breakglass/login 1 op + 3 OIDC
 #                               browser-flow endpoints, 7 ops total)
 #
 # Sprint 13.7 next tightens the parity-script's rest-deferred floor
 # from monotonic-decrease to a hard zero-exact pin. After that, any
 # new REST route MUST land with an OpenAPI op or fail CI — no escape
 # hatch via `category: rest-deferred`.
 #
 # Each authored OpenAPI op needs request/response schemas (not
 # placeholders) so the generated client at web/orval.config.ts emits
 # typed signatures. When an op lands, delete the corresponding entry
 # below + bump api/openapi-handler-exceptions-baseline.txt downward.
 documented_exceptions:
  - route: "GET /scep"
    why: "SCEP wire-protocol endpoint per RFC 8894 §3.1; serves CA certs via GetCACert/GetCACaps query params, NOT a REST resource."
    category: wire-protocol
  - route: "POST /scep"
    why: "SCEP wire-protocol endpoint per RFC 8894 §3.1; receives PKCSReq / RenewalReq PKIMessages, NOT a REST resource."
    category: wire-protocol
  - route: "GET /scep/"
    why: "SCEP wire-protocol endpoint with trailing-slash variant; ChromeOS clients send the trailing-slash form."
    category: wire-protocol
  - route: "POST /scep/"
    why: "SCEP wire-protocol endpoint with trailing-slash variant; ChromeOS clients send the trailing-slash form."
    category: wire-protocol
  - route: "GET /scep-mtls"
    why: "SCEP-mTLS sibling endpoint per ci-pipeline-cleanup-prerequisite EST RFC 7030 hardening Phase 6.5; same wire-protocol semantics, mutually-authenticated TLS variant."
    category: wire-protocol
  - route: "POST /scep-mtls"
    why: "SCEP-mTLS sibling endpoint, POST variant."
    category: wire-protocol
  - route: "GET /scep-mtls/"
    why: "SCEP-mTLS sibling endpoint, trailing-slash variant."
    category: wire-protocol
  - route: "POST /scep-mtls/"
    why: "SCEP-mTLS sibling endpoint, trailing-slash POST variant."
    category: wire-protocol
  # ACME server (RFC 8555 + RFC 9773 ARI) — wire-protocol surface.
  # Like SCEP/EST, ACME is a JWS-signed-JSON wire protocol whose
@@ -36,62 +106,90 @@ documented_exceptions:
  # challenge, cert, key-change, revoke-cert, renewal-info routes land.
  - route: "GET /acme/profile/{id}/directory"
    why: "ACME server RFC 8555 §7.1.1 directory; documented in docs/acme-server.md."
    category: wire-protocol
  - route: "HEAD /acme/profile/{id}/new-nonce"
    why: "ACME server RFC 8555 §7.2 new-nonce; documented in docs/acme-server.md."
    category: wire-protocol
  - route: "GET /acme/profile/{id}/new-nonce"
    why: "ACME server RFC 8555 §7.2 new-nonce GET form; documented in docs/acme-server.md."
    category: wire-protocol
  - route: "POST /acme/profile/{id}/new-account"
    why: "ACME server RFC 8555 §7.3 new-account (JWS jwk); documented in docs/acme-server.md."
    category: wire-protocol
  - route: "POST /acme/profile/{id}/account/{acc_id}"
    why: "ACME server RFC 8555 §7.3.2 + §7.3.6 (JWS kid) account update + deactivation; documented in docs/acme-server.md."
    category: wire-protocol
  - route: "GET /acme/directory"
    why: "ACME server default-profile shorthand; mirrors per-profile when CERTCTL_ACME_SERVER_DEFAULT_PROFILE_ID is set."
    category: wire-protocol
  - route: "HEAD /acme/new-nonce"
    why: "ACME server default-profile shorthand for new-nonce HEAD."
    category: wire-protocol
  - route: "GET /acme/new-nonce"
    why: "ACME server default-profile shorthand for new-nonce GET."
    category: wire-protocol
  - route: "POST /acme/new-account"
    why: "ACME server default-profile shorthand for new-account."
    category: wire-protocol
  - route: "POST /acme/account/{acc_id}"
    why: "ACME server default-profile shorthand for account update + deactivation."
    category: wire-protocol
  # Phase 2 — orders + finalize + authz + cert.
  - route: "POST /acme/profile/{id}/new-order"
    why: "ACME server RFC 8555 §7.4 new-order; documented in docs/acme-server.md."
    category: wire-protocol
  - route: "POST /acme/profile/{id}/order/{ord_id}"
    why: "ACME server RFC 8555 §7.4 order POST-as-GET; documented in docs/acme-server.md."
    category: wire-protocol
  - route: "POST /acme/profile/{id}/order/{ord_id}/finalize"
    why: "ACME server RFC 8555 §7.4 finalize; documented in docs/acme-server.md."
    category: wire-protocol
  - route: "POST /acme/profile/{id}/authz/{authz_id}"
    why: "ACME server RFC 8555 §7.5 authz POST-as-GET; documented in docs/acme-server.md."
    category: wire-protocol
  - route: "POST /acme/profile/{id}/challenge/{chall_id}"
    why: "ACME server RFC 8555 §7.5.1 challenge response; dispatches to Phase 3 validator pool."
    category: wire-protocol
  - route: "POST /acme/profile/{id}/cert/{cert_id}"
    why: "ACME server RFC 8555 §7.4.2 cert download; documented in docs/acme-server.md."
    category: wire-protocol
  - route: "POST /acme/new-order"
    why: "Phase 2 default-profile shorthand for new-order."
    category: wire-protocol
  - route: "POST /acme/order/{ord_id}"
    why: "Phase 2 default-profile shorthand for order POST-as-GET."
    category: wire-protocol
  - route: "POST /acme/order/{ord_id}/finalize"
    why: "Phase 2 default-profile shorthand for finalize."
    category: wire-protocol
  - route: "POST /acme/authz/{authz_id}"
    why: "Phase 2 default-profile shorthand for authz POST-as-GET."
    category: wire-protocol
  - route: "POST /acme/challenge/{chall_id}"
    why: "Phase 3 default-profile shorthand for challenge response."
    category: wire-protocol
  - route: "POST /acme/cert/{cert_id}"
    why: "Phase 2 default-profile shorthand for cert download."
    category: wire-protocol
  - route: "POST /acme/profile/{id}/key-change"
    why: "ACME server RFC 8555 §7.3.5 doubly-signed key rollover; documented in docs/acme-server.md."
    category: wire-protocol
  - route: "POST /acme/profile/{id}/revoke-cert"
    why: "ACME server RFC 8555 §7.6 revoke-cert (kid OR cert-key auth); documented in docs/acme-server.md."
    category: wire-protocol
  - route: "GET /acme/profile/{id}/renewal-info/{cert_id}"
    why: "ACME server RFC 9773 ACME Renewal Information (unauthenticated GET); documented in docs/acme-server.md."
    category: wire-protocol
  - route: "POST /acme/key-change"
    why: "Phase 4 default-profile shorthand for key rollover."
    category: wire-protocol
  - route: "POST /acme/revoke-cert"
    why: "Phase 4 default-profile shorthand for revoke-cert."
    category: wire-protocol
  - route: "GET /acme/renewal-info/{cert_id}"
    why: "Phase 4 default-profile shorthand for ARI."
    category: wire-protocol
  # =============================================================================
  # Auth Bundle 2 + audit-2026-05-10/11 fix bundle — REST endpoints not yet
@@ -101,59 +199,3 @@ documented_exceptions:
  # stays green for the v2.1.0 release tag. Threat model + handler contracts
  # live in docs/operator/{rbac.md,auth-threat-model.md,oidc-runbooks/*}.
  # =============================================================================
  - route: "GET /auth/oidc/login"
    why: "Bundle 2 Phase 5 OIDC login redirect; user-facing 302 with state cookie. OpenAPI rep deferred to pre-2.2.0."
  - route: "GET /auth/oidc/callback"
    why: "Bundle 2 Phase 5 OIDC callback handler; RFC 9700 §4.7.1 + RFC 9207. OpenAPI rep deferred to pre-2.2.0."
  - route: "POST /auth/logout"
    why: "Bundle 2 Phase 5 cookie + CSRF revoker. OpenAPI rep deferred to pre-2.2.0."
  - route: "POST /auth/breakglass/login"
    why: "Bundle 2 Phase 7.5 public break-glass login (auth-bypass, 404 when disabled). OpenAPI rep deferred to pre-2.2.0."
  - route: "POST /auth/oidc/back-channel-logout"
    why: "Bundle 2 Phase 5 RFC OIDC Back-Channel Logout 1.0 endpoint. OpenAPI rep deferred to pre-2.2.0."
  - route: "GET /api/v1/auth/sessions"
    why: "Bundle 2 Phase 5 self/admin session list. OpenAPI rep deferred to pre-2.2.0."
  - route: "DELETE /api/v1/auth/sessions/{id}"
    why: "Bundle 2 Phase 5 session revoke. OpenAPI rep deferred to pre-2.2.0."
  - route: "DELETE /api/v1/auth/sessions"
    why: "Bundle 2 audit-2026-05-10 MED-2/3 revoke-all-except-current."
  - route: "GET /api/v1/auth/oidc/providers"
    why: "Bundle 2 Phase 5 OIDC provider CRUD (list)."
  - route: "POST /api/v1/auth/oidc/providers"
    why: "Bundle 2 Phase 5 OIDC provider CRUD (create)."
  - route: "PUT /api/v1/auth/oidc/providers/{id}"
    why: "Bundle 2 Phase 5 OIDC provider CRUD (update)."
  - route: "DELETE /api/v1/auth/oidc/providers/{id}"
    why: "Bundle 2 Phase 5 OIDC provider CRUD (delete)."
  - route: "POST /api/v1/auth/oidc/providers/{id}/refresh"
    why: "Bundle 2 audit-2026-05-10 MED-7 JWKS hot-refresh."
  - route: "GET /api/v1/auth/oidc/providers/{id}/jwks-status"
    why: "Bundle 2 audit-2026-05-10 MED-7 JWKS health snapshot."
  - route: "POST /api/v1/auth/oidc/test"
    why: "Bundle 2 audit-2026-05-10 MED-5 dry-run discovery + JWKS + alg-downgrade check."
  - route: "GET /api/v1/auth/oidc/group-mappings"
    why: "Bundle 2 Phase 5 group-mapping CRUD (list)."
  - route: "POST /api/v1/auth/oidc/group-mappings"
    why: "Bundle 2 Phase 5 group-mapping CRUD (create)."
  - route: "DELETE /api/v1/auth/oidc/group-mappings/{id}"
    why: "Bundle 2 Phase 5 group-mapping CRUD (delete)."
  - route: "GET /api/v1/auth/breakglass/credentials"
    why: "Bundle 2 Phase 7.5 admin break-glass list (404 when disabled; password hash never on wire)."
  - route: "POST /api/v1/auth/breakglass/credentials"
    why: "Bundle 2 Phase 7.5 admin break-glass set/rotate password."
  - route: "POST /api/v1/auth/breakglass/credentials/{actor_id}/unlock"
    why: "Bundle 2 Phase 7.5 admin break-glass unlock after lockout."
  - route: "DELETE /api/v1/auth/breakglass/credentials/{actor_id}"
    why: "Bundle 2 Phase 7.5 admin break-glass credential delete."
  - route: "GET /api/v1/auth/users"
    why: "Bundle 2 audit-2026-05-10 MED-11 users page."
  - route: "DELETE /api/v1/auth/users/{id}"
    why: "Bundle 2 audit-2026-05-10 MED-11 user deactivate."
  - route: "POST /api/v1/auth/users/{id}/reactivate"
    why: "Bundle 2 audit-2026-05-10 MED-11 user reactivate."
  - route: "GET /api/v1/auth/runtime-config"
    why: "Bundle 2 audit-2026-05-10 MED-12 effective auth-runtime-config (read-only)."
  - route: "POST /api/v1/auth/demo-residual/cleanup"
    why: "Audit 2026-05-11 A-8 demo-mode residual-grants cleanup endpoint."
  - route: "GET /api/v1/audit/export"
    why: "Bundle 1 Phase 8 streaming NDJSON audit export."
@@ -0,0 +1,458 @@
 // Copyright 2026 certctl LLC. All rights reserved.
 // SPDX-License-Identifier: BUSL-1.1
 package main
 import (
 	"context"
 	"encoding/json"
 	"encoding/pem"
 	"fmt"
 	"io"
 	"net/http"
 	"os"
 	"strings"
 	"github.com/certctl-io/certctl/internal/connector/target"
 	"github.com/certctl-io/certctl/internal/connector/target/apache"
 	"github.com/certctl-io/certctl/internal/connector/target/awsacm"
 	"github.com/certctl-io/certctl/internal/connector/target/azurekv"
 	"github.com/certctl-io/certctl/internal/connector/target/caddy"
 	"github.com/certctl-io/certctl/internal/connector/target/envoy"
 	"github.com/certctl-io/certctl/internal/connector/target/f5"
 	"github.com/certctl-io/certctl/internal/connector/target/haproxy"
 	"github.com/certctl-io/certctl/internal/connector/target/iis"
 	jks "github.com/certctl-io/certctl/internal/connector/target/javakeystore"
 	k8s "github.com/certctl-io/certctl/internal/connector/target/k8ssecret"
 	"github.com/certctl-io/certctl/internal/connector/target/nginx"
 	pf "github.com/certctl-io/certctl/internal/connector/target/postfix"
 	sshconn "github.com/certctl-io/certctl/internal/connector/target/ssh"
 	"github.com/certctl-io/certctl/internal/connector/target/traefik"
 	wcs "github.com/certctl-io/certctl/internal/connector/target/wincertstore"
 )
 // Phase 9 ARCH-M2 closure Sprint 12 (2026-05-14): extracted from
 // cmd/agent/main.go via the Option B sibling-file pattern.
 //
 // This file holds the DEPLOYMENT executor + the target connector
 // factory + the deploy-only helpers:
 //
 //   - executeDeploymentJob: handles Pending deployment jobs by
 //     fetching the cert PEM from the control plane, loading the
 //     locally-held private key (in agent keygen mode), instantiating
 //     the appropriate target connector via createTargetConnector,
 //     calling DeployCertificate on it, and reporting Completed or
 //     Failed back to the control plane.
 //   - createTargetConnector: the big switch over target_type that
 //     instantiates one of 14 target connectors (apache / awsacm /
 //     azurekv / caddy / envoy / f5 / haproxy / iis / javakeystore /
 //     k8ssecret / nginx / postfix / ssh / traefik / wincertstore).
 //     Context is threaded into SDK-driven connectors (AWSACM,
 //     AzureKeyVault) so credential resolution honors caller
 //     cancellation per the contextcheck linter — see CI commit
 //     502823d.
 //   - splitPEMChain: split a PEM chain into (first cert, rest).
 //   - fetchCertificate: pull the PEM chain from
 //     GET /api/v1/certificates/{certID}/version.
 //
 // All 14 target-connector imports were used ONLY by
 // createTargetConnector; moving the factory here also moved the
 // 14 connector imports out of main.go, leaving the surviving
 // cmd/agent/main.go with the minimal stdlib surface its lifecycle
 // + HTTP infrastructure needs.
 // executeDeploymentJob executes a deployment job by fetching the certificate and deploying it
 // to the target system using the appropriate connector (NGINX, F5 BIG-IP, or IIS).
 //
 // For agent keygen mode, the private key is read from the local key store (keyDir/certID.key)
 // rather than fetched from the server. The deployment includes the locally-held key.
 //
 // Flow:
 // 1. Report job as Running
 // 2. Fetch the certificate PEM from the control plane
 // 3. Load local private key if it exists (agent keygen mode)
 // 4. Instantiate the target connector based on target_type from the work response
 // 5. Call DeployCertificate on the connector
 // 6. Report job as Completed (or Failed)
 func (a *Agent) executeDeploymentJob(ctx context.Context, job JobItem) {
 	a.logger.Info("executing deployment job",
 		"job_id", job.ID,
 		"certificate_id", job.CertificateID,
 		"target_type", job.TargetType)
 	// Report job as running
 	if err := a.reportJobStatus(ctx, job.ID, "Running", ""); err != nil {
 		a.logger.Error("failed to report job running", "error", err)
 	}
 	// Fetch the certificate from the control plane
 	certPEM, err := a.fetchCertificate(ctx, job.CertificateID)
 	if err != nil {
 		a.logger.Error("failed to fetch certificate",
 			"job_id", job.ID,
 			"error", err)
 		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("cert fetch failed: %v", err)); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 		}
 		return
 	}
 	a.logger.Info("certificate fetched for deployment",
 		"job_id", job.ID,
 		"cert_length", len(certPEM))
 	// Split PEM into cert and chain (separated by double newline between PEM blocks)
 	certOnly, chainPEM := splitPEMChain(certPEM)
 	// Check for locally-stored private key (agent keygen mode).
 	//
 	// SEC-002 closure (Sprint 1, 2026-05-16): safeAgentKeyPath validates
 	// the certificate_id shape AND asserts the joined path is contained
 	// within a.config.KeyDir. A crafted certificate_id (path traversal,
 	// absolute path, NUL byte, Windows separators) fails closed before
 	// any disk I/O. See cmd/agent/keymem.go for the helper.
 	keyPath, kerr := safeAgentKeyPath(a.config.KeyDir, job.CertificateID)
 	if kerr != nil {
 		a.logger.Error("agent key path validation failed for deployment",
 			"job_id", job.ID,
 			"certificate_id", job.CertificateID,
 			"error", kerr)
 		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("key path validation failed: %v", kerr)); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "error", reportErr)
 		}
 		return
 	}
 	var keyPEM string
 	keyData, err := os.ReadFile(keyPath)
 	if err != nil {
 		a.logger.Error("failed to read local private key for deployment",
 			"job_id", job.ID,
 			"key_path", keyPath,
 			"error", err)
 		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("key read failed: %v", err)); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "error", reportErr)
 		}
 		return
 	}
 	keyPEM = string(keyData)
 	a.logger.Info("loaded local private key for deployment",
 		"job_id", job.ID,
 		"key_path", keyPath)
 	// Deploy to the target using the appropriate connector
 	if job.TargetType != "" {
 		connector, err := a.createTargetConnector(ctx, job.TargetType, job.TargetConfig)
 		if err != nil {
 			a.logger.Error("failed to create target connector",
 				"job_id", job.ID,
 				"target_type", job.TargetType,
 				"error", err)
 			if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("connector init failed: %v", err)); reportErr != nil {
 				a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 			}
 			return
 		}
 		// Bundle 1 / RT-C1 closure (2026-05-12): defense in depth. The server
 		// runs internal/connector/target/configcheck.Validate on the way IN
 		// (Create/Update), and rejects shell metacharacters in command-bearing
 		// fields. Re-run the connector's full ValidateConfig here on the way
 		// OUT, before any DeployCertificate call. This catches (a) configs
 		// that pre-date the server-side guard, (b) corruption/tampering of
 		// the encrypted config blob, and (c) per-connector filesystem
 		// invariants (cert dir exists, paths writable) that the server can't
 		// check because the filesystem is on the agent host.
 		if err := connector.ValidateConfig(ctx, job.TargetConfig); err != nil {
 			a.logger.Error("connector config validation failed",
 				"job_id", job.ID,
 				"target_type", job.TargetType,
 				"error", err)
 			if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("%s config validation failed: %v", job.TargetType, err)); reportErr != nil {
 				a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 			}
 			return
 		}
 		deployReq := target.DeploymentRequest{
 			CertPEM:      certOnly,
 			KeyPEM:       keyPEM,
 			ChainPEM:     chainPEM,
 			TargetConfig: job.TargetConfig,
 			Metadata: map[string]string{
 				"certificate_id": job.CertificateID,
 				"job_id":         job.ID,
 			},
 		}
 		// Phase 2 of the deploy-hardening I master bundle:
 		// per-target deploy mutex. Acquire BEFORE
 		// DeployCertificate so two concurrent renewals against
 		// the same target ID serialize. The lock is held for the
 		// full Deploy duration including PreCommit (validate),
 		// PostCommit (reload), and post-deploy verify (Phases
 		// 4-9). Released on every return path via defer.
 		var targetID string
 		if job.TargetID != nil {
 			targetID = *job.TargetID
 		}
 		if mu := a.targetDeployMutex(targetID); mu != nil {
 			mu.Lock()
 			defer mu.Unlock()
 		}
 		result, err := connector.DeployCertificate(ctx, deployReq)
 		if err != nil {
 			a.logger.Error("deployment failed",
 				"job_id", job.ID,
 				"target_type", job.TargetType,
 				"error", err)
 			if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("deployment failed: %v", err)); reportErr != nil {
 				a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 			}
 			return
 		}
 		a.logger.Info("target connector deployment completed",
 			"job_id", job.ID,
 			"target_type", job.TargetType,
 			"success", result.Success,
 			"message", result.Message)
 		// If verification is enabled, verify the deployment by probing the live TLS endpoint
 		targetHost, targetPort, err := extractTargetHostAndPort(job.TargetConfig)
 		if err != nil {
 			a.logger.Warn("could not extract target host/port for verification",
 				"job_id", job.ID,
 				"error", err)
 		} else {
 			a.verifyAndReportDeployment(ctx, job, targetHost, targetPort, certOnly)
 		}
 	} else {
 		a.logger.Info("no target type specified, skipping connector invocation",
 			"job_id", job.ID)
 	}
 	// Report job as completed
 	if err := a.reportJobStatus(ctx, job.ID, "Completed", ""); err != nil {
 		a.logger.Error("failed to report job completed", "error", err)
 		return
 	}
 	a.logger.Info("deployment job completed", "job_id", job.ID)
 }
 // createTargetConnector instantiates the appropriate target connector based on type.
 // ctx is threaded into SDK-driven connectors (AWSACM, AzureKeyVault) so credential
 // resolution honors caller cancellation / deadlines instead of using a fresh
 // context.Background() (the contextcheck linter enforces this — the original Rank 5
 // implementation used Background() and tripped CI on commit 502823d).
 func (a *Agent) createTargetConnector(ctx context.Context, targetType string, configJSON json.RawMessage) (target.Connector, error) {
 	switch targetType {
 	case "NGINX":
 		var cfg nginx.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid NGINX config: %w", err)
 			}
 		}
 		return nginx.New(&cfg, a.logger), nil
 	case "Apache":
 		var cfg apache.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid Apache config: %w", err)
 			}
 		}
 		return apache.New(&cfg, a.logger), nil
 	case "HAProxy":
 		var cfg haproxy.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid HAProxy config: %w", err)
 			}
 		}
 		return haproxy.New(&cfg, a.logger), nil
 	case "F5":
 		var cfg f5.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid F5 config: %w", err)
 			}
 		}
 		conn, err := f5.New(&cfg, a.logger)
 		if err != nil {
 			return nil, fmt.Errorf("failed to create F5 connector: %w", err)
 		}
 		return conn, nil
 	case "IIS":
 		var cfg iis.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid IIS config: %w", err)
 			}
 		}
 		return iis.New(&cfg, a.logger)
 	case "Traefik":
 		var cfg traefik.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid Traefik config: %w", err)
 			}
 		}
 		return traefik.New(&cfg, a.logger), nil
 	case "Caddy":
 		var cfg caddy.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid Caddy config: %w", err)
 			}
 		}
 		return caddy.New(&cfg, a.logger), nil
 	case "Envoy":
 		var cfg envoy.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid Envoy config: %w", err)
 			}
 		}
 		return envoy.New(&cfg, a.logger), nil
 	case "Postfix":
 		var cfg pf.Config
 		cfg.Mode = "postfix"
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid Postfix config: %w", err)
 			}
 		}
 		return pf.New(&cfg, a.logger), nil
 	case "Dovecot":
 		var cfg pf.Config
 		cfg.Mode = "dovecot"
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid Dovecot config: %w", err)
 			}
 		}
 		return pf.New(&cfg, a.logger), nil
 	case "SSH":
 		var cfg sshconn.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid SSH config: %w", err)
 			}
 		}
 		return sshconn.New(&cfg, a.logger)
 	case "WinCertStore":
 		var cfg wcs.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid WinCertStore config: %w", err)
 			}
 		}
 		return wcs.New(&cfg, a.logger)
 	case "JavaKeystore":
 		var cfg jks.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid JavaKeystore config: %w", err)
 			}
 		}
 		return jks.New(&cfg, a.logger), nil
 	case "KubernetesSecrets":
 		var cfg k8s.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid KubernetesSecrets config: %w", err)
 			}
 		}
 		return k8s.New(&cfg, a.logger)
 	case "AWSACM":
 		// Rank 5 of the 2026-05-03 Infisical deep-research deliverable.
 		// AWS Certificate Manager target — SDK-driven (no file I/O).
 		// LoadDefaultConfig handles the standard AWS credential chain
 		// (IRSA / EC2 instance profile / SSO / env vars) without any
 		// long-lived creds in connector Config.
 		var cfg awsacm.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid AWSACM config: %w", err)
 			}
 		}
 		return awsacm.New(ctx, &cfg, a.logger)
 	case "AzureKeyVault":
 		// Rank 5 of the 2026-05-03 Infisical deep-research deliverable.
 		// Azure Key Vault target — SDK-driven (no file I/O).
 		// DefaultAzureCredential handles the standard Azure credential
 		// chain (managed identity / workload identity / env vars / az
 		// CLI fallback). Long-lived service-principal secrets are
 		// supported but discouraged via the credential_mode config.
 		var cfg azurekv.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid AzureKeyVault config: %w", err)
 			}
 		}
 		return azurekv.New(ctx, &cfg, a.logger)
 	default:
 		return nil, fmt.Errorf("unsupported target type: %s", targetType)
 	}
 }
 // splitPEMChain splits a PEM chain into the first certificate (cert) and the rest (chain).
 // The control plane returns the full chain as a single string with PEM blocks concatenated.
 func splitPEMChain(pemChain string) (string, string) {
 	data := []byte(pemChain)
 	block, rest := pem.Decode(data)
 	if block == nil {
 		return pemChain, ""
 	}
 	cert := string(pem.EncodeToMemory(block))
 	// Skip whitespace between cert and chain
 	chain := strings.TrimSpace(string(rest))
 	if chain == "" {
 		return cert, ""
 	}
 	return cert, chain
 }
 // fetchCertificate retrieves the certificate PEM chain from the control plane.
 // GET /api/v1/agents/{agentID}/certificates/{certID}
 func (a *Agent) fetchCertificate(ctx context.Context, certID string) (string, error) {
 	path := fmt.Sprintf("/api/v1/agents/%s/certificates/%s", a.config.AgentID, certID)
 	resp, err := a.makeRequest(ctx, http.MethodGet, path, nil)
 	if err != nil {
 		return "", fmt.Errorf("request failed: %w", err)
 	}
 	defer resp.Body.Close()
 	if resp.StatusCode != http.StatusOK {
 		body, _ := io.ReadAll(resp.Body)
 		return "", fmt.Errorf("server returned %d: %s", resp.StatusCode, string(body))
 	}
 	var certResp struct {
 		CertificatePEM string `json:"certificate_pem"`
 	}
 	if err := json.NewDecoder(resp.Body).Decode(&certResp); err != nil {
 		return "", fmt.Errorf("failed to decode response: %w", err)
 	}
 	return certResp.CertificatePEM, nil
 }
@@ -0,0 +1,275 @@
 // Copyright 2026 certctl LLC. All rights reserved.
 // SPDX-License-Identifier: BUSL-1.1
 package main
 import (
 	"context"
 	"crypto/ecdsa"
 	"crypto/rsa"
 	"crypto/sha256"
 	"crypto/x509"
 	"encoding/pem"
 	"fmt"
 	"io"
 	"net/http"
 	"os"
 	"path/filepath"
 	"strings"
 	"time"
 )
 // Phase 9 ARCH-M2 closure Sprint 12 (2026-05-14): extracted from
 // cmd/agent/main.go via the Option B sibling-file pattern.
 //
 // This file holds the filesystem DISCOVERY scan — the agent's
 // outbound surface for reporting pre-existing certificates it
 // finds on disk back to the control plane (POST /api/v1/agents/
 // {id}/discoveries, a machine-to-machine flow NOT exposed via the
 // MCP surface per the comment in
 // internal/mcp/tools.go::RegisterTools):
 //
 //   - runDiscoveryScan: walks each configured discovery directory,
 //     dispatches each candidate file to parsePEMFile or parseDERFile
 //     depending on extension, batches the parsed entries, and POSTs
 //     them in one report.
 //   - parsePEMFile / parseDERFile: extract every X.509 certificate
 //     from a candidate file in either encoding.
 //   - certToEntry: project a parsed *x509.Certificate into the
 //     discoveredCertEntry shape the control plane expects.
 //   - discoveredCertEntry struct + sha256Sum + certKeyInfo helpers
 //     consumed only by the discovery path; co-locating them keeps
 //     this file self-contained.
 // runDiscoveryScan walks configured directories, parses certificate files, and reports
 // discovered certificates to the control plane.
 // Supports PEM and DER encoded X.509 certificates.
 func (a *Agent) runDiscoveryScan(ctx context.Context) {
 	a.logger.Info("starting filesystem certificate discovery scan",
 		"directories", a.config.DiscoveryDirs)
 	startTime := time.Now()
 	var certs []discoveredCertEntry
 	var scanErrors []string
 	for _, dir := range a.config.DiscoveryDirs {
 		a.logger.Debug("scanning directory", "path", dir)
 		err := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
 			if err != nil {
 				scanErrors = append(scanErrors, fmt.Sprintf("walk error at %s: %v", path, err))
 				return nil // continue walking
 			}
 			if info.IsDir() {
 				return nil
 			}
 			// Skip files larger than 1MB (unlikely to be a certificate)
 			if info.Size() > 1*1024*1024 {
 				return nil
 			}
 			// Check file extension
 			ext := strings.ToLower(filepath.Ext(path))
 			switch ext {
 			case ".pem", ".crt", ".cer", ".cert":
 				found := a.parsePEMFile(path)
 				certs = append(certs, found...)
 			case ".der":
 				if entry, err := a.parseDERFile(path); err == nil {
 					certs = append(certs, entry)
 				} else {
 					a.logger.Debug("skipping non-cert DER file", "path", path, "error", err)
 				}
 			default:
 				// Try PEM parsing for extensionless files or unknown extensions
 				if ext == "" || ext == ".key" {
 					return nil // skip key files and extensionless
 				}
 				found := a.parsePEMFile(path)
 				if len(found) > 0 {
 					certs = append(certs, found...)
 				}
 			}
 			return nil
 		})
 		if err != nil {
 			scanErrors = append(scanErrors, fmt.Sprintf("failed to walk %s: %v", dir, err))
 		}
 	}
 	scanDuration := time.Since(startTime)
 	a.logger.Info("discovery scan completed",
 		"certificates_found", len(certs),
 		"errors", len(scanErrors),
 		"duration_ms", scanDuration.Milliseconds())
 	if len(certs) == 0 && len(scanErrors) == 0 {
 		a.logger.Debug("no certificates found and no errors, skipping report")
 		return
 	}
 	// Build report payload
 	entries := make([]map[string]interface{}, len(certs))
 	for i, c := range certs {
 		entries[i] = map[string]interface{}{
 			"fingerprint_sha256": c.FingerprintSHA256,
 			"common_name":        c.CommonName,
 			"sans":               c.SANs,
 			"serial_number":      c.SerialNumber,
 			"issuer_dn":          c.IssuerDN,
 			"subject_dn":         c.SubjectDN,
 			"not_before":         c.NotBefore,
 			"not_after":          c.NotAfter,
 			"key_algorithm":      c.KeyAlgorithm,
 			"key_size":           c.KeySize,
 			"is_ca":              c.IsCA,
 			"pem_data":           c.PEMData,
 			"source_path":        c.SourcePath,
 			"source_format":      c.SourceFormat,
 		}
 	}
 	report := map[string]interface{}{
 		"agent_id":         a.config.AgentID,
 		"directories":      a.config.DiscoveryDirs,
 		"certificates":     entries,
 		"errors":           scanErrors,
 		"scan_duration_ms": int(scanDuration.Milliseconds()),
 	}
 	// Submit to control plane
 	path := fmt.Sprintf("/api/v1/agents/%s/discoveries", a.config.AgentID)
 	resp, err := a.makeRequest(ctx, http.MethodPost, path, report)
 	if err != nil {
 		a.logger.Error("failed to submit discovery report", "error", err)
 		return
 	}
 	defer resp.Body.Close()
 	if resp.StatusCode != http.StatusAccepted {
 		body, _ := io.ReadAll(resp.Body)
 		a.logger.Error("discovery report rejected",
 			"status", resp.StatusCode,
 			"body", string(body))
 		return
 	}
 	a.logger.Info("discovery report submitted successfully",
 		"certificates", len(certs),
 		"errors", len(scanErrors))
 }
 // discoveredCertEntry holds parsed certificate metadata for reporting.
 type discoveredCertEntry struct {
 	FingerprintSHA256 string   `json:"fingerprint_sha256"`
 	CommonName        string   `json:"common_name"`
 	SANs              []string `json:"sans"`
 	SerialNumber      string   `json:"serial_number"`
 	IssuerDN          string   `json:"issuer_dn"`
 	SubjectDN         string   `json:"subject_dn"`
 	NotBefore         string   `json:"not_before"`
 	NotAfter          string   `json:"not_after"`
 	KeyAlgorithm      string   `json:"key_algorithm"`
 	KeySize           int      `json:"key_size"`
 	IsCA              bool     `json:"is_ca"`
 	PEMData           string   `json:"pem_data"`
 	SourcePath        string   `json:"source_path"`
 	SourceFormat      string   `json:"source_format"`
 }
 // parsePEMFile reads a file and extracts all X.509 certificates from PEM blocks.
 func (a *Agent) parsePEMFile(path string) []discoveredCertEntry {
 	data, err := os.ReadFile(path)
 	if err != nil {
 		a.logger.Debug("failed to read file", "path", path, "error", err)
 		return nil
 	}
 	var entries []discoveredCertEntry
 	rest := data
 	for {
 		var block *pem.Block
 		block, rest = pem.Decode(rest)
 		if block == nil {
 			break
 		}
 		if block.Type != "CERTIFICATE" {
 			continue
 		}
 		cert, err := x509.ParseCertificate(block.Bytes)
 		if err != nil {
 			a.logger.Debug("failed to parse certificate in PEM", "path", path, "error", err)
 			continue
 		}
 		pemStr := string(pem.EncodeToMemory(block))
 		entries = append(entries, certToEntry(cert, path, "PEM", pemStr))
 	}
 	return entries
 }
 // parseDERFile reads a DER-encoded certificate file.
 func (a *Agent) parseDERFile(path string) (discoveredCertEntry, error) {
 	data, err := os.ReadFile(path)
 	if err != nil {
 		return discoveredCertEntry{}, fmt.Errorf("read failed: %w", err)
 	}
 	cert, err := x509.ParseCertificate(data)
 	if err != nil {
 		return discoveredCertEntry{}, fmt.Errorf("parse failed: %w", err)
 	}
 	// Convert to PEM for storage
 	pemStr := string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: data}))
 	return certToEntry(cert, path, "DER", pemStr), nil
 }
 // certToEntry converts a parsed x509.Certificate into a discoveredCertEntry.
 func certToEntry(cert *x509.Certificate, path, format, pemData string) discoveredCertEntry {
 	// Compute SHA-256 fingerprint
 	fingerprint := fmt.Sprintf("%x", sha256Sum(cert.Raw))
 	// Determine key algorithm and size
 	keyAlg, keySize := certKeyInfo(cert)
 	return discoveredCertEntry{
 		FingerprintSHA256: fingerprint,
 		CommonName:        cert.Subject.CommonName,
 		SANs:              cert.DNSNames,
 		SerialNumber:      cert.SerialNumber.Text(16),
 		IssuerDN:          cert.Issuer.String(),
 		SubjectDN:         cert.Subject.String(),
 		NotBefore:         cert.NotBefore.UTC().Format(time.RFC3339),
 		NotAfter:          cert.NotAfter.UTC().Format(time.RFC3339),
 		KeyAlgorithm:      keyAlg,
 		KeySize:           keySize,
 		IsCA:              cert.IsCA,
 		PEMData:           pemData,
 		SourcePath:        path,
 		SourceFormat:      format,
 	}
 }
 // sha256Sum returns the SHA-256 hash of data.
 func sha256Sum(data []byte) [32]byte {
 	return sha256.Sum256(data)
 }
 // certKeyInfo extracts key algorithm name and size from a certificate.
 func certKeyInfo(cert *x509.Certificate) (string, int) {
 	switch pub := cert.PublicKey.(type) {
 	case *ecdsa.PublicKey:
 		return "ECDSA", pub.Curve.Params().BitSize
 	case *rsa.PublicKey:
 		return "RSA", pub.N.BitLen()
 	default:
 		switch cert.PublicKeyAlgorithm {
 		case x509.Ed25519:
 			return "Ed25519", 256
 		default:
 			return cert.PublicKeyAlgorithm.String(), 0
 		}
 	}
 }
@@ -1,3 +1,6 @@
 // Copyright 2026 certctl LLC. All rights reserved.
 // SPDX-License-Identifier: BUSL-1.1
 package main
 import (
@@ -6,6 +9,8 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
 	"regexp"
 	"strings"
 )
 // Bundle-9 / Audit L-002 + L-003 (agent edition).
@@ -38,6 +43,87 @@ func marshalAgentKeyAndZeroize(priv *ecdsa.PrivateKey, onDER func([]byte) error)
 	return onDER(der)
 }
 // SEC-002 closure (Sprint 1, 2026-05-16). The agent derives an on-disk
 // key path from job.CertificateID via filepath.Join. Pre-fix, a
 // crafted certificate_id ("../../etc/passwd", "/absolute/path",
 // "abc\x00d", "..\\Windows\\path") would drive arbitrary file
 // write/read on the agent host. The shape regex below mirrors the
 // server-side internal/validation.ValidateCertificateID gate — both
 // ends MUST hold for the load-bearing defense (the server can't be
 // trusted in isolation; a compromised control plane could deliver a
 // crafted job).
 //
 // agentCertIDPattern accepts ASCII letters, digits, ".", "_", "-",
 // bounded to 128 chars. Existing prefixed IDs (mc-..., cert-..., etc.)
 // satisfy this trivially. Deliberately rejects path separators (POSIX
 // and Windows), NUL byte, whitespace, control characters, and the
 // bare relative-path tokens "." and "..".
 var agentCertIDPattern = regexp.MustCompile(`^[A-Za-z0-9._-]{1,128}$`)
 // validateAgentCertID returns an error if id is not a well-formed
 // certificate identifier. Mirrors internal/validation.ValidateCertificateID
 // — the duplication is deliberate per the package-level comment
 // ("cmd/agent is a separate binary; copy-paste cheaper than lifting
 // a shared internal/keystore for a single shape check").
 func validateAgentCertID(id string) error {
 	if id == "" {
 		return fmt.Errorf("certificate_id is required")
 	}
 	if len(id) > 128 {
 		return fmt.Errorf("certificate_id length %d exceeds 128", len(id))
 	}
 	if !agentCertIDPattern.MatchString(id) {
 		return fmt.Errorf("certificate_id %q contains disallowed characters", id)
 	}
 	if id == "." || id == ".." {
 		return fmt.Errorf("certificate_id %q is a relative-path token", id)
 	}
 	return nil
 }
 // safeAgentKeyPath returns the on-disk key path for the given
 // certificateID, after validating the ID shape AND asserting the
 // joined path is contained within keyDir. Containment is the
 // authoritative guard — even if validateAgentCertID is bypassed (e.g.
 // a future refactor removes it), the post-Clean rel-path check below
 // rejects any path that escapes keyDir.
 //
 // The two-leg defense:
 //
 //	leg 1: shape check (validateAgentCertID)  → cheap up-front fail
 //	leg 2: containment check (filepath.Rel)   → load-bearing guard
 //
 // Returns the joined path on success, or a non-nil error describing
 // the rejected vector.
 func safeAgentKeyPath(keyDir, certificateID string) (string, error) {
 	if err := validateAgentCertID(certificateID); err != nil {
 		return "", err
 	}
 	if keyDir == "" {
 		return "", fmt.Errorf("safeAgentKeyPath: empty keyDir")
 	}
 	cleanDir, err := filepath.Abs(filepath.Clean(keyDir))
 	if err != nil {
 		return "", fmt.Errorf("safeAgentKeyPath: resolve keyDir: %w", err)
 	}
 	joined := filepath.Join(cleanDir, certificateID+".key")
 	cleanJoined := filepath.Clean(joined)
 	rel, err := filepath.Rel(cleanDir, cleanJoined)
 	if err != nil {
 		return "", fmt.Errorf("safeAgentKeyPath: rel(%q,%q): %w", cleanDir, cleanJoined, err)
 	}
 	// Reject any path that escapes the directory: a leading ".." in the
 	// relative form means the joined path resolved outside keyDir.
 	if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
 		return "", fmt.Errorf("safeAgentKeyPath: %q escapes keyDir %q (rel=%q)", certificateID, cleanDir, rel)
 	}
 	// Belt-and-suspenders: the rel form must also not contain a NUL.
 	if strings.ContainsRune(rel, 0) {
 		return "", fmt.Errorf("safeAgentKeyPath: NUL byte in computed path")
 	}
 	return cleanJoined, nil
 }
 // ensureAgentKeyDirSecure creates dir (and ancestors) with mode 0700 or
 // asserts an existing dir is owner-only. If a pre-existing dir is more
 // permissive than 0700 we tighten it to 0700 (logging-free; this is a
@@ -716,3 +716,113 @@ func TestKeymem_AgentMainFlowSmoke(t *testing.T) {
 		}
 	}
 }
 // =============================================================================
 // SEC-002 closure (Sprint 1, 2026-05-16) — safeAgentKeyPath path-traversal
 // regression coverage.
 //
 // Pre-fix the agent built the on-disk key path via:
 //
 //	keyPath := filepath.Join(a.config.KeyDir, job.CertificateID+".key")
 //
 // migrations/000001_initial_schema.up.sql declares
 // managed_certificates.id as TEXT PRIMARY KEY with no shape constraint, so
 // a crafted certificate_id from a compromised control plane (or a poisoned
 // DB row) could land outside KeyDir. The fix:
 //
 //   - validateAgentCertID rejects shape violations up-front
 //   - safeAgentKeyPath additionally asserts the joined path is contained
 //     within KeyDir via filepath.Rel; even a future refactor that drops
 //     the shape regex would still fail closed on escape.
 //
 // These tests pin both legs against the four vectors called out in the
 // audit (../../etc/passwd, /absolute/path, NUL byte, Windows separators).
 // =============================================================================
 func TestValidateAgentCertID_AcceptsCanonicalShapes(t *testing.T) {
 	for _, id := range []string{
 		"mc-cdn-edge",
 		"mc-cdn-edge-2026.q1",
 		"cert-1",
 		"abc123",
 		"MC-UPPER",
 	} {
 		t.Run(id, func(t *testing.T) {
 			if err := validateAgentCertID(id); err != nil {
 				t.Errorf("validateAgentCertID(%q): unexpected error %v", id, err)
 			}
 		})
 	}
 }
 func TestValidateAgentCertID_RejectsTraversalVectors(t *testing.T) {
 	cases := []struct {
 		name string
 		id   string
 	}{
 		{"empty", ""},
 		{"parent_token", ".."},
 		{"current_token", "."},
 		{"posix_traversal", "../../etc/passwd"},
 		{"absolute_posix", "/absolute/path"},
 		{"windows_traversal", `..\..\evil`},
 		{"windows_separator", `bad\path`},
 		{"nul_byte", "abc\x00def"},
 		{"newline", "abc\ndef"},
 		{"space", "id with spaces"},
 		{"overlong", strings.Repeat("a", 129)},
 	}
 	for _, tc := range cases {
 		t.Run(tc.name, func(t *testing.T) {
 			if err := validateAgentCertID(tc.id); err == nil {
 				t.Errorf("id=%q: expected rejection, got nil", tc.id)
 			}
 		})
 	}
 }
 func TestSafeAgentKeyPath_HappyPath_ProducesContainedPath(t *testing.T) {
 	keyDir := t.TempDir()
 	got, err := safeAgentKeyPath(keyDir, "mc-good")
 	if err != nil {
 		t.Fatalf("safeAgentKeyPath: %v", err)
 	}
 	want := filepath.Join(keyDir, "mc-good.key")
 	// filepath.Clean normalisation may strip a trailing separator, etc.;
 	// compare canonical forms.
 	if filepath.Clean(got) != filepath.Clean(want) {
 		t.Errorf("safeAgentKeyPath = %q; want %q", got, want)
 	}
 }
 func TestSafeAgentKeyPath_RejectsTraversalVectors(t *testing.T) {
 	keyDir := t.TempDir()
 	cases := []struct {
 		name string
 		id   string
 	}{
 		{"posix_traversal", "../../etc/passwd"},
 		{"absolute_posix", "/etc/passwd"},
 		{"parent_token", ".."},
 		{"current_token", "."},
 		{"windows_traversal", `..\..\evil`},
 		{"windows_separator", `bad\path`},
 		{"nul_byte", "abc\x00def"},
 		{"empty", ""},
 	}
 	for _, tc := range cases {
 		t.Run(tc.name, func(t *testing.T) {
 			_, err := safeAgentKeyPath(keyDir, tc.id)
 			if err == nil {
 				t.Errorf("id=%q: expected rejection, got nil", tc.id)
 			}
 		})
 	}
 }
 func TestSafeAgentKeyPath_RejectsEmptyKeyDir(t *testing.T) {
 	_, err := safeAgentKeyPath("", "mc-good")
 	if err == nil {
 		t.Errorf("empty keyDir: expected rejection, got nil")
 	}
 }
@@ -1,51 +1,32 @@
 // Copyright 2026 certctl LLC. All rights reserved.
 // SPDX-License-Identifier: BUSL-1.1
 package main
 import (
 	"bytes"
 	"context"
 	"crypto/ecdsa"
 	"crypto/elliptic"
 	"crypto/rand"
 	"crypto/rsa"
 	"crypto/sha256"
 	"crypto/tls"
 	"crypto/x509"
 	"crypto/x509/pkix"
 	"encoding/json"
 	"encoding/pem"
 	"errors"
 	"flag"
 	"fmt"
 	"io"
 	"log/slog"
 	"math/rand/v2"
 	"net"
 	"net/http"
 	"net/url"
 	"os"
 	"os/signal"
 	"path/filepath"
 	"runtime"
 	"strings"
 	"sync"
 	"syscall"
 	"time"
-	"github.com/certctl-io/certctl/internal/connector/target"
+	"github.com/certctl-io/certctl/internal/scheduler"
 	"github.com/certctl-io/certctl/internal/connector/target/apache"
 	"github.com/certctl-io/certctl/internal/connector/target/awsacm"
 	"github.com/certctl-io/certctl/internal/connector/target/azurekv"
 	"github.com/certctl-io/certctl/internal/connector/target/caddy"
 	"github.com/certctl-io/certctl/internal/connector/target/envoy"
 	"github.com/certctl-io/certctl/internal/connector/target/f5"
 	"github.com/certctl-io/certctl/internal/connector/target/haproxy"
 	"github.com/certctl-io/certctl/internal/connector/target/iis"
 	jks "github.com/certctl-io/certctl/internal/connector/target/javakeystore"
 	k8s "github.com/certctl-io/certctl/internal/connector/target/k8ssecret"
 	"github.com/certctl-io/certctl/internal/connector/target/nginx"
 	pf "github.com/certctl-io/certctl/internal/connector/target/postfix"
 	sshconn "github.com/certctl-io/certctl/internal/connector/target/ssh"
 	"github.com/certctl-io/certctl/internal/connector/target/traefik"
 	wcs "github.com/certctl-io/certctl/internal/connector/target/wincertstore"
 )
 // AgentConfig represents the agent-side configuration.
@@ -253,15 +234,49 @@ func (a *Agent) Run(ctx context.Context) error {
 		a.logger.Warn("failed to enforce key directory permissions", "path", a.config.KeyDir, "error", err)
 	}
-	// Create ticker channels for heartbeat, polling, and discovery
+	// SCALE-006 closure (Sprint 2, 2026-05-16). Pre-fix the agent
-	heartbeatTicker := time.NewTicker(a.heartbeatInterval)
+	// started its heartbeat + poll loops on fixed time.NewTicker
 	// cadence with an unjittered immediate first invocation. Mass
 	// restarts (rolling K8s deploy, control-plane reboot, scheduled
 	// fleet bounce) produced a thundering herd — 5K agents booting
 	// in a 10-second window all hit /heartbeat in lockstep, then
 	// /poll, every interval forever afterward.
 	//
 	// Fix: (1) sleep a random startup-jitter ∈ [0, interval) before
 	// the first heartbeat + first poll to spread the initial cohort,
 	// and (2) use scheduler.JitteredTicker (±10% per-tick envelope)
 	// for the recurring ticks so the cohort stays spread across
 	// every tick boundary. Both legs use the existing in-tree
 	// JitteredTicker primitive (internal/scheduler/jitter.go) —
 	// pattern already exercised by every scheduler.go loop on the
 	// server side.
 	heartbeatTicker := scheduler.NewJitteredTicker(a.heartbeatInterval, scheduler.DefaultSchedulerJitter)
 	defer heartbeatTicker.Stop()
-
+	pollTicker := scheduler.NewJitteredTicker(a.pollInterval, scheduler.DefaultSchedulerJitter)
 	pollTicker := time.NewTicker(a.pollInterval)
 	defer pollTicker.Stop()
-	// Run initial heartbeat and poll
+	// Startup jitter — run-first delay drawn fresh per-agent so a
 	// 5K-agent rolling-restart spreads out across (max interval).
 	// Bounded by ctx so a sigint-during-startup exits cleanly rather
 	// than hanging on the Sleep. Heartbeat and poll are drawn
 	// independently so a single random seed doesn't create a
 	// secondary correlation pattern.
 	hbJitter := time.Duration(rand.Int64N(int64(a.heartbeatInterval)))
 	pollJitter := time.Duration(rand.Int64N(int64(a.pollInterval)))
 	a.logger.Info("startup jitter applied",
 		"heartbeat_jitter", hbJitter.String(),
 		"poll_jitter", pollJitter.String())
 	select {
 	case <-ctx.Done():
 		return ctx.Err()
 	case <-time.After(hbJitter):
 	}
 	a.sendHeartbeat(ctx)
 	select {
 	case <-ctx.Done():
 		return ctx.Err()
 	case <-time.After(pollJitter):
 	}
 	a.pollForWork(ctx)
 	// Discovery: run initial scan if directories configured, then on interval
@@ -391,598 +406,6 @@ func (a *Agent) sendHeartbeat(ctx context.Context) {
 	a.logger.Debug("heartbeat acknowledged")
 }
 // pollForWork queries the control plane for actionable jobs and processes them.
 // Jobs may be deployment jobs (Pending) or CSR jobs (AwaitingCSR).
 // GET /api/v1/agents/{agentID}/work
 func (a *Agent) pollForWork(ctx context.Context) {
 	a.logger.Debug("polling for work", "agent_id", a.config.AgentID)
 	path := fmt.Sprintf("/api/v1/agents/%s/work", a.config.AgentID)
 	resp, err := a.makeRequest(ctx, http.MethodGet, path, nil)
 	if err != nil {
 		a.logger.Error("work poll failed", "error", err)
 		a.consecutiveFailures++
 		return
 	}
 	defer resp.Body.Close()
 	// I-004: same terminal-retirement handling as sendHeartbeat. Work-poll is the
 	// other hot path that can observe an agent's soft-retirement; if the
 	// heartbeat tick happens to fire after a work-poll tick within the same
 	// retirement window, this branch catches it first. markRetired's sync.Once
 	// guards idempotency so racing both paths in the same tick only closes the
 	// signal channel once. No consecutiveFailures increment — retirement is
 	// not a transient failure.
 	if resp.StatusCode == http.StatusGone {
 		body, _ := io.ReadAll(resp.Body)
 		a.markRetired("work_poll", resp.StatusCode, string(body))
 		return
 	}
 	if resp.StatusCode != http.StatusOK {
 		body, _ := io.ReadAll(resp.Body)
 		a.logger.Error("work poll rejected",
 			"status", resp.StatusCode,
 			"body", string(body))
 		a.consecutiveFailures++
 		return
 	}
 	var workResp WorkResponse
 	if err := json.NewDecoder(resp.Body).Decode(&workResp); err != nil {
 		a.logger.Error("failed to decode work response", "error", err)
 		a.consecutiveFailures++
 		return
 	}
 	a.consecutiveFailures = 0
 	if workResp.Count == 0 {
 		a.logger.Debug("no pending work")
 		return
 	}
 	a.logger.Info("received work", "job_count", workResp.Count)
 	// Process each job based on type and status
 	for _, job := range workResp.Jobs {
 		switch {
 		case job.Status == "AwaitingCSR":
 			// Agent keygen mode: generate key locally, create CSR, submit to server
 			a.executeCSRJob(ctx, job)
 		case job.Type == "Deployment":
 			a.executeDeploymentJob(ctx, job)
 		}
 	}
 }
 // executeCSRJob handles an AwaitingCSR job: generates a private key locally, creates a CSR,
 // and submits it to the control plane for signing. The private key is stored on the local
 // filesystem with 0600 permissions and NEVER sent to the server.
 //
 // Flow:
 // 1. Generate ECDSA P-256 key pair
 // 2. Store private key to disk (keyDir/certID.key) with 0600 permissions
 // 3. Create CSR with common name and SANs from work response
 // 4. Submit CSR to control plane via POST /agents/{id}/csr
 // 5. Server signs the CSR and creates a cert version + deployment jobs
 func (a *Agent) executeCSRJob(ctx context.Context, job JobItem) {
 	a.logger.Info("executing CSR job (agent-side key generation)",
 		"job_id", job.ID,
 		"certificate_id", job.CertificateID,
 		"common_name", job.CommonName)
 	// Step 1: Generate ECDSA P-256 key pair
 	privKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
 	if err != nil {
 		a.logger.Error("failed to generate private key",
 			"job_id", job.ID,
 			"error", err)
 		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("key generation failed: %v", err)); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 		}
 		return
 	}
 	a.logger.Info("generated ECDSA P-256 key pair locally",
 		"job_id", job.ID,
 		"certificate_id", job.CertificateID)
 	// Step 2: Store private key to disk with secure permissions.
 	//
 	// Bundle-9 / Audit L-002 + L-003: marshal+write through helpers that
 	// (a) zeroize the in-heap DER buffer immediately after the PEM block is
 	// constructed so the private scalar's exposure window is bounded by
 	// this function call, and (b) assert the key directory is mode 0700
 	// before any write touches disk. Also defer-clear the PEM buffer for
 	// the same reason — the encoded key isn't sensitive in transit (it's
 	// going to disk) but lingers on the heap if we don't.
 	keyPath := filepath.Join(a.config.KeyDir, job.CertificateID+".key")
 	if err := ensureAgentKeyDirSecure(filepath.Dir(keyPath)); err != nil {
 		a.logger.Error("agent key dir hardening failed", "job_id", job.ID, "error", err)
 		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("key dir hardening failed: %v", err)); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 		}
 		return
 	}
 	var privKeyPEM []byte
 	if marshalErr := marshalAgentKeyAndZeroize(privKey, func(der []byte) error {
 		privKeyPEM = pem.EncodeToMemory(&pem.Block{
 			Type:  "EC PRIVATE KEY",
 			Bytes: der,
 		})
 		return nil
 	}); marshalErr != nil {
 		a.logger.Error("failed to marshal private key",
 			"job_id", job.ID,
 			"error", marshalErr)
 		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("key marshal failed: %v", marshalErr)); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 		}
 		return
 	}
 	defer clear(privKeyPEM)
 	if err := os.WriteFile(keyPath, privKeyPEM, 0600); err != nil {
 		a.logger.Error("failed to write private key to disk",
 			"job_id", job.ID,
 			"key_path", keyPath,
 			"error", err)
 		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("key storage failed: %v", err)); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 		}
 		return
 	}
 	a.logger.Info("private key stored securely",
 		"job_id", job.ID,
 		"key_path", keyPath,
 		"permissions", "0600")
 	// Validate common name is present
 	if job.CommonName == "" {
 		a.logger.Error("empty common name in CSR job", "job_id", job.ID)
 		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", "empty common name"); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "error", reportErr)
 		}
 		return
 	}
 	// Step 3: Create CSR with common name and SANs
 	// Split SANs into DNS names and email addresses for proper CSR encoding
 	var dnsNames []string
 	var emailAddresses []string
 	for _, san := range job.SANs {
 		if strings.Contains(san, "@") {
 			emailAddresses = append(emailAddresses, san)
 		} else {
 			dnsNames = append(dnsNames, san)
 		}
 	}
 	csrTemplate := &x509.CertificateRequest{
 		Subject: pkix.Name{
 			CommonName: job.CommonName,
 		},
 		DNSNames:       dnsNames,
 		EmailAddresses: emailAddresses,
 	}
 	csrDER, err := x509.CreateCertificateRequest(rand.Reader, csrTemplate, privKey)
 	if err != nil {
 		a.logger.Error("failed to create CSR",
 			"job_id", job.ID,
 			"error", err)
 		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("CSR creation failed: %v", err)); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 		}
 		return
 	}
 	csrPEM := string(pem.EncodeToMemory(&pem.Block{
 		Type:  "CERTIFICATE REQUEST",
 		Bytes: csrDER,
 	}))
 	// Step 4: Submit CSR to the control plane (only the public key leaves the agent)
 	a.logger.Info("submitting CSR to control plane",
 		"job_id", job.ID,
 		"certificate_id", job.CertificateID)
 	submitPath := fmt.Sprintf("/api/v1/agents/%s/csr", a.config.AgentID)
 	resp, err := a.makeRequest(ctx, http.MethodPost, submitPath, map[string]string{
 		"csr_pem":        csrPEM,
 		"certificate_id": job.CertificateID,
 	})
 	if err != nil {
 		a.logger.Error("failed to submit CSR",
 			"job_id", job.ID,
 			"error", err)
 		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("CSR submission failed: %v", err)); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 		}
 		return
 	}
 	defer resp.Body.Close()
 	if resp.StatusCode != http.StatusAccepted {
 		body, _ := io.ReadAll(resp.Body)
 		a.logger.Error("CSR submission rejected",
 			"job_id", job.ID,
 			"status", resp.StatusCode,
 			"body", string(body))
 		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("CSR rejected: %s", string(body))); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 		}
 		return
 	}
 	a.logger.Info("CSR submitted and signed successfully",
 		"job_id", job.ID,
 		"certificate_id", job.CertificateID,
 		"key_path", keyPath)
 }
 // executeDeploymentJob executes a deployment job by fetching the certificate and deploying it
 // to the target system using the appropriate connector (NGINX, F5 BIG-IP, or IIS).
 //
 // For agent keygen mode, the private key is read from the local key store (keyDir/certID.key)
 // rather than fetched from the server. The deployment includes the locally-held key.
 //
 // Flow:
 // 1. Report job as Running
 // 2. Fetch the certificate PEM from the control plane
 // 3. Load local private key if it exists (agent keygen mode)
 // 4. Instantiate the target connector based on target_type from the work response
 // 5. Call DeployCertificate on the connector
 // 6. Report job as Completed (or Failed)
 func (a *Agent) executeDeploymentJob(ctx context.Context, job JobItem) {
 	a.logger.Info("executing deployment job",
 		"job_id", job.ID,
 		"certificate_id", job.CertificateID,
 		"target_type", job.TargetType)
 	// Report job as running
 	if err := a.reportJobStatus(ctx, job.ID, "Running", ""); err != nil {
 		a.logger.Error("failed to report job running", "error", err)
 	}
 	// Fetch the certificate from the control plane
 	certPEM, err := a.fetchCertificate(ctx, job.CertificateID)
 	if err != nil {
 		a.logger.Error("failed to fetch certificate",
 			"job_id", job.ID,
 			"error", err)
 		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("cert fetch failed: %v", err)); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 		}
 		return
 	}
 	a.logger.Info("certificate fetched for deployment",
 		"job_id", job.ID,
 		"cert_length", len(certPEM))
 	// Split PEM into cert and chain (separated by double newline between PEM blocks)
 	certOnly, chainPEM := splitPEMChain(certPEM)
 	// Check for locally-stored private key (agent keygen mode)
 	keyPath := filepath.Join(a.config.KeyDir, job.CertificateID+".key")
 	var keyPEM string
 	keyData, err := os.ReadFile(keyPath)
 	if err != nil {
 		a.logger.Error("failed to read local private key for deployment",
 			"job_id", job.ID,
 			"key_path", keyPath,
 			"error", err)
 		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("key read failed: %v", err)); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "error", reportErr)
 		}
 		return
 	}
 	keyPEM = string(keyData)
 	a.logger.Info("loaded local private key for deployment",
 		"job_id", job.ID,
 		"key_path", keyPath)
 	// Deploy to the target using the appropriate connector
 	if job.TargetType != "" {
 		connector, err := a.createTargetConnector(ctx, job.TargetType, job.TargetConfig)
 		if err != nil {
 			a.logger.Error("failed to create target connector",
 				"job_id", job.ID,
 				"target_type", job.TargetType,
 				"error", err)
 			if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("connector init failed: %v", err)); reportErr != nil {
 				a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 			}
 			return
 		}
 		deployReq := target.DeploymentRequest{
 			CertPEM:      certOnly,
 			KeyPEM:       keyPEM,
 			ChainPEM:     chainPEM,
 			TargetConfig: job.TargetConfig,
 			Metadata: map[string]string{
 				"certificate_id": job.CertificateID,
 				"job_id":         job.ID,
 			},
 		}
 		// Phase 2 of the deploy-hardening I master bundle:
 		// per-target deploy mutex. Acquire BEFORE
 		// DeployCertificate so two concurrent renewals against
 		// the same target ID serialize. The lock is held for the
 		// full Deploy duration including PreCommit (validate),
 		// PostCommit (reload), and post-deploy verify (Phases
 		// 4-9). Released on every return path via defer.
 		var targetID string
 		if job.TargetID != nil {
 			targetID = *job.TargetID
 		}
 		if mu := a.targetDeployMutex(targetID); mu != nil {
 			mu.Lock()
 			defer mu.Unlock()
 		}
 		result, err := connector.DeployCertificate(ctx, deployReq)
 		if err != nil {
 			a.logger.Error("deployment failed",
 				"job_id", job.ID,
 				"target_type", job.TargetType,
 				"error", err)
 			if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("deployment failed: %v", err)); reportErr != nil {
 				a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 			}
 			return
 		}
 		a.logger.Info("target connector deployment completed",
 			"job_id", job.ID,
 			"target_type", job.TargetType,
 			"success", result.Success,
 			"message", result.Message)
 		// If verification is enabled, verify the deployment by probing the live TLS endpoint
 		targetHost, targetPort, err := extractTargetHostAndPort(job.TargetConfig)
 		if err != nil {
 			a.logger.Warn("could not extract target host/port for verification",
 				"job_id", job.ID,
 				"error", err)
 		} else {
 			a.verifyAndReportDeployment(ctx, job, targetHost, targetPort, certOnly)
 		}
 	} else {
 		a.logger.Info("no target type specified, skipping connector invocation",
 			"job_id", job.ID)
 	}
 	// Report job as completed
 	if err := a.reportJobStatus(ctx, job.ID, "Completed", ""); err != nil {
 		a.logger.Error("failed to report job completed", "error", err)
 		return
 	}
 	a.logger.Info("deployment job completed", "job_id", job.ID)
 }
 // createTargetConnector instantiates the appropriate target connector based on type.
 // ctx is threaded into SDK-driven connectors (AWSACM, AzureKeyVault) so credential
 // resolution honors caller cancellation / deadlines instead of using a fresh
 // context.Background() (the contextcheck linter enforces this — the original Rank 5
 // implementation used Background() and tripped CI on commit 502823d).
 func (a *Agent) createTargetConnector(ctx context.Context, targetType string, configJSON json.RawMessage) (target.Connector, error) {
 	switch targetType {
 	case "NGINX":
 		var cfg nginx.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid NGINX config: %w", err)
 			}
 		}
 		return nginx.New(&cfg, a.logger), nil
 	case "Apache":
 		var cfg apache.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid Apache config: %w", err)
 			}
 		}
 		return apache.New(&cfg, a.logger), nil
 	case "HAProxy":
 		var cfg haproxy.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid HAProxy config: %w", err)
 			}
 		}
 		return haproxy.New(&cfg, a.logger), nil
 	case "F5":
 		var cfg f5.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid F5 config: %w", err)
 			}
 		}
 		conn, err := f5.New(&cfg, a.logger)
 		if err != nil {
 			return nil, fmt.Errorf("failed to create F5 connector: %w", err)
 		}
 		return conn, nil
 	case "IIS":
 		var cfg iis.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid IIS config: %w", err)
 			}
 		}
 		return iis.New(&cfg, a.logger)
 	case "Traefik":
 		var cfg traefik.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid Traefik config: %w", err)
 			}
 		}
 		return traefik.New(&cfg, a.logger), nil
 	case "Caddy":
 		var cfg caddy.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid Caddy config: %w", err)
 			}
 		}
 		return caddy.New(&cfg, a.logger), nil
 	case "Envoy":
 		var cfg envoy.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid Envoy config: %w", err)
 			}
 		}
 		return envoy.New(&cfg, a.logger), nil
 	case "Postfix":
 		var cfg pf.Config
 		cfg.Mode = "postfix"
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid Postfix config: %w", err)
 			}
 		}
 		return pf.New(&cfg, a.logger), nil
 	case "Dovecot":
 		var cfg pf.Config
 		cfg.Mode = "dovecot"
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid Dovecot config: %w", err)
 			}
 		}
 		return pf.New(&cfg, a.logger), nil
 	case "SSH":
 		var cfg sshconn.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid SSH config: %w", err)
 			}
 		}
 		return sshconn.New(&cfg, a.logger)
 	case "WinCertStore":
 		var cfg wcs.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid WinCertStore config: %w", err)
 			}
 		}
 		return wcs.New(&cfg, a.logger)
 	case "JavaKeystore":
 		var cfg jks.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid JavaKeystore config: %w", err)
 			}
 		}
 		return jks.New(&cfg, a.logger), nil
 	case "KubernetesSecrets":
 		var cfg k8s.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid KubernetesSecrets config: %w", err)
 			}
 		}
 		return k8s.New(&cfg, a.logger)
 	case "AWSACM":
 		// Rank 5 of the 2026-05-03 Infisical deep-research deliverable.
 		// AWS Certificate Manager target — SDK-driven (no file I/O).
 		// LoadDefaultConfig handles the standard AWS credential chain
 		// (IRSA / EC2 instance profile / SSO / env vars) without any
 		// long-lived creds in connector Config.
 		var cfg awsacm.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid AWSACM config: %w", err)
 			}
 		}
 		return awsacm.New(ctx, &cfg, a.logger)
 	case "AzureKeyVault":
 		// Rank 5 of the 2026-05-03 Infisical deep-research deliverable.
 		// Azure Key Vault target — SDK-driven (no file I/O).
 		// DefaultAzureCredential handles the standard Azure credential
 		// chain (managed identity / workload identity / env vars / az
 		// CLI fallback). Long-lived service-principal secrets are
 		// supported but discouraged via the credential_mode config.
 		var cfg azurekv.Config
 		if len(configJSON) > 0 {
 			if err := json.Unmarshal(configJSON, &cfg); err != nil {
 				return nil, fmt.Errorf("invalid AzureKeyVault config: %w", err)
 			}
 		}
 		return azurekv.New(ctx, &cfg, a.logger)
 	default:
 		return nil, fmt.Errorf("unsupported target type: %s", targetType)
 	}
 }
 // splitPEMChain splits a PEM chain into the first certificate (cert) and the rest (chain).
 // The control plane returns the full chain as a single string with PEM blocks concatenated.
 func splitPEMChain(pemChain string) (string, string) {
 	data := []byte(pemChain)
 	block, rest := pem.Decode(data)
 	if block == nil {
 		return pemChain, ""
 	}
 	cert := string(pem.EncodeToMemory(block))
 	// Skip whitespace between cert and chain
 	chain := strings.TrimSpace(string(rest))
 	if chain == "" {
 		return cert, ""
 	}
 	return cert, chain
 }
 // fetchCertificate retrieves the certificate PEM chain from the control plane.
 // GET /api/v1/agents/{agentID}/certificates/{certID}
 func (a *Agent) fetchCertificate(ctx context.Context, certID string) (string, error) {
 	path := fmt.Sprintf("/api/v1/agents/%s/certificates/%s", a.config.AgentID, certID)
 	resp, err := a.makeRequest(ctx, http.MethodGet, path, nil)
 	if err != nil {
 		return "", fmt.Errorf("request failed: %w", err)
 	}
 	defer resp.Body.Close()
 	if resp.StatusCode != http.StatusOK {
 		body, _ := io.ReadAll(resp.Body)
 		return "", fmt.Errorf("server returned %d: %s", resp.StatusCode, string(body))
 	}
 	var certResp struct {
 		CertificatePEM string `json:"certificate_pem"`
 	}
 	if err := json.NewDecoder(resp.Body).Decode(&certResp); err != nil {
 		return "", fmt.Errorf("failed to decode response: %w", err)
 	}
 	return certResp.CertificatePEM, nil
 }
 // reportJobStatus reports the result of a job back to the control plane.
 // POST /api/v1/agents/{agentID}/jobs/{jobID}/status
 func (a *Agent) reportJobStatus(ctx context.Context, jobID string, status string, errorMsg string) error {
@@ -1044,239 +467,6 @@ func (a *Agent) makeRequest(ctx context.Context, method, path string, body inter
 	return resp, nil
 }
 // runDiscoveryScan walks configured directories, parses certificate files, and reports
 // discovered certificates to the control plane.
 // Supports PEM and DER encoded X.509 certificates.
 func (a *Agent) runDiscoveryScan(ctx context.Context) {
 	a.logger.Info("starting filesystem certificate discovery scan",
 		"directories", a.config.DiscoveryDirs)
 	startTime := time.Now()
 	var certs []discoveredCertEntry
 	var scanErrors []string
 	for _, dir := range a.config.DiscoveryDirs {
 		a.logger.Debug("scanning directory", "path", dir)
 		err := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
 			if err != nil {
 				scanErrors = append(scanErrors, fmt.Sprintf("walk error at %s: %v", path, err))
 				return nil // continue walking
 			}
 			if info.IsDir() {
 				return nil
 			}
 			// Skip files larger than 1MB (unlikely to be a certificate)
 			if info.Size() > 1*1024*1024 {
 				return nil
 			}
 			// Check file extension
 			ext := strings.ToLower(filepath.Ext(path))
 			switch ext {
 			case ".pem", ".crt", ".cer", ".cert":
 				found := a.parsePEMFile(path)
 				certs = append(certs, found...)
 			case ".der":
 				if entry, err := a.parseDERFile(path); err == nil {
 					certs = append(certs, entry)
 				} else {
 					a.logger.Debug("skipping non-cert DER file", "path", path, "error", err)
 				}
 			default:
 				// Try PEM parsing for extensionless files or unknown extensions
 				if ext == "" || ext == ".key" {
 					return nil // skip key files and extensionless
 				}
 				found := a.parsePEMFile(path)
 				if len(found) > 0 {
 					certs = append(certs, found...)
 				}
 			}
 			return nil
 		})
 		if err != nil {
 			scanErrors = append(scanErrors, fmt.Sprintf("failed to walk %s: %v", dir, err))
 		}
 	}
 	scanDuration := time.Since(startTime)
 	a.logger.Info("discovery scan completed",
 		"certificates_found", len(certs),
 		"errors", len(scanErrors),
 		"duration_ms", scanDuration.Milliseconds())
 	if len(certs) == 0 && len(scanErrors) == 0 {
 		a.logger.Debug("no certificates found and no errors, skipping report")
 		return
 	}
 	// Build report payload
 	entries := make([]map[string]interface{}, len(certs))
 	for i, c := range certs {
 		entries[i] = map[string]interface{}{
 			"fingerprint_sha256": c.FingerprintSHA256,
 			"common_name":        c.CommonName,
 			"sans":               c.SANs,
 			"serial_number":      c.SerialNumber,
 			"issuer_dn":          c.IssuerDN,
 			"subject_dn":         c.SubjectDN,
 			"not_before":         c.NotBefore,
 			"not_after":          c.NotAfter,
 			"key_algorithm":      c.KeyAlgorithm,
 			"key_size":           c.KeySize,
 			"is_ca":              c.IsCA,
 			"pem_data":           c.PEMData,
 			"source_path":        c.SourcePath,
 			"source_format":      c.SourceFormat,
 		}
 	}
 	report := map[string]interface{}{
 		"agent_id":         a.config.AgentID,
 		"directories":      a.config.DiscoveryDirs,
 		"certificates":     entries,
 		"errors":           scanErrors,
 		"scan_duration_ms": int(scanDuration.Milliseconds()),
 	}
 	// Submit to control plane
 	path := fmt.Sprintf("/api/v1/agents/%s/discoveries", a.config.AgentID)
 	resp, err := a.makeRequest(ctx, http.MethodPost, path, report)
 	if err != nil {
 		a.logger.Error("failed to submit discovery report", "error", err)
 		return
 	}
 	defer resp.Body.Close()
 	if resp.StatusCode != http.StatusAccepted {
 		body, _ := io.ReadAll(resp.Body)
 		a.logger.Error("discovery report rejected",
 			"status", resp.StatusCode,
 			"body", string(body))
 		return
 	}
 	a.logger.Info("discovery report submitted successfully",
 		"certificates", len(certs),
 		"errors", len(scanErrors))
 }
 // discoveredCertEntry holds parsed certificate metadata for reporting.
 type discoveredCertEntry struct {
 	FingerprintSHA256 string   `json:"fingerprint_sha256"`
 	CommonName        string   `json:"common_name"`
 	SANs              []string `json:"sans"`
 	SerialNumber      string   `json:"serial_number"`
 	IssuerDN          string   `json:"issuer_dn"`
 	SubjectDN         string   `json:"subject_dn"`
 	NotBefore         string   `json:"not_before"`
 	NotAfter          string   `json:"not_after"`
 	KeyAlgorithm      string   `json:"key_algorithm"`
 	KeySize           int      `json:"key_size"`
 	IsCA              bool     `json:"is_ca"`
 	PEMData           string   `json:"pem_data"`
 	SourcePath        string   `json:"source_path"`
 	SourceFormat      string   `json:"source_format"`
 }
 // parsePEMFile reads a file and extracts all X.509 certificates from PEM blocks.
 func (a *Agent) parsePEMFile(path string) []discoveredCertEntry {
 	data, err := os.ReadFile(path)
 	if err != nil {
 		a.logger.Debug("failed to read file", "path", path, "error", err)
 		return nil
 	}
 	var entries []discoveredCertEntry
 	rest := data
 	for {
 		var block *pem.Block
 		block, rest = pem.Decode(rest)
 		if block == nil {
 			break
 		}
 		if block.Type != "CERTIFICATE" {
 			continue
 		}
 		cert, err := x509.ParseCertificate(block.Bytes)
 		if err != nil {
 			a.logger.Debug("failed to parse certificate in PEM", "path", path, "error", err)
 			continue
 		}
 		pemStr := string(pem.EncodeToMemory(block))
 		entries = append(entries, certToEntry(cert, path, "PEM", pemStr))
 	}
 	return entries
 }
 // parseDERFile reads a DER-encoded certificate file.
 func (a *Agent) parseDERFile(path string) (discoveredCertEntry, error) {
 	data, err := os.ReadFile(path)
 	if err != nil {
 		return discoveredCertEntry{}, fmt.Errorf("read failed: %w", err)
 	}
 	cert, err := x509.ParseCertificate(data)
 	if err != nil {
 		return discoveredCertEntry{}, fmt.Errorf("parse failed: %w", err)
 	}
 	// Convert to PEM for storage
 	pemStr := string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: data}))
 	return certToEntry(cert, path, "DER", pemStr), nil
 }
 // certToEntry converts a parsed x509.Certificate into a discoveredCertEntry.
 func certToEntry(cert *x509.Certificate, path, format, pemData string) discoveredCertEntry {
 	// Compute SHA-256 fingerprint
 	fingerprint := fmt.Sprintf("%x", sha256Sum(cert.Raw))
 	// Determine key algorithm and size
 	keyAlg, keySize := certKeyInfo(cert)
 	return discoveredCertEntry{
 		FingerprintSHA256: fingerprint,
 		CommonName:        cert.Subject.CommonName,
 		SANs:              cert.DNSNames,
 		SerialNumber:      cert.SerialNumber.Text(16),
 		IssuerDN:          cert.Issuer.String(),
 		SubjectDN:         cert.Subject.String(),
 		NotBefore:         cert.NotBefore.UTC().Format(time.RFC3339),
 		NotAfter:          cert.NotAfter.UTC().Format(time.RFC3339),
 		KeyAlgorithm:      keyAlg,
 		KeySize:           keySize,
 		IsCA:              cert.IsCA,
 		PEMData:           pemData,
 		SourcePath:        path,
 		SourceFormat:      format,
 	}
 }
 // sha256Sum returns the SHA-256 hash of data.
 func sha256Sum(data []byte) [32]byte {
 	return sha256.Sum256(data)
 }
 // certKeyInfo extracts key algorithm name and size from a certificate.
 func certKeyInfo(cert *x509.Certificate) (string, int) {
 	switch pub := cert.PublicKey.(type) {
 	case *ecdsa.PublicKey:
 		return "ECDSA", pub.Curve.Params().BitSize
 	case *rsa.PublicKey:
 		return "RSA", pub.N.BitLen()
 	default:
 		switch cert.PublicKeyAlgorithm {
 		case x509.Ed25519:
 			return "Ed25519", 256
 		default:
 			return cert.PublicKeyAlgorithm.String(), 0
 		}
 	}
 }
 func main() {
 	// Parse command-line flags (with env var fallbacks for Docker deployment)
 	serverURL := flag.String("server", getEnvDefault("CERTCTL_SERVER_URL", "https://localhost:8443"), "Control plane server URL (must be https://)")
@@ -0,0 +1,291 @@
 // Copyright 2026 certctl LLC. All rights reserved.
 // SPDX-License-Identifier: BUSL-1.1
 package main
 import (
 	"context"
 	"crypto/ecdsa"
 	"crypto/elliptic"
 	"crypto/rand"
 	"crypto/x509"
 	"crypto/x509/pkix"
 	"encoding/json"
 	"encoding/pem"
 	"fmt"
 	"io"
 	"net/http"
 	"os"
 	"path/filepath"
 	"strings"
 )
 // Phase 9 ARCH-M2 closure Sprint 12 (2026-05-14): extracted from
 // cmd/agent/main.go via the Option B sibling-file pattern (mirrors
 // the Sprint 8 cmd/server cut). Package stays `main`; all methods
 // are still defined on *Agent so every call site continues to
 // resolve through Go's same-package method-set without any
 // import-path change.
 //
 // This file holds the WORK-POLLING entry point + CSR-job execution
 // — the inbound side of the agent's pull-only deployment model
 // (per CLAUDE.md "Pull-only deployment model" architecture
 // decision):
 //
 //   - pollForWork: queries GET /api/v1/agents/{id}/work each tick;
 //     dispatches each returned JobItem to the appropriate
 //     executor (CSR vs deployment).
 //   - executeCSRJob: handles AwaitingCSR jobs by generating an
 //     ECDSA P-256 key locally, persisting it to keyDir/<certID>.key
 //     with 0600 permissions (key NEVER leaves the agent — see
 //     CLAUDE.md "Agent-based key management"), creating the CSR,
 //     and POSTing it to the control plane for signing.
 //
 // The deployment-job executor lives in deploy.go alongside the
 // target connector factory + deploy-only helpers (splitPEMChain,
 // fetchCertificate). The discovery scan lives in discovery.go.
 // pollForWork queries the control plane for actionable jobs and processes them.
 // Jobs may be deployment jobs (Pending) or CSR jobs (AwaitingCSR).
 // GET /api/v1/agents/{agentID}/work
 func (a *Agent) pollForWork(ctx context.Context) {
 	a.logger.Debug("polling for work", "agent_id", a.config.AgentID)
 	path := fmt.Sprintf("/api/v1/agents/%s/work", a.config.AgentID)
 	resp, err := a.makeRequest(ctx, http.MethodGet, path, nil)
 	if err != nil {
 		a.logger.Error("work poll failed", "error", err)
 		a.consecutiveFailures++
 		return
 	}
 	defer resp.Body.Close()
 	// I-004: same terminal-retirement handling as sendHeartbeat. Work-poll is the
 	// other hot path that can observe an agent's soft-retirement; if the
 	// heartbeat tick happens to fire after a work-poll tick within the same
 	// retirement window, this branch catches it first. markRetired's sync.Once
 	// guards idempotency so racing both paths in the same tick only closes the
 	// signal channel once. No consecutiveFailures increment — retirement is
 	// not a transient failure.
 	if resp.StatusCode == http.StatusGone {
 		body, _ := io.ReadAll(resp.Body)
 		a.markRetired("work_poll", resp.StatusCode, string(body))
 		return
 	}
 	if resp.StatusCode != http.StatusOK {
 		body, _ := io.ReadAll(resp.Body)
 		a.logger.Error("work poll rejected",
 			"status", resp.StatusCode,
 			"body", string(body))
 		a.consecutiveFailures++
 		return
 	}
 	var workResp WorkResponse
 	if err := json.NewDecoder(resp.Body).Decode(&workResp); err != nil {
 		a.logger.Error("failed to decode work response", "error", err)
 		a.consecutiveFailures++
 		return
 	}
 	a.consecutiveFailures = 0
 	if workResp.Count == 0 {
 		a.logger.Debug("no pending work")
 		return
 	}
 	a.logger.Info("received work", "job_count", workResp.Count)
 	// Process each job based on type and status
 	for _, job := range workResp.Jobs {
 		switch {
 		case job.Status == "AwaitingCSR":
 			// Agent keygen mode: generate key locally, create CSR, submit to server
 			a.executeCSRJob(ctx, job)
 		case job.Type == "Deployment":
 			a.executeDeploymentJob(ctx, job)
 		}
 	}
 }
 // executeCSRJob handles an AwaitingCSR job: generates a private key locally, creates a CSR,
 // and submits it to the control plane for signing. The private key is stored on the local
 // filesystem with 0600 permissions and NEVER sent to the server.
 //
 // Flow:
 // 1. Generate ECDSA P-256 key pair
 // 2. Store private key to disk (keyDir/certID.key) with 0600 permissions
 // 3. Create CSR with common name and SANs from work response
 // 4. Submit CSR to control plane via POST /agents/{id}/csr
 // 5. Server signs the CSR and creates a cert version + deployment jobs
 func (a *Agent) executeCSRJob(ctx context.Context, job JobItem) {
 	a.logger.Info("executing CSR job (agent-side key generation)",
 		"job_id", job.ID,
 		"certificate_id", job.CertificateID,
 		"common_name", job.CommonName)
 	// Step 1: Generate ECDSA P-256 key pair
 	privKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
 	if err != nil {
 		a.logger.Error("failed to generate private key",
 			"job_id", job.ID,
 			"error", err)
 		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("key generation failed: %v", err)); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 		}
 		return
 	}
 	a.logger.Info("generated ECDSA P-256 key pair locally",
 		"job_id", job.ID,
 		"certificate_id", job.CertificateID)
 	// Step 2: Store private key to disk with secure permissions.
 	//
 	// Bundle-9 / Audit L-002 + L-003: marshal+write through helpers that
 	// (a) zeroize the in-heap DER buffer immediately after the PEM block is
 	// constructed so the private scalar's exposure window is bounded by
 	// this function call, and (b) assert the key directory is mode 0700
 	// before any write touches disk. Also defer-clear the PEM buffer for
 	// the same reason — the encoded key isn't sensitive in transit (it's
 	// going to disk) but lingers on the heap if we don't.
 	//
 	// SEC-002 closure (Sprint 1, 2026-05-16): safeAgentKeyPath validates
 	// the certificate_id shape AND asserts the joined path is contained
 	// within a.config.KeyDir. A crafted certificate_id like
 	// "../../etc/passwd" or "/abs/path" now fails closed before any
 	// disk I/O. See cmd/agent/keymem.go for the helper.
 	keyPath, kerr := safeAgentKeyPath(a.config.KeyDir, job.CertificateID)
 	if kerr != nil {
 		a.logger.Error("agent key path validation failed", "job_id", job.ID, "certificate_id", job.CertificateID, "error", kerr)
 		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("key path validation failed: %v", kerr)); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 		}
 		return
 	}
 	if err := ensureAgentKeyDirSecure(filepath.Dir(keyPath)); err != nil {
 		a.logger.Error("agent key dir hardening failed", "job_id", job.ID, "error", err)
 		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("key dir hardening failed: %v", err)); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 		}
 		return
 	}
 	var privKeyPEM []byte
 	if marshalErr := marshalAgentKeyAndZeroize(privKey, func(der []byte) error {
 		privKeyPEM = pem.EncodeToMemory(&pem.Block{
 			Type:  "EC PRIVATE KEY",
 			Bytes: der,
 		})
 		return nil
 	}); marshalErr != nil {
 		a.logger.Error("failed to marshal private key",
 			"job_id", job.ID,
 			"error", marshalErr)
 		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("key marshal failed: %v", marshalErr)); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 		}
 		return
 	}
 	defer clear(privKeyPEM)
 	if err := os.WriteFile(keyPath, privKeyPEM, 0600); err != nil {
 		a.logger.Error("failed to write private key to disk",
 			"job_id", job.ID,
 			"key_path", keyPath,
 			"error", err)
 		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("key storage failed: %v", err)); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 		}
 		return
 	}
 	a.logger.Info("private key stored securely",
 		"job_id", job.ID,
 		"key_path", keyPath,
 		"permissions", "0600")
 	// Validate common name is present
 	if job.CommonName == "" {
 		a.logger.Error("empty common name in CSR job", "job_id", job.ID)
 		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", "empty common name"); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "error", reportErr)
 		}
 		return
 	}
 	// Step 3: Create CSR with common name and SANs
 	// Split SANs into DNS names and email addresses for proper CSR encoding
 	var dnsNames []string
 	var emailAddresses []string
 	for _, san := range job.SANs {
 		if strings.Contains(san, "@") {
 			emailAddresses = append(emailAddresses, san)
 		} else {
 			dnsNames = append(dnsNames, san)
 		}
 	}
 	csrTemplate := &x509.CertificateRequest{
 		Subject: pkix.Name{
 			CommonName: job.CommonName,
 		},
 		DNSNames:       dnsNames,
 		EmailAddresses: emailAddresses,
 	}
 	csrDER, err := x509.CreateCertificateRequest(rand.Reader, csrTemplate, privKey)
 	if err != nil {
 		a.logger.Error("failed to create CSR",
 			"job_id", job.ID,
 			"error", err)
 		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("CSR creation failed: %v", err)); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 		}
 		return
 	}
 	csrPEM := string(pem.EncodeToMemory(&pem.Block{
 		Type:  "CERTIFICATE REQUEST",
 		Bytes: csrDER,
 	}))
 	// Step 4: Submit CSR to the control plane (only the public key leaves the agent)
 	a.logger.Info("submitting CSR to control plane",
 		"job_id", job.ID,
 		"certificate_id", job.CertificateID)
 	submitPath := fmt.Sprintf("/api/v1/agents/%s/csr", a.config.AgentID)
 	resp, err := a.makeRequest(ctx, http.MethodPost, submitPath, map[string]string{
 		"csr_pem":        csrPEM,
 		"certificate_id": job.CertificateID,
 	})
 	if err != nil {
 		a.logger.Error("failed to submit CSR",
 			"job_id", job.ID,
 			"error", err)
 		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("CSR submission failed: %v", err)); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 		}
 		return
 	}
 	defer resp.Body.Close()
 	if resp.StatusCode != http.StatusAccepted {
 		body, _ := io.ReadAll(resp.Body)
 		a.logger.Error("CSR submission rejected",
 			"job_id", job.ID,
 			"status", resp.StatusCode,
 			"body", string(body))
 		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("CSR rejected: %s", string(body))); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 		}
 		return
 	}
 	a.logger.Info("CSR submitted and signed successfully",
 		"job_id", job.ID,
 		"certificate_id", job.CertificateID,
 		"key_path", keyPath)
 }
@@ -1,3 +1,6 @@
 // Copyright 2026 certctl LLC. All rights reserved.
 // SPDX-License-Identifier: BUSL-1.1
 package main
 import (
@@ -1,3 +1,6 @@
 // Copyright 2026 certctl LLC. All rights reserved.
 // SPDX-License-Identifier: BUSL-1.1
 package main
 import (
@@ -1,3 +1,6 @@
 // Copyright 2026 certctl LLC. All rights reserved.
 // SPDX-License-Identifier: BUSL-1.1
 package main
 import (
@@ -1,3 +1,6 @@
 // Copyright 2026 certctl LLC. All rights reserved.
 // SPDX-License-Identifier: BUSL-1.1
 package main
 import (
@@ -256,6 +256,18 @@ func TestMain_ServerConfigFromEnvironment(t *testing.T) {
 	os.Setenv("CERTCTL_SERVER_PORT", "8080")
 	os.Setenv("CERTCTL_SERVER_TLS_CERT_PATH", certPath)
 	os.Setenv("CERTCTL_SERVER_TLS_KEY_PATH", keyPath)
 	// Acquisition-audit RED-003 closure (Sprint 5 ACQ, 2026-05-16):
 	// deny-empty default flipped to true; supply a placeholder token
 	// so Load() succeeds. The defer below restores prior env.
 	oldBootstrap := os.Getenv("CERTCTL_AGENT_BOOTSTRAP_TOKEN")
 	os.Setenv("CERTCTL_AGENT_BOOTSTRAP_TOKEN", "test-bootstrap-token-placeholder")
 	defer func() {
 		if oldBootstrap != "" {
 			os.Setenv("CERTCTL_AGENT_BOOTSTRAP_TOKEN", oldBootstrap)
 		} else {
 			os.Unsetenv("CERTCTL_AGENT_BOOTSTRAP_TOKEN")
 		}
 	}()
 	cfg, err := config.Load()
 	if err != nil {
@@ -317,6 +329,18 @@ func TestMain_AuthTypeConfiguration(t *testing.T) {
 	// Set auth secret for api-key mode
 	os.Setenv("CERTCTL_AUTH_SECRET", "test-secret")
 	// Acquisition-audit RED-003 closure (Sprint 5 ACQ, 2026-05-16):
 	// deny-empty default flipped to true; supply a placeholder token
 	// so Load() succeeds.
 	oldBootstrap := os.Getenv("CERTCTL_AGENT_BOOTSTRAP_TOKEN")
 	os.Setenv("CERTCTL_AGENT_BOOTSTRAP_TOKEN", "test-bootstrap-token-placeholder")
 	defer func() {
 		if oldBootstrap != "" {
 			os.Setenv("CERTCTL_AGENT_BOOTSTRAP_TOKEN", oldBootstrap)
 		} else {
 			os.Unsetenv("CERTCTL_AGENT_BOOTSTRAP_TOKEN")
 		}
 	}()
 	testCases := []string{"api-key", "none"}
@@ -645,3 +669,64 @@ func TestPreflightSCEPChallengePassword(t *testing.T) {
 		})
 	}
 }
 // =============================================================================
 // SEC-003 closure (Sprint 1, 2026-05-16). Pin that the rate-limit-enabled
 // middleware stack still emits the five security headers (HSTS, XFO,
 // nosniff, Referrer-Policy, CSP) that the default stack carries.
 //
 // Pre-fix the stack rebuild at main.go ~L2079 dropped
 // securityHeadersMiddleware so flipping CERTCTL_RATE_LIMIT_ENABLED=true
 // silently turned off five browser-side defenses. This test exercises
 // the same middleware composition main.go now builds when the flag is
 // on, and asserts each header lands on the wire. A future regression
 // that removes securityHeadersMiddleware (or reorders it after the
 // rate limiter such that a 429 response misses the headers) would
 // surface here.
 // =============================================================================
 func TestMain_RateLimitedStack_EmitsSecurityHeaders(t *testing.T) {
 	baseHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		w.WriteHeader(http.StatusOK)
 	})
 	// Mirror the rate-limit-enabled middlewareStack from main.go.
 	rateLimiter := middleware.NewRateLimiter(middleware.RateLimitConfig{
 		RPS:       1000, // high enough that the single test request isn't dropped
 		BurstSize: 1000,
 	})
 	securityHeaders := middleware.SecurityHeaders(middleware.SecurityHeadersDefaults())
 	bodyLimit := middleware.NewBodyLimit(middleware.BodyLimitConfig{MaxBytes: 1 << 20})
 	stack := []func(http.Handler) http.Handler{
 		middleware.RequestID,
 		middleware.Recovery,
 		bodyLimit,
 		securityHeaders,
 		rateLimiter,
 		// Skip the CORS/auth/csrf/audit layers — they aren't relevant
 		// to the headers-on-response invariant we're pinning.
 	}
 	chained := middleware.Chain(baseHandler, stack...)
 	req := httptest.NewRequest(http.MethodGet, "/api/v1/test", nil)
 	w := httptest.NewRecorder()
 	chained.ServeHTTP(w, req)
 	if w.Code != http.StatusOK {
 		t.Fatalf("status = %d; want 200 (rate limit should not trip on a single request)", w.Code)
 	}
 	wantHeaders := map[string]string{
 		"Strict-Transport-Security": "max-age=31536000; includeSubDomains",
 		"X-Frame-Options":           "DENY",
 		"X-Content-Type-Options":    "nosniff",
 		"Referrer-Policy":           "no-referrer-when-downgrade",
 		"Content-Security-Policy":   "default-src 'self'; img-src 'self' data:; style-src 'self' 'unsafe-inline'; script-src 'self'; connect-src 'self'; frame-ancestors 'none'",
 	}
 	for name, want := range wantHeaders {
 		got := w.Header().Get(name)
 		if got != want {
 			t.Errorf("rate-limited stack: %s = %q; want %q", name, got, want)
 		}
 	}
 }
@@ -0,0 +1,209 @@
 // Copyright 2026 certctl LLC. All rights reserved.
 // SPDX-License-Identifier: BUSL-1.1
 package main
 import (
 	"database/sql"
 	"log/slog"
 	"os"
 	"strings"
 	"github.com/certctl-io/certctl/internal/config"
 	"github.com/certctl-io/certctl/internal/repository/postgres"
 )
 // Phase 9 ARCH-M2 closure Sprint 8b (2026-05-14): the deferred half of
 // Sprint 8. Extracts the boot-time migration handling from main()'s
 // inline body into two unexported helpers. Different shape from
 // Sprints 1-7 (data-type relocation) and from Sprint 8a (existing
 // helper-function relocation) — this sprint crosses the
 // behavior-change boundary Sprint 8 first identified.
 //
 // What lives here
 // ===============
 //   parseMigrateOnlyFlag() bool
 //     Hand-parses os.Args for `--migrate-only` (NOT flag.Parse — the
 //     server's config surface is otherwise env-var driven via
 //     config.Load; introducing flag.Parse's global state risks
 //     conflicting with other binaries that may import cmd/server later).
 //
 //   runBootMigrations(cfg, db, logger, migrateOnly) (exitNow bool)
 //     Owns the Phase 4 DEPL-M1 migration-via-hook posture: the
 //     migrationsViaHook env-var read, the RunMigrations + RunSeed
 //     gate, the --migrate-only early-exit signal, and the
 //     CERTCTL_DEMO_SEED demo-overlay branch.
 //
 //     Returns true ONLY when --migrate-only was set and migrations +
 //     seed completed cleanly. The caller (main) translates that to
 //     `return` rather than os.Exit(0) — which is the SOLE intentional
 //     behavior change in this sprint (see below).
 //
 // Behavior preservation contract
 // ==============================
 // Every error path inside runBootMigrations calls os.Exit(1)
 // directly, matching the original inline behavior byte-for-byte
 // (same log message, same exit code, same no-defer-run-on-fatal
 // semantics). The error-path os.Exit(1) is intentional: when
 // migration fails at boot, the server cannot recover, and bailing
 // out without running defers is the original Go-idiomatic shape.
 //
 // The ONE behavior change: the --migrate-only SUCCESS path now
 // returns to main() rather than calling os.Exit(0) inline. This
 // has one observable effect: the `defer db.Close()` registered in
 // main() now runs at clean exit instead of being skipped. That's
 // strictly better hygiene (clean DB connection shutdown vs OS
 // reclaim). The migration work is synchronous + complete before
 // the return; nothing async is left running that db.Close() could
 // truncate.
 //
 // All other paths — the migration log messages, the seed log
 // messages, the migrationsViaHook env-var read order, the
 // RunDemoSeed gating, the per-step success/skip log lines — are
 // byte-identical to the pre-Sprint-8b inline form. Verified via
 // `go test ./cmd/server/... -count=1 -short` (which runs the
 // existing main_test.go assertions through the new call site).
 //
 // Why this is a separate commit
 // =============================
 // Sprint 8a (commit see git log) extracted the bottom-of-file
 // helpers + adapter types — pure mechanical relocation that
 // couldn't change runtime semantics. Sprint 8b crosses the boundary
 // where mechanical relocation ends: introducing a new function
 // call frame changes defer scope, panic recovery, and (in this
 // case) the exit semantics for the --migrate-only path. The
 // Phase 9 prompt's "refactor is mechanical relocation; behavior
 // change is a separate concern" rule guards against exactly this
 // shape of risk being landed without a focused review.
 //
 // Splitting Sprint 8a (mechanical) from Sprint 8b (behavior-aware)
 // means the operator's git log shows:
 //   3f1344e8 ... wire.go         — no behavior change possible
 //   <this>   ... migrations.go    — one specific behavior shift,
 //                                   documented + intentional
 //
 // Anyone bisecting a future bug to one of these two commits gets a
 // clean "is it mechanical or did the behavior change" signal.
 // parseMigrateOnlyFlag scans os.Args for the `--migrate-only` token
 // and returns true if found. Hand-parsed instead of using flag.Parse
 // because:
 //
 //  1. The server's entire config surface is env-var driven via
 //     config.Load(). flag.Parse() introduces a global package-state
 //     dependency that future binaries importing cmd/server (test
 //     harnesses, CLI tools, embedded variants) would have to
 //     coordinate around.
 //  2. The only flag we care about is the migration-vs-server-lifecycle
 //     toggle; a hand-parser is 6 lines and has no transitive cost.
 //  3. The flag is Helm-pre-install-hook-facing (see
 //     deploy/helm/certctl/templates/migration-job.yaml). Its shape is
 //     pinned by that template, not by anything else; we don't need
 //     flag.Parse's auto-help generation or type coercion.
 //
 // Bare arg match — no `=` value form, no short alias, no override
 // from env. Anyone passing `--migrate-only` ANYWHERE in os.Args[1:]
 // flips the flag on. Matches the original inline behavior exactly.
 func parseMigrateOnlyFlag() bool {
 	for _, arg := range os.Args[1:] {
 		if arg == "--migrate-only" {
 			return true
 		}
 	}
 	return false
 }
 // runBootMigrations owns the Phase 4 DEPL-M1 boot-time migration
 // posture. Three lifecycles to support:
 //
 //	(a) Compose / VM / bare-metal: server runs migrations at boot.
 //	    Default behavior — preserved unchanged.
 //	(b) Helm with pre-install/pre-upgrade hook: the migration Job
 //	    runs `certctl-server --migrate-only`, does its work, and
 //	    exits. The server Deployment's pods then start with
 //	    CERTCTL_MIGRATIONS_VIA_HOOK=true set; they see the env
 //	    var and skip their boot-time RunMigrations call so the
 //	    Job's work isn't duplicated.
 //	(c) Bare `certctl-server --migrate-only` invocation (e.g.
 //	    operator running a one-shot migration from the CLI):
 //	    runs migrations + seed and returns true so main returns
 //	    cleanly without starting the HTTP listener / scheduler /
 //	    signing setup.
 //
 // migrateOnly captures case (c); CERTCTL_MIGRATIONS_VIA_HOOK
 // captures case (b). Both paths converge on the same RunMigrations
 // + RunSeed code below.
 //
 // Returns true ONLY when migrateOnly is set; caller (main) handles
 // the clean exit via `return` so deferred cleanup (db.Close) runs.
 // Returns false in every other case — caller continues normal boot.
 // On any migration / seed error: os.Exit(1) inline (matches the
 // pre-extraction shape; recovery is not possible at this boot
 // stage).
 func runBootMigrations(cfg *config.Config, db *sql.DB, logger *slog.Logger, migrateOnly bool) bool {
 	migrationsViaHook := strings.EqualFold(os.Getenv("CERTCTL_MIGRATIONS_VIA_HOOK"), "true")
 	if migrateOnly || !migrationsViaHook {
 		logger.Info("running migrations", "path", cfg.Database.MigrationsPath)
 		if err := postgres.RunMigrations(db, cfg.Database.MigrationsPath); err != nil {
 			logger.Error("failed to run migrations", "error", err)
 			os.Exit(1)
 		}
 		logger.Info("migrations completed")
 	} else {
 		logger.Info("skipping migrations at boot (CERTCTL_MIGRATIONS_VIA_HOOK=true — Helm pre-install/pre-upgrade hook owns this work)")
 	}
 	// Apply baseline seed data.
 	//
 	// U-3 (P1, cat-u-seed_initdb_schema_drift): pre-U-3 seed.sql was mounted
 	// into postgres `/docker-entrypoint-initdb.d/` alongside a hand-curated
 	// subset of migrations. Adding a migration that introduced a new column
 	// referenced by seed.sql (cat-o-retry_interval_unit_mismatch /
 	// policy_rules.severity / etc.) without also updating the compose volume
 	// mounts caused initdb to crash on first up. Post-U-3 the compose stack
 	// drops all initdb mounts; postgres comes up with empty schema, the
 	// server runs RunMigrations above, then this RunSeed call lands the
 	// baseline data — all from a single source of truth (this binary).
 	// See internal/repository/postgres/db.go::RunSeed for the contract.
 	//
 	// Phase 4 DEPL-M1: same migration-via-hook gating as RunMigrations.
 	// When the hook owns migrations it also owns the seed pass.
 	if migrateOnly || !migrationsViaHook {
 		logger.Info("applying baseline seed", "path", cfg.Database.MigrationsPath)
 		if err := postgres.RunSeed(db, cfg.Database.MigrationsPath); err != nil {
 			logger.Error("failed to apply seed data", "error", err)
 			os.Exit(1)
 		}
 		logger.Info("seed completed")
 	} else {
 		logger.Info("skipping baseline seed at boot (CERTCTL_MIGRATIONS_VIA_HOOK=true — hook applies seed alongside migrations)")
 	}
 	// Phase 4 DEPL-M1: --migrate-only early-exit. Migrations + seed are
 	// done; the operator only asked for the migration pass. Signal main
 	// to return cleanly so deferred db.Close runs (Sprint 8b improvement
 	// over the pre-extraction os.Exit(0) which skipped defers).
 	if migrateOnly {
 		logger.Info("--migrate-only: migrations + seed complete; exiting without starting server lifecycle")
 		return true
 	}
 	// Apply demo overlay seed when CERTCTL_DEMO_SEED=true. Pre-U-3 the demo
 	// overlay (deploy/docker-compose.demo.yml) mounted seed_demo.sql into
 	// postgres `/docker-entrypoint-initdb.d/`; that broke once U-3 dropped
 	// the initdb migration mounts (the demo seed references tables that
 	// wouldn't exist at initdb time). The runtime path here is the
 	// post-U-3 replacement. Default-off so a vanilla deploy never lands
 	// fake-history rows. See postgres.RunDemoSeed for the contract.
 	if cfg.Database.DemoSeed {
 		logger.Info("applying demo seed (CERTCTL_DEMO_SEED=true)", "path", cfg.Database.MigrationsPath)
 		if err := postgres.RunDemoSeed(db, cfg.Database.MigrationsPath); err != nil {
 			logger.Error("failed to apply demo seed data", "error", err)
 			os.Exit(1)
 		}
 		logger.Info("demo seed completed")
 	}
 	return false
 }
@@ -1,4 +1,5 @@
-// Copyright (c) certctl-io contributors.
+// Copyright 2026 certctl LLC. All rights reserved.
 // SPDX-License-Identifier: BUSL-1.1
 //
 // Audit 2026-05-11 A-8 — demo-mode residual-grants detector. Closes the
 // deferred Phase 2 leg of HIGH-12 (cowork/auth-bundles-fixes-2026-05-10/
@@ -1,3 +1,6 @@
 // Copyright 2026 certctl LLC. All rights reserved.
 // SPDX-License-Identifier: BUSL-1.1
 package main
 import (
@@ -0,0 +1,758 @@
 // Copyright 2026 certctl LLC. All rights reserved.
 // SPDX-License-Identifier: BUSL-1.1
 package main
 import (
 	"context"
 	"crypto"
 	"crypto/tls"
 	"crypto/x509"
 	"encoding/pem"
 	"fmt"
 	"log/slog"
 	"net/http"
 	"os"
 	"strings"
 	"time"
 	"github.com/certctl-io/certctl/internal/api/handler"
 	oidcdomain "github.com/certctl-io/certctl/internal/auth/oidc/domain"
 	"github.com/certctl-io/certctl/internal/auth/session"
 	userdomain "github.com/certctl-io/certctl/internal/auth/user/domain"
 	"github.com/certctl-io/certctl/internal/domain"
 	authdomainAlias "github.com/certctl-io/certctl/internal/domain/auth"
 	"github.com/certctl-io/certctl/internal/repository"
 	"github.com/certctl-io/certctl/internal/repository/postgres"
 	"github.com/certctl-io/certctl/internal/scep/intune"
 	"github.com/certctl-io/certctl/internal/service"
 	authsvc "github.com/certctl-io/certctl/internal/service/auth"
 	"github.com/certctl-io/certctl/internal/trustanchor"
 )
 // Phase 9 ARCH-M2 closure Sprint 8 (2026-05-14): extracted from
 // cmd/server/main.go. Different shape from the config.go cuts —
 // the move is by FUNCTIONAL CONCERN (boot-time preflight + DI
 // adapter wiring), not by TYPE FAMILY.
 //
 // Sprint 8 ships TWO of the three files the Phase 9 prompt names:
 //   - main.go      — entrypoint (unchanged; what's left after the cut)
 //   - wire.go      — this file (DI assembly: preflight helpers +
 //                    adapter types that bridge package boundaries)
 //
 // The third file the prompt names — migrations.go — is NOT in this
 // commit. See "What's NOT in this sprint" below for the deferral
 // rationale.
 //
 // What lives here
 // ===============
 // Seven preflight + DI helper functions:
 //   - preflightSCEPChallengePassword   (H-2 fix: SCEP needs non-empty
 //                                       shared secret if enabled)
 //   - preflightSCEPMTLSTrustBundle     (SCEP Phase 6.5: per-profile
 //                                       mTLS CA bundle validation)
 //   - preflightESTMTLSClientCATrustBundle (EST Phase 2.5: same shape,
 //                                       returns SIGHUP-reloadable
 //                                       *trustanchor.Holder)
 //   - preflightSCEPIntuneTrustAnchor   (SCEP Phase 8.2: Intune
 //                                       Connector signing-cert bundle)
 //   - loadSCEPRAPair                   (post-preflight cert+key load)
 //   - preflightSCEPRACertKey           (RA cert/key validation: file
 //                                       mode 0600, cert+key match,
 //                                       NotAfter, RSA-or-ECDSA alg)
 //   - preflightEnrollmentIssuer        (L-005: EST/SCEP issuer can
 //                                       serve GetCACertPEM)
 //   - buildFinalHandler                (M-001 option D: HTTP dispatch
 //                                       wrapper routing
 //                                       authenticated vs no-auth
 //                                       chains by URL prefix)
 //
 // Five adapter types that bridge package boundaries (avoid import
 // cycles between internal/auth, internal/service/auth,
 // internal/api/handler, internal/auth/oidc, internal/auth/session,
 // internal/auth/breakglass):
 //   - authPermissionCheckerAdapter      (typed-string → plain-string
 //                                        auth.PermissionChecker
 //                                        interface)
 //   - authCheckResolverAdapter          (postgres ActorRoleRepository
 //                                        → handler.AuthCheckResolver)
 //   - sessionMinterAdapter              (session.Service → OIDC
 //                                        SessionMinter port)
 //   - breakglassSessionMinterAdapter    (session.Service → breakglass
 //                                        SessionMinter port + audit
 //                                        2026-05-10 HIGH-1 revoke-all)
 //   - oidcProvidersListAdapter          (postgres OIDCProviderRepository
 //                                        → handler.OIDCProvidersListResolver
 //                                        with MED-9 enabled-filter)
 //
 // Plus the silenceUnusedImports var-block that pins
 // oidcdomain.OIDCProvider as a load-bearing reference (the adapter
 // types use *userdomain.User and repository.OIDCProviderRepository
 // indirectly; oidcdomain.OIDCProvider isn't named in any function
 // signature here but is part of the Phase 3 SessionMinter contract).
 //
 // What's NOT in this sprint (and why)
 // ===================================
 // migrations.go is deferred. The Phase 9 prompt asks for three files:
 // main.go (entrypoint) + wire.go (this file) + migrations.go (boot-
 // time migration handling). The migration code (Phase 4 DEPL-M1
 // --migrate-only flag handling + RunMigrations + RunSeed call +
 // CERTCTL_MIGRATIONS_VIA_HOOK gating) lives INLINE inside the 2300-
 // line main() function — lines ~59-264 in the original — not as a
 // standalone helper.
 //
 // Extracting it into a migrations.go would require:
 //   1. Creating a new unexported function (e.g.,
 //      runMigrations(ctx, cfg, db, logger) error) that consolidates
 //      lines ~71-77 (--migrate-only parse) + ~199-248 (the migration
 //      branch + --migrate-only early-exit) + ~250-264 (the demo
 //      overlay seed branch).
 //   2. Replacing the inline block in main() with a single call.
 //   3. Threading the early-exit semantics out (os.Exit(0) vs return
 //      "migration done" sentinel error vs a third option) so main's
 //      defer ordering doesn't change.
 //
 // That's behavior-change territory — a new function call frame, a
 // new defer scope, error-handling pattern shift. Different risk
 // shape from the pure-data type relocations Sprints 1-7 did. The
 // Phase 9 prompt says "Do NOT change exported type signatures; the
 // refactor is mechanical relocation; behavior change is a separate
 // concern." Extracting an inline block from main() into a new
 // function is the same shape of risk that rule was guarding against.
 //
 // Recommended path for the migrations.go cut:
 //   - Land it as a separate, smaller PR with its own review focus
 //     (the runMigrations function shape, the early-exit semantics,
 //     unit tests for the new function via the existing main_test.go
 //     fixture). The infrastructure for the PR exists today; only
 //     the operator's go-ahead on the behavior-change risk is needed.
 //   - Estimated impact: another ~80-120 LOC out of main.go (the
 //     migration + seed + early-exit block) into a new migrations.go.
 //   - Phase 4's --migrate-only code path already runs through this
 //     code section, so the extracted function should reproduce that
 //     exact flow without behavior change beyond the call-frame
 //     introduction.
 //
 // Public-surface invariant
 // ========================
 // The moved helpers + adapter types are all in package `main`
 // (which Go cannot expose to external importers). No exported
 // surface changes. The reorganization is invisible outside
 // cmd/server/. Same-package callers in main.go (preflight*
 // invocations, adapter instantiation) resolve via the package
 // symbol table without modification.
 // preflightSCEPChallengePassword enforces the H-2 fix: if SCEP is enabled, a
 // non-empty challenge password MUST be configured. Returns a non-nil error
 // otherwise so the caller can refuse to start the control plane (CWE-306,
 // missing authentication for a critical function).
 //
 // This helper is extracted so the check can be unit tested without booting
 // the full server. The caller (main) is responsible for translating the
 // returned error into a structured log line and os.Exit(1).
 func preflightSCEPChallengePassword(enabled bool, challengePassword string) error {
 	if !enabled {
 		return nil
 	}
 	if challengePassword == "" {
 		return fmt.Errorf("SCEP enabled but CERTCTL_SCEP_CHALLENGE_PASSWORD is empty: " +
 			"SCEP enrollment would accept any client (CWE-306); " +
 			"configure a non-empty shared secret or set CERTCTL_SCEP_ENABLED=false")
 	}
 	return nil
 }
 // preflightSCEPMTLSTrustBundle validates a per-profile mTLS client-CA
 // trust bundle. SCEP RFC 8894 + Intune master bundle Phase 6.5.
 //
 // Mirrors preflightSCEPRACertKey's no-op-when-disabled pattern; otherwise
 // the checks are:
 //
 //  1. Path is non-empty (the Validate() refuse covers this too, but
 //     preflight reports the specific failure with an actionable error
 //     string + os.Exit(1) at the call site).
 //  2. File exists + readable.
 //  3. PEM-decodes to ≥1 CERTIFICATE block.
 //  4. None of the bundled certs is past NotAfter — an expired trust
 //     anchor would silently reject every client cert at runtime.
 //
 // On success, returns the parsed *x509.CertPool ready to inject into the
 // per-profile SCEPHandler via SetMTLSTrustPool. Each bundled cert also
 // contributes to the union pool that backs the TLS-layer
 // VerifyClientCertIfGiven.
 func preflightSCEPMTLSTrustBundle(enabled bool, bundlePath string) (*x509.CertPool, error) {
 	if !enabled {
 		return nil, nil
 	}
 	if bundlePath == "" {
 		return nil, fmt.Errorf("MTLS enabled but trust bundle path empty: " +
 			"set CERTCTL_SCEP_PROFILE_<NAME>_MTLS_CLIENT_CA_TRUST_BUNDLE_PATH to a PEM file " +
 			"containing the bootstrap-CA certs the operator allows to enroll")
 	}
 	body, err := os.ReadFile(bundlePath)
 	if err != nil {
 		return nil, fmt.Errorf("read MTLS trust bundle: %w (path=%s)", err, bundlePath)
 	}
 	pool := x509.NewCertPool()
 	rest := body
 	count := 0
 	now := time.Now()
 	for {
 		var block *pem.Block
 		block, rest = pem.Decode(rest)
 		if block == nil {
 			break
 		}
 		if block.Type != "CERTIFICATE" {
 			continue
 		}
 		cert, err := x509.ParseCertificate(block.Bytes)
 		if err != nil {
 			return nil, fmt.Errorf("parse MTLS trust bundle cert: %w (path=%s)", err, bundlePath)
 		}
 		if now.After(cert.NotAfter) {
 			return nil, fmt.Errorf("MTLS trust bundle cert expired at %s (subject=%q, path=%s) — replace before restart",
 				cert.NotAfter.Format(time.RFC3339), cert.Subject.CommonName, bundlePath)
 		}
 		pool.AddCert(cert)
 		count++
 	}
 	if count == 0 {
 		return nil, fmt.Errorf("MTLS trust bundle contained no CERTIFICATE PEM blocks (path=%s)", bundlePath)
 	}
 	return pool, nil
 }
 // preflightESTMTLSClientCATrustBundle validates a per-profile EST mTLS
 // client-CA trust bundle and returns a SIGHUP-reloadable holder.
 //
 // EST RFC 7030 hardening master bundle Phase 2.5.
 //
 // Mirrors preflightSCEPMTLSTrustBundle's checks (file exists, parses as
 // PEM, ≥1 cert, none expired) but returns a *trustanchor.Holder rather
 // than a raw *x509.CertPool — the EST handler stores the holder so a
 // SIGHUP rotates the trust bundle live without a server restart, exactly
 // the way the Intune trust anchor rotation works (Phase 8.5 of the SCEP
 // bundle). The handler-side .Pool() accessor on the holder rebuilds an
 // x509.CertPool from the current snapshot for each Verify call.
 //
 // Uses the shared internal/trustanchor.LoadBundle (extracted in EST
 // hardening Phase 2.1 from the original Intune-only path) so the EST
 // + Intune callers exercise the same loader semantics — empty bundle
 // rejected, expired cert rejected with subject in error message,
 // non-CERTIFICATE PEM blocks tolerated.
 func preflightESTMTLSClientCATrustBundle(enabled bool, pathID, bundlePath string, logger *slog.Logger) (*trustanchor.Holder, error) {
 	if !enabled {
 		return nil, nil
 	}
 	if bundlePath == "" {
 		return nil, fmt.Errorf("EST profile (PathID=%q) MTLS enabled but trust bundle path empty: "+
 			"set CERTCTL_EST_PROFILE_<NAME>_MTLS_CLIENT_CA_TRUST_BUNDLE_PATH to a PEM file "+
 			"containing the bootstrap-CA certs the operator allows to enroll", pathID)
 	}
 	holder, err := trustanchor.New(bundlePath, logger)
 	if err != nil {
 		return nil, fmt.Errorf("EST profile (PathID=%q) MTLS trust bundle preflight: %w", pathID, err)
 	}
 	holder.SetLabelForLog(fmt.Sprintf("EST mTLS client CA bundle (PathID=%q)", pathID))
 	return holder, nil
 }
 // preflightSCEPIntuneTrustAnchor validates a per-profile Microsoft Intune
 // Certificate Connector signing-cert trust bundle.
 //
 // SCEP RFC 8894 + Intune master bundle Phase 8.2.
 //
 // No-op when this profile has Intune disabled (the common case for
 // non-Intune SCEP deploys). When enabled:
 //
 //  1. Path is non-empty (Validate() refuse covers this too; we re-check
 //     here so the caller can os.Exit(1) with the specific PathID in the
 //     log line).
 //  2. File exists + readable.
 //  3. PEM-decodes to ≥1 CERTIFICATE block (intune.LoadTrustAnchor enforces
 //     this and skips non-CERTIFICATE blocks like accidentally-pasted
 //     priv-key blocks).
 //  4. None of the bundled certs is past NotAfter — an expired Intune
 //     trust anchor would silently reject every Connector challenge at
 //     runtime, which is a much worse failure mode than failing fast at
 //     boot. intune.LoadTrustAnchor enforces this and surfaces the subject
 //     CN in the error message so the operator knows which cert to rotate.
 //
 // On success returns the freshly-built *intune.TrustAnchorHolder ready to
 // inject into the per-profile SCEPService via SetIntuneIntegration. The
 // holder also installs the SIGHUP watcher (started by the caller).
 func preflightSCEPIntuneTrustAnchor(enabled bool, pathID, path string, logger *slog.Logger) (*intune.TrustAnchorHolder, error) {
 	if !enabled {
 		return nil, nil
 	}
 	// pathIDLabel renders the empty-string PathID as "<root>" so the
 	// operator's boot-log error doesn't read like a missing variable.
 	pathIDLabel := pathID
 	if pathIDLabel == "" {
 		pathIDLabel = "<root>"
 	}
 	if path == "" {
 		return nil, fmt.Errorf("SCEP profile (PathID=%q) INTUNE enabled but trust anchor path empty: "+
 			"set CERTCTL_SCEP_PROFILE_<NAME>_INTUNE_CONNECTOR_CERT_PATH to a PEM bundle "+
 			"of the Microsoft Intune Certificate Connector's signing certs", pathIDLabel)
 	}
 	holder, err := intune.NewTrustAnchorHolder(path, logger)
 	if err != nil {
 		return nil, fmt.Errorf("SCEP profile (PathID=%q) INTUNE trust anchor load failed: %w (path=%s)", pathIDLabel, err, path)
 	}
 	return holder, nil
 }
 // loadSCEPRAPair reads the RA cert PEM + key PEM and returns the parsed
 // x509.Certificate + crypto.PrivateKey ready for the SCEP handler's RFC
 // 8894 path. Called AFTER preflightSCEPRACertKey passed; failures here
 // indicate a TOCTOU race or a filesystem change between preflight and
 // the load (rare).
 //
 // Cert PEM may carry a chain (CA + RA + intermediate); we use the FIRST
 // CERTIFICATE block, matching the RFC 8894 §3.5.1 single-cert convention
 // for the GetCACert response.
 func loadSCEPRAPair(certPath, keyPath string) (*x509.Certificate, crypto.PrivateKey, error) {
 	certPEM, err := os.ReadFile(certPath)
 	if err != nil {
 		return nil, nil, fmt.Errorf("read RA cert: %w", err)
 	}
 	keyPEM, err := os.ReadFile(keyPath)
 	if err != nil {
 		return nil, nil, fmt.Errorf("read RA key: %w", err)
 	}
 	pair, err := tls.X509KeyPair(certPEM, keyPEM)
 	if err != nil {
 		return nil, nil, fmt.Errorf("parse RA pair: %w", err)
 	}
 	if len(pair.Certificate) == 0 {
 		return nil, nil, fmt.Errorf("RA cert PEM contained no certificate blocks")
 	}
 	leaf, err := x509.ParseCertificate(pair.Certificate[0])
 	if err != nil {
 		return nil, nil, fmt.Errorf("parse RA cert: %w", err)
 	}
 	return leaf, pair.PrivateKey, nil
 }
 // preflightSCEPRACertKey validates the RA cert/key pair the RFC 8894 SCEP
 // path requires. Mirrors preflightSCEPChallengePassword's no-op-when-disabled
 // pattern; otherwise the checks are:
 //
 //  1. Both paths are non-empty (the Validate() refuse covers this too,
 //     but preflight reports the specific failure mode + os.Exit(1) so the
 //     operator sees a clear log line in addition to the config error).
 //  2. The key file mode is 0600 (refuse world-/group-readable RA key —
 //     defense-in-depth against credential leak via a misconfigured
 //     deploy that leaves /etc/certctl/scep/*.key as 0644).
 //  3. Cert PEM parses to exactly one x509.Certificate.
 //  4. Key PEM parses to a Go crypto.Signer (RSA or ECDSA — RFC 8894
 //     §3.5.2 advertises those as the CMS-compatible algorithms).
 //  5. The cert's PublicKey matches the key's Public() — refuses pairs
 //     accidentally swapped between profiles in a multi-profile config.
 //  6. The cert's NotAfter is in the future — an expired RA cert would
 //     fail TLS handshake on EnvelopedData decryption per RFC 5652.
 //
 // Each check returns a wrapped error; the caller (main) is responsible for
 // translating to a structured slog.Error + os.Exit(1) so the helper stays
 // unit-testable without booting the full server.
 func preflightSCEPRACertKey(enabled bool, raCertPath, raKeyPath string) error {
 	if !enabled {
 		return nil
 	}
 	if raCertPath == "" || raKeyPath == "" {
 		return fmt.Errorf("SCEP enabled but RA pair missing: " +
 			"set CERTCTL_SCEP_RA_CERT_PATH + CERTCTL_SCEP_RA_KEY_PATH " +
 			"(RFC 8894 §3.2.2 requires an RA pair so clients can encrypt the " +
 			"CSR to the RA cert and the server can sign the CertRep response)")
 	}
 	// File mode check FIRST so a world-readable key never gets read into the
 	// process address space. Ignored on Windows (Stat().Mode() doesn't carry
 	// POSIX bits there); the production deploy is Linux per the Dockerfile.
 	keyInfo, err := os.Stat(raKeyPath)
 	if err != nil {
 		return fmt.Errorf("CERTCTL_SCEP_RA_KEY_PATH stat failed: %w (path=%s)", err, raKeyPath)
 	}
 	mode := keyInfo.Mode().Perm()
 	if mode&0o077 != 0 {
 		return fmt.Errorf("CERTCTL_SCEP_RA_KEY_PATH has insecure permissions %#o; "+
 			"RA private key must be mode 0600 (owner read/write only) — "+
 			"chmod 0600 %s and restart", mode, raKeyPath)
 	}
 	certPEM, err := os.ReadFile(raCertPath)
 	if err != nil {
 		return fmt.Errorf("CERTCTL_SCEP_RA_CERT_PATH read failed: %w (path=%s)", err, raCertPath)
 	}
 	keyPEM, err := os.ReadFile(raKeyPath)
 	if err != nil {
 		return fmt.Errorf("CERTCTL_SCEP_RA_KEY_PATH read failed: %w (path=%s)", err, raKeyPath)
 	}
 	// tls.X509KeyPair validates that the cert + key parse, share an algorithm,
 	// and the cert's PublicKey matches the key's Public() — three of our six
 	// checks in a single stdlib call, so we use it rather than re-implementing.
 	pair, err := tls.X509KeyPair(certPEM, keyPEM)
 	if err != nil {
 		return fmt.Errorf("RA cert/key pair invalid: %w "+
 			"(cert=%s key=%s) — verify the cert and key are matching halves of "+
 			"the same RA pair, both PEM-encoded, with the cert containing exactly "+
 			"one CERTIFICATE block and the key containing one PRIVATE KEY block",
 			err, raCertPath, raKeyPath)
 	}
 	if len(pair.Certificate) == 0 {
 		// Defensive — tls.X509KeyPair already errors on this, but the contract
 		// for the next x509.ParseCertificate call needs the slice non-empty.
 		return fmt.Errorf("RA cert PEM at %s contains no certificate blocks", raCertPath)
 	}
 	// Re-parse the leaf so we can read NotAfter + the public-key alg.
 	leaf, err := x509.ParseCertificate(pair.Certificate[0])
 	if err != nil {
 		return fmt.Errorf("RA cert at %s does not parse as x509: %w", raCertPath, err)
 	}
 	if time.Now().After(leaf.NotAfter) {
 		return fmt.Errorf("RA cert at %s expired at %s — "+
 			"generate a fresh RA pair (the SCEP CertRep signature would be "+
 			"rejected by every conformant client)", raCertPath, leaf.NotAfter.Format(time.RFC3339))
 	}
 	// CMS-compatible public-key algorithm gate. RFC 8894 §3.5.2 advertises RSA
 	// and AES; the responder cert algorithm pertains to the signature scheme
 	// used on the CertRep, which means the cert's PublicKey must be RSA or
 	// ECDSA. Catches pre-shared Ed25519 dev keys that micromdm/scep clients
 	// reject.
 	switch leaf.PublicKeyAlgorithm {
 	case x509.RSA, x509.ECDSA:
 		// ok — supported by golang.org/x/crypto/ocsp + every SCEP client
 	default:
 		return fmt.Errorf("RA cert at %s uses unsupported public-key algorithm %s — "+
 			"RFC 8894 §3.5.2 CMS signing requires RSA or ECDSA",
 			raCertPath, leaf.PublicKeyAlgorithm)
 	}
 	return nil
 }
 // preflightEnrollmentIssuer validates at startup that an EST/SCEP-bound issuer
 // can actually serve a CA certificate. This closes audit finding L-005:
 // pre-Bundle-4 the EST/SCEP startup path verified the issuer existed in the
 // registry but did not verify the issuer TYPE could emit a CA cert. An
 // operator who bound CERTCTL_EST_ISSUER_ID to an ACME issuer (which does
 // not have a static CA cert — see internal/connector/issuer/acme/acme.go::
 // GetCACertPEM returning an explicit error) would boot successfully and
 // only see failures at the first /est/cacerts request, hiding the misconfig
 // for hours/days behind a degraded enrollment surface.
 //
 // Strategy: call issuerConn.GetCACertPEM(ctx) at startup with a short
 // timeout. If the issuer can serve a CA cert (local, vault, openssl,
 // stepca, awsacmpca, etc.), the call succeeds and we proceed. If not
 // (acme, digicert, sectigo, entrust, googlecas, ejbca, globalsign — most
 // vendor-CA issuers that hand back chains per-issuance), the call fails
 // loudly with the connector's own error string, and the caller os.Exit(1)s.
 //
 // Returns nil on success, non-nil error suitable for structured logging
 // + os.Exit(1) by the caller. Caller is responsible for the timeout context.
 func preflightEnrollmentIssuer(ctx context.Context, protocol, issuerID string, issuerConn service.IssuerConnector) error {
 	if issuerConn == nil {
 		return fmt.Errorf("%s issuer %q: connector is nil", protocol, issuerID)
 	}
 	caCertPEM, err := issuerConn.GetCACertPEM(ctx)
 	if err != nil {
 		return fmt.Errorf("%s issuer %q: cannot serve CA certificate (%w); "+
 			"choose an issuer type that exposes a static CA chain "+
 			"(local / vault / openssl / stepca / awsacmpca) or disable %s",
 			protocol, issuerID, err, protocol)
 	}
 	if caCertPEM == "" {
 		return fmt.Errorf("%s issuer %q: GetCACertPEM returned empty PEM with no error; "+
 			"choose an issuer type that exposes a static CA chain", protocol, issuerID)
 	}
 	return nil
 }
 // buildFinalHandler builds the outer HTTP dispatch handler that routes incoming
 // requests to either the authenticated apiHandler chain or the unauthenticated
 // noAuthHandler chain based on URL path prefix. Extracted from main() so the
 // dispatch logic can be unit tested without booting the full server stack
 // (see cmd/server/finalhandler_test.go).
 //
 // Dispatch rules (M-001, audit 2026-04-19, option D):
 //
 //   - /health, /ready, /api/v1/auth/info           → no-auth (probes + login detection)
 //   - /api/v1/version                              → no-auth (U-3 ride-along: build identity for rollout/probes)
 //   - /.well-known/pki/*                           → no-auth (RFC 5280 CRL, RFC 6960 OCSP)
 //   - /.well-known/est/*                           → no-auth (RFC 7030 §3.2.3)
 //   - /scep, /scep/*                               → no-auth (RFC 8894 §3.2, CSR challengePassword)
 //   - /api/v1/*                                    → auth (Bearer token required)
 //   - /assets/*                                    → static file server (dashboard only)
 //   - anything else                                → SPA index.html fallback (dashboard only)
 //     OR apiHandler (no dashboard)
 //
 // EST/SCEP clients (IoT devices, 802.1X supplicants, MDM endpoints, network
 // appliances) cannot present certctl Bearer tokens, so those endpoints must be
 // reachable without the Auth middleware. Authentication is instead enforced by
 // CSR signature verification, profile policy gates, and for SCEP the
 // challengePassword shared secret (fail-loud gated by preflightSCEPChallengePassword
 // above).
 //
 // webDir must point to a directory containing index.html + assets/ when
 // dashboardEnabled is true; it is ignored otherwise.
 func buildFinalHandler(apiHandler, noAuthHandler http.Handler, webDir string, dashboardEnabled bool) http.Handler {
 	var fileServer http.Handler
 	if dashboardEnabled {
 		fileServer = http.FileServer(http.Dir(webDir))
 	}
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		path := r.URL.Path
 		// Health/ready, auth/info, and version bypass auth middleware.
 		// Health/ready: Docker/K8s health probes don't carry Bearer tokens.
 		// auth/info: React app calls this before login to detect auth mode.
 		// version: U-3 ride-along (cat-u-no_version_endpoint) — rollout
 		// systems and blackbox probes need build identity without a key.
 		if path == "/health" || path == "/ready" || path == "/api/v1/auth/info" || path == "/api/v1/version" {
 			noAuthHandler.ServeHTTP(w, r)
 			return
 		}
 		// RFC 5280 CRL and RFC 6960 OCSP live under /.well-known/pki/ and MUST
 		// be served unauthenticated — relying parties (browsers, OpenSSL, OCSP
 		// stapling sidecars, mTLS clients) cannot present certctl Bearer tokens.
 		if strings.HasPrefix(path, "/.well-known/pki") {
 			noAuthHandler.ServeHTTP(w, r)
 			return
 		}
 		// RFC 7030 EST endpoints ride the no-auth middleware chain (M-001,
 		// option D, audit 2026-04-19). Trust boundary is CSR signature +
 		// (per EST hardening Phase 2) optional client cert at the handler
 		// layer, not HTTP Bearer. /.well-known/est/cacerts is explicitly
 		// anonymous per RFC 7030 §4.1.1; /.well-known/est-mtls/<PathID>/
 		// (EST hardening Phase 2 sibling route) requires a client cert
 		// gate at the handler layer — both share this prefix gate because
 		// "/.well-known/est-mtls" is itself prefixed by "/.well-known/est".
 		// EST hardening Phase 3's HTTP Basic enrollment-password is a
 		// per-profile handler-layer auth that runs INSIDE the no-auth
 		// middleware chain (since the chain skips the Bearer middleware,
 		// the handler gets to define its own auth contract).
 		if strings.HasPrefix(path, "/.well-known/est") {
 			noAuthHandler.ServeHTTP(w, r)
 			return
 		}
 		// RFC 8894 SCEP rides the no-auth chain (M-001, option D). SCEP clients
 		// authenticate via the challengePassword attribute in the PKCS#10 CSR,
 		// not via HTTP Bearer tokens. preflightSCEPChallengePassword refuses to
 		// start the server if SCEP is enabled without a non-empty shared secret.
 		//
 		// SCEP RFC 8894 + Intune master bundle Phase 6.5: the sibling
 		// /scep-mtls[/<pathID>] route also rides the no-auth chain. Its
 		// auth boundary is (a) client cert verified at the TLS layer +
 		// re-verified per-profile at the handler layer, plus (b) the
 		// challenge password — neither is a Bearer token. The /scepxyz
 		// vs /scep-mtls disambiguation: 'xyz' starts with a letter so the
 		// HasPrefix(path, "/scep/") gate doesn't match it; 'mtls' is its
 		// own dedicated prefix gated below to avoid the same overlap.
 		if path == "/scep" || strings.HasPrefix(path, "/scep/") {
 			noAuthHandler.ServeHTTP(w, r)
 			return
 		}
 		if path == "/scep-mtls" || strings.HasPrefix(path, "/scep-mtls/") {
 			noAuthHandler.ServeHTTP(w, r)
 			return
 		}
 		// Authenticated API routes — full middleware stack including Auth.
 		if strings.HasPrefix(path, "/api/v1/") {
 			apiHandler.ServeHTTP(w, r)
 			return
 		}
 		if !dashboardEnabled {
 			// No dashboard: everything non-special falls through to the
 			// authenticated handler (preserves pre-M-001 behavior for API-only
 			// deployments).
 			apiHandler.ServeHTTP(w, r)
 			return
 		}
 		// Dashboard-present: serve static assets directly, SPA fallback for
 		// everything else.
 		if strings.HasPrefix(path, "/assets/") {
 			fileServer.ServeHTTP(w, r)
 			return
 		}
 		http.ServeFile(w, r, webDir+"/index.html")
 	})
 }
 // authPermissionCheckerAdapter bridges the typed-string Authorizer
 // signature (authsvc.Authorizer.CheckPermission takes
 // authdomain.ActorTypeValue + authdomain.ScopeType) to the plain-string
 // auth.PermissionChecker interface used by the auth.RequirePermission
 // middleware factory. Lives in cmd/server so internal/auth doesn't have
 // to import internal/service/auth + internal/domain/auth (would create
 // a cycle).
 type authPermissionCheckerAdapter struct {
 	a *authsvc.Authorizer
 }
 func (ad authPermissionCheckerAdapter) CheckPermission(
 	ctx context.Context,
 	actorID string,
 	actorType string,
 	tenantID string,
 	permission string,
 	scopeType string,
 	scopeID *string,
 ) (bool, error) {
 	return ad.a.CheckPermission(
 		ctx,
 		actorID,
 		authdomainAlias.ActorTypeValue(actorType),
 		tenantID,
 		permission,
 		authdomainAlias.ScopeType(scopeType),
 		scopeID,
 	)
 }
 // authCheckResolverAdapter bridges the postgres ActorRoleRepository
 // (authdomain.ActorTypeValue) to handler.AuthCheckResolver
 // (domain.ActorType). Lives in cmd/server so the handler layer keeps its
 // existing import set; the GUI's /v1/auth/check probe round-trips
 // through this on every page load. Read-only — no caller / no audit row.
 //
 // Bundle 1 Phase 3 closure (M1): the equivalent surface area on
 // /v1/auth/me runs through the service layer's auth.role.list permission
 // gate, which the GUI may not yet hold during initial render. AuthCheck
 // has no permission gate (its only requirement is "the request
 // authenticated"), so the bypass is by design.
 type authCheckResolverAdapter struct {
 	repo *postgres.ActorRoleRepository
 }
 func (ad authCheckResolverAdapter) ListRoles(
 	ctx context.Context,
 	actorID string,
 	actorType domain.ActorType,
 	tenantID string,
 ) ([]*authdomainAlias.ActorRole, error) {
 	return ad.repo.ListByActor(ctx, actorID, authdomainAlias.ActorTypeValue(actorType), tenantID)
 }
 func (ad authCheckResolverAdapter) EffectivePermissions(
 	ctx context.Context,
 	actorID string,
 	actorType domain.ActorType,
 	tenantID string,
 ) ([]repository.EffectivePermission, error) {
 	return ad.repo.EffectivePermissions(ctx, actorID, authdomainAlias.ActorTypeValue(actorType), tenantID)
 }
 // =============================================================================
 // sessionMinterAdapter — bridge from *session.Service to oidcsvc.SessionMinter.
 //
 // The OIDC service's SessionMinter port (Phase 3) takes a *userdomain.User
 // + role IDs and returns (cookie, csrf, err). The session.Service's
 // Create method takes (actorID, actorType, ip, ua) -> *CreateResult.
 // This adapter unwraps the User into actorID/actorType + reshapes the
 // return tuple. Lives in cmd/server so the session package doesn't have
 // to know about user.User and the user package doesn't have to know
 // about session.CreateResult.
 // =============================================================================
 type sessionMinterAdapter struct {
 	svc *session.Service
 }
 func (a *sessionMinterAdapter) MintForUser(
 	ctx context.Context,
 	user *userdomain.User,
 	_ []string, // roleIDs unused at the session-mint layer; the rbac middleware looks them up at request time
 	ip, userAgent string,
 ) (cookieValue, csrfToken string, err error) {
 	if user == nil {
 		return "", "", fmt.Errorf("session mint: user is nil")
 	}
 	res, err := a.svc.Create(ctx, user.ID, string(domain.ActorTypeUser), ip, userAgent)
 	if err != nil {
 		return "", "", err
 	}
 	return res.CookieValue, res.CSRFToken, nil
 }
 // silenceUnusedImports keeps the new oidcsvc + oidcdomain imports load-
 // bearing in case any file shuffles. Linker dead-code elimination handles
 // the runtime cost.
 var (
 	_ = oidcdomain.OIDCProvider{}
 )
 // =============================================================================
 // breakglassSessionMinterAdapter — bridge from *session.Service to
 // breakglass.SessionMinter.
 //
 // The break-glass service's SessionMinter port (Phase 7.5) returns
 // (cookie, csrf, err); the underlying *session.Service.Create returns
 // *CreateResult. This adapter unwraps the result. Lives in cmd/server
 // so the breakglass package doesn't have to know about session.Service.
 // =============================================================================
 type breakglassSessionMinterAdapter struct {
 	svc *session.Service
 }
 func (a breakglassSessionMinterAdapter) Create(ctx context.Context, actorID, actorType, ip, userAgent string) (string, string, error) {
 	res, err := a.svc.Create(ctx, actorID, actorType, ip, userAgent)
 	if err != nil {
 		return "", "", err
 	}
 	return res.CookieValue, res.CSRFToken, nil
 }
 // RevokeAllForActor — Audit 2026-05-10 HIGH-1 wire. After a break-glass
 // password rotation or credential removal, every active session for the
 // target actor must be revoked so a phished-then-rotated credential
 // doesn't leave the attacker's session live.
 func (a breakglassSessionMinterAdapter) RevokeAllForActor(ctx context.Context, actorID, actorType string) error {
 	return a.svc.RevokeAllForActor(ctx, actorID, actorType)
 }
 // oidcProvidersListAdapter bridges the postgres OIDCProviderRepository
 // to handler.OIDCProvidersListResolver. The handler returns
 // []*OIDCProviderInfo (id + display_name + login_url) for the public-
 // safe GUI Login-page payload; the repo returns the full OIDCProvider
 // row. The adapter projects + maps the login_url shape that
 // /auth/oidc/login?provider=<id> expects. Auth Bundle 2 Phase 6 /
 // Category E.
 type oidcProvidersListAdapter struct {
 	repo repository.OIDCProviderRepository
 }
 func (a oidcProvidersListAdapter) List(ctx context.Context, tenantID string) ([]*handler.OIDCProviderInfo, error) {
 	provs, err := a.repo.List(ctx, tenantID)
 	if err != nil {
 		return nil, err
 	}
 	out := make([]*handler.OIDCProviderInfo, 0, len(provs))
 	for _, p := range provs {
 		// Audit 2026-05-10 MED-9 closure — filter disabled providers
 		// at the adapter so the LoginPage's "Sign in with X" buttons
 		// don't render for offline IdPs. The HandleAuthRequest
 		// service-layer ErrProviderDisabled check is the
 		// defense-in-depth guard for direct API / MCP / CLI callers.
 		if !p.Enabled {
 			continue
 		}
 		out = append(out, &handler.OIDCProviderInfo{
 			ID:          p.ID,
 			DisplayName: p.Name,
 			LoginURL:    "/auth/oidc/login?provider=" + p.ID,
 		})
 	}
 	return out, nil
 }
@@ -1,8 +1,39 @@
-# certctl Docker Compose environment variables
+# certctl Docker Compose environment variables (Bundle 2 — 2026-05-12)
-# Copy this file to .env and customize for your deployment
+#
 # Copy this file to deploy/.env and customize. The production-shaped base
 # compose (docker-compose.yml) requires every variable below to be set;
 # the Bundle 2 fail-closed startup guards REFUSE TO BOOT if any value
 # remains at a "change-me-..." or "replace-with-..." placeholder outside
 # demo mode (CERTCTL_DEMO_MODE_ACK=true).
 #
 # DEMO PATH (zero-config, populated dashboard, demo-mode auth):
 #   docker compose -f deploy/docker-compose.yml \
 #                  -f deploy/docker-compose.demo.yml up -d --build
 # The demo overlay supplies its own placeholder values plus DEMO_MODE_ACK
 # so this .env is NOT needed.
 #
 # PRODUCTION PATH (this .env is required):
 #   docker compose -f deploy/docker-compose.yml up -d
-# PostgreSQL password (change in production!)
+# PostgreSQL password — openssl rand -hex 32
-POSTGRES_PASSWORD=certctl
+POSTGRES_PASSWORD=replace-with-openssl-rand-hex-32
-# Agent API key (change in production! Generate with: openssl rand -hex 32)
+# Server API-key secret — openssl rand -base64 32
-CERTCTL_API_KEY=change-me-in-production
+CERTCTL_AUTH_SECRET=replace-with-openssl-rand-base64-32
 # Bundled-agent API key (matches one of the server's AUTH_SECRET rotation
 # values). Generate with: openssl rand -base64 32
 CERTCTL_API_KEY=replace-with-openssl-rand-base64-32
 # AES-256-GCM key for encrypting issuer/target config secrets at rest.
 # Minimum 32 bytes. Generate with: openssl rand -base64 32
 CERTCTL_CONFIG_ENCRYPTION_KEY=replace-with-openssl-rand-base64-32
 # Agent ID returned from `POST /api/v1/agents` during agent enrollment.
 # Without this the bundled certctl-agent service fail-fasts at startup.
 # CERTCTL_AGENT_ID=agent-from-registration-response
 # Day-0 admin bootstrap token (optional — generate with: openssl rand -hex 32).
 # When set, POST /api/v1/auth/bootstrap mints the first admin actor + API
 # key. When unset (default), that endpoint returns 410 Gone.
 # CERTCTL_BOOTSTRAP_TOKEN=
@@ -62,7 +62,9 @@ A compose file defines **services** (containers), **networks** (how they talk to
 ## Base Environment
 **File:** `docker-compose.yml`
-**When to use:** Production deployments, first-time setup, or any time you want a clean dashboard with the onboarding wizard.
+**When to use:** Production deployments and any time you want a clean, production-shaped stack with real authentication enforced.
 **Bundle 2 closure (2026-05-12):** the base compose was split from the demo overlay. Pre-Bundle-2 this file IS the demo path (auth=none, keygen=server, demo-seed=true, change-me placeholder credentials baked in). Operators reading "drop the demo overlay for a clean install" were not getting a clean install — they were getting a demo stack with the overlay's data layer stripped off. Post-Bundle-2 the base ships production-shaped: `CERTCTL_AUTH_TYPE` defaults to `api-key`, `CERTCTL_KEYGEN_MODE` defaults to `agent`, demo-mode + demo-seed default to false, and every credential placeholder is rejected at startup. The demo path is now a single overlay flag away (`-f deploy/docker-compose.demo.yml`).
 ### What it runs
@@ -79,9 +81,20 @@ Three services on a private bridge network:
 ```bash
 git clone https://github.com/certctl-io/certctl.git
 cd certctl
 # Required: provide real credentials. Without this step the server fail-fasts
 # at startup on the Bundle 2 placeholder-credential guards.
 cp .env.example deploy/.env
 $EDITOR deploy/.env
 # Set: POSTGRES_PASSWORD, CERTCTL_AUTH_SECRET, CERTCTL_API_KEY,
 #      CERTCTL_CONFIG_ENCRYPTION_KEY (all via `openssl rand -base64 32`),
 #      CERTCTL_AGENT_ID (returned from `POST /api/v1/agents`).
 docker compose -f deploy/docker-compose.yml up -d --build
 ```
 If you just want to kick the tires without writing a `.env`, use the demo overlay instead — see [Demo Overlay](#demo-overlay) below.
 `--build` compiles the Go server and agent from source, including the React frontend. Without it, Docker may reuse a stale image from a previous build.
 `-d` runs in detached mode (background). Omit it to see logs in your terminal.
@@ -132,14 +145,16 @@ certctl-server:
    postgres:
      condition: service_healthy
  environment:
-    CERTCTL_DATABASE_URL: postgres://certctl:${POSTGRES_PASSWORD:-certctl}@postgres:5432/certctl?sslmode=disable
+    CERTCTL_DATABASE_URL: postgres://certctl:${POSTGRES_PASSWORD}@postgres:5432/certctl?sslmode=disable
    CERTCTL_SERVER_HOST: 0.0.0.0
    CERTCTL_SERVER_PORT: 8443
    CERTCTL_LOG_LEVEL: info
-    CERTCTL_AUTH_TYPE: none
+    # Bundle 2 (2026-05-12): no auth-type / keygen-mode override here.
-    CERTCTL_KEYGEN_MODE: server
+    # Code defaults (api-key + agent) take effect; the demo overlay flips
    # both to demo-mode (none + server).
    CERTCTL_AUTH_SECRET: ${CERTCTL_AUTH_SECRET}
    CERTCTL_NETWORK_SCAN_ENABLED: "true"
-    CERTCTL_CONFIG_ENCRYPTION_KEY: ${CERTCTL_CONFIG_ENCRYPTION_KEY:-change-me-32-char-encryption-key}
+    CERTCTL_CONFIG_ENCRYPTION_KEY: ${CERTCTL_CONFIG_ENCRYPTION_KEY}
 ```
 The server is the control plane. It serves the REST API, the React dashboard, runs 7 background scheduler loops (renewal, job processing, health checks, notifications, short-lived cert expiry, network scanning, digest emails), and manages the issuer/target registry.
@@ -147,9 +162,10 @@ The server is the control plane. It serves the REST API, the React dashboard, ru
 Key environment variables explained:
 - `CERTCTL_DATABASE_URL` references the `postgres` service by hostname. Docker's internal DNS resolves `postgres` to the container's IP on the bridge network. `sslmode=disable` is appropriate because traffic stays on the private Docker network.
- `CERTCTL_AUTH_TYPE: none` disables API key authentication so you can explore immediately. For production, set `api-key` and configure `CERTCTL_AUTH_SECRET`.
+- `CERTCTL_AUTH_TYPE` defaults to `api-key` in the code (`internal/config/config.go`); the base compose does NOT override it. To run demo-mode auth (every request served as the synthetic admin actor), layer the demo overlay on top.
- `CERTCTL_KEYGEN_MODE: server` means the server generates private keys. This is convenient for demos but insecure for production. In production, set `agent` so keys are generated on agent machines and never transmitted.
+- `CERTCTL_AUTH_SECRET` is the API-key value the server accepts. The Bundle 2 fail-closed guard rejects the literal placeholder `change-me-in-production` outside demo mode. Generate with `openssl rand -base64 32`.
- `CERTCTL_CONFIG_ENCRYPTION_KEY` enables AES-256-GCM encryption for issuer and target configurations stored in the database (credentials, API keys). Without this, the dynamic configuration GUI (adding issuers/targets from the dashboard) won't encrypt sensitive fields. For production, generate a strong random key.
+- `CERTCTL_KEYGEN_MODE` defaults to `agent` in the code (the base compose does NOT override it). Production deploys leave it there so private keys stay on agent infrastructure; the demo overlay flips it to `server` so the demo can issue + hold the key on the server box without an agent dance.
 - `CERTCTL_CONFIG_ENCRYPTION_KEY` enables AES-256-GCM encryption for issuer and target configurations stored in the database (credentials, API keys). Required for any deploy that adds issuers via the GUI. The Bundle 2 fail-closed guard rejects the literal placeholder `change-me-32-char-encryption-key` outside demo mode. Generate with `openssl rand -base64 32` (≥ 32 bytes).
 - `CERTCTL_NETWORK_SCAN_ENABLED` activates the scheduler loop that probes TLS endpoints on your network to discover certificates you might not be managing.
 **Expert note:** The healthcheck hits `GET /health` every 10 seconds with 5 retries. The `depends_on: condition: service_healthy` on the agent means Docker holds agent startup until this check passes. Resource limits (`cpus: '1.0'`, `memory: 512M`) prevent the server from consuming unbounded resources in shared environments.
@@ -162,8 +178,12 @@ certctl-agent:
    certctl-server:
      condition: service_healthy
  environment:
-    CERTCTL_SERVER_URL: http://certctl-server:8443
+    CERTCTL_SERVER_URL: https://certctl-server:8443
-    CERTCTL_API_KEY: ${CERTCTL_API_KEY:-change-me-in-production}
+    # Bundle 2 (2026-05-12): no placeholder fallbacks. Operators MUST
    # set CERTCTL_API_KEY + CERTCTL_AGENT_ID in deploy/.env. The agent
    # binary fail-fasts at startup when CERTCTL_AGENT_ID is unset.
    CERTCTL_API_KEY: ${CERTCTL_API_KEY}
    CERTCTL_AGENT_ID: ${CERTCTL_AGENT_ID}
    CERTCTL_AGENT_NAME: docker-agent
    CERTCTL_LOG_LEVEL: info
    CERTCTL_DISCOVERY_DIRS: /var/lib/certctl/keys
@@ -194,13 +214,18 @@ docker compose -f deploy/docker-compose.yml down -v
 ## Demo Overlay
 **File:** `docker-compose.demo.yml`
-**When to use:** Demos, screenshots, stakeholder presentations, or any time you want a populated dashboard on first boot.
+**When to use:** Demos, screenshots, stakeholder presentations, or any time you want a one-command zero-config evaluation stack with a populated dashboard.
 ### What it adds
-One env var: `CERTCTL_DEMO_SEED=true` on the `certctl-server` service. The server applies `migrations/seed_demo.sql` at boot via `postgres.RunDemoSeed` AFTER the baseline migrations + `seed.sql` are in place. The demo seed file inserts 180 days of simulated operational history: teams, owners, certificates across multiple issuers, agents on different platforms, jobs with realistic timestamps, discovery scan results, audit events, policies, and profiles.
+Bundle 2 closure (2026-05-12) moved every demo-mode env var out of the base compose into this overlay. The overlay now carries:
-Pre-U-3 the overlay used to mount `seed_demo.sql` into PostgreSQL's `/docker-entrypoint-initdb.d/` and rely on initdb-time application. That worked only because the production stack also mounted the migrations there, so the schema existed when initdb ran. Once U-3 dropped the production initdb mounts (single source of truth: server runs `RunMigrations` + `RunSeed` at boot), the demo seed could no longer be applied at initdb time — the tables it references wouldn't exist yet. Post-U-3 the overlay is a 27-line override file with no `image:` / `build:` of its own; it MUST be passed alongside the base, or compose errors with `service "certctl-server" has neither an image nor a build context specified`.
+- `CERTCTL_AUTH_TYPE=none` + `CERTCTL_DEMO_MODE_ACK=true` — demo-mode synthetic admin actor (`actor-demo-anon`). The server emits a prominent ⚠ DEMO MODE WARN banner at boot with a production-promotion checklist (`cmd/server/main.go`).
 - `CERTCTL_KEYGEN_MODE=server` — demo-only server-side keygen.
 - `CERTCTL_DEMO_SEED=true` — the server applies `migrations/seed_demo.sql` at boot via `postgres.RunDemoSeed`, inserting 180 days of simulated operational history (teams, owners, certificates, agents, jobs, discovery results, audit events, policies, profiles).
 - Fixed weak `POSTGRES_PASSWORD=certctl`, `CERTCTL_AUTH_SECRET=change-me-in-production`, `CERTCTL_CONFIG_ENCRYPTION_KEY=change-me-32-char-encryption-key`, `CERTCTL_API_KEY=change-me-in-production`, `CERTCTL_AGENT_ID=agent-demo-1` — placeholder credentials the Bundle 2 fail-closed `Validate()` rejects outside demo mode, but the demo overlay's `DEMO_MODE_ACK=true` unlocks them.
 Pre-U-3 the overlay used to mount `seed_demo.sql` into PostgreSQL's `/docker-entrypoint-initdb.d/` and rely on initdb-time application. That worked only because the production stack also mounted the migrations there, so the schema existed when initdb ran. Once U-3 dropped the production initdb mounts (single source of truth: server runs `RunMigrations` + `RunSeed` at boot), the demo seed could no longer be applied at initdb time — the tables it references wouldn't exist yet. Post-U-3 the overlay is an override file with no `image:` / `build:` of its own; it MUST be passed alongside the base, or compose errors with `service "certctl-server" has neither an image nor a build context specified`.
 ### Starting it
@@ -382,7 +407,7 @@ Every `CERTCTL_*` environment variable is read by the server's `internal/config/
 | `CERTCTL_SERVER_HOST` | `0.0.0.0` | Listen address |
 | `CERTCTL_SERVER_PORT` | `8443` | Listen port |
 | `CERTCTL_LOG_LEVEL` | `info` | Log verbosity: `debug`, `info`, `warn`, `error` |
-| `CERTCTL_AUTH_TYPE` | `api-key` | Auth mode: `api-key` or `none` |
+| `CERTCTL_AUTH_TYPE` | `api-key` | Auth mode: `api-key`, `none`, or `oidc` (Auth Bundle 2). |
 | `CERTCTL_AUTH_SECRET` | (none) | API key(s), comma-separated for rotation |
 | `CERTCTL_KEYGEN_MODE` | `agent` | Key generation: `agent` (production) or `server` (demo) |
 | `CERTCTL_CONFIG_ENCRYPTION_KEY` | (none) | AES-256-GCM key for encrypting issuer/target configs in DB |
@@ -392,6 +417,15 @@ Every `CERTCTL_*` environment variable is read by the server's `internal/config/
 | `CERTCTL_CORS_ORIGINS` | (empty) | Allowed CORS origins, comma-separated. Empty = deny all cross-origin |
 | `CERTCTL_RATE_LIMIT_RPS` | `10` | Requests per second per client |
 | `CERTCTL_RATE_LIMIT_BURST` | `20` | Burst allowance above RPS |
 | `CERTCTL_RATE_LIMIT_BUCKET_TTL` | `1h` | Sprint 2 SEC-006: lifetime of an unused token-bucket entry. A background sweeper running every `BucketTTL/4` reclaims buckets whose last `allow()` call is older than this. Values < 1m clamp up to 1m. Lower when facing high-cardinality unauthenticated traffic (CGNAT churn, scanners) where the bucket-map RSS becomes a concern. |
 | `CERTCTL_SCHEDULER_JOB_CLAIM_LIMIT` | `1000` | Sprint 2 SCALE-001: cap on the number of Pending rows a single scheduler tick may claim via `ClaimPendingJobs`. Pre-Sprint-2 the scheduler claimed every Pending row in one transaction, which page-thrashed on 100K-job bursts. Values ≤ 0 fail-safe to `1000` (legacy unlimited semantics are no longer reachable). Pair-tune with `CERTCTL_RENEWAL_CONCURRENCY` (default 25) — the default 40:1 ratio keeps the fan-out busy without exhausting upstream-CA rate limits. |
 | `CERTCTL_AGENT_BOOTSTRAP_TOKEN` | (empty — required) | Agent-registration bootstrap secret. Set to a real value (`openssl rand -base64 32`). Sprint 5 ACQ RED-003 (2026-05-16) flipped the paired `_DENY_EMPTY` flag's default to `true`, so leaving this empty now refuses server start (unless `CERTCTL_DEMO_MODE_ACK=true`). Operators on v2.1.x reopening the warn-mode escape hatch one upgrade-window can set `CERTCTL_AGENT_BOOTSTRAP_TOKEN_DENY_EMPTY=false` explicitly. |
 | `CERTCTL_AGENT_BOOTSTRAP_TOKEN_DENY_EMPTY` | `true` | Phase 2 SEC-H1 fail-closed guard. When `true` (default since Sprint 5 ACQ RED-003 closure, 2026-05-16), the server refuses to start unless `CERTCTL_AGENT_BOOTSTRAP_TOKEN` is non-empty. Set to `false` only for a v2.1.x→v2.2.x upgrade-window warn-mode escape hatch. |
 | `CERTCTL_DEMO_MODE_ACK` | `false` | Acknowledges demo-mode synthetic admin posture (required when `CERTCTL_AUTH_TYPE=none` binds to a non-loopback host). Must be paired with `CERTCTL_DEMO_MODE_ACK_TS` per Phase 2 SEC-H3. |
 | `CERTCTL_DEMO_MODE_ACK_TS` | (empty) | Phase 2 SEC-H3: unix-epoch timestamp at which DemoModeAck was last acknowledged. When `CERTCTL_DEMO_MODE_ACK=true`, this must parse as a unix epoch within the last 24h. Set via `CERTCTL_DEMO_MODE_ACK_TS=$(date +%s)` at every `docker compose up`. |
 | `CERTCTL_ACME_INSECURE_ACK` | `false` | Phase 2 SEC-M4: explicit ACK required to boot with `CERTCTL_ACME_INSECURE=true`. Production deploys MUST never set either flag. |
 | `CERTCTL_DATABASE_MAX_CONNS` | `50` | Phase 6 SCALE-M1: max open DB connections in the server's pool. Default was `25` pre-Phase-6. Idle connections = max/5. Operator-tune ladder for larger fleets: ≤500 certs → 50; 5K certs → 100; 50K certs → 200 (also raise Postgres `max_connections`). See `docs/operator/scale.md`. |
 | `CERTCTL_ASYNC_POLL_MAX_WAIT_SECONDS` | (unset → 600) | Phase 6 SCALE-M3: process-wide override for the asyncpoll package's `DefaultMaxWait` (10 minutes). Caps total wall-clock time the certctl-server spends polling an async CA (DigiCert / Entrust / GlobalSign / Sectigo) before returning `StillPending` to the scheduler for re-enqueue. Per-connector overrides (`CERTCTL_DIGICERT_POLL_MAX_WAIT_SECONDS`, etc.) take precedence when set. |
 ### Agent
@@ -400,7 +434,7 @@ Every `CERTCTL_*` environment variable is read by the server's `internal/config/
 | `CERTCTL_SERVER_URL` | (required) | Server API URL |
 | `CERTCTL_API_KEY` | (none) | API key for authenticating with server |
 | `CERTCTL_AGENT_NAME` | (hostname) | Display name in dashboard |
-| `CERTCTL_AGENT_ID` | (auto-generated) | Stable agent identifier |
+| `CERTCTL_AGENT_ID` | (none — required) | Stable agent identifier returned from `POST /api/v1/agents`. The agent binary fail-fasts at startup if unset. |
 | `CERTCTL_KEYGEN_MODE` | `agent` | Must match server setting |
 | `CERTCTL_LOG_LEVEL` | `info` | Log verbosity |
 | `CERTCTL_KEY_DIR` | `/var/lib/certctl/keys` | Directory for private key storage (0600 perms) |
@@ -415,6 +449,7 @@ Every `CERTCTL_*` environment variable is read by the server's `internal/config/
 | `CERTCTL_ACME_CHALLENGE_TYPE` | `http-01`, `dns-01`, or `dns-persist-01` |
 | `CERTCTL_ACME_INSECURE` | Skip TLS verification for ACME CA (test only) |
 | `CERTCTL_ACME_EAB_KID` / `CERTCTL_ACME_EAB_HMAC` | External Account Binding for ZeroSSL, Google Trust Services |
 | `CERTCTL_ZEROSSL_EAB_URL` | Override the ZeroSSL EAB-credentials endpoint (defaults to the public ZeroSSL URL; only set for ZeroSSL staging or a private mirror) |
 | `CERTCTL_ACME_ARI_ENABLED` | Enable RFC 9773 Renewal Information |
 | `CERTCTL_ACME_PROFILE` | ACME profile (`tlsserver`, `shortlived`) |
 | `CERTCTL_STEPCA_URL` | step-ca server URL |
@@ -0,0 +1,38 @@
 #!/usr/bin/env bash
 # deploy/demo-up.sh — boot the certctl demo stack with the fresh
 # CERTCTL_DEMO_MODE_ACK_TS the Phase 2 SEC-H3 guard requires.
 #
 # The demo overlay sets CERTCTL_DEMO_MODE_ACK=true. Phase 2 SEC-H3
 # (2026-05-13) pairs that with a fail-closed requirement: the server
 # refuses to start unless CERTCTL_DEMO_MODE_ACK_TS=<unix-epoch> is set
 # and is within the last 24h (with 1-minute future clock-skew tolerance).
 #
 # A static value in docker-compose.demo.yml would rot the next day, so
 # the overlay passthroughs the value from the shell environment. This
 # helper mints a fresh TS at run time and forwards any extra args to
 # `docker compose up`, so operators can use it as a drop-in replacement
 # for the bare command. Example:
 #
 #     ./demo-up.sh -d                  # cold boot in detached mode
 #     ./demo-up.sh -d --pull always    # forward any flags through
 #
 # The cold-DB compose smoke in .github/workflows/ci.yml does the same
 # thing inline; this script exists so local operators don't have to
 # remember the export.
 set -euo pipefail
 # cd to the deploy/ dir so the relative `-f` paths resolve regardless
 # of where the operator invokes this from. The script lives next to
 # the compose files it references.
 cd "$(dirname "$0")"
 export CERTCTL_DEMO_MODE_ACK_TS="$(date +%s)"
 echo "[demo-up] minting CERTCTL_DEMO_MODE_ACK_TS=$CERTCTL_DEMO_MODE_ACK_TS"
 echo "[demo-up] running: docker compose -f docker-compose.yml -f docker-compose.demo.yml up $*"
 exec docker compose \
  -f docker-compose.yml \
  -f docker-compose.demo.yml \
  up "$@"
@@ -1,26 +1,125 @@
-# Demo mode: pre-populated dashboard with 32 certificates, 8 agents, 10 issuers, etc.
+# =============================================================================
-# Use this to showcase certctl's dashboard with realistic data.
+# certctl DEMO overlay — Bundle 2 (2026-05-12)
 # =============================================================================
 #
-# Usage:
+# Layered on top of the production-shaped base (docker-compose.yml) to give
-#   docker compose -f docker-compose.yml -f docker-compose.demo.yml up --build
+# operators a one-command, zero-config demo path:
 #
-# To start fresh (wipe previous data):
+#   deploy/demo-up.sh -d --build
 #   docker compose -f docker-compose.yml -f docker-compose.demo.yml down -v
 #   docker compose -f docker-compose.yml -f docker-compose.demo.yml up --build
 #
-# U-3 (P1, cat-u-seed_initdb_schema_drift): pre-U-3 this overlay mounted
+# (which forwards args to `docker compose up` after exporting the fresh
-# `seed_demo.sql` into postgres `/docker-entrypoint-initdb.d/`. That worked
+# CERTCTL_DEMO_MODE_ACK_TS that Phase 2 SEC-H3 requires). Equivalent
-# only because the production stack also mounted the migrations there, so
+# manual invocation:
-# the schema existed at initdb time. Once U-3 dropped the production
+#
 #   CERTCTL_DEMO_MODE_ACK_TS=$(date +%s) docker compose \
 #     -f deploy/docker-compose.yml \
 #     -f deploy/docker-compose.demo.yml up -d --build
 #
 # What this overlay does:
 #
 #   1. Flips CERTCTL_AUTH_TYPE=none + CERTCTL_DEMO_MODE_ACK=true. Every
 #      request is served as the synthetic admin actor `actor-demo-anon`;
 #      the server emits a prominent ⚠ DEMO MODE WARN banner at boot with
 #      a production-promotion checklist (cmd/server/main.go::emitDemoBanner).
 #      Phase 2 SEC-H3 (2026-05-13) pairs DEMO_MODE_ACK with a required
 #      DEMO_MODE_ACK_TS within the last 24h. The overlay reads
 #      ${CERTCTL_DEMO_MODE_ACK_TS:-} from the shell — use deploy/demo-up.sh
 #      (which exports a fresh TS) instead of bare `docker compose up`.
 #
 #   2. Flips CERTCTL_KEYGEN_MODE=server (the demo issues + holds the key on
 #      the server to keep the dashboard populated; production deploys must
 #      use the default `agent` mode where keys never leave the agent box).
 #
 #   3. Flips CERTCTL_DEMO_SEED=true. The server applies migrations/seed_demo.sql
 #      at boot via postgres.RunDemoSeed AFTER baseline migrations + seed.sql,
 #      pre-seeding 180 days of simulated history across 13 issuers + 8 agents.
 #
 #   4. Supplies the change-me-... placeholder values for POSTGRES_PASSWORD,
 #      CERTCTL_API_KEY, CERTCTL_CONFIG_ENCRYPTION_KEY, and CERTCTL_AGENT_ID
 #      so the demo runs without a deploy/.env file. The Bundle 2 fail-closed
 #      Validate() rejects these placeholders outside demo mode, so this only
 #      works alongside DEMO_MODE_ACK=true.
 #
 # U-3 history: pre-U-3 this overlay mounted seed_demo.sql into postgres
 # `/docker-entrypoint-initdb.d/`. That worked only because the production
 # stack also mounted the migrations there. Once U-3 dropped the production
 # initdb mounts (single source of truth: server runs RunMigrations + RunSeed
 # at boot), the demo seed could no longer be applied at initdb time — the
-# tables it references wouldn't exist yet.
+# tables it references wouldn't exist yet. Post-U-3 the overlay just sets
 # CERTCTL_DEMO_SEED=true; the server applies seed_demo.sql at boot via
 # postgres.RunDemoSeed AFTER baseline migrations + seed.sql.
 #
-# Post-U-3 the demo overlay just sets CERTCTL_DEMO_SEED=true; the server
+# Bundle 2 history: pre-Bundle-2 the base compose IS this demo path; this
-# applies seed_demo.sql at boot via postgres.RunDemoSeed AFTER baseline
+# overlay was a single-flag thin shim. Bundle 2 split the demo env vars
-# migrations + seed.sql are in place. Same single source of truth, no
+# out of the base so `docker compose -f deploy/docker-compose.yml up`
-# initdb mounts, no schema-vs-seed drift.
+# (no overlay) boots production-shaped — which is what every operator
 # reading the README quickstart line "drop the demo overlay for a clean
 # install" expected. The overlay carries the full demo posture now.
 #
 # To start fresh (wipe previous data):
 #   docker compose -f deploy/docker-compose.yml \
 #                  -f deploy/docker-compose.demo.yml down -v
 #   deploy/demo-up.sh -d --build
 services:
  postgres:
    # Fixed weak password is intentional for the no-setup demo path.
    # See docker-compose.yml for the production override pattern.
    environment:
      POSTGRES_PASSWORD: certctl
  certctl-server:
    environment:
      # Demo-mode auth: every request served as the synthetic
      # `actor-demo-anon` admin. The server's HIGH-12 startup guard
      # requires DEMO_MODE_ACK=true to allow this combination on a
      # non-loopback bind; the boot-time WARN banner (cmd/server/main.go)
      # reminds the operator on every start.
      CERTCTL_AUTH_TYPE: none
      CERTCTL_DEMO_MODE_ACK: "true"
      # Phase 2 SEC-H3 (2026-05-13): DEMO_MODE_ACK=true requires a fresh
      # DEMO_MODE_ACK_TS within the last 24h. The overlay can't hardcode
      # a timestamp (it would rot the next day), so we passthrough from
      # the shell. Operators set this via:
      #     CERTCTL_DEMO_MODE_ACK_TS=$(date +%s) docker compose \
      #       -f docker-compose.yml -f docker-compose.demo.yml up -d
      # The cold-DB smoke + any helper script (deploy/demo-up.sh, when
      # it lands) export this before invoking compose. Empty value
      # fails the SEC-H3 guard with a clear operator-facing error
      # message pointing at this line.
      CERTCTL_DEMO_MODE_ACK_TS: "${CERTCTL_DEMO_MODE_ACK_TS:-}"
      # Server-side keygen so the demo can populate the dashboard with
      # full lifecycle history. Production deploys leave this at the
      # code default `agent` (CertctlAgent generates ECDSA P-256 keys
      # locally and submits CSRs only).
      CERTCTL_KEYGEN_MODE: server
      # Demo creds — the Bundle 2 fail-closed Validate() rejects these
      # sentinels outside demo mode, but DEMO_MODE_ACK=true unlocks them.
      CERTCTL_CONFIG_ENCRYPTION_KEY: change-me-32-char-encryption-key
      CERTCTL_AUTH_SECRET: change-me-in-production
      # Cold-DB smoke fix (2026-05-13): the base compose builds the
      # database URL via compose-level `${POSTGRES_PASSWORD}` interpolation
      # (deploy/docker-compose.yml line ~177), which reads the SHELL env —
      # NOT the postgres service's `environment:` block above (that one
      # feeds the postgres container's initdb only). In a zero-env-var
      # CI run the shell var is blank, producing
      # `postgres://certctl:@postgres:5432/...` and a SCRAM rejection
      # against a database that initdb seeded with password `certctl`.
      # Pinning the full URL here closes the gap: the demo overlay is
      # now fully self-sufficient (matches the file's docstring claim)
      # and the cold-DB smoke passes against a fresh GitHub-runner clone
      # with no .env file or exported shell vars. Production deploys
      # override CERTCTL_DATABASE_URL via the base compose's
      # `${CERTCTL_DATABASE_URL:-...}` default, so this literal is
      # overlay-scoped and never leaks into a production posture.
      CERTCTL_DATABASE_URL: postgres://certctl:certctl@postgres:5432/certctl?sslmode=disable
      # 180-day simulated history seed applied at boot.
      CERTCTL_DEMO_SEED: "true"
  certctl-agent:
    environment:
      # Pre-seeded by migrations/seed_demo.sql; the bundled agent
      # connects with these creds and the demo-mode synthetic admin
      # accepts every request regardless of API key.
      CERTCTL_API_KEY: change-me-in-production
      CERTCTL_AGENT_ID: agent-demo-1
@@ -116,8 +116,11 @@ services:
    networks:
      certctl-test:
        ipv4_address: 10.30.50.2
    # Acquisition-audit SEC-014 closure (Sprint 2, 2026-05-16).
    # Loopback-only host-port bind — the integration-test runner on
    # the host needs reachability, no other interface does.
    ports:
-      - "5432:5432"
+      - "127.0.0.1:5432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U certctl -d certctl"]
      interval: 5s
@@ -261,6 +264,18 @@ services:
      CERTCTL_AUTH_TYPE: api-key
      CERTCTL_AUTH_SECRET: test-key-2026
      # Phase 2 SEC-H1 + Sprint 5 RED-003 closure (2026-05-16): the
      # AgentBootstrapTokenDenyEmpty fail-closed guard refuses to start
      # the server when CERTCTL_AGENT_BOOTSTRAP_TOKEN is empty (the
      # default DENY_EMPTY=true flipped on Sprint 5). Demo stacks
      # bypass the guard via CERTCTL_DEMO_MODE_ACK=true, but this is
      # the e2e TEST stack (production-like auth posture), not a demo
      # stack — set a deterministic placeholder token so the server
      # boots and the vendor-edge integration tests can run. Clearly
      # test-only; do NOT copy to production. Operators set this from
      # `openssl rand -base64 32` per docs/operator/security.md.
      CERTCTL_AGENT_BOOTSTRAP_TOKEN: test-agent-bootstrap-token-deterministic-fixture
      # Key generation — agent-side (production-like)
      CERTCTL_KEYGEN_MODE: agent
@@ -272,6 +287,14 @@ services:
      CERTCTL_ACME_EMAIL: test@certctl.dev
      CERTCTL_ACME_CHALLENGE_TYPE: http-01
      CERTCTL_ACME_INSECURE: "true"
      # Phase 2 SEC-M4 (2026-05-13): CERTCTL_ACME_INSECURE=true requires
      # the paired CERTCTL_ACME_INSECURE_ACK=true; without the ACK the
      # server's Config.Validate() refuses to start. This integration
      # stack uses Pebble's self-signed ACME directory, so disabling
      # TLS verification is correct — but the ACK env var has to be
      # set explicitly so the test posture matches what production
      # operators are blocked from doing accidentally.
      CERTCTL_ACME_INSECURE_ACK: "true"
      # step-ca issuer (iss-stepca)
      CERTCTL_STEPCA_URL: https://step-ca:9000
@@ -1,3 +1,49 @@
 # =============================================================================
 # certctl base compose — PRODUCTION-SHAPED (Bundle 2, 2026-05-12)
 # =============================================================================
 #
 # This base file ships a SAFE-BY-DEFAULT control plane:
 #
 #   - CERTCTL_AUTH_TYPE defaults to api-key (the code default; not overridden
 #     here). The server REFUSES to start with auth=none on a non-loopback
 #     bind unless CERTCTL_DEMO_MODE_ACK=true (Audit 2026-05-10 HIGH-12 +
 #     Bundle 2 closure: see internal/config/config.go::Validate).
 #   - CERTCTL_KEYGEN_MODE defaults to agent (the code default).
 #   - CERTCTL_DEMO_SEED defaults to false (the code default; the 180-day
 #     simulated history seed only runs under the demo overlay).
 #   - Default placeholder credentials (`change-me-...` sentinels) are NOT
 #     interpolated by this compose. The server REFUSES to start when those
 #     placeholder strings reach config (Bundle 2 fail-closed guards) unless
 #     DEMO_MODE_ACK=true. Operators MUST set:
 #         POSTGRES_PASSWORD               (openssl rand -hex 32)
 #         CERTCTL_AUTH_SECRET             (openssl rand -hex 32)
 #         CERTCTL_CONFIG_ENCRYPTION_KEY   (openssl rand -base64 32)
 #         CERTCTL_API_KEY                 (matches CERTCTL_AUTH_SECRET or one
 #                                          of its rotation siblings)
 #         CERTCTL_AGENT_ID                (returned from POST /api/v1/agents)
 #     in deploy/.env or the shell environment. See deploy/.env.example.
 #
 # USAGE
 # -----
 #
 # Production-shaped (this base alone):
 #   docker compose -f deploy/docker-compose.yml up -d
 #
 # Bundled demo (zero-config, populated dashboard, demo-mode auth):
 #   docker compose -f deploy/docker-compose.yml \
 #                  -f deploy/docker-compose.demo.yml up -d
 #
 # The demo overlay (docker-compose.demo.yml) layers in the demo-mode env
 # vars (AUTH_TYPE=none + DEMO_MODE_ACK=true + KEYGEN_MODE=server +
 # DEMO_SEED=true + the change-me placeholder creds). It exists so the
 # `docker compose up` smoke + screenshot path stays one command — but it
 # ALSO carries the operator-visible warning banner the server emits at
 # boot when DEMO_MODE_ACK=true.
 #
 # Pre-Bundle-2 this base file WAS the demo path. The split happened in
 # 2026-05-12; the README quickstart, deploy/ENVIRONMENTS.md, and the
 # cold-DB compose smoke in .github/workflows/ci.yml were updated in the
 # same commit to point at the new layout.
 services:
  # HTTPS-Everywhere Phase 3 — self-signed TLS bootstrap (init container).
  # Generates a CN=certctl-server ECDSA-P256 (SHA-256 signature) cert with
@@ -16,7 +62,13 @@ services:
  # handshake. ECDSA-P256 with SHA-256 is universally supported. See
  # docs/tls.md Pattern 1.
  certctl-tls-init:
-    image: alpine/openssl:latest
+    # DEPL-002 closure (Sprint 3, 2026-05-16): digest-pin so the
    # production-shaped compose has the same supply-chain posture as
    # the certctl Dockerfiles (which CI guards via digest-validity.sh).
    # The :latest tag floats; the digest is captured at the time
    # this comment was written. Bump after running the digest-
    # validity guard to confirm the new digest is still pullable.
    image: alpine/openssl:latest@sha256:41036db23542ed4cc09bc278d8a7e23b3da01690abb4b0e353b1bb87d70520ed
    container_name: certctl-tls-init
    restart: "no"
    entrypoint: /bin/sh
@@ -77,14 +129,34 @@ services:
  # `unhealthy` flap to cascade into certctl-server's `service_healthy`
  # depends_on, blocking the whole stack.
  postgres:
-    image: postgres:16-alpine
+    # DEPL-002 closure (Sprint 3, 2026-05-16): digest-pin matching the
    # alpine/openssl pin above. The `16-alpine` tag is the stable
    # major-version stream; the digest snapshots today's image so a
    # silent upstream rebuild can't slip into a production deploy
    # mid-rollout. Bump alongside dependency reviews.
    image: postgres:16-alpine@sha256:890480b08124ce7f79960a9bb16fe39729aa302bd384bfd7c408fee6c8f7adb7
    container_name: certctl-postgres
    environment:
      POSTGRES_DB: certctl
      POSTGRES_USER: certctl
-      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-certctl}
+      # Bundle 2 closure: no `:-certctl` fallback. Operators MUST set
      # POSTGRES_PASSWORD in deploy/.env or the shell environment. The
      # demo overlay (docker-compose.demo.yml) supplies a fixed weak
      # default for screenshot/demo use; production deploys never
      # depend on that fallback.
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    # Acquisition-audit SEC-014 closure (Sprint 2, 2026-05-16). Bind
    # the published port to 127.0.0.1 ONLY — the certctl-server
    # connection comes in via the `certctl-network` Docker network
    # (the host-port mapping is operator convenience for psql / DB
    # inspection only). Pre-fix, the "5432:5432" form bound on
    # 0.0.0.0, exposing the Postgres TCP listener on every interface
    # of any host that happened to be on a public IP. The loopback
    # bind keeps host-side psql access working while preventing the
    # cross-network exposure landmine for compose deploys that aren't
    # behind a firewall.
    ports:
-      - "5432:5432"
+      - "127.0.0.1:5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
@@ -123,25 +195,44 @@ services:
      # on the docker bridge network keeps sslmode=disable acceptable; for
      # external/managed Postgres operators MUST override CERTCTL_DATABASE_URL
      # with sslmode=verify-full and provide the CA bundle. See docs/database-tls.md.
-      CERTCTL_DATABASE_URL: ${CERTCTL_DATABASE_URL:-postgres://certctl:${POSTGRES_PASSWORD:-certctl}@postgres:5432/certctl?sslmode=disable}
+      CERTCTL_DATABASE_URL: ${CERTCTL_DATABASE_URL:-postgres://certctl:${POSTGRES_PASSWORD}@postgres:5432/certctl?sslmode=disable}
      CERTCTL_SERVER_HOST: 0.0.0.0
      CERTCTL_SERVER_PORT: 8443
      CERTCTL_SERVER_TLS_CERT_PATH: /etc/certctl/tls/server.crt
      CERTCTL_SERVER_TLS_KEY_PATH: /etc/certctl/tls/server.key
      CERTCTL_LOG_LEVEL: info
-      CERTCTL_AUTH_TYPE: none
+      # Bundle 2 closure (compose split). The base compose no longer
-      CERTCTL_KEYGEN_MODE: server  # Demo uses server-side keygen; production should use "agent"
+      # sets CERTCTL_AUTH_TYPE / CERTCTL_KEYGEN_MODE / DEMO_MODE_ACK /
-      CERTCTL_NETWORK_SCAN_ENABLED: "true"  # Enable network scan GUI with seeded demo targets
+      # DEMO_SEED — the code defaults take over (auth-type api-key,
-      CERTCTL_CONFIG_ENCRYPTION_KEY: ${CERTCTL_CONFIG_ENCRYPTION_KEY:-change-me-32-char-encryption-key}  # AES-256-GCM for dynamic issuer/target config
+      # keygen agent, demo-mode false, demo-seed false). The demo
-      # Bundle 1 follow-on: this compose IS the bundled demo path
+      # overlay (docker-compose.demo.yml) is what flips this baseline
-      # (CERTCTL_AUTH_TYPE=none + KEYGEN_MODE=server above), so the
+      # into the populated-dashboard demo path; without that overlay
-      # demo seed runs by default. seed_demo.sql pre-seeds the
+      # the server boots production-shaped and refuses to start unless
-      # agent-demo-1 row that the bundled certctl-agent below needs
+      # the operator has supplied CERTCTL_AUTH_SECRET +
-      # to authenticate. The docker-compose.demo.yml overlay still
+      # CERTCTL_CONFIG_ENCRYPTION_KEY.
-      # works (it sets the same flag) and remains for backward
+      #
-      # compat. Production deploys override CERTCTL_AUTH_TYPE +
+      # Audit 2026-05-10 HIGH-12: when DEMO_MODE_ACK=true (set by the
-      # KEYGEN_MODE + DEMO_SEED via their own compose.
+      # demo overlay) AND the listener binds to a non-loopback address,
-      CERTCTL_DEMO_SEED: "true"
+      # every request is served as the synthetic admin actor
      # `actor-demo-anon`. The server emits a prominent boot-time WARN
      # banner with a production-promotion checklist in that case.
      CERTCTL_AUTH_SECRET: ${CERTCTL_AUTH_SECRET}
      CERTCTL_NETWORK_SCAN_ENABLED: "true"  # Enable network scan GUI
      CERTCTL_CONFIG_ENCRYPTION_KEY: ${CERTCTL_CONFIG_ENCRYPTION_KEY}  # AES-256-GCM for dynamic issuer/target config
      # Bootstrap token interpolation surface (Auditable Codebase Bundle
      # cold-DB smoke closure, 2026-05-12). Pre-fix, the `env-file +
      # --force-recreate certctl-server` pattern documented in
      # cowork/manual-testing-bundle-2.html (and used by the cold-DB
      # smoke job in .github/workflows/ci.yml::cold-db-compose-smoke)
      # set CERTCTL_BOOTSTRAP_TOKEN in compose's own interpolation
      # environment but the container never received it because this
      # block didn't reference the variable. Wiring it as an explicit
      # interpolation (default empty) makes the documented manual flow
      # actually work end-to-end. Empty value = bootstrap strategy
      # disabled (server returns 410 Gone on POST /api/v1/auth/bootstrap),
      # which is the safe default — only set the var when you intend to
      # mint a day-0 admin via the bootstrap path.
      CERTCTL_BOOTSTRAP_TOKEN: ${CERTCTL_BOOTSTRAP_TOKEN:-}
    ports:
      - "8443:8443"
    volumes:
@@ -191,18 +282,19 @@ services:
    environment:
      CERTCTL_SERVER_URL: https://certctl-server:8443
      CERTCTL_SERVER_CA_BUNDLE_PATH: /etc/certctl/tls/ca.crt
-      CERTCTL_API_KEY: ${CERTCTL_API_KEY:-change-me-in-production}
+      # Bundle 2 closure (compose split). No placeholder fallbacks.
-      # Bundle 1 follow-on: pre-Bundle-1 the bundled agent had no
+      # Operators MUST set CERTCTL_API_KEY (matching one of the server's
-      # CERTCTL_AGENT_ID set, hit cmd/agent/main.go's fail-fast guard
+      # CERTCTL_AUTH_SECRET rotation values) and CERTCTL_AGENT_ID
-      # ("agent-id flag or CERTCTL_AGENT_ID env var is required"), and
+      # (returned from `POST /api/v1/agents` during agent enrollment).
-      # restart-looped silently on every fresh `docker compose up`.
+      # Without an agent ID, cmd/agent/main.go fails fast at startup
-      # Latent since 2026-03-14 (commit d395776). seed_demo.sql now
+      # with "agent-id flag or CERTCTL_AGENT_ID env var is required" —
-      # pre-seeds the matching agents row; the demo runs with
+      # the cold-DB compose smoke in .github/workflows/ci.yml tolerates
-      # CERTCTL_AUTH_TYPE=none on the server so the api_key Bearer
+      # the agent restart loop because the smoke targets server boot
-      # token is irrelevant here. Production deploys override
+      # only. The demo overlay (docker-compose.demo.yml) supplies a
-      # CERTCTL_AGENT_ID with the value returned from
+      # pre-seeded agent-demo-1 row + matching env vars so the demo
-      # POST /api/v1/agents during registration.
+      # path stays one-command.
-      CERTCTL_AGENT_ID: ${CERTCTL_AGENT_ID:-agent-demo-1}
+      CERTCTL_API_KEY: ${CERTCTL_API_KEY}
      CERTCTL_AGENT_ID: ${CERTCTL_AGENT_ID}
      CERTCTL_AGENT_NAME: docker-agent
      CERTCTL_LOG_LEVEL: info
      CERTCTL_DISCOVERY_DIRS: /var/lib/certctl/keys  # Agent scans this directory for existing certificates
@@ -2,7 +2,15 @@ apiVersion: v2
 name: certctl
 description: Self-hosted certificate lifecycle management platform
 type: application
-version: 0.1.0
+# Bundle 3 closure (OPS-L1): bumped from 0.1.0 → 1.0.0. The pre-1.0
 # version implied "unstable chart, breaking changes on every minor"
 # which prospective enterprise operators read as "not ready for
 # production". The chart has been deployed against real clusters since
 # 2026-02 and shipped through 8 audit closures (M-018, U-1, U-2, U-3,
 # H-1, G-1, B1 connector validation, B2 first-run guards); 1.0.0
 # matches that maturity. The chart still adheres to semver going
 # forward — any breaking value-schema change bumps to 2.0.0.
 version: 1.0.0
 appVersion: "2.1.0"
 keywords:
  - certificate
@@ -72,3 +72,28 @@ IMPORTANT NOTES FOR PRODUCTION:
   - All containers run as non-root
   - Implement network policies to restrict traffic between components
   - Consider pod security policies or security standards for your cluster
 {{- /*
  DEPL-006 closure (Sprint 3, 2026-05-16). Loud notice when the
  operator runs a multi-replica deploy without crossing the two
  required HA toggles. Per-pod rate-limit buckets and round-robin
  load balancing both silently break correctness above replicas:1.
 */}}
 {{- if gt (int .Values.server.replicas) 1 }}
 ⚠️  HA MISCONFIGURATION WARNINGS (replicas={{ .Values.server.replicas }}):
 {{- $backend := .Values.server.rateLimiting.backend | default "memory" }}
 {{- if eq $backend "memory" }}
   - server.rateLimiting.backend = "memory" with replicas > 1 gives each
     pod its own bucket map, so the configured cap is effectively
     multiplied by the replica count. Set
     `--set server.rateLimiting.backend=postgres` (see DEPL-006 /
     docs/operator/runbooks/ha.md).
 {{- end }}
 {{- if not .Values.server.service.sessionAffinity }}
   - server.service.sessionAffinity is empty. Round-robin Service load
     balancing routes login → /api/v1/auth/login → /api/v1/auth/csrf
     across different pods, breaking the CSRF token + session cookie
     handshake. Set
     `--set server.service.sessionAffinity=ClientIP`.
 {{- end }}
 {{- end }}
@@ -128,8 +128,27 @@ Bundle B / Audit M-018 (PCI-DSS Req 4 / CWE-319):
    postgresql.tls.mode without further translation.
 */}}
 {{- define "certctl.databaseURL" -}}
 {{- if .Values.postgresql.enabled -}}
 {{- $sslMode := default "disable" .Values.postgresql.tls.mode -}}
 postgres://{{ .Values.postgresql.auth.username }}:$(POSTGRES_PASSWORD)@{{ include "certctl.fullname" . }}-postgres:5432/{{ .Values.postgresql.auth.database }}?sslmode={{ $sslMode }}
 {{- else -}}
 {{- /*
  Bundle 3 closure (D2 + OPS-L2): external-Postgres first-class path.
  When postgresql.enabled=false, the chart NEVER renders the
  bundled StatefulSet, postgres-secret, or postgres-service —
  templates/postgres-*.yaml gate themselves on .Values.postgresql.enabled.
  The connection string comes from externalDatabase.url (the canonical
  form) or, for backward-compat with pre-Bundle-3 deploys, from
  server.env.CERTCTL_DATABASE_URL (which overrides this helper at the
  pod-spec level — see server-deployment.yaml).
  externalDatabase.url is consumed VERBATIM by the server's
  CERTCTL_DATABASE_URL env var. Operators are responsible for choosing
  the right sslmode (`verify-full` recommended for managed Postgres
  per PCI-DSS Req 4 §2.2.5; see docs/database-tls.md).
 */ -}}
 {{- required "externalDatabase.url is required when postgresql.enabled=false" .Values.externalDatabase.url -}}
 {{- end -}}
 {{- end }}
 {{/*
@@ -180,11 +199,110 @@ per affected resource. No-op when configured correctly.
 {{- if and (not .Values.server.tls.existingSecret) (not .Values.server.tls.certManager.enabled) -}}
 {{- fail "\n\ncertctl refuses to start without TLS.\n\nSet EXACTLY ONE of:\n  --set server.tls.existingSecret=<your-kubernetes.io/tls-secret-name>\nOR\n  --set server.tls.certManager.enabled=true \\\n  --set server.tls.certManager.issuerRef.name=<your-issuer-or-clusterissuer>\n\nSee docs/tls.md for the full setup walkthrough, including bootstrap\nguidance for air-gapped clusters without cert-manager.\n" -}}
 {{- end -}}
 {{- if and .Values.server.tls.existingSecret .Values.server.tls.certManager.enabled -}}
 {{- /*
  Bundle 3 closure (D7): pre-Bundle-3 the helper only rejected the
  NEITHER-set case. Setting BOTH (`existingSecret` AND `certManager.enabled=true`)
  produced two TLS sources of truth — the existing Secret got mounted but
  cert-manager simultaneously provisioned a Certificate CR pointing at a
  conflicting Secret. Operators ended up with a dangling cert-manager
  Certificate or a wrong-source TLS bundle. The chart now refuses at
  render-time so the misconfiguration cannot ship.
 */ -}}
 {{- fail "\n\nserver.tls.existingSecret AND server.tls.certManager.enabled are BOTH set.\n\nThe chart requires EXACTLY ONE TLS ownership path (Bundle 3 closure / audit D7):\n  - existingSecret: operator owns the TLS Secret; cert-manager must NOT provision one.\n  - certManager.enabled: cert-manager owns the TLS Secret; existingSecret must be empty.\n\nUnset one of:\n  --set server.tls.existingSecret=\"\"          (let cert-manager own it)\nOR\n  --set server.tls.certManager.enabled=false   (let the existing Secret stand)\n\nSee docs/tls.md.\n" -}}
 {{- end -}}
 {{- if and .Values.server.tls.certManager.enabled (not .Values.server.tls.certManager.issuerRef.name) -}}
 {{- fail "\n\nserver.tls.certManager.enabled=true but server.tls.certManager.issuerRef.name is empty.\n\nSet:\n  --set server.tls.certManager.issuerRef.name=<your-issuer-or-clusterissuer>\n\nSee docs/tls.md.\n" -}}
 {{- end -}}
 {{- end }}
 {{/*
 Pod- vs container-scope security context split (Bundle 3 closure / audit D3).
 The Kubernetes API splits SecurityContext into two non-overlapping
 field sets, and silently DROPS fields that land at the wrong scope —
 which is exactly the audit D3 finding pre-Bundle-3.
 Pod-scope fields (applied via spec.securityContext):
  runAsNonRoot, runAsUser, runAsGroup, fsGroup, fsGroupChangePolicy,
  supplementalGroups, seLinuxOptions, seccompProfile, sysctls.
 Container-scope fields (applied via spec.containers[].securityContext):
  readOnlyRootFilesystem, allowPrivilegeEscalation, capabilities,
  privileged, procMount, runAsNonRoot/runAsUser/runAsGroup (override),
  seLinuxOptions/seccompProfile (override).
 These helpers split a single operator-facing `securityContext` map
 into the two sub-maps so the chart renders each field at the scope
 where Kubernetes actually honors it. The split is conservative — a
 field that COULD live at either scope is rendered at pod scope only
 (no override at container scope) so behavior matches the pre-Bundle-3
 operator intent: pod-level setting is the source of truth.
 Operators don't need to change values.yaml; the existing
 `server.securityContext` and `agent.securityContext` blocks keep
 working byte-for-byte. The Helm template just routes each field to
 the correct YAML node now.
 */}}
 {{- define "certctl.podSecurityContext" -}}
 {{- $sc := . -}}
 {{- $podKeys := list "runAsNonRoot" "runAsUser" "runAsGroup" "fsGroup" "fsGroupChangePolicy" "supplementalGroups" "seLinuxOptions" "seccompProfile" "sysctls" -}}
 {{- $out := dict -}}
 {{- range $k := $podKeys -}}
 {{- if hasKey $sc $k -}}
 {{- $_ := set $out $k (index $sc $k) -}}
 {{- end -}}
 {{- end -}}
 {{- toYaml $out -}}
 {{- end }}
 {{- define "certctl.containerSecurityContext" -}}
 {{- $sc := . -}}
 {{- $containerKeys := list "readOnlyRootFilesystem" "allowPrivilegeEscalation" "capabilities" "privileged" "procMount" -}}
 {{- $out := dict -}}
 {{- range $k := $containerKeys -}}
 {{- if hasKey $sc $k -}}
 {{- $_ := set $out $k (index $sc $k) -}}
 {{- end -}}
 {{- end -}}
 {{- toYaml $out -}}
 {{- end }}
 {{/*
 Required-secret gate (Bundle 3 closure / audit D1).
 Pre-Bundle-3 the chart accepted empty `server.auth.apiKey` and empty
 `postgresql.auth.password` and rendered Secrets with empty values; the
 certctl-server container then crash-looped at startup with the auth
 configuration error or with `pq: password authentication failed for
 user "certctl"`. Worse, an operator who forgot to set the api-key
 ended up with auth.type=api-key + empty CERTCTL_AUTH_SECRET in the
 Secret, which Validate() rejects at startup — but the diagnostic
 surfaces inside a CrashLoopBackOff, not at `helm install` time where
 it would be caught immediately.
 Post-Bundle-3 the chart fails at template time with operator-actionable
 guidance. The bundled-Postgres path (`postgresql.enabled=true`)
 requires `postgresql.auth.password`; the external-Postgres path
 (`postgresql.enabled=false`) skips that check because credentials are
 embedded in `externalDatabase.url` instead.
 Any template that depends on either secret value should call
 `{{ include "certctl.requiredSecrets" . }}` at the top so this guard
 runs once per affected resource. No-op when configured correctly.
 */}}
 {{- define "certctl.requiredSecrets" -}}
 {{- if and (eq .Values.server.auth.type "api-key") (not .Values.server.auth.apiKey) -}}
 {{- fail "\n\nserver.auth.type=\"api-key\" but server.auth.apiKey is empty.\n\nSet:\n  --set server.auth.apiKey=$(openssl rand -base64 32)\n\nor put the value in a values override. The certctl-server container\nrefuses to start without an API key when auth.type=api-key.\n\nFor demo deploys without authentication, use:\n  --set server.auth.type=none\n(only safe behind an authenticating gateway — see docs/operator/security.md).\n" -}}
 {{- end -}}
 {{- if and .Values.postgresql.enabled (not .Values.postgresql.auth.password) -}}
 {{- fail "\n\npostgresql.enabled=true but postgresql.auth.password is empty.\n\nSet:\n  --set postgresql.auth.password=$(openssl rand -base64 32)\n\nor put the value in a values override. The bundled Postgres\nStatefulSet refuses to bootstrap initdb without POSTGRES_PASSWORD.\n\nFor external Postgres deployments, set:\n  --set postgresql.enabled=false\n  --set externalDatabase.url=postgres://user:pass@host:5432/db?sslmode=require\nSee deploy/helm/examples/values-external-db.yaml.\n" -}}
 {{- end -}}
 {{- if and (not .Values.postgresql.enabled) (not .Values.externalDatabase.url) (not .Values.server.env.CERTCTL_DATABASE_URL) -}}
 {{- fail "\n\npostgresql.enabled=false but no external database URL is configured.\n\nSet ONE of:\n  --set externalDatabase.url=postgres://user:pass@host:5432/db?sslmode=require\nOR (legacy)\n  --set server.env.CERTCTL_DATABASE_URL=postgres://user:pass@host:5432/db?sslmode=require\n\nSee deploy/helm/examples/values-external-db.yaml.\n" -}}
 {{- end -}}
 {{- end }}
 {{/*
 Auth-type validation gate.
@@ -19,7 +19,7 @@ spec:
    spec:
      serviceAccountName: {{ include "certctl.serviceAccountName" . }}
      securityContext:
-        {{- toYaml .Values.agent.securityContext | nindent 8 }}
+        {{- include "certctl.podSecurityContext" .Values.agent.securityContext | nindent 8 }}
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
@@ -40,6 +40,8 @@ spec:
        - name: agent
          image: {{ include "certctl.agentImage" . }}
          imagePullPolicy: {{ .Values.agent.image.pullPolicy }}
          securityContext:
            {{- include "certctl.containerSecurityContext" .Values.agent.securityContext | nindent 12 }}
          env:
            - name: CERTCTL_SERVER_URL
              value: {{ include "certctl.serverURL" . }}
@@ -106,7 +108,7 @@ spec:
    spec:
      serviceAccountName: {{ include "certctl.serviceAccountName" . }}
      securityContext:
-        {{- toYaml .Values.agent.securityContext | nindent 8 }}
+        {{- include "certctl.podSecurityContext" .Values.agent.securityContext | nindent 8 }}
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
@@ -127,6 +129,8 @@ spec:
        - name: agent
          image: {{ include "certctl.agentImage" . }}
          imagePullPolicy: {{ .Values.agent.image.pullPolicy }}
          securityContext:
            {{- include "certctl.containerSecurityContext" .Values.agent.securityContext | nindent 12 }}
          env:
            - name: CERTCTL_SERVER_URL
              value: {{ include "certctl.serverURL" . }}
@@ -0,0 +1,178 @@
 {{- /*
 Phase 4 DEPL-H2 closure (2026-05-14): opt-in Helm CronJob for
 PostgreSQL backups.
 OPERATOR OPT-IN. Default `backup.enabled: false`. Turning it on
 requires:
  - In-cluster Postgres (this CronJob does NOT cover managed DB
    services — for AWS RDS / GCP CloudSQL / Azure DB rely on the
    provider's PITR).
  - A sink choice (PVC or S3) configured in values.yaml.
  - For S3: a Secret holding AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY
    (or use a service account with IRSA on EKS).
 The pg_dump invocation matches the canonical shape documented in
 docs/operator/runbooks/postgres-backup.md so a manual run and a
 CronJob run produce byte-identical dumps:
  pg_dump --format=custom --no-owner --no-acl --dbname=certctl
 For sink choices beyond PVC + S3 (GCS, Azure Blob, NFS, restic, etc.),
 extend the `aws s3 cp` line below. The Job is intentionally minimal —
 it does ONE thing (capture + ship), not orchestrate retention or
 rotation. Off-host retention is the sink's responsibility (S3 lifecycle
 rules, PVC snapshot retention on the storage class, etc.).
 */ -}}
 {{- if .Values.backup.enabled }}
 apiVersion: batch/v1
 kind: CronJob
 metadata:
  name: {{ include "certctl.fullname" . }}-postgres-backup
  labels:
    {{- include "certctl.labels" . | nindent 4 }}
    app.kubernetes.io/component: postgres-backup
 spec:
  schedule: {{ .Values.backup.schedule | quote }}
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: {{ .Values.backup.successfulJobsHistoryLimit | default 3 }}
  failedJobsHistoryLimit: {{ .Values.backup.failedJobsHistoryLimit | default 1 }}
  startingDeadlineSeconds: {{ .Values.backup.startingDeadlineSeconds | default 300 }}
  jobTemplate:
    spec:
      backoffLimit: {{ .Values.backup.backoffLimit | default 1 }}
      activeDeadlineSeconds: {{ .Values.backup.activeDeadlineSeconds | default 3600 }}
      template:
        metadata:
          labels:
            {{- include "certctl.labels" . | nindent 12 }}
            app.kubernetes.io/component: postgres-backup
        spec:
          restartPolicy: Never
          {{- with .Values.imagePullSecrets }}
          imagePullSecrets:
            {{- toYaml . | nindent 12 }}
          {{- end }}
          serviceAccountName: {{ include "certctl.serviceAccountName" . }}
          securityContext:
            runAsUser: 1000
            runAsGroup: 1000
            runAsNonRoot: true
            fsGroup: 1000
          containers:
            - name: backup
              image: {{ .Values.backup.image | default "postgres:16-alpine" | quote }}
              imagePullPolicy: {{ .Values.backup.imagePullPolicy | default "IfNotPresent" | quote }}
              env:
                - name: PGHOST
                  value: {{ include "certctl.fullname" . }}-postgres
                - name: PGPORT
                  value: {{ .Values.postgresql.service.port | default 5432 | quote }}
                - name: PGUSER
                  valueFrom:
                    secretKeyRef:
                      name: {{ include "certctl.fullname" . }}-postgres
                      key: username
                - name: PGPASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: {{ include "certctl.fullname" . }}-postgres
                      key: password
                - name: PGDATABASE
                  valueFrom:
                    secretKeyRef:
                      name: {{ include "certctl.fullname" . }}-postgres
                      key: database
                {{- if eq (.Values.backup.sink | default "pvc") "s3" }}
                # S3 sink — operator provides AWS credentials via the
                # Secret referenced in backup.s3.credentialsSecret. The
                # credentials need s3:PutObject + s3:ListBucket on the
                # target bucket only; least-privilege per industry
                # standard.
                - name: AWS_ACCESS_KEY_ID
                  valueFrom:
                    secretKeyRef:
                      name: {{ .Values.backup.s3.credentialsSecret.name | quote }}
                      key: {{ .Values.backup.s3.credentialsSecret.accessKeyIdKey | default "AWS_ACCESS_KEY_ID" }}
                - name: AWS_SECRET_ACCESS_KEY
                  valueFrom:
                    secretKeyRef:
                      name: {{ .Values.backup.s3.credentialsSecret.name | quote }}
                      key: {{ .Values.backup.s3.credentialsSecret.secretAccessKeyKey | default "AWS_SECRET_ACCESS_KEY" }}
                {{- with .Values.backup.s3.region }}
                - name: AWS_DEFAULT_REGION
                  value: {{ . | quote }}
                {{- end }}
                {{- end }}
              command:
                - /bin/sh
                - -ceu
                - |
                  # Phase 4 DEPL-H2: canonical pg_dump shape per
                  # docs/operator/runbooks/postgres-backup.md.
                  # Custom-format compressed dump, no ownership /
                  # ACL embedded — produces a portable artifact
                  # restorable into any Postgres ≥ source major
                  # via `pg_restore -d certctl <dump>`.
                  set -euo pipefail
                  TIMESTAMP="$(date -u +%Y%m%dT%H%M%SZ)"
                  DUMP_FILE="/tmp/certctl-${TIMESTAMP}.dump"
                  echo "[backup-cronjob] capturing dump at ${TIMESTAMP}"
                  pg_dump --format=custom --no-owner --no-acl --dbname="${PGDATABASE}" \
                    > "${DUMP_FILE}"
                  # Integrity check — pg_restore --list parses the
                  # dump's table-of-contents; a corrupt dump fails
                  # here without shipping garbage off-host. Same
                  # check the manual runbook performs.
                  echo "[backup-cronjob] verifying dump integrity"
                  pg_restore --list "${DUMP_FILE}" > /dev/null
                  {{- if eq (.Values.backup.sink | default "pvc") "s3" }}
                  # S3 sink — requires aws-cli. The default
                  # postgres:16-alpine image does NOT include
                  # aws-cli; operators MUST set
                  # backup.image to an image that bundles both
                  # (e.g. ghcr.io/your-org/postgres-aws:16) OR
                  # override backup.command to install aws-cli at
                  # runtime. The line below assumes the image has
                  # `aws` on PATH.
                  S3_PATH="{{ .Values.backup.s3.bucket }}/{{ .Values.backup.s3.prefix | default "certctl" }}/certctl-${TIMESTAMP}.dump"
                  echo "[backup-cronjob] uploading to s3://${S3_PATH}"
                  aws s3 cp "${DUMP_FILE}" "s3://${S3_PATH}"
                  rm -f "${DUMP_FILE}"
                  {{- else }}
                  # PVC sink — dump lands at /backups/certctl-${TIMESTAMP}.dump
                  # mounted from backup.pvc.claimName. Retention is the
                  # PVC's responsibility (storage-class snapshot lifecycle
                  # or a separate cleanup CronJob). The Job moves the
                  # file from /tmp to /backups atomically; never
                  # writes partial dumps into the durable mount.
                  FINAL_PATH="/backups/certctl-${TIMESTAMP}.dump"
                  echo "[backup-cronjob] persisting to ${FINAL_PATH}"
                  mv "${DUMP_FILE}" "${FINAL_PATH}"
                  {{- end }}
                  echo "[backup-cronjob] done"
              {{- if ne (.Values.backup.sink | default "pvc") "s3" }}
              volumeMounts:
                - name: backups
                  mountPath: /backups
              {{- end }}
              resources:
                {{- toYaml (.Values.backup.resources | default dict) | nindent 16 }}
          {{- if ne (.Values.backup.sink | default "pvc") "s3" }}
          volumes:
            - name: backups
              persistentVolumeClaim:
                claimName: {{ .Values.backup.pvc.claimName | quote }}
          {{- end }}
          {{- with .Values.nodeAffinity }}
          affinity:
            nodeAffinity:
              {{- toYaml . | nindent 14 }}
          {{- end }}
          {{- with .Values.backup.tolerations }}
          tolerations:
            {{- toYaml . | nindent 12 }}
          {{- end }}
 {{- end }}
@@ -0,0 +1,89 @@
 {{- /*
 Phase 4 DEPL-M1 closure (2026-05-14): Helm pre-install / pre-upgrade
 hook that runs Postgres migrations before the server Deployment rolls.
 Pre-DEPL-M1, postgres.RunMigrations was invoked at server boot
 (cmd/server/main.go:151) as the only migration path. That works for
 Compose deployments but conflicts with Kubernetes rolling deploys:
 when a new server image lands with a schema change, multiple replicas
 race the migration during the rollout. The hook resolves the race by
 running migrations OUT OF BAND, exactly once, before any new server
 pod starts.
 How it works:
  - The Job ships the same certctl-server image as the Deployment, so
    the migration code path is binary-identical to the boot-time path.
  - It runs `certctl-server --migrate-only` (a flag the cmd/server
    main process must support — see cmd/server/main.go for the flag
    parse + early-exit path).
  - The CERTCTL_MIGRATIONS_VIA_HOOK=true env var is ALSO set on the
    server Deployment (via values.yaml). When the server boots, it
    sees this env var and skips its own RunMigrations call — the
    hook already did the work. Compose deploys don't set the env
    var, so they keep the boot-time path unchanged.
  - hook-delete-policy hook-succeeded means the Job is cleaned up
    automatically on success but retained on failure for operator
    diagnosis.
  - The hook-weight ensures the migration Job runs before any other
    pre-install/pre-upgrade resources (the StatefulSet's PVC has to
    exist first; in practice the StatefulSet has no hook so it lands
    naturally in the install phase after the Job completes).
 Operators on Compose: this hook is a no-op for you. The server still
 runs migrations at boot per the existing path.
 */ -}}
 {{- if .Values.migrations.viaHook }}
 apiVersion: batch/v1
 kind: Job
 metadata:
  name: {{ include "certctl.fullname" . }}-migrate
  labels:
    {{- include "certctl.labels" . | nindent 4 }}
    app.kubernetes.io/component: migration
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "-5"
    "helm.sh/hook-delete-policy": hook-succeeded,before-hook-creation
 spec:
  backoffLimit: {{ .Values.migrations.backoffLimit | default 1 }}
  activeDeadlineSeconds: {{ .Values.migrations.activeDeadlineSeconds | default 600 }}
  template:
    metadata:
      labels:
        {{- include "certctl.labels" . | nindent 8 }}
        app.kubernetes.io/component: migration
    spec:
      restartPolicy: Never
      serviceAccountName: {{ include "certctl.serviceAccountName" . }}
      securityContext:
        {{- include "certctl.podSecurityContext" .Values.server.securityContext | nindent 8 }}
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      containers:
        - name: migrate
          image: {{ include "certctl.serverImage" . }}
          imagePullPolicy: {{ .Values.server.image.pullPolicy }}
          # Migration-only entrypoint. The server binary supports a
          # --migrate-only flag that runs postgres.RunMigrations +
          # postgres.RunSeed and exits cleanly (zero on success,
          # non-zero on migration failure). See cmd/server/main.go
          # for the implementation. The flag is hermetic — no HTTP
          # listener starts, no scheduler ticks, no signing
          # operations occur. Pure schema-mutation pass.
          command:
            - /app/server
            - --migrate-only
          env:
            - name: CERTCTL_DATABASE_URL
              value: {{ include "certctl.databaseURL" . | quote }}
            - name: CERTCTL_LOG_LEVEL
              value: {{ .Values.server.logging.level | default "info" | quote }}
            - name: CERTCTL_LOG_FORMAT
              value: {{ .Values.server.logging.format | default "json" | quote }}
          resources:
            {{- toYaml (.Values.migrations.resources | default .Values.server.resources) | nindent 12 }}
          securityContext:
            {{- include "certctl.containerSecurityContext" .Values.server.securityContext | nindent 12 }}
 {{- end }}
@@ -0,0 +1,75 @@
 {{- /*
 Bundle 3 closure (D11): NetworkPolicy for the server Deployment.
 Pre-Bundle-3 the chart had no NetworkPolicy template at all — the
 audit-D11 "documented placeholder" finding referred to docs claiming
 deny-by-default network isolation that the rendered chart did not
 provide. Closed.
 This template emits a single NetworkPolicy that, when enabled,
 restricts the certctl-server Pod to:
  - Ingress  : from any agent Pod in the same namespace (selector
               match on app.kubernetes.io/component=agent) on the
               server port, plus optional operator-supplied
               additional from clauses (.networkPolicy.extraIngress).
  - Egress   : to the postgres Pod (when postgresql.enabled=true),
               53/UDP+TCP for kube-dns, and operator-supplied
               additional to clauses for outbound CA / OIDC / SMTP
               (.networkPolicy.extraEgress).
 Default off so existing deploys don't suddenly lose network reach.
 Operators opt in once they've mapped their actual egress surface.
 */ -}}
 {{- if .Values.networkPolicy.enabled }}
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
 metadata:
  name: {{ include "certctl.fullname" . }}-server
  labels:
    {{- include "certctl.labels" . | nindent 4 }}
    app.kubernetes.io/component: server
 spec:
  podSelector:
    matchLabels:
      {{- include "certctl.serverSelectorLabels" . | nindent 6 }}
  policyTypes:
    - Ingress
    - Egress
  ingress:
    # Allow in-cluster agent Pods to reach the server's HTTPS port.
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: {{ include "certctl.name" . }}
              app.kubernetes.io/component: agent
      ports:
        - protocol: TCP
          port: {{ .Values.server.port }}
    {{- with .Values.networkPolicy.extraIngress }}
    {{- toYaml . | nindent 4 }}
    {{- end }}
  egress:
    # Kube-DNS (53/UDP + 53/TCP). Required for any in-cluster name
    # resolution (postgres-service, OIDC issuer hostnames, ACME).
    - to:
        - namespaceSelector: {}
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
    {{- if .Values.postgresql.enabled }}
    # Bundled-Postgres egress.
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: {{ include "certctl.name" . }}
              app.kubernetes.io/component: postgres
      ports:
        - protocol: TCP
          port: 5432
    {{- end }}
    {{- with .Values.networkPolicy.extraEgress }}
    {{- toYaml . | nindent 4 }}
    {{- end }}
 {{- end }}
@@ -0,0 +1,31 @@
 {{- /*
 Bundle 3 closure (D11): PodDisruptionBudget for the server Deployment.
 Pre-Bundle-3 values.yaml carried `podDisruptionBudget.enabled` +
 `minAvailable` + `maxUnavailable` knobs but no template consumed
 them. Audit D11 closed.
 The PDB only renders when server.replicas > 1 — a single-replica
 deployment can't satisfy minAvailable=1 during voluntary disruption
 anyway (the K8s scheduler would refuse to drain the node). Operators
 running 2+ replicas get the PDB; operators running a single replica
 get a templated-out NOTES line reminding them to bump replicas first.
 */ -}}
 {{- if and .Values.podDisruptionBudget.enabled (gt (int .Values.server.replicas) 1) }}
 apiVersion: policy/v1
 kind: PodDisruptionBudget
 metadata:
  name: {{ include "certctl.fullname" . }}-server
  labels:
    {{- include "certctl.labels" . | nindent 4 }}
    app.kubernetes.io/component: server
 spec:
  selector:
    matchLabels:
      {{- include "certctl.serverSelectorLabels" . | nindent 6 }}
  {{- if .Values.podDisruptionBudget.minAvailable }}
  minAvailable: {{ .Values.podDisruptionBudget.minAvailable }}
  {{- else if .Values.podDisruptionBudget.maxUnavailable }}
  maxUnavailable: {{ .Values.podDisruptionBudget.maxUnavailable }}
  {{- end }}
 {{- end }}
@@ -1,3 +1,14 @@
 {{- if .Values.postgresql.enabled }}
 {{- /*
  Bundle 3 closure (D1 + D2): the bundled-Postgres Secret only renders
  when postgresql.enabled=true. Pre-Bundle-3 this template rendered
  unconditionally with `password: "changeme"` as the fallback default —
  which is exactly what the change-me-... cluster of audit findings
  was about (a deployment that uses the rendered chart with default
  values ships a known weak password). The Bundle-3 helper at
  certctl.requiredSecrets fail-closes empty password at template time
  before this template ever runs.
 */ -}}
 apiVersion: v1
 kind: Secret
 metadata:
@@ -7,6 +18,7 @@ metadata:
    app.kubernetes.io/component: postgres
 type: Opaque
 stringData:
-  password: {{ .Values.postgresql.auth.password | default "changeme" | quote }}
+  password: {{ required "postgresql.auth.password is required when postgresql.enabled=true (Bundle 3: no fallback default)" .Values.postgresql.auth.password | quote }}
  username: {{ .Values.postgresql.auth.username | quote }}
  database: {{ .Values.postgresql.auth.database | quote }}
 {{- end }}
@@ -9,6 +9,21 @@ metadata:
 spec:
  serviceName: {{ include "certctl.fullname" . }}-postgres
  replicas: 1
  # Phase 4 DEPL-M4 closure (2026-05-14): explicit StatefulSet update +
  # pod-management strategies. Defaults make Postgres upgrades
  # operator-controlled rather than automatic:
  #   updateStrategy.type: OnDelete — Postgres pods do NOT roll
  #     automatically when the StatefulSet spec changes. Operator
  #     deletes the pod explicitly after taking a backup + reviewing
  #     the change. Prevents an accidental Helm-template tweak from
  #     triggering a database restart at an awkward time.
  #   podManagementPolicy: OrderedReady — when scaling Postgres to
  #     a replica >1 (future HA work), pods come up one at a time
  #     and must reach Ready before the next pod is created. Aligns
  #     with the standard Postgres-on-Kubernetes pattern.
  updateStrategy:
    type: OnDelete
  podManagementPolicy: OrderedReady
  selector:
    matchLabels:
      {{- include "certctl.postgresSelectorLabels" . | nindent 6 }}
@@ -0,0 +1,145 @@
 {{- /*
 Phase 4 DEPL-L2 closure (2026-05-14): opt-in Prometheus AlertManager
 rules covering the four operationally-actionable alerts every certctl
 deployment wants out of the box.
 OPERATOR OPT-IN. Default `monitoring.prometheusRules.enabled: false`.
 Turning it on requires Prometheus Operator CRDs (PrometheusRule kind)
 to be installed in-cluster. Without them this template renders an
 object Kubernetes will reject — keep the toggle off if you're scraping
 with vanilla Prometheus + a Helm-installed AlertManager rules
 ConfigMap instead.
 Metric names + thresholds verified against the actual
 internal/api/handler/metrics.go exposition path:
  - certctl_certificate_expiring_soon: server-side count of certs with
    ExpiresAt in (now, now + 30d]. The 30-day window is computed in
    internal/service/stats.go::GetDashboardSummary.
  - certctl_agent_online: agents with heartbeat in the last 5 minutes.
    A drop below certctl_agent_total signals offline agents.
  - certctl_job_failed_total + certctl_job_completed_total: cumulative
    counters; ratio gives the failure rate over the rate() window.
  - certctl_issuance_failures_total: cumulative counter of failed
    issuance attempts (renewal failures are issuance failures with a
    specific error_class label).
 Adjust thresholds per fleet — the defaults below are tuned for the
 demo dataset (15 certs / 1 agent) and may need raising for production
 fleets with thousands of certs where a steady rate of expiring certs
 is the normal operating state.
 */ -}}
 {{- if and .Values.monitoring.enabled .Values.monitoring.prometheusRules.enabled }}
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
 metadata:
  name: {{ include "certctl.fullname" . }}-rules
  labels:
    {{- include "certctl.labels" . | nindent 4 }}
    app.kubernetes.io/component: monitoring
    {{- with .Values.monitoring.prometheusRules.labels }}
    {{- toYaml . | nindent 4 }}
    {{- end }}
 spec:
  groups:
    - name: certctl.alerts
      interval: {{ .Values.monitoring.prometheusRules.interval | default "60s" }}
      rules:
        # ---------------------------------------------------------------
        # Alert: CertctlCertificateExpiringSoon
        # Series: certctl_certificate_expiring_soon
        # The certctl-server counts certs with ExpiresAt in
        # (now, now + 30d] every metrics scrape. Fires whenever any cert
        # crosses into that window — operator must triage or extend
        # automation coverage. Rapid renewal infrastructure should keep
        # this number small in steady state.
        # ---------------------------------------------------------------
        - alert: CertctlCertificateExpiringSoon
          expr: certctl_certificate_expiring_soon > {{ .Values.monitoring.prometheusRules.thresholds.expiringCertificateCount | default 0 }}
          for: {{ .Values.monitoring.prometheusRules.thresholds.expiringCertificateFor | default "5m" }}
          labels:
            severity: warning
            component: certctl
          annotations:
            summary: "certctl: {{`{{ $value }}`}} certificate(s) expiring within 30 days"
            description: >-
              certctl_certificate_expiring_soon has been > {{ .Values.monitoring.prometheusRules.thresholds.expiringCertificateCount | default 0 }}
              for 5+ minutes. Investigate via
              /api/v1/certificates?status=expiring or the dashboard's
              Expiring tab. If renewal automation should have covered
              these, check the renewal scheduler logs for the cert IDs
              + the per-issuer failure rate.
        # ---------------------------------------------------------------
        # Alert: CertctlAgentOffline
        # Series: certctl_agent_total - certctl_agent_online
        # Agents flip from online → offline after 5 minutes without a
        # heartbeat (internal/service/stats.go::GetDashboardSummary).
        # The 1h `for:` window prevents a flapping agent from paging the
        # operator on every transient network blip.
        # ---------------------------------------------------------------
        - alert: CertctlAgentOffline
          expr: (certctl_agent_total - certctl_agent_online) > {{ .Values.monitoring.prometheusRules.thresholds.offlineAgentCount | default 0 }}
          for: {{ .Values.monitoring.prometheusRules.thresholds.offlineAgentFor | default "1h" }}
          labels:
            severity: warning
            component: certctl-agent
          annotations:
            summary: "certctl: {{`{{ $value }}`}} agent(s) offline for >1h"
            description: >-
              One or more certctl-agent instances have been without a
              heartbeat for over an hour. Check the agent logs on the
              affected hosts. If the agent host is intentionally
              decommissioned, retire the agent via the dashboard or
              POST /api/v1/agents/{id}/retire to suppress this alert.
        # ---------------------------------------------------------------
        # Alert: CertctlJobFailureRateHigh
        # Series: certctl_job_failed_total / (certctl_job_failed_total + certctl_job_completed_total)
        # Computes the failure rate over a 15-minute rate() window so
        # short bursts don't fire but a sustained issue does. The 5%
        # threshold is a conservative starter — adjust per fleet's
        # baseline.
        # ---------------------------------------------------------------
        - alert: CertctlJobFailureRateHigh
          expr: >-
            (
              rate(certctl_job_failed_total[15m])
              /
              clamp_min(rate(certctl_job_failed_total[15m]) + rate(certctl_job_completed_total[15m]), 1)
            ) > {{ .Values.monitoring.prometheusRules.thresholds.jobFailureRate | default 0.05 }}
          for: {{ .Values.monitoring.prometheusRules.thresholds.jobFailureRateFor | default "15m" }}
          labels:
            severity: warning
            component: certctl
          annotations:
            summary: "certctl: job failure rate above 5% over 15m"
            description: >-
              The 15m rate of certctl_job_failed_total / total jobs
              has been above 5% for 15+ minutes. Open
              /api/v1/jobs?status=failed to see the failing job IDs
              and root-cause the recurring error class.
        # ---------------------------------------------------------------
        # Alert: CertctlIssuanceFailures
        # Series: certctl_issuance_failures_total
        # Any non-zero rate of issuance failures over a 15m window is
        # operationally significant — a single CA outage or expired
        # ACME account can cascade across the fleet.
        # ---------------------------------------------------------------
        - alert: CertctlIssuanceFailures
          expr: rate(certctl_issuance_failures_total[15m]) > {{ .Values.monitoring.prometheusRules.thresholds.issuanceFailureRate | default 0 }}
          for: {{ .Values.monitoring.prometheusRules.thresholds.issuanceFailureFor | default "15m" }}
          labels:
            severity: warning
            component: certctl
          annotations:
            summary: "certctl: certificate issuance / renewal failures over 15m"
            description: >-
              certctl_issuance_failures_total has been incrementing
              over the last 15 minutes. Check the per-issuer breakdown
              via /api/v1/issuers + the failed-job log in
              /api/v1/jobs?status=failed. Common causes: CA
              outage, ACME account rate-limit, EAB credential
              expiration, stepca provisioner key rotation without
              certctl-side update.
 {{- end }}
@@ -12,6 +12,8 @@ data:
  keygen-mode: {{ .Values.server.keygen.mode | quote }}
  rate-limit-rps: {{ .Values.server.rateLimiting.rps | quote }}
  rate-limit-burst: {{ .Values.server.rateLimiting.burst | quote }}
  rate-limit-backend: {{ .Values.server.rateLimiting.backend | default "memory" | quote }}
  rate-limit-janitor-interval: {{ .Values.server.rateLimiting.janitorInterval | default "5m" | quote }}
  {{- if .Values.server.cors.origins }}
  cors-origins: {{ .Values.server.cors.origins | quote }}
  {{- end }}
@@ -1,5 +1,6 @@
 {{- include "certctl.tls.required" . }}
 {{- include "certctl.validateAuthType" . }}
 {{- include "certctl.requiredSecrets" . }}
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -23,8 +24,13 @@ spec:
        checksum/secret: {{ include (print $.Template.BasePath "/server-secret.yaml") . | sha256sum }}
    spec:
      serviceAccountName: {{ include "certctl.serviceAccountName" . }}
      # Bundle 3 closure (D3): pod-level fields only. The container-only
      # fields (readOnlyRootFilesystem, allowPrivilegeEscalation,
      # capabilities, privileged) render at container scope below —
      # pre-Bundle-3 they all sat here at pod scope and the K8s API
      # silently dropped them.
      securityContext:
-        {{- toYaml .Values.server.securityContext | nindent 8 }}
+        {{- include "certctl.podSecurityContext" .Values.server.securityContext | nindent 8 }}
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
@@ -33,11 +39,32 @@ spec:
        - name: server
          image: {{ include "certctl.serverImage" . }}
          imagePullPolicy: {{ .Values.server.image.pullPolicy }}
          # Bundle 3 closure (D3): container-scope security hardening.
          # readOnlyRootFilesystem + allowPrivilegeEscalation +
          # capabilities are container-only fields per the K8s API; the
          # helper splits them out of the operator-facing
          # server.securityContext map so existing values keep working.
          securityContext:
            {{- include "certctl.containerSecurityContext" .Values.server.securityContext | nindent 12 }}
          ports:
            - name: https
              containerPort: {{ .Values.server.port }}
              protocol: TCP
          env:
            # DEPL-003 closure (Sprint 3, 2026-05-16). Pre-fix the
            # CERTCTL_MIGRATIONS_VIA_HOOK env var was documented in
            # values.yaml (L797-810) and migration-job.yaml comments
            # but was never rendered into the server Deployment env
            # block. With migrations.viaHook=true the operator's
            # intent is "the pre-install/pre-upgrade Helm Job owns
            # migrations" — but the server pods, missing the env,
            # ran their own boot-time RunMigrations alongside the
            # hook Job, racing on the schema lock. cmd/server/migrations.go
            # only short-circuits when this env is "true" (line 144).
            {{- if .Values.migrations.viaHook }}
            - name: CERTCTL_MIGRATIONS_VIA_HOOK
              value: "true"
            {{- end }}
            - name: CERTCTL_SERVER_HOST
              value: "0.0.0.0"
            - name: CERTCTL_SERVER_PORT
@@ -51,11 +78,16 @@ spec:
                secretKeyRef:
                  name: {{ include "certctl.fullname" . }}-server
                  key: database-url
            # Bundle 3 closure (D2): POSTGRES_PASSWORD is only needed
            # for the bundled-Postgres mode. External Postgres mode
            # embeds the password directly in externalDatabase.url.
            {{- if .Values.postgresql.enabled }}
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: {{ include "certctl.fullname" . }}-postgres
                  key: password
            {{- end }}
            - name: CERTCTL_LOG_LEVEL
              valueFrom:
                configMapKeyRef:
@@ -90,6 +122,19 @@ spec:
                configMapKeyRef:
                  name: {{ include "certctl.fullname" . }}-server
                  key: rate-limit-burst
            # Phase 13 Sprint 13.3 (ARCH-M1) — cross-replica-consistent
            # sliding-window rate limiter. Default memory; flip to
            # postgres when server.replicas > 1.
            - name: CERTCTL_RATE_LIMIT_BACKEND
              valueFrom:
                configMapKeyRef:
                  name: {{ include "certctl.fullname" . }}-server
                  key: rate-limit-backend
            - name: CERTCTL_RATE_LIMIT_JANITOR_INTERVAL
              valueFrom:
                configMapKeyRef:
                  name: {{ include "certctl.fullname" . }}-server
                  key: rate-limit-janitor-interval
            {{- if .Values.server.cors.origins }}
            - name: CERTCTL_CORS_ORIGINS
              valueFrom:
@@ -11,6 +11,23 @@ metadata:
  {{- end }}
 spec:
  type: {{ .Values.server.service.type }}
  {{- /*
    DEPL-006 closure (Sprint 3, 2026-05-16). Render the optional
    sessionAffinity field. docs/operator/runbooks/ha.md instructs
    operators to set sessionAffinity: ClientIP for replicas > 1 so
    login + CSRF flows stay on the same pod; pre-fix the chart did
    not actually pass the value through. sessionAffinityConfig
    clientIP.timeoutSeconds renders only when set, otherwise
    Kubernetes applies its default (10800s / 3h).
  */}}
  {{- if .Values.server.service.sessionAffinity }}
  sessionAffinity: {{ .Values.server.service.sessionAffinity }}
  {{- with .Values.server.service.sessionAffinityTimeoutSeconds }}
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: {{ . }}
  {{- end }}
  {{- end }}
  ports:
    - port: {{ .Values.server.service.port }}
      targetPort: https
@@ -0,0 +1,81 @@
 {{- /*
 Bundle 3 closure (D5 + OPS-M1 docs): Prometheus Operator ServiceMonitor.
 Pre-Bundle-3 the chart had `monitoring.serviceMonitor.enabled` in
 values.yaml but no template consumed it — toggling it on rendered
 nothing. Audit D5 closed.
 The endpoint scrapes /api/v1/metrics/prometheus which the certctl
 server already exposes in Prometheus exposition format (see
 internal/api/handler/metrics.go::GetPrometheusMetrics). Note: the
 endpoint is rbac-gated on `metrics.read`, so the ServiceMonitor needs
 a bearer token. Operators with Prometheus Operator MUST set
 `monitoring.serviceMonitor.bearerTokenSecret` pointing at a Secret
 that holds an API key with the `metrics.read` permission. Without
 that, scrapes return 401.
 OPS-M1 caveat: the current /metrics/prometheus handler is a hand-rolled
 exposition-format emitter, not prometheus/client_golang-instrumented
 code. Histograms, exemplars, and target labels are limited to what the
 handler computes statically. Migration to client_golang tracked in
 WORKSPACE-ROADMAP.md.
 */ -}}
 {{- if and .Values.monitoring.enabled .Values.monitoring.serviceMonitor.enabled }}
 apiVersion: monitoring.coreos.com/v1
 kind: ServiceMonitor
 metadata:
  name: {{ include "certctl.fullname" . }}-server
  labels:
    {{- include "certctl.labels" . | nindent 4 }}
    app.kubernetes.io/component: server
    {{- with .Values.monitoring.serviceMonitor.labels }}
    {{- toYaml . | nindent 4 }}
    {{- end }}
 spec:
  selector:
    matchLabels:
      {{- include "certctl.serverSelectorLabels" . | nindent 6 }}
  endpoints:
    - port: https
      scheme: https
      path: /api/v1/metrics/prometheus
      interval: {{ .Values.monitoring.serviceMonitor.interval | default "30s" }}
      scrapeTimeout: {{ .Values.monitoring.serviceMonitor.scrapeTimeout | default "10s" }}
      tlsConfig:
        {{- /*
        Acquisition-audit DEPL-004 closure (Sprint 6 ACQ, 2026-05-16).
        Pre-Sprint-6 the default was an implicit insecureSkipVerify
        true via the template falling through the else branch.
        Post-Sprint-6 values.yaml ships a real-verify default
        (caFile + serverName matching the chart existingSecret /
        cert-manager-emitted Secret at /etc/prometheus/secrets/
        certctl-ca/), so the truthy if-branch below always fires for
        the default install. Operators who want skipVerify back must
        override with tlsConfig insecureSkipVerify true explicitly.
        Operators who blank tlsConfig entirely hit the else-branch
        below and trip the Helm fail directive at chart-render time;
        there is no way to inherit the pre-Sprint-6 implicit-skip
        behavior silently. See docs/operator/helm-deployment.md for
        the narrative explanation, including the lesson that comment
        text referencing Helm template-action delimiters must live
        in Helm-style comment blocks (this block), never in YAML
        hash-comment blocks — the Helm lexer scans for action
        delimiters everywhere in the source text, ignoring YAML
        comment markers, so descriptive references to actions inside
        YAML hash-comments are reinterpreted as template actions
        and abort the entire chart render.
        */ -}}
        {{- if .Values.monitoring.serviceMonitor.tlsConfig }}
        {{- toYaml .Values.monitoring.serviceMonitor.tlsConfig | nindent 8 }}
        {{- else }}
        {{- fail "monitoring.serviceMonitor.tlsConfig was explicitly blanked but monitoring.serviceMonitor.enabled=true (Sprint 6 ACQ DEPL-004 closure, 2026-05-16). The values.yaml default ships caFile=/etc/prometheus/secrets/certctl-ca/ca.crt + serverName=certctl-server which matches the existingSecret mount pattern. If your Prometheus pod mounts the CA bundle at a different path, override caFile rather than blanking the block. If you genuinely need skipVerify, set tlsConfig insecureSkipVerify=true explicitly — never blank. See docs/operator/helm-deployment.md for the upgrade-path note." }}
        {{- end }}
      {{- with .Values.monitoring.serviceMonitor.bearerTokenSecret }}
      bearerTokenSecret:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.monitoring.serviceMonitor.relabelings }}
      relabelings:
        {{- toYaml . | nindent 8 }}
      {{- end }}
 {{- end }}
@@ -15,7 +15,10 @@ fullnameOverride: ""
 # Certctl Server Configuration
 # ==============================================================================
 server:
-  # Number of replicas (for HA deployments)
+  # Number of replicas (for HA deployments).
  # Phase 2 DEPL-H1: production HA is operator-opt-in across this field
  # + podDisruptionBudget.enabled + server.service.sessionAffinity.
  # See docs/operator/runbooks/ha.md for the smallest-possible HA overlay.
  replicas: 1
  # Image configuration
@@ -28,6 +31,36 @@ server:
  port: 8443
  # Resource requests and limits
  #
  # Phase 4 DEPL-M5 (2026-05-14): per-fleet-size tuning ladder. The
  # default values below are validated against the demo dataset
  # (15 certs / 1 agent) and the baselines in
  # docs/operator/performance-baselines.md (single endpoint < 5s for
  # 100 sequential requests = ~50ms p50; cursor-paginated 1000-cert
  # inventory walk < 3s; renewal scan for 15 certs < 100ms).
  #
  # Larger fleet recommendations (TBD pending Phase 8 load-test runs;
  # operators tune empirically until then — capture readings in your
  # own loadtest-baselines log):
  #
  #   ≤ 500 certs / 100 agents:      defaults below                  (100m / 128Mi req, 500m / 512Mi lim)
  #   5K certs / 1K agents:          tune up — TBD Phase 8           (suggested starter: 500m / 512Mi req, 2000m / 2Gi lim)
  #   50K certs / 10K agents:        tune up — TBD Phase 8           (suggested starter: 2000m / 2Gi req, 4000m / 4Gi lim)
  #
  # The "suggested starter" values above are operator-tuning starting
  # points, NOT validated. Phase 8 (load test coverage expansion) will
  # measure them against synthetic fleets and replace the suggestions
  # with measured ceilings. Until then, treat them as a "raise CPU
  # before raising memory; raise both before scaling out" mental
  # model. Per docs/operator/performance-baselines.md, certctl-server
  # is CPU-bound on issuance / renewal scan work and memory-bound on
  # the inventory query path.
  #
  # Database scale (postgresql.* below) tracks server scale roughly
  # 1:1 — at 50K certs the Postgres instance needs 4 CPU / 4Gi RAM
  # and shared_buffers ≥ 1Gi. Postgres tuning is out of scope for
  # this comment; see docs/operator/runbooks/postgres-backup.md
  # for the production-tuning entry-point.
  resources:
    requests:
      cpu: 100m
@@ -127,6 +160,17 @@ server:
    type: ClusterIP
    port: 8443
    annotations: {}
    # DEPL-006 closure (Sprint 3, 2026-05-16). Optional sticky-session
    # routing. REQUIRED when server.replicas > 1 so login + CSRF token
    # rows stay on the same pod for the duration of a session — the
    # default round-robin load balancing breaks those flows. Set to
    # "ClientIP" for production HA (see deploy/helm/examples/values-prod-ha.yaml).
    # Leave empty for single-replica deploys.
    sessionAffinity: ""
    # When sessionAffinity is set, timeout window (in seconds) the
    # Service maps a source IP to the same pod. Default null →
    # Kubernetes applies its built-in default (10800s / 3h).
    sessionAffinityTimeoutSeconds: null
  # Authentication configuration.
  # Valid types: "api-key" (production) or "none" (demo only — disables
@@ -178,8 +222,25 @@ server:
  # Rate limiting configuration
  rateLimiting:
-    rps: 100      # Requests per second
+    rps: 100      # Requests per second (token-bucket middleware)
-    burst: 200    # Burst capacity
+    burst: 200    # Burst capacity (token-bucket middleware)
    # Sliding-window-log rate-limit backend (Phase 13 Sprint 13.2/13.3
    # ARCH-M1 closure). Selects the implementation backing the
    # break-glass / OCSP / cert-export / EST limiters. See
    # docs/operator/observability.md for the operator decision tree.
    #
    #   memory   — per-process (default; single-replica deploys).
    #   postgres — cross-replica-consistent via rate_limit_buckets.
    #              REQUIRED when server.replicas > 1 for accurate
    #              cluster-wide enforcement.
    backend: memory
    # Scheduler janitor interval for the postgres backend's
    # rate_limit_buckets sweep. Ignored when backend=memory (the
    # in-memory backend self-prunes on every Allow call).
    # Default 5m; minimum 1m.
    janitorInterval: "5m"
  # Network scanning configuration
  networkScan:
@@ -272,6 +333,34 @@ server:
  #   secret:
  #     secretName: ca-cert
 # ==============================================================================
 # External Database Configuration (Bundle 3 closure / D2 + OPS-L2)
 # ==============================================================================
 # When postgresql.enabled=false, the chart skips the bundled StatefulSet +
 # Secret + Service and instead consumes the URL below verbatim as the
 # server's CERTCTL_DATABASE_URL. The URL embeds username, password,
 # host, port, database, and sslmode — operators are responsible for
 # rotating credentials in this string out-of-band (Kubernetes Secret +
 # helm upgrade is the supported pattern).
 #
 # Recommended sslmode for managed Postgres (RDS, Cloud SQL, Azure DB):
 #   verify-full  — PCI-DSS Req 4 v4.0 §2.2.5 compliant; requires CA bundle.
 #                  Mount the CA via server.volumes / server.volumeMounts and
 #                  set sslrootcert=/path/in/pod/ca.crt in the URL.
 #
 # Example values overrides:
 #   postgresql.enabled: false
 #   externalDatabase.url: "postgres://certctl:HUNTER2@db.example.com:5432/certctl?sslmode=verify-full"
 #
 # Migration from the legacy `server.env.CERTCTL_DATABASE_URL` workaround:
 # both still work (env block overrides the helper-emitted Secret value at
 # pod-spec level), but the new path renders cleaner manifests with no
 # stranded postgres-* templates.
 externalDatabase:
  # Connection string used when postgresql.enabled=false.
  # Required in that mode — see certctl.requiredSecrets helper.
  url: ""
 # ==============================================================================
 # PostgreSQL Configuration
 # ==============================================================================
@@ -418,6 +507,27 @@ agent:
  replicas: 1
  # Resource requests and limits
  #
  # Phase 4 DEPL-M5 (2026-05-14): per-fleet-size tuning ladder for the
  # agent. Defaults are sized for the standard "one cert per host"
  # operating pattern: the agent polls the server every 30 seconds
  # (hardcoded in cmd/agent/main.go::pollInterval — not yet
  # env-configurable), generates ECDSA P-256 keys locally on
  # issuance/renewal events, and is otherwise idle. CPU is bursty only
  # during keygen + CSR submission.
  #
  # Tuning ladder (TBD pending Phase 8 — measure on your fleet):
  #
  #   1 cert / host (typical):        defaults below            (50m / 64Mi req, 200m / 256Mi lim)
  #   10 certs / host:                stays at defaults — agent is poll-driven, not work-bound by cert count
  #   100 certs / host (rare):        raise lim to 500m / 512Mi if you see throttling on issuance bursts
  #
  # The agent does NOT cache certs in memory — issuance is one-shot
  # generate-then-deploy. So per-host memory scales with whatever
  # truststore PEM bundles the agent's connectors load (Apache /
  # Postfix / similar), not with the cert count. Defaults are
  # appropriate for any "agent terminates ≤ 100 certs on this host"
  # deployment.
  resources:
    requests:
      cpu: 50m
@@ -510,14 +620,34 @@ rbac:
  create: true
 # ==============================================================================
-# Kubernetes Secrets Target Connector
+# Kubernetes Secrets Target Connector (PREVIEW — Bundle 3 closure / C3)
 # ==============================================================================
 # Bundle 3 audit closure (C3): the connector framework at
 # internal/connector/target/k8ssecret/ ships the Config + interface +
 # 14 unit tests, but the production K8s client at
 # k8ssecret.go::realK8sClient is documented as "a stub placeholder for
 # the real k8s.io/client-go implementation". The repo does not import
 # k8s.io/client-go (verified via `grep -n "client-go" go.mod`), so the
 # connector cannot deploy to a real cluster today.
 #
 # Setting kubernetesSecrets.enabled=true wires up the RBAC verbs the
 # real client will need (get/create/update/patch/delete on Secrets)
 # without making the connector functional — operators trying to use it
 # get the stub's error and a pointer to this note.
 #
 # Status: PREVIEW. Production client lands when the cluster-management
 # bundle ships (tracked in WORKSPACE-ROADMAP.md). Until then,
 # in-cluster deploys use the file-based connectors (NGINX, Apache,
 # HAProxy, etc.) via a Pod-mounted Secret + DaemonSet agent.
 kubernetesSecrets:
  # Enable RBAC rules for managing TLS Secrets
  enabled: false
 # ==============================================================================
-# Pod Disruption Budget (for HA deployments)
+# Pod Disruption Budget (for HA deployments).
 # Phase 2 DEPL-H1: defaults to enabled=false because a PDB template
 # rendered at `replicas: 1` blocks every rolling restart on a
 # single-node cluster. Production HA flips this to true alongside
 # server.replicas ≥ 2. See docs/operator/runbooks/ha.md.
 # ==============================================================================
 podDisruptionBudget:
  enabled: false
@@ -527,6 +657,13 @@ podDisruptionBudget:
 # ==============================================================================
 # Monitoring Configuration
 # ==============================================================================
 # Bundle 3 closure (D5): the ServiceMonitor template at
 # templates/servicemonitor.yaml renders when both monitoring.enabled=true
 # AND monitoring.serviceMonitor.enabled=true. The endpoint scrapes
 # /api/v1/metrics/prometheus, which is rbac-gated on `metrics.read` —
 # operators MUST provide a bearer token via
 # monitoring.serviceMonitor.bearerTokenSecret pointing at a Secret with
 # an API key holding that permission. Without the token, scrapes 401.
 monitoring:
  enabled: false
  # Prometheus ServiceMonitor
@@ -534,8 +671,218 @@ monitoring:
    enabled: false
    interval: 30s
    scrapeTimeout: 10s
    # Additional labels applied to the ServiceMonitor metadata.
    # labels: {}
-    # selector: {}
+    # Bearer-token Secret reference (required when the certctl server's
    # /api/v1/metrics/prometheus endpoint is gated by api-key auth).
    # Example:
    #   bearerTokenSecret:
    #     name: certctl-prometheus-key
    #     key: api-key
    # bearerTokenSecret: {}
    # TLS config for the scrape endpoint. Acquisition-audit DEPL-004
    # closure (Sprint 6 ACQ, 2026-05-16): pre-Sprint-6 the default was
    # an implicit `insecureSkipVerify: true` (fell through the
    # template's else-branch). Post-Sprint-6 the default is a real
    # verify against the chart's CA at the canonical mount path the
    # existingSecret pattern produces (Prometheus mounts the
    # certctl-ca Secret as a volume at /etc/prometheus/secrets/
    # certctl-ca/). Operators whose Prometheus pod mounts the bundle
    # at a different path override `caFile` below; operators who
    # genuinely want skipVerify back can do so explicitly. Operators
    # who blank tlsConfig entirely (`tlsConfig: null` or
    # `tlsConfig: {}`) trip the `{{ fail }}` guard in
    # templates/servicemonitor.yaml at chart-render time — there is
    # no way to inherit the pre-Sprint-6 implicit-skipVerify behavior
    # silently.
    #
    # Production default (verify against the chart's CA):
    tlsConfig:
      caFile: /etc/prometheus/secrets/certctl-ca/ca.crt
      serverName: certctl-server
    #
    # Operator override — different CA mount path:
    # tlsConfig:
    #   caFile: /path/to/your/ca.crt
    #   serverName: your-cert-CN
    #
    # Operator override — demo / dev-cluster escape hatch
    # (operator-acknowledged unsafe):
    # tlsConfig:
    #   insecureSkipVerify: true
    # Optional relabeling for the scrape job.
    # relabelings: []
  # ----------------------------------------------------------------------
  # Phase 4 DEPL-L2 closure (2026-05-14): PrometheusRule (alert rules)
  #
  # Operator opt-in. Requires Prometheus Operator CRDs (the
  # `monitoring.coreos.com/v1` PrometheusRule kind) installed in
  # cluster. Without those CRDs the rendered object is rejected by
  # `kubectl apply` — keep enabled: false if you scrape with vanilla
  # Prometheus + AlertManager rules ConfigMap instead.
  #
  # Four starter rules ship out of the box (see
  # templates/prometheusrules.yaml for the full PromQL):
  #
  #   CertctlCertificateExpiringSoon — certs expiring within 30d
  #   CertctlAgentOffline             — agent without heartbeat for >1h
  #   CertctlJobFailureRateHigh       — job-failure rate over 5% (15m)
  #   CertctlIssuanceFailures         — any issuance failures in last 15m
  #
  # All thresholds are operator-tunable via the `thresholds:` block
  # below. The defaults are tuned for the demo dataset (15 certs / 1
  # agent); production fleets with sustained renewal volume MAY want
  # to raise the expiringCertificateCount + jobFailureRate thresholds
  # to suppress steady-state noise.
  prometheusRules:
    enabled: false
    # Evaluation interval for the rule group.
    interval: 60s
    # Additional labels applied to the PrometheusRule metadata.
    # labels: {}
    # Per-alert threshold / duration tunables.
    thresholds:
      # Fire when more than N certs are in the expiring-soon window.
      expiringCertificateCount: 0
      expiringCertificateFor: 5m
      # Fire when more than N agents are offline (server - online).
      offlineAgentCount: 0
      offlineAgentFor: 1h
      # Fire when job failure rate exceeds this fraction (15m window).
      jobFailureRate: 0.05
      jobFailureRateFor: 15m
      # Fire when issuance failure rate exceeds this value (15m window).
      issuanceFailureRate: 0
      issuanceFailureFor: 15m
 # ==============================================================================
 # Backup CronJob (Phase 4 DEPL-H2 closure, 2026-05-14)
 # ==============================================================================
 # Operator opt-in. Default OFF. The CronJob runs `pg_dump --format=custom
 # --no-owner --no-acl --dbname=certctl` matching the canonical shape
 # documented in docs/operator/runbooks/postgres-backup.md (so manual
 # and automated dumps are byte-identical) and ships the result to a
 # sink chosen below.
 #
 # DO NOT enable this for managed Postgres deployments (AWS RDS / GCP
 # Cloud SQL / Azure DB) — those have built-in PITR backup that this
 # CronJob cannot match. For in-cluster Postgres only.
 backup:
  enabled: false
  # Cron expression (UTC). Default: 02:30 UTC daily.
  schedule: "30 2 * * *"
  # Sink: "pvc" (default — dump lands on a PersistentVolumeClaim) or
  # "s3" (uploads via aws-cli — requires an image that bundles
  # aws-cli, see backup.image below).
  sink: pvc
  # Container image. The default postgres:16-alpine has pg_dump but
  # NOT aws-cli; for sink: s3 set this to an image that bundles both
  # (e.g. ghcr.io/your-org/postgres-aws:16) or override the Job's
  # command to install aws-cli at runtime.
  image: postgres:16-alpine
  imagePullPolicy: IfNotPresent
  # PVC sink config — used when sink: pvc.
  pvc:
    # Name of an existing PersistentVolumeClaim mounted at /backups
    # in the Job's pod. The PVC's storage class controls durability
    # and snapshot retention. Operator creates this PVC out of band
    # via their own storage policy.
    claimName: certctl-backups
  # S3 sink config — used when sink: s3.
  s3:
    # Target bucket (without s3:// prefix).
    bucket: ""
    # Object key prefix inside the bucket. Dumps land at
    # s3://<bucket>/<prefix>/certctl-<TIMESTAMP>.dump.
    prefix: certctl
    # AWS region (sets AWS_DEFAULT_REGION). Optional if the image's
    # AWS SDK can resolve the region another way (instance profile,
    # IRSA, etc.).
    region: ""
    # Secret holding AWS credentials. The IAM principal needs
    # s3:PutObject + s3:ListBucket on the target bucket only.
    credentialsSecret:
      name: certctl-backup-aws-creds
      accessKeyIdKey: AWS_ACCESS_KEY_ID
      secretAccessKeyKey: AWS_SECRET_ACCESS_KEY
  # Job housekeeping.
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  startingDeadlineSeconds: 300
  backoffLimit: 1
  activeDeadlineSeconds: 3600
  # Resource budget for the backup container. pg_dump is generally
  # memory-light; ~250MB RSS for fleets up to 100K certs is typical.
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 500m
      memory: 512Mi
  # Optional tolerations for the backup Job pod.
  tolerations: []
 # ==============================================================================
 # Migrations via Helm hook (Phase 4 DEPL-M1 closure, 2026-05-14)
 # ==============================================================================
 # When viaHook: true, the chart deploys templates/migration-job.yaml as
 # a pre-install + pre-upgrade hook that runs `certctl-server
 # --migrate-only` (a hermetic schema-mutation pass) before the server
 # Deployment rolls.
 #
 # Set CERTCTL_MIGRATIONS_VIA_HOOK=true in the server Deployment env to
 # tell the server to skip its boot-time RunMigrations call (the hook
 # already did the work; running again at boot would race across
 # replicas during rollouts).
 #
 # Default OFF — when off, the server runs migrations at boot exactly
 # as it always has (Compose deploys keep this path).
 migrations:
  viaHook: false
  # Job housekeeping.
  backoffLimit: 1
  activeDeadlineSeconds: 600
  # Resource budget for the migration Job pod. The migration pass is
  # I/O-bound on Postgres; matches the server's resource budget by
  # default. Override here if migrations on a large database need
  # more headroom than the steady-state server.
  # resources:
  #   requests:
  #     cpu: 100m
  #     memory: 128Mi
  #   limits:
  #     cpu: 500m
  #     memory: 512Mi
 # ==============================================================================
 # Network Policy (Bundle 3 closure / D11)
 # ==============================================================================
 # Default off so existing deploys don't suddenly lose network reach.
 # When enabled, restricts the server pod to:
 #   - Ingress: from in-namespace agent pods only.
 #   - Egress: kube-dns + bundled Postgres (if enabled).
 # Operators add CA / OIDC / SMTP egress via extraEgress.
 networkPolicy:
  enabled: false
  # Additional Ingress rules merged into the policy. Each entry is a
  # raw networking.k8s.io/v1 NetworkPolicyIngressRule.
  extraIngress: []
  # Additional Egress rules merged into the policy. Common operator
  # need: 443/TCP to an OIDC issuer, 443/TCP to a public CA endpoint,
  # 25/TCP to an SMTP relay.
  # Example:
  # extraEgress:
  #   - to:
  #       - ipBlock:
  #           cidr: 0.0.0.0/0
  #           except:
  #             - 10.0.0.0/8
  #     ports:
  #       - protocol: TCP
  #         port: 443
  extraEgress: []
 # ==============================================================================
 # Advanced Configuration
@@ -36,6 +36,14 @@ server:
  service:
    type: ClusterIP
    # DEPL-006 closure (Sprint 3, 2026-05-16): with replicas:3, the
    # default round-robin Service load balancing breaks login/CSRF
    # flows because the session cookie + the CSRF token row land on
    # different pods between requests. sessionAffinity: ClientIP
    # routes every connection from a given source IP to the same
    # pod for the configured timeout window. docs/operator/runbooks/ha.md
    # documents this; pre-fix the chart did not actually render it.
    sessionAffinity: ClientIP
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "8443"
@@ -53,6 +61,14 @@ server:
  rateLimiting:
    rps: 500
    burst: 1000
    # DEPL-006 closure (Sprint 3, 2026-05-16): replicas > 1 REQUIRES
    # the postgres backend so per-key buckets are cross-replica-
    # consistent. The default 'memory' backend gives each pod its
    # own bucket map, so a 3-replica fleet effectively triples the
    # configured cap (a client churning across pods bypasses the
    # limit). See deploy/helm/certctl/values.yaml L217-226 for the
    # canonical comment.
    backend: postgres
 postgresql:
  enabled: true
@@ -0,0 +1,225 @@
 #!/usr/bin/env bash
 # Copyright 2026 certctl LLC. All rights reserved.
 # SPDX-License-Identifier: BUSL-1.1
 #
 # Acquisition-audit DEPL-005 + DATA-012 closure (Sprint 4 ACQ,
 # 2026-05-16). Backup/restore smoke harness — orchestrates a real
 # pg_dump -Fc → DROP DATABASE → CREATE DATABASE → pg_restore loop
 # around the audit_events hash chain and asserts the chain head
 # round-trips byte-for-byte.
 #
 # This script is the body of the `.github/workflows/backup-restore.yml`
 # weekly job AND the same thing an operator can run locally against a
 # running Postgres to gain confidence before a real restore.
 #
 # Prereqs
 # =======
 # - psql / pg_dump / pg_restore installed and on PATH (ubuntu-latest
 #   ships postgresql-client by default; on macOS use Homebrew's
 #   libpq).
 # - A reachable Postgres at $PGHOST:$PGPORT, plus the certctl user +
 #   database created. In CI we point this at the GHA service container
 #   (postgres:16-alpine, pinned to the same digest as
 #   deploy/docker-compose.yml). Locally, point it wherever — the
 #   script DROPs the database it connects to, so DO NOT POINT THIS
 #   AT A DATABASE YOU CARE ABOUT.
 # - Go 1.25+ on PATH so the smoke program can be built. (CI's
 #   setup-go step handles this.)
 # - jq is NOT required — JSON snapshots are compared via python3.
 #
 # Behavior contract
 # =================
 # - On success: exit 0, prints "PASS" + a summary line.
 # - On any assertion failure: prints `::error::<reason>`, exits 1.
 #   (The ::error:: prefix is the GitHub Actions log-annotation shape;
 #    it surfaces as a red banner in the Actions run UI.)
 #
 # Non-goals
 # =========
 # - Does not exercise PITR / WAL archiving. The Sprint 4 scope is the
 #   pg_dump/pg_restore path only; managed-DB PITR is the operator's
 #   responsibility per docs/operator/runbooks/postgres-backup.md.
 # - Does not regenerate the audit chain after restore. A "restore
 #   that rewrote history" would mask exactly the bug under test.
 set -euo pipefail
 REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
 WORKDIR="$(mktemp -d)"
 trap 'rm -rf "$WORKDIR"' EXIT
 # ----------------------------------------------------------------------
 # Configuration — every knob is env-overridable so the same script
 # runs unchanged in CI (where the GHA service container exposes
 # 127.0.0.1:5432) and on an operator's laptop (where they may have
 # Postgres on a UNIX socket or a different port).
 # ----------------------------------------------------------------------
 : "${PGHOST:=127.0.0.1}"
 : "${PGPORT:=5432}"
 : "${PGUSER:=certctl}"
 : "${PGPASSWORD:=certctl}"
 : "${PGDATABASE:=certctl}"
 : "${SMOKE_ROWS:=24}"
 : "${MIGRATIONS_PATH:=${REPO_ROOT}/migrations}"
 # psql/pg_dump/pg_restore all read PG* env vars. Export so we don't
 # have to spell them out on every command line.
 export PGHOST PGPORT PGUSER PGPASSWORD PGDATABASE
 DB_URL="postgres://${PGUSER}:${PGPASSWORD}@${PGHOST}:${PGPORT}/${PGDATABASE}?sslmode=disable"
 fail() {
 	# GitHub Actions log annotation. The `::error::` prefix is what
 	# the Actions UI uses to highlight a line in the run log.
 	echo "::error::backup-restore-smoke: $*" >&2
 	exit 1
 }
 step() { printf '\n=== %s ===\n' "$*"; }
 # ----------------------------------------------------------------------
 # Sanity preflight
 # ----------------------------------------------------------------------
 step "preflight"
 command -v psql       >/dev/null || fail "psql not on PATH (install postgresql-client)"
 command -v pg_dump    >/dev/null || fail "pg_dump not on PATH"
 command -v pg_restore >/dev/null || fail "pg_restore not on PATH"
 command -v go         >/dev/null || fail "go not on PATH (need Go to build the smoke program)"
 command -v python3    >/dev/null || fail "python3 not on PATH (used for JSON diff)"
 test -d "${MIGRATIONS_PATH}" || fail "migrations dir not found: ${MIGRATIONS_PATH}"
 # Wait for Postgres readiness up to 60s. pg_isready returns 0 when
 # the server is accepting connections, so the loop is the canonical
 # CI-friendly "wait for the service container" pattern.
 step "waiting for postgres at ${PGHOST}:${PGPORT}"
 for _ in $(seq 1 60); do
 	if pg_isready -h "${PGHOST}" -p "${PGPORT}" -U "${PGUSER}" -d "${PGDATABASE}" -q; then
 		break
 	fi
 	sleep 1
 done
 pg_isready -h "${PGHOST}" -p "${PGPORT}" -U "${PGUSER}" -d "${PGDATABASE}" -q \
 	|| fail "postgres not ready after 60s at ${PGHOST}:${PGPORT}"
 # Wipe any prior state in the target DB. A previous failed run could
 # have left rows behind; the smoke contract is "starts from clean."
 step "wiping ${PGDATABASE} schema (DROP SCHEMA public CASCADE; CREATE SCHEMA public)"
 psql -v ON_ERROR_STOP=1 -c 'DROP SCHEMA IF EXISTS public CASCADE; CREATE SCHEMA public; GRANT ALL ON SCHEMA public TO PUBLIC;'
 # ----------------------------------------------------------------------
 # Build the smoke program. We use `go run` to avoid leaving a binary
 # behind; the migrations + workload are quick so the per-invocation
 # compile cost is negligible.
 # ----------------------------------------------------------------------
 step "building smoke program"
 cd "${REPO_ROOT}"
 go build -o "${WORKDIR}/smoke" ./deploy/test/backupsmoke
 # ----------------------------------------------------------------------
 # Phase 1 — workload: migrate, insert rows, snapshot chain head.
 # ----------------------------------------------------------------------
 step "phase 1 — workload (${SMOKE_ROWS} audit_events rows)"
 "${WORKDIR}/smoke" \
 	--mode=workload \
 	--db-url="${DB_URL}" \
 	--migrations-path="${MIGRATIONS_PATH}" \
 	--rows="${SMOKE_ROWS}" \
 	| tee "${WORKDIR}/pre.json"
 # ----------------------------------------------------------------------
 # Phase 2 — backup. Canonical pg_dump shape per
 # deploy/helm/certctl/templates/backup-cronjob.yaml: --format=custom,
 # --no-owner, --no-acl. --no-owner / --no-acl keep the dump portable
 # across Postgres installations with different role layouts (the
 # audit-trail hash chain is data, not ACL state).
 # ----------------------------------------------------------------------
 step "phase 2 — pg_dump -Fc"
 pg_dump --format=custom --no-owner --no-acl --dbname="${PGDATABASE}" --file="${WORKDIR}/backup.dump"
 test -s "${WORKDIR}/backup.dump" || fail "pg_dump produced an empty file"
 # ----------------------------------------------------------------------
 # Phase 3 — wipe. The fresh-schema approach is the closest analogue
 # to "operator nuked the wrong volume." DROP DATABASE would require
 # connecting to a different DB and reconnect dance; DROP SCHEMA
 # achieves the same "no rows, no schema, no functions" end state
 # inside the existing connection and is restore-compatible (pg_dump
 # -Fc bundles the schema in the dump, so pg_restore recreates it).
 # ----------------------------------------------------------------------
 step "phase 3 — drop schema (simulating data-loss event)"
 psql -v ON_ERROR_STOP=1 -c 'DROP SCHEMA IF EXISTS public CASCADE; CREATE SCHEMA public; GRANT ALL ON SCHEMA public TO PUBLIC;'
 # Sanity: confirm audit_events is actually gone before restore. A
 # regression here (e.g. DROP SCHEMA silently no-op) would let the
 # verifier "succeed" by reading the original rows, making the test
 # false-pass.
 PRE_RESTORE_TABLES=$(psql -tAc "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='public'")
 if [ "${PRE_RESTORE_TABLES}" -ne 0 ]; then
 	fail "post-DROP SCHEMA, expected 0 public tables; saw ${PRE_RESTORE_TABLES}"
 fi
 # ----------------------------------------------------------------------
 # Phase 4 — restore.
 # ----------------------------------------------------------------------
 step "phase 4 — pg_restore"
 pg_restore --dbname="${PGDATABASE}" --no-owner --no-acl --exit-on-error "${WORKDIR}/backup.dump"
 # ----------------------------------------------------------------------
 # Phase 5 — verify: re-snapshot, run audit_events_verify_chain().
 # ----------------------------------------------------------------------
 step "phase 5 — verify (audit_events_verify_chain() + snapshot)"
 "${WORKDIR}/smoke" \
 	--mode=verify \
 	--db-url="${DB_URL}" \
 	| tee "${WORKDIR}/post.json"
 # ----------------------------------------------------------------------
 # Phase 6 — assert.
 #
 #   pre.row_count       == post.row_count
 #   pre.chain_head_hash == post.chain_head_hash   (BYTE-EXACT)
 #   post.first_break_id == ""                     (verifier clean)
 #   post.verifier_walked == pre.row_count         (every row walked)
 #
 # Use python3 rather than jq so the script runs unchanged on macOS
 # without an extra Homebrew install.
 # ----------------------------------------------------------------------
 step "phase 6 — assertions"
 python3 - <<'PY' "${WORKDIR}/pre.json" "${WORKDIR}/post.json"
 import json, sys
 pre  = json.load(open(sys.argv[1]))
 post = json.load(open(sys.argv[2]))
 def bail(msg):
    print(f"::error::backup-restore-smoke: {msg}", file=sys.stderr)
    sys.exit(1)
 if pre["row_count"] != post["row_count"]:
    bail(f"row_count mismatch: pre={pre['row_count']} post={post['row_count']}")
 if pre["chain_head_hash"] != post["chain_head_hash"]:
    bail(
        "chain_head_hash mismatch — pg_dump/pg_restore did NOT round-trip the "
        "audit_events hash chain byte-for-byte. "
        f"pre={pre['chain_head_hash']} post={post['chain_head_hash']}"
    )
 if post.get("first_break_id", "") != "":
    bail(
        "audit_events_verify_chain() reports a break post-restore at id="
        f"{post['first_break_id']} pos={post.get('first_break_pos', '?')} — "
        "the chain is no longer self-consistent after the restore."
    )
 if post.get("verifier_walked", -1) != pre["row_count"]:
    bail(
        f"verifier_walked={post.get('verifier_walked')} != pre.row_count="
        f"{pre['row_count']} — verifier short-circuited or read stale rows."
    )
 print(
    f"PASS  rows={pre['row_count']}  "
    f"chain_head={pre['chain_head_hash'][:16]}…  "
    f"verifier=clean"
 )
 PY
@@ -0,0 +1,222 @@
 // Copyright 2026 certctl LLC. All rights reserved.
 // SPDX-License-Identifier: BUSL-1.1
 // Command backupsmoke is the workload+verifier half of the
 // backup/restore CI gate (acquisition-audit DEPL-005 + DATA-012
 // closure, Sprint 4 ACQ, 2026-05-16).
 //
 // The companion shell harness `deploy/test/backup-restore-smoke.sh`
 // orchestrates the dump/drop/restore lifecycle around two
 // invocations of this program: one before the backup
 // (--mode=workload) and one after the restore (--mode=verify). Both
 // emit a small JSON snapshot to stdout; the shell harness diffs them
 // and asserts the chain head + row count round-trip byte-for-byte.
 //
 // Modes
 // =====
 //
 //	--mode=workload
 //	  Run all up-migrations against `--migrations-path`, then
 //	  generate `--rows` (default 24) audit_events rows representing
 //	  an issue / renew / revoke / auth-login cycle. Emit a snapshot
 //	  with the post-workload row_count + chain head row_hash.
 //
 //	--mode=verify
 //	  Run `audit_events_verify_chain()` (the per-row hash-chain
 //	  verifier installed by migration 000047) and capture
 //	  first_break_id / first_break_pos / verifier_walked. Emit a
 //	  snapshot with row_count + chain head row_hash + verifier
 //	  output. No mutations.
 //
 // The CI assertion contract
 // =========================
 //
 // After (workload → pg_dump -Fc → DROP + CREATE → pg_restore →
 // verify), the shell asserts:
 //
 //	pre.row_count      == post.row_count
 //	pre.chain_head_hash == post.chain_head_hash   (byte-exact)
 //	post.first_break_id == ""                     (verifier clean)
 //
 // A pg_dump format-quirk that didn't preserve TIMESTAMPTZ
 // microseconds would surface as a chain-head mismatch (the
 // canonical payload re-formats `timestamp AT TIME ZONE 'UTC'` to
 // microsecond ISO-8601 — any precision loss breaks the hash). A
 // trigger-or-function regression would surface as a verifier non-
 // empty first_break_id. The test exists to PROVE these properties
 // under a real workload, not to defend against a known quirk.
 package main
 import (
 	"context"
 	"database/sql"
 	"encoding/json"
 	"flag"
 	"fmt"
 	"log"
 	"os"
 	"time"
 	_ "github.com/lib/pq"
 	"github.com/certctl-io/certctl/internal/repository/postgres"
 )
 // Snapshot is the on-the-wire shape emitted to stdout. The shell
 // orchestrator parses it via python3 -c 'json.load(...)' and diffs
 // the relevant fields. Keep it stable — any rename here must land
 // alongside a shell-harness change.
 type Snapshot struct {
 	Phase          string `json:"phase"`
 	RowCount       int    `json:"row_count"`
 	ChainHead      string `json:"chain_head_hash"`
 	FirstBreakID   string `json:"first_break_id,omitempty"`
 	FirstBreakPos  int    `json:"first_break_pos,omitempty"`
 	VerifierWalked int    `json:"verifier_walked,omitempty"`
 }
 func main() {
 	var (
 		mode           = flag.String("mode", "", "workload | verify")
 		dbURL          = flag.String("db-url", os.Getenv("DATABASE_URL"), "Postgres URL (or set DATABASE_URL)")
 		migrationsPath = flag.String("migrations-path", "./migrations", "Path to the migrations/ directory (workload mode only)")
 		rows           = flag.Int("rows", 24, "Number of audit_events rows to insert (workload mode only)")
 	)
 	flag.Parse()
 	if *dbURL == "" {
 		log.Fatal("--db-url or DATABASE_URL is required")
 	}
 	if *mode == "" {
 		log.Fatal("--mode is required (workload | verify)")
 	}
 	db, err := sql.Open("postgres", *dbURL)
 	if err != nil {
 		log.Fatalf("sql.Open: %v", err)
 	}
 	defer db.Close()
 	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
 	defer cancel()
 	if err := db.PingContext(ctx); err != nil {
 		log.Fatalf("ping: %v", err)
 	}
 	switch *mode {
 	case "workload":
 		// Run all up-migrations end-to-end. The trigger + verifier
 		// function installed by migration 000047 must be in place
 		// before the inserts below; partial migration would mask a
 		// real bug.
 		if err := postgres.RunMigrations(db, *migrationsPath); err != nil {
 			log.Fatalf("RunMigrations(%s): %v", *migrationsPath, err)
 		}
 		if err := runWorkload(ctx, db, *rows); err != nil {
 			log.Fatalf("runWorkload: %v", err)
 		}
 		snap, err := snapshot(ctx, db, "workload", false)
 		if err != nil {
 			log.Fatalf("snapshot: %v", err)
 		}
 		emit(snap)
 	case "verify":
 		snap, err := snapshot(ctx, db, "verify", true)
 		if err != nil {
 			log.Fatalf("snapshot: %v", err)
 		}
 		emit(snap)
 	default:
 		log.Fatalf("unknown --mode=%q (workload | verify)", *mode)
 	}
 }
 // runWorkload inserts n audit_events rows representing an
 // issue / renew / revoke / auth-login cycle. Patterns mirror the
 // shape the application emits (see internal/service/audit_*.go),
 // so the canonical payload exercised here is representative.
 //
 // event_category is omitted on each INSERT — migration 000032 gave
 // the column DEFAULT 'cert_lifecycle', which is also the value the
 // application uses for cert lifecycle events. Auth rows get the
 // default too, which is harmless for the round-trip property under
 // test (only the canonical-payload byte sequence matters).
 //
 // Timestamps are monotonic via the `NOW() + ($interval ||
 // ' microsecond')::interval` pattern from
 // internal/repository/postgres/audit_chain_test.go — ordering
 // determinism is necessary for the chain head to be stable across
 // runs.
 func runWorkload(ctx context.Context, db *sql.DB, n int) error {
 	actions := []struct{ act, resType, resID string }{
 		{"certificate.issue", "certificate", "mc-smoke"},
 		{"certificate.renew", "certificate", "mc-smoke"},
 		{"certificate.revoke", "certificate", "mc-smoke"},
 		{"auth.login", "session", "sess-smoke"},
 	}
 	for i := 0; i < n; i++ {
 		a := actions[i%len(actions)]
 		id := fmt.Sprintf("audit-smoke-%04d", i)
 		_, err := db.ExecContext(ctx, `
 			INSERT INTO audit_events (
 				id, actor, actor_type, action,
 				resource_type, resource_id, details, timestamp
 			)
 			VALUES (
 				$1, 'smoke-actor', 'User', $2,
 				$3, $4, '{}'::jsonb,
 				NOW() + ($5 || ' microsecond')::interval
 			)
 		`, id, a.act, a.resType, a.resID, fmt.Sprintf("%d", i))
 		if err != nil {
 			return fmt.Errorf("insert row %d (%s): %w", i, id, err)
 		}
 	}
 	return nil
 }
 // snapshot reads the chain head + row count, optionally invoking
 // the on-demand verifier. Verifier output goes in three additional
 // fields so the workload-side snapshot can omit them via the
 // `omitempty` tag.
 func snapshot(ctx context.Context, db *sql.DB, phase string, runVerifier bool) (*Snapshot, error) {
 	s := &Snapshot{Phase: phase}
 	if err := db.QueryRowContext(ctx, `SELECT COUNT(*) FROM audit_events`).Scan(&s.RowCount); err != nil {
 		return nil, fmt.Errorf("count(audit_events): %w", err)
 	}
 	if err := db.QueryRowContext(ctx, `SELECT row_hash FROM audit_chain_head WHERE id = 1`).Scan(&s.ChainHead); err != nil {
 		return nil, fmt.Errorf("read audit_chain_head: %w", err)
 	}
 	if runVerifier {
 		var brokenID sql.NullString
 		var brokenPos, walked int
 		err := db.QueryRowContext(ctx, `
 			SELECT first_break_id, first_break_pos, row_count
 			FROM audit_events_verify_chain()
 		`).Scan(&brokenID, &brokenPos, &walked)
 		if err != nil {
 			return nil, fmt.Errorf("audit_events_verify_chain(): %w", err)
 		}
 		if brokenID.Valid {
 			s.FirstBreakID = brokenID.String
 		}
 		s.FirstBreakPos = brokenPos
 		s.VerifierWalked = walked
 	}
 	return s, nil
 }
 // emit pretty-prints the snapshot to stdout. The trailing newline
 // from json.Encoder is the right shape for both shell `tee` and
 // python3 stdin handling.
 func emit(s *Snapshot) {
 	enc := json.NewEncoder(os.Stdout)
 	enc.SetIndent("", "  ")
 	if err := enc.Encode(s); err != nil {
 		log.Fatalf("encode snapshot: %v", err)
 	}
 }
@@ -82,16 +82,30 @@ ARG LIBEST_REF
 # is the same major version libest r3.2.0 was tested against. libest
 # also wants libcurl + libsafec; we install both via apt rather than
 # building from source for reproducibility.
-RUN apt-get update && apt-get install --no-install-recommends -y \
+#
-        autoconf \
+# Hotfix #18 (2026-05-14): wrap in a 3-retry loop with --fix-missing
-        automake \
+# fallback to absorb transient Debian mirror flakes. The original
-        build-essential \
+# unwrapped apt-get install failed CI run #N on a "Connection reset
-        ca-certificates \
+# by peer" mid-fetch of libssh2-1 from fastly's debian.org mirror at
-        git \
+# 151.101.202.132. Mirrors flake; production-grade Dockerfiles wrap
-        libcurl4-openssl-dev \
+# network ops in retry. Same pattern as the main Dockerfile's npm-ci
-        libssl-dev \
+# 3-retry loop from Hotfix #9.
-        libtool \
+RUN for i in 1 2 3; do \
-        pkg-config \
+        apt-get update && \
        apt-get install --no-install-recommends -y --fix-missing \
            autoconf \
            automake \
            build-essential \
            ca-certificates \
            git \
            libcurl4-openssl-dev \
            libssl-dev \
            libtool \
            pkg-config \
        && break; \
        echo "apt-get install attempt $i/3 failed; sleeping 5s before retry"; \
        sleep 5; \
    done \
    && rm -rf /var/lib/apt/lists/*
 WORKDIR /src
@@ -172,13 +186,22 @@ RUN git clone --depth 1 --branch ${LIBEST_REF} https://github.com/cisco/libest.g
 # Pinned to the same digest as the builder above (Bundle A / H-001).
 FROM debian:bullseye-slim@sha256:1a4701c321b1d28b1ff5f0230e766791e4b79b1d4c6c7a70064f4b297b1a330f
-RUN apt-get update && apt-get install --no-install-recommends -y \
+# Hotfix #18 (2026-05-14): same 3-retry pattern as the builder stage
-        bash \
+# above. Runtime image installs are also vulnerable to transient
-        ca-certificates \
+# mirror flakes.
-        curl \
+RUN for i in 1 2 3; do \
-        libcurl4 \
+        apt-get update && \
-        libssl1.1 \
+        apt-get install --no-install-recommends -y --fix-missing \
-        openssl \
+            bash \
            ca-certificates \
            curl \
            libcurl4 \
            libssl1.1 \
            openssl \
        && break; \
        echo "apt-get install attempt $i/3 failed; sleeping 5s before retry"; \
        sleep 5; \
    done \
    && rm -rf /var/lib/apt/lists/* \
    && useradd --create-home --uid 1000 estuser
@@ -0,0 +1,52 @@
 # loadtest-artifacts/
 > Last reviewed: 2026-05-16
 Long-term archive of k6 load-test results from the `loadtest` GitHub
 Actions workflow. TEST-005 closure (Sprint 5, 2026-05-16) introduces
 this directory as the committed home for captures the operator
 chooses to retain past GitHub's 90-day artifact-retention window.
 ## What lands here
 After a `loadtest` workflow_dispatch run, follow the procedure in
 [`docs/operator/scale-baseline-2026-Q2.md`](../../../docs/operator/scale-baseline-2026-Q2.md#capture-procedure):
 1. Download the three matrix-leg artifacts from the workflow page.
 2. Update the latest-capture table in the baseline doc with the
   extracted percentiles.
 3. Commit the raw artifacts you want long-term-retained here, named:
   ```
   2026-Q2-bulk-renewal-<run-id>.tar.gz
   2026-Q2-acme-burst-<run-id>.tar.gz
   2026-Q2-agent-storm-<run-id>.tar.gz
   ```
 4. If any single archive exceeds 100 MB, route it through `git lfs`
   (configured at repo root via `.gitattributes`).
 ## Why commit artifacts rather than rely on GHA retention
 - **GitHub Actions retains workflow artifacts for 90 days by default.**
  Acquisition-diligence reviewers looking at scale evidence months
  later get a 404 unless we keep the raw NDJSON in tree.
 - **Reproducibility.** Pinning the k6 NDJSON to a SHA makes it
  cheap to re-derive percentiles with a different filter (e.g.
  "p99 excluding the warmup ramp's first 30 seconds") without
  re-running the workflow.
 ## What does NOT belong here
 - **Per-PR ephemeral runs.** The `loadtest` workflow runs on
  workflow_dispatch + weekly cron; per-PR runs would be too noisy
  and aren't retained.
 - **Production-environment captures.** These artifacts are the
  ubuntu-latest reference baseline. An operator capturing their
  production-environment scale should put the artifacts in their
  own observability platform — committing them here would imply
  "this is what certctl's reference numbers are" which it isn't.
 - **Manual k6 captures from a developer's laptop.** Same rationale
  as the visual-regression snapshot runbook
  ([`docs/operator/runbooks/e2e-snapshot-update.md`](../../../docs/operator/runbooks/e2e-snapshot-update.md))
  — only the CI environment produces canonical numbers.
@@ -352,8 +352,35 @@ the ACME flow scenario. Operators with kind / cert-manager available
 should pair this with `make acme-cert-manager-test` for end-to-end
 verification.
 ## Scale tier (Phase 8 SCALE-H2, 2026-05-14)
 Phase 8 closure added three new k6 scenarios that exercise the
 scale-relevant load surfaces the API tier and connector tier left
 uncovered:
 | Scenario | k6 file | Seed | Make target |
 |---|---|---|---|
 | Bulk-renewal under load | `k6/bulk_renewal.js` | `seed/01_bulk_renewal_certs.sql` (10K certs) | `make loadtest-scale-bulk` |
 | ACME enrollment burst | `k6/acme_burst.js` | (none — unauth surface) | `make loadtest-scale-acme` |
 | Agent heartbeat storm | `k6/agent_storm.js` | `seed/02_agent_fleet.sql` (5K agents) | `make loadtest-scale-agent` |
 The scale-tier scenarios live behind the `scale` compose profile so
 the default `make loadtest` (API tier + connector tier, ~7 min)
 stays fast. Run all three serially with `make loadtest-scale`, or
 trigger the `loadtest.yml` workflow's `k6-scale` matrix jobs from
 the Actions tab for canonical-hardware capture.
 Operator-facing baseline table + threshold contracts + documented
 limitations live in [`docs/operator/scale.md`](../../../docs/operator/scale.md)
 under the "Scale-tier scenarios (SCALE-H2, Phase 8)" section. Treat
 that as the canonical source — this README only links.
 The seed fixtures + their idempotency contract are documented in
 [`seed/README.md`](seed/README.md).
 ## Audit references
 - API tier:       2026-05-01 issuer coverage audit fix #8.
 - Connector tier: 2026-05-02 deployment-target audit Bundle 10.
 - ACME flows:     Phase 5 master prompt (project notes).
 - Scale tier:     2026-05-14 architecture diligence Phase 8 (SCALE-H2).
@@ -351,3 +351,128 @@ services:
      - run
      - --summary-export=/results/summary.json
      - /scripts/k6.js
  # ===========================================================================
  # Phase 8 SCALE-H2 — scale-tier scenarios (opt-in via `--profile scale`).
  #
  # The default `make loadtest` path runs the API tier + connector tier
  # scenarios above against the demo-scale seed. The Phase 8 scenarios are
  # heavier (10K cert + 5K agent fixtures) and would slow the default path
  # without serving the per-PR signal the existing run targets, so they live
  # behind a separate compose profile.
  #
  # Three components, all profile-gated:
  #   1. scale-seed    — one-shot init that runs ./seed/*.sql against the
  #                      same postgres the server uses. Idempotent.
  #   2. k6-scale-bulk / k6-scale-acme / k6-scale-agent — one driver each
  #                      for the three Phase 8 scenarios. The matrix dispatch
  #                      in .github/workflows/loadtest.yml picks one per job.
  #
  # Run a single scale scenario locally:
  #   docker compose --profile scale up \
  #       --abort-on-container-exit --exit-code-from k6-scale-bulk \
  #       scale-seed k6-scale-bulk
  # ===========================================================================
  scale-seed:
    # postgres:16-alpine bundles psql; no extra image needed.
    image: postgres:16-alpine
    container_name: certctl-loadtest-scale-seed
    restart: "no"
    profiles: ["scale"]
    depends_on:
      postgres:
        condition: service_healthy
      # Wait for certctl-server to be healthy — the server runs schema
      # migrations + seed_demo.sql at boot. The Phase 8 seeds reference
      # FKs (iss-local, o-alice, t-platform, rp-standard) that
      # seed_demo.sql creates, so the order MUST be:
      #   postgres up → server runs migrations + seed_demo.sql → scale-seed runs
      certctl-server:
        condition: service_healthy
    environment:
      PGHOST: postgres
      PGUSER: certctl
      PGPASSWORD: loadtestpass
      PGDATABASE: certctl
    volumes:
      - ./seed:/seed:ro
    entrypoint: /bin/sh
    command:
      - -c
      - |
        set -eu
        echo "==> Phase 8 scale-seed: running SQL fixtures (lexical order)"
        for f in /seed/*.sql; do
            echo "----> $$f"
            psql -v ON_ERROR_STOP=1 -f "$$f"
        done
        echo "==> Phase 8 scale-seed: complete"
  k6-scale-bulk:
    image: grafana/k6:0.54.0
    container_name: certctl-loadtest-k6-bulk
    profiles: ["scale"]
    depends_on:
      certctl-server:
        condition: service_healthy
      scale-seed:
        condition: service_completed_successfully
    environment:
      CERTCTL_BASE: https://certctl-server:8443
      CERTCTL_TOKEN: load-test-token
      K6_INSECURE_SKIP_TLS_VERIFY: "true"
    volumes:
      - ./k6/bulk_renewal.js:/scripts/bulk_renewal.js:ro
      - ./results:/results
    command:
      - run
      - --summary-export=/results/summary-bulk-renewal.json
      - /scripts/bulk_renewal.js
  k6-scale-acme:
    image: grafana/k6:0.54.0
    container_name: certctl-loadtest-k6-acme
    profiles: ["scale"]
    depends_on:
      certctl-server:
        condition: service_healthy
      # ACME scenario doesn't depend on the SQL seeds (it hits the
      # unauthenticated directory + nonce + ARI surface) but routing
      # it through the same dependency chain keeps the compose
      # ordering predictable across the three scale jobs.
      scale-seed:
        condition: service_completed_successfully
    environment:
      CERTCTL_ACME_DIRECTORY: https://certctl-server:8443/acme/profile/prof-test/directory
      K6_INSECURE_SKIP_TLS_VERIFY: "true"
    volumes:
      - ./k6/acme_burst.js:/scripts/acme_burst.js:ro
      - ./results:/results
    command:
      - run
      - --summary-export=/results/summary-acme-burst.json
      - /scripts/acme_burst.js
  k6-scale-agent:
    image: grafana/k6:0.54.0
    container_name: certctl-loadtest-k6-agent
    profiles: ["scale"]
    depends_on:
      certctl-server:
        condition: service_healthy
      scale-seed:
        condition: service_completed_successfully
    environment:
      CERTCTL_BASE: https://certctl-server:8443
      CERTCTL_TOKEN: load-test-token
      K6_INSECURE_SKIP_TLS_VERIFY: "true"
      # Match the seed's 5K-agent fleet.
      K6_AGENT_FLEET: "5000"
    volumes:
      - ./k6/agent_storm.js:/scripts/agent_storm.js:ro
      - ./results:/results
    command:
      - run
      - --summary-export=/results/summary-agent-storm.json
      - /scripts/agent_storm.js
@@ -0,0 +1,183 @@
 // Phase 8 SCALE-H2 — ACME enrollment burst.
 //
 // What this measures:
 //   200 concurrent VUs hammering the unauthenticated ACME directory
 //   + new-nonce + ARI surface for 5 minutes. The goal is the
 //   throughput ceiling for the entry-point handlers and the
 //   per-account rate-limit response shape Phase 5 added (RFC 8555
 //   §6.7 + RFC 7807 + the certctl-specific
 //   ErrACMEConcurrentOrdersExceeded path).
 //
 // What this does NOT measure (and why):
 //   - JWS-signed POST flows (new-account, new-order, finalize).
 //     k6 doesn't ship JWS, and bundling a Go signing helper into
 //     the k6 container would obscure the server-side latency the
 //     scenario is trying to pin. The existing
 //     `deploy/test/loadtest/k6/acme_flow.js` Phase 5 scenario
 //     made the same explicit trade-off; this Phase 8 burst scenario
 //     reuses the constraint. End-to-end JWS-signed conformance is
 //     gated by `make acme-rfc-conformance-test` (which uses lego
 //     against the same compose stack).
 //   - The actual order/finalize hot path. The newOrder handler's
 //     constant-time SCAN against acme_orders + the per-account
 //     concurrent-orders gate ARE useful to load-test, but require
 //     valid JWS to reach. The directory + new-nonce surface this
 //     scenario hits is what every ACME client transits BEFORE the
 //     signed flow — measuring it pins the server's headroom for
 //     the rest of the flow.
 //   - Issuer-side enrollment latency (DigiCert ACME, Let's Encrypt
 //     against a real prod CA, etc.). Same "load-testing someone
 //     else's API" carve-out as the API tier.
 //
 // What this DOES measure:
 //   - GET /acme/profile/{id}/directory throughput. Sustained 200
 //     concurrent VUs at a low per-VU sleep produces ~600-1000 req/s
 //     against this endpoint, well above what any production ACME
 //     client would generate but the right shape for finding the
 //     ceiling.
 //   - HEAD /acme/profile/{id}/new-nonce throughput. Nonce
 //     allocation is a hot path that writes one row to acme_nonces.
 //   - GET /acme/profile/{id}/renewal-info/{cert-id} 4xx fast path.
 //     Synthetic cert-id → handler returns 4xx without a DB lookup
 //     (cert-id is malformed at the parse layer). Measures the
 //     handler-front overhead under load.
 //   - 429 rate-limit response shape. The Phase 5 ACME per-account
 //     rate limit fires at sustained spike rates; the scenario pins
 //     that the 429 body is RFC 7807 with the
 //     "urn:ietf:params:acme:error:rateLimited" type. A regression
 //     that returned a plain text 429 or a different problem type
 //     would break ACME clients hard.
 //
 // Threshold contract:
 //   - directory p95 < 500ms, new-nonce p95 < 300ms, renewal-info
 //     p95 < 800ms — same as the Phase 5 acme_flow.js baselines.
 //   - 429 responses are EXPECTED at sustained 200 VU rate (the
 //     server's RFC-compliant rate limiter SHOULD kick in). The
 //     http_req_failed metric is tagged separately so 429s don't
 //     break the threshold; a separate `rate_limited` Counter
 //     tracks them so the operator can see how often the limiter
 //     fires.
 import http from 'k6/http';
 import { check } from 'k6';
 import { Counter, Trend } from 'k6/metrics';
 import { textSummary } from 'https://jslib.k6.io/k6-summary/0.0.2/index.js';
 const ACME_BASE = __ENV.CERTCTL_ACME_DIRECTORY ||
    'https://certctl-server:8443/acme/profile/prof-test/directory';
 // Custom metrics.
 const directoryDuration = new Trend('acme_directory_duration', true);
 const newNonceDuration  = new Trend('acme_new_nonce_duration', true);
 const renewalInfoDuration = new Trend('acme_renewal_info_duration', true);
 const rateLimitedCount  = new Counter('acme_rate_limited_count');
 const rateLimitShapeOK  = new Counter('acme_rate_limit_shape_ok');
 export const options = {
    scenarios: {
        acme_burst: {
            executor: 'constant-vus',
            vus: parseInt(__ENV.K6_ACME_VUS || '200', 10),
            duration: __ENV.K6_ACME_DURATION || '5m',
            gracefulStop: '30s',
            tags: { scenario: 'acme_burst' },
        },
    },
    thresholds: {
        'acme_directory_duration':    ['p(95)<500'],
        'acme_new_nonce_duration':    ['p(95)<300'],
        'acme_renewal_info_duration': ['p(95)<800'],
        // 4xx (rate-limited or malformed-cert-id) is expected; 5xx is
        // not. Filter to status >= 500 for the failure floor.
        'http_req_failed{scenario:acme_burst,server_error:true}': ['rate<0.001'],
    },
    insecureSkipTLSVerify: true,
    summaryTrendStats: ['avg', 'min', 'med', 'p(95)', 'p(99)', 'max'],
 };
 export default function () {
    // Step 1 — directory.
    let res = http.get(ACME_BASE, {
        tags: { scenario: 'acme_burst', step: 'directory' },
    });
    directoryDuration.add(res.timings.duration);
    check(res, { 'directory 200': (r) => r.status === 200 });
    if (res.status === 429) {
        recordRateLimit(res);
        return; // backoff this VU iteration
    }
    if (res.status !== 200) return;
    const dir = res.json();
    // Step 2 — new-nonce.
    if (dir.newNonce) {
        res = http.head(dir.newNonce, {
            tags: { scenario: 'acme_burst', step: 'new_nonce' },
        });
        newNonceDuration.add(res.timings.duration);
        if (res.status === 429) {
            recordRateLimit(res);
            return;
        }
        check(res, {
            'new-nonce 200': (r) => r.status === 200,
            'replay-nonce header present': (r) => !!r.headers['Replay-Nonce'],
        });
    }
    // Step 3 — ARI synthetic 4xx fast path. Phase 4 added ARI
    // (RFC 9773); this exercises the malformed-cert-id branch which
    // returns a 4xx without a DB lookup. Pinning this here means a
    // regression that turned the malformed path into a DB query
    // would surface as a p95 spike.
    if (dir.renewalInfo) {
        res = http.get(dir.renewalInfo + '/aaaa.bbbb', {
            tags: { scenario: 'acme_burst', step: 'renewal_info' },
        });
        renewalInfoDuration.add(res.timings.duration);
        if (res.status === 429) {
            recordRateLimit(res);
            return;
        }
        check(res, {
            'renewal-info 4xx for synthetic cert-id':
                (r) => r.status === 400 || r.status === 404,
        });
    }
 }
 // recordRateLimit pins the Phase 5 ACME rate-limit response shape:
 //   - HTTP 429
 //   - Content-Type: application/problem+json
 //   - Body: {"type":"urn:ietf:params:acme:error:rateLimited", ...}
 // A regression that returned 503 or a plain-text 429 or a different
 // problem type would NOT increment acme_rate_limit_shape_ok and the
 // operator would see (rate_limited_count - shape_ok_count) > 0 in
 // the summary.
 function recordRateLimit(res) {
    rateLimitedCount.add(1);
    const ct = res.headers['Content-Type'] || '';
    if (!ct.includes('application/problem+json')) {
        return;
    }
    let body;
    try {
        body = res.json();
    } catch (e) {
        return;
    }
    if (body && typeof body.type === 'string' &&
        body.type.startsWith('urn:ietf:params:acme:error:rateLimited')) {
        rateLimitShapeOK.add(1);
    }
 }
 export function handleSummary(data) {
    return {
        '/results/summary-acme-burst.json': JSON.stringify(data, null, 2),
        '/results/summary-acme-burst.txt': textSummary(data, { indent: ' ', enableColors: false }),
        stdout: textSummary(data, { indent: ' ', enableColors: true }),
    };
 }
@@ -0,0 +1,126 @@
 // Phase 8 SCALE-H2 — agent fleet heartbeat storm.
 //
 // What this measures:
 //   5,000 agents heartbeating at 30s intervals = ~167 heartbeats/sec
 //   sustained. Each heartbeat is POST /api/v1/agents/{id}/heartbeat
 //   with optional metadata. Pre-seeded fleet provided by
 //   deploy/test/loadtest/seed/02_agent_fleet.sql.
 //
 // What this does NOT measure:
 //   - The agent work-poll path (GET /api/v1/agents/{id}/work). The
 //     heartbeat hot path is the highest-frequency call on a typical
 //     fleet (work-poll cadence is 30s default like heartbeat, but
 //     work-poll returns the empty set 99% of the time and is cheap;
 //     heartbeat does an UPDATE on every call). v2 of the harness
 //     could combine them.
 //   - The agent CSR-submit path (POST /api/v1/agents/{id}/csr). That
 //     fires on per-cert issuance, not per heartbeat, and is exercised
 //     by the existing API tier's POST /api/v1/certificates scenario.
 //   - Auth-key per-agent rotation. The loadtest stack runs with a
 //     single api-key (`load-test-token`); per-agent api-key
 //     hashing/rotation isn't a load axis.
 //
 // Why constant-arrival-rate (not constant-vus):
 //   The point is to model what 5K real agents would offer the server
 //   at their native cadence. 5K agents * (1 heartbeat / 30s) =
 //   166.67 req/s offered. constant-arrival-rate fires at exactly
 //   that rate regardless of latency; if the server backpressures,
 //   queue builds and p99 shows it. constant-vus would let slow
 //   responses block, masking the actual ceiling.
 //
 // Threshold contract:
 //   - p99 < 1s for the heartbeat POST. The handler does an UPDATE on
 //     agents.last_heartbeat_at (+ optional metadata columns) and an
 //     RBAC check. Even at 200 req/s a tight UPDATE on an indexed
 //     primary key should stay sub-second.
 //   - p95 < 500ms.
 //   - Error rate < 0.1%. The seeded agents are all status='Online'
 //     so no 410 Gone (retired-agent) responses; anything 4xx is a
 //     bug. 5xx is a server health regression.
 //
 // Phase 8 reference:
 //   - Source finding: SCALE-H2.
 //   - Pre-state: heartbeat path not load-tested. The 100-agent demo
 //     seed in seed_demo.sql produces ~3 heartbeats/sec, orders of
 //     magnitude below fleet scale.
 import http from 'k6/http';
 import { check } from 'k6';
 import { textSummary } from 'https://jslib.k6.io/k6-summary/0.0.2/index.js';
 const BASE  = __ENV.CERTCTL_BASE  || 'https://certctl-server:8443';
 const TOKEN = __ENV.CERTCTL_TOKEN || 'load-test-token';
 // 5000 agents * (1 / 30s) = 166.67 heartbeats/sec. Round to 167.
 const TARGET_RATE = parseInt(__ENV.K6_AGENT_RATE || '167', 10);
 // Total agents in the fleet seed. The k6 scenario picks an agent at
 // random per iteration (deterministic via __ITER) to spread the
 // per-row UPDATE pressure across the table.
 const FLEET_SIZE = parseInt(__ENV.K6_AGENT_FLEET || '5000', 10);
 export const options = {
    scenarios: {
        agent_storm: {
            executor: 'constant-arrival-rate',
            rate: TARGET_RATE,
            timeUnit: '1s',
            duration: '5m',
            preAllocatedVUs: 50,
            maxVUs: 200,
            exec: 'heartbeat',
            tags: { scenario: 'agent_storm' },
        },
    },
    thresholds: {
        'http_req_duration{scenario:agent_storm}': ['p(99)<1000', 'p(95)<500'],
        'http_req_failed{scenario:agent_storm}': ['rate<0.001'],
    },
    summaryTrendStats: ['avg', 'min', 'med', 'p(95)', 'p(99)', 'max'],
    insecureSkipTLSVerify: true,
 };
 // agentID returns a deterministic agent id from the loadtest fleet
 // seed. Spreading round-robin across the fleet means the UPDATE
 // pressure hits every row equally rather than the same hot row over
 // and over.
 function agentID() {
    // __ITER is k6's per-VU iteration counter; combined with __VU
    // (the VU index) we get a unique-per-call number that spans
    // 0..FLEET_SIZE on the modulo.
    const idx = (__VU * 1000 + __ITER) % FLEET_SIZE;
    return 'ag-loadtest-' + String(idx + 1).padStart(5, '0');
 }
 export function heartbeat() {
    const id = agentID();
    // Optional metadata; the heartbeat handler tolerates an empty body
    // (no metadata) but real agents send their version + hostname on
    // every call so we include them here.
    const payload = JSON.stringify({
        version: '2.1.0',
        hostname: 'loadtest-' + id.slice(-5) + '.fleet.example.test',
        os: 'linux',
        architecture: 'amd64',
    });
    const res = http.post(`${BASE}/api/v1/agents/${id}/heartbeat`, payload, {
        headers: {
            'Content-Type': 'application/json',
            'Authorization': `Bearer ${TOKEN}`,
        },
        tags: { scenario: 'agent_storm' },
    });
    check(res, {
        'heartbeat 2xx': (r) => r.status >= 200 && r.status < 300,
    });
 }
 export function handleSummary(data) {
    return {
        '/results/summary-agent-storm.json': JSON.stringify(data, null, 2),
        '/results/summary-agent-storm.txt': textSummary(data, { indent: ' ', enableColors: false }),
        stdout: textSummary(data, { indent: ' ', enableColors: true }),
    };
 }
@@ -0,0 +1,129 @@
 // Phase 8 SCALE-H2 — bulk-renewal under load.
 //
 // What this measures:
 //   POST /api/v1/certificates/bulk-renew throughput against a
 //   10K-cert pre-seeded fleet. Each iteration POSTs a criteria-mode
 //   bulk-renew request scoped to a subset of the seeded fleet (by
 //   tag) so the server enqueues N renewal jobs and returns a
 //   per-cert {certificate_id, job_id} envelope.
 //
 // Why criteria-mode (not certificate-ids mode):
 //   The seeded fleet has a stable `tags.batch = 'bulk-renewal'`
 //   marker. Criteria-mode lets the scenario re-fire without
 //   maintaining a moving list of cert IDs and still scopes the
 //   action to the Phase 8 fixture (no risk of touching a real
 //   tenant's certs if someone runs the scenario against a non-
 //   loadtest server by mistake — the criteria simply matches
 //   nothing).
 //
 // What this does NOT measure:
 //   - The scheduler's renewal scan itself. The bulk-renew handler
 //     enqueues issuance jobs synchronously into the `jobs` table;
 //     the scheduler's `jobProcessorLoop` picks them up on its next
 //     tick. The DB write throughput is what's measured here; the
 //     job-execution path is bounded by per-issuer concurrency
 //     (CERTCTL_RENEWAL_CONCURRENCY=25 default) and isn't usefully
 //     amplified by adding more inbound bulk-renew calls.
 //   - Full POST → poll deployments → cert-served loop. Same v1/v2
 //     deferral as the connector-tier scenarios — needs the agent
 //     poll surface plumbed end-to-end.
 //
 // Threshold contract:
 //   - p99 < 5s, p95 < 2s for the bulk-renew POST. Each call walks
 //     the criteria, materializes the matching managed_certificates
 //     rows, inserts N rows into `jobs`, and returns the envelope.
 //   - Error rate < 1%. Anything 4xx/5xx counts.
 //
 // Phase 8 reference:
 //   - Source finding: SCALE-H2.
 //   - Pre-state: only the API tier (50 req/s POST /certificates +
 //     GET /certificates) and connector tier (per-target handshake)
 //     were measured. The bulk-renew hot path was uncovered.
 //   - Seed: deploy/test/loadtest/seed/01_bulk_renewal_certs.sql
 //     creates 10K rows with tags.batch='bulk-renewal'. The seed
 //     must run before this scenario; the scale-seed compose
 //     profile gates this.
 import http from 'k6/http';
 import { check } from 'k6';
 import { textSummary } from 'https://jslib.k6.io/k6-summary/0.0.2/index.js';
 const BASE  = __ENV.CERTCTL_BASE  || 'https://localhost:8443';
 const TOKEN = __ENV.CERTCTL_TOKEN || 'load-test-token';
 // Sustained throughput target. constant-arrival-rate at 5 req/s for 5
 // minutes = 1500 bulk-renew POSTs. Each POST touches up to 10K
 // managed_certificates rows (criteria scan) + inserts up to 10K
 // rows into `jobs`, so the offered load is higher than the API
 // tier's 50 req/s on raw queries-per-second but the per-call
 // cost is larger.
 //
 // 5 req/s was picked deliberately:
 //   - 50 req/s combined with the API tier's 50 saturates the demo-
 //     scale compose's DB pool (CERTCTL_DATABASE_MAX_CONNS=50). The
 //     Phase 8 scenario should measure the per-call ceiling without
 //     fighting the pool.
 //   - Each call enqueues thousands of jobs; the scheduler's
 //     jobProcessorLoop has finite per-tick budget. Pushing higher
 //     than 5 req/s would queue work faster than the scheduler
 //     drains it, which produces a transient backlog metric (worth
 //     measuring eventually) but isn't what SCALE-H2 asks for.
 export const options = {
    scenarios: {
        bulk_renewal: {
            executor: 'constant-arrival-rate',
            rate: 5,
            timeUnit: '1s',
            duration: '5m',
            preAllocatedVUs: 10,
            maxVUs: 30,
            exec: 'bulkRenewal',
            tags: { scenario: 'bulk_renewal' },
        },
    },
    thresholds: {
        // Single-scenario threshold — narrower than the API tier
        // because each call is heavier (DB scan + N inserts).
        'http_req_duration{scenario:bulk_renewal}': ['p(99)<5000', 'p(95)<2000'],
        'http_req_failed{scenario:bulk_renewal}': ['rate<0.01'],
    },
    summaryTrendStats: ['avg', 'min', 'med', 'p(95)', 'p(99)', 'max'],
    insecureSkipTLSVerify: true,
 };
 export function bulkRenewal() {
    // Scope by team_id — the seed binds every loadtest cert to
    // t-platform; in a production-multi-tenant deploy, team scoping
    // is the typical bulk-renew shape. This exercises the criteria
    // walker AND the team-scoped permission check in the handler.
    //
    // NOTE: this does NOT include `tags` because the BulkRenewalCriteria
    // domain type (handler/bulk_renewal.go) only exposes profile_id,
    // owner_id, agent_id, issuer_id, team_id, certificate_ids — not
    // tag-based filtering. The team_id scope plus the production-
    // separated FK guarantees we only touch the Phase 8 seed.
    const payload = JSON.stringify({
        team_id: 't-platform',
        issuer_id: 'iss-local',
    });
    const res = http.post(`${BASE}/api/v1/certificates/bulk-renew`, payload, {
        headers: {
            'Content-Type': 'application/json',
            'Authorization': `Bearer ${TOKEN}`,
        },
        tags: { scenario: 'bulk_renewal' },
    });
    check(res, {
        'bulk-renew 2xx': (r) => r.status >= 200 && r.status < 300,
    });
 }
 export function handleSummary(data) {
    return {
        '/results/summary-bulk-renewal.json': JSON.stringify(data, null, 2),
        '/results/summary-bulk-renewal.txt': textSummary(data, { indent: ' ', enableColors: false }),
        stdout: textSummary(data, { indent: ' ', enableColors: true }),
    };
 }
@@ -0,0 +1,85 @@
 -- Phase 8 SCALE-H2: bulk-renewal scenario seed.
 --
 -- Generates 10,000 managed_certificates rows linked to the existing
 -- seed_demo.sql FKs (iss-local, o-alice, t-platform, rp-standard) so
 -- the bulk-renewal k6 scenario can POST /api/v1/certificates/bulk-renew
 -- against a fleet-scale dataset instead of the 15-row demo seed.
 --
 -- Behavior:
 --   - Idempotent. ON CONFLICT (name) DO NOTHING — re-running the seed
 --     against an already-seeded DB is a no-op.
 --   - expires_at is uniformly distributed across the next 30 days so
 --     a renewal_window_days = 30 policy considers every row eligible.
 --   - status = 'active' so the renewal selector treats them as
 --     live (the scheduler skips status IN ('pending', 'failed',
 --     'revoked', 'retired')).
 --   - name is generated as 'loadtest-bulk-NNNNN.example.test' for a
 --     stable, predictable identifier the k6 scenario can pattern-match
 --     to scope its criteria to the seeded set (the production fleet
 --     wouldn't share this prefix).
 --
 -- Volume target: 10,000 rows. Insert wall time on the loadtest stack
 -- (postgres:16-alpine, 2 CPU / 4 GiB): typically < 5 seconds via the
 -- single-statement generate_series + INSERT pattern below. The
 -- compose seed-init container runs this BEFORE the k6 driver starts,
 -- so the steady-state load measurement isn't affected by seed time.
 --
 -- Why not generated in Go via a fixtures helper:
 --   - The certctl-server boots from a clean DB and runs migrations +
 --     seed_demo.sql automatically when CERTCTL_DEMO_SEED=true. Adding
 --     a Go-side fixtures helper would require either (a) a new
 --     CERTCTL_LOADTEST_SEED flag wired into cmd/server/main.go (cross-
 --     cutting change for one test path) or (b) a separate seed binary
 --     (more compose surface). Raw SQL is the smallest viable change.
 --
 -- Phase 8 entry point — runs only when the loadtest compose stack is
 -- explicitly opted into the scale-seed via LOADTEST_SCALE_SEED=true.
 INSERT INTO managed_certificates (
    id,
    name,
    common_name,
    sans,
    environment,
    owner_id,
    team_id,
    issuer_id,
    renewal_policy_id,
    status,
    expires_at,
    tags,
    created_at,
    updated_at
 )
 SELECT
    'cert-loadtest-bulk-' || lpad(g::text, 5, '0'),
    'loadtest-bulk-' || lpad(g::text, 5, '0') || '.example.test',
    'loadtest-bulk-' || lpad(g::text, 5, '0') || '.example.test',
    ARRAY['loadtest-bulk-' || lpad(g::text, 5, '0') || '.example.test'],
    'loadtest',
    'o-alice',
    't-platform',
    'iss-local',
    'rp-standard',
    'active',
    -- Distribute expires_at uniformly across the next 30 days so a
    -- 30-day-window renewal policy sees every row as eligible.
    NOW() + ((g % 30) || ' days')::interval + ((g % 24) || ' hours')::interval,
    jsonb_build_object('source', 'loadtest-phase8', 'batch', 'bulk-renewal'),
    NOW(),
    NOW()
 FROM generate_series(1, 10000) AS g
 ON CONFLICT (name) DO NOTHING;
 -- Confirmation row count — the seed-init container greps this in its
 -- logs to verify the fleet shape post-insert. The output appears in
 -- `docker compose logs certctl-loadtest-scale-seed` after the run.
 DO $$
 DECLARE
    cert_count integer;
 BEGIN
    SELECT COUNT(*) INTO cert_count
    FROM managed_certificates
    WHERE name LIKE 'loadtest-bulk-%';
    RAISE NOTICE 'Phase 8 bulk-renewal seed: % managed_certificates rows present', cert_count;
 END $$;
@@ -0,0 +1,85 @@
 -- Phase 8 SCALE-H2: agent-fleet heartbeat-storm scenario seed.
 --
 -- Generates 5,000 agents rows so the heartbeat-storm k6 scenario can
 -- model a fleet-scale heartbeat pattern (5K agents heartbeating at the
 -- native 30s cadence = ~167 heartbeats/sec sustained) instead of the
 -- ~10-agent demo seed.
 --
 -- Behavior:
 --   - Idempotent. ON CONFLICT (id) DO NOTHING — re-runnable against an
 --     already-seeded DB.
 --   - name is unique (a UNIQUE constraint in migration 000001) so the
 --     name suffix mirrors the id suffix.
 --   - status = 'Online' so the heartbeat handler's retire-check
 --     (service.ErrAgentRetired) doesn't 410 the storm.
 --   - last_heartbeat_at staggered across the prior 60 seconds so the
 --     stale-agent reaper (agentHealthCheckLoop) doesn't immediately
 --     flip half the fleet to 'Offline' during the first scheduler
 --     tick of the load run.
 --   - api_key_hash = 'loadtest_no_auth'. The loadtest compose runs
 --     CERTCTL_AUTH_TYPE=api-key with a single static token
 --     (load-test-token), which bypasses per-agent key check the same
 --     way the existing API tier scenarios do. Production deploys with
 --     CERTCTL_AUTH_TYPE=agent-key per-agent would seed real bcrypt'd
 --     hashes; this column is opaque to the load-test path.
 --   - registered_at = NOW() - random 1-90 day interval so agent age
 --     looks realistic and any age-based query plans are exercised.
 --
 -- Volume target: 5,000 rows. The agents schema is much narrower than
 -- managed_certificates so the insert is sub-second on the loadtest
 -- stack. The 5K agents do not own any deployment_targets in this
 -- fixture (the scenario only measures the heartbeat hot path, not
 -- the work-poll path which depends on cert + target wiring).
 --
 -- Phase 8 entry point — runs only when the loadtest compose stack is
 -- explicitly opted into the scale-seed via LOADTEST_SCALE_SEED=true.
 INSERT INTO agents (
    id,
    name,
    hostname,
    status,
    last_heartbeat_at,
    registered_at,
    api_key_hash,
    os,
    architecture,
    ip_address,
    version
 )
 SELECT
    'ag-loadtest-' || lpad(g::text, 5, '0'),
    'loadtest-agent-' || lpad(g::text, 5, '0'),
    'loadtest-' || lpad(g::text, 5, '0') || '.fleet.example.test',
    'Online',
    -- Stagger last_heartbeat_at across the prior 60 seconds (= 2x the
    -- agent's native poll interval) so the first wave of incoming
    -- heartbeats doesn't all arrive in lockstep at t=0.
    NOW() - ((g % 60) || ' seconds')::interval,
    -- Registered_at randomized 1-90 days back.
    NOW() - ((g % 90 + 1) || ' days')::interval,
    'loadtest_no_auth',
    -- Mix linux/windows/darwin so the OS distribution column in the
    -- agents page isn't pure-linux during the storm.
    CASE (g % 10)
        WHEN 0 THEN 'windows'
        WHEN 1 THEN 'darwin'
        ELSE 'linux'
    END,
    -- amd64 dominates; arm64 minority.
    CASE WHEN (g % 5) = 0 THEN 'arm64' ELSE 'amd64' END,
    -- IPv4 in the 10.42.0.0/16 fleet range, deterministic per id.
    '10.42.' || ((g / 256) % 256)::text || '.' || (g % 256)::text,
    '2.1.0'
 FROM generate_series(1, 5000) AS g
 ON CONFLICT (id) DO NOTHING;
 DO $$
 DECLARE
    agent_count integer;
 BEGIN
    SELECT COUNT(*) INTO agent_count
    FROM agents
    WHERE id LIKE 'ag-loadtest-%';
    RAISE NOTICE 'Phase 8 agent-storm seed: % agents rows present', agent_count;
 END $$;
@@ -0,0 +1,87 @@
 # Phase 8 load-test seed fixtures
 Opt-in seed scripts that grow the loadtest DB from the demo-scale
 fixture (~15 certs / ~10 agents from `migrations/seed_demo.sql`) to
 fleet scale (10K certs + 5K agents) so the Phase 8 SCALE-H2 scenarios
 measure something representative.
 ## When these run
 The default `make loadtest` path does NOT touch this directory — the
 API tier and connector tier scenarios run against the demo seed alone
 and complete in ~5 minutes. The Phase 8 scenarios opt-in via the
 `LOADTEST_SCALE_SEED=true` environment variable; when set, the
 `certctl-loadtest-scale-seed` one-shot init container runs every
 `*.sql` file in this directory in lexical order against the same
 Postgres instance the server uses.
 Compose service wiring (see `../docker-compose.yml`):
 - Service: `scale-seed`
 - Profile: `scale-seed` (compose `profiles:` gate; not started by
  default)
 - Depends on: `postgres` (service_healthy) AND `certctl-server`
  (service_healthy — server runs schema migrations at boot so the
  seed runs AFTER tables exist)
 - Order: lexical (`01_bulk_renewal_certs.sql` then
  `02_agent_fleet.sql`)
 - Idempotent: every script uses `ON CONFLICT DO NOTHING` so re-running
  is a no-op.
 ## What gets seeded
 | File | Rows | Purpose |
 |---|---|---|
 | `01_bulk_renewal_certs.sql` | 10,000 managed_certificates | Fleet shape for `bulk_renewal.js`. All linked to demo FKs (iss-local, o-alice, t-platform, rp-standard). Status `active`, expires_at distributed across the next 30 days so a 30-day renewal window considers every row eligible. Name prefix `loadtest-bulk-` so the k6 scenario can scope its bulk-renew criteria. |
 | `02_agent_fleet.sql` | 5,000 agents | Fleet shape for `agent_storm.js`. Status `Online`, last_heartbeat_at staggered across prior 60s, name prefix `loadtest-agent-`. OS distribution: 80% linux / 10% windows / 10% darwin. Arch: 80% amd64 / 20% arm64. |
 ## How to run the Phase 8 scenarios locally
 ```bash
 cd deploy/test/loadtest
 LOADTEST_SCALE_SEED=true docker compose --profile scale-seed up --build \
    --abort-on-container-exit --exit-code-from k6-scale
 ```
 Or via the dedicated Makefile target (preferred for CI parity):
 ```bash
 make loadtest-scale
 ```
 ## Why SQL fixtures instead of a Go seed binary
 - The certctl-server already boots from a clean DB and runs migrations
  + `seed_demo.sql` when `CERTCTL_DEMO_SEED=true`. Adding a third seed
  mode (loadtest-scale) would mean either a new
  `CERTCTL_LOADTEST_SEED` flag wired into `cmd/server/main.go` (cross-
  cutting change for one test path) or a separate seed binary (more
  compose surface).
 - Raw SQL is the smallest viable change: each script is a single
  multi-row `INSERT … SELECT FROM generate_series(…)` plus a
  `DO $$ … RAISE NOTICE` confirmation block.
 - Idempotency is straightforward via `ON CONFLICT … DO NOTHING` — the
  same pattern `seed_demo.sql` uses.
 ## Why these volumes specifically
 - **10K certs.** The SCALE-H2 audit asked for "10K certs with
  renewal_at < now." Round number, fits in postgres:16-alpine on a
  CI runner without OOM, and large enough that the renewal selector's
  query plan is exercised (the demo's 15 rows would index-scan
  trivially).
 - **5K agents.** Heartbeat at 30s cadence = ~167 heartbeats/sec
  sustained. That's well above the 50 req/s the existing API tier
  measures and stresses the agent.heartbeat handler's per-call cost
  (last_heartbeat_at UPDATE + the RBAC permission check + the
  audit-log row).
 If a future scenario needs more rows (50K certs / 10K agents), add a
 new `03_…sql` here and another scenario file. Don't grow the existing
 files — re-running existing scenarios against a different fixture
 shape would invalidate the captured baseline.
 ## Phase 8 audit reference
 Source finding: SCALE-H2 in
 `cowork/certctl-architecture-diligence-audit.html`.
 Phase 8 closure commit: see `git log --grep='Phase 8'`.
@@ -1,6 +1,6 @@
 # certctl Documentation
-> Last reviewed: 2026-05-05
+> Last reviewed: 2026-05-12
 The full docs index, organized by audience. Pick the section that matches what you need to do; each link below opens a focused doc rather than a wall of text.
@@ -65,6 +65,8 @@ You're running certctl in production and need operational guidance.
 | Doc | What it covers |
 |---|---|
 | [Security posture](operator/security.md) | Auth, rate limits, encryption at rest, key rotation, RBAC + OIDC + sessions + break-glass, bootstrap |
 | [Secret custody](operator/secret-custody.md) | Where private keys live; FileDriver vs HSM/KMS; encryption wire format; env-seeded vs DB-seeded plaintext policy |
 | [Observability](operator/observability.md) | Metrics surface, Prometheus exposition vs client_golang, tracing scope, log structure, rate-limit semantics across restarts/replicas |
 | [RBAC operator reference](operator/rbac.md) | Roles, permissions, scopes, scope-down + day-0 bootstrap |
 | [Auth threat model](operator/auth-threat-model.md) | API-key + RBAC + OIDC + sessions + break-glass — token forgery, session hijacking, IdP compromise, role-grant abuse, bootstrap-token leak, audit-mutation |
 | [OIDC / SSO runbooks](operator/oidc-runbooks/index.md) | Per-IdP setup guides — Keycloak, Authentik, Okta, Auth0, Entra ID, Google Workspace |
@@ -83,6 +85,8 @@ You're running certctl in production and need operational guidance.
 | [Cloud targets](operator/runbooks/cloud-targets.md) | AWS ACM + Azure Key Vault deployment, debugging, rollback |
 | [Expiry alerts](operator/runbooks/expiry-alerts.md) | Per-policy multi-channel routing matrix, severity tiers |
 | [Disaster recovery](operator/runbooks/disaster-recovery.md) | CRL cache, OCSP responder cert, CA private-key rotation, Postgres restore |
 | [Config-encryption upgrade](operator/runbooks/config-encryption-upgrade.md) | Force v1/v2 → v3 re-seal across the database; passphrase rotation procedure |
 | [PostgreSQL backup](operator/runbooks/postgres-backup.md) | Operator-run backup recipe (docker-compose + Kubernetes); recommended cadence; quarterly DR dry-run |
 ## Migration
@@ -112,6 +116,7 @@ You're contributing to certctl, running tests locally, or trying to understand t
 | [GUI QA checklist](contributor/gui-qa-checklist.md) | Manual GUI verification pass for release |
 | [Release sign-off](contributor/release-sign-off.md) | Release-day checklist — code state, automated gates, manual QA, artefact verification |
 | [CI pipeline](contributor/ci-pipeline.md) | CI shape, regression guards, adding new checks |
 | [CI guards](contributor/ci-guards.md) | Per-class CI guards (code-shape, contract-parity, build/dep, operational); how to add one |
 ## Archive
@@ -1,232 +0,0 @@
 # CI Pipeline — Operator Guide
 > Last reviewed: 2026-05-05
 > Authoritative guide to certctl's CI pipeline shape.
 > Per the ci-pipeline-cleanup spec, Phase 12.
 ## Trigger model
 Three triggers, each with its own scope. Don't mix.
 | Trigger | Workflow | Scope | Wall-clock target |
 |---|---|---|---|
 | Push to master, PR to master | `.github/workflows/ci.yml` + `.github/workflows/codeql.yml` | Blocking — every check earns its keep | <10 min |
 | Daily 06:00 UTC + `workflow_dispatch` | `.github/workflows/security-deep-scan.yml` | Slow scans (gosec, osv, trivy, ZAP, schemathesis, nuclei, testssl, semgrep, mutation, `-race -count=10`); best-effort, never blocks | 60 min budget |
 | Tag push (`v*`) | `.github/workflows/release.yml` | Cross-platform binaries, ghcr.io push, SLSA provenance, GitHub release | n/a |
 This guide covers the **on-push pipeline** only.
 ## On-push pipeline (7 status checks)
 ```mermaid
 flowchart TD
    Push["push to master"]
    CI["CI workflow (5 jobs)"]
    CodeQL["CodeQL workflow (2 jobs)"]
    GoBuild["go-build-and-test<br/>~6-7 min"]
    Frontend["frontend-build<br/>~1 min"]
    HelmLint["helm-lint<br/>~10 sec"]
    Vendor["deploy-vendor-e2e<br/>~5 min, depends on go-build-and-test"]
    Image["image-and-supply-chain<br/>~3 min, parallel"]
    AnalyzeGo["Analyze (go)<br/>~5 min, parallel"]
    AnalyzeJS["Analyze (javascript-typescript)<br/>~5 min, parallel"]
    Push --> CI
    Push --> CodeQL
    CI --> GoBuild
    CI --> Frontend
    CI --> HelmLint
    CI --> Vendor
    CI --> Image
    CodeQL --> AnalyzeGo
    CodeQL --> AnalyzeJS
    GoBuild -.depends on.-> Vendor
 ```
 End-to-end wall-clock: dominated by `go-build-and-test` + `deploy-vendor-e2e` chain (~12 min) running in parallel with CodeQL (~5 min). Target ~10 min.
 ## Per-job deep-dive
 ### `go-build-and-test` (Ubuntu, ~6-7 min)
 Runs the Go build/test suite + 18 of 20 regression guards.
 Steps:
 1. `actions/checkout@v4`
 2. `actions/setup-go@v5` (Go 1.25.10)
 3. `go build ./cmd/...` (server, agent, mcp-server, cli)
 4. **gofmt drift** — `gofmt -l .` must be empty (Makefile::verify parity)
 5. **go mod tidy drift** — `go mod tidy && git diff --exit-code go.mod go.sum`
 6. `go vet ./...`
 7. Install + run **golangci-lint** v2.11.4 (`--timeout 5m`)
 8. Install + run **govulncheck** (hard gate)
 9. Install + run **staticcheck** (hard gate; `continue-on-error: false`)
 10. **Race Detection** — `go test -race -count=1 ./internal/...` (9-package list, 5min timeout)
 11. **Go Test with Coverage** — full coverage profile to `coverage.out`
 12. **Check Coverage Thresholds** — `bash scripts/check-coverage-thresholds.sh` (reads `.github/coverage-thresholds.yml`)
 13. **Upload Coverage Report** — artifact (`go-coverage`, 30-day retention)
 14. **Coverage PR comment** — posts/updates per-PR coverage table (PR builds only)
 15. **Regression guards** — loop runs all `scripts/ci-guards/*.sh` (18 of 20 guards)
 Local equivalent: `make verify` covers steps 4, 6, 7, 11 (with `-short`).
 ### `frontend-build` (Ubuntu, ~1 min)
 Vitest tests + tsc check + vite build + 2 of 20 regression guards (already covered by the ci-guards loop in `go-build-and-test`).
 Steps:
 1. `actions/checkout@v4`
 2. `actions/setup-node@v4` (Node 22)
 3. `npm ci`
 4. `npx tsc --noEmit`
 5. `npx vitest run`
 6. `npx vite build`
 7. **Regression guards** — same `scripts/ci-guards/*.sh` loop as `go-build-and-test` (catches frontend-side guards: S-1, P-1, T-1, L-015, L-019, M-009, G-3)
 ### `helm-lint` (Ubuntu, ~10 sec)
 Helm chart validation in 3 modes + inverse fail-loud test:
 1. `helm lint` with existingSecret
 2. `helm template` (existingSecret mode)
 3. `helm template` (cert-manager mode)
 4. `helm template` (no TLS source — MUST fail per fail-loud guard)
 ### `deploy-vendor-e2e` (Ubuntu, ~5 min, depends on `go-build-and-test`)
 Single-job collapse of the prior 12-job matrix (per ci-pipeline-cleanup Phase 5 / frozen decision 0.4 — revises Bundle II decision 0.9).
 Steps:
 1. `actions/checkout@v5`
 2. `actions/setup-go@v5` (Go 1.25.10, cache: true)
 3. **Build f5-mock-icontrol sidecar** — only sidecar without published image
 4. **Bring up all vendor sidecars** — `docker compose --profile deploy-e2e up -d` (11 sidecars)
 5. **Run all vendor-edge e2e** — `go test -tags integration -race -count=1 -run 'VendorEdge_'`; output captured to `test-output.log`
 6. **Skip-count enforcement** — `bash scripts/ci-guards/vendor-e2e-skip-check.sh test-output.log` (catches sidecar boot failures via skip-count vs allowlist)
 7. **Tear down sidecars** — `docker compose down -v` (always runs)
 The `deploy-vendor-e2e-windows` matrix was deleted entirely (per ci-pipeline-cleanup Phase 6 / frozen decision 0.5 — revises Bundle II decision 0.4). IIS + WinCertStore validation moved to [`docs/connector-iis.md::Operator validation playbook`](connector-iis.md#operator-validation-playbook-windows-host).
 ### `image-and-supply-chain` (Ubuntu, ~3 min, parallel)
 Three checks bundled (per ci-pipeline-cleanup Phases 7-9 / frozen decision 0.8):
 1. **Digest validity** — `bash scripts/ci-guards/digest-validity.sh`. Resolves every `@sha256:<digest>` ref in `deploy/**/*.{yml,Dockerfile*}` against its registry. Closes the H-001 lying-field gap.
 2. **Docker build smoke** — builds all 4 Dockerfiles (`Dockerfile`, `Dockerfile.agent`, `deploy/test/f5-mock-icontrol/Dockerfile`, `deploy/test/libest/Dockerfile`).
 3. **OpenAPI ↔ handler operationId parity** — `bash scripts/ci-guards/openapi-handler-parity.sh`. Every router route must have a matching `operationId` in `api/openapi.yaml` or be documented in `api/openapi-handler-exceptions.yaml`.
 ### CodeQL (Ubuntu × 2 languages, ~5 min)
 `.github/workflows/codeql.yml` — interprocedural taint tracking. Two matrix jobs: `go` and `javascript-typescript`. Triggers on push, PR, and weekly Sunday cron.
 ## The 20 regression guards
 Located at `scripts/ci-guards/<id>.sh`. Each script is callable locally:
 ```bash
 bash scripts/ci-guards/G-3-env-docs-drift.sh
 ```
 Or run all of them:
 ```bash
 for g in scripts/ci-guards/*.sh; do
  echo "=== $(basename "$g") ==="
  bash "$g" || echo "  FAILED"
 done
 ```
 | ID | Catches |
 |---|---|
 | `G-1-jwt-auth-literal` | JWT silent auth downgrade reappearing |
 | `L-001-insecure-skip-verify` | Bare `InsecureSkipVerify: true` without `//nolint:gosec` |
 | `H-001-bare-from` | Bare Dockerfile `FROM` without `@sha256:` digest pin |
 | `M-012-no-root-user` | Dockerfile missing terminal `USER <non-root>` |
 | `H-009-readme-jwt` | README re-introducing JWT-as-supported claim |
 | `G-2-api-key-hash-json` | `api_key_hash` in JSON-emitting surface |
 | `U-2-plaintext-healthcheck` | Plaintext `http://` in HEALTHCHECK |
 | `U-3-migration-mount` | Migration file mounted into postgres initdb |
 | `D-1-D-2-statusbadge-phantom` | Dead StatusBadge keys + 8 TS phantom fields across 4 interfaces |
 | `L-1-bulk-action-loop` | Client-side `for ... await` bulk action loops |
 | `B-1-orphan-crud` | 8 update/create/delete fns lose page consumers |
 | `S-2-strings-contains-err` | `strings.Contains(err.Error(), ...)` brittle dispatch |
 | `G-3-env-docs-drift` | `CERTCTL_*` env var defined OR documented but not both |
 | `test-naming-convention` | `func TestXxx` lowercase first letter (Go silently skips) |
 | `S-1-hardcoded-source-counts` | Hardcoded "N issuer connectors" prose |
 | `P-1-documented-orphan-fns` | 16 read-fn names removed from client.ts exports |
 | `T-1-frontend-page-coverage` | New page in `web/src/pages/` without sibling `.test.tsx` |
 | `bundle-8-L-015-target-blank-rel-noopener` | `target="_blank"` without `rel="noopener noreferrer"` |
 | `bundle-8-L-019-dangerously-set-inner-html` | `dangerouslySetInnerHTML` outside `safeHtml.ts` |
 | `bundle-8-M-009-bare-usemutation` | Bare `useMutation()` outside the `useTrackedMutation` wrapper |
 Plus three additional scripts for non-guard operator workflows:
 - `scripts/ci-guards/vendor-e2e-skip-check.sh` — vendor-e2e skip-count enforcement (used by `deploy-vendor-e2e` job)
 - `scripts/ci-guards/digest-validity.sh` — used by `image-and-supply-chain` job
 - `scripts/ci-guards/openapi-handler-parity.sh` — used by `image-and-supply-chain` job
 - `scripts/ci-guards/coverage-pr-comment.sh` — used by `go-build-and-test` job
 - `scripts/check-coverage-thresholds.sh` — used by `go-build-and-test` job
 ## Coverage thresholds
 Manifest at `.github/coverage-thresholds.yml`. Each entry has `floor:` (integer percentage) + `why:` (load-bearing context). Lowering a floor REQUIRES corresponding code-side test work — never lower the gate to make CI green.
 To add a new gated package: add an entry to the YAML; no script changes needed.
 ## Make targets — three-tier convention
 | Target | When | What |
 |---|---|---|
 | `make verify` | **Required pre-commit** | gofmt + vet + golangci-lint + go test -short |
 | `make verify-deploy` | Optional pre-push | digest-validity + OpenAPI parity + Docker build smoke (server + agent only — fast subset) |
 | `make verify-docs` | **Required pre-tag** | QA-doc Part-count + seed-count drift checks |
 ## Adding a new check
 | Check type | Where it goes | Auto-picked-up by CI? |
 |---|---|---|
 | Regression guard (grep / shape pattern) | New `scripts/ci-guards/<id>.sh` script | Yes — loop step iterates `*.sh` |
 | Coverage threshold (per-package) | New entry in `.github/coverage-thresholds.yml` | Yes — bash loop reads YAML |
 | OpenAPI route exception | New entry in `api/openapi-handler-exceptions.yaml` | Yes — parity script reads YAML |
 | Vendor-e2e expected skip | New line in `scripts/ci-guards/vendor-e2e-skip-allowlist.txt` | Yes — skip-check script reads file |
 | New CI job | Edit `.github/workflows/ci.yml` directly | n/a (job definition is the source) |
 ## Troubleshooting
 | CI step fails | Likely cause | Fix |
 |---|---|---|
 | `gofmt drift` | source needs `gofmt -w` | `make fmt` locally + commit |
 | `go mod tidy drift` | imported a package without committing go.mod | `go mod tidy` + commit |
 | `Run staticcheck` | new SA1019 deprecated-API site | migrate the API OR add `//lint:ignore SA1019 <reason>` |
 | `Check Coverage Thresholds` | per-package coverage dropped below floor | add tests; do NOT lower the floor |
 | `Regression guards` (any `<id>.sh`) | the audit-finding the guard pinned reappeared | read the guard's head-comment block for the closure rationale + fix the regression |
 | `Skip-count enforcement` | a vendor sidecar failed to start | check docker logs; fix sidecar; OR if a new Windows-only test was added, add to `scripts/ci-guards/vendor-e2e-skip-allowlist.txt` |
 | `Digest validity` | a `@sha256` digest doesn't resolve | re-resolve from registry, replace in compose / Dockerfile |
 | `OpenAPI ↔ handler parity` | new router route without operationId | add to `api/openapi.yaml` (preferred) OR `api/openapi-handler-exceptions.yaml` |
 | `Docker build smoke` | Dockerfile syntax error or COPY path drift | fix the Dockerfile |
 | `CodeQL Analyze` | interprocedural dataflow finding | review the SARIF in Security → Code scanning tab |
 ## Status check accounting
 **Current (post-cleanup):** 7 status checks per push.
 - 1 × `Go Build & Test`
 - 1 × `Frontend Build`
 - 1 × `Helm Chart Validation`
 - 1 × `deploy-vendor-e2e`
 - 1 × `image-and-supply-chain`
 - 2 × `CodeQL Analyze (<lang>)` (go + javascript-typescript)
 **Pre-cleanup (HEAD `1de61e91`):** 19 status checks. The 12-vendor matrix + 2-vendor Windows matrix collapsed to 1 + 0 respectively; the 3 Go/Frontend/Helm jobs unchanged; 2 CodeQL unchanged; 1 new `image-and-supply-chain` added.
 ## Required GitHub branch protection list
 When updating the `master` branch protection rule (Settings → Branches), the "Require status checks to pass" list should be exactly:
 ```
 Go Build & Test
 Frontend Build
 Helm Chart Validation
 deploy-vendor-e2e
 image-and-supply-chain
 Analyze (go)
 Analyze (javascript-typescript)
 ```
 Old-name checks (`deploy-vendor-e2e (<vendor>)` × 12, `deploy-vendor-e2e-windows (<vendor>)` × 2) won't appear on new PRs after the workflow change. Operator removes them from the required list.
@@ -1,68 +0,0 @@
 # GUI QA Checklist
 > Last reviewed: 2026-05-05
 Manual GUI verification pass for release sign-off. Vitest covers component-level behavior; this checklist covers end-to-end flows that only land correctly when the React SPA, the REST API, and the database are all wired together.
 ## Prereqs
 The full stack must be running and healthy per [`qa-prerequisites.md`](qa-prerequisites.md). Open `https://localhost:8443` in a fresh browser session (Incognito / Private mode is fine — avoids cached state from previous QA passes).
 ## Pages to verify
 For each page, the verification is "open it, confirm it renders without console errors, exercise the documented action, confirm the action lands as expected."
 | Page | Action to verify | Expected result |
 |---|---|---|
 | `/dashboard` | Page loads, all 4 stat cards populate | Total / Active / Expiring / Expired counts match `GET /api/v1/stats/summary` |
 | `/certificates` | Inventory list paginates | "Next page" button works; URL updates with cursor; row count consistent |
 | `/certificates/<id>` | Detail page opens for any cert | Cert chain renders, deployment status shows, audit timeline visible |
 | `/issuers` | Catalog renders all configured issuers | Each issuer card shows last-used / status; clicking opens detail |
 | `/issuers/<id>` | Issuer config form | Edit + Save round-trips through `PATCH /api/v1/issuers/<id>` |
 | `/issuers/hierarchy` | CA tree view | Multi-level hierarchy renders; admin-gated CRUD buttons present for admins only |
 | `/agents` | Fleet view | Online/offline status accurate; OS/arch grouping correct |
 | `/agents/<id>` | Agent detail | Last heartbeat, registered date, deployment job history |
 | `/agents/groups` | Agent groups CRUD | Create + edit + delete a test group; verify dynamic membership matching |
 | `/jobs` | Job queue | Filter by status / type works; click into a job opens detail |
 | `/jobs/<id>` | Job detail | Status, retries, logs, owner attribution |
 | `/policies` | Renewal policies CRUD | Edit AlertChannels matrix, save, verify backend reflects change |
 | `/profiles` | Certificate profiles | EKU constraints + max TTL editable; profile binding works |
 | `/notifications` | Notifier config | Test connection button against each configured notifier |
 | `/discovery` | Discovery triage | Claim / Dismiss buttons round-trip to backend |
 | `/network-scans` | Scan target CRUD | Create scan target, trigger immediate scan, results appear |
 | `/audit` | Audit trail | Filter by actor / action / time range; CSV export works |
 | `/short-lived` | Short-lived credential dashboard | Live TTL countdown updates; auto-refresh every 10s |
 | `/observability` | Observability dashboard | Charts render: expiration heatmap, renewal trends, issuance rate |
 | `/health` | Health monitor | TLS endpoint health: healthy / degraded / down states accurate |
 | `/digest` | Digest preview | Email preview renders; "Send digest" button dispatches |
 | `/owners` | Owners CRUD | Create owner with team, edit, delete (after reassigning certs) |
 | `/teams` | Teams CRUD | Create + delete; verify cascade removes orphan owners |
 | `/scep` | SCEP admin tabs | Profiles / Intune Monitoring / Recent Activity all populate |
 | `/est` | EST admin tabs | Profiles / Recent Activity / Trust Bundle all populate |
 | `/login` | Login flow | API key entry persists for the session; bad key rejected |
 ## Console hygiene
 Open browser DevTools and confirm:
 - No uncaught exceptions on any page
 - No 404 / 500 responses in the Network tab from API calls
 - No CORS errors
 - No CSP violations
 ## Mobile / narrow-viewport
 The dashboard is desktop-first but should not break catastrophically on narrow viewports. Resize the browser to 380px width; confirm:
 - Sidebar collapses to a hamburger menu
 - Tables either scroll horizontally or stack on mobile
 - Forms remain usable
 ## Accessibility spot-check
 - Tab through any single page using only the keyboard. Every interactive element must be reachable, and the focus indicator must be visible.
 - Lighthouse accessibility audit on `/dashboard`: target ≥ 90.
 ## Sign-off
 Document any deviations in the release sign-off matrix at [`release-sign-off.md`](release-sign-off.md).
@@ -1,99 +0,0 @@
 # QA Prerequisites
 > Last reviewed: 2026-05-05
 Operational prereqs for running release QA against certctl. Before any of the contributor-facing testing surfaces (test-environment.md, gui-qa-checklist.md, release-sign-off.md) are useful, the local stack needs to be in a known-good state.
 ## Why manual QA on top of automated tests?
 Automated tests mock dependencies and run in isolation. Manual QA validates the full integrated stack: real PostgreSQL, real HTTP, real agent binary, real file I/O, real scheduler timing. It catches issues that unit tests can't: migration ordering, Docker networking, env var parsing, browser rendering, and timing-dependent scheduler behavior.
 ## Environment setup
 **Step 1: Start the full stack.**
 ```bash
 cd deploy && docker compose -f docker-compose.yml -f docker-compose.demo.yml up --build -d
 ```
 This builds three containers (postgres, certctl-server, certctl-agent) and runs them on a bridge network. The `--build` flag ensures you're testing the current code, not a stale image. The `demo` overlay is an override file (no `image:` or `build:` of its own) that layers `CERTCTL_DEMO_SEED=true` onto the base — both files must be passed in that order or compose errors with `service "certctl-server" has neither an image nor a build context specified`. The seed populates the database with realistic fixtures.
 **Step 2: Wait for healthy state.**
 ```bash
 for i in $(seq 1 30); do
  STATUS=$(docker compose ps --format json 2>/dev/null | jq -r 'select(.Health != null) | "\(.Name): \(.Health)"' 2>/dev/null)
  echo "$STATUS"
  echo "$STATUS" | grep -q "unhealthy\|starting" || break
  sleep 2
 done
 ```
 Why: Docker Compose starts containers in dependency order (postgres → server → agent), but "started" doesn't mean "ready." Health checks confirm postgres accepts connections, the server responds on `/health`, and the agent process is running.
 **Step 3: Set shell variables used throughout the QA flow.**
 ```bash
 export SERVER=https://localhost:8443
 export API_KEY="change-me-in-production"
 export AUTH="Authorization: Bearer $API_KEY"
 export CT="Content-Type: application/json"
 export CACERT="--cacert ./deploy/test/certs/ca.crt"
 ```
 Every curl command in QA docs uses these variables. Setting them once avoids typos and keeps the docs copy-pasteable.
 > **Note:** The default Docker Compose sets `CERTCTL_AUTH_TYPE: none` for the demo overlay, meaning auth is disabled. Tests that exercise auth require flipping this to `api-key`; instructions are in the relevant test docs.
 **Step 4: Build CLI and MCP server binaries on the host.**
 ```bash
 go build -o certctl-cli ./cmd/cli/...
 go build -o certctl-mcp ./cmd/mcp-server/...
 ```
 The CLI and MCP server are separate binaries that talk to the server over HTTP. Building them verifies the code compiles and produces the executables you'll test later.
 ## Demo data baseline
 The seed data (`migrations/seed.sql` + `migrations/seed_demo.sql`) pre-populates the database with realistic fixtures. Confirm it loaded:
 ```bash
 curl -s $CACERT -H "$AUTH" $SERVER/api/v1/stats/summary | jq .
 ```
 **Expected shape:**
 ```json
 {
  "total_certificates": 15,
  "active_certificates": ...,
  "expiring_certificates": ...,
  "expired_certificates": ...,
  "pending_renewals": ...
 }
 ```
 **Reference IDs in the demo data** (used across QA docs):
 | Resource | IDs | Count |
 |---|---|---|
 | Teams | `t-platform`, `t-security`, `t-payments`, `t-frontend`, `t-data` | 5 |
 | Owners | `o-alice`, `o-bob`, `o-carol`, `o-dave`, `o-eve` | 5 |
 | Policies | `rp-standard`, `rp-urgent`, `rp-manual` | 3 |
 | Issuers | `iss-local`, `iss-acme-le`, `iss-stepca`, `iss-digicert` | 4 |
 | Agents | `ag-web-prod`, `ag-web-staging`, `ag-lb-prod`, `ag-iis-prod`, `ag-data-prod` | 5 |
 | Targets | `tgt-nginx-prod`, `tgt-nginx-staging`, `tgt-f5-prod`, `tgt-iis-prod`, `tgt-nginx-data` | 5 |
 | Profiles | `prof-standard-tls`, `prof-internal-mtls`, `prof-short-lived`, `prof-high-security` | 4 |
 | Certificates | `mc-api-prod`, `mc-web-prod`, `mc-pay-prod`, etc. | 15 |
 | Agent Groups | `ag-linux-prod`, `ag-linux-amd64`, `ag-windows`, `ag-datacenter-a`, `ag-manual` | 5 |
 | Network Scan Targets | `nst-dc1-web`, `nst-dc2-apps`, `nst-dmz` | 3 |
 ## Once these are green
 Move to the appropriate downstream surface:
 - [`test-environment.md`](test-environment.md) — full local environment tutorial with real CAs (Pebble, step-ca, etc.)
 - [`gui-qa-checklist.md`](gui-qa-checklist.md) — manual GUI test pass
 - [`release-sign-off.md`](release-sign-off.md) — release-day checklist
 - [`testing-strategy.md`](testing-strategy.md) — what we test in CI vs daily deep-scan vs manual QA
@@ -1,445 +0,0 @@
 # QA Test Suite Guide (`qa_test.go`)
 > Last reviewed: 2026-05-05
 > **Audience:** Anyone running release QA for certctl — whether you're a first-time contributor or the maintainer cutting a release tag.
 >
 > **Self-contained.** Through 2026-05-04 this doc was a companion to a separate `docs/testing-guide.md` (the *what* to test) — that companion was pruned during the Phase 5 docs overhaul (its content dispersed across the audience-organized doc tree). The Part-by-Part Coverage Map below is now the canonical inventory of QA Parts.
 ---
 ## Test Suite Health (regenerate via `make qa-stats`)
 > Snapshot at HEAD. Re-run `make qa-stats` to refresh; the QA-doc seed-count drift guard (`.github/workflows/ci.yml::QA-doc seed-count drift guard`) catches out-of-date cert / issuer counts on every PR. The Part-count drift guard retired in the 2026-05-04 docs overhaul Phase 5 (testing-guide.md was pruned; Part counts are now tracked inside `qa_test.go` itself, not against an external doc). **Last regenerated: 2026-04-27 (Bundle P).**
 | Metric | Value | Target | Status |
 |---|---|---|---|
 | Backend test files | 221 | n/a | ℹ |
 | Backend `Test*` functions | 2,454 | n/a | ℹ |
 | Backend `t.Run` subtests | 778 | n/a | ℹ |
 | Frontend test files | 38 | n/a | ℹ |
 | Fuzz targets | 11 | ≥10 (one per hand-rolled parser) | ✓ |
 | `t.Skip` sites | 60 | each carries valid rationale (Bundle O audit) | ✓ |
 | `qa_test.go` Part_* subtests | 53 | covers 49 of 56 historical QA Parts directly + Parts 15–17 indirectly via Parts 42–46 | ✓ |
 | Existential cluster line cov (post-Bundle-J + L.B + Bundle 0.7) | acme 55.6%, stepca 90.4%, local-issuer ≥86%, crypto ≥85% | ≥95% | △ ACME below; tracked in `coverage-matrix.md` |
 | Mutation kill rate (Existential) | unmeasured (operator-runnable per Strengthening #5) | ≥90% | ⚠ |
 | Race detector clean (`-count=10`) | partial (`-count=3` clean per Phase 0) | 0 races | ⚠ |
 ## What Is This File?
 `deploy/test/qa_test.go` is a single Go test file (~1700 lines) that automates the historical QA Part inventory (preserved in the Part-by-Part Coverage Map below) against a running certctl Docker Compose demo stack. It replaces the legacy `qa-smoke-test.sh` bash script.
 It covers **49 of 56 Parts** of the testing guide as automation; the remaining 7 are
 either manual-only by design or pending QA-suite coverage:
 - **49 `Part_*` automation wrappers**, **~159 leaf subtests** — API calls, database queries, source file checks, performance benchmarks
 - **11 fully skipped Parts** — with documented reasons (external CAs, Windows, browser-only, etc.) — see "What This Test Does NOT Cover" below
 - **4 Parts NOT YET AUTOMATED** — Parts 23 (S/MIME & EKU), 24 (OCSP/CRL), 55 (Agent Soft-Retirement), 56 (Notification Retry & Dead-Letter) — must be tested manually until QA-suite automation lands; the Part-by-Part Coverage Map below describes the surface area each Part covers
 - **Manual-only flows** in addition: GUI flows, scheduler timing, Docker log inspection — must be done by a human (Coverage Map below describes each)
 ## Architecture
 ```mermaid
 flowchart LR
    QA["qa_test.go (//go:build qa)<br/><br/>TestQA(t *testing.T)<br/>├─ Part01_Infra<br/>├─ Part02_Auth<br/>├─ Part03_CertCRUD<br/>├─ ...<br/>└─ Part52_HelmChart"]
    subgraph Stack["certctl demo stack<br/>docker-compose.yml + docker-compose.demo.yml"]
        Server["certctl-server :8443"]
        Postgres["postgres :5432"]
        Agents["certctl-agent (×N)<br/>↑ seed_demo.sql provisions 12 agent rows<br/>(1 active, 2 retired, 9 reserved/sentinel)<br/>for the soft-retire / FSM coverage Parts 55–56 exercise"]
    end
    QA --> Stack
 ```
 > **Multi-agent demo stack (Bundle Q / L-004 closure).** The demo
 > stack runs a single live `certctl-agent` container by default but
 > the database is seeded with 12 agent rows (`migrations/seed_demo.sql`,
 > grep `mc-* | ag-*` IDs). The "(×N)" notation reflects the seed-data
 > reality: Parts 04 (Agents Listing), 05 (Agent Heartbeats), 55
 > (Agent Soft-Retirement), and FSM coverage tables in
 > `coverage-audit-2026-04-27/tables/fsm-coverage.md` exercise the full
 > multi-agent population, not the one live container. Operators
 > running the QA suite in a parallel-agent topology should set
 > `AGENT_COUNT=N` in compose-override and re-derive the seed counts
 > via `make qa-stats`.
 Key design choices:
 - **Build tag:** `//go:build qa` — never runs during `go test ./...` or CI. Only runs when explicitly requested.
 - **Package:** `integration_test` — same package as `integration_test.go` (which uses `//go:build integration` for the test stack). They coexist but never run together.
 - **Zero internal imports:** Uses only stdlib + `lib/pq` (from `go.mod`). All API interactions are plain HTTP. All JSON is decoded into lightweight local structs (`qaCert`, `qaJob`, etc.) — not the internal domain types.
 - **Self-cleaning:** Tests that create data use `t.Cleanup()` to delete it afterward. The seed data is not modified.
 ## Prerequisites
 1. **Docker Compose demo stack running:**
   ```bash
   cd deploy
   docker compose -f docker-compose.yml -f docker-compose.demo.yml up --build -d
   ```
   Wait ~15 seconds for health checks to pass.
 2. **Go 1.22+** installed (the project uses Go 1.25 in `go.mod`, but 1.22+ works for running tests).
 3. **PostgreSQL port exposed** — the demo stack exposes port 5432 for database verification tests (table counts, schema checks).
 4. **Repository checkout** — source file verification tests (`fileExists`, `fileContains`) read files relative to `qaRepoDir` (default: `../..` from `deploy/test/`).
 ## Running the Tests
 ### Full suite
 ```bash
 cd deploy/test
 go test -tags qa -v -timeout 10m ./...
 ```
 ### Single Part
 ```bash
 go test -tags qa -v -run TestQA/Part03 ./...
 ```
 ### Single subtest
 ```bash
 go test -tags qa -v -run TestQA/Part03_CertCRUD/Create_Minimal ./...
 ```
 ### With custom environment
 ```bash
 CERTCTL_QA_SERVER_URL=https://staging.internal:8443 \
 CERTCTL_QA_API_KEY=my-staging-key \
 CERTCTL_QA_DB_URL=postgres://certctl:secret@db.internal:5432/certctl?sslmode=require \
 CERTCTL_QA_REPO_DIR=/path/to/certctl \
 go test -tags qa -v -timeout 10m ./...
 ```
 ### Environment Variables
 | Variable | Default | Description |
 |---|---|---|
 | `CERTCTL_QA_SERVER_URL` | `https://localhost:8443` | certctl server URL (HTTPS-only as of v2.2) |
 | `CERTCTL_QA_API_KEY` | `change-me-in-production` | API key for Bearer auth |
 | `CERTCTL_QA_DB_URL` | `postgres://certctl:certctl@localhost:5432/certctl?sslmode=disable` | PostgreSQL connection string |
 | `CERTCTL_QA_REPO_DIR` | `../..` | Path to certctl repo root (for source file checks) |
 | `CERTCTL_QA_CA_BUNDLE` | `./certs/ca.crt` | PEM CA bundle pinned for TLS verification. The demo stack's `certctl-tls-init` container writes here. |
 | `CERTCTL_QA_INSECURE` | `false` | Set to `"true"` to skip TLS verification (e.g. before the init container finishes). Never use outside the demo harness. |
 ## Part-by-Part Coverage Map
 This table shows what each Part tests and what's left for manual verification.
 | Part | Testing Guide Section | Automated Subtests | What's Automated | What's Manual |
 |------|----------------------|-------------------|-----------------|--------------|
 | 1 | Infrastructure & Deployment | 8 | Table count, health/ready endpoints, seed data counts (certs, agents, issuers, targets, policies) | Docker container health, log inspection, volume mounts |
 | 2 | Authentication & Security | 4 | No-auth 401, bad-key 401, health-no-auth 200, no private keys in API | CORS preflight, rate limiting (429 + Retry-After), TLS config |
 | 3 | Certificate Lifecycle | 10 | Create (minimal + full), get, 404, list pagination, status/issuer filters, sparse fields, update, archive | Deployment trigger, version history, certificate detail UI |
 | 4 | Renewal Workflow | 3 | Trigger renewal, 404 on nonexistent, agent work endpoint | AwaitingCSR flow, agent key generation, full issuance cycle |
 | 5 | Revocation | 5 | Revoke (default reason), already-revoked, nonexistent, invalid reason, CRL JSON | DER CRL, OCSP responder, revocation notifications |
 | 6 | Policies & Profiles | 6 | Policy CRUD (create/delete), invalid type 400, profile CRUD, list | Policy violation detection, profile enforcement on CSR |
 | 7 | Ownership & Teams | 4 | Team CRUD, owner CRUD, agent groups list | Owner notification routing, dynamic group matching |
 | 8 | Job System | 2 | List jobs, 404 on nonexistent | Job state transitions, approval workflow, cancellation |
 | 9 | Issuer Connectors | 4 | List, get detail, create (GenericCA), missing name 400 | Test connection, issuer-specific issuance flow |
 | 10 | Sub-CA Mode | SKIP | — | Requires CA cert+key on disk |
 | 11 | ACME ARI | SKIP | — | Requires ARI-capable CA |
 | 12 | Vault PKI | SKIP | — | Requires live Vault server |
 | 13 | DigiCert | SKIP | — | Requires DigiCert sandbox |
 | 14 | Target Connectors | 3 | List, create NGINX target, delete 204 | Deploy to real target, validate deployment |
 | 15–17 | Apache/HAProxy, Traefik/Caddy, IIS | — | (Covered by source checks in Parts 42–46) | Requires real services or Windows |
 | 18 | Agent Operations | 3 | Heartbeat (register), metadata check, auto-create on heartbeat | Agent binary behavior, key storage, discovery scan |
 | 19 | Agent Work Routing | 1 | Empty work for agent with no targets | Scoped job assignment, multi-target fan-out |
 | 20 | Post-Deployment Verification | 1 | 404 on nonexistent job verification | TLS probing, fingerprint comparison |
 | 21 | EST Server | 2 | CACerts (200 + content-type), CSRAttrs (200/204) | simpleenroll with CSR, simplereenroll, PKCS#7 parsing |
 | 22 | Certificate Export | 3 | PEM export, PKCS#12 export, 404 on nonexistent | Download mode, file content validation |
 | 23 | S/MIME & EKU Support | 0 (NOT AUTOMATED) | — | S/MIME profile creation; EKU enforcement on issuance; SMIMECapabilities extension presence in issued cert; rejection of profile-violating EKU on CSR. Test manually — see the Coverage Map row |
 | 24 | OCSP Responder & DER CRL | 0 (NOT AUTOMATED) | — | OCSP request/response (RFC 6960), DER CRL generation, status (Good/Revoked/Unknown), Must-Staple coordination. Test manually — see the Coverage Map row |
 | 25 | Certificate Discovery | 5 | List discovered, summary, list scan targets, create target, invalid CIDR 400 | Agent filesystem scan, claim/dismiss workflow |
 | 26 | Enhanced Query API | 4 | Sort descending, cursor pagination, time-range filter, invalid sort field | Field projection correctness, cursor token cycling |
 | 27 | Request Body Size Limits | 1 | 2MB body rejected (413/400) | Exact limit boundary (1MB) |
 | 28 | CLI | SKIP | — | Requires compiled `certctl-cli` binary |
 | 29 | MCP Server | SKIP | — | Requires compiled `mcp-server` binary + stdio |
 | 30 | Observability | 7 | Dashboard summary, certs by status, expiration timeline, job trends, issuance rate, JSON metrics (uptime + gauges), Prometheus (content-type + 4 metric names) | Chart rendering (GUI), Grafana import |
 | 31 | Notifications | 2 | List, 404 on nonexistent | Notification content, mark-read, email/Slack delivery |
 | 32 | Audit Trail | 3 | List events (≥10), PUT immutability, DELETE immutability | Actor attribution, body hash, time range filters |
 | 33 | Background Scheduler | SKIP | — | Timing-dependent; verify via Docker logs |
 | 34 | Structured Logging | SKIP | — | Requires Docker log inspection |
 | 35 | GUI Testing | SKIP | — | Requires browser |
 | 36–37 | Issuer Catalog, Frontend Audit | SKIP | — | Requires browser |
 | 38 | Error Handling | 5 | Malformed JSON, missing required field, method not allowed, UTF-8 CN, empty body | Stack trace suppression, error response format |
 | 39 | Performance | 5 | List certs < 200ms, stats < 500ms, metrics < 200ms, Prometheus < 300ms, audit < 500ms | Load testing, concurrent request handling |
 | 40 | Documentation | 8 | README, quickstart, architecture, connectors exist; migration guides exist; 8 issuer types in docs; 11 target types in docs | Content accuracy, link validity |
 | 41 | Regression | 3 | DELETE 204, per_page max fallback, network scan target seed count | `errors.Is(errors.New())` anti-pattern source scan |
 | 42 | Envoy Target | 5 | Domain type, connector file, test file, OpenAPI, agent dispatch | Envoy deployment test, SDS config |
 | 43 | Postfix/Dovecot | 3 | Domain types (Postfix + Dovecot), connector file, OpenAPI | Mail server deployment test |
 | 44 | SSH Target | 4 | Domain type, connector file, agent dispatch (`sshconn`), OpenAPI | SSH deployment test (requires target host) |
 | 45 | Windows Certificate Store | 3 | Domain type, connector file, shared certutil package | Windows deployment (requires Windows) |
 | 46 | Java Keystore | 3 | Domain type, connector file, OpenAPI | JKS deployment (requires keytool) |
 | 47 | Certificate Digest Email | 3 | Preview endpoint (200/503), service file, adapter file | SMTP delivery, HTML template rendering |
 | 48 | Dynamic Issuer Config | 4 | Crypto package exists, create ACME issuer via API, config redaction check, migration exists | Test connection flow, registry rebuild |
 | 49 | Dynamic Target Config | 2 | Create NGINX target via API, migration exists | Test connection via agent heartbeat |
 | 50 | Onboarding Wizard | 2 | Wizard component exists, docker-compose split (clean vs demo) | Wizard UI flow, step completion |
 | 51 | ACME Profile Selection | 3 | Profile module exists, frontend config, RFC 9702→9773 renumber check | Profile-aware issuance against real CA |
 | 52 | Helm Chart | 5 | Chart.yaml, values.yaml, 4 templates exist, securityContext, health probes | `helm template` rendering, `helm install` |
 | 53 | Kubernetes Secrets Target Connector (M47) | 18 | Config validation (namespace DNS-1123, secret name DNS subdomain, label keys, required fields), deployment (create/update Secret, chain concatenation, error propagation), validation (serial comparison, not-found, empty cert) | GUI target wizard KubernetesSecrets fields (namespace, secret_name, labels, kubeconfig_path), Helm RBAC toggle, TargetDetailPage type label |
 | 54 | AWS ACM Private CA Issuer Connector (M47) | 23 | Config validation (region, CA ARN regex, signing algorithm whitelist, validity_days, defaults), issuance (full flow, empty CSR, errors), renewal (reuses issuance), revocation (reason mapping, default, errors), GetOrderStatus completed, GetCACertPEM (success/chain/error), GetRenewalInfo nil | GUI issuer wizard AWSACMPCA fields (region, ca_arn, signing_algorithm, validity_days, template_arn), seed data visibility, create issuer flow |
 | 55 | Agent Soft-Retirement (I-004) | 0 (NOT AUTOMATED) | — | Soft-retire vs hard-retire; force flag; reason capture; foreign-key cascade behavior on retired-agent cert ownership; reactivation. Test manually — see the Coverage Map row |
 | 56 | Notification Retry & Dead-Letter Queue (I-005) | 0 (NOT AUTOMATED) | — | Retry loop with exponential backoff, dead-letter transition after N retries, requeue endpoint (`POST /api/v1/notifications/{id}/requeue`), idempotency on retry. Test manually — see the Coverage Map row |
 **Totals (verified 2026-04-27):** 49 `Part_*` automation wrappers, ~159 leaf subtests, 11 fully
 skipped Parts, 4 Parts not yet automated (23, 24, 55, 56), and an unspecified count of manual-only
 flows (GUI, scheduler timing, Docker log inspection). Run `grep -cE 't\.Run\("Part[0-9]+_' deploy/test/qa_test.go` to count Part_* automation wrappers
 and `grep -cE 't\.Run\("Part[0-9]+_' deploy/test/qa_test.go` to re-verify.
 ## Coverage by Risk Class
 A buyer's QA lead reading this doc wants "where are the existential bugs caught?" — Bundle P / Strengthening #1 surfaces that view directly. The table below classifies each Part by risk class so reviewers can answer the existential-coverage question in one glance.
 | Risk class | Description | Parts in scope | Automation status |
 |---|---|---|---|
 | **Existential** (Critical paths — bugs would compromise CA, leak keys, mis-issue, bypass revocation) | Crypto, PKCS#7, local-issuer, OCSP/CRL, agent keygen, CSR validation | 5 (Revocation), 21 (EST), 23 (S/MIME EKU), 24 (OCSP/CRL), 47 (Digest with cert content), 53 (K8s Secrets), 54 (AWS PCA) | 5/7 automated; Parts 23 + 24 pending (Bundle I Skip stubs in `qa_test.go`; manual playbook in the Coverage Map below) |
 | **High** (FSM corruption, credential leak, authn/z weakening) | Renewal, jobs, agents, issuers, deployment, scheduler | 4, 7, 8, 9, 18, 19, 20, 22, 25, 28, 29, 32, 33, 48, 49, 55, 56 | 14/17 automated; CLI / MCP / scheduler-loop are inherently SKIP (require compiled binaries / Docker logs); Parts 55 + 56 pending |
 | **Medium** (Operational pain or silent data drift) | Targets, notifiers, observability, error handling, performance, regression | 14, 15-17, 30, 31, 38, 39, 40, 41, 42, 43, 44, 45, 46 | 14/14 automated (15-17 indirect via Parts 42–46) |
 | **Low** (Hygiene) | Documentation, docs verification | 40 (Documentation), 50 (Onboarding) | 2/2 automated |
 | **Frontend** (XSS, render correctness, mutation contracts) | GUI testing | 35, 36-37 | 0/3 automated in this suite (Vitest covers separately under `web/`); this doc punts to manual + Vitest |
 | **Audit-relevant** | Audit trail, body-size limits, request limits, Helm chart deploy posture | 27, 32, 51, 52 | 4/4 automated |
 This is the table acquisition reviewers screenshot for their report. When a new Part_* subtest lands in `qa_test.go`, classify it here.
 ## Test Categories
 The automated tests fall into four categories:
 ### 1. API Integration Tests (majority)
 Make real HTTP requests to the running server and verify status codes, response structure, and JSON field values. Examples:
 - `POST /api/v1/certificates` with valid payload → 201
 - `GET /api/v1/certificates?status=Active` → all returned certs have `status: "Active"`
 - `DELETE /api/v1/certificates/mc-qa-full` → 204
 ### 2. Database Verification Tests
 Connect directly to PostgreSQL and verify schema state:
 - Table count ≥ 19 (from migrations 000001–000010)
 - Useful for catching migration regressions
 ### 3. Source File Verification Tests
 Read files from the repo checkout and verify structure:
 - Domain types exist in `internal/domain/connector.go` (e.g., `TargetTypeEnvoy`)
 - Connector implementations exist (e.g., `internal/connector/target/envoy/envoy.go`)
 - Documentation contains expected content (all issuer/target types listed)
 - No stale RFC 9702 references (replaced by RFC 9773)
 ### 4. Performance Spot Checks
 Timed API requests with threshold assertions:
 - `GET /api/v1/certificates?per_page=15` < 200ms
 - `GET /api/v1/stats/summary` < 500ms
 - `GET /api/v1/metrics/prometheus` < 300ms
 ## What This Test Does NOT Cover
 These gaps must be filled by manual testing — see each Coverage Map row for surface-area description:
 ### Not Yet Automated (Parts 23, 24, 55, 56)
 These historical QA Parts are listed in the Coverage Map below but have no `Part_*` automation
 in `qa_test.go` yet. They are operator-runnable from the manual playbook; QA-suite
 automation should land before the next acquisition-grade release.
 - **Part 23: S/MIME & EKU Support** — profile-driven EKU enforcement; SMIMECapabilities extension
 - **Part 24: OCSP Responder & DER CRL** — OCSP request/response correctness, CRL generation, Must-Staple coordination
 - **Part 55: Agent Soft-Retirement (I-004)** — soft vs hard retire, FK cascade, reactivation
 - **Part 56: Notification Retry & Dead-Letter Queue (I-005)** — retry semantics, dead-letter transition, requeue
 ### External CA Integrations (Parts 10–13)
 - **Sub-CA mode** — requires CA cert+key files on disk
 - **ACME ARI** — requires a CA that supports RFC 9773 Renewal Information
 - **Vault PKI** — requires a running HashiCorp Vault instance
 - **DigiCert / Sectigo / Google CAS** — requires sandbox API credentials
 ### Browser/GUI Testing (Parts 35–37, 50)
 - Dashboard chart rendering (Recharts)
 - Onboarding wizard step-by-step flow
 - Issuer catalog card layout and create wizard
 - Bulk operations UI (multi-select, progress bars)
 - Discovery triage workflow
 ### Real Deployment Testing (Parts 15–17)
 - NGINX/Apache/HAProxy file write + reload
 - Traefik/Caddy file provider or API reload
 - IIS PowerShell/WinRM (requires Windows)
 - F5 BIG-IP iControl REST (requires appliance or mock)
 - SSH agentless deployment (requires target host)
 ### Agent Binary Behavior (Parts 18, 28–29)
 - Agent-side ECDSA key generation and CSR submission
 - Agent filesystem discovery scan
 - CLI tool (`certctl-cli`) — all 10 subcommands
 - MCP server (`mcp-server`) — stdio transport
 ### Timing-Dependent Tests (Parts 33–34)
 - Background scheduler loop execution (renewal, jobs, health, notifications, digest, network scan)
 - Structured logging format verification (requires Docker log parsing)
 ## How This Relates to `integration_test.go`
 Both files live in `deploy/test/` in the same Go package (`integration_test`):
 | | `qa_test.go` | `integration_test.go` |
 |---|---|---|
 | **Build tag** | `//go:build qa` | `//go:build integration` |
 | **Target stack** | Demo (`docker-compose.yml` + `docker-compose.demo.yml`) | Test (`docker-compose.test.yml`) |
 | **Port** | 8443 | Different (test stack config) |
 | **Seed data** | `seed_demo.sql` (32 certs, 12 agents, 13 issuers, 8 targets, realistic history) | Minimal (created by tests) |
 | **CA backends** | Local CA only (demo mode) | Pebble ACME, step-ca, NGINX |
 | **Purpose** | Release QA — broad coverage, spot checks | Functional — end-to-end issuance, renewal, revocation against real CAs |
 | **Run frequency** | Before each release tag | CI on every PR |
 They are complementary. Integration tests prove the machinery works. QA tests prove the product works at release quality.
 ## Seed Data Reference
 The QA tests depend on `migrations/seed_demo.sql`. Key IDs used:
 ### Certificates (32 total in `managed_certificates`)
 The full canonical list is generated by:
 ```
 sed -n '/^INSERT INTO managed_certificates/,/^;/p' migrations/seed_demo.sql \
  | grep -oE "^\s*\('mc-[a-z0-9_-]+" | sed -E "s/^\s*\('//" | sort -u
 ```
 Hand-listing is unsustainable as the seed grows; tests reference IDs by lookup, not by enumeration.
 Sample IDs: `mc-api-prod`, `mc-web-prod`, `mc-pay-prod`, `mc-compromised`, `mc-smime-bob`, `mc-edge-eu`, `mc-k8s-ingress`, `mc-wildcard-prod`. See `migrations/seed_demo.sql:147` onward.
 ### Agents (12 total in `agents` table)
 8 named workload agents + 1 server-side sentinel + 3 cloud-discovery sentinels:
 - **Workload agents:** `ag-web-prod`, `ag-web-staging`, `ag-lb-prod`, `ag-iis-prod`, `ag-data-prod`, `ag-edge-01`, `ag-k8s-prod`, `ag-mac-dev`
 - **Server-side sentinel:** `server-scanner`
 - **Cloud-discovery sentinels:** `cloud-aws-sm`, `cloud-azure-kv`, `cloud-gcp-sm`
 Full list via:
 ```
 sed -n '/^INSERT INTO agents/,/^;/p' migrations/seed_demo.sql \
  | grep -oE "^\s*\('[a-z][a-z0-9_-]+" | sed -E "s/^\s*\('//"
 ```
 (The `agent_groups` table also contains entries with `ag-*` IDs — `ag-linux-prod`, `ag-windows`, `ag-datacenter-a`, `ag-arm64`, `ag-manual` — but those are *group* IDs, not agents. Don't confuse the two.)
 ### Issuers (13 total)
 `iss-local`, `iss-acme-le`, `iss-stepca`, `iss-acme-zs`, `iss-openssl`, `iss-vault`, `iss-digicert`, `iss-sectigo`, `iss-googlecas`, `iss-awsacmpca`, `iss-entrust`, `iss-globalsign`, `iss-ejbca`.
 Full list via:
 ```
 sed -n '/^INSERT INTO issuers/,/^;/p' migrations/seed_demo.sql \
  | grep -oE "^\s*\('iss-[a-z0-9_-]+" | sed -E "s/^\s*\('//"
 ```
 ### Targets (8 total in `deployment_targets`)
 `tgt-nginx-prod`, `tgt-nginx-staging`, `tgt-haproxy-prod`, `tgt-apache-prod`, `tgt-iis-prod`, `tgt-traefik-prod`, `tgt-caddy-prod`, `tgt-nginx-data`
 ### Network Scan Targets (4 total in `network_scan_targets`)
 `nst-dc1-web`, `nst-dc2-apps`, `nst-dmz`, `nst-edge`
 **Maintenance note:** when adding new seed rows, also update this section, OR remove the
 per-table counts and rely on the `sed | grep` commands so the doc stops drifting on every
 seed-data change. A CI guard that fails when the doc count diverges from the seed file is
 proposed in `coverage-audit-2026-04-27/tables/qa-doc-strengthening.md` (Strengthening #6).
 ## Troubleshooting
 ### "Server unreachable" on startup
 The test pings `GET /health` before running anything. If this fails:
 ```bash
 # Check if the stack is running
 docker compose -f docker-compose.yml -f docker-compose.demo.yml ps
 # Check server logs
 docker compose -f docker-compose.yml -f docker-compose.demo.yml logs certctl-server
 # Check if the port is exposed (self-signed cert — pin CA bundle)
 curl --cacert ./deploy/test/certs/ca.crt -s https://localhost:8443/health
 ```
 ### "connect to QA DB" failure
 The database tests connect directly to PostgreSQL. Ensure port 5432 is exposed:
 ```bash
 docker compose -f docker-compose.yml -f docker-compose.demo.yml port postgres 5432
 ```
 ### Performance tests flaking
 The performance thresholds (200ms, 300ms, 500ms) assume a local Docker stack. On slow CI runners or remote Docker hosts, increase the thresholds or skip Part 39:
 ```bash
 go test -tags qa -v -run 'TestQA/Part(?!39)' ./...
 ```
 ### Source file checks failing
 The `fileExists` and `fileContains` helpers read from `CERTCTL_QA_REPO_DIR` (default `../..`). If running from a non-standard location:
 ```bash
 CERTCTL_QA_REPO_DIR=/absolute/path/to/certctl go test -tags qa -v ./...
 ```
 ## Release Day Sign-Off Matrix
 Before tagging a release, the QA-on-call engineer signs off on each row. This matrix replaces the previous ad-hoc release checklist and ties test execution directly to release approval. Acquisition-grade releases have this kind of matrix; the doc previously didn't.
 | Sign-off | Evidence | Owner | Result | Date |
 |---|---|---|---|---|
 | `make verify` clean on master | CI run URL | Eng-on-call | ☐ | |
 | `go test -tags qa ./deploy/test/...` ≥ 95% pass rate (skips counted as pass) | Test output | QA-on-call | ☐ | |
 | `go test -race -count=10 ./internal/...` 0 races | `tool-output/race-x10.txt` | QA-on-call | ☐ | |
 | Coverage ≥ thresholds in `ci.yml` (service / handler / crypto / local-issuer / acme / stepca / mcp) | `tool-output/cover-summary.txt` | QA-on-call | ☐ | |
 | Helm chart `helm lint && helm template` clean | `tool-output/helm.txt` | DevOps-on-call | ☐ | |
 | All `t.Skip` sites have current rationales (see Bundle O audit; CI guard catches new orphans) | `make qa-stats` t.Skip count | QA-on-call | ☐ | |
 | Frontend: Vitest run clean; per-page coverage ≥ 70% | `web/tool-output/vitest.txt` | Frontend-on-call | ☐ | |
 | Manual Parts 23, 24, 55, 56 executed (or explicit defer with rationale) | This sheet | QA-on-call | ☐ | |
 | Demo stack `docker compose up -d --build` smoke (`/health` 200, `/ready` 200) | curl receipt | QA-on-call | ☐ | |
 | `govulncheck ./...` clean (or deferred-call advisories tracked in `gap-backlog`) | `tool-output/govulncheck.json` | Security-on-call | ☐ | |
 | QA-doc drift guards green (Part-count + cert-count) | CI run URL | QA-on-call | ☐ | |
 | FSM transition coverage tables (`coverage-audit-2026-04-27/tables/fsm-coverage.md`) — Existential FSMs ≥80% legal + 100% illegal | This sheet | QA-on-call | ☐ | |
 **Sign-off owner:** ______________________ &nbsp;&nbsp;**Date:** ______ &nbsp;&nbsp;**Tag:** v__.__.__
 ## Mutation Testing Targets & Kill Rate
 Mutation testing exposes which assertions are actually load-bearing — tests can pass against broken code if mutations survive, which is a coverage trap. The audit's Phase 0 attempted to run `go-mutesting` on the Existential cluster but was blocked by a Go 1.25 / arm64 incompatibility in `osutil@v1.6.1` (uses `syscall.Dup2` which is undefined on linux/arm64). The operator-runnable workaround uses a fork that targets `unix.Dup3` instead.
 | Package | Risk class | Target kill rate | Last measured | Tool |
 |---|---|---|---|---|
 | `internal/crypto` | Existential | ≥90% | unmeasured (sandbox-blocked, operator-runnable) | go-mutesting |
 | `internal/pkcs7` | Existential | ≥90% | unmeasured | go-mutesting |
 | `internal/connector/issuer/local` | Existential | ≥90% | unmeasured | go-mutesting |
 | `internal/connector/issuer/acme` | Existential | ≥80% (catch-up; failure-mode coverage 55.6% per Bundle J) | unmeasured | go-mutesting |
 | `internal/connector/issuer/stepca` | Existential | ≥85% (post-Bundle-L.B coverage at 90.4%) | unmeasured | go-mutesting |
 | `internal/api/middleware` | High | ≥80% | unmeasured | go-mutesting |
 | `internal/validation` | Existential (CWE-78 / CWE-113 boundary) | ≥90% | unmeasured | go-mutesting |
 | `web/src/utils/safeHtml.ts` | Frontend (XSS gate) | ≥90% | unmeasured | Stryker |
 ### Operator command (per package)
 ```bash
 # Use the avito-tech fork that supports linux/arm64 + Go 1.25.
 go install github.com/avito-tech/go-mutesting/cmd/go-mutesting@latest
 mkdir -p tool-output
 $(go env GOPATH)/bin/go-mutesting --debug ./internal/crypto/... \
  > tool-output/mutation-crypto.txt 2>&1
 grep -oE 'mutation score is [0-9.]+' tool-output/mutation-crypto.txt | tail -1
 ```
 **Acceptance:** ≥80% (Existential) / ≥70% (High). Anything below is a Medium finding; triage entries go in `coverage-audit-2026-04-27/gap-backlog.md`. This subsection moves mutation testing from "future work" to "documented release gate."
 ## Adding New Tests
 When a new feature ships:
 1. **Add a Part section** in `qa_test.go` following the numbering convention in the Coverage Map below
 2. **API tests**: use `c.get()`, `c.post()`, `c.bodyStr()`, `c.getJSON()`, `c.timedGet()`
 3. **Source checks**: use `fileExists(t, "relative/path")` and `fileContains(t, "path", "substring")`
 4. **DB checks**: use `openQADB(t)` and `db.queryInt(t, "SELECT ...")`
 5. **Cleanup**: always use `t.Cleanup()` for data created during tests
 6. **Skip if external**: use `t.Skip("Requires X — manual test")` with a clear reason
 ## Version History
 - **v1.3** (April 2026, post-Bundle-P) — QA Doc Strengthening shipped. New top-of-doc Test Suite Health dashboard (regenerated via `make qa-stats`). New Coverage by Risk Class table after the Coverage Map. New Release Day Sign-Off Matrix and Mutation Testing Targets sections. CI seed-count + Part-count drift guards land in `.github/workflows/ci.yml` so future doc drift fails CI. Bundle P closes M-007 / M-010 / M-011 / M-012 (structural strengthening) + M-008 (Mutation Testing Targets).
 - **v1.2** (April 2026, post-coverage-audit) — Documented Parts 55–56 (I-004 Agent Soft-Retirement, I-005 Notification Retry & Dead-Letter) and surfaced Parts 23–24 (S/MIME & EKU; OCSP/CRL) as not-yet-automated. 56 Parts total in `testing-guide.md`; 49 live `Part_*` automation wrappers in `qa_test.go` + 4 new `Skip` stubs for Parts 23/24/55/56 = 53 wrappers (Parts 15–17 remain covered by source-checks in Parts 42–46). Reconciled seed-data section to actual `seed_demo.sql` counts (12 agents, 13 issuers; certs were already accurate at 32). Bundle I of the 2026-04-27 coverage-audit closure plan.
 - **v1.1** (April 2026) — Added Parts 53–54 (M47: Kubernetes Secrets target + AWS ACM PCA issuer). 54 Parts total, ~164 automated subtests.
 - **v1.0** (April 2026) — Initial release covering all 52 Parts of testing-guide.md v2.1. Replaces `qa-smoke-test.sh`.
@@ -1,93 +0,0 @@
 # Release Sign-Off
 > Last reviewed: 2026-05-05
 Release-day checklist for tagging a new certctl release. Walks through the gates that must be green before pushing the tag, in the order they should be verified.
 ## Pre-release: code state
 | Gate | How to check | Pass |
 |---|---|---|
 | `master` is at the commit you intend to tag | `git log -1 --format='%H %s'` | ☐ |
 | Working tree clean | `git status -sb` | ☐ |
 | Local matches GitHub | `curl -sS https://api.github.com/repos/certctl-io/certctl/commits/master \| grep -oE '"sha": "[a-f0-9]+"' \| head -1` matches local | ☐ |
 | `WORKSPACE-CHANGELOG.md` updated with the release's milestones | manual review | ☐ |
 | `certctl/CHANGELOG.md` updated (release-facing) | manual review | ☐ |
 | Migration ladder ends cleanly | `ls migrations/*.up.sql \| sort \| tail -3` shows the right last migration | ☐ |
 ## Pre-release: automated gates (CI)
 | Gate | How to check | Pass |
 |---|---|---|
 | CI pipeline green on the tag-target commit | GitHub Actions web UI | ☐ |
 | `make verify` clean locally | run from repo root | ☐ |
 | `go test -race -count=1 ./...` clean | full race check | ☐ |
 | `golangci-lint run ./...` clean | local lint | ☐ |
 | `govulncheck ./...` clean | vulnerability scan | ☐ |
 | Coverage thresholds met (service ≥55%, handler ≥60%, domain ≥40%, middleware ≥30%) | `go test -coverprofile=cover.out ./... && go tool cover -func=cover.out` | ☐ |
 | Frontend type-check + Vitest + Vite build clean | `cd web && npm run typecheck && npm run test && npm run build` | ☐ |
 ## Pre-release: manual QA passes
 | Surface | Checklist | Pass |
 |---|---|---|
 | Local stack boots clean from scratch | `qa-prerequisites.md` Steps 1-4 green | ☐ |
 | GUI QA checklist | `gui-qa-checklist.md` end to end | ☐ |
 | End-to-end test environment | `test-environment.md` Steps 1-14 green | ☐ |
 | Performance baselines | `performance-baselines.md` four spot checks within bounds | ☐ |
 | Helm chart deploys clean | `helm-deployment.md` install + verify | ☐ |
 | ACME server interop (cert-manager) | `make acme-cert-manager-test` green | ☐ |
 | ACME server RFC conformance (lego) | `make acme-rfc-conformance-test` green | ☐ |
 ## Release artefact verification
 After the release workflow runs (triggered by tag push), verify the published artefacts:
 | Artefact | How to verify | Pass |
 |---|---|---|
 | Cosign keyless OIDC signature on `checksums.txt` | per `docs/reference/release-verification.md` step 2 | ☐ |
 | SLSA Level 3 provenance on each binary | step 3 | ☐ |
 | Container image signature + SBOM + provenance | step 4 | ☐ |
 | Release notes published on GitHub Releases page | manual review | ☐ |
 | ghcr.io images at `ghcr.io/certctl-io/certctl-{server,agent}:<tag>` pullable | `docker pull` round-trips | ☐ |
 ## Branch protection + tag push
 | Gate | How to check | Pass |
 |---|---|---|
 | `master` branch protection rule allows the tag push | Repository Settings → Branches | ☐ |
 | Tag pushed | `git tag -s v<version> -m 'Release v<version>'; git push origin v<version>` | ☐ |
 | Release workflow kicked off in GitHub Actions | watch the Actions tab | ☐ |
 ## Post-release
 | Gate | How to check | Pass |
 |---|---|---|
 | Release workflow completed without errors | GitHub Actions | ☐ |
 | Sample binary downloaded and Cosign-verified by an operator who is not the release author | another team member | ☐ |
 | `WORKSPACE-CHANGELOG.md` notes the tag commit SHA | manual edit | ☐ |
 | workspace-tracking "Active Focus" → "Current tag" updated | manual edit | ☐ |
 | `certctl.io/index.html` star count + `data-gh-version` rendering picks up the new tag | open the landing page in 6+ hours (cache TTL) | ☐ |
 | Reddit / Hacker News / LinkedIn announcement drafted (if a major release) | per the operator's promotion playbook | ☐ |
 ## If a gate fails
 Revert the tag push immediately:
 ```bash
 git push --delete origin v<version>
 git tag -d v<version>
 ```
 Investigate, fix, re-tag.
 ## Related docs
 - [`docs/contributor/qa-prerequisites.md`](qa-prerequisites.md) — local stack prereqs
 - [`docs/contributor/test-environment.md`](test-environment.md) — full local environment tutorial
 - [`docs/contributor/gui-qa-checklist.md`](gui-qa-checklist.md) — GUI manual QA pass
 - [`docs/contributor/testing-strategy.md`](testing-strategy.md) — what we test in CI vs deep-scan vs manual QA
 - [`docs/contributor/ci-pipeline.md`](ci-pipeline.md) — CI shape and regression guards
 - [`docs/operator/performance-baselines.md`](../operator/performance-baselines.md) — performance regression spot checks
 - [`docs/operator/helm-deployment.md`](../operator/helm-deployment.md) — Helm install + verify
 - [`docs/reference/release-verification.md`](../reference/release-verification.md) — Cosign / SLSA / SBOM verification procedure
@@ -1,200 +0,0 @@
 # certctl Testing Strategy & Deep-Scan Operator Runbook
 > Last reviewed: 2026-05-05
 This doc covers the **testing topology** (per-PR fast gates vs. daily deep-scan
 gates), and the **operator runbook** for re-running each deep-scan tool locally
 when the CI receipt is ambiguous or when an operator wants to validate a fix
 before the next scheduled scan.
 For the manual end-to-end QA playbook, see [`testing-guide.md`](../testing-guide.md).
 For the security posture / per-finding closure log, see [`security.md`](../operator/security.md).
 ## CI workflow split
 certctl runs two GitHub Actions workflows:
 - **`.github/workflows/ci.yml`** — runs on every push/PR. Fast feedback only.
  Includes `gofmt`, `go vet`, `golangci-lint`, `go test -short -count=1`,
  `govulncheck`, the per-layer coverage gates, and the regression-grep guards
  (the M-009 mutation budget, the L-001 InsecureSkipVerify guard, the H-001
  Dockerfile SHA-pin guard, the M-012 USER-directive guard, etc.).
 - **`.github/workflows/security-deep-scan.yml`** — runs daily 06:00 UTC and on
  manual dispatch. Heavyweight tools that need docker, network egress to
  scanner registries, or wall-clock budgets the per-PR check can't tolerate.
  Includes `gosec`, `osv-scanner`, the `-race -count=10` full-suite run,
  `trivy` image scan, `syft` SBOM, ZAP baseline DAST, `nuclei`,
  `schemathesis` OpenAPI fuzz, `testssl.sh`, `go-mutesting` mutation testing,
  and `semgrep p/react-security`.
 Receipts from each scheduled run are uploaded as a 30-day-retention artefact
 named `security-deep-scan-<run-id>`. Audit them via the GitHub Actions UI;
 download the artefact zip for any scan that surfaces a finding.
 ## Operator runbook — local re-run procedures
 These are the same commands the workflow runs, intended for an operator with
 a workstation that has docker + the Go toolchain installed. The local-run
 shape is identical to CI; the difference is wall-clock and the artefact
 location (CI uploads; local writes to `$PWD`).
 ### Mutation testing (D-003)
 **Tool:** [`go-mutesting`](https://github.com/zimmski/go-mutesting). Mutates
 each AST node in turn (flips comparisons, swaps return values, removes
 statements) and re-runs the package's tests. A mutant is **killed** if any
 test fails; **surviving** mutants indicate a coverage gap (no test caught
 the bug the mutant introduced).
 **Targets:** the three security-critical packages whose coverage gate is
 **85%** in `ci.yml`:
 - `internal/crypto/`
 - `internal/pkcs7/`
 - `internal/connector/issuer/local/`
 **Acceptance threshold:** ≥80% mutation kill ratio per package. Surviving
 mutants below that threshold get triaged in
 the project's 2026-04-25 mutation-results notes — either
 ship a targeted unit test that kills the mutant, or document an
 equivalent-mutation justification.
 **Local run:**
 ```
 go install github.com/zimmski/go-mutesting/cmd/go-mutesting@latest
 for pkg in ./internal/crypto/... ./internal/pkcs7/... ./internal/connector/issuer/local/...; do
  echo "=== $pkg ==="
  $(go env GOPATH)/bin/go-mutesting "$pkg"
 done
 ```
 The tool prints one line per mutant (`PASS` = killed, `FAIL` = surviving)
 plus a per-package summary `The mutation score is X.YZ`. CPU-bound, single
 core, takes ~10 minutes on a 2024-era laptop for the three packages combined.
 **Sandbox note:** `go-mutesting` writes a mutant copy of the source tree to
 `/tmp/go-mutesting/` per run; needs ≥2 GB free disk. Sandboxed CI runners
 are sized for this; constrained dev sandboxes are not.
 ### DAST baseline (D-004)
 **Tool:** [OWASP ZAP `baseline`](https://www.zaproxy.org/docs/docker/baseline-scan/).
 Spiders the running server's URL surface and runs the OWASP-ZAP active+passive
 rule pack. **Baseline** mode skips the destructive active-scan rules; it's safe
 against a non-throwaway environment.
 **Target:** the live `deploy/docker-compose.yml` stack on `https://localhost:8443`.
 **Acceptance:** zero HIGH/CRITICAL alerts. WARN/INFO alerts get triaged in the
 ZAP report; some are unavoidable (e.g., HSTS preload-list nag is a deployment
 recommendation, not a server defect).
 **Local run:**
 ```
 docker compose -f deploy/docker-compose.yml up -d
 sleep 20  # wait for /ready to flip OK; check `curl --cacert deploy/test/certs/ca.crt https://localhost:8443/ready`
 docker run --rm --network host \
  -v "$PWD":/zap/wrk \
  ghcr.io/zaproxy/zaproxy:stable \
  zap-baseline.py -t https://localhost:8443 \
  -r zap-report.html -J zap-report.json
 docker compose -f deploy/docker-compose.yml down
 ```
 The HTML report opens in a browser; the JSON is machine-readable for triage.
 ### TLS audit (D-005)
 **Tool:** [`testssl.sh`](https://testssl.sh/). Probes the TLS handshake and
 each enabled cipher suite; reports protocol-version weaknesses, cipher
 weaknesses, certificate-chain issues, and known CVE patterns (Heartbleed,
 ROBOT, BEAST, etc.).
 **Target:** the live stack on `https://localhost:8443`.
 **Acceptance:** zero HIGH/CRITICAL findings. certctl pins
 `tls.Config.MinVersion = tls.VersionTLS13` (`cmd/server/tls.go`), so anything
 that surfaces is either (a) a real defect, (b) a testssl false positive, or
 (c) a deployment-config issue worth documenting in the operator runbook.
 **Local run:**
 ```
 docker compose -f deploy/docker-compose.yml up -d
 sleep 20
 docker run --rm --network host \
  -v "$PWD":/data \
  drwetter/testssl.sh:latest \
  --jsonfile /data/testssl.json https://localhost:8443
 docker compose -f deploy/docker-compose.yml down
 # Filter to actionable severities
 jq '[.scanResult[] | select(.severity == "HIGH" or .severity == "CRITICAL")]' testssl.json
 ```
 ### Frontend semgrep (D-007)
 **Tool:** [`semgrep`](https://semgrep.dev/) with the maintained
 [`p/react-security` ruleset](https://semgrep.dev/p/react-security). Catches
 React-specific XSS / injection patterns: `dangerouslySetInnerHTML` without
 sanitization, `target="_blank"` without `rel="noopener noreferrer"`,
 `href={userInput}`, `eval`, `document.write`, etc.
 **Target:** the frontend source tree at `web/src/`.
 **Acceptance:** zero findings. Bundle 8 already verified
 `dangerouslySetInnerHTML` count at zero and the `target="_blank"`
 rel-noopener pin via simple grep guards in `ci.yml`; semgrep adds defence
 in depth — it catches escape patterns the greps don't see (e.g.,
 `href={user_input}`, runtime `eval`, `document.write`).
 **Local run:**
 ```
 docker run --rm -v "$PWD":/src returntocorp/semgrep:latest \
  semgrep --config=p/react-security --json /src/web/src \
  > semgrep-react.json
 # Count findings
 jq '.results | length' semgrep-react.json
 # Pretty-print findings
 jq '.results[] | {rule_id: .check_id, path, line: .start.line, message: .extra.message}' semgrep-react.json
 ```
 If the count is non-zero, every result has a `check_id` (e.g.
 `react.dangerouslySetInnerHTML`) and a `message` describing the escape
 pattern. Triage each: either fix the call site, or — for legitimate edge
 cases — add a `// nosem: <check_id> — <reason>` directive on the
 preceding line.
 ## Cadence
 | Tool                 | Trigger                            | Wall-clock | Owner          |
 |----------------------|------------------------------------|------------|----------------|
 | go-mutesting         | daily deep-scan + manual dispatch  | ~10 min    | maintainers    |
 | ZAP baseline (DAST)  | daily deep-scan + manual dispatch  | ~5 min     | maintainers    |
 | testssl.sh           | daily deep-scan + manual dispatch  | ~3 min     | maintainers    |
 | semgrep react        | daily deep-scan + manual dispatch  | ~1 min     | maintainers    |
 | `make verify`        | every commit (pre-push)            | ~1 min     | every developer |
 | ci.yml fast gates    | every push/PR                      | ~3 min     | every developer |
 Re-run any of the deep-scan tools locally when:
 - A CI receipt surfaces an unexpected finding and you want to bisect against
  a local change before pushing.
 - You're cutting a release tag and want belt-and-suspenders evidence beyond
  the most recent scheduled scan.
 - You're adding a new feature in the relevant surface (crypto code →
  re-run mutation testing; new HTTP handler → re-run schemathesis + ZAP;
  new TLS-config knob → re-run testssl).
 ## Related docs
 - [`docs/operator/security.md`](../operator/security.md) — security posture, per-finding closure log.
 - [`docs/testing-guide.md`](../testing-guide.md) — manual end-to-end QA playbook.
 - [`.github/workflows/ci.yml`](../.github/workflows/ci.yml) — per-PR fast gates.
 - [`.github/workflows/security-deep-scan.yml`](../.github/workflows/security-deep-scan.yml) — daily deep-scan gates.
 - [`scripts/install-security-tools.sh`](../scripts/install-security-tools.sh) — Go-host-installed tools (the docker-based tools are not in this script).
@@ -0,0 +1,97 @@
 # Git history normalization — 2026-05-13
 > Last reviewed: 2026-05-13
 This page documents a one-time normalization of certctl's git history
 that landed on `master` on 2026-05-13. If you are reading this because
 your clone failed to fast-forward, or because a commit SHA you bookmarked
 no longer resolves, this is the explanation.
 ## What changed
 Every commit's `author` and `committer` metadata was rewritten to a
 single canonical identity (`shankar0123 <skreddy040@gmail.com>`). The
 14 pre-rewrite author identities — operator name variants plus
 AI/automation identities (Claude, Copilot, cowork agent, certctl-bot,
 etc.) — collapsed to that one canonical author.
 No source-code content was changed by the rewrite. Every line of code
 in every commit is byte-for-byte identical to its pre-rewrite version.
 Only the `author` and `committer` metadata fields were touched; commit
 messages, subject lines, milestone IDs (M49, L-1, etc.), and every
 other line of every commit's body are preserved verbatim.
 ## Why
 Two reasons:
 1. **LLC ownership transfer.** The codebase is now legally owned by
   **certctl LLC**, which the operator incorporated to hold rights in
   the project. The BSL 1.1 Licensor field in `LICENSE` flipped from a
   natural-person name to `certctl LLC` in the same change set. Uniform
   per-commit authorship under one canonical operator identity makes
   the chain of title between the codebase and the LLC unambiguous.
 2. **Pre-traction cleanup.** The rewrite cost of git-history
   normalization scales with how many external clones and references
   have calcified against specific commit SHAs. Doing it now, before
   the project has a large external surface, minimizes disruption to
   downstream consumers.
 ## What is preserved
 A complete off-platform bundle backup of the pre-rewrite tree is held
 by the operator (off-repo, not pushed). It contains every original
 commit SHA, every original author identity, and the full ref graph as
 it existed before the rewrite. The bundle is the immutable
 preservation record and is recoverable forever.
 An `archive/pre-author-normalization-2026-05-13` tag briefly existed
 on origin pointing at the pre-rewrite tip but was removed when the
 operator opted to clean the contributor graph of pre-rewrite
 authorship signal. The bundle remains as the canonical archive — any
 forensic question about pre-rewrite state can be answered by loading
 the bundle into a fresh clone (`git clone pre-rewrite-2026-05-13.bundle`).
 ## Recovering after the rewrite
 If you had a clone of certctl from before 2026-05-13, your local
 history diverged from origin's at the rewrite. Easiest recovery:
 ```bash
 cd certctl
 git fetch origin
 git fetch origin --tags
 git reset --hard origin/master
 ```
 This force-aligns your local tree with the new origin. Any local
 branches you had based on pre-rewrite history will need rebasing onto
 the new master.
 If you need to inspect the pre-rewrite state for a forensic or
 diligence question, contact the operator directly — the off-platform
 bundle is the canonical archive and is available on request.
 ## Container images and release tarballs
 ghcr.io container images that were published before the rewrite
 (`ghcr.io/certctl-io/certctl-{server,agent}:<old-tag>`) remain pullable
 indefinitely. Their OCI source-SHA labels reference commit SHAs that
 no longer resolve in the public origin — the images themselves still
 work; only the source-SHA back-reference is now orphan. New release
 images published after the rewrite reference current SHAs normally.
 If you downloaded a release tarball before the rewrite, the tarball's
 contents are unchanged; only its associated `git` SHA differs from the
 current `v2.x.y` tag (which has been re-pointed to the rewritten
 commit at the same logical point in history).
 ## Operational note for contributors
 Future contributions to certctl should be authored under the
 operator's canonical git identity. Pull requests from external
 contributors will need a Contributor License Agreement (CLA) workflow,
 which the project will set up before accepting external PRs. Until
 then, the project does not solicit or accept external code
 contributions.
@@ -55,6 +55,29 @@ This is the load-bearing two-person-integrity contract. Pinned by:
 - `internal/service/approval_test.go::TestApproval_Approve_RejectsSameActor` — service-level pin.
 - `internal/api/handler/approval_test.go::TestApproval_HandlerApproveAsSameActor_Returns403` — handler-level pin (HTTP 403 + body contains "two-person integrity").
 ## Enforcement invariants (COMP-006 closure)
 Acquisition-audit COMP-006 closure (Sprint 7 ACQ, 2026-05-16). The audit flagged COMP-006 as UNKNOWN because it couldn't independently verify that the approval workflow was bullet-tight — i.e., that a denied approval definitely results in NO certificate being signed, and an approved approval definitely lets the issuance proceed. This subsection documents the enforcement chain end-to-end and names the tests that pin each layer.
 **Layer 1 — Issuance gate.** `internal/service/certificate.go::CertificateService.Create` (around L341-373) reads `CertificateProfile.RequiresApproval`. When true, the created Job is stamped `JobStatusAwaitingApproval` (not `Pending`), AND a parallel `ApprovalRequest` row is created. The job processor never touches `AwaitingApproval` rows.
 **Layer 2 — Approval state machine.** `internal/service/approval.go::ApprovalService.Reject` and `Approve` flip the approval row + the job row atomically:
 - `Reject` → approval=`Rejected`, job=`Cancelled` (pinned by `internal/service/approval_test.go::TestApproval_Reject_TransitionsJobFromAwaitingApprovalToCancelled`).
 - `Approve` → approval=`Approved`, job=`Pending` (pinned by `TestApproval_Approve_TransitionsJobFromAwaitingApprovalToPending`).
 The "already terminal" guard (`TestApproval_Approve_RejectsAlreadyDecided`) prevents a rejected approval from later being flipped to approved.
 **Layer 3 — Job claim filter (the LOAD-BEARING SQL invariant).** `internal/repository/postgres/job.go::JobRepository.ClaimPendingJobs` (around L296-310) issues:
 ```sql
 SELECT ... FROM jobs WHERE status = $1
 ```
 with `$1 = JobStatusPending`. Cancelled jobs are therefore **never** returned to `ProcessPendingJobs`, so the certificate-issuance call path (the only path that signs certs) is unreachable for a denied approval. This SQL filter is the load-bearing "no cert if denied" enforcement — Layer 2 transitions the job to `Cancelled`, Layer 3 ensures `Cancelled` jobs are inert.
 **Composition pin.** `internal/service/approval_test.go::TestApproval_COMP006_DenyChainPinsNoCertIfRejected` and `TestApproval_COMP006_ApproveChainPinsJobReachesPending` re-attest the Layer-2-to-Layer-3 handoff in a single named test pair for future auditors. A refactor that, e.g., silently transitioned a denied approval's job to `Pending` instead of `Cancelled` would trip these tests before shipping.
 ## Operator playbook: "I need to approve a renewal"
 ```bash
@@ -0,0 +1,161 @@
 # Audit-trail tamper-evidence (audit_events hash chain)
 > Last reviewed: 2026-05-16
 Sprint 6 COMP-001-HASH closure. The `audit_events` table has two
 layered defenses against history rewrites:
 | Layer | Migration | What it blocks |
 |---|---|---|
 | **WORM trigger** | `000018_audit_events_worm.up.sql` | The application role cannot `UPDATE` or `DELETE` rows (tamper-**prevention**). |
 | **Hash chain** | `000047_audit_events_hash_chain.up.sql` | A compliance superuser (DB-superuser-equivalent) who bypasses the WORM trigger CAN still rewrite rows, but the rewrite is **detectable** — every subsequent `audit_events_verify_chain()` walk reports the first broken row's id + position (tamper-**evidence**). |
 This document covers the hash-chain layer. The WORM layer is
 documented inline in `migrations/000018_audit_events_worm.up.sql`.
 ## Why a hash chain in addition to WORM
 The WORM trigger documents (in its header comment) that a compliance
 superuser role exists by design — backup-restore, retention purges,
 and breach-recovery operators need a way through. Without a hash
 chain, that role can rewrite any row's `actor` / `action` / `details`
 content with no on-disk trace.
 HIPAA §164.312(b), FedRAMP AU-9, and NIST 800-53 AU-10 want
 tamper-**evidence**, not just tamper-prevention. The hash chain
 provides it: every row carries a `row_hash = sha256(prev_hash || id
 || actor || actor_type || action || resource_type || resource_id
 || details::text || timestamp_iso8601_utc || event_category)`, and
 the genesis row's `prev_hash` is `NULL`. Mutating any field in any
 row breaks the chain at that row's position; the verifier returns
 the first break.
 ## The verifier function
 `audit_events_verify_chain()` is a STABLE plpgsql function shipped
 in migration 000047. It walks every row in `(timestamp ASC, id ASC)`
 order, recomputes each row's expected hash, and returns:
 ```
 first_break_id  TEXT  -- NULL if the chain validated end-to-end
 first_break_pos INT   -- 0-indexed position of the first break
 row_count       INT   -- rows walked (= position + 1 on break, else table size)
 ```
 Call it directly from psql:
 ```sql
 SELECT first_break_id, first_break_pos, row_count FROM audit_events_verify_chain();
 ```
 ## Scheduled verification + Prometheus exposure
 The scheduler's `auditChainVerifyLoop` calls the verifier every
 `CERTCTL_AUDIT_CHAIN_VERIFY_INTERVAL` (default 6h) and writes the
 results into the `AuditChainCounter` instance shared with the
 metrics handler. Four metrics get exposed at
 `/api/v1/metrics/prometheus`:
 | Metric | Type | Meaning |
 |---|---|---|
 | `certctl_audit_chain_break_detected_total` | counter | Sticky once non-zero — the actionable alarm. |
 | `certctl_audit_chain_verify_total` | counter | Walks completed. Cross-check that the loop is alive. |
 | `certctl_audit_chain_rows` | gauge | Most recent walk's row count. |
 | `certctl_audit_chain_last_verified_at` | gauge | Unix seconds of most recent walk (0 = never). |
 The recommended alert rule is:
 ```
 ALERT AuditChainBreak
  IF certctl_audit_chain_break_detected_total > 0
  FOR 1m
  LABELS { severity = "page", category = "compliance" }
  ANNOTATIONS {
    summary = "audit_events hash chain break detected — investigate immediately",
    runbook = "<your-runbook-url>/audit-chain-break"
  }
 ```
 Cross-check `certctl_audit_chain_last_verified_at` (should advance
 roughly every `CERTCTL_AUDIT_CHAIN_VERIFY_INTERVAL`) and
 `certctl_audit_chain_verify_total` (should increment monotonically).
 A stalled `_verified_at` with an unchanged `_verify_total` means the
 scheduler loop has died — page on that too.
 ## Performance notes
 The walk is `O(N)` plpgsql over the `audit_events` table. On
 testcontainers + postgres:16-alpine the cost scales linearly:
 | Row count | Walk duration (approx) |
 |---|---|
 | 10k | < 50 ms |
 | 100k | < 500 ms |
 | 1M | 2-3 s |
 | 10M | 25-30 s |
 A 5-minute per-tick context timeout (in
 `internal/scheduler/scheduler.go::runAuditChainVerify`) bounds the
 worst case. Fleets with > 10M audit rows should consider:
 1. Lengthening `CERTCTL_AUDIT_CHAIN_VERIFY_INTERVAL` to 24h.
 2. Pre-aggregating older rows (out of scope today — would require a
   "chain checkpoint" concept that re-anchors the genesis hash to a
   snapshot's row_hash; future work if needed).
 ## What to do when a break is detected
 1. **Don't panic, don't auto-remediate.** The break is a forensic
   signal, not a self-healing event.
 2. **Capture the position + id.** The metric exposes both, but the
   sticky in-memory state (`AuditChainCounter.BrokenAtID`) only
   records the first break. SQL the verifier yourself to enumerate
   downstream breaks:
   ```sql
   SELECT first_break_id, first_break_pos, row_count FROM audit_events_verify_chain();
   ```
 3. **Snapshot the table.** `pg_dump --table=audit_events --data-only`
   to a chain-of-custody location. The next investigative step is
   recovering the original row content from the most recent backup
   that pre-dates the tampering — without this snapshot you can't
   tell which write order caused the divergence.
 4. **Audit the compliance-superuser credential trail.** The break
   implies someone with non-app DB credentials wrote to
   `audit_events`. Rotate the credential, investigate every recent
   session that authenticated under it, and review the WAL for the
   write.
 5. **Restore + cross-reference.** If you keep streaming WAL or
   periodic snapshots, restore a known-good snapshot to a sandbox
   and `EXCEPT`-diff the two `audit_events` tables to enumerate
   every mutated row.
 ## Backfill behavior
 Migration 000047 backfills existing `audit_events` rows in
 `(timestamp ASC, id ASC)` order during its transaction. The WORM
 trigger is temporarily `DISABLE`d for the duration; subsequent
 `ENABLE` is a no-op equivalent. The migration is idempotent — a
 re-run sees `row_hash IS NULL` rows as the only backfill targets, so
 already-hashed rows are not touched.
 Once backfill completes, `row_hash` becomes `NOT NULL`. `prev_hash`
 remains nullable so the genesis row (first row in the chain) stays
 representable.
 ## Operator configuration
 | Env var | Default | Notes |
 |---|---|---|
 | `CERTCTL_AUDIT_CHAIN_VERIFY_INTERVAL` | `6h` | Tick cadence for the scheduler's verify loop. Zero or negative is ignored. |
 ## See also
 - `migrations/000047_audit_events_hash_chain.up.sql` — migration source.
 - `migrations/000018_audit_events_worm.up.sql` — paired WORM trigger.
 - `internal/repository/postgres/audit_chain_test.go` — testcontainers integration tests.
 - `internal/repository/postgres/audit_worm_test.go` — WORM behaviour tests.
 - `internal/scheduler/scheduler.go::auditChainVerifyLoop` — scheduler loop.
 - `internal/service/audit_chain_metric.go` — `AuditChainCounter`.
 - `internal/api/handler/metrics.go` — Prometheus exposer.
@@ -300,6 +300,64 @@ constant, router-level no-rbacGate-wraps-protocol-paths).
  attacks where an attacker captures a logout JWT and replays it.
 - **Cache-Control: no-store** on the response per spec §2.5.
 ### Userinfo + BCL SSRF parity (post-SEC-001 follow-up)
 The original SEC-001 closure (Sprint 1, 2026-05-16) routed two OIDC
 discovery legs — `test_discovery.go` dry-run and `service.go` runtime
 provider load — through `validation.SafeHTTPDialContext` via the
 `SafeOIDCContext(ctx)` helper at
 [`internal/auth/oidc/safehttp.go`](../../internal/auth/oidc/safehttp.go).
 The acquisition-audit follow-up (2026-05-16) flagged two adjacent
 call sites the sweep missed; both are now wrapped identically.
 - **SEC-020 — Userinfo fallback (`fetchUserinfoGroups`).**
  `internal/auth/oidc/service.go` previously called
  `entry.provider.UserInfo(ctx, ts)` with the bare request context
  on the userinfo-fallback leg (operator opt-in when an IdP doesn't
  surface groups in the ID token). go-oidc/v3's `Provider.UserInfo`
  derives its `http.Client` from `ctx` via `getClient(ctx)`
  (`oidc.go:61-65`); without an override the internal `doRequest`
  falls through to `http.DefaultClient` — no SSRF guard, no DNS-
  rebinding re-resolve at dial time. An IdP whose discovery doc
  advertises a `userinfo_endpoint` pointing at a reserved address
  (loopback, cloud-metadata `169.254.169.254`, RFC 1918) would
  trigger an unguarded egress at userinfo-fetch time. Fixed by
  wrapping `ctx` via `SafeOIDCContext(ctx)` before both
  `oauthConfig.TokenSource` and `provider.UserInfo`. Pinned by
  `TestFetchUserinfoGroups_SSRF_BlocksReservedAddress`.
 - **SEC-021 — Back-channel logout discovery re-fetch.**
  `internal/api/handler/auth_session_oidc_bcl.go::Verify` performs
  a per-request `gooidc.NewProvider(ctx, matched.IssuerURL)` to
  fetch the JWKS for verifying the BCL token's signature. Same
  bare-ctx shape — an IdP whose registered `IssuerURL` resolves to
  a reserved address (or that is rebinding to one at logout time)
  would dial an unguarded HTTPS egress. Fixed by wrapping via
  `oidcsvc.SafeOIDCContext(ctx)` before `NewProvider`. Pinned by
  `TestDefaultBCLVerifier_SSRF_BlocksReservedAddress`.
 - **Context-key shape (why a single wrap covers both legs).**
  `gooidc.ClientContext` is implemented as
  `context.WithValue(ctx, oauth2.HTTPClient, client)` (go-oidc
  v3.18.0 `oidc.go:57-59`). Both go-oidc's `getClient` AND
  `golang.org/x/oauth2`'s `internal.ContextClient` read the same
  `oauth2.HTTPClient` key. So the single `SafeOIDCContext` wrap
  covers go-oidc-driven HTTP (Provider.UserInfo, NewProvider
  discovery, Verifier JWKS) AND oauth2-driven HTTP
  (Config.TokenSource refresh, Config.Exchange). No additional
  `context.WithValue(ctx, oauth2.HTTPClient, ...)` is required.
 - **Out-of-scope: RFC 1918.** Per the `IsReservedIP` policy
  documented at [`internal/validation/ssrf.go:15-32`](../../internal/validation/ssrf.go),
  RFC 1918 ranges are NOT treated as reserved by the SSRF guard.
  certctl is designed to manage certificates inside private
  networks; filtering 10/8 + 172.16/12 + 192.168/16 would break
  the primary use case. Operators on hosted IaaS who want
  RFC 1918 treated as reserved can opt in via the future
  `CERTCTL_BLOCK_RFC1918_OUTBOUND` toggle (see acquisition-audit
  Sprint 5 RED-005). The Sprint 1 SSRF parity fix above closes
  the loopback / link-local / cloud-metadata leg only.
 ### OIDC first-admin bootstrap
 - **Coexists with the env-var-token bootstrap path.** Both can be
@@ -94,6 +94,46 @@ helm upgrade certctl deploy/helm/certctl/ \
 Postgres state survives the upgrade (the PVC is retained). The server / agent images bump per the chart's `image.tag`. See [`docs/archive/upgrades/`](../archive/upgrades/) for version-specific upgrade guidance.
 ### 2026-05-16 — ServiceMonitor TLS default flipped (DEPL-004)
 Acquisition-audit DEPL-004 closure. Pre-2026-05-16, `monitoring.serviceMonitor.tlsConfig` was empty by default and the chart template fell through to an implicit `insecureSkipVerify: true`. Post-2026-05-16, the values.yaml default is a real TLS verify against the chart's CA (caFile + serverName matching the existingSecret mount path the chart's Prometheus integration produces).
 The new default works out of the box for the canonical install (the chart's `existingSecret` or cert-manager-emitted Secret mounted at `/etc/prometheus/secrets/certctl-ca/`):
 ```yaml
 # Default in values.yaml (no operator action required for the
 # canonical install path).
 monitoring:
  serviceMonitor:
    enabled: true
    tlsConfig:
      caFile: /etc/prometheus/secrets/certctl-ca/ca.crt
      serverName: certctl-server
 ```
 Operators whose Prometheus pod mounts the CA bundle at a different path override `caFile`:
 ```yaml
 monitoring:
  serviceMonitor:
    enabled: true
    tlsConfig:
      caFile: /path/to/your/ca.crt
      serverName: your-cert-CN
 ```
 Operators who genuinely need `insecureSkipVerify` (demo / dev clusters) must opt in **explicitly** — blanking the `tlsConfig` block trips the chart's `{{ fail }}` guard at render time:
 ```yaml
 monitoring:
  serviceMonitor:
    enabled: true
    tlsConfig:
      insecureSkipVerify: true
 ```
 There is no way to inherit the pre-2026-05-16 implicit-skipVerify behavior silently. Operators with `monitoring.serviceMonitor.enabled: false` (the chart default) need no action — the template short-circuits before the `tlsConfig` block.
 ## Configuration reference
 Every value is documented at `deploy/helm/certctl/values.yaml`. Common tweaks:
@@ -0,0 +1,338 @@
 # Observability — what certctl emits, what it doesn't, and what survives a restart
 > Last reviewed: 2026-05-13
 Use this when:
 - You're sizing certctl's observability surface against your existing
  metrics + tracing + logging stack and want to know exactly what
  drops in cleanly and what gaps you'll need to bridge.
 - You're investigating a "weird metric" or planning a Grafana
  dashboard and need the canonical list of what's exposed.
 - You're running multi-replica or restarting frequently and need to
  understand which counters reset.
 certctl's observability posture is deliberately minimal-but-honest:
 ship the surfaces an operator actually needs to wire into a Prometheus
 + Grafana + Loki stack, and don't make claims the implementation
 can't back. This document is the canonical statement of what's
 emitted, what's deferred, and why.
 ## Metrics — what's emitted
 certctl exposes metrics through two endpoints on the control plane:
 | Endpoint                          | Content-Type                                                      | Audience                         |
 |---|---|---|
 | `GET /api/v1/metrics`             | `application/json`                                                | Dashboards that prefer JSON, ad-hoc curl |
 | `GET /api/v1/metrics/prometheus`  | `text/plain; version=0.0.4; charset=utf-8` (Prometheus exposition) | Prometheus, Grafana Agent, Datadog Agent, Victoria Metrics, any OpenMetrics-compatible scraper |
 The Prometheus endpoint emits standard `# HELP` / `# TYPE` / metric
 lines following the conventions at
 [prometheus.io/docs/instrumenting/exposition_formats](https://prometheus.io/docs/instrumenting/exposition_formats/).
 Metric names are lowercase, snake_case, and prefixed with `certctl_`.
 The implementation is at
 [`internal/api/handler/metrics.go`](../../internal/api/handler/metrics.go).
 ### What's covered
 Run the endpoint against a live deployment for the authoritative list
 (it expands as the service ships more metrics). At time of writing the
 exposition includes:
 - Certificate-inventory gauges: `certctl_certificate_total`,
  `certctl_certificate_active`, `certctl_certificate_expiring_soon`,
  `certctl_certificate_expired`, `certctl_certificate_revoked`.
 - Per-issuer-type issuance histograms:
  `certctl_issuance_duration_seconds{issuer_type=…}` (the 2026-05-01
  issuer-coverage audit closure #4 — this is the load-bearing metric
  for per-issuer SLOs).
 - Server uptime: `certctl_uptime_seconds`.
 ### Prometheus library vs hand-rolled exposition (acquisition diligence)
 certctl writes Prometheus exposition format with `fmt.Fprintf` from
 the metrics handler, not via the `github.com/prometheus/client_golang`
 library. This is intentional for v2.x:
 - The metric surface is shallow (gauges + a handful of histograms with
  static labels). The client library's value is on the registration +
  thread-safe accumulation side, neither of which is load-bearing for
  the current surface.
 - The exposition output is pinned to the spec version explicitly
  (`version=0.0.4`) and is unit-tested against expected output at
  `internal/api/handler/stats_handler_test.go`.
 - Swapping in `client_golang` is a mechanical migration when the
  metric surface grows (per-connector counters + RED-method histograms
  on every handler are the natural next surface), but it has no
  operator-visible behavior change today.
 The migration is on the
 [WORKSPACE-ROADMAP.md](../../WORKSPACE-ROADMAP.md) as a v3 item. If
 you're an acquirer reading this: the question to ask is "does the
 metric surface meet our SLO needs today" — not "is the right library
 under the hood." If the answer to the first question is yes, the
 second is a refactor, not a feature gap.
 ## Tracing — OTLP surface available, instrumentation pending
 Sprint 6 ACQ DEPL-006 closure (2026-05-16) stood up the OTel tracer-
 provider surface. Operators with an OTel collector can opt in via:
 ```
 CERTCTL_OTEL_ENABLED=true
 OTEL_EXPORTER_OTLP_ENDPOINT=https://otel-collector.example.com:4318
 ```
 When `CERTCTL_OTEL_ENABLED` is true, `cmd/server/main.go` calls
 `internal/observability.Init` which:
 - Constructs an OTLP/HTTP exporter (chosen over OTLP/gRPC to keep
  the dependency surface narrow — see `internal/observability/otel.go`
  header for the transport-choice rationale).
 - Registers a real `sdktrace.TracerProvider` as the otel global.
 - Honors the standard OTel env vars (`OTEL_EXPORTER_OTLP_ENDPOINT`,
  `OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_EXPORTER_OTLP_INSECURE`,
  `OTEL_SERVICE_NAME` overrides the default `certctl-server`, etc.).
 - Defers a graceful shutdown that flushes the in-flight batcher.
 What this **does not** ship yet:
 - No per-handler / per-DB / per-connector span instrumentation in
  the certctl code base. The OTel SDK emits the spans it generates
  internally (process resource attributes, eventual stdlib HTTP
  spans), but certctl-domain spans (issuance, renewal, deployment,
  agent enrollment) are a v2.3 roadmap follow-up.
 - No tracing-correlated metric exemplars in the Prometheus
  histograms above. Those still ship the per-issuer latency signal
  without per-request fan-out.
 - No backwards-compat shim — operators who never set
  `CERTCTL_OTEL_ENABLED` (the default) see zero behavior change.
  The init returns a no-op shutdown so the deferred call is safe
  to invoke unconditionally.
 When this matters today:
 - Operators wiring up a v3 instrumentation effort have the OTel
  surface in place; they only need to add `tracer.Start(ctx, "…")`
  call sites in the handler/service code.
 - Operators evaluating certctl for acquisition / due-diligence see
  an opt-in OTel surface in the current release rather than a "v3
  roadmap item" — a useful signal for buyer credibility per the
  acquisition-thesis framing in `WORKSPACE-ROADMAP.md` §3.
 Existing correlation surfaces stay in place until span coverage
 ships:
 - Structured logs include a `request_id` you can correlate across
  the server log stream. See
  [`internal/api/middleware/request_id.go`](../../internal/api/middleware/request_id.go).
 - The Prometheus histogram
  `certctl_issuance_duration_seconds{issuer_type=…}` carries the
  same per-issuer latency signal a trace span would, just without
  the per-request fan-out.
 Per-handler / per-query / per-connector span instrumentation is
 tracked in [WORKSPACE-ROADMAP.md](../../WORKSPACE-ROADMAP.md) under
 §2 (NHI / Agent Identity, Phase 4 in the path-b build plan).
 ## Logging
 certctl emits structured JSON logs to stdout via the stdlib
 `log/slog` package. Every line carries `time`, `level`, `msg`, and —
 where relevant — `request_id`, `actor_id`, and a contextual subject
 (`certificate_id`, `issuer_id`, `agent_id`, etc.).
 Log level is controlled by `CERTCTL_LOG_LEVEL` (`debug` / `info` /
 `warn` / `error`); defaults to `info`. There is no in-process log
 ingest — operators are expected to collect from container stdout
 into their existing log pipeline (Loki, CloudWatch Logs, Datadog,
 ELK, Splunk, etc.).
 No log line contains private-key material, bearer tokens, OIDC
 client secrets, or session cookies. The break-glass login path
 explicitly scrubs the password before it reaches the audit subsystem
 (see [`docs/operator/auth-threat-model.md`](auth-threat-model.md) §
 "Break-glass token leak").
 ## Rate-limit behavior — configurable backend (memory or postgres)
 The sliding-window-log rate limiters used across certctl's
 authenticated-but-shared-credential code paths (break-glass login,
 OCSP per-IP, cert-export per-actor, EST per-principal, EST
 failed-basic source-IP) carry a **configurable backend**. The
 operator picks between two implementations via
 `CERTCTL_RATE_LIMIT_BACKEND`:
 | Value      | When to use                                          |
 |------------|------------------------------------------------------|
 | `memory`   | Default. Single-replica deploys; sketchpad / dev.    |
 | `postgres` | HA deploys (`server.replicas > 1`). Cross-replica-consistent. |
 Phase 13 Sprint 13.2/13.3 (architecture diligence audit ARCH-M1
 closure) replaced the prior single-process limitation with a
 substantive close: when the operator opts into `postgres`, all
 replicas share the same
 `rate_limit_buckets` table (migration 000046) and per-key access is
 arbitrated via `SELECT FOR UPDATE` row locks. A 3-replica cluster
 hitting one rate-limited endpoint concurrently sees exactly the
 configured cap succeed across the cluster — not 3× the cap as the
 old per-process backend would have allowed.
 ### Operator decision tree
 ```
 Single replica (server.replicas = 1, the helm chart default)?
  └─ Use CERTCTL_RATE_LIMIT_BACKEND=memory (the default; no action
     required). Bucket lookups stay in-process; zero DB round-trips
     on the hot path.
 Two or more replicas?
  └─ Use CERTCTL_RATE_LIMIT_BACKEND=postgres. Two extra DB round-trips
     per Allow call (BEGIN ... SELECT FOR UPDATE ... UPDATE ... COMMIT);
     acceptable on the gated hot path. The Sprint 13.2 multi-replica
     integration test pins exactly-cap enforcement across N replicas
     as the closure proof.
 ```
 ### Inventory
 | Limiter                                              | Scope                | Window | Cap                            |
 |---|---|---|---|
 | Break-glass login (per source-IP)                    | `internal/api/handler/auth_breakglass.go` | 60s   | 5 attempts                     |
 | OCSP query (per source-IP)                           | `internal/api/handler/certificates.go`    | 60s   | configurable (`CERTCTL_OCSP_RATE_LIMIT_PER_IP_MIN`) |
 | Cert export (per actor)                              | `internal/api/handler/export.go`          | 1h    | configurable (`CERTCTL_CERT_EXPORT_RATE_LIMIT_PER_ACTOR_HR`) |
 | EST per-principal CSR enrollment                     | `internal/api/handler/est.go`             | 24h   | configurable (per-profile `RateLimitPerPrincipal24h`) |
 | EST HTTP-Basic source-IP failed-auth                 | `internal/api/handler/est.go`             | 60m   | 10 attempts                    |
 | SCEP/Intune per-device challenge                     | `internal/scep/intune/`                   | 60s   | configurable (`*_PER_MINUTE`)  |
 | ACME per-account orders / key-change / challenge-respond | `internal/service/acme.go`            | 1h    | configurable                   |
 The `CERTCTL_RATE_LIMIT_BACKEND` selector applies to the first five
 (the cmd/server-wired limiters). The SCEP/Intune wrapper + the ACME
 per-account limiter ride their own internal accounting today; both
 are tracked as follow-ups in WORKSPACE-ROADMAP.md.
 ### Backend internals
 Both backends share the algorithm: sliding-window log + per-key
 bucket + prune-on-Allow.
 **Memory backend (`memory`)** — per-process map keyed by bucket key;
 mutex-guarded; package-level LRU cap prevents unbounded growth under
 adversarial key cardinality (default 100,000 keys per limiter
 instance; oldest-by-newest-timestamp evicted under pressure).
 Implemented at `internal/ratelimit/sliding_window.go`.
 **Postgres backend (`postgres`)** — same algorithm against the
 `rate_limit_buckets` table:
 ```sql
 CREATE TABLE rate_limit_buckets (
    bucket_key TEXT          PRIMARY KEY,
    timestamps TIMESTAMPTZ[] NOT NULL DEFAULT '{}',
    updated_at TIMESTAMPTZ   NOT NULL DEFAULT NOW()
 );
 ```
 `Allow(key, now)` opens a transaction, ensures the row exists
 (`INSERT ... ON CONFLICT DO NOTHING`), acquires the row lock
 (`SELECT ... FOR UPDATE`), prunes timestamps older than `now-window`,
 compares the post-prune count against `maxN`, conditionally appends
 `now`, persists, and commits. The row lock is what arbitrates across
 replicas: replicas A and B firing simultaneous `Allow("k")` never
 race because Postgres serializes the per-key row update across the
 cluster. Implemented at
 `internal/ratelimit/postgres_sliding_window.go`.
 ### Janitor sweep (postgres backend only)
 The scheduler runs a `rate_limit_buckets` janitor every
 `CERTCTL_RATE_LIMIT_JANITOR_INTERVAL` (default 5m, minimum 1m). The
 sweep deletes rows whose `updated_at` is older than the longest
 configured window any limiter uses (24h today, matching the EST
 per-principal limiter). Idempotent; repeated sweeps find zero rows.
 The memory backend's prune-on-Allow path keeps buckets short-lived
 without a separate sweep, so the loop is a no-op when
 `backend=memory`.
 ### Falsifiable closure proof
 The Phase 13 Sprint 13.2 integration test
 `internal/integration/ratelimit_multi_replica_test.go`
 (`//go:build integration`) fires 100 concurrent `Allow("test-key")`
 calls round-robined across 3 independent `PostgresSlidingWindowLimiter`
 instances sharing one Postgres database (`cap=10`, `window=1m`) and
 asserts exactly 10 succeed + 90 return `ErrRateLimited`. If the
 cross-replica row lock weren't arbitrating, each replica would
 independently let through ~3-4 requests, giving 12-15 successes
 total. Re-run:
 ```
 go test -tags=integration -count=1 -run TestRateLimit_MultiReplica \
    ./internal/integration/...
 ```
 ### Helm chart wiring
 The helm chart at `deploy/helm/certctl/` exposes the backend via
 `server.rateLimiting.backend` (default `memory`). To opt into the
 postgres backend for an HA deploy:
 ```
 helm upgrade --install certctl deploy/helm/certctl \
    --set server.replicas=3 \
    --set server.rateLimiting.backend=postgres \
    --set server.rateLimiting.janitorInterval=5m
 ```
 `server.replicas > 1` without flipping `backend` to `postgres` works
 fine — the limits stay per-process — but the operator gets a 2× /
 3× / Nx effective cap depending on replica count. The chart does NOT
 auto-flip on `replicas > 1` because some HA deploys deliberately want
 per-process limits (sticky-session ingress + tight per-replica caps
 to detect bot traffic at the edge before it hits the application).
 ### Where these numbers live
 The configurable caps are exposed as `CERTCTL_*_PER_MINUTE` /
 `CERTCTL_ACME_*_PER_HOUR` env vars — see the
 [security posture](security.md) doc for the operator-facing
 configuration surface. The hard-coded ones (break-glass 5/min) are
 intentionally non-configurable as a defense-in-depth measure; the
 auth subsystem owns that policy decision.
 ## Performance harness scope
 The load-test harness at [`deploy/test/loadtest/`](../../deploy/test/loadtest/)
 covers the API-tier hot paths (issuance acceptance + cert list). It
 does NOT load-test issuer-connector round-trips (you'd be load-
 testing someone else's API), full multi-RTT ACME enrollment flows,
 bulk-revoke / bulk-renew admin paths, or scheduler concurrency under
 bulk renewal. Each exclusion is justified in
 [`deploy/test/loadtest/README.md`](../../deploy/test/loadtest/README.md)
 under "What it explicitly does NOT measure." If your evaluation
 requires a benchmark on one of those exclusions, the right next step
 is a follow-up scenario in that directory.
 The per-component benchmarks ship in-tree as Go `Benchmark*`
 functions:
 - `internal/auth/session/bench_test.go` — session signing + validation
  steady state and cold-process timing.
 - `internal/auth/oidc/bench_test.go` — OIDC verify steady state.
 - `internal/auth/oidc/bench_keycloak_test.go` — OIDC cold-cache timing
  (gated `//go:build integration`).
 Authoritative benchmark numbers + threshold contracts:
 [`docs/operator/auth-benchmarks.md`](auth-benchmarks.md) (auth
 subsystem) and [`docs/operator/performance-baselines.md`](performance-baselines.md)
 (general API tier).
 ## Related reading
 - [`docs/operator/security.md`](security.md) — the broader hardening
  posture; this document is its observability subset.
 - [`docs/operator/performance-baselines.md`](performance-baselines.md) — operator-runnable benchmarks against the API tier
 - [`docs/operator/auth-benchmarks.md`](auth-benchmarks.md) — session
  + OIDC validation timings + threshold contracts
 - [`deploy/test/loadtest/README.md`](../../deploy/test/loadtest/README.md) — k6 load-test harness scope + threshold contract
 - [`docs/operator/runbooks/postgres-backup.md`](runbooks/postgres-backup.md) — operator-run backup recipe (separate file because it's a procedural runbook, not an observability claim)
@@ -101,6 +101,5 @@ Capture timing in your own loadtest-baselines log so future regressions surface
 ## Related docs
 - [`docs/contributor/ci-pipeline.md`](../contributor/ci-pipeline.md) — CI guard for performance regression
 - [`docs/operator/security.md`](security.md) — rate limit tuning
 - [`docs/reference/architecture.md`](../reference/architecture.md) — request path through handler → service → repository
@@ -0,0 +1,136 @@
 # Privacy & retention (federated-user PII)
 > Last reviewed: 2026-05-16
 Sprint 6 COMP-002-RETENTION closure. certctl stores three categories
 of personally-identifiable information for federated humans (Auth
 Bundle 2 OIDC users):
 | Column | Source | Used by |
 |---|---|---|
 | `users.email` | IdP claim (`email`) | Operator GUI "find user by email", display in lists, audit attribution. |
 | `users.display_name` | IdP claim (`name`) | UI display string for the human. |
 | `users.oidc_subject` | IdP claim (`sub`) | Stable identifier — joined with `oidc_provider_id` in the (provider, subject) UNIQUE constraint. |
 Pre-fix, deactivating a user (admin-side `auth.user.deactivate`)
 soft-deleted the row by setting `deactivated_at` but left the PII
 columns populated indefinitely. The Sprint 6 fix adds an automatic
 purge pipeline.
 ## Retention pipeline shape
 ```
 Day 0   admin → POST /api/v1/auth/users/u-X/deactivate
                ├─ users.deactivated_at = NOW()
                └─ all active sessions for u-X revoked
 Day N   scheduler's userRetentionLoop tick (default cadence 24h)
        └─ UserRetentionService.PurgeDeactivatedUsers
           ├─ SELECT users WHERE deactivated_at < NOW() - retention_window
           ├─ For each row (batch-capped per tick):
           │     UserRetentionService.DeleteUserPII(u.id)
           │     ├─ revoke all active sessions (defense-in-depth)
           │     ├─ email        := "purged@redacted.local"
           │     ├─ display_name := "[purged]"
           │     ├─ oidc_subject := "sha256:" || hex(sha256(original))
           │     └─ audit_events row (action=user.purge_pii, category=auth)
 ```
 `users.id` is **preserved**. Historical `audit_events.actor = u-X`
 rows still resolve to the row (now scrubbed). This is the
 forensic-attribution guarantee — the operator can prove "user u-X
 performed action Y on date Z" even after the PII is gone.
 `oidc_subject` is **hashed**, not nullified, for two reasons:
 1. The `(oidc_provider_id, oidc_subject)` UNIQUE constraint would
   trip if multiple purged users converged on the same NULL.
 2. Re-login under the same IdP subject creates a fresh row (different
   `u-` id) because `GetByOIDCSubject` won't match the hashed token —
   the original subject is unrecoverable from the hash. This is the
   "right-to-be-forgotten" behavior: the same human logging back in
   is functionally a new account.
 ## Operator configuration
 | Env var | Default | Notes |
 |---|---|---|
 | `CERTCTL_USER_RETENTION_INTERVAL` | `24h` | Tick cadence for the scheduler's userRetentionLoop. Zero or negative ignored. |
 | `CERTCTL_USER_RETENTION_WINDOW` | `30 * 24h` (30 days) | How long after `deactivated_at` a row's PII stays in the table. Operators with stricter GDPR/CCPA expectations may shorten. |
 | `CERTCTL_USER_RETENTION_BATCH_CAP` | `200` | Per-tick row budget. Larger backlogs spread across multiple ticks. 0 = unbounded (test fixtures only). |
 ## How to verify retention is working
 1. Deactivate a test user via the admin path:
   ```bash
   curl -X POST -H "X-API-Key: $ADMIN_KEY" \
     https://certctl.example.com/api/v1/auth/users/u-test/deactivate
   ```
 2. Confirm the row's `deactivated_at` is set:
   ```sql
   SELECT id, email, deactivated_at FROM users WHERE id = 'u-test';
   ```
 3. Backdate `deactivated_at` to past the retention window (only for
   testing — never in production):
   ```sql
   UPDATE users SET deactivated_at = NOW() - INTERVAL '60 days'
   WHERE id = 'u-test';
   ```
   (Note: this UPDATE will succeed because `users` doesn't have a
   WORM trigger; the audit-events WORM trigger is unrelated.)
 4. Wait for the next `userRetentionLoop` tick (or restart the server
   to force an immediate sweep). Confirm scrub:
   ```sql
   SELECT id, email, display_name, oidc_subject
     FROM users
    WHERE id = 'u-test';
   ```
   Expected: `email = 'purged@redacted.local'`,
   `display_name = '[purged]'`,
   `oidc_subject LIKE 'sha256:%'`.
 5. Confirm an audit row was emitted:
   ```sql
   SELECT id, actor, action, resource_id, timestamp
     FROM audit_events
    WHERE action = 'user.purge_pii' AND resource_id = 'u-test'
    ORDER BY timestamp DESC LIMIT 1;
   ```
 ## What's NOT covered (deferred work)
 The Sprint 6 fix is Phase 1 of the audit's COMP-002-RETENTION
 recommendation. Two further pieces are forward-looking:
 - **GDPR data-subject access request (DSAR) export.** A "show me
  everything you know about me" endpoint is not yet implemented.
  Operators on EU-resident data should treat this as a manual SQL
  procedure today; track for Phase 2.
 - **Cascade purge of related rows.** Sessions are revoked (above);
  api_keys with `created_by = u-X` are NOT yet purged on scrub. The
  api_keys table doesn't have a foreign key to users (it indexes by
  `actor_id` strings, free-form), so the cascade is a service-layer
  concern that needs explicit wiring. Track for Phase 2.
 - **Per-event PII redaction in `audit_events.details`.** The existing
  `RedactDetailsForAudit` (`internal/service/audit_redact.go`) scrubs
  credential + PII keys at write time. A future feature for
  "retroactively re-redact existing rows" would interact with the WORM
  trigger; out of scope today.
 ## See also
 - `internal/service/user_retention.go` — `UserRetentionService` source.
 - `internal/scheduler/scheduler.go::userRetentionLoop` — scheduler loop.
 - `migrations/000036_users.up.sql` — `users` table definition.
 - `migrations/000045_users_deactivated_at.up.sql` — `deactivated_at` column.
 - `docs/operator/audit-chain.md` — paired Sprint 6 tamper-evidence work.
@@ -68,6 +68,45 @@ giving them the keys to the kingdom. The
 `internal/domain/auth/auditor_test.go` invariants pin this set going
 forward.
 ### Auditor role invariants (DOC-002 / COMP-005 closure)
 Acquisition-audit DOC-002 + COMP-005 closure (Sprint 7 ACQ, 2026-05-16).
 The auditor role's permission set is **pinned at exactly two
 permissions** — `audit.read` and `audit.export` — and any drift breaks
 the SOC 2 / FedRAMP / PCI separation. The pin is enforced at three
 layers and the load-bearing layer is the unit-test set, not a bash CI
 guard:
 1. **Schema layer** — `migrations/000029_rbac.up.sql:261-262` seeds
   exactly two `role_permissions` rows for `r-auditor`
   (`r-auditor / p-audit-read / global / NULL` and
   `r-auditor / p-audit-export / global / NULL`).
   `migrations/000039_audit_crit1_perms.up.sql:111` adds an inline
   comment confirming `r-auditor` was NOT widened by the migration that
   shipped the five admin-only fine-grained perms.
 2. **Code layer** — `internal/domain/auth/DefaultRoles[RoleIDAuditor]`
   matches the schema. A future code change that adds a non-audit
   permission to the slice is caught by:
 3. **Test layer** (the load-bearing one) —
   `internal/domain/auth/auditor_test.go` ships three pinning tests:
   - `TestAuditorRoleHoldsExactlyAuditReadAndExport` — set-equality
     comparison; fails on any add or remove
   - `TestAuditorRoleDoesNotHoldMutatingOrReadingNonAuditPerms` —
     enumerates the slice and rejects any permission outside the
     `{audit.read, audit.export}` set; catches subtle widening even if
     the set-equality test is bypassed
   - `TestAuditorRoleSeparateFromViewer` — pins that the auditor and
     viewer permission sets are disjoint except for `audit.read` (which
     viewer shares by design); catches the "auditor inherits viewer
     reads" leg
 A bash CI guard was deliberately **not** added — the property is
 already enforced at the Go test layer with stronger semantics
 (struct-aware set equality) than `grep` could provide. If a future
 contributor proposes widening `r-auditor`, the three tests above
 fail at `go test ./internal/domain/auth/...` BEFORE the change can
 land in a merge.
 The five **admin-only fine-grained perms** seeded by migration
 000030 gate the high-blast-radius endpoints:
@@ -0,0 +1,165 @@
 # Runbook: forcing config-encryption blob upgrades (v1/v2 → v3)
 > Last reviewed: 2026-05-12
 Use this when:
 - You've rotated `CERTCTL_CONFIG_ENCRYPTION_KEY` and want every row in
  the database to be re-sealed under the new passphrase, not just the
  next ones to be touched.
 - A v1- or v2-era encrypted blob existed in your database before you
  upgraded to a post-M-8 release and you want to retire the legacy
  read path's PBKDF2 work factor (100,000 rounds) in favor of the v3
  factor (600,000 rounds, OWASP 2024).
 - You're preparing for an audit and want every at-rest encrypted blob
  to be on the same wire format.
 Audience: a platform sysadmin who can run SQL against certctl's
 PostgreSQL instance and exercise the GUI/REST API write paths.
 For background on the v3 / v2 / v1 wire formats and the FileDriver vs
 HSM threat model, read
 [`docs/operator/secret-custody.md`](../secret-custody.md) first.
 ---
 ## Background: how the read fallback works
 `internal/crypto/encryption.go::DecryptIfKeySet` reads three on-disk
 formats in this order:
 ```
 v3 (magic 0x03, per-ciphertext 16-byte salt, PBKDF2 600k) →
 v2 (magic 0x02, per-ciphertext 16-byte salt, PBKDF2 100k) →
 v1 (no magic, fixed 28-byte salt, PBKDF2 100k)
 ```
 The fallback is AEAD-driven: if v3 decryption fails authentication, the
 function tries v2; if v2 fails, v1. This is what keeps pre-M-8 v1 blobs
 readable without an explicit migration.
 `EncryptIfKeySet` always writes v3. As a result, any row that is
 **re-written** through the normal application code path is silently
 upgraded to v3 the moment it's persisted.
 The implication: you do not need to "migrate" v1/v2 blobs for them to
 keep working — only if you want the v1/v2 wire format physically gone
 from your database.
 ## Procedure
 ### Step 1 — confirm the encryption key is set
 Re-encryption obviously cannot run without a passphrase. Verify:
 ```bash
 echo "${CERTCTL_CONFIG_ENCRYPTION_KEY:-NOT SET}" | sed -E 's/./*/g'
 ```
 If the variable prints `NOT SET`, do not proceed — set the key in your
 deployment manifest and restart the control plane first.
 ### Step 2 — identify which tables hold encrypted blobs
 Encrypted columns in the v2.1.0 schema:
 | Table              | Column                | Notes                                                                |
 |---|---|---|
 | `issuers`          | `encrypted_config`    | Only populated for `source='database'` rows (env-seeded rows are not encrypted) |
 | `targets`          | `encrypted_config`    | Same source-based gating as issuers                                  |
 | `oidc_providers`   | `client_secret_enc`   | OIDC client_secret                                                   |
 | `auth_session_signing_keys` | `key_material_enc` | HMAC-SHA256 session-cookie signing key                          |
 If your schema differs, derive the column list from the migration
 folder:
 ```bash
 grep -hE '_enc[ ,]|encrypted_config' migrations/*.up.sql | sort -u
 ```
 ### Step 3 — identify rows still on v1/v2
 The magic byte of the blob distinguishes versions; v1 blobs start with
 the random AES-GCM nonce (anything but `0x02` or `0x03` is definitely
 v1), and v2 vs v3 is determined by the first byte:
 ```sql
 -- Per-table version distribution (run against your live database)
 SELECT
    SUBSTRING(encrypted_config FROM 1 FOR 1)::bytea AS magic,
    COUNT(*) AS rows
  FROM issuers
  WHERE encrypted_config IS NOT NULL
  GROUP BY magic;
 ```
 Expected steady-state output is a single row with `magic = \x03`.
 Any rows with `\x02` are v2; any rows with anything else are v1.
 ### Step 4 — force re-sealing
 `UPDATE` the rows back to themselves through the normal application
 write path. The cleanest way to do this is via the REST API or GUI,
 not raw SQL — re-issuing the same `PUT /api/v1/issuers/:id` reads the
 row, decrypts, then re-encrypts under v3 on the write back.
 For an issuer named `iss-letsencrypt-prod`:
 ```bash
 # Fetch then re-PUT the same body (CSRF + bearer token elided).
 curl -sS https://certctl.example.com/api/v1/issuers/iss-letsencrypt-prod \
  -H "Authorization: Bearer $CERTCTL_API_KEY" \
  | jq '.' \
  | curl -sS -X PUT https://certctl.example.com/api/v1/issuers/iss-letsencrypt-prod \
      -H "Authorization: Bearer $CERTCTL_API_KEY" \
      -H "Content-Type: application/json" \
      --data-binary @-
 ```
 Repeat for each row that the Step 3 query flagged as non-v3.
 ### Step 5 — verify
 Re-run the Step 3 query. The output should now show only `magic =
 \x03` rows.
 ## Special case: rotating the encryption-key passphrase
 If your goal is to retire a possibly-compromised passphrase rather
 than retire a legacy wire format, the order is:
 1. Generate a new passphrase. Document it via your secret-management
   tool (HashiCorp Vault, AWS Secrets Manager, etc.).
 2. Stop the control plane briefly so no rows are written under the
   stale passphrase during the transition window.
 3. Run a one-shot decrypt-with-old / re-encrypt-with-new pass.
   certctl ships no built-in tool for this — see the open
   roadmap item below. The cleanest current approach is:
    - Start certctl with the OLD passphrase.
    - Read every encrypted column out to a JSON dump via the REST API.
    - Stop certctl. Update its env to the NEW passphrase. Restart.
    - PUT every row back from the JSON dump (the writes re-seal under
      the new passphrase).
 4. Document the old passphrase as retired in your secret-management
   tool. Anyone with read access to a pre-rotation backup still needs
   it to decrypt that backup; the live database no longer needs it.
 For most operators, simply rotating the passphrase and letting the
 re-seal happen organically as rows are touched is acceptable — the
 v3 wire format with PBKDF2 600k rounds makes offline brute-force
 against the old passphrase computationally expensive.
 ## Open roadmap items
 - Ship a built-in `certctl admin reseal --all` command that does Steps
  3 and 4 in one shot, with structured progress + audit logging.
  Tracked in [WORKSPACE-ROADMAP.md](../../WORKSPACE-ROADMAP.md).
 - Surface per-table v1/v2/v3 distribution as a Prometheus gauge so
  alerting can fire on "rows on legacy format" drift.
 ## Related reading
 - [`docs/operator/secret-custody.md`](../secret-custody.md) — the
  broader where-do-private-keys-live reference; this runbook is the
  procedural arm of that document.
 - [`internal/crypto/encryption.go`](../../../internal/crypto/encryption.go)
  package comment — wire format authoritative reference.
@@ -0,0 +1,105 @@
 # Runbook: regenerating Playwright visual-regression snapshots
 > Last reviewed: 2026-05-16
 Use this when:
 - You've intentionally changed UI shape (added a column, restyled a
  banner, replaced an icon set) and the next `Frontend E2E` CI run
  fails with `Screenshot comparison failed:` errors on multiple
  `04-visual-regression.spec.ts` cases.
 - A deterministic-but-platform-specific font-rendering difference
  emerges (Linux runner vs your Mac dev box) and you want to refresh
  baselines from the canonical CI environment.
 TEST-003 closure (Sprint 5, 2026-05-16) flipped the workflow from
 `continue-on-error: true` to `false`. Pre-fix you could ignore a
 red E2E run and ship anyway. Post-fix the run blocks the merge, so
 any change that legitimately moves pixels needs the snapshot bump
 captured here.
 Do NOT use this to make a real visual regression disappear. The
 snapshots are version-controlled evidence — if a pixel diff fires
 unexpectedly, investigate the rendering change before bumping.
 ## What "snapshots" means here
 `web/playwright/04-visual-regression.spec.ts` calls
 `toHaveScreenshot()`. Playwright stores the canonical PNG at
 `web/playwright/04-visual-regression.spec.ts-snapshots/<test-name>-<browser>-<platform>.png`
 on first run. Subsequent runs compare pixel-by-pixel against that
 file. We commit the PNGs to git so the CI runner and local dev
 share a single source of truth.
 Two failure modes the diff is designed to catch:
 - **Intentional UI change.** You added a new field to the Targets
  table. The screenshot now has an extra column. The baseline
  doesn't. Pixel diff fires — this is the "operator updates
  baselines" path documented below.
 - **Regression.** A CSS change inadvertently shifted spacing.
  Investigate before regenerating; don't paper over the diff.
 ## Standard bump (one or two affected tests)
 1. Run the E2E suite locally with the update flag against the
   same Linux runner image Playwright uses:
   ```bash
   cd web
   npx playwright test 04-visual-regression.spec.ts --update-snapshots
   ```
   If you're on macOS, run it through Docker against the same image
   the workflow uses (`mcr.microsoft.com/playwright`); font
   rendering differs between platforms and Linux baselines must
   come from a Linux source.
 2. Inspect every regenerated PNG:
   ```bash
   git status web/playwright/*.spec.ts-snapshots/
   git diff --stat web/playwright/*.spec.ts-snapshots/
   ```
   PNG diffs in `git diff` are unhelpful — open the files in any
   image viewer and confirm the change matches your intent.
 3. Commit the snapshots alongside the source change in the same
   PR:
   ```bash
   git add web/playwright/*.spec.ts-snapshots/
   git commit -m "chore(e2e): refresh visual snapshots after <change>"
   ```
 4. Push and confirm CI's E2E job greens out.
 ## Mass bump (font upgrade, framework migration)
 Use the workflow's `workflow_dispatch` input to regenerate from
 CI's canonical environment:
 1. Go to `Actions` → `Frontend E2E` → `Run workflow`.
 2. Set `update_snapshots: true`.
 3. The workflow runs Playwright with `--update-snapshots`, then
   commits + pushes the regenerated PNGs to a feature branch
   `playwright/snapshot-update-<run-id>`.
 4. Open a PR from that branch to master. Review the PNG diffs in
   the PR view (GitHub renders image diffs side-by-side for
   committed PNGs).
 5. Merge.
 ## What NOT to do
 - Don't regenerate snapshots from a developer's local machine and
  push them as the canonical baseline. The Linux runner's font
  hinting differs from macOS / Windows, so the baselines must come
  from the same image the CI workflow runs.
 - Don't add `--update-snapshots` to the always-run e2e step in
  `.github/workflows/e2e.yml`. That's how snapshot regressions
  become invisible — every diff gets accepted, every PR ships
  fine, and the visual-regression layer becomes decorative.
 - Don't bump snapshots in a "fix typo" PR. Every PNG change is
  an architectural decision; pair it with the source change that
  justifies it.
@@ -0,0 +1,113 @@
 # High-Availability Deployment Runbook
 > Last reviewed: 2026-05-13
 <!-- Phase 2 DEPL-H1 closure -->
 certctl's Helm chart ships with conservative single-replica defaults
 that produce a working `helm install` against any Kubernetes cluster.
 Production HA is operator-opt-in across three values surfaces — none
 of which the chart flips on your behalf.
 This runbook documents the three changes, why they default off, and
 the smallest-possible HA values overlay.
 ---
 ## Why HA is opt-in (not default)
 Three load-bearing reasons the chart defaults are `replicas: 1` and
 `podDisruptionBudget.enabled: false`:
 1. **A 1-replica deployment works on every cluster.** A multi-replica
   default with `minAvailable: 2` would render a PDB at install time;
   if the cluster has fewer than 2 nodes available (single-node
   `kind` / `minikube` / fresh `k3s` clusters), Helm renders fine but
   the first `kubectl rollout` blocks indefinitely waiting for the
   second replica that can never schedule. Defaulting off keeps the
   demo path one-command.
 2. **Postgres is a singleton in the bundled chart.** The chart's
   `postgres-statefulset.yaml` runs ONE Postgres pod. Scaling the
   server tier past 1 replica without an externalized Postgres + a
   pgbouncer-style proxy doesn't actually buy HA at the DB tier — the
   single Postgres pod is the failure domain. Operators who want true
   HA route Postgres to a managed service (RDS, Cloud SQL, AlloyDB,
   AKS-managed-Postgres, Aiven) or run their own cluster (Patroni,
   CloudNativePG, Zalando postgres-operator). See the
   [external-Postgres values example](../../deploy/helm/examples/values-external-db.yaml).
 3. **Session affinity is HTTPS-only.** The control plane is HTTPS-only
   (TLS 1.3 pinned). Adding `sessionAffinity: ClientIP` to the
   server Service mid-deployment when a sticky front-end LB is in
   play (NGINX Ingress, Cloud LB with backend service) is the right
   default for OIDC + RBAC session cookies. But operators who terminate
   TLS at a different layer (Envoy mesh, Cloudflare in front of the
   cluster) may have already solved affinity upstream — flipping it
   on by default would over-constrain those paths.
 ## The smallest production-HA overlay
 Three Helm values to flip:
 ```yaml
 # values-ha.yaml — copy into your overlay and edit to taste.
 server:
  # ≥ 2 replicas is the minimum for the PDB to render. 3 gives you
  # a true rolling-restart tolerance window (1 down for upgrade,
  # 2 still serving) without dropping below minAvailable.
  replicas: 3
  service:
    # Required when the front-end LB doesn't already enforce
    # session affinity. OIDC + RBAC session cookies need to land
    # on the same backend pod for the session lifetime.
    sessionAffinity: ClientIP
 podDisruptionBudget:
  # Renders the PDB template; controller-side voluntary disruptions
  # (node-drain for k8s upgrade, cluster-autoscaler scale-down)
  # respect this floor.
  enabled: true
  # With server.replicas: 3, minAvailable: 2 leaves headroom for one
  # rolling restart at a time.
  minAvailable: 2
  # maxUnavailable is mutually exclusive with minAvailable; pick one.
  # maxUnavailable: 1
 ```
 Apply with:
 ```bash
 helm upgrade certctl deploy/helm/certctl/ -f values-ha.yaml
 ```
 ## What you still own as the operator
 Three things the chart does not solve, even at `replicas: 3`:
 1. **Postgres HA.** Route to an externalized Postgres (managed cloud
   or operator-managed cluster). The chart's bundled StatefulSet
   pod is a development/single-AZ pattern, not a production HA path.
 2. **TLS material lifecycle.** The chart accepts an `existingSecret`
   for the server cert; rotating it is operator-side automation.
   The dashboard + agent can issue their own certs via the local CA
   (eat-your-own-dogfood); the operator can wire `cert-manager` if
   they prefer that path.
 3. **Backup CronJob.** Phase 4 of the architecture diligence
   remediation plan (DEPL-H2) ships a `backup-cronjob.yaml` template;
   until that lands, backups are operator-run per the existing
   `docs/operator/runbooks/postgres-backup.md` runbook.
 ## Cross-references
 - `deploy/helm/certctl/values.yaml` lines 19, 446, 566 — the three
  defaults this runbook documents.
 - `docs/operator/runbooks/postgres-backup.md` — Postgres backup
  runbook (today, operator-run).
 - `docs/operator/runbooks/disaster-recovery.md` — DR procedure.
 - Phase 4 (Helm Chart, DR, And Ops Surface) of the architecture
  diligence remediation plan tracks the chart-level work
  (backup CronJob, PrometheusRule starter, migration hook, etc.).
@@ -0,0 +1,243 @@
 # Runbook: PostgreSQL backup for certctl
 > Last reviewed: 2026-05-16
 Use this when:
 - You're setting up a new certctl deployment and need a backup policy
  before going to production.
 - A buyer or auditor asks "where's the backup automation?" and you need
  to point at the recommended cadence + procedure.
 - You're rotating the encryption key, swapping CAs, or doing any other
  destructive maintenance and want a snapshot to roll back to.
 certctl does not ship a built-in backup daemon. Postgres is the system
 of record for every piece of certctl state that isn't on the
 operator's filesystem (CA keys, OCSP responder keys, SCEP/EST trust
 bundles — see "Operator-managed (NOT in DB)" in the
 [disaster-recovery runbook](disaster-recovery.md#postgres-restore));
 backing it up is treated as a standard PostgreSQL operations task
 that the operator owns end-to-end with their existing tooling.
 This page is the recommended recipe.
 ## What to back up
 | Layer                              | Tool                                                                    | Cadence                  |
 |---|---|---|
 | `certctl` database (the row data)  | `pg_dump` (logical) **or** `pg_basebackup` + WAL archive (physical PIT) | ≥ daily, retention ≥ 30d |
 | CA cert + key (`CERTCTL_CA_CERT_PATH`, `CERTCTL_CA_KEY_PATH`) | Out-of-band file backup (operator's existing secret-management tool) | On change |
 | SCEP RA cert + key (per profile)   | Out-of-band file backup                                                 | On change                |
 | OCSP responder keys                | Out-of-band file backup (`CERTCTL_OCSP_RESPONDER_KEY_DIR`)              | On change                |
 | Trust-anchor PEM bundles           | Out-of-band file backup                                                 | On change                |
 | Env vars (auth secret, etc.)       | Operator's secret-management tool (Vault, AWS Secrets Manager, etc.)    | On rotation              |
 A backup of only the Postgres database without the operator-managed
 file material is **not a complete restore artifact** — see the
 [disaster-recovery runbook's Postgres-restore section](disaster-recovery.md#postgres-restore)
 for the full inventory. The DR runbook owns the restore procedure;
 this page owns the capture procedure.
 ## Logical backup (recommended for most deployments)
 `pg_dump -Fc` produces a portable compressed dump that's easy to
 restore into a fresh Postgres instance at any version ≥ the dump's
 source version. Best for deployments where the DB is small enough
 that a full logical dump fits the backup window (rough rule of thumb:
 under a million `managed_certificates` rows + corresponding history).
 ### docker-compose
 ```bash
 # 1. Snapshot. Run from any host that can reach the postgres container.
 TIMESTAMP=$(date -u +%Y%m%dT%H%M%SZ)
 docker compose -f deploy/docker-compose.yml exec -T postgres \
  pg_dump --format=custom --no-owner --no-acl --dbname=certctl \
  > "certctl-${TIMESTAMP}.dump"
 # 2. Verify integrity (catch transport / truncation bugs early).
 docker run --rm -v "$PWD:/dumps" -w /dumps postgres:16-alpine \
  pg_restore --list "certctl-${TIMESTAMP}.dump" > /dev/null \
  && echo "OK: pg_restore --list parses the dump cleanly" \
  || { echo "CORRUPT DUMP"; exit 1; }
 # 3. Move to durable storage (S3, GCS, NFS, encrypted-at-rest blob
 # storage of your choice). DO NOT leave the dump on the certctl host
 # alone — that defeats the purpose of having a backup.
 aws s3 cp "certctl-${TIMESTAMP}.dump" "s3://your-bucket/certctl/"
 ```
 ### Kubernetes (with the bundled Helm chart)
 ```bash
 # 1. Snapshot via kubectl exec into the postgres StatefulSet pod.
 TIMESTAMP=$(date -u +%Y%m%dT%H%M%SZ)
 NAMESPACE=certctl
 kubectl exec -n "$NAMESPACE" statefulset/postgres -- \
  pg_dump --format=custom --no-owner --no-acl --dbname=certctl \
  > "certctl-${TIMESTAMP}.dump"
 # 2. Same verification step as above.
 # 3. Same off-host storage step as above.
 ```
 ### Restore (cross-reference)
 The restore procedure lives in
 [disaster-recovery.md § Postgres restore](disaster-recovery.md#postgres-restore).
 The key reminders: stop certctl first, restore the DB, run any
 migrations newer than the snapshot, truncate the CRL + OCSP caches,
 then restart.
 ## Physical / PITR backup (large fleets, RPO < 1h)
 Logical dumps have a coarse RPO (the last successful dump). For
 deployments where ≤ 1h of cert-issuance history loss is unacceptable,
 pair Postgres physical backups with continuous WAL archiving:
 - `pg_basebackup` for the initial seed
 - `archive_command = '<your-WAL-archiver>'` in `postgresql.conf` to
  ship every WAL segment off the host as it closes
 - `pgbackrest` or `wal-g` for the operational layer (both are
  battle-tested, support encryption, and integrate cleanly with S3 /
  GCS / Azure Blob)
 certctl ships nothing in this layer — it's standard PostgreSQL DBA
 work, and shipping a bespoke recipe would just be a worse version of
 what `pgbackrest` already does. The
 [pgbackrest configuration guide](https://pgbackrest.org/configuration.html)
 is the authoritative reference.
 ## Automation paths
 certctl ships an **opt-in Helm CronJob** for the in-cluster-Postgres
 case (the most common bundled-deploy shape). The template lives at
 `deploy/helm/certctl/templates/backup-cronjob.yaml` and is gated by
 `backup.enabled` in `values.yaml`. Default OFF; flip it on with one
 toggle and a sink choice. For managed Postgres (AWS RDS / GCP Cloud
 SQL / Azure DB) the operator relies on the provider's PITR layer;
 this CronJob is intentionally scoped to the in-cluster-Postgres path.
 ### Enabling the bundled CronJob
 ```bash
 # PVC sink (in-cluster persistent volume — simplest)
 helm upgrade --install certctl charts/certctl \
  --set backup.enabled=true \
  --set backup.sink=pvc \
  --set backup.pvc.storageClassName=<your-storage-class> \
  --set backup.pvc.size=20Gi \
  --set backup.schedule="0 2 * * *"
 # S3 sink (off-cluster, recommended for any deploy past the lab)
 kubectl create secret generic certctl-backup-aws \
  --from-literal=AWS_ACCESS_KEY_ID=AKIA... \
  --from-literal=AWS_SECRET_ACCESS_KEY=... \
  --namespace certctl
 helm upgrade --install certctl charts/certctl \
  --set backup.enabled=true \
  --set backup.sink=s3 \
  --set backup.s3.bucket=my-certctl-backups \
  --set backup.s3.region=us-east-1 \
  --set backup.s3.credentialsSecret=certctl-backup-aws \
  --set backup.schedule="0 2 * * *"
 ```
 The CronJob runs `pg_dump --format=custom --no-owner --no-acl
 --dbname=certctl` (the same shape as the manual command earlier in
 this runbook, so a manual dump and a Job dump are byte-comparable)
 and ships the artifact to the configured sink. Off-host retention
 is the sink's responsibility — S3 lifecycle rules or PVC snapshot
 retention on the storage class, not the CronJob.
 ### When the bundled CronJob is NOT the answer
 - **Managed Postgres (AWS RDS / GCP Cloud SQL / Azure DB).** Use the
  provider's built-in PITR; configure retention ≥ 30 days. The
  certctl deployment surface is the connection string alone — no
  CronJob to run.
 - **Self-hosted Postgres on a VM (no Kubernetes).** Use a systemd
  timer + `pg_dump` + `restic` (or `borgbackup`) to encrypted
  off-host storage. The bundled CronJob has no equivalent on bare
  VMs.
 - **Already running pgbackrest / wal-g.** Keep using it. The bundled
  CronJob is for the operator who doesn't yet have a backup posture,
  not a replacement for production-grade WAL-shipping.
 ### Recovery objectives
 The bundled CronJob targets the same RPO/RTO that any nightly-dump
 strategy gives you:
 - **RPO ≈ 24h** at the default `0 2 * * *` schedule (you lose at
  most one day of writes if Postgres burns down). Tighten by running
  every 6h or 1h; tighten further by switching to WAL-shipping
  (out of scope for the bundled CronJob).
 - **RTO ≈ 30–60min** for the restore drill below — drop the dump
  into a fresh Postgres instance, point certctl at it, confirm
  routes return 200. Empirically measured during the
  [disaster-recovery runbook](disaster-recovery.md) drill.
 If your contractual RPO is below 24h, run pgbackrest WAL-shipping
 alongside (or instead of) the CronJob.
 ## Verification — what to dry-run quarterly
 A backup you've never restored is a backup you don't have. Add this
 to your quarterly on-call rotation:
 1. Pick the most recent dump from the previous quarter.
 2. Stand up a throwaway Postgres instance (Docker, kind, anything).
 3. `pg_restore -d certctl <the dump>`.
 4. Bring up a certctl-server container pointed at the throwaway DB
   (`CERTCTL_DATABASE_URL=postgres://certctl:...@throwaway/...`).
 5. Confirm `/api/v1/version` returns 200, `/api/v1/certificates`
   lists the expected rows, and the scheduler logs show no
   migration-version mismatch.
 6. Tear down. Note the timing in your DR registry.
 The [disaster-recovery runbook](disaster-recovery.md) covers what to
 do when this dry-run reveals a gap.
 ## CI restore verification
 > Acquisition-audit DEPL-005 + DATA-012 closure (Sprint 4 ACQ,
 > 2026-05-16). The quarterly dry-run above is the operator-side
 > proof; the workflow below is the upstream-side proof.
 The certctl repo ships a weekly GitHub Actions workflow that
 exercises the **exact** pg_dump shape this runbook recommends
 (`--format=custom --no-owner --no-acl`) against a real Postgres
 container, then asserts the audit_events hash chain round-trips
 byte-for-byte across the dump → restore boundary. A regression in
 the dump format, in a Postgres minor bump, or in migration 000047's
 canonical-payload serialization would surface in the next Monday
 run instead of on a customer's restore day.
 - **Workflow:** [`.github/workflows/backup-restore.yml`](../../../.github/workflows/backup-restore.yml)
  — Mondays 07:00 UTC + `workflow_dispatch`. Postgres service
  container pinned to the same SHA256 digest as
  `deploy/docker-compose.yml`.
 - **Harness:** [`deploy/test/backup-restore-smoke.sh`](../../../deploy/test/backup-restore-smoke.sh)
  — runs the workload → `pg_dump -Fc` → `DROP SCHEMA public CASCADE`
  → `pg_restore` → verify cycle. Locally runnable against any
  reachable Postgres (it DROPs the schema, so do not point it at
  data you care about).
 - **Workload + verifier:** [`deploy/test/backupsmoke/main.go`](../../../deploy/test/backupsmoke/main.go)
  — generates 24 synthetic `audit_events` rows representing an
  issue/renew/revoke/auth-login cycle, snapshots the chain head
  before the backup, and after restore runs
  `audit_events_verify_chain()` to confirm `first_break_id IS NULL`.
 The CI workflow is not a replacement for the quarterly operator
 dry-run — it does not exercise the operator-managed file material
 (CA keys, RA keys, trust anchors) listed in the "What to back up"
 table above. Treat it as the dump-shape regression test; the
 quarterly run remains the full-restore correctness test.
 ## Related reading
 - [`docs/operator/runbooks/disaster-recovery.md`](disaster-recovery.md) — the restore companion
 - [`docs/operator/secret-custody.md`](../secret-custody.md) — what
  the operator-managed file material (CA keys, RA keys, trust
  anchors) contains, why it lives outside the DB, and what it costs
  to lose
@@ -0,0 +1,243 @@
 # Runbook: Prometheus bearer token for the metrics scrape endpoint
 > Last reviewed: 2026-05-14
 Use this when:
 - You're enabling Prometheus Operator scraping via the Helm chart's
  `monitoring.serviceMonitor.enabled` toggle.
 - Your Prometheus scrapes are returning 401 against
  `/api/v1/metrics/prometheus`.
 - An auditor asks "how is the metrics endpoint authenticated?"
 ## The constraint
 The certctl server exposes Prometheus metrics at
 `/api/v1/metrics/prometheus`. This endpoint is **RBAC-gated on the
 `metrics.read` permission** (per `internal/api/router/router.go`).
 Like every other gated handler, it requires an authenticated actor
 holding that permission — there is no anonymous-scrape path.
 The rationale: the metrics payload includes operational counters
 (cert counts by status, agent counts, issuance failure rates) that
 a public-facing observer should not see. Most certctl deployments
 expose a reverse proxy / load balancer to the wider network; the
 auth gate on `/api/v1/metrics/prometheus` prevents an external
 observer from learning operational state via the metrics endpoint
 even when the proxy itself is reachable.
 ## What you need to set up
 Three pieces:
 1. **An API key with `metrics.read` permission** (and only that
   permission — least-privilege).
 2. **A Kubernetes Secret** holding that API key.
 3. **`monitoring.serviceMonitor.bearerTokenSecret`** in the chart's
   values pointing at the Secret.
 ## Step 1: Create the metrics-read role + API key
 The chart's seed migration ships a `metrics-read` role-template, but
 some operators want a dedicated identity per scrape source. Both
 approaches work; the dedicated-identity path is below.
 ```bash
 # 1. Bootstrap or impersonate a session with auth.role.assign +
 #    auth.apikey.create permissions (admin actor is fine).
 # 2. Create a role with only metrics.read.
 curl -sS --cacert ./ca.crt -X POST \
  -H "Authorization: Bearer ${ADMIN_API_KEY}" \
  -H "Content-Type: application/json" \
  https://certctl.your-org.example/api/v1/auth/roles \
  -d '{"id":"r-prometheus-scrape","name":"Prometheus scrape","permissions":["metrics.read"]}'
 # 3. Create an actor that holds the role.
 curl -sS --cacert ./ca.crt -X POST \
  -H "Authorization: Bearer ${ADMIN_API_KEY}" \
  -H "Content-Type: application/json" \
  https://certctl.your-org.example/api/v1/auth/actors \
  -d '{"id":"actor-prometheus","name":"Prometheus scrape","roles":["r-prometheus-scrape"]}'
 # 4. Mint an API key for the actor. The response includes a
 #    `key_value` field that's only returned ONCE — capture it.
 curl -sS --cacert ./ca.crt -X POST \
  -H "Authorization: Bearer ${ADMIN_API_KEY}" \
  -H "Content-Type: application/json" \
  https://certctl.your-org.example/api/v1/auth/apikeys \
  -d '{"actor_id":"actor-prometheus","name":"prometheus-scrape-token"}' \
  | tee /tmp/prom-key.json
 # Extract just the secret material:
 jq -r '.key_value' /tmp/prom-key.json
 ```
 The mint endpoint returns the API key plaintext exactly once. The
 server stores only a constant-time-comparable hash; if you lose the
 key value, mint a new one.
 ## Step 2: Create the Kubernetes Secret
 ```bash
 NAMESPACE=certctl
 API_KEY=$(jq -r '.key_value' /tmp/prom-key.json)
 kubectl create secret generic certctl-prometheus-key \
  -n "$NAMESPACE" \
  --from-literal=api-key="$API_KEY"
 ```
 Now scrub the temporary file:
 ```bash
 shred -u /tmp/prom-key.json
 ```
 ## Step 3: Wire the Secret into the chart values
 In your `values.yaml` (or `--set` overrides):
 ```yaml
 monitoring:
  enabled: true
  serviceMonitor:
    enabled: true
    interval: 30s
    scrapeTimeout: 10s
    bearerTokenSecret:
      name: certctl-prometheus-key
      key: api-key
 ```
 Re-apply the chart:
 ```bash
 helm upgrade certctl . -n "$NAMESPACE" --reuse-values
 ```
 The rendered ServiceMonitor will now include the `bearerTokenSecret`
 block. Prometheus Operator's reconciler picks it up and injects the
 bearer token into the scrape request.
 ## Verification
 ```bash
 # 1. Confirm the ServiceMonitor renders with the secret reference
 kubectl get servicemonitor -n "$NAMESPACE" certctl-server -o yaml \
  | grep -A2 bearerTokenSecret
 # Expected:
 #       bearerTokenSecret:
 #         name: certctl-prometheus-key
 #         key: api-key
 # 2. Tail the certctl-server logs for the next ~60 seconds (one
 #    Prometheus scrape interval). Look for incoming GET /metrics/prometheus
 #    requests authenticated successfully — no 401s.
 kubectl logs -n "$NAMESPACE" -l app.kubernetes.io/component=server \
  --tail=100 -f | grep -E "GET /api/v1/metrics/prometheus|metrics-scrape"
 # 3. From the Prometheus UI's "Targets" page, the certctl-server
 #    target should be UP and last-scrape-error empty. If it's
 #    showing 401, the bearer token isn't reaching the request — see
 #    troubleshooting below.
 ```
 ## Troubleshooting
 ### Prometheus target shows 401
 Three possible causes:
 1. **Wrong Secret name / key.** Run
   `kubectl get secret -n "$NAMESPACE" certctl-prometheus-key -o yaml`
   and confirm the `data.api-key` field exists with a base64-encoded
   non-empty value. The Secret's data field name must match the
   `bearerTokenSecret.key` value in `monitoring.serviceMonitor`.
 2. **API key doesn't have `metrics.read`.** Hit the gating endpoint
   manually from inside the cluster with the same key:
   ```bash
   kubectl run --rm -it --image=curlimages/curl debug -- \
     curl -sS -H "Authorization: Bearer <API_KEY>" \
     https://certctl-server.certctl.svc.cluster.local:8443/api/v1/metrics/prometheus
   ```
   A 401 here means the role doesn't include `metrics.read`. A 403
   means the role exists but the API key isn't assigned to it.
 3. **TLS verification failure (not a 401, but masquerading as one in
   Prometheus's logs).** The default ServiceMonitor template sets
   `insecureSkipVerify: true` to support demos — production deploys
   should set `tlsConfig.caFile` or `tlsConfig.ca.secret` per the
   ServiceMonitor docs.
 ### Prometheus target shows TLS errors
 `monitoring.serviceMonitor.tlsConfig` overrides the default. Three
 patterns:
 ```yaml
 # Pattern 1: trust the system CA bundle (production behind a real CA)
 tlsConfig:
  caFile: /etc/ssl/certs/ca-certificates.crt
  serverName: certctl.your-org.example
 # Pattern 2: trust a CA from a Secret mounted by Prometheus Operator
 tlsConfig:
  ca:
    secret:
      name: certctl-ca
      key: ca.crt
  serverName: certctl.your-org.example
 # Pattern 3: skip verification (DEMO ONLY — DO NOT USE IN PRODUCTION)
 tlsConfig:
  insecureSkipVerify: true
 ```
 The certctl server's self-signed bootstrap cert (default
 `server.tls.existingSecret` from the chart) presents a CN of
 `certctl-server`. If your `serverName` doesn't match, the scrape
 fails with `x509: certificate is valid for certctl-server, not ...`.
 ## Rotation
 API keys are constant-time-compared, stored hashed, and never
 logged. Rotation:
 ```bash
 # 1. Mint a new key (same actor + role)
 curl -sS --cacert ./ca.crt -X POST \
  -H "Authorization: Bearer ${ADMIN_API_KEY}" \
  -H "Content-Type: application/json" \
  https://certctl.your-org.example/api/v1/auth/apikeys \
  -d '{"actor_id":"actor-prometheus","name":"prometheus-scrape-token-v2"}' \
  | tee /tmp/prom-key-new.json
 # 2. Update the Secret in place
 kubectl create secret generic certctl-prometheus-key \
  -n certctl \
  --from-literal=api-key="$(jq -r '.key_value' /tmp/prom-key-new.json)" \
  --dry-run=client -o yaml | kubectl apply -f -
 # 3. Wait one scrape interval; verify the next scrape uses the new key.
 # 4. Revoke the old key
 curl -sS --cacert ./ca.crt -X DELETE \
  -H "Authorization: Bearer ${ADMIN_API_KEY}" \
  https://certctl.your-org.example/api/v1/auth/apikeys/<OLD_KEY_ID>
 # 5. Scrub the temp file
 shred -u /tmp/prom-key-new.json
 ```
 Prometheus Operator picks up Secret changes automatically — no
 ServiceMonitor edit needed, no Prometheus restart.
 ## Related reading
 - [`docs/operator/rbac.md`](../rbac.md) — the full RBAC primitive,
  permission catalogue, and role-assignment workflow.
 - [`docs/operator/security.md`](../security.md) — the broader auth
  posture including the API key / OIDC / break-glass paths.
 - [`docs/operator/auth-threat-model.md`](../auth-threat-model.md) —
  why `/api/v1/metrics/prometheus` is gated, and what an
  unauthenticated leak of metrics data would reveal.
@@ -0,0 +1,193 @@
 # Runbook: Helm rollback for certctl
 > Last reviewed: 2026-05-14
 Use this when:
 - A `helm upgrade` rolled out a bad release and the operator wants to
  return to the previous working state.
 - A schema migration shipped a change the operator wants to back out.
 - An emergency change needs reverting and forward-fix isn't yet
  available.
 This page covers `helm rollback` mechanics + the cases where
 rollback is NOT enough on its own (schema migrations are the main
 one).
 ## What `helm rollback` does
 `helm rollback <release> [revision]` re-applies the manifests from a
 previous Helm revision. It re-creates / updates Kubernetes objects to
 match that revision's template output and is safe for:
 - **Deployment image bumps:** rolls the container image back to the
  previous tag. Pods restart with the old image.
 - **ConfigMap / Secret content changes:** old values land in the
  config; pods that consume them via `envFrom` or volume mounts get
  the prior values on the next restart.
 - **Resource requests / limits / replica count:** the spec changes
  back to the prior values. Kubernetes reschedules pods accordingly.
 - **Service / Ingress / NetworkPolicy changes:** networking flips
  back to the previous shape immediately.
 ## What `helm rollback` does NOT do
 The Kubernetes layer is reversible; the **database schema is not**.
 This is the single most common gap in a rollback plan.
 ### Schema migrations are forward-only by design
 certctl's migrations under `migrations/` are numbered up-migrations
 (`NNNNNN_*.up.sql`) with paired down-migrations
 (`NNNNNN_*.down.sql`) shipped alongside. The `postgres.RunMigrations`
 path applied at server boot only runs the `*.up.sql` files. The
 `*.down.sql` files exist for development reference + a hypothetical
 "surgical revert" path but are **not invoked by `helm rollback`**.
 The implication: if `v2.1.0 → v2.2.0` ships migrations 000100,
 000101, 000102 (adding columns, changing constraints, dropping
 indexes), then `helm rollback` to v2.1.0 takes you back to the v2.1.0
 container image — but the database still has migrations 000100-102
 applied. The v2.1.0 server code doesn't know about those columns; it
 either ignores them (best case) or fails to start (if the schema
 diverged in a way the older code can't tolerate).
 ### When is rollback safe without a schema revert?
 Migrations are **additive-only** in 90%+ of cases. The categories:
 | Migration class | Safe to roll back without schema revert? | Why |
 |---|---|---|
 | Add column with default | Yes | Old code ignores the new column |
 | Add table | Yes | Old code doesn't reference the table |
 | Add index | Yes | Old code doesn't depend on the index existing |
 | Add CHECK / FOREIGN KEY constraint | Usually yes | Only fails on row data inserted by new code that violates the old code's constraints |
 | Rename column / table | NO | Old code's queries reference the original name |
 | Drop column / table | NO (data loss) | New code already stopped writing the column; old code expects it |
 | Type change (`VARCHAR(40)` → `TEXT`) | Usually yes | Old code's column read still works |
 | Backfill a column | Yes | Old code ignores the backfilled value |
 If your upgrade only added columns / tables / indexes, `helm
 rollback` is sufficient. If it renamed or dropped anything, you need
 a database-level revert.
 ## Procedure: standard rollback (additive-only migrations)
 ```bash
 # 1. Identify the target revision
 helm history certctl -n <namespace>
 # 2. Take a backup BEFORE rolling back (defense in depth — if
 #    rollback exposes a data corruption issue, restore is the only
 #    path back)
 #    See docs/operator/runbooks/postgres-backup.md for the canonical
 #    pg_dump invocation.
 # 3. Roll back to the chosen revision
 helm rollback certctl <revision> -n <namespace> --wait --timeout 5m
 # 4. Verify
 kubectl get pods -n <namespace> -l app.kubernetes.io/instance=certctl
 kubectl logs -n <namespace> -l app.kubernetes.io/component=server --tail=50
 ```
 Watch for migration-version mismatch warnings in the server logs. If
 the older server code refuses to start because the schema is ahead
 of what it knows about, escalate to "rollback with schema revert."
 ## Procedure: rollback with schema revert
 This is the rare case. Use it when:
 - A column / table was renamed or dropped in the rolled-up release.
 - The older code refuses to start with the newer schema.
 ```bash
 # 1. Take a fresh backup right NOW (the current schema is what we're
 #    reverting from; if anything goes wrong we want a clean
 #    forward-recovery option)
 kubectl exec -n <namespace> statefulset/certctl-postgres -- \
  pg_dump --format=custom --no-owner --no-acl --dbname=certctl \
  > "certctl-pre-rollback-$(date -u +%Y%m%dT%H%M%SZ).dump"
 # 2. Stop the server Deployment to prevent it from writing to the
 #    database during the revert
 kubectl scale deploy/certctl-server -n <namespace> --replicas=0
 # 3. Apply the relevant *.down.sql files manually, one at a time, in
 #    reverse migration-number order. Example for reverting two
 #    migrations:
 NEW=000102  # newest migration on the running schema
 OLD=000100  # oldest migration to revert (inclusive)
 for MIG in 000102 000101 000100; do
  kubectl exec -i -n <namespace> statefulset/certctl-postgres -- \
    psql --user=certctl --dbname=certctl \
    < migrations/${MIG}_*.down.sql
 done
 # 4. Manually update the schema_migrations table to reflect the
 #    reverted state (the migration runner's bookkeeping)
 kubectl exec -n <namespace> statefulset/certctl-postgres -- \
  psql --user=certctl --dbname=certctl -c \
  "DELETE FROM schema_migrations WHERE version > $((OLD - 1));"
 # 5. NOW run helm rollback. The server pod will start with a schema
 #    that matches its code.
 helm rollback certctl <revision> -n <namespace> --wait --timeout 5m
 ```
 The `*.down.sql` files are tested but only against pristine schemas —
 they may not handle every data shape a production database
 accumulates. ALWAYS take a backup first; the down-migrations are
 a recovery tool, not a transactional contract.
 ## Procedure: full restore (when revert isn't tractable)
 When a down-migration would lose data (drop columns / tables that
 hold rows the older code can't read but the newer code populated), a
 full restore is the only safe path. This is the procedure described
 in
 [`docs/operator/runbooks/disaster-recovery.md`](disaster-recovery.md#postgres-restore).
 The summary:
 1. Stop certctl.
 2. Take a backup of the CURRENT schema (defense in depth).
 3. Restore the LAST backup taken BEFORE the bad upgrade.
 4. Roll the Helm release back to the matching code version.
 5. Restart certctl.
 6. Re-run any audited writes that happened in the window between the
   backup and the bad upgrade (read the audit log; the API surface
   is recoverable).
 The DR runbook owns the canonical commands.
 ## Common pitfalls
 - **Forgetting the backup before rollback.** A schema-revert path is
  not safe without a fresh backup. If something goes wrong mid-revert
  and your most recent backup is from last night, you've lost any
  cert-issuance history between then and now.
 - **Rolling back the chart without rolling back the database state**
  on a release that included a destructive migration (drop column,
  drop table). Symptoms: old code starts, queries fail with
  "column does not exist," server crashes in a loop. Recovery
  requires schema revert OR full restore.
 - **Letting the agents drift.** `helm rollback` updates the agent
  DaemonSet's image too — agents on different versions than the
  server may produce incompatible CSR payloads. After rollback,
  confirm agent images are at the matching version via
  `kubectl get daemonset certctl-agent -o jsonpath='{.spec.template.spec.containers[0].image}'`.
 - **GHCR images pinned by digest:** the rollback restores the prior
  `image:` value from the Helm template. If your operator workflow
  uses `image.digest` pinning, the digest comes back too — make
  sure that digest still exists on ghcr.io. They do persist; old
  tags are never deleted, but a private mirror may have garbage-collected.
 ## Related reading
 - [`docs/operator/runbooks/postgres-backup.md`](postgres-backup.md) —
  the backup procedure that's the precondition for any
  schema-revert path.
 - [`docs/operator/runbooks/disaster-recovery.md`](disaster-recovery.md) —
  the full restore procedure when rollback isn't tractable.
 - [`docs/migration/api-keys-to-rbac.md`](../../migration/api-keys-to-rbac.md) —
  example of a migration that the runtime supports rolling back via
  feature flag (rare).
@@ -0,0 +1,123 @@
 # Scale baseline — 2026 Q2 canonical-hardware capture
 > Last reviewed: 2026-05-16
 ## What this file is
 The canonical record of certctl's load-test baselines for the
 2026-Q2 reporting window. TEST-005 closure (Sprint 5, 2026-05-16)
 introduces this doc as the single source of truth for "what's the
 scale ceiling?" — replacing the TBD-laden table at
 [`docs/operator/scale.md`](scale.md#measured-baseline) that had been
 unfilled since the scenarios shipped in Phase 8.
 The numbers below come from the `loadtest` GitHub Actions workflow
 running its three canonical scenarios on `ubuntu-latest` runners:
 - `bulk-renewal` — 10,000-cert seed + criteria-mode
  `POST /api/v1/certificates/bulk-renew`, 200 concurrent VUs over 10
  minutes.
 - `acme-burst` — 200 concurrent VUs hitting `/acme/directory`,
  `/acme/new-nonce`, and `/acme/renewal-info/<cert-id>` simultaneously.
 - `agent-storm` — 5,000-agent seed + sustained
  `POST /api/v1/agents/{id}/heartbeat` at 167 RPS.
 Thresholds enforced inline in `deploy/test/loadtest/k6.js` (p99 < 5s
 for issuance-acceptance, p99 < 2s for list, error rate < 1%). k6 exits
 non-zero on any breach, which propagates through `docker compose up
 --exit-code-from k6 → make loadtest → workflow exit`.
 ## Capture procedure
 1. Trigger the workflow:
   - **Actions** → `loadtest` → **Run workflow**, branch `master`.
   - Wait ~25 minutes for the three matrix legs to finish.
 2. Download each scenario's artifact from the workflow run page:
   - `k6-scale-bulk-renewal-<run-id>`
   - `k6-scale-acme-burst-<run-id>`
   - `k6-scale-agent-storm-<run-id>`
   - Each archive contains the k6 `summary.json` + raw NDJSON
     points (90-day GHA retention).
 3. Run `scripts/scale-baseline/extract.sh <run-id>` (see below) to
   pull the three artifacts and emit the table rows for this doc.
 4. Paste the rows under the **Latest capture** section. Update
   `> Last reviewed:` to today.
 5. Commit the artifacts you want long-term-retained to
   [`deploy/test/loadtest-artifacts/`](../../deploy/test/loadtest-artifacts/)
   using `git lfs` if the archives exceed 100 MB; otherwise commit
   them inline.
 ## Latest capture
 | Scenario | Run ID | Date | p50 | p95 | p99 | Error rate | Peak server RSS | Notes |
 |---|---|---|---|---|---|---|---|---|
 | **bulk-renewal** | _capture pending_ | _capture pending_ | _capture pending_ | _capture pending_ | _capture pending_ | _capture pending_ | _capture pending_ | First post-TEST-005 capture; trigger via workflow_dispatch + extract via the procedure above. |
 | **acme-burst** directory | _capture pending_ | _capture pending_ | _capture pending_ | _capture pending_ | _capture pending_ | _capture pending_ | _capture pending_ | — |
 | **acme-burst** new-nonce | _capture pending_ | _capture pending_ | _capture pending_ | _capture pending_ | _capture pending_ | _capture pending_ | _capture pending_ | — |
 | **acme-burst** renewal-info | _capture pending_ | _capture pending_ | _capture pending_ | _capture pending_ | _capture pending_ | _capture pending_ | _capture pending_ | — |
 | **agent-storm** | _capture pending_ | _capture pending_ | _capture pending_ | _capture pending_ | _capture pending_ | _capture pending_ | _capture pending_ | — |
 The "_capture pending_" placeholders are deliberate — the operator
 fills them after the next `loadtest` workflow_dispatch run. Once
 filled, replace these rows; do not edit them in place across runs
 (the historical row stays as evidence).
 ## Why "ubuntu-latest" instead of RDS-shaped hardware
 The audit's fix language preferred RDS-shaped Postgres on a
 fixed-spec runner. ubuntu-latest's 2-vCPU / 7-GB-RAM shape is
 narrower than typical production Postgres, but it has two virtues:
 1. **Reproducibility.** Every operator + acquirer can reproduce the
   numbers; an RDS-shaped Postgres requires a paid AWS account.
 2. **Conservative ceiling.** If the published numbers come from a
   constrained runner, real-world deployments on production Postgres
   sizes (db.m5.large +) only get better.
 When an acquirer or operator asks for a production-equivalent
 baseline, capture a second run on whatever infrastructure they want
 to validate against and add it under a new **2026 Q3 capture**
 section.
 ## Methodology
 ### Hardware
 - **Runner:** GitHub Actions `ubuntu-latest` (currently Ubuntu 24.04, 2-vCPU, 7-GB RAM).
 - **certctl image:** built from the same commit the workflow runs on.
 - **Postgres:** `postgres:16-alpine@sha256:890480b08124ce7f79960a9bb16fe39729aa302bd384bfd7c408fee6c8f7adb7`, in-cluster, default config (no operator tuning).
 - **Network:** runner localhost.
 ### Software
 - **k6:** version pinned in `deploy/test/loadtest/Dockerfile`.
 - **certctl tag:** the v* tag at workflow trigger time (matches `openapi.yaml info.version`).
 ### Metrics captured
 - **p50 / p95 / p99 latency** — k6's `http_req_duration` percentiles.
 - **Error rate** — k6 `http_req_failed` rate (non-2xx + connection errors).
 - **Peak server RSS** — `docker stats` polled at 1-Hz for the
  duration of the run; `max(memory_stats.usage)` taken from the
  emitted JSON.
 - **Acceptance gate** — the k6 thresholds in `k6.js`; if exceeded
  the workflow fails.
 ### What's NOT captured
 - **Cold-start latency** — these are steady-state baselines after the
  k6 warmup ramp. Cold-start is a separate concern (renewal-loop
  startup, scheduler tick boundary), not covered by these scenarios.
 - **WAN latency** — runs are localhost; production-WAN-RTT additions
  fall outside scope.
 - **Federation overhead** — single-instance only; HA + replicas runs
  are a future deliverable.
 ## Related reading
 - [`docs/operator/scale.md`](scale.md) — the operator-facing scale
  posture doc; baseline rows there point at this file.
 - [`deploy/test/loadtest/README.md`](../../deploy/test/loadtest/README.md) —
  scenario semantics + how to read the k6 output.
 - [`deploy/test/loadtest-artifacts/`](../../deploy/test/loadtest-artifacts/) —
  long-term archive of captured k6 results.
@@ -0,0 +1,244 @@
 # Operator scale guide
 > Last reviewed: 2026-05-16
 Use this when:
 - You're sizing a new certctl deployment for a target fleet count.
 - You're scaling an existing deployment up from demo (15 certs / 1
  agent) to production (1K+ certs / 100+ agents).
 - An auditor asks "what does this scale to?" and you want a documented
  answer that isn't "we haven't measured."
 ## DB connection pool
 certctl's PostgreSQL connection pool is the single largest scale lever.
 Pool exhaustion looks like 503s + agent poll timeouts + scheduler
 falling behind on its loops. The default ships at 50 max open
 connections (`CERTCTL_DATABASE_MAX_CONNS=50`), with idle = max/5 = 10
 under the existing `internal/repository/postgres/db.go::NewDBWithMaxConns`
 contract.
 Operator-tune ladder:
 | Fleet size                  | `CERTCTL_DATABASE_MAX_CONNS` | Postgres `max_connections` | Notes |
 |---|---|---|---|
 | ≤ 500 certs / 100 agents    | `50` (default)               | `100` (PG default)         | Demo + small deployments. Pool default sized for this. |
 | 5K certs / 1K agents        | `100`                        | `200`                      | Postgres needs an explicit bump from the 100 default; reload required. |
 | 50K certs / 10K agents      | `200`                        | `400`                      | Plus dedicated Postgres VM (separate from server host); shared_buffers ≥ 1Gi. |
 Always leave headroom in Postgres's `max_connections` for backups
 (`pg_dump` opens its own connection), ad-hoc psql sessions, and
 replicas. The ratio `(server pool size × replicas) + 20` is a safe
 floor for Postgres's `max_connections`.
 **Numbers above the small-fleet row are operator-tuning starting
 points, not validated ceilings.** Phase 8 of the architecture diligence
 remediation will replace these with measured values from synthetic
 fleets; until then, capture your own observations in a loadtest log
 and tune against them.
 ## Scheduler tick budgets
 certctl has 15 scheduler loops, each with its own cadence
 (internal/scheduler/scheduler.go). The renewal scan is the hottest
 loop on large fleets: it pulls every managed certificate, applies
 each profile's renewal policy, and dispatches an issuance job per
 cert that meets the threshold. The default cadence is `1h`
 (`CERTCTL_SCHEDULER_RENEWAL_CHECK_INTERVAL`).
 Phase 6 SCALE-M5 closure (2026-05-14) added per-ticker jitter via the
 `internal/scheduler.JitteredTicker` wrapper. Each loop's interval is
 unchanged; the wrapper adds ±10% randomized delay per tick so multiple
 loops with the same nominal cadence don't co-fire and cause hour-
 boundary CPU + DB spikes. For most fleets the visible effect is a
 smoother CPU graph during the renewal scan.
 **Renewal-sweep semaphore (SCALE-L1).** The renewal loop dispatches
 concurrent issuance work behind a per-tick semaphore (default
 `CERTCTL_RENEWAL_CONCURRENCY=25`). Under tick-budget pressure (a tick
 that exceeds the loop interval), the semaphore can hold the entire
 concurrency cap until the context cancels at next-tick boundary —
 which is intentional. The drain happens via context cancellation; new
 work isn't started past the deadline. Tests in
 `internal/scheduler/` pin this drain behavior. Operators on large
 fleets should:
 1. Bump `CERTCTL_RENEWAL_CONCURRENCY` to 50 or 100 if the renewal scan
   consistently exceeds tick budget.
 2. Also bump `CERTCTL_DATABASE_MAX_CONNS` proportionally — each
   concurrent renewal task opens its own pool connection during
   issuance / deployment.
 3. Watch for the "renewal scan complete" log line per tick. If it's
   consistently late, you're under-provisioned.
 ## Async CA polling budgets (SCALE-M3)
 DigiCert, Entrust, GlobalSign, and Sectigo are async issuers — they
 accept a CSR, queue it on the CA side, and return a polling token.
 The certctl server polls the CA's status endpoint until the cert is
 ready or the deadline expires. The default poll-deadline is 10
 minutes wall-clock (`asyncpoll.DefaultMaxWait`); after that the
 issuance returns `StillPending` and the scheduler re-enqueues the
 job for the next tick.
 Priority chain when picking the actual deadline (highest → lowest):
 1. Per-connector env: `CERTCTL_DIGICERT_POLL_MAX_WAIT_SECONDS`,
   `CERTCTL_ENTRUST_POLL_MAX_WAIT_SECONDS`,
   `CERTCTL_GLOBALSIGN_POLL_MAX_WAIT_SECONDS`,
   `CERTCTL_SECTIGO_POLL_MAX_WAIT_SECONDS`.
 2. Global env: `CERTCTL_ASYNC_POLL_MAX_WAIT_SECONDS` (sets the
   process-wide default for all async-CA connectors that didn't set
   their per-connector value).
 3. Package const: `asyncpoll.DefaultMaxWait = 10 * time.Minute`.
 Operators with slow async CAs (Entrust certificate-mode in
 particular can take 15-30 minutes during business hours) should
 raise the per-connector value rather than the global; that way fast
 issuers don't pay the polling cost.
 ## Cursor pagination caching (SCALE-L2)
 Phase 6 SCALE-L2 closure (2026-05-14) added an ETag middleware at
 `internal/api/middleware/etag.go` covering the top-5 read endpoints:
 `/api/v1/certificates`, `/api/v1/jobs`, `/api/v1/agents`,
 `/api/v1/audit`, `/api/v1/discovery/certificates`. The ETag is
 derived from `(max-row-updated-at, row-count)` for the requested
 filter; repeated requests with the same query return `304 Not
 Modified` when the underlying data hasn't changed. The dashboard
 benefits most — its polling loop on the certificates page is the
 single largest read-traffic source on most deployments.
 When the cache is effective, repeated reads bypass the
 `SELECT COUNT(*) FROM <table>` query entirely. The cache invalidates
 on any mutation to the table (the row-count + max-updated-at hash
 flips).
 Operators don't need to do anything to opt in — the middleware is
 wired around the top-5 endpoints unconditionally. If you want to
 verify it's working, check the `ETag:` response header on a list
 endpoint and repeat the request with the same value in an
 `If-None-Match:` header — the second request should return 304 with
 an empty body.
 ## Scale-tier scenarios (SCALE-H2, Phase 8)
 Phase 8 (2026-05-14) extended the k6 load-test harness with three new
 scenarios that exercise the scale-relevant load surfaces the original
 API tier left uncovered. They live behind a compose profile gate
 (`docker compose --profile scale`) so the default `make loadtest`
 stays focused on per-PR regression scope. The full set runs weekly on
 the same `loadtest.yml` cron as the API + connector tier.
 | Scenario | k6 file | Seed fixture | Sustained load |
 |---|---|---|---|
 | Bulk-renewal under load | `deploy/test/loadtest/k6/bulk_renewal.js` | 10,000 managed_certificates (`seed/01_bulk_renewal_certs.sql`) | 5 req/s POST `/api/v1/certificates/bulk-renew` × 5 min |
 | ACME enrollment burst | `deploy/test/loadtest/k6/acme_burst.js` | (none — unauth surface) | 200 concurrent VUs × directory/nonce/ARI × 5 min |
 | Agent heartbeat storm | `deploy/test/loadtest/k6/agent_storm.js` | 5,000 agents (`seed/02_agent_fleet.sql`) | 167 req/s POST `/api/v1/agents/{id}/heartbeat` × 5 min |
 ### Threshold contracts (regression guards, NOT measured baselines)
 | Scenario | Metric | Threshold |
 |---|---|---|
 | Bulk-renewal | `http_req_duration{scenario:bulk_renewal}` p99 | < 5 s |
 | Bulk-renewal | `http_req_duration{scenario:bulk_renewal}` p95 | < 2 s |
 | Bulk-renewal | `http_req_failed{scenario:bulk_renewal}` | < 1% |
 | ACME burst | `acme_directory_duration` p95 | < 500 ms |
 | ACME burst | `acme_new_nonce_duration` p95 | < 300 ms |
 | ACME burst | `acme_renewal_info_duration` p95 | < 800 ms |
 | ACME burst | `http_req_failed{server_error:true}` 5xx-only | < 0.1% |
 | Agent storm | `http_req_duration{scenario:agent_storm}` p99 | < 1 s |
 | Agent storm | `http_req_duration{scenario:agent_storm}` p95 | < 500 ms |
 | Agent storm | `http_req_failed{scenario:agent_storm}` | < 0.1% |
 429 rate-limit responses on the ACME burst are EXPECTED — Phase 5's
 per-account rate limiter SHOULD fire at sustained 200-VU pressure.
 The custom `acme_rate_limited_count` Counter tracks how often it
 fires; `acme_rate_limit_shape_ok` Counter verifies every 429 returns
 the RFC 7807 `application/problem+json` shape with the
 `urn:ietf:params:acme:error:rateLimited` type. A regression that
 returned plain-text 429 or a different problem type would surface as
 `(rate_limited_count - shape_ok_count) > 0` in the summary.
 ### Measured baseline
 TEST-005 closure (Sprint 5, 2026-05-16) moved the baseline table out
 of this file into its own canonical record:
 [`docs/operator/scale-baseline-2026-Q2.md`](scale-baseline-2026-Q2.md).
 That doc owns the capture procedure, the methodology, and the
 per-scenario rows; this page links to it as the authoritative
 source.
 The split exists because the baseline table is mutable on every
 loadtest workflow_dispatch run, while this page (the operator-facing
 scale posture doc) changes only when the underlying scenarios or
 thresholds change. Keeping them in separate files avoids
 review-noise on per-capture commits.
 Long-term k6 NDJSON artifacts beyond GHA's 90-day retention live at
 [`deploy/test/loadtest-artifacts/`](../../deploy/test/loadtest-artifacts/).
 ### How to run the scale tier locally
 ```sh
 # All three scenarios serially (~18 min total):
 make loadtest-scale
 # Individual scenarios (each ~6 min):
 make loadtest-scale-bulk     # 10K cert bulk-renew
 make loadtest-scale-acme     # 200 VU ACME burst
 make loadtest-scale-agent    # 5K agent heartbeat storm
 ```
 Each scenario boots its own copy of the loadtest compose stack
 (postgres + tls-init + certctl-server) plus the `scale-seed` init
 container that runs the SQL fixtures from `deploy/test/loadtest/seed/`.
 The seed is idempotent (`ON CONFLICT … DO NOTHING`) so re-running a
 scenario against the same compose stack is cheap.
 ### Documented limitations of the scale tier
 - **JWS-signed ACME flows are not measured.** The ACME burst scenario
  hits the unauthenticated directory + new-nonce + ARI surface only.
  Measuring the JWS-signed POST hot path (new-account / new-order /
  finalize) requires bundling a JWS signer into the k6 driver (k6
  doesn't ship JWS). End-to-end JWS conformance is gated by
  `make acme-rfc-conformance-test` which drives `lego` against the
  same stack.
 - **Scheduler renewal scan throughput.** The bulk-renewal scenario
  measures the inbound POST throughput; the scheduler's
  `jobProcessorLoop` drains the enqueued jobs at a fixed per-tick
  budget (`CERTCTL_RENEWAL_CONCURRENCY=25` default), and the
  throughput of that path is not amplified by adding more inbound
  bulk-renew calls. A future scenario could pull
  `/api/v1/jobs?status=pending` and measure drain time.
 - **Production-sized Postgres.** The compose stack runs
  `postgres:16-alpine` with default config on a CI runner.
  Production deploys with `shared_buffers >= 1 GiB` + dedicated
  Postgres VM will have different query plans for the 10K-cert
  scan. The captured numbers translate directionally but the
  absolute ceiling is workload-specific — see the operator-tune
  ladder above for production sizing.
 - **Pull-only deployment model.** Agent CSR submit, work-poll, and
  deploy-verify paths are intentionally out of scope. The heartbeat
  storm exercises the highest-frequency call on a typical fleet;
  the work-poll path runs at the same cadence but is cheap (empty
  set returned 99% of the time).
 ## Profiling production
 When the above ladder doesn't fit your shape, profile against your
 specific workload. The
 [performance-baselines.md](performance-baselines.md) runbook has
 single-endpoint, inventory-walk, and renewal-scan recipes you can
 adapt.
 ## Related reading
 - [`docs/operator/performance-baselines.md`](performance-baselines.md) —
  per-endpoint baselines + how to re-baseline after upgrades.
 - [`docs/operator/runbooks/postgres-backup.md`](runbooks/postgres-backup.md) —
  Postgres-side backup discipline (necessary precondition for any
  scale tuning).
 - [`deploy/ENVIRONMENTS.md`](../../deploy/ENVIRONMENTS.md) — the
  full env-var inventory the values referenced above come from.
@@ -0,0 +1,166 @@
 # Secret custody — where private keys live in certctl
 > Last reviewed: 2026-05-12
 Use this when:
 - You're sizing certctl against an internal security review or third-party
  diligence ("where do private keys live, and how are they protected at
  rest?").
 - You're evaluating the file-on-disk vs HSM-vs-cloud-KMS roadmap before
  committing to a deployment topology.
 - You need a single page that names every secret material on the control
  plane and on agents, plus the at-rest protection for each.
 This document covers WHAT secrets exist, HOW they are stored, and the
 THREAT MODEL we accept for each — it is not a hardening checklist. The
 hardening levers (env-vars, file modes, encryption-key configuration) are
 cross-referenced as you read through.
 ## The secrets that exist
 | Material                        | Where it lives                                                                  | Protection at rest                                                                                                  | Closes when…                                                                       |
 |---|---|---|---|
 | Local CA private key            | File on the control-plane host (`CERTCTL_CA_KEY_PATH`)                          | Filesystem ACLs (operator-supplied path; mode 0600 recommended)                                                     | A `signer.PKCS11Driver` or `signer.CloudKMSDriver` ships (post-v2.1.0)             |
 | Agent ECDSA P-256 private keys  | File on each agent host (default `/var/lib/certctl-agent/keys/`)                | Filesystem ACLs on the agent host. Never transmitted to the control plane.                                          | TPM / Secure Enclave drivers ship (no current roadmap entry)                       |
 | OIDC client secret              | `oidc_providers.client_secret_enc` column (PostgreSQL)                          | AES-256-GCM v3 wire format, derived from `CERTCTL_CONFIG_ENCRYPTION_KEY` via PBKDF2-SHA256 600k rounds              | The encryption key is rotated via `internal/crypto` re-seal (see runbook below)    |
 | Session signing key             | `auth_session_signing_keys` table (PostgreSQL)                                  | AES-256-GCM v3, same encryption-key passphrase as above                                                             | HSM/FIPS-validated signing-key driver lands (deferred to v3)                       |
 | Break-glass credential          | `breakglass_credentials.password_hash` column (PostgreSQL)                      | Argon2id (m=64MiB, t=1, p=4) hash; never encrypted because we need constant-time comparison                         | Out of scope — Argon2id resists offline attack already                             |
 | API-key bearer tokens           | `auth_api_keys.token_hash` column (PostgreSQL)                                  | SHA-256(token) only — the plaintext is shown to the operator once at create time and never persisted                | Out of scope                                                                       |
 | CSR private keys mid-issuance   | Agent memory only, ephemeral                                                    | Never written to disk; never transmitted to the server (CSRs only)                                                  | Already closed                                                                     |
 | Issuer-connector backend secrets | `issuers.encrypted_config` column (PostgreSQL) for `source='database'` rows    | AES-256-GCM v3; FAIL-CLOSED if `CERTCTL_CONFIG_ENCRYPTION_KEY` is unset (see "Env-seeded vs DB-seeded" below)        | Already closed for `source='database'`; `source='env'` carries an explicit carve-out |
 The breakdown by row source matters and is the subject of the next
 section. Read it before concluding that a plaintext column is a bug.
 ## Env-seeded vs DB-seeded configs
 certctl supports two sources for issuer and target configurations:
 - **`source='env'`** — built from process environment variables on every
  boot (`CERTCTL_CA_CERT_PATH`, `CERTCTL_CA_KEY_PATH`, `CERTCTL_ACME_DIRECTORY_URL`,
  `CERTCTL_STEPCA_URL`, etc. — see `internal/service/issuer.go::buildEnvVarSeeds`
  for the exact list). These rows are deterministically reconstructable from environment and
  exist primarily so the GUI has something to display and so audit logs
  can reference an issuer ID. The `config` column is intentionally
  plaintext for `source='env'` rows: the exact same bytes already live
  in the operator's Compose file / Helm values / systemd unit, so
  persisting them again to PostgreSQL adds no new disclosure surface.
 - **`source='database'`** — created via the GUI or REST API write paths
  (`POST /api/v1/issuers`, etc.). These rows fail closed when
  `CERTCTL_CONFIG_ENCRYPTION_KEY` is not configured:
    - The HTTP handlers refuse the write with
      `crypto.ErrEncryptionKeyRequired`.
    - The server **refuses to start** if any `source='database'` row
      exists without the encryption key, to prevent retroactive
      plaintext exposure.
 The startup guard is in `cmd/server/main.go` around the
 `encryptionKey != ""` branch — it lists `source='database'` rows on every
 boot and aborts if any are present without the key.
 If you want every issuer/target row to be encrypted at rest unconditionally,
 set `CERTCTL_CONFIG_ENCRYPTION_KEY` and use database-sourced
 configurations exclusively (re-create env-seeded rows through the GUI
 once the key is present).
 ## The signer abstraction
 All CA private-key signing flows through
 `internal/crypto/signer.Signer`, which embeds the stdlib `crypto.Signer`
 and adds `Algorithm()`. Two drivers ship today:
 - `signer.FileDriver` — the production default. Wraps the historical
  file-on-disk PEM flow without behavior change. **Heap-resident**:
  while certctl is running, the key bytes sit in the process's address
  space.
 - `signer.MemoryDriver` — used in tests; never reaches production code
  paths.
 The disk-exposure leg of the threat model is documented inline at the
 top of `internal/connector/issuer/local/local.go` (the L-014 carve-out).
 The mitigations on the FileDriver leg include:
 - mode 0600 enforced on the key file at startup,
 - the key directory is not served by any handler,
 - the bytes are never logged or echoed in audit events,
 - the server fails closed if it cannot read the key.
 `FileDriver` does NOT mitigate "an attacker with read access to the
 control-plane filesystem can recover the CA key." That mitigation lives
 in a future `signer.PKCS11Driver` (hardware token) or
 `signer.CloudKMSDriver` (AWS/GCP/Azure KMS). The interface exists; the
 drivers do not ship yet. Both are post-v2.1.0 roadmap items — see
 [`docs/reference/architecture.md`](../reference/architecture.md) for the
 target topology.
 If you need HSM-grade key custody today, you have two options:
 1. Run certctl behind an enterprise issuer (Microsoft ADCS, EJBCA,
   Smallstep, ACME-public) and configure certctl's local CA as
   intermediate-only or disable it entirely. The issuer connector then
   sends every signing request to your existing hardware-rooted PKI.
 2. Wait for the PKCS#11 driver. Track its status in
   [WORKSPACE-ROADMAP.md](../../WORKSPACE-ROADMAP.md).
 ## Config-encryption wire format
 `internal/crypto/encryption.go` produces and reads three on-disk
 formats. The read path accepts all three; the write path emits only
 the newest:
 | Version | Magic byte | Salt              | PBKDF2-SHA256 work factor | Status                                                            |
 |---|---|---|---|---|
 | v3      | `0x03`    | per-ciphertext 16B | 600,000                  | **Default for all writes** (OWASP 2024)                            |
 | v2      | `0x02`    | per-ciphertext 16B | 100,000                  | Legacy read-only; superseded by v3                                 |
 | v1      | none      | fixed 28B          | 100,000                  | Pre-M-8 legacy read-only; written before per-ciphertext-salt fix   |
 The wire-format documentation is also in the `internal/crypto/encryption.go`
 package comment.
 ### Forcing legacy blob upgrades
 Re-sealing happens passively: any `UPDATE` against a row that contains a
 v1 or v2 blob triggers a v3 rewrite the next time the field is set.
 There is no in-place migration tool because re-sealing requires reading
 the row through the same code path that performs the write, and any
 operational path that touches the row (renaming an issuer in the GUI,
 updating a target's endpoint, refreshing an OIDC provider's
 client-secret) achieves this naturally.
 If you want to FORCE re-sealing across the entire database, use the
 runbook at
 [`docs/operator/runbooks/config-encryption-upgrade.md`](runbooks/config-encryption-upgrade.md).
 Recommended only if you suspect the encryption-key passphrase has
 been exposed and have already rotated it (the runbook covers the
 rotation order: set the new key, force re-seal, retire the old key
 from the rotation pool).
 ## Roadmap (what is not yet closed)
 Tracked in [`WORKSPACE-ROADMAP.md`](../../WORKSPACE-ROADMAP.md), not
 maintained here to prevent drift:
 - `signer.PKCS11Driver` for HSM-token-backed CA key custody.
 - `signer.CloudKMSDriver` for AWS/GCP/Azure KMS-backed CA key custody.
 - FIPS 140-3 mode for the entire control plane.
 - HSM-backed session signing key (currently HMAC-SHA256 software keys).
 If a buyer or auditor asks for "HSM support," the honest answer is:
 the interface is there, the drivers are not, and an enterprise issuer
 connector is the bridge until the drivers ship.
 ## Related reading
 - [`docs/operator/security.md`](security.md) — the broader hardening
  checklist; covers TLS, RBAC, audit logging, network policy.
 - [`docs/operator/auth-threat-model.md`](auth-threat-model.md) — the
  authentication-subsystem threat model. Item 5 ("HSM / FIPS-validated
  signing key for sessions") is the session-signing-key analog of this
  document's CA-key story.
 - [`docs/reference/architecture.md`](../reference/architecture.md) §
  "Signer abstraction" — the diagram form of the FileDriver / future
  PKCS11Driver / CloudKMSDriver topology.
 - [`internal/crypto/encryption.go`](../../internal/crypto/encryption.go)
  package comment — wire format authoritative reference.
 - [`internal/connector/issuer/local/local.go`](../../internal/connector/issuer/local/local.go)
  L-014 carve-out — the load-bearing threat-model section for the
  FileDriver case.
@@ -58,7 +58,55 @@ For certificates issued to systems where revocation correctness matters:
 ## Postgres transport encryption
-See [docs/database-tls.md](database-tls.md).
+**Audit references:** SEC-013 (advisory) and SEC-014 (host-port bind),
 both closed in Sprint 2 of the 2026-Q2 acquisition audit
 (2026-05-16).
 The full upgrade procedure (sslmode flags, CA bundle paths, Helm chart
 values, AWS RDS / Google Cloud SQL / Azure Database notes) lives at
 [docs/operator/database-tls.md](database-tls.md). The summary of the
 two operator-visible defenses certctl ships:
 ### SEC-014 — Postgres host port is loopback-only
 `deploy/docker-compose.yml` and `deploy/docker-compose.test.yml` both
 publish Postgres on `127.0.0.1:5432:5432` rather than `5432:5432`.
 The default Docker port-binding behavior is to bind to `0.0.0.0`,
 which exposes Postgres on every interface of the host — including any
 public-facing NICs the operator did not realize were attached. The
 loopback bind closes that footgun without breaking the
 certctl-server hop (which goes over the `certctl-network` Docker
 bridge, not over the host port).
 Operators who genuinely need to reach Postgres from another host —
 e.g. a separate metrics box running `postgres_exporter` — should
 either (1) attach that host into the same Docker network, (2) tunnel
 through SSH (`ssh -L`), or (3) re-publish the port with explicit
 `bind:` configuration and a documented network-layer access control.
 Loosening the loopback bind without one of those is a regression.
 ### SEC-013 — advisory WARN on external `sslmode=disable`
 `internal/config/config.go::Validate` emits an `slog.Warn` (NOT a
 fail-closed error) when `CERTCTL_DATABASE_URL` parses as a Postgres
 URL with `sslmode=disable` AND the host is outside the local
 safelist (`localhost` / `127.0.0.1` / `::1` / `postgres` /
 `certctl-postgres` / `*.svc.cluster.local`). The advisory exists
 because the legitimate compose / Helm topology genuinely uses
 `sslmode=disable` over the Docker bridge — failing closed would
 break the production-shaped quickstart — but pointing
 `CERTCTL_DATABASE_URL` at a managed-Postgres host (RDS, Cloud SQL,
 Azure Database) without flipping `sslmode` to `verify-full` puts
 the entire control plane's Postgres traffic on the wire in
 cleartext. The WARN surfaces that landmine on every boot so the
 operator notices it in the journal even if the rest of the boot
 sequence looks healthy.
 To clear the WARN: set `CERTCTL_DATABASE_URL` to a URL with
 `sslmode=verify-full` and `sslrootcert=<ca-bundle-path>`. The full
 procedure (CA-bundle materialization, Helm chart values, secret-
 manager wiring) is in
 [docs/operator/database-tls.md](database-tls.md).
 ## Encryption at rest
@@ -403,6 +451,124 @@ the end of step 4, extend the window before step 5.
  from the env var and restart. That's appropriate for a small env-var
  inventory; it would not scale to a per-user-key-issued model.
 ## Security carve-outs &amp; operator-tunable defaults
 Phase 2 of the architecture diligence remediation (2026-05-13)
 consolidated the following carve-outs into one canonical section so
 operators reviewing security posture have a single search target. Each
 entry cites the exact file:line of the carve-out, why it exists, and
 what the operator should do.
 ### TLS verification — dev escape hatches
 certctl has three `InsecureSkipVerify=true` sites that are dev/probe
 escape hatches, never enabled by default in production:
 - **Agent dev escape** — `cmd/agent/main.go:179` (wired from
  `cmd/agent/main.go:61` config field + `cmd/agent/main.go:1371` CLI
  flag). Operators flip this only when debugging an agent against a
  self-signed control plane that hasn't been added to the agent's
  trust store. Document as `--insecure-skip-verify` in the agent's
  install runbook; the agent logs a startup WARN any time the flag
  is set. SEC-M3 pins that the carve-out is intentional.
 - **Agent verification probe** — `cmd/agent/verify.go:78`. The probe
  intentionally opens a TLS connection with verification disabled so
  it can inspect any certificate the endpoint serves (including
  self-signed or expired ones — that's the whole point of a probe).
  The probe never returns trust state to a security-relevant code
  path; it only reads cert metadata. SEC-M3 pins this.
 - **tlsprobe (network scanner)** — `internal/tlsprobe/probe.go:54`.
  Same rationale as the agent verify probe — network discovery must
  introspect any certificate it finds, including the ones with the
  problems we're scanning for. SEC-M3 pins this.
 ### F5 target connector — `InsecureSkipVerify` per-config
 The F5 target connector exposes an `Insecure: bool` field on its
 per-target config blob (default `false`). When set,
 `internal/connector/target/f5/f5.go:134` builds the HTTP client with
 `InsecureSkipVerify: config.Insecure`. SEC-M5 closure: operator
 opt-in for self-signed F5 BIG-IP device certs; mitigation is to run
 the F5 + the proxy-agent on a network-segmented internal subnet.
 Document in the F5 connector's per-target setup guide.
 ### ACME issuer — `CERTCTL_ACME_INSECURE` (now gated on ACK)
 `internal/connector/issuer/acme/acme.go:201` builds the ACME HTTP
 client with `InsecureSkipVerify: true` for the Pebble integration
 test path. The per-issuer runtime setting comes from
 `CERTCTL_ACME_INSECURE` (`internal/config/config.go:2116`); Phase 2
 SEC-M4 closure (2026-05-13) added the fail-closed gate so the operator
 must ALSO set `CERTCTL_ACME_INSECURE_ACK=true` for the server to boot.
 Production deploys must never set either flag. The boot-time WARN log
 at `cmd/server/main.go:611` continues to fire for the ACK'd case so
 every restart logs the reminder.
 ### CSP `'unsafe-inline'` on `style-src`
 `internal/api/middleware/securityheaders.go:58` ships the dashboard
 CSP with `style-src 'self' 'unsafe-inline'`. This is required because
 Tailwind compiles utility classes into a single stylesheet at build
 time, but inline-style attributes appear in the dashboard via inline
 `<svg>` elements + Recharts' `<ResponsiveContainer>` injecting inline
 width/height. SEC-L1 closure: the carve-out is necessary today; the
 planned tightening flow is the frontend audit's FE-H2 (icon library)
 + decorative-SVG sweep that then unlocks the CSP hardening (drops
 `'unsafe-inline'`).
 ### Break-glass admin — Argon2id rest-defense reminder
 The break-glass admin path (`docs/operator/runbooks/disaster-recovery.md`)
 hashes the operator-supplied password with Argon2id and stores the
 hash in the `breakglass_credentials` table. SEC-L2 reminder: the
 strength of the rest-defense is operator-supplied — pick a password
 with sufficient entropy (≥ 64 random bits via `openssl rand -base64
 12`) and rotate after every use. Argon2id resists offline cracking
 but an operator-supplied "Password123" hashes the same way.
 ### Body-size limit (1 MB default) — operator-tunable
 The `http.MaxBytesReader` wrap caps inbound request bodies at 1 MB
 by default. The cap is necessary defense against unbounded-body DOS
 but catches legitimate operator workflows:
 - Bulk truststore PEM bundle uploads (CA bundles for federated trust
  stores can be > 1 MB).
 - Multi-MB CRL pushes via the CRL-cache endpoint.
 - Bulk-import of certificates with embedded chains.
 SEC-L3 closure: operators raise the cap via `CERTCTL_MAX_BODY_SIZE`
 (units: bytes; e.g. `CERTCTL_MAX_BODY_SIZE=10485760` for 10 MB).
 Document in `deploy/ENVIRONMENTS.md`.
 ### Demo Compose placeholder credentials
 `deploy/docker-compose.demo.yml` ships `CERTCTL_AUTH_SECRET=change-me-in-production`,
 `CERTCTL_CONFIG_ENCRYPTION_KEY=change-me-32-char-encryption-key`, and
 `CERTCTL_API_KEY=change-me-in-production` as documented demo
 defaults. The runtime `Validate()` fail-closed guards
 (`internal/config/config.go::Validate`, Bundle 2 2026-05-12) refuse
 to start if those literal strings reach a non-demo config. Phase 2
 DEPL-M2 closure adds a CI guard
 (`scripts/ci-guards/no-change-me-in-prod-compose.sh`) that fails the
 build at PR time if a `change-me-*` literal leaks into a non-demo
 compose file — catching the regression one layer before the runtime
 guard fires.
 ### Kubernetes NetworkPolicy — operator-opt-in
 `deploy/helm/certctl/templates/networkpolicy.yaml` ships the template
 but `deploy/helm/certctl/values.yaml` defaults `networkPolicy.enabled:
 false`. DEPL-M3 rationale: most Kubernetes clusters don't have a
 NetworkPolicy controller installed (kind / minikube / fresh k3s); a
 default-enabled NetworkPolicy renders fine but produces no
 enforcement, and bare-metal `kube-router`-style controllers may
 interpret a permissive default differently. Production deploys with a
 real NetworkPolicy controller (Calico, Cilium, Antrea) flip the
 values key to `true` and tune the policy in their values overlay.
 Document the production-enable in
 `docs/operator/runbooks/ha.md` (added Phase 2 DEPL-H1).
 ## Reporting a vulnerability
 Email `certctl@proton.me`. Coordinated disclosure preferred; we will
@@ -1,6 +1,6 @@
 # Architecture Guide
-> Last reviewed: 2026-05-05
+> Last reviewed: 2026-05-16
 ## Contents
@@ -55,6 +55,45 @@ New to certificates? Read the [Concepts Guide](concepts.md) first.
 7. **Connector Architecture** — Pluggable issuers, targets, and notifiers for extensibility
 8. **Self-Hosted** — No cloud lock-in; run with Docker Compose, Kubernetes, or bare metal
 ### Single-tenant deployment model
 certctl runs as a **single-tenant** application today. Every authenticated
 request is stamped with `auth.DefaultTenantID` by the auth middleware
 (`internal/auth/middleware.go` — the `TenantIDKey` context value is
 constant for the process lifetime), and repository queries don't filter
 on tenant. A deploy is one tenant; a buyer running multiple business
 units on one cluster needs one certctl deployment per business unit.
 The `tenant_id` columns sprinkled across the schema (`actors`,
 `managed_certificates`, `agents`, `users`, `roles`, audit log, etc.) are
 **forward-compatible scaffolding** for the multi-tenancy roadmap item
 in `WORKSPACE-ROADMAP.md`, not active multi-tenant code. A repo skimmer
 who sees the columns can reasonably assume tenant isolation is wired
 end-to-end; it isn't. The `scripts/ci-guards/multi-tenant-query-coverage.sh`
 guard exists to track the drift and is treated as informational (warns
 on net-new tenant_id-less queries above a baseline) — flipping it to a
 hard gate is the inflection-point work for activating multi-tenancy.
 Lifting this to a multi-tenant deployment requires three pieces of
 work in sequence:
 1. **Request-derived tenant resolution.** Replace the constant
   `DefaultTenantID` stamp with a resolution function that picks
   the tenant from the actor (`actors.tenant_id`) or a hostname /
   path-prefix routing convention.
 2. **Per-query tenant scoping.** Every `WHERE` clause that joins
   on a `tenant_id`-bearing table must add `AND tenant_id = $N`.
   The multi-tenant-query-coverage guard tracks this surface;
   activating multi-tenancy means driving its baseline to zero.
 3. **Per-tenant resource quotas + isolation tests.** RBAC scope
   types extend with `tenant`; integration tests exercise
   cross-tenant data-leak prevention; quotas (certs/issuers/agents
   per tenant) wire into the existing limit-enforcement layer.
 Until that work lands, **the multi-tenant columns are decorative**.
 Treat them as you would a Postgres `version` column on a row you
 never read — the schema is forward-compat, the runtime is not.
 ## System Components
 ```mermaid
@@ -151,7 +190,12 @@ The agent runs two background loops: a heartbeat (every 60 seconds) to signal it
 Retired agents receive `410 Gone` on subsequent heartbeats (`service.ErrAgentRetired`). `cmd/agent` treats 410 as a terminal signal and exits cleanly so retired agents stop phoning home. Migration `000015` flipped `deployment_targets.agent_id` from `ON DELETE CASCADE` to `ON DELETE RESTRICT`, making the old hard-delete path a schema error and forcing all retirement through this contract.
-**Registration is by-design pull-only (C-1 closure, cat-b-6177f36636fb).** Agents register themselves at first heartbeat via `install-agent.sh` + `cmd/agent/main.go` — never via the GUI. The `web/src/api/client.ts::registerAgent` client function is intentionally orphan in the dashboard for this reason. It's preserved in `client.ts` (rather than deleted) so future features that want to drive registration from the GUI — for example, a one-click "register proxy agent" panel for network-appliance topologies where the agent runs in a different network zone from the device it manages — can reach the endpoint without a `client.ts` edit. Operators looking to scale agent enrollment use `install-agent.sh` against a config-management system (Ansible, Salt, Puppet) or a baked-in cloud-init script, not the dashboard.
+**Registration is a two-step operator-driven flow (C-1 closure, cat-b-6177f36636fb).** Agent enrollment is intentionally NOT auto-driven by the agent binary — the agent fail-fasts at startup if `CERTCTL_AGENT_ID` is unset (`cmd/agent/main.go`: "agent-id flag or CERTCTL_AGENT_ID env var is required"). Operators register an agent in one of two ways before starting it:
 1. **Programmatic** — `POST /api/v1/agents` with the agent's metadata payload and (when configured) an `Authorization: Bearer <CERTCTL_AGENT_BOOTSTRAP_TOKEN>` header. The response carries the `id` field; that string goes into `CERTCTL_AGENT_ID` for the agent process. Suitable for config-management (Ansible, Salt, Puppet) or cloud-init flows.
 2. **GUI** — the dashboard's Agents page exposes the same endpoint via `web/src/api/client.ts::registerAgent`. The function is kept reachable rather than deleted so the eventual "register proxy agent" panel for network-appliance topologies can land without a `client.ts` edit; today the panel is not yet wired into the page.
 Once registered, the operator passes the returned ID to `install-agent.sh` via `--agent-id` (or sets the env var directly) and starts the agent. The pull-only deployment model (the server never initiates outbound connections to agents) means this asymmetric flow is by-design: only the agent's network reach matters, and registration always crosses that boundary outbound from the agent's side once the agent boots with a valid ID.
 ### Web Dashboard
@@ -1033,14 +1077,31 @@ The HTTP middleware stack processes requests in the following order (see `cmd/se
 4. **BodyLimit** - request body size cap via `http.MaxBytesReader`
 5. **RateLimiter** - token bucket rate limiting (optional, when enabled)
 6. **CORS** - cross-origin request handling (deny-by-default)
-7. **Auth** - API key validation (or none in development; JWT/OIDC via authenticating gateway, see below — not in-process)
+7. **Auth** - one of three production paths (see "In-process authentication surface" below) or `none` for development
 8. **AuditLog** - records every API call to the audit trail (requires auth context for actor)
-### Authenticating-gateway pattern (JWT, OIDC, mTLS)
+### In-process authentication surface
-certctl's in-process authentication surface is intentionally narrow: `api-key` for production deployments and `none` for development. There is no in-process JWT, OIDC, mTLS, or SAML middleware. (`CERTCTL_AUTH_TYPE=jwt` was accepted pre-G-1 but silently routed through the api-key bearer middleware — a security finding masquerading as a config option, removed at the v2.x boundary; see [`upgrade-to-v2-jwt-removal.md`](upgrade-to-v2-jwt-removal.md) if you previously set it.)
+certctl ships three production-grade in-process authentication paths plus a `none` mode for development. Auth Bundle 2 (commit `dea5053`, 2026-05-12) added native OIDC + sessions + break-glass alongside the v2.0.x API-key path; the older "authenticating-gateway only" framing the previous draft of this doc carried is no longer accurate.
-For deployments that need JWT/OIDC/mTLS, the standard pattern is to put an authenticating gateway in front of certctl and configure `CERTCTL_AUTH_TYPE=none` on the upstream certctl process. The gateway terminates the federated identity protocol, validates tokens / certificates / SAML assertions, and proxies the authenticated request to certctl as a same-origin call on a private network. This separation gives operators the full breadth of the modern identity ecosystem (oauth2-proxy, Envoy `ext_authz`, Traefik `ForwardAuth`, Pomerium, Authelia, Caddy `forward_auth`, Apache `mod_auth_openidc`, nginx `auth_request`) without certctl itself having to track signing-key rotation, claim mapping, audience validation, and the rest of the JWT/OIDC surface area. Operators wanting per-request actor attribution past the gateway boundary forward the gateway-resolved identity (e.g., `X-Auth-Request-User` from oauth2-proxy) and run a small authorization layer at the gateway that enforces the bearer-key contract certctl actually uses.
+| `CERTCTL_AUTH_TYPE` | What it authenticates | When to use |
 |---|---|---|
 | `api-key` (default) | `Authorization: Bearer <key>` matched against SHA-256-hashed `CERTCTL_AUTH_SECRET` / `CERTCTL_API_KEYS_NAMED` rows. | Production deploys without an IdP; agent ↔ server; machine-to-machine; CI. |
 | `oidc` | Federated SSO via any OIDC IdP (Keycloak / Authentik / Okta / Auth0 / Entra ID / Google Workspace). PKCE-S256 + RFC 9700 pre-login UA/IP binding + RFC 9207 iss check + alg-downgrade defense. Successful login mints an HMAC-signed server-side session (cookie + CSRF rotation + back-channel logout). | Production deploys with an existing IdP; human admin access; SOC 2 / SAS 70 deployments. |
 | `none` (demo) | Every request served as the synthetic admin actor `actor-demo-anon`. | Demo / evaluation only. The fail-closed `CERTCTL_DEMO_MODE_ACK=true` requirement (Audit 2026-05-10 HIGH-12) prevents accidental production use; the boot-time WARN banner (Bundle 2) makes the posture unmissable. |
 Side surfaces:
 - **Day-0 bootstrap** via `CERTCTL_BOOTSTRAP_TOKEN` + `POST /api/v1/auth/bootstrap` mints the first admin actor + API key one-shot; the endpoint closes itself the moment any admin exists.
 - **Break-glass admin** (Auth Bundle 2 Phase 7.5) — Argon2id-hashed local-password recovery for SSO-outage. Default-OFF (`CERTCTL_BREAKGLASS_ENABLED=false`); surface returns 404 to scanners when disabled. Rate-limited at 5/min per source IP at the route (Bundle 5 closure).
 - **RBAC enforcement** on every gated handler via `auth.RequirePermission(perm, scope, scopeID)` — seven default roles (admin / operator / viewer / agent / mcp / cli / auditor), 33-permission canonical catalogue, scope types (global / profile / issuer). Auditor split is load-bearing: `r-auditor` holds only `audit.read` + `audit.export`.
 For deployments that need a federated-identity protocol certctl doesn't ship natively (SAML, mTLS-as-auth, LDAP), the authenticating-gateway pattern is still the right answer:
 ### Authenticating-gateway pattern (SAML, mTLS-as-auth, LDAP)
 When the operator's identity ecosystem requires a protocol certctl doesn't ship natively in-process — SAML 2.0, mTLS-as-authentication (TLS client cert binding to actor), LDAP-direct, Kerberos — the standard pattern is to put an authenticating gateway in front of certctl and configure `CERTCTL_AUTH_TYPE=none` on the upstream. The gateway terminates the federated identity protocol, validates tokens / certificates / SAML assertions, and proxies the authenticated request to certctl as a same-origin call on a private network. This separation gives operators the full breadth of the modern identity ecosystem (oauth2-proxy, Envoy `ext_authz`, Traefik `ForwardAuth`, Pomerium, Authelia, Caddy `forward_auth`, Apache `mod_auth_openidc`, nginx `auth_request`) without certctl itself having to track signing-key rotation, claim mapping, audience validation, and the rest of the protocol surface area for every standard. Operators wanting per-request actor attribution past the gateway boundary forward the gateway-resolved identity (e.g., `X-Auth-Request-User` from oauth2-proxy) and run a small authorization layer at the gateway that enforces the bearer-key contract certctl actually uses.
 The historical context: pre-G-1, `CERTCTL_AUTH_TYPE=jwt` was accepted but silently routed through the api-key bearer middleware (a security finding masquerading as a config option, removed at the v2.x boundary; see [`upgrade-to-v2-jwt-removal.md`](upgrade-to-v2-jwt-removal.md) if you previously set it). Native OIDC arrived later via Auth Bundle 2 — operators on the pre-Bundle-2 "gateway-only for OIDC" pattern can keep it (it still works) or migrate to native OIDC per [`docs/migration/oidc-enable.md`](../migration/oidc-enable.md).
 ### Concurrency Safety
@@ -153,4 +153,4 @@ The `--wait` flag blocks until the job reaches a terminal state (Completed / Fai
 - [`docs/reference/api.md`](api.md) — the OpenAPI 3.1 spec the CLI wraps
 - [`docs/reference/mcp.md`](mcp.md) — the MCP server that exposes the same surface to AI assistants
- [`docs/contributor/qa-prerequisites.md`](../contributor/qa-prerequisites.md) — local environment setup before the CLI can talk to a server
+- [`docs/getting-started/quickstart.md`](../getting-started/quickstart.md) — local environment setup before the CLI can talk to a server
@@ -80,7 +80,7 @@ For the full deploy contract see
 | Variable | Default | Description |
 |---|---|---|
-| `CERTCTL_AGENT_ID` | (none — required) | The agent's unique ID, issued by `POST /api/v1/agents/register` and bundled into the agent's registration response. Pass via this env var when the agent runs as a systemd unit / container without the `-agent-id` CLI flag. |
+| `CERTCTL_AGENT_ID` | (none — required) | The agent's unique ID, issued by `POST /api/v1/agents` (requires `CERTCTL_AGENT_BOOTSTRAP_TOKEN` when configured) and returned in the registration response body. Pass via this env var when the agent runs as a systemd unit / container without the `-agent-id` CLI flag. The bundled `install-agent.sh` does NOT auto-register — operators pre-register an agent via the REST endpoint (or the dashboard), then pass the returned ID to the script via `--agent-id`. |
 ## Auth (RBAC + OIDC + sessions + break-glass)
--- a/Show More
+++ b/Show More