Merge fix/coverage-N.AB-ci-fix-2: digicert QF1002 4th hit fixed

CI flagged one more QF1002 hit at digicert_failure_test.go:102:5
that I missed in the prior fix (only got the three at 32/51/70).
Same fix: 'switch { case r.URL.Path == "/user/me" }' →
'switch r.URL.Path { case "/user/me" }'.

The remaining switches in this file (lines 126, 149) mix
r.URL.Path == "x" with strings.Contains(r.URL.Path, "..."),
which can't be expressed as tagged switches — staticcheck
correctly does not flag those (same shape as the sectigo
switches that pass clean).

Verification: go test -short -count=1 ./internal/connector/issuer/
digicert/... PASS in 0.6s.

Bundle: N.AB-ci-fix-2

.env.example

@@ -13,22 +13,43 @@ POSTGRES_PASSWORD=change-me-in-production
 # Certctl Server
 # All server vars use the CERTCTL_ prefix (see internal/config/config.go)
 # ==============================================================================
-CERTCTL_DATABASE_URL=postgres://certctl:certctl@postgres:5432/certctl?sslmode=disable
+# IMPORTANT: keep the password segment of CERTCTL_DATABASE_URL in sync with
+# POSTGRES_PASSWORD above. If you deploy via `deploy/docker-compose.yml`,
+# this value is *overridden* by the compose file's
+# `postgres://certctl:${POSTGRES_PASSWORD:-certctl}@postgres:5432/...`
+# interpolation — but if you run the binary directly with this .env loaded
+# (e.g. `set -a; source .env; ./certctl-server`), update *both* lines.
+# Background: editing POSTGRES_PASSWORD after the postgres data directory
+# has been initialized once does NOT rotate the password — initdb only
+# seeds pg_authid on first boot of an empty volume. See docs/quickstart.md
+# "Warning" callout and `internal/repository/postgres/db.go::wrapPingError`
+# for the SQLSTATE 28P01 diagnostic that fires when the two drift.
+CERTCTL_DATABASE_URL=postgres://certctl:change-me-in-production@postgres:5432/certctl?sslmode=disable
 CERTCTL_SERVER_HOST=0.0.0.0
 CERTCTL_SERVER_PORT=8443
 CERTCTL_LOG_LEVEL=info
 CERTCTL_LOG_FORMAT=json
 
-# Auth type: "api-key", "jwt", or "none" (for demo/development)
+# Auth type: "api-key" (production) or "none" (demo/development).
+# For JWT/OIDC, run an authenticating gateway in front of certctl
+# (oauth2-proxy / Envoy ext_authz / Traefik ForwardAuth / Pomerium) and
+# set CERTCTL_AUTH_TYPE=none on the upstream — see
+# docs/architecture.md "Authenticating-gateway pattern". G-1 removed
+# the in-process "jwt" option (no JWT middleware shipped — silent auth
+# downgrade); see docs/upgrade-to-v2-jwt-removal.md if you previously
+# set CERTCTL_AUTH_TYPE=jwt.
 CERTCTL_AUTH_TYPE=none
-# Required when CERTCTL_AUTH_TYPE is "api-key" or "jwt"
+# Required when CERTCTL_AUTH_TYPE is "api-key".
 # Generate with: openssl rand -base64 32
 # CERTCTL_AUTH_SECRET=change-me-in-production
 
 # ==============================================================================
 # Certctl Agent
 # ==============================================================================
-CERTCTL_SERVER_URL=http://localhost:8443
+# HTTPS-only as of v2.2 (TLS 1.3 pinned). Agents reject http:// URLs at
+# startup. Use the docker-compose self-signed bootstrap CA bundle from
+# `deploy/test/certs/ca.crt` or supply your own via CERTCTL_SERVER_CA_BUNDLE_PATH.
+CERTCTL_SERVER_URL=https://localhost:8443
 CERTCTL_API_KEY=change-me-in-production
 CERTCTL_AGENT_NAME=local-agent
 

.github/workflows/release.yml

@@ -7,14 +7,225 @@ on:
 
 env:
   REGISTRY: ghcr.io
+  # Keep in lock-step with .github/workflows/ci.yml (M-3).
+  GO_VERSION: '1.25.9'
+  IMAGE_NAMESPACE: shankar0123
 
 jobs:
-  build-and-push:
+  # ----------------------------------------------------------------------
+  # build-binaries (M-3): matrix build every (binary × OS × arch) tuple.
+  # For each tuple we produce: the binary, a SPDX-JSON SBOM, a keyless
+  # Cosign signature + certificate bundle, and a single-line sha256sum
+  # file. All artefacts are uploaded to a workflow-scoped artifact; the
+  # aggregate-checksums job fans them back in for release upload.
+  # ----------------------------------------------------------------------
+  build-binaries:
+    name: Build ${{ matrix.binary }} (${{ matrix.os }}/${{ matrix.arch }})
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      id-token: write  # Cosign keyless OIDC identity token
+    strategy:
+      fail-fast: false
+      matrix:
+        binary: [agent, server, cli, mcp-server]
+        os: [linux, darwin]
+        arch: [amd64, arm64]
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: ${{ env.GO_VERSION }}
+
+      - name: Extract version from tag
+        id: version
+        run: echo "VERSION=${GITHUB_REF#refs/tags/}" >> "$GITHUB_OUTPUT"
+
+      - name: Install govulncheck
+        # Bundle D / Audit L-008: release.yml previously had no vulnerability
+        # scan, so a release tag could in principle ship a binary with a
+        # known CVE in transitive deps that ci.yml's govulncheck would have
+        # caught on master. Pre-build scan blocks the release if anything
+        # surfaced post-merge. Pinned to the same major as ci.yml.
+        run: go install golang.org/x/vuln/cmd/govulncheck@latest
+
+      - name: Run govulncheck (release gate)
+        # govulncheck distinguishes called-vs-uncalled vulnerable functions.
+        # Default exit code (0 unless an actual call site lands in a vuln
+        # function) is the right gate for release; deferred-call advisories
+        # are tracked separately on master via L-021. If a release-time
+        # scan surfaces a NEW called-vuln, the release is blocked until the
+        # bump lands on master and a new tag is cut.
+        run: govulncheck ./...
+
+      - name: Build binary
+        id: build
+        env:
+          GOOS: ${{ matrix.os }}
+          GOARCH: ${{ matrix.arch }}
+          CGO_ENABLED: '0'
+          VERSION: ${{ steps.version.outputs.VERSION }}
+        run: |
+          set -euo pipefail
+          OUTPUT_NAME="certctl-${{ matrix.binary }}-${{ matrix.os }}-${{ matrix.arch }}"
+          mkdir -p dist
+          go build \
+            -trimpath \
+            -ldflags="-w -s -X main.Version=${VERSION}" \
+            -o "dist/${OUTPUT_NAME}" \
+            "./cmd/${{ matrix.binary }}"
+          ls -lh "dist/${OUTPUT_NAME}"
+          echo "output_name=${OUTPUT_NAME}" >> "$GITHUB_OUTPUT"
+
+      - name: Generate SBOM (SPDX-JSON)
+        uses: anchore/sbom-action@e22c389904149dbc22b58101806040fa8d37a610  # v0.24.0
+        with:
+          file: dist/${{ steps.build.outputs.output_name }}
+          format: spdx-json
+          output-file: dist/${{ steps.build.outputs.output_name }}.sbom.spdx.json
+          upload-artifact: false
+          upload-release-assets: false
+
+      - name: Install Cosign
+        uses: sigstore/cosign-installer@cad07c2e89fa2edd6e2d7bab4c1aa38e53f76003  # v4.1.1
+
+      - name: Keyless-sign binary with Cosign
+        env:
+          OUTPUT_NAME: ${{ steps.build.outputs.output_name }}
+        run: |
+          set -euo pipefail
+          # Cosign v3.0 (shipped by cosign-installer@v4.1.1 default
+          # cosign-release=v3.0.5) removed --output-signature/--output-certificate
+          # on sign-blob. The replacement is --bundle, which emits a unified
+          # Sigstore bundle (signature + cert chain + Rekor inclusion proof) as
+          # a single .sigstore.json artefact. M-11.
+          cosign sign-blob \
+            --yes \
+            --bundle "dist/${OUTPUT_NAME}.sigstore.json" \
+            "dist/${OUTPUT_NAME}"
+
+      - name: Compute SHA-256 sidecar
+        env:
+          OUTPUT_NAME: ${{ steps.build.outputs.output_name }}
+        run: |
+          set -euo pipefail
+          cd dist
+          sha256sum "${OUTPUT_NAME}" > "${OUTPUT_NAME}.sha256"
+          cat "${OUTPUT_NAME}.sha256"
+
+      - name: Upload build artefacts
+        uses: actions/upload-artifact@v4
+        with:
+          name: binary-${{ steps.build.outputs.output_name }}
+          path: |
+            dist/${{ steps.build.outputs.output_name }}
+            dist/${{ steps.build.outputs.output_name }}.sigstore.json
+            dist/${{ steps.build.outputs.output_name }}.sbom.spdx.json
+            dist/${{ steps.build.outputs.output_name }}.sha256
+          if-no-files-found: error
+          retention-days: 7
+
+  # ----------------------------------------------------------------------
+  # aggregate-checksums (M-3): fan in every matrix artefact, produce a
+  # single checksums.txt (sha256sum format, compatible with `sha256sum
+  # -c`), sign it with Cosign, upload everything to the GitHub Release,
+  # and emit a base64-encoded hash manifest for the SLSA generator.
+  # ----------------------------------------------------------------------
+  aggregate-checksums:
+    name: Aggregate checksums & sign
+    runs-on: ubuntu-latest
+    needs: [build-binaries]
+    permissions:
+      contents: write
+      id-token: write  # Cosign keyless OIDC identity token
+    outputs:
+      hashes: ${{ steps.hashes.outputs.hashes }}
+    steps:
+      - name: Download binary artefacts
+        uses: actions/download-artifact@v4
+        with:
+          pattern: binary-*
+          path: artifacts
+          merge-multiple: true
+
+      - name: Aggregate SHA-256 sums
+        id: hashes
+        run: |
+          set -euo pipefail
+          cd artifacts
+          : > checksums.txt
+          for f in certctl-*; do
+            case "$f" in
+              *.sigstore.json|*.sbom.spdx.json|*.sha256|checksums.txt)
+                continue ;;
+            esac
+            sha256sum "$f" >> checksums.txt
+          done
+          echo "=== checksums.txt ==="
+          cat checksums.txt
+          # base64 hashes (single line, no wrapping) for SLSA generator.
+          HASHES=$(base64 -w0 < checksums.txt)
+          echo "hashes=${HASHES}" >> "$GITHUB_OUTPUT"
+
+      - name: Install Cosign
+        uses: sigstore/cosign-installer@cad07c2e89fa2edd6e2d7bab4c1aa38e53f76003  # v4.1.1
+
+      - name: Keyless-sign checksums.txt
+        run: |
+          set -euo pipefail
+          cd artifacts
+          # Cosign v3.0 --bundle replaces the removed v2 flag pair
+          # --output-signature / --output-certificate. See M-11.
+          cosign sign-blob \
+            --yes \
+            --bundle checksums.txt.sigstore.json \
+            checksums.txt
+
+      - name: Upload artefacts to GitHub Release
+        uses: softprops/action-gh-release@v2
+        if: startsWith(github.ref, 'refs/tags/')
+        with:
+          files: |
+            artifacts/certctl-*
+            artifacts/checksums.txt
+            artifacts/checksums.txt.sigstore.json
+
+  # ----------------------------------------------------------------------
+  # provenance-binaries (M-3): SLSA Level 3 provenance for every binary.
+  # The SLSA generic generator reusable workflow runs in a hermetic
+  # workflow run, producing multiple.intoto.jsonl from the base64 hash
+  # manifest and uploading it as a release asset.
+  # ----------------------------------------------------------------------
+  provenance-binaries:
+    name: SLSA provenance (binaries)
+    needs: [aggregate-checksums]
+    permissions:
+      actions: read
+      id-token: write
+      contents: write
+    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v2.1.0
+    with:
+      base64-subjects: "${{ needs.aggregate-checksums.outputs.hashes }}"
+      upload-assets: true
+      provenance-name: multiple.intoto.jsonl
+
+  # ----------------------------------------------------------------------
+  # build-and-push-docker: push container images to GHCR with native
+  # SLSA L3 provenance (mode=max) and SBOM attestations emitted by
+  # docker/build-push-action@v6, plus a keyless Cosign signature on the
+  # image digest for identity-bound verification. The M-4 proxy-propagation
+  # build-args block is retained verbatim — M-3 only adds supply-chain
+  # steps; it never touches M-4 wiring.
+  # ----------------------------------------------------------------------
+  build-and-push-docker:
     name: Build & Push Docker Images
     runs-on: ubuntu-latest
     permissions:
       contents: write
       packages: write
+      id-token: write  # Cosign keyless OIDC identity token
 
     steps:
       - uses: actions/checkout@v4
@@ -28,48 +239,146 @@ jobs:
 
       - name: Extract version from tag
         id: version
-        run: echo "VERSION=${GITHUB_REF#refs/tags/}" >> $GITHUB_OUTPUT
+        run: echo "VERSION=${GITHUB_REF#refs/tags/}" >> "$GITHUB_OUTPUT"
 
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
 
+      - name: Install Cosign
+        uses: sigstore/cosign-installer@cad07c2e89fa2edd6e2d7bab4c1aa38e53f76003  # v4.1.1
+
       - name: Build and push server image
+        id: server-push
         uses: docker/build-push-action@v6
         with:
           context: .
           file: ./Dockerfile
           push: true
           tags: |
-            ${{ env.REGISTRY }}/shankar0123/certctl-server:${{ steps.version.outputs.VERSION }}
-            ${{ env.REGISTRY }}/shankar0123/certctl-server:latest
+            ${{ env.REGISTRY }}/${{ env.IMAGE_NAMESPACE }}/certctl-server:${{ steps.version.outputs.VERSION }}
+            ${{ env.REGISTRY }}/${{ env.IMAGE_NAMESPACE }}/certctl-server:latest
+          # Proxy propagation (M-4, Issue #9) — forwards runner-level proxy
+          # secrets into the Docker build so self-hosted runners behind
+          # corporate proxies can reach public registries. GitHub-hosted
+          # runners don't need proxies, so the secrets are optional and
+          # resolve to empty strings when unset — byte-identical to the
+          # pre-fix behaviour for the public-runner path.
+          build-args: |
+            HTTP_PROXY=${{ secrets.HTTP_PROXY }}
+            HTTPS_PROXY=${{ secrets.HTTPS_PROXY }}
+            NO_PROXY=${{ secrets.NO_PROXY }}
+          # Supply-chain hardening (M-3): emit native SLSA L3 provenance
+          # and SBOM attestations bound to the image manifest.
+          provenance: mode=max
+          sbom: true
           cache-from: type=gha
           cache-to: type=gha,mode=max
 
+      - name: Keyless-sign server image with Cosign
+        env:
+          DIGEST: ${{ steps.server-push.outputs.digest }}
+          IMAGE: ${{ env.REGISTRY }}/${{ env.IMAGE_NAMESPACE }}/certctl-server
+        run: |
+          set -euo pipefail
+          cosign sign --yes "${IMAGE}@${DIGEST}"
+
       - name: Build and push agent image
+        id: agent-push
         uses: docker/build-push-action@v6
         with:
           context: .
           file: ./Dockerfile.agent
           push: true
           tags: |
-            ${{ env.REGISTRY }}/shankar0123/certctl-agent:${{ steps.version.outputs.VERSION }}
-            ${{ env.REGISTRY }}/shankar0123/certctl-agent:latest
+            ${{ env.REGISTRY }}/${{ env.IMAGE_NAMESPACE }}/certctl-agent:${{ steps.version.outputs.VERSION }}
+            ${{ env.REGISTRY }}/${{ env.IMAGE_NAMESPACE }}/certctl-agent:latest
+          # Proxy propagation (M-4, Issue #9) — see server-image step for
+          # rationale. Empty secrets resolve to empty build args, leaving
+          # the un-proxied code path byte-identical to the pre-fix tree.
+          build-args: |
+            HTTP_PROXY=${{ secrets.HTTP_PROXY }}
+            HTTPS_PROXY=${{ secrets.HTTPS_PROXY }}
+            NO_PROXY=${{ secrets.NO_PROXY }}
+          # Supply-chain hardening (M-3): emit native SLSA L3 provenance
+          # and SBOM attestations bound to the image manifest.
+          provenance: mode=max
+          sbom: true
           cache-from: type=gha
           cache-to: type=gha,mode=max
 
-      - name: Create GitHub Release
+      - name: Keyless-sign agent image with Cosign
+        env:
+          DIGEST: ${{ steps.agent-push.outputs.digest }}
+          IMAGE: ${{ env.REGISTRY }}/${{ env.IMAGE_NAMESPACE }}/certctl-agent
+        run: |
+          set -euo pipefail
+          cosign sign --yes "${IMAGE}@${DIGEST}"
+
+  # ----------------------------------------------------------------------
+  # create-release: stamp the release body. The actual asset uploads are
+  # handled by aggregate-checksums (binaries, SBOMs, sigs, certs,
+  # checksums.txt + signature) and the SLSA generator (multiple.intoto.jsonl).
+  # ----------------------------------------------------------------------
+  create-release:
+    name: Create Release Notes
+    runs-on: ubuntu-latest
+    needs: [build-binaries, aggregate-checksums, provenance-binaries, build-and-push-docker]
+    permissions:
+      contents: write
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Extract version from tag
+        id: version
+        run: echo "VERSION=${GITHUB_REF#refs/tags/}" >> "$GITHUB_OUTPUT"
+
+      - name: Create release with notes
         uses: softprops/action-gh-release@v2
         with:
           generate_release_notes: true
           body: |
-            ## Docker Images
+            ## Installation
+
+            ### Quick Install (Linux/macOS)
 
             ```bash
-            docker pull shankar0123.docker.scarf.sh/certctl-server:${{ steps.version.outputs.VERSION }}
-            docker pull shankar0123.docker.scarf.sh/certctl-agent:${{ steps.version.outputs.VERSION }}
+            curl -sSL https://raw.githubusercontent.com/shankar0123/certctl/master/install-agent.sh | bash
             ```
 
-            ## Quick Start
+            ### Manual Binary Download
+
+            Download the appropriate binary for your OS and architecture:
+
+            - **Linux x86_64**: `certctl-agent-linux-amd64`
+            - **Linux ARM64**: `certctl-agent-linux-arm64`
+            - **macOS x86_64**: `certctl-agent-darwin-amd64`
+            - **macOS ARM64 (Apple Silicon)**: `certctl-agent-darwin-arm64`
+
+            Then make it executable and start the service:
+
+            ```bash
+            chmod +x certctl-agent-linux-amd64
+            sudo mv certctl-agent-linux-amd64 /usr/local/bin/certctl-agent
+            ```
+
+            ## Docker Images
+
+            Pull pre-built Docker images for server and agent:
+
+            ```bash
+            docker pull ghcr.io/shankar0123/certctl-server:${{ steps.version.outputs.VERSION }}
+            docker pull ghcr.io/shankar0123/certctl-agent:${{ steps.version.outputs.VERSION }}
+            ```
+
+            Or use the latest tag:
+
+            ```bash
+            docker pull ghcr.io/shankar0123/certctl-server:latest
+            docker pull ghcr.io/shankar0123/certctl-agent:latest
+            ```
+
+            ## Docker Compose Quick Start
 
             ```bash
             git clone https://github.com/shankar0123/certctl.git
@@ -77,3 +386,92 @@ jobs:
             cp deploy/.env.example deploy/.env
             docker compose -f deploy/docker-compose.yml up -d
             ```
+
+            ## Server Binaries
+
+            Pre-compiled server binaries are also available for direct installation:
+
+            - **Linux x86_64**: `certctl-server-linux-amd64`
+            - **Linux ARM64**: `certctl-server-linux-arm64`
+            - **macOS x86_64**: `certctl-server-darwin-amd64`
+            - **macOS ARM64 (Apple Silicon)**: `certctl-server-darwin-arm64`
+
+            ## CLI & MCP Server Binaries
+
+            The `certctl-cli` (REST API wrapper) and `certctl-mcp-server` (Model Context
+            Protocol bridge) binaries ship for all four platforms as well:
+
+            - `certctl-cli-{linux,darwin}-{amd64,arm64}`
+            - `certctl-mcp-server-{linux,darwin}-{amd64,arm64}`
+
+            ## Verifying this release
+
+            Every binary, `checksums.txt`, and container image is signed with Cosign
+            keyless OIDC. Each binary ships with a SPDX-JSON SBOM. Binaries are covered
+            by SLSA Level 3 provenance; container images carry native SLSA L3 provenance
+            and SBOM attestations (docker/build-push-action `provenance: mode=max`,
+            `sbom: true`) in addition to a Cosign signature on the digest.
+
+            **1. Verify SHA-256 checksums:**
+
+            ```bash
+            sha256sum -c checksums.txt
+            ```
+
+            **2. Verify the Cosign signature on checksums.txt (keyless OIDC):**
+
+            ```bash
+            cosign verify-blob \
+              --bundle checksums.txt.sigstore.json \
+              --certificate-identity-regexp '^https://github\.com/shankar0123/certctl/\.github/workflows/release\.yml@refs/tags/' \
+              --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
+              checksums.txt
+            ```
+
+            Replace `checksums.txt` with any individual binary name to verify that
+            artefact directly (each binary ships with its own `.sigstore.json`
+            bundle, e.g. `cosign verify-blob --bundle certctl-agent-linux-amd64.sigstore.json …`).
+
+            **3. Verify SLSA Level 3 provenance (binaries):**
+
+            ```bash
+            slsa-verifier verify-artifact \
+              --provenance-path multiple.intoto.jsonl \
+              --source-uri github.com/shankar0123/certctl \
+              --source-tag ${{ steps.version.outputs.VERSION }} \
+              certctl-agent-linux-amd64
+            ```
+
+            **4. Verify container image signature and attestations:**
+
+            ```bash
+            IMAGE=ghcr.io/shankar0123/certctl-server:${{ steps.version.outputs.VERSION }}
+            cosign verify \
+              --certificate-identity-regexp '^https://github\.com/shankar0123/certctl/\.github/workflows/release\.yml@refs/tags/' \
+              --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
+              "$IMAGE"
+
+            # SBOM attestation (SPDX-JSON) emitted by docker/build-push-action
+            cosign verify-attestation --type spdxjson \
+              --certificate-identity-regexp '^https://github\.com/shankar0123/certctl/' \
+              --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
+              "$IMAGE"
+
+            # SLSA provenance attestation (mode=max)
+            cosign verify-attestation --type slsaprovenance \
+              --certificate-identity-regexp '^https://github\.com/shankar0123/certctl/' \
+              --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
+              "$IMAGE"
+            ```
+
+            ## Helm Chart
+
+            Deploy certctl to Kubernetes using Helm:
+
+            ```bash
+            helm repo add certctl https://github.com/shankar0123/certctl/tree/master/deploy/helm
+            helm repo update
+            helm install certctl certctl/certctl
+            ```
+
+            See `deploy/helm/certctl/` for values customization.

.github/workflows/security-deep-scan.yml

@@ -0,0 +1,194 @@
+name: security-deep-scan
+
+# Bundle-7 / Audit D-001..D-007:
+# Slow / containerized scans on a daily schedule + manual dispatch.
+# Per-PR fast gates live in ci.yml; this workflow runs the heavyweight
+# tools that need docker, network egress to scanner registries, or
+# longer wall-clock budgets than a per-PR check tolerates.
+#
+# Scope:
+#   trivy image          container CVE + secret scan
+#   syft SBOM            CycloneDX SBOM artefact upload
+#   ZAP baseline         DAST baseline against a live deploy_test stack (D-004)
+#   nuclei               template-based vuln scan against the same stack
+#   schemathesis         OpenAPI fuzz against the running server
+#   testssl.sh           TLS configuration audit (D-005)
+#   race detector x10    full -count=10 race run on the entire test suite (D-002)
+#   gosec                Go security static analysis (slow first run)
+#   go-mutesting         mutation testing on crypto cluster (D-003)
+#   semgrep p/react-security  frontend XSS / dangerouslySetInnerHTML / target=_blank ruleset (D-007)
+#
+# Each step is best-effort — failures are uploaded as artefacts but do
+# NOT block the workflow. Triage happens via the Bundle-7 receipt
+# directory under cowork/comprehensive-audit-2026-04-25/tool-output/.
+
+on:
+  schedule:
+    - cron: '0 6 * * *'   # daily 06:00 UTC
+  workflow_dispatch: {}
+
+permissions:
+  contents: read
+  security-events: write   # SARIF upload to GitHub code scanning
+
+jobs:
+  deep-scan:
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.25'
+
+      - name: Install Go-based tools
+        run: bash scripts/install-security-tools.sh
+        continue-on-error: true
+
+      # --- Static analysis (slow paths) ---
+
+      - name: gosec
+        run: |
+          $(go env GOPATH)/bin/gosec -fmt sarif -out gosec.sarif ./... || true
+        continue-on-error: true
+
+      - name: osv-scanner (multi-ecosystem CVE)
+        run: |
+          $(go env GOPATH)/bin/osv-scanner -r --format json --output osv-scanner.json . || true
+        continue-on-error: true
+
+      # --- Race detector at -count=10 (D-002) ---
+
+      - name: go test -race -count=10 (full suite)
+        run: |
+          go test -race -count=10 -short ./... 2>&1 | tee go-test-race.txt
+        continue-on-error: true
+
+      # --- Coverage receipts for crypto cluster (H-005) ---
+
+      - name: go test -cover (crypto cluster)
+        run: |
+          go test -cover -covermode=atomic \
+            ./internal/crypto/... \
+            ./internal/pkcs7/... \
+            ./internal/connector/issuer/local/... \
+            2>&1 | tee go-test-cover.txt
+
+      # --- Mutation testing on crypto cluster (D-003) ---
+      #
+      # Operator runbook: docs/testing-strategy.md::Mutation testing.
+      # Tool: go-mutesting (https://github.com/zimmski/go-mutesting). Each
+      # package is mutated independently; the per-package summary line
+      # (`The mutation score is X.YZ`) is grep-extracted into the receipt.
+      # Acceptance threshold: ≥80% kill ratio per package; surviving
+      # mutants get triaged in cowork/comprehensive-audit-2026-04-25/
+      # d003-mutation-results.md (per-mutant action item or
+      # equivalent-mutation justification).
+
+      - name: Install go-mutesting
+        run: go install github.com/zimmski/go-mutesting/cmd/go-mutesting@latest
+        continue-on-error: true
+
+      - name: go-mutesting (crypto cluster)
+        run: |
+          : > go-mutesting.txt
+          for pkg in ./internal/crypto/... ./internal/pkcs7/... ./internal/connector/issuer/local/...; do
+            echo "=== $pkg ===" | tee -a go-mutesting.txt
+            $(go env GOPATH)/bin/go-mutesting "$pkg" 2>&1 | tee -a go-mutesting.txt || true
+          done
+        continue-on-error: true
+
+      # --- Container + supply chain (D-001 partial, D-006 partial) ---
+
+      - name: Build certctl image
+        run: docker build -t certctl:deep-scan .
+        continue-on-error: true
+
+      - name: trivy image scan
+        run: |
+          docker run --rm -v "$PWD":/src aquasec/trivy:latest image \
+            --format json --output /src/trivy.json certctl:deep-scan || true
+        continue-on-error: true
+
+      - name: syft SBOM
+        run: |
+          docker run --rm -v "$PWD":/src anchore/syft:latest dir:/src \
+            -o cyclonedx-json > syft.cyclonedx.json || true
+        continue-on-error: true
+
+      # --- DAST against a live stack (D-004) ---
+
+      - name: docker compose up (test stack)
+        run: |
+          docker compose -f deploy/docker-compose.yml up -d
+          sleep 20
+        continue-on-error: true
+
+      - name: ZAP baseline
+        uses: zaproxy/action-baseline@v0.10.0
+        with:
+          target: 'https://localhost:8443'
+        continue-on-error: true
+
+      - name: schemathesis (OpenAPI fuzz)
+        run: |
+          pip install schemathesis
+          schemathesis run --base-url https://localhost:8443 \
+            --hypothesis-max-examples=50 api/openapi.yaml || true
+        continue-on-error: true
+
+      - name: nuclei
+        run: |
+          docker run --rm --network host projectdiscovery/nuclei:latest \
+            -u https://localhost:8443 -j -o nuclei.json || true
+        continue-on-error: true
+
+      # --- TLS audit (D-005) ---
+
+      - name: testssl.sh
+        run: |
+          docker run --rm -v "$PWD":/data drwetter/testssl.sh:latest \
+            --jsonfile /data/testssl.json https://localhost:8443 || true
+        continue-on-error: true
+
+      - name: docker compose down
+        run: docker compose -f deploy/docker-compose.yml down || true
+        if: always()
+
+      # --- Frontend XSS / unsafe-link ruleset (D-007) ---
+      #
+      # Operator runbook: docs/testing-strategy.md::Frontend semgrep.
+      # Bundle 8 already verified `dangerouslySetInnerHTML` count at
+      # zero and the `target="_blank"` rel-noopener pin via grep
+      # guards in ci.yml — semgrep p/react-security adds defence in
+      # depth (it catches escape patterns the grep guards don't see,
+      # e.g., href={user_input}, eval, document.write).
+
+      - name: semgrep p/react-security (frontend)
+        run: |
+          docker run --rm -v "$PWD":/src returntocorp/semgrep:latest \
+            semgrep --config=p/react-security --json /src/web/src \
+            > semgrep-react.json 2>semgrep-react.stderr || true
+        continue-on-error: true
+
+      # --- Upload everything as artefacts ---
+
+      - name: Upload deep-scan receipts
+        uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: security-deep-scan-${{ github.run_id }}
+          path: |
+            gosec.sarif
+            osv-scanner.json
+            go-test-race.txt
+            go-test-cover.txt
+            go-mutesting.txt
+            trivy.json
+            syft.cyclonedx.json
+            nuclei.json
+            testssl.json
+            semgrep-react.json
+            semgrep-react.stderr
+          retention-days: 30

.gitignore

@@ -43,6 +43,11 @@ vendor/
 tmp/
 temp/
 *.log
+*.bak
+
+# Private keys (agent-generated, never commit)
+cmd/agent/*.key
+cmd/agent/*.pem
 
 # Database
 *.db
@@ -57,12 +62,29 @@ certctl-agent
 certctl-cli
 /server
 /agent
+/cli
+/mcp-server
 
 # Private strategy docs
-roadmap.md
 SECURITY_REMEDIATION.md
 
 # OS
 .DS_Store
 Thumbs.db
-mcp-server
+
+# Local Go build/module caches (session-scoped, never committed)
+/.gocache/
+/.gomodcache/
+/.gopath/
+/.gomodcache-gopath/
+
+# Design scratch files (session-scoped)
+/.i004-design.md
+/.i005-design.md
+
+# HTTPS-Everywhere (M-007) Phase 6: the docker-compose.test.yml tls-init
+# container writes ca.crt / server.crt / server.key into this directory so
+# the host-side integration_test.go binary can pin the CA via
+# CERTCTL_TEST_CA_BUNDLE=./certs/ca.crt. Material is regenerated on every
+# `docker compose up` and never belongs in git.
+/deploy/test/certs/

.golangci.yml

@@ -6,6 +6,7 @@ run:
 linters:
   default: none
   enable:
+    - contextcheck
     - govet
     - staticcheck
     - unused

.govulnignore

@@ -0,0 +1,21 @@
+# Bundle-7 / Audit D-001 / govulncheck suppressions.
+#
+# Format: one OSV ID per line, with a comment justifying the suppression.
+# Every entry needs:
+#   - the OSV ID (GO-YYYY-NNNN)
+#   - one-line "what is it"
+#   - one-line "why we're not affected" (must reference call-graph evidence)
+#   - "review-by" date (YYYY-MM-DD) — re-triage on/after this date
+#
+# Triage rule: only suppress an advisory if `govulncheck ./...` (NOT
+# verbose) reports it as a deferred-call vulnerability ("packages you
+# import" or "modules you require", not "Your code is affected by").
+#
+# At Bundle-7 time (2026-04-26): the 5 advisories surfaced are all in
+# transitive deps and govulncheck confirms our code does not call them.
+# Documented here for tracking; no entries needed because the default
+# fail-on-non-zero gate already passes (govulncheck distinguishes
+# called vs uncalled and only exits non-zero when the latter calls in).
+#
+# Example (do not enable unless the advisory becomes call-affected):
+# GO-2026-4441  # transitive: golang.org/x/crypto pre-v0.40 — net/ssh terrapin downgrade; we don't use net/ssh; review 2026-07-01

Dockerfile

@@ -1,18 +1,80 @@
 # Multi-stage build for certctl server
+#
+# Bundle A / Audit H-001 (CWE-829): every FROM line is pinned to an
+# immutable digest in addition to the human-readable tag. The tag is
+# advisory; the digest is what Docker actually pulls. A registry-side
+# tag swap (the documented prior-art for tag-only pulls being unsafe)
+# can no longer change the build.
+#
+# Bump procedure (operator):
+#   1. Quarterly cadence (or sooner if a CVE lands on a base image).
+#   2. For each FROM:
+#        docker pull <image>:<tag>
+#        docker manifest inspect <image>:<tag> | grep -m1 digest
+#      OR via Docker Hub Registry API:
+#        curl -sSL https://hub.docker.com/v2/repositories/library/<image>/tags/<tag> \
+#          | jq -r .digest
+#   3. Replace the @sha256:... portion of the FROM line.
+#   4. Run `docker build` locally + verify CI.
+#   5. Commit with the bump procedure cited in the message body.
+#
+# The CI step "Forbidden bare FROM regression guard (H-001)" rejects
+# any future commit that lands a FROM without an @sha256 pin.
 
 # Stage 1: Build frontend
-FROM node:20-alpine AS frontend
+FROM node:20-alpine@sha256:fb4cd12c85ee03686f6af5362a0b0d56d50c58a04632e6c0fb8363f609372293 AS frontend
+
+# Proxy propagation (M-4, Issue #9) — defaulted to empty so un-proxied builds
+# behave identically to the pre-fix tree. When `HTTP_PROXY`/`HTTPS_PROXY`/
+# `NO_PROXY` are forwarded via `docker build --build-arg` (or compose
+# `build.args`), they are re-exported as ENV with both upper- and lower-case
+# names because npm/apk/curl read the lowercase variants while Go, Node, and
+# most HTTP libraries read the uppercase ones.
+ARG HTTP_PROXY=
+ARG HTTPS_PROXY=
+ARG NO_PROXY=
+ENV HTTP_PROXY=${HTTP_PROXY} \
+    HTTPS_PROXY=${HTTPS_PROXY} \
+    NO_PROXY=${NO_PROXY} \
+    http_proxy=${HTTP_PROXY} \
+    https_proxy=${HTTPS_PROXY} \
+    no_proxy=${NO_PROXY}
 
 WORKDIR /app/web
 
-COPY web/package.json web/package-lock.json ./
-RUN npm ci
-
 COPY web/ .
-RUN npm run build
+# Bundle A / Audit M-014: explicit retry loop for `npm ci`. Pre-bundle
+# this was `npm ci || npm ci && tsc && build` — the bash precedence is
+# `A || (B && C && D)` so the second `npm ci` only ran on the failure
+# path of the first, but the `tsc && build` chain only ran on the
+# success path of the second. Net effect: a transient registry blip
+# turned the build into a silent skip of the production step.
+#
+# New shape: a deterministic 3-attempt retry with 5-second backoff and
+# an explicit `[ -d node_modules ]` post-check so a silent failure is
+# impossible.
+RUN for i in 1 2 3; do \
+        npm ci --include=dev && break; \
+        echo "npm ci attempt $i failed; sleeping 5s before retry"; \
+        sleep 5; \
+    done && \
+    [ -d node_modules ] || (echo "ERROR: npm ci failed after 3 attempts; node_modules missing" && exit 1) && \
+    node_modules/.bin/tsc --version && \
+    npm run build
 
 # Stage 2: Build Go binary
-FROM golang:1.25-alpine AS builder
+FROM golang:1.25-alpine@sha256:5caaf1cca9dc351e13deafbc3879fd4754801acba8653fa9540cea125d01a71f AS builder
+
+# Proxy propagation (M-4, Issue #9) — see Stage 1 rationale.
+ARG HTTP_PROXY=
+ARG HTTPS_PROXY=
+ARG NO_PROXY=
+ENV HTTP_PROXY=${HTTP_PROXY} \
+    HTTPS_PROXY=${HTTPS_PROXY} \
+    NO_PROXY=${NO_PROXY} \
+    http_proxy=${HTTP_PROXY} \
+    https_proxy=${HTTPS_PROXY} \
+    no_proxy=${NO_PROXY}
 
 RUN apk add --no-cache git ca-certificates tzdata
 
@@ -31,7 +93,7 @@ RUN CGO_ENABLED=0 GOOS=linux GOARCH=${TARGETARCH} go build \
     ./cmd/server
 
 # Stage 3: Runtime
-FROM alpine:3.19
+FROM alpine:3.19@sha256:6baf43584bcb78f2e5847d1de515f23499913ac9f12bdf834811a3145eb11ca1
 
 RUN apk add --no-cache ca-certificates tzdata curl
 
@@ -50,7 +112,34 @@ USER certctl
 
 EXPOSE 8443
 
+# Image-level HEALTHCHECK for bare `docker run` / Docker Swarm / Nomad / ECS.
+#
+# U-2 (P1, cat-u-healthcheck_protocol_mismatch): pre-U-2 this probe used
+# `curl -f http://localhost:8443/health`, which always failed against the
+# HTTPS-only listener (HTTPS-Everywhere milestone, v2.2 / tag v2.0.47 —
+# `cmd/server/main.go::ListenAndServeTLS`, no plaintext fallback, TLS 1.3
+# pinned). Operators outside docker-compose / Helm saw permanent
+# `unhealthy` status and a restart-loop the first time they pulled the
+# image. The compose stack overrides this HEALTHCHECK with `--cacert` to
+# the bootstrap CA bundle (deploy/docker-compose.yml:126); the Helm chart
+# uses explicit `httpGet` probes with `scheme: HTTPS` and ignores Docker's
+# HEALTHCHECK; every example compose file in `examples/*/docker-compose.yml`
+# overrides with `curl -sfk https://localhost:8443/health`. This image-
+# level probe is for the bare-`docker run` consumer ONLY.
+#
+# `-k` (insecure) is acceptable here because the probe is localhost-to-
+# localhost: the same process serving the cert is being probed; the probe
+# never traverses a network. Pinning a `--cacert` is not viable for the
+# published image because the bootstrap cert is per-deploy (generated into
+# the `certs` named volume on first up; operator-supplied via Helm's
+# `existingSecret` or cert-manager). Compose / Helm / examples already
+# perform full cert-chain validation and are unaffected.
+#
+# CI grep guardrail at .github/workflows/ci.yml ("Forbidden plaintext
+# HEALTHCHECK regression guard (U-2)") blocks reintroduction of the
+# `http://` shape. Image-level integration test in
+# deploy/test/healthcheck_test.go pins the contract end-to-end.
 HEALTHCHECK --interval=10s --timeout=5s --start-period=5s --retries=5 \
-    CMD curl -f http://localhost:8443/health || exit 1
+    CMD curl -fsk https://localhost:8443/health || exit 1
 
 ENTRYPOINT ["/app/server"]

Dockerfile.agent

@@ -1,6 +1,27 @@
 # Multi-stage build for certctl agent
+#
+# Bundle A / Audit H-001 (CWE-829): every FROM line is pinned to an
+# immutable digest. See Dockerfile (server) for the bump-procedure
+# operator runbook; the pins here MUST be bumped in the same pass.
+
 # Stage 1: Build
-FROM golang:1.25-alpine AS builder
+FROM golang:1.25-alpine@sha256:5caaf1cca9dc351e13deafbc3879fd4754801acba8653fa9540cea125d01a71f AS builder
+
+# Proxy propagation (M-4, Issue #9) — defaulted to empty so un-proxied builds
+# behave identically to the pre-fix tree. When `HTTP_PROXY`/`HTTPS_PROXY`/
+# `NO_PROXY` are forwarded via `docker build --build-arg` (or compose
+# `build.args`), they are re-exported as ENV with both upper- and lower-case
+# names because apk and curl read the lowercase variants while Go reads the
+# uppercase ones.
+ARG HTTP_PROXY=
+ARG HTTPS_PROXY=
+ARG NO_PROXY=
+ENV HTTP_PROXY=${HTTP_PROXY} \
+    HTTPS_PROXY=${HTTPS_PROXY} \
+    NO_PROXY=${NO_PROXY} \
+    http_proxy=${HTTP_PROXY} \
+    https_proxy=${HTTPS_PROXY} \
+    no_proxy=${NO_PROXY}
 
 RUN apk add --no-cache git ca-certificates
 
@@ -18,9 +39,16 @@ RUN CGO_ENABLED=0 GOOS=linux GOARCH=${TARGETARCH} go build \
     ./cmd/agent
 
 # Stage 2: Runtime
-FROM alpine:3.19
+FROM alpine:3.19@sha256:6baf43584bcb78f2e5847d1de515f23499913ac9f12bdf834811a3145eb11ca1
 
-RUN apk add --no-cache ca-certificates curl
+# U-2: `procps` ships pgrep, which the HEALTHCHECK below uses to verify the
+# agent process is alive. Pre-U-2 the deploy/docker-compose.yml agent
+# HEALTHCHECK called `pgrep -f certctl-agent` against this image but
+# pgrep wasn't installed — the compose probe was a latent always-fail.
+# Adding procps here fixes both the new image-level HEALTHCHECK and the
+# pre-existing compose override. Adds ~250KB to the image; acceptable for
+# observability parity with the server image.
+RUN apk add --no-cache ca-certificates curl procps
 
 RUN addgroup -g 1000 certctl && \
     adduser -D -u 1000 -G certctl certctl
@@ -35,4 +63,19 @@ RUN mkdir -p /var/lib/certctl/keys && \
 
 USER certctl
 
+# Image-level HEALTHCHECK for bare `docker run` / Docker Swarm / Nomad / ECS.
+#
+# U-2 (P1, cat-u-healthcheck_protocol_mismatch — adjacent fix): the agent
+# has no HTTP listener (it polls the server via outbound HTTPS), so a
+# process-presence check is the correct primitive. Pre-U-2 the agent image
+# shipped with no HEALTHCHECK at all, so bare-`docker run` operators got
+# zero health signal and orchestrators that key off Docker's HEALTHCHECK
+# (Swarm, Nomad, ECS) saw the container reported as `none`. The compose
+# override at deploy/docker-compose.yml:173 used the same `pgrep -f
+# certctl-agent` shape; we mirror it here so the published image has
+# parity with the compose stack and the override on docker-compose.yml
+# becomes redundant-but-correct rather than load-bearing.
+HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
+    CMD pgrep -f certctl-agent > /dev/null || exit 1
+
 ENTRYPOINT ["/app/agent"]

LICENSE

@@ -6,20 +6,27 @@ Licensor:             Shankar Reddy
 Licensed Work:        certctl
                       The Licensed Work is (c) 2026 Shankar Reddy.
 Additional Use Grant: You may make use of the Licensed Work, provided that
-                      you may not use the Licensed Work for a Certificate
-                      Management Service. A "Certificate Management Service"
-                      is a commercial offering that allows third parties
-                      (other than your employees and contractors acting on
-                      your behalf) to access and/or use the Licensed Work's
-                      certificate lifecycle management functionality as part
-                      of a hosted or managed service.
+                      you may not use the Licensed Work for a Commercial
+                      Certificate Service. A "Commercial Certificate Service"
+                      is any product, service, or offering in which a third
+                      party (other than your employees and contractors
+                      acting on your behalf) accesses, uses, or benefits
+                      from the Licensed Work's certificate management
+                      functionality — including but not limited to lifecycle
+                      management, discovery, monitoring, alerting, renewal
+                      automation, deployment, and revocation — as part of
+                      or in connection with an offering for which
+                      compensation is received. This restriction applies
+                      regardless of whether the Licensed Work is hosted,
+                      managed, embedded, bundled, or integrated with
+                      another product or service.
 
-Change Date:          March 14, 2033
+Change Date:          March 14, 2126
 
 Change License:       Apache License, Version 2.0
 
 For information about alternative licensing arrangements for the Licensed Work,
-please contact: skreddy040@gmail.com
+please contact: certctl@proton.me
 
 Notice
 

Makefile

@@ -1,4 +1,4 @@
-.PHONY: help build run test lint clean docker-up docker-down migrate-up migrate-down generate test-cover frontend-build
+.PHONY: help build run test lint verify clean docker-up docker-down migrate-up migrate-down generate test-cover frontend-build qa-stats
 
 # Default target - show help
 help:
@@ -15,6 +15,7 @@ help:
 	@echo "  make test-verbose   Run tests with verbose output"
 	@echo "  make lint           Run linter (golangci-lint)"
 	@echo "  make fmt            Format code with gofmt"
+	@echo "  make verify         Pre-commit gate: fmt + vet + lint + test (CI-parity)"
 	@echo ""
 	@echo "Database:"
 	@echo "  make migrate-up     Run migrations (requires DB_URL)"
@@ -97,6 +98,24 @@ vet:
 	@echo "Running go vet..."
 	go vet ./...
 
+# verify: aggregate pre-commit gate. Mirrors what CI enforces, so
+# running `make verify` locally before committing prevents the
+# class of breakages that ship green-locally / red-on-CI (e.g.
+# Bundle-9's ST1018 invisible-Unicode-literal hits, which `go vet`
+# alone cannot catch — staticcheck under golangci-lint does).
+verify:
+	@echo "==> fmt"
+	@go fmt ./... | { ! grep -q '.'; } || (echo "gofmt produced changes — commit them" && exit 1)
+	@echo "==> go vet ./..."
+	@go vet ./...
+	@echo "==> golangci-lint run ./... (incl. staticcheck ST*)"
+	@which golangci-lint > /dev/null || (echo "Installing golangci-lint..." && go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest)
+	@golangci-lint run ./... --timeout 5m
+	@echo "==> go test -short ./..."
+	@go test -short -count=1 ./...
+	@echo ""
+	@echo "verify: PASS — safe to commit"
+
 # Database targets (requires migrate tool)
 migrate-up:
 	@echo "Running migrations..."
@@ -162,6 +181,29 @@ frontend-build:
 	cd web && npm ci && npx vite build
 	@echo "Frontend build complete"
 
+# QA Suite Stats — Bundle P / Strengthening #8.
+# Single source-of-truth for every count claim in docs/qa-test-guide.md +
+# docs/testing-guide.md. The Strengthening #6 CI drift guards consume the
+# same numbers, eliminating the doc-drift class structurally.
+qa-stats:
+	@echo "=== certctl QA Suite Stats ==="
+	@echo "Date: $$(date +%Y-%m-%d)"
+	@echo "HEAD: $$(git rev-parse HEAD 2>/dev/null || echo 'not-a-git-repo')"
+	@echo ""
+	@echo "Backend test files: $$(find . -name '*_test.go' -not -path './web/*' 2>/dev/null | wc -l | tr -d ' ')"
+	@echo "Backend Test functions: $$(find . -name '*_test.go' -not -path './web/*' 2>/dev/null | xargs grep -c '^func Test' 2>/dev/null | awk -F: '{s+=$$2} END{print s+0}')"
+	@echo "Backend t.Run subtests: $$(find . -name '*_test.go' -not -path './web/*' 2>/dev/null | xargs grep -c 't\.Run(' 2>/dev/null | awk -F: '{s+=$$2} END{print s+0}')"
+	@echo "Frontend test files: $$(find web/src -name '*.test.ts' -o -name '*.test.tsx' 2>/dev/null | wc -l | tr -d ' ')"
+	@echo "Fuzz targets: $$(grep -rE 'func Fuzz[A-Z]' --include='*_test.go' . 2>/dev/null | wc -l | tr -d ' ')"
+	@echo "t.Skip sites: $$(grep -rE 't\.Skip(Now|f)?\(' --include='*_test.go' . 2>/dev/null | wc -l | tr -d ' ')"
+	@echo "qa_test.go Part_ subtests: $$(grep -cE 't\.Run\(\"Part[0-9]+_' deploy/test/qa_test.go 2>/dev/null || echo 0)"
+	@echo "testing-guide.md Parts: $$(grep -cE '^## Part [0-9]+:' docs/testing-guide.md 2>/dev/null || echo 0)"
+	@echo "Seed unique mc-* IDs:  $$(grep -oE "mc-[a-z0-9_-]+" migrations/seed_demo.sql 2>/dev/null | sort -u | wc -l | tr -d ' ')"
+	@echo "Seed unique ag-* IDs:  $$(grep -oE "ag-[a-z0-9_-]+" migrations/seed_demo.sql 2>/dev/null | sort -u | wc -l | tr -d ' ') (incl. agent_groups; agents-table count is 12)"
+	@echo "Seed unique iss-* IDs: $$(grep -oE "iss-[a-z0-9_-]+" migrations/seed_demo.sql 2>/dev/null | sort -u | wc -l | tr -d ' ') (issuers table count is 13)"
+	@echo "Seed unique tgt-* IDs: $$(grep -oE "tgt-[a-z0-9_-]+" migrations/seed_demo.sql 2>/dev/null | sort -u | wc -l | tr -d ' ')"
+	@echo "Seed unique nst-* IDs: $$(grep -oE "nst-[a-z0-9_-]+" migrations/seed_demo.sql 2>/dev/null | sort -u | wc -l | tr -d ' ')"
+
 # Cleanup
 clean:
 	@echo "Cleaning build artifacts..."

README.md

@@ -7,97 +7,130 @@
 
 # certctl — Self-Hosted Certificate Lifecycle Platform
 
-```mermaid
-timeline
-    title TLS Certificate Maximum Lifespan (CA/Browser Forum Ballot SC-081v3)
-    2015 : 5 years
-    2018 : 825 days
-    2020 : 398 days
-    March 2026 : 200 days
-    March 2027 : 100 days
-    March 2029 : 47 days
-```
-
-TLS certificate lifespans are shrinking fast. The CA/Browser Forum passed [Ballot SC-081v3](https://cabforum.org/2025/04/11/ballot-sc081v3-introduce-schedule-of-reducing-validity-and-data-reuse-periods/) unanimously in April 2025, setting a phased reduction: **200 days** by March 2026, **100 days** by March 2027, and **47 days** by March 2029. Organizations managing dozens or hundreds of certificates can no longer rely on spreadsheets, calendar reminders, or manual renewal workflows. The math doesn't work — at 47-day lifespans, a team managing 100 certificates is processing 7+ renewals per week, every week, forever.
-
-certctl is a self-hosted platform that automates the entire certificate lifecycle — from issuance through renewal to deployment — with zero human intervention. It works with any certificate authority, deploys to any server, and keeps private keys on your infrastructure where they belong.
-
 [![License](https://img.shields.io/badge/license-BSL%201.1-blue.svg)](LICENSE)
 [![Go Report Card](https://goreportcard.com/badge/github.com/shankar0123/certctl)](https://goreportcard.com/report/github.com/shankar0123/certctl)
 [![GitHub Release](https://img.shields.io/github/v/release/shankar0123/certctl)](https://github.com/shankar0123/certctl/releases)
+[![GitHub Stars](https://img.shields.io/github/stars/shankar0123/certctl?style=flat&logo=github)](https://github.com/shankar0123/certctl/stargazers)
+
+TLS certificate lifespans are shrinking fast. The CA/Browser Forum passed [Ballot SC-081v3](https://cabforum.org/2025/04/11/ballot-sc081v3-introduce-schedule-of-reducing-validity-and-data-reuse-periods/) unanimously in April 2025, setting a phased reduction: **200 days** by March 2026, **100 days** by March 2027, and **47 days** by March 2029. Organizations managing dozens or hundreds of certificates can no longer rely on spreadsheets, calendar reminders, or manual renewal workflows. The math doesn't work — at 47-day lifespans, a team managing 100 certificates is processing 7+ renewals per week, every week, forever.
+
+certctl is a self-hosted platform that automates the entire certificate lifecycle — from issuance through renewal to deployment — with zero human intervention. It works with any certificate authority, deploys to any server, and keeps private keys on your infrastructure where they belong. It's free, self-hosted, and covers the same lifecycle that enterprise platforms charge $100K+/year for.
+
+```mermaid
+gantt
+    title TLS Certificate Maximum Lifespan — CA/Browser Forum Ballot SC-081v3
+    dateFormat YYYY-MM-DD
+    axisFormat
+    todayMarker off
+    section 2015
+        5 years (1825 days)    :done, 2020-01-01, 1825d
+    section 2018
+        825 days               :done, 2020-01-01, 825d
+    section 2020
+        398 days               :active, 2020-01-01, 398d
+    section 2026
+        200 days               :crit, 2020-01-01, 200d
+    section 2027
+        100 days               :crit, 2020-01-01, 100d
+    section 2029
+        47 days                :crit, 2020-01-01, 47d
+```
+
+> **Actively maintained — shipping weekly.** Found something? [Open a GitHub issue](https://github.com/shankar0123/certctl/issues) — issues get triaged same-day. CI runs the full test suite with race detection, static analysis, and vulnerability scanning on every commit.
+
+**Ready to try it?** Jump to the [Quick Start](#quick-start) — you'll have a running dashboard in under 5 minutes.
 
 ## Documentation
 
 | Guide | Description |
 |-------|-------------|
-| [Why certctl?](docs/why-certctl.md) | Competitive positioning — how certctl compares to open-source and enterprise certificate management platforms |
+| [Why certctl?](docs/why-certctl.md) | How certctl compares to ACME clients, agent-based SaaS, and enterprise platforms |
 | [Concepts](docs/concepts.md) | TLS certificates explained from scratch — for beginners who know nothing about certs |
-| [Quick Start](docs/quickstart.md) | Get running in 5 minutes — dashboard, API, CLI, discovery, stakeholder demo flow |
+| [Quick Start](docs/quickstart.md) | 5-minute setup — dashboard, API, CLI, discovery, stakeholder demo flow |
+| [Docker Compose Environments](deploy/ENVIRONMENTS.md) | Service-by-service walkthrough of all 4 compose files, env var reference |
+| [Deployment Examples](docs/examples.md) | 5 turnkey scenarios (ACME+NGINX, wildcard DNS-01, private CA, step-ca, multi-issuer) with migration guides |
 | [Advanced Demo](docs/demo-advanced.md) | Issue a certificate end-to-end with technical deep-dives |
 | [Architecture](docs/architecture.md) | System design, data flow diagrams, security model |
-| [Feature Inventory](docs/features.md) | Complete reference of all V2 capabilities, API endpoints, and configuration |
-| [Connectors](docs/connectors.md) | Build custom issuer, target, and notifier connectors |
+| [Feature Inventory](docs/features.md) | Complete reference of all capabilities, API endpoints, and configuration |
+| [Connector Reference](docs/connectors.md) | Configuration for all issuer, target, and notifier connectors |
+| [MCP Server](docs/mcp.md) | AI integration via Model Context Protocol — setup, available tools, examples |
+| [OpenAPI 3.1 Spec](docs/openapi.md) | API reference guide with endpoint overview ([raw spec](api/openapi.yaml)) |
 | [Compliance Mapping](docs/compliance.md) | SOC 2 Type II, PCI-DSS 4.0, NIST SP 800-57 alignment guides |
-
-## Why certctl Exists
-
-Certificate lifecycle tooling today falls into two camps: expensive enterprise platforms (Venafi, Keyfactor, Sectigo) that cost six figures and take months to deploy, or single-purpose tools (cert-manager, certbot) that handle one slice of the problem. If you run a mixed infrastructure — some NGINX, some Apache, a few HAProxy nodes, maybe an F5 — and you need to manage certificates from multiple CAs, there's nothing self-hosted that covers the full lifecycle without vendor lock-in.
-
-certctl fills that gap. It's **CA-agnostic** — the issuer connector interface means you can plug in any certificate authority: a self-signed local CA for dev, Let's Encrypt via ACME for public certs, Smallstep step-ca for your private PKI, your enterprise ADCS via sub-CA mode, or any custom CA through a shell script adapter. You're never locked to a single CA vendor, and you can run multiple issuers simultaneously for different certificate types.
-
-It's also **target-agnostic**. Agents deploy certificates to NGINX, Apache, HAProxy, Traefik, and Caddy — all using the same pluggable connector model for any server that accepts cert files. The control plane never initiates outbound connections — agents poll for work, which means certctl works behind firewalls, across network zones, and in air-gapped environments.
-
-For a detailed comparison with CertKit, KeyTalk, and enterprise platforms (Venafi, Keyfactor), see [Why certctl?](docs/why-certctl.md)
-
-## What It Does
-
-certctl gives you a single pane of glass for every TLS certificate in your organization:
-
-- **Web dashboard** — full certificate inventory with status, ownership, expiration heatmaps, and bulk operations
-- **REST API** — 93 endpoints under `/api/v1/` + `/.well-known/est/` for complete automation
-- **Agents** — generate private keys locally, discover existing certs on disk, submit CSRs (private keys never leave your servers)
-- **Network scanner** — discovers certificates on TLS endpoints across CIDR ranges without requiring agents
-- **EST server** (RFC 7030) — device and WiFi certificate enrollment via industry-standard protocol
-- **Approval workflows** — require human sign-off on renewals before deployment
-- **Background scheduler** — watches expiration dates and triggers renewals automatically, handling constant rotation at 47-day lifespans without human involvement
-
-For the full capability breakdown — revocation infrastructure, policy engine, observability, EST enrollment, and more — see the [Feature Inventory](docs/features.md).
+| [Migrate from certbot](docs/migrate-from-certbot.md) | Step-by-step migration from certbot cron jobs to certctl |
+| [Migrate from acme.sh](docs/migrate-from-acmesh.md) | Migration guide for acme.sh users, DNS hook compatibility |
+| [certctl for cert-manager users](docs/certctl-for-cert-manager-users.md) | How certctl complements cert-manager for mixed infrastructure |
+| [Test Environment](docs/test-env.md) | Docker Compose test environment with real CA backends |
+| [Testing Guide](docs/testing-guide.md) | Comprehensive test procedures, smoke tests, and release sign-off checklist |
 
 ## Supported Integrations
 
 ### Certificate Issuers
-| Issuer | Status | Type |
-|--------|--------|------|
-| Local CA (self-signed + sub-CA) | Implemented | `GenericCA` |
-| ACME v2 (Let's Encrypt, Sectigo) | Implemented (HTTP-01 + DNS-01 + DNS-PERSIST-01) | `ACME` |
-| ACME EAB (ZeroSSL, Google Trust) | Implemented (auto-fetch EAB from ZeroSSL) | `ACME` |
-| step-ca | Implemented | `StepCA` |
-| OpenSSL / Custom CA | Implemented | `OpenSSL` |
-| Vault PKI | Future | — |
-| DigiCert | Future | — |
 
-**Note:** ADCS integration is handled via the Local CA's sub-CA mode — certctl operates as a subordinate CA with its signing certificate issued by ADCS. Any CA with a shell-accessible signing interface can be integrated today via the OpenSSL/Custom CA connector.
+| Issuer | Type | Notes |
+|--------|------|-------|
+| Local CA (self-signed + sub-CA) | `GenericCA` | Sub-CA mode chains to enterprise root (ADCS, etc.) |
+| ACME v2 (Let's Encrypt, ZeroSSL, etc.) | `ACME` | HTTP-01, DNS-01, DNS-PERSIST-01 challenges. EAB auto-fetch from ZeroSSL. Profile selection (`tlsserver`, `shortlived`). |
+| step-ca (Smallstep) | `StepCA` | JWK provisioner auth, issuance + renewal + revocation |
+| OpenSSL / Custom CA | `OpenSSL` | Shell script adapter — any CA with a CLI |
+| HashiCorp Vault PKI | `VaultPKI` | Token auth, synchronous issuance, CRL/OCSP delegated to Vault |
+| DigiCert CertCentral | `DigiCert` | Async order model, OV/EV support, PEM bundle parsing |
+| Sectigo SCM | `Sectigo` | 3-header auth, DV/OV/EV, collect-not-ready graceful handling |
+| Google Cloud CAS | `GoogleCAS` | OAuth2 service account, synchronous issuance, CA pool selection |
+| AWS ACM Private CA | `AWSACMPCA` | Synchronous issuance, configurable signing algorithm/template ARN |
+| Entrust Certificate Services | `Entrust` | mTLS client certificate auth, synchronous/approval-pending issuance |
+| GlobalSign Atlas HVCA | `GlobalSign` | mTLS + API key/secret dual auth, serial-based tracking |
+| EJBCA (Keyfactor) | `EJBCA` | Dual auth (mTLS or OAuth2), self-hosted open-source CA |
+
+**Note:** ADCS integration is handled via the Local CA's sub-CA mode — certctl operates as a subordinate CA with its signing certificate issued by ADCS. Any CA with a shell-accessible signing interface can be integrated via the OpenSSL/Custom CA connector.
 
 ### Deployment Targets
-| Target | Status | Type |
-|--------|--------|------|
-| NGINX | Implemented | `NGINX` |
-| Apache httpd | Implemented | `Apache` |
-| HAProxy | Implemented | `HAProxy` |
-| Traefik | Implemented | `Traefik` |
-| Caddy | Implemented | `Caddy` |
-| F5 BIG-IP | Interface only | `F5` |
-| Microsoft IIS | Interface only | `IIS` |
+
+| Target | Type | Notes |
+|--------|------|-------|
+| NGINX | `NGINX` | File write, config validation, reload |
+| Apache httpd | `Apache` | Separate cert/chain/key files, configtest, graceful reload |
+| HAProxy | `HAProxy` | Combined PEM file, validate, reload |
+| Traefik | `Traefik` | File provider deployment, auto-reload via filesystem watch |
+| Caddy | `Caddy` | Dual-mode: admin API hot-reload or file-based |
+| Envoy | `Envoy` | File-based with optional SDS JSON config |
+| Postfix | `Postfix` | Mail server TLS, pairs with S/MIME support |
+| Dovecot | `Dovecot` | Mail server TLS, pairs with S/MIME support |
+| Microsoft IIS | `IIS` | Local PowerShell or remote WinRM, PEM→PFX, SNI support |
+| F5 BIG-IP | `F5` | iControl REST via proxy agent, transaction-based atomic updates |
+| SSH (Agentless) | `SSH` | SFTP cert/key deployment to any Linux/Unix server |
+| Windows Certificate Store | `WinCertStore` | PowerShell Import-PfxCertificate, configurable store/location |
+| Java Keystore | `JavaKeystore` | PEM→PKCS#12→keytool pipeline, JKS and PKCS12 formats |
+| Kubernetes Secrets | `KubernetesSecrets` | `kubernetes.io/tls` Secrets, in-cluster or kubeconfig auth |
+
+### Enrollment Protocols
+
+| Protocol | Standard | Use Case |
+|----------|----------|----------|
+| EST (Enrollment over Secure Transport) | RFC 7030 | Device enrollment, WiFi/802.1X, IoT |
+| SCEP (Simple Certificate Enrollment Protocol) | RFC 8894 | MDM platforms (Jamf, Intune), network devices |
+| ACME v2 | RFC 8555 | Public CA automated issuance (Let's Encrypt, ZeroSSL) |
+| ACME ARI (Renewal Information) | RFC 9773 | CA-directed renewal timing — the CA tells you when to renew |
+
+### Standards & Revocation
+
+| Capability | Standard | Notes |
+|------------|----------|-------|
+| DER-encoded X.509 CRL | RFC 5280 | Per-issuer, signed by issuing CA, 24h validity |
+| Embedded OCSP responder | RFC 6960 | Good/revoked/unknown status per issuer |
+| S/MIME certificates | RFC 8551 | Email protection EKU, adaptive KeyUsage flags |
+| Certificate export | — | PEM (JSON/file) and PKCS#12 formats |
+| ACME DNS-PERSIST-01 | IETF draft | Standing validation record, no per-renewal DNS updates |
 
 ### Notifiers
-| Notifier | Status | Type |
-|----------|--------|------|
-| Email (SMTP) | Implemented | `Email` |
-| Webhooks | Implemented | `Webhook` |
-| Slack | Implemented | `Slack` |
-| Microsoft Teams | Implemented | `Teams` |
-| PagerDuty | Implemented | `PagerDuty` |
-| OpsGenie | Implemented | `OpsGenie` |
+
+| Notifier | Type |
+|----------|------|
+| Email (SMTP) | `Email` |
+| Webhooks | `Webhook` |
+| Slack | `Slack` |
+| Microsoft Teams | `Teams` |
+| PagerDuty | `PagerDuty` |
+| OpsGenie | `OpsGenie` |
 
 All connectors are pluggable — build your own by implementing the [connector interface](docs/connectors.md).
 
@@ -105,41 +138,57 @@ All connectors are pluggable — build your own by implementing the [connector i
 
 <table>
 <tr>
-<td><a href="docs/screenshots/v2-dashboard.png"><img src="docs/screenshots/v2-dashboard.png" width="270" alt="Dashboard"></a><br><b>Dashboard</b><br><sub>Stats, expiration heatmap, renewal trends</sub></td>
-<td><a href="docs/screenshots/v2-certificates.png"><img src="docs/screenshots/v2-certificates.png" width="270" alt="Certificates"></a><br><b>Certificates</b><br><sub>Inventory with status, owner, team filters</sub></td>
-<td><a href="docs/screenshots/v2-agents.png"><img src="docs/screenshots/v2-agents.png" width="270" alt="Agents"></a><br><b>Agents</b><br><sub>Fleet health, OS/arch, IP, version</sub></td>
+<td><a href="docs/screenshots/v2-dashboard.png"><img src="docs/screenshots/v2-dashboard.png" width="400" alt="Dashboard"></a><br><b>Dashboard</b><br><sub>Stats, expiration heatmap, renewal trends, issuance rate</sub></td>
+<td><a href="docs/screenshots/v2-certificates.png"><img src="docs/screenshots/v2-certificates.png" width="400" alt="Certificates"></a><br><b>Certificates</b><br><sub>Inventory with bulk ops, status filters, owner/team columns</sub></td>
 </tr>
 <tr>
-<td><a href="docs/screenshots/v2-fleet.png"><img src="docs/screenshots/v2-fleet.png" width="270" alt="Fleet Overview"></a><br><b>Fleet Overview</b><br><sub>OS distribution, status breakdown</sub></td>
-<td><a href="docs/screenshots/v2-jobs.png"><img src="docs/screenshots/v2-jobs.png" width="270" alt="Jobs"></a><br><b>Jobs</b><br><sub>Issuance, renewal, deployment queue</sub></td>
-<td><a href="docs/screenshots/v2-notifications.png"><img src="docs/screenshots/v2-notifications.png" width="270" alt="Notifications"></a><br><b>Notifications</b><br><sub>Expiration warnings, renewal results</sub></td>
-</tr>
-<tr>
-<td><a href="docs/screenshots/v2-policies.png"><img src="docs/screenshots/v2-policies.png" width="270" alt="Policies"></a><br><b>Policies</b><br><sub>Ownership, lifetime, renewal rules</sub></td>
-<td><a href="docs/screenshots/v2-profiles.png"><img src="docs/screenshots/v2-profiles.png" width="270" alt="Profiles"></a><br><b>Profiles</b><br><sub>Key types, max TTL, crypto constraints</sub></td>
-<td><a href="docs/screenshots/v2-issuers.png"><img src="docs/screenshots/v2-issuers.png" width="270" alt="Issuers"></a><br><b>Issuers</b><br><sub>Local CA, ACME, step-ca connectors</sub></td>
-</tr>
-<tr>
-<td><a href="docs/screenshots/v2-targets.png"><img src="docs/screenshots/v2-targets.png" width="270" alt="Targets"></a><br><b>Targets</b><br><sub>NGINX, Apache, HAProxy, Traefik, Caddy deployment</sub></td>
-<td><a href="docs/screenshots/v2-owners.png"><img src="docs/screenshots/v2-owners.png" width="270" alt="Owners"></a><br><b>Owners</b><br><sub>Cert ownership with team assignment</sub></td>
-<td><a href="docs/screenshots/v2-teams.png"><img src="docs/screenshots/v2-teams.png" width="270" alt="Teams"></a><br><b>Teams</b><br><sub>Org grouping for notification routing</sub></td>
-</tr>
-<tr>
-<td><a href="docs/screenshots/v2-agent-groups.png"><img src="docs/screenshots/v2-agent-groups.png" width="270" alt="Agent Groups"></a><br><b>Agent Groups</b><br><sub>Dynamic grouping by OS, arch, CIDR</sub></td>
-<td><a href="docs/screenshots/v2-audit-trail.png"><img src="docs/screenshots/v2-audit-trail.png" width="270" alt="Audit Trail"></a><br><b>Audit Trail</b><br><sub>Immutable log, CSV/JSON export</sub></td>
-<td><a href="docs/screenshots/v2-short-lived.png"><img src="docs/screenshots/v2-short-lived.png" width="270" alt="Short-Lived"></a><br><b>Short-Lived Creds</b><br><sub>Ephemeral certs with live TTL countdown</sub></td>
+<td><a href="docs/screenshots/v2-issuers.png"><img src="docs/screenshots/v2-issuers.png" width="400" alt="Issuers"></a><br><b>Issuers</b><br><sub>Catalog with 10 CA types, GUI config, test connection</sub></td>
+<td><a href="docs/screenshots/v2-jobs.png"><img src="docs/screenshots/v2-jobs.png" width="400" alt="Jobs"></a><br><b>Jobs</b><br><sub>Issuance, renewal, deployment queue with approval workflow</sub></td>
 </tr>
 </table>
 
+**[See all screenshots →](docs/screenshots/)**
+
+## Why certctl
+
+Certificate lifecycle tooling falls into two camps: enterprise platforms (Venafi, Keyfactor) that cost six figures and take months to deploy, or single-purpose tools (certbot, cert-manager) that handle one slice of the problem. certctl fills the gap — full lifecycle automation, self-hosted, free, CA-agnostic, and target-agnostic. If you're running certbot cron jobs, manually renewing certs, or stitching together scripts across mixed infrastructure, certctl replaces all of that.
+
+Built for **platform engineering and DevOps teams** managing 10–500+ certificates, **security and compliance teams** who need audit trails and policy enforcement for SOC 2, PCI-DSS 4.0, or NIST SP 800-57 ([compliance mapping included](docs/compliance.md)), and **small teams without enterprise budgets** who need Venafi-grade automation for a 50-server environment. For a detailed comparison, see [Why certctl?](docs/why-certctl.md)
+
+**Architecture.** Go 1.25 control plane with handler→service→repository layering, PostgreSQL 16 backend (21 tables), and a pull-only deployment model — the server never initiates outbound connections. Agents poll for work. For network appliances and agentless servers, a proxy agent in the same network zone handles deployment via the target's API (WinRM, iControl REST, SSH/SFTP). Background scheduler runs 7 loops: renewal with ARI integration (1h), job processing (30s), agent health (2m), notifications (1m), short-lived cert expiry (30s), network scanning (6h), certificate digest (24h). See [Architecture Guide](docs/architecture.md) for full system diagrams.
+
+**Security-first.** Agents generate ECDSA P-256 keys locally — private keys never touch the control plane. API key auth enforced by default with SHA-256 hashing and constant-time comparison. CORS deny-by-default. Shell injection prevention on all connector scripts. SSRF protection (reserved IP filtering) on the network scanner. Atomic idempotency guards on scheduler loops. Issuer and target credentials encrypted at rest with AES-256-GCM. Every API call recorded to an immutable audit trail with actor attribution, body hash, and latency tracking. CI runs race detection, 11 linters, and vulnerability scanning on every commit.
+
+**Key design decisions.** TEXT primary keys — human-readable prefixed IDs (`mc-api-prod`, `t-platform`, `o-alice`) so you can identify resources at a glance in logs and queries. Idempotent migrations (`IF NOT EXISTS`, `ON CONFLICT DO NOTHING`) safe for repeated execution. Dynamic configuration via GUI with AES-256-GCM encrypted credential storage and env var backward compatibility. Handlers define their own service interfaces for clean dependency inversion.
+
+## What It Does
+
+**Automated lifecycle.** Certificates renew and deploy themselves. The scheduler monitors expiration, issues through your CA, and deploys to targets — zero human intervention. ACME ARI (RFC 9773) lets the CA direct renewal timing. Ready for 47-day (SC-081v3) and 6-day (Let's Encrypt shortlived) certificate lifetimes.
+
+**Operational dashboard.** 26-page GUI covers the entire lifecycle: certificate inventory with bulk ops, deployment timeline with rollback, discovery triage, network scan management, agent fleet health, short-lived credential countdown, approval workflows, and observability metrics. Configure issuers and targets from the dashboard — no env var editing, no server restarts.
+
+**Private keys stay on your servers.** Agents generate ECDSA P-256 keys locally, submit only the CSR. The control plane never touches private keys. After deployment, agents probe the live TLS endpoint and compare SHA-256 fingerprints to confirm the right certificate is actually being served.
+
+**Discovery.** Agents scan filesystems for existing PEM/DER certificates. The network scanner probes TLS endpoints across CIDR ranges without agents. Cloud discovery finds certificates in AWS Secrets Manager, Azure Key Vault, and GCP Secret Manager. Continuous TLS health monitoring tracks endpoint status (healthy/degraded/down/cert_mismatch) with configurable thresholds and historical probe data. All discovery modes feed into a unified triage workflow — claim, dismiss, or import what you find.
+
+**Policy engine.** Certificate profiles constrain key types, max TTL, and EKUs — with crypto policy enforcement that validates every CSR against profile rules before it reaches the issuer. MaxTTL caps are enforced per issuer connector. Approval workflows pause jobs for human review. Ownership tracking routes notifications to the right team. Agent groups match devices by OS, architecture, IP CIDR, and version.
+
+**Enrollment protocols.** EST server (RFC 7030) for device and WiFi enrollment. SCEP server (RFC 8894) for MDM platforms and network devices. S/MIME issuance with email protection EKU.
+
+**Revocation.** Single and bulk revocation (by profile, owner, agent, or issuer). DER-encoded X.509 CRL per issuer, signed by the issuing CA. Embedded OCSP responder. RFC 5280 reason codes. Short-lived certs (TTL < 1 hour) are exempt — expiry is sufficient revocation.
+
+**Audit and observability.** Immutable append-only audit trail records every lifecycle action, every API call, and every approval decision. Prometheus metrics endpoint. Scheduled certificate digest emails. Continuous endpoint health monitoring with state machine transitions and real-time alerts.
+
+**Notifications.** Slack, Teams, PagerDuty, OpsGenie, SMTP, webhooks. Routed by certificate owner. Daily digest emails with stats and expiring certs.
+
+**Multiple interfaces.** REST API (111 routes), CLI (12 commands), MCP server (80 tools for Claude, Cursor, Windsurf), Helm chart, web dashboard. Certificate export in PEM and PKCS#12.
+
+**First-run onboarding.** Wizard guides you through connecting a CA, deploying an agent, and issuing your first certificate. Or start with the pre-populated demo — 32 certificates, 10 issuers, 180 days of history.
+
+For the complete capability breakdown, see the [Feature Inventory](docs/features.md).
+
 ## Quick Start
 
-### Docker Pull
-
-```bash
-docker pull shankar0123.docker.scarf.sh/certctl-server
-docker pull shankar0123.docker.scarf.sh/certctl-agent
-```
-
 ### Docker Compose (Recommended)
 
 ```bash
@@ -148,247 +197,129 @@ cd certctl
 docker compose -f deploy/docker-compose.yml up -d --build
 ```
 
-Wait ~30 seconds, then open **http://localhost:8443** in your browser.
+Wait ~30 seconds, then open **https://localhost:8443** in your browser. (The shipped `docker-compose.yml` self-signs a cert via the `certctl-tls-init` init container on first boot — accept the browser warning for the demo, or feed the generated `ca.crt` to your client.) The onboarding wizard walks you through connecting a CA, deploying an agent, and issuing your first certificate.
 
-The dashboard comes pre-loaded with 15 demo certificates, 5 agents, policy rules, audit events, and notifications — a realistic snapshot of a certificate inventory so you can explore immediately.
+**Want a pre-populated demo instead?** Add the demo override to see 32 certificates across 10 issuers, 8 agents, and 180 days of realistic history:
 
-Verify the API:
 ```bash
-curl http://localhost:8443/health
+docker compose -f deploy/docker-compose.yml -f deploy/docker-compose.demo.yml up -d --build
+```
+
+The `deploy/` directory has four compose files: `docker-compose.yml` (base platform), `docker-compose.demo.yml` (demo data overlay), `docker-compose.dev.yml` (PgAdmin + debug logging), and `docker-compose.test.yml` (standalone integration tests with real CA backends). See the [Docker Compose Environments Guide](deploy/ENVIRONMENTS.md) for a service-by-service walkthrough, or the [Quick Start](docs/quickstart.md#docker-compose-environments) for a summary.
+
+```bash
+curl --cacert $(docker compose -f deploy/docker-compose.yml exec -T certctl-server cat /etc/certctl/tls/ca.crt) https://localhost:8443/health
 # {"status":"healthy"}
-
-curl -s http://localhost:8443/api/v1/certificates | jq '.total'
-# 15
 ```
 
-### Manual Build
+The control plane is HTTPS-only (TLS 1.3, no plaintext listener). See [`docs/tls.md`](docs/tls.md) for cert provisioning patterns and [`docs/upgrade-to-tls.md`](docs/upgrade-to-tls.md) if you're upgrading from a pre-v2.2 release.
+
+### Agent Install (One-Liner)
 
 ```bash
-# Prerequisites: Go 1.25+, PostgreSQL 16+
-go mod download
-make build
-
-# Set up database
-export CERTCTL_DATABASE_URL="postgres://certctl:certctl@localhost:5432/certctl?sslmode=disable"
-export CERTCTL_AUTH_TYPE=none
-make migrate-up
-
-# Start server
-./bin/server
-
-# Start agent (separate terminal)
-export CERTCTL_SERVER_URL=http://localhost:8443
-export CERTCTL_API_KEY=change-me-in-production
-export CERTCTL_AGENT_NAME=local-agent
-export CERTCTL_AGENT_ID=agent-local-01
-./bin/agent --agent-id=agent-local-01
+curl -sSL https://raw.githubusercontent.com/shankar0123/certctl/master/install-agent.sh | bash
 ```
 
-## Architecture
+Detects your OS and architecture, downloads the binary, configures systemd (Linux) or launchd (macOS), and starts the agent. See [install-agent.sh](install-agent.sh) for details.
 
-**Control plane** (Go 1.25 net/http) → **PostgreSQL 16** (21 tables, TEXT primary keys) → **Agents** (key generation, CSR submission, cert deployment). Background scheduler runs 6 loops: renewal checks (1h), job processing (30s), agent health (2m), notifications (1m), short-lived cert expiry (30s), network scanning (6h). See [Architecture Guide](docs/architecture.md) for full system diagrams and data flow.
-
-### Key Design Decisions
-
-- **Private keys isolated from the control plane.** Agents generate ECDSA P-256 keys locally and submit CSRs (public key only). The server signs the CSR and returns the certificate — private keys never touch the control plane. Server-side keygen is available via `CERTCTL_KEYGEN_MODE=server` for demo/development only.
-- **TEXT primary keys, not UUIDs.** IDs are human-readable prefixed strings (`mc-api-prod`, `t-platform`, `o-alice`) so you can identify resource types at a glance in logs and queries.
-- **Handler → Service → Repository layering.** Handlers define their own service interfaces for clean dependency inversion. No global service singletons.
-- **Idempotent migrations.** All schema uses `IF NOT EXISTS` and seed data uses `ON CONFLICT (id) DO NOTHING`, safe for repeated execution.
-
-PostgreSQL 16 with 21 tables covering certificates, versions, policies, issuers, targets, agents, jobs, teams, owners, profiles, agent groups, revocations, discovery, network scans, and audit events. See the [Architecture Guide](docs/architecture.md) for the full schema.
-
-## Configuration
-
-All environment variables use the `CERTCTL_` prefix. Full reference below (39 variables across server, agent, and connector config).
-
-### Server — Core
-
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `CERTCTL_SERVER_HOST` | `127.0.0.1` | Server bind address |
-| `CERTCTL_SERVER_PORT` | `8080` | Server listen port (1–65535) |
-| `CERTCTL_DATABASE_URL` | `postgres://localhost/certctl` | PostgreSQL connection string (required) |
-| `CERTCTL_DATABASE_MAX_CONNS` | `25` | PostgreSQL connection pool size (min 1) |
-| `CERTCTL_DATABASE_MIGRATIONS_PATH` | `./migrations` | Path to migration SQL files |
-| `CERTCTL_MAX_BODY_SIZE` | `1048576` | Max HTTP request body in bytes (default 1MB) |
-| `CERTCTL_LOG_LEVEL` | `info` | Log verbosity: `debug`, `info`, `warn`, `error` |
-| `CERTCTL_LOG_FORMAT` | `json` | Log format: `json` (structured) or `text` (human-readable) |
-
-### Server — Auth, CORS, Rate Limiting
-
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `CERTCTL_AUTH_TYPE` | `api-key` | Auth mode: `api-key`, `jwt`, or `none` (demo only) |
-| `CERTCTL_AUTH_SECRET` | — | Required for `api-key` and `jwt` auth types |
-| `CERTCTL_CORS_ORIGINS` | *(empty = deny all)* | Comma-separated allowed origins, or `*` for dev |
-| `CERTCTL_RATE_LIMIT_ENABLED` | `true` | Enable token bucket rate limiting |
-| `CERTCTL_RATE_LIMIT_RPS` | `50` | Requests per second per client |
-| `CERTCTL_RATE_LIMIT_BURST` | `100` | Max burst size |
-| `CERTCTL_KEYGEN_MODE` | `agent` | Key generation: `agent` (production) or `server` (demo only) |
-
-### Server — Scheduler
-
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `CERTCTL_SCHEDULER_RENEWAL_CHECK_INTERVAL` | `1h` | How often to check expiring certs (min 1m) |
-| `CERTCTL_SCHEDULER_JOB_PROCESSOR_INTERVAL` | `30s` | How often to process pending jobs (min 1s) |
-| `CERTCTL_SCHEDULER_AGENT_HEALTH_CHECK_INTERVAL` | `2m` | Agent heartbeat check frequency (min 1s) |
-| `CERTCTL_SCHEDULER_NOTIFICATION_PROCESS_INTERVAL` | `1m` | Notification send frequency (min 1s) |
-
-### Server — Sub-CA Mode
-
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `CERTCTL_CA_CERT_PATH` | — | PEM-encoded CA certificate for sub-CA mode |
-| `CERTCTL_CA_KEY_PATH` | — | PEM-encoded CA private key (RSA, ECDSA, PKCS#8) |
-
-### Server — Feature Flags
-
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `CERTCTL_EST_ENABLED` | `false` | Enable RFC 7030 EST enrollment endpoints |
-| `CERTCTL_EST_ISSUER_ID` | `iss-local` | Which issuer processes EST enrollments |
-| `CERTCTL_EST_PROFILE_ID` | — | Constrain EST to a specific certificate profile |
-| `CERTCTL_NETWORK_SCAN_ENABLED` | `false` | Enable server-side TLS network scanning |
-| `CERTCTL_NETWORK_SCAN_INTERVAL` | `6h` | How often scheduled scans run |
-| `CERTCTL_VERIFY_DEPLOYMENT` | `true` | TLS verification after certificate deployment |
-| `CERTCTL_VERIFY_TIMEOUT` | `10s` | TLS probe timeout |
-| `CERTCTL_VERIFY_DELAY` | `2s` | Delay before verification probe |
-
-### Server — Notification Connectors
-
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `CERTCTL_SLACK_WEBHOOK_URL` | — | Slack incoming webhook URL (enables Slack) |
-| `CERTCTL_SLACK_CHANNEL` | — | Override default webhook channel |
-| `CERTCTL_SLACK_USERNAME` | `certctl` | Bot display name |
-| `CERTCTL_TEAMS_WEBHOOK_URL` | — | Microsoft Teams webhook URL (enables Teams) |
-| `CERTCTL_PAGERDUTY_ROUTING_KEY` | — | PagerDuty Events API v2 key (enables PagerDuty) |
-| `CERTCTL_PAGERDUTY_SEVERITY` | `warning` | Event severity: `info`, `warning`, `error`, `critical` |
-| `CERTCTL_OPSGENIE_API_KEY` | — | OpsGenie Alert API key (enables OpsGenie) |
-| `CERTCTL_OPSGENIE_PRIORITY` | `P3` | Alert priority: `P1`–`P5` |
-
-### Agent
-
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `CERTCTL_SERVER_URL` | `http://localhost:8080` | Control plane URL |
-| `CERTCTL_API_KEY` | — | Agent API key for authentication |
-| `CERTCTL_AGENT_ID` | — | Registered agent ID (required) |
-| `CERTCTL_KEY_DIR` | `/var/lib/certctl/keys` | Private key storage directory (0600 perms) |
-| `CERTCTL_DISCOVERY_DIRS` | — | Directories to scan for existing certs (comma-separated) |
-
-Docker Compose overrides for the demo stack are in `deploy/docker-compose.yml`.
-
-## Development
+### Helm Chart (Kubernetes)
 
 ```bash
-# Install dev tools (golangci-lint, migrate CLI, air)
-make install-tools
-
-# Run tests
-make test
-
-# Run tests with race detection (same as CI)
-go test -race ./internal/service/... ./internal/api/handler/... ./internal/api/middleware/... ./internal/scheduler/... ./internal/connector/... ./internal/domain/... ./internal/validation/...
-
-# Run with coverage
-make test-coverage
-
-# Lint (runs golangci-lint with project config)
-make lint
-
-# Vulnerability scan
-govulncheck ./...
-
-# Format
-make fmt
+helm install certctl deploy/helm/certctl/ \
+  --set server.apiKey=your-api-key \
+  --set postgres.password=your-db-password
 ```
 
-### CI Pipeline
+Production-ready chart with Server Deployment, PostgreSQL StatefulSet, Agent DaemonSet, health probes, security contexts (non-root, read-only rootfs), and optional Ingress. See [values.yaml](deploy/helm/certctl/values.yaml) for all configuration options.
 
-Every push and PR runs: `go vet`, `go test -race` (race detection), `golangci-lint` (11 linters including gosec and bodyclose), `govulncheck` (dependency CVE scanning), and per-layer coverage thresholds (service 60%, handler 60%, domain 40%, middleware 50%). Frontend CI runs TypeScript type checking, Vitest tests, and Vite production build. See `.github/workflows/ci.yml` for details.
-
-### Docker Compose
+### Docker Pull
 
 ```bash
-make docker-up          # Start stack (server + postgres + agent)
-make docker-down        # Stop stack
-make docker-logs-server # Server logs
-make docker-logs-agent  # Agent logs
-make docker-clean       # Stop + remove volumes
+docker pull shankar0123.docker.scarf.sh/certctl-server
+docker pull shankar0123.docker.scarf.sh/certctl-agent
 ```
 
-## Security
+## Verifying this release
 
-### Private Key Management
-- **Agent keygen mode (default)**: Agents generate ECDSA P-256 keys locally and store them with 0600 permissions in `CERTCTL_KEY_DIR` (default `/var/lib/certctl/keys`). Only the CSR (public key) is sent to the control plane. Private keys never leave agent infrastructure.
-- **Server keygen mode (demo only)**: Set `CERTCTL_KEYGEN_MODE=server` for development/demo with Local CA. The control plane generates RSA-2048 keys server-side. A log warning is emitted at startup.
+Every `v*` tag publishes signed, attested release artefacts. Binaries
+(`certctl-agent`, `certctl-server`, `certctl-cli`, `certctl-mcp-server` for
+`linux|darwin × amd64|arm64`) ship alongside a `checksums.txt`, per-binary
+SPDX-JSON SBOMs, Cosign signatures, and SLSA Level 3 provenance. Container
+images on `ghcr.io/shankar0123/certctl-{server,agent}` are built with
+`docker/build-push-action` `provenance: mode=max` + `sbom: true` and are
+additionally signed with Cosign at the image digest.
 
-### Authentication
-- Agent-to-server: API key (registered at agent creation)
-- API key and JWT auth types supported; `none` for demo/development
-- Auth type and secret configured via `CERTCTL_AUTH_TYPE` and `CERTCTL_AUTH_SECRET`
+All signatures use Cosign keyless OIDC; the signing identity is the
+release workflow running on a signed tag.
 
-### CORS
-- **Deny-by-default**: Empty `CERTCTL_CORS_ORIGINS` blocks all cross-origin requests. Operators must explicitly list allowed origins (comma-separated) or set `*` for development.
+**1. Verify SHA-256 checksums:**
 
-### Input Validation
-- Shell command injection prevention on all connector scripts (strict character whitelist, no metacharacters)
-- RFC 1123 domain name validation, base64url ACME token validation
-- SSRF protection in network scanner (loopback, link-local, multicast, broadcast ranges filtered)
-
-### Concurrency Safety
-- Scheduler loops protected by `sync/atomic.Bool` idempotency guards — duplicate ticks are skipped
-- Graceful shutdown waits up to 30 seconds for in-flight work before database close
-
-### Audit Trail
-- Immutable append-only log in PostgreSQL (`audit_events` table)
-- Every lifecycle action attributed to an actor with timestamp and resource reference
-- No update or delete operations on audit records
-- Every API call recorded to audit trail with method, path, actor, SHA-256 body hash, response status, and latency
-
-## API Overview
-
-93 endpoints under `/api/v1/` + `/.well-known/est/`, all returning JSON. List endpoints support pagination, sparse field selection (`?fields=`), sort (`?sort=-notAfter`), time-range filters, and cursor-based pagination. Full request/response schemas in the [OpenAPI 3.1 spec](api/openapi.yaml).
-
-### Key Endpoints
-```
-# Certificate lifecycle
-GET    /api/v1/certificates              List (filter, sort, cursor, sparse fields)
-POST   /api/v1/certificates/{id}/renew   Trigger renewal → 202 Accepted
-POST   /api/v1/certificates/{id}/revoke  Revoke with RFC 5280 reason code
-GET    /api/v1/crl/{issuer_id}           DER-encoded X.509 CRL
-GET    /api/v1/ocsp/{issuer_id}/{serial} OCSP responder (good/revoked/unknown)
-
-# Agent operations
-POST   /api/v1/agents/{id}/csr           Submit CSR for issuance
-GET    /api/v1/agents/{id}/work          Poll for pending deployment jobs
-POST   /api/v1/agents/{id}/discoveries   Submit certificate discovery scan results
-
-# Discovery & network scanning
-GET    /api/v1/discovered-certificates   List discovered certs (?agent_id, ?status)
-POST   /api/v1/discovered-certificates/{id}/claim  Link to managed cert
-POST   /api/v1/network-scan-targets/{id}/scan      Trigger immediate TLS scan
-
-# Jobs & approval
-POST   /api/v1/jobs/{id}/approve         Approve interactive renewal
-POST   /api/v1/jobs/{id}/reject          Reject interactive renewal
-
-# Post-deployment verification
-POST   /api/v1/jobs/{id}/verify          Submit TLS verification result
-GET    /api/v1/jobs/{id}/verification    Get verification status
-
-# Observability
-GET    /api/v1/metrics/prometheus         Prometheus exposition format
-GET    /api/v1/stats/summary             Dashboard summary
-
-# EST enrollment (RFC 7030)
-POST   /.well-known/est/simpleenroll     Device certificate enrollment
-GET    /.well-known/est/cacerts          CA certificate chain (PKCS#7)
+```bash
+sha256sum -c checksums.txt
 ```
 
-Full CRUD is available for certificates, agents, issuers, targets, teams, owners, policies, profiles, agent groups, notifications, and audit events. See the [OpenAPI spec](api/openapi.yaml) or [Feature Inventory](docs/features.md) for the complete endpoint reference.
+**2. Verify the Cosign signature on `checksums.txt`:**
+
+```bash
+cosign verify-blob \
+  --bundle checksums.txt.sigstore.json \
+  --certificate-identity-regexp '^https://github\.com/shankar0123/certctl/\.github/workflows/release\.yml@refs/tags/' \
+  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
+  checksums.txt
+```
+
+Every individual binary ships with its own `.sigstore.json` bundle
+(unified Sigstore bundle containing signature, certificate chain, and
+Rekor inclusion proof). Swap `checksums.txt` for any binary name and
+point `--bundle` at the matching `<binary>.sigstore.json` to verify it
+directly.
+
+**3. Verify SLSA Level 3 provenance on a binary:**
+
+```bash
+slsa-verifier verify-artifact \
+  --provenance-path multiple.intoto.jsonl \
+  --source-uri github.com/shankar0123/certctl \
+  --source-tag v2.1.0 \
+  certctl-agent-linux-amd64
+```
+
+**4. Verify a container image signature and its SBOM / provenance attestations:**
+
+```bash
+IMAGE=ghcr.io/shankar0123/certctl-server:v2.1.0
+
+cosign verify \
+  --certificate-identity-regexp '^https://github\.com/shankar0123/certctl/\.github/workflows/release\.yml@refs/tags/' \
+  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
+  "$IMAGE"
+
+# SBOM attestation (SPDX-JSON, emitted by docker/build-push-action)
+cosign verify-attestation --type spdxjson \
+  --certificate-identity-regexp '^https://github\.com/shankar0123/certctl/' \
+  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
+  "$IMAGE"
+
+# SLSA provenance attestation (docker/build-push-action `provenance: mode=max`)
+cosign verify-attestation --type slsaprovenance \
+  --certificate-identity-regexp '^https://github\.com/shankar0123/certctl/' \
+  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
+  "$IMAGE"
+```
+
+## Examples
+
+Pick the scenario closest to your setup and have it running in 2 minutes.
+
+| Example | Scenario |
+|---------|----------|
+| [`examples/acme-nginx/`](examples/acme-nginx/) | Let's Encrypt + NGINX, HTTP-01 challenges |
+| [`examples/acme-wildcard-dns01/`](examples/acme-wildcard-dns01/) | Wildcard certs via DNS-01 (Cloudflare hook included) |
+| [`examples/private-ca-traefik/`](examples/private-ca-traefik/) | Local CA (self-signed or sub-CA) + Traefik file provider |
+| [`examples/step-ca-haproxy/`](examples/step-ca-haproxy/) | Smallstep step-ca + HAProxy combined PEM |
+| [`examples/multi-issuer/`](examples/multi-issuer/) | ACME for public + Local CA for internal, one dashboard |
+
+Each directory contains a `docker-compose.yml` and a `README.md` explaining the scenario, prerequisites, and customization.
 
 ## CLI
 
@@ -397,44 +328,36 @@ Full CRUD is available for certificates, agents, issuers, targets, teams, owners
 go install github.com/shankar0123/certctl/cmd/cli@latest
 
 # Configure
-export CERTCTL_SERVER_URL=http://localhost:8443
+export CERTCTL_SERVER_URL=https://localhost:8443
 export CERTCTL_API_KEY=your-api-key
+export CERTCTL_SERVER_CA_BUNDLE_PATH=/path/to/ca.crt   # or --ca-bundle on the CLI; --insecure for dev self-signed
 
-# Certificate commands
+# Usage
 certctl-cli certs list                    # List all certificates
-certctl-cli certs get mc-api-prod         # Get certificate details
 certctl-cli certs renew mc-api-prod       # Trigger renewal
 certctl-cli certs revoke mc-api-prod --reason keyCompromise
-
-# Agent and job commands
 certctl-cli agents list                   # List registered agents
 certctl-cli jobs list                     # List jobs
-certctl-cli jobs cancel job-123           # Cancel a pending job
-
-# Operations
 certctl-cli status                        # Server health + summary stats
 certctl-cli import certs.pem              # Bulk import from PEM file
-
-# Output formats
 certctl-cli certs list --format json      # JSON output (default: table)
 ```
 
 ## MCP Server (AI Integration)
 
-certctl ships a standalone MCP (Model Context Protocol) server that exposes all 78 API endpoints as tools for AI assistants — Claude, Cursor, Windsurf, OpenClaw, VS Code Copilot, and any MCP-compatible client.
+certctl ships a standalone MCP (Model Context Protocol) server that exposes all 80 API endpoints as tools for AI assistants — Claude, Cursor, Windsurf, OpenClaw, VS Code Copilot, and any MCP-compatible client.
 
 ```bash
-# Install
+# Install and run
 go install github.com/shankar0123/certctl/cmd/mcp-server@latest
-
-# Configure
-export CERTCTL_SERVER_URL=http://localhost:8443
+export CERTCTL_SERVER_URL=https://localhost:8443
 export CERTCTL_API_KEY=your-api-key
-
-# Run (stdio transport — add to your AI client config)
+export CERTCTL_SERVER_CA_BUNDLE_PATH=/path/to/ca.crt   # required for self-signed bootstrap
 mcp-server
 ```
 
+The MCP server is env-vars-only — there are no CLI flags for TLS. If you must bypass verification for local development against a self-signed cert, set `CERTCTL_SERVER_TLS_INSECURE_SKIP_VERIFY=true`. Never set that in production.
+
 **Claude Desktop** (`claude_desktop_config.json`):
 ```json
 {
@@ -442,54 +365,59 @@ mcp-server
     "certctl": {
       "command": "mcp-server",
       "env": {
-        "CERTCTL_SERVER_URL": "http://localhost:8443",
-        "CERTCTL_API_KEY": "your-api-key"
+        "CERTCTL_SERVER_URL": "https://localhost:8443",
+        "CERTCTL_API_KEY": "your-api-key",
+        "CERTCTL_SERVER_CA_BUNDLE_PATH": "/path/to/ca.crt"
       }
     }
   }
 }
 ```
 
+## Development
+
+```bash
+make build              # Build server + agent binaries
+make test               # Run tests
+make lint               # golangci-lint (11 linters)
+govulncheck ./...       # Vulnerability scan
+make docker-up          # Start Docker Compose stack
+```
+
+CI runs on every push: `go vet`, `go test -race`, `golangci-lint`, `govulncheck`, and per-layer coverage thresholds (service 55%, handler 60%, domain 40%, middleware 30%). Frontend CI runs TypeScript type checking, Vitest tests, and Vite production build. 1,668 Go test functions with 625+ subtests, plus frontend test suite.
+
 ## Roadmap
 
-### V1 (v1.0.0)
+### V1 (v1.0.0) — Shipped
 Core lifecycle management — Local CA + ACME v2 issuers, NGINX target connector, agent-side key generation, API auth + rate limiting, React dashboard, CI pipeline with coverage gates, Docker images on GHCR.
 
-### V2: Operational Maturity
-
-18 milestones complete, 1100+ tests. See the [Feature Inventory](docs/features.md) for details on every capability.
-
-**What shipped (all ✅):**
-
-- **Issuers** — Sub-CA mode (enterprise root chains), ACME DNS-01 + DNS-PERSIST-01 (wildcard certs, any DNS provider), step-ca (native /sign API), OpenSSL/Custom CA (script-based signing)
-- **Revocation** — RFC 5280 reason codes, DER-encoded X.509 CRL, embedded OCSP responder, short-lived cert exemption
-- **Profiles + Ownership** — certificate profiles (key types, max TTL, crypto constraints), ownership tracking (owners + teams), dynamic agent groups, interactive renewal approval
-- **GUI Operations** — bulk renew/revoke/reassign, deployment timeline, inline policy editor, target wizard, audit export (CSV/JSON), short-lived credentials view
-- **Discovery** — filesystem scanning (PEM/DER) + network TLS scanning (CIDR ranges), triage workflow (claim/dismiss), network scan target management
-- **Observability** — Prometheus + JSON metrics, 5 stats API endpoints, dashboard charts (heatmap, trends, distribution), agent fleet overview, structured logging
-- **EST Server** (RFC 7030) — device/WiFi certificate enrollment, PKCS#7 wire format, configurable issuer + profile binding
-- **MCP Server** — 78 API operations as AI tools for Claude, Cursor, and any MCP-compatible client
-- **CLI** — 12 subcommands (list/get/renew/revoke certs, agents, jobs, import, status), JSON/table output
-- **Notifications** — Slack, Microsoft Teams, PagerDuty, OpsGenie connectors
-- **API Enhancements** — sparse fields, sort, time-range filters, cursor pagination, immutable API audit logging
-- **Compliance Mapping** — SOC 2 Type II, PCI-DSS 4.0, NIST SP 800-57 alignment guides
-
-- **Post-Deployment TLS Verification** — agent-side TLS probe confirms the target is serving the correct certificate by SHA-256 fingerprint match
-- **Traefik + Caddy Targets** — Traefik (file provider, auto-reload) and Caddy (Admin API hot-reload or file-based)
-
-**Coming next:**
-
-- **Certificate Export** (v2.1.x) — single-cert download in PFX/PKCS12, DER, and PEM formats
-- **S/MIME Support** (v2.2.x) — profile EKU constraints for S/MIME (emailProtection), code signing, and custom EKUs
+### V2: Operational Maturity — Shipped
+30+ milestones shipping enterprise-grade features for free. Sub-CA mode, ACME DNS-01/DNS-PERSIST-01/EAB/ARI (RFC 9773)/profile selection, step-ca, Vault PKI, DigiCert CertCentral, Sectigo SCM, Google CAS, AWS ACM PCA, Entrust, GlobalSign, EJBCA, OpenSSL/Custom CA issuers. NGINX, Apache, HAProxy, Traefik, Caddy, Envoy, Postfix, Dovecot, IIS (WinRM), F5 BIG-IP, SSH, Windows Certificate Store, Java Keystore, Kubernetes Secrets targets. EST server (RFC 7030) and SCEP server (RFC 8894) enrollment protocols. RFC 5280 revocation with DER CRL + embedded OCSP responder. Certificate profiles, ownership tracking, team assignment, agent groups, interactive approval workflows. Filesystem, network, and cloud secret manager (AWS SM, Azure KV, GCP SM) certificate discovery with triage GUI. Dynamic issuer/target configuration via GUI with AES-256-GCM encrypted storage. First-run onboarding wizard. Post-deployment TLS verification. Certificate export (PEM/PKCS#12). S/MIME support. Prometheus metrics. Scheduled certificate digest emails. Slack, Teams, PagerDuty, OpsGenie, SMTP notifications. MCP server (80 tools), CLI (12 commands), Helm chart. Compliance mapping (SOC 2, PCI-DSS 4.0, NIST SP 800-57). 5 turnkey deployment examples. Agent install script. Migration guides from certbot, acme.sh, and cert-manager. See the [Feature Inventory](docs/features.md) for details.
 
 ### V3: certctl Pro
+Enterprise capabilities for larger deployments are available in the commercial tier.
 
-Team access controls, identity provider integration, enterprise deployment targets, compliance and risk scoring, advanced fleet operations, event-driven architecture, advanced search, real-time operational views, and premium CA integrations.
-
-### V4+: Cloud, Scale & Passive Discovery
-Passive network discovery (TLS listener), Kubernetes integration (cert-manager external issuer, Secrets target), cloud infrastructure targets (AWS ALB/ACM, Azure Key Vault), extended CA support (Vault PKI, Google CAS, EJBCA), and platform-scale features (Terraform provider, multi-tenancy, HSM support).
+### V4+: Cloud & Scale
+Kubernetes cert-manager external issuer, cloud infrastructure targets, extended CA support, and platform-scale features.
 
 ## License
 
-Certctl is licensed under the [Business Source License 1.1](LICENSE). The source code is publicly available and free to use, modify, and self-host. The one restriction: you may not offer certctl as a managed/hosted certificate management service to third parties.
+Certctl is licensed under the [Business Source License 1.1](LICENSE). The source code is publicly available and free to use, modify, and self-host. The one restriction: you may not use certctl's certificate management functionality as part of a commercial offering to third parties, whether hosted, managed, embedded, bundled, or integrated.
 
+For licensing inquiries: certctl@proton.me
+
+## Dependencies
+
+Backend dependency footprint is auditable on demand:
+
+```
+go list -m all | wc -l   # total module count (direct + transitive)
+go mod why <path>        # explain why a particular module is pulled in
+govulncheck ./...        # vulnerability scan (CI runs this on every commit)
+```
+
+The release-time SBOM is published as a syft-produced cyclonedx file alongside each release artifact in `.github/workflows/release.yml`.
+
+---
+
+If certctl solves a problem you have, [star the repo](https://github.com/shankar0123/certctl) to help others find it. Questions, bugs, or feature requests — [open an issue](https://github.com/shankar0123/certctl/issues).

cmd/agent/dispatch_test.go

@@ -0,0 +1,638 @@
+package main
+
+import (
+	"context"
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/x509"
+	"crypto/x509/pkix"
+	"encoding/json"
+	"encoding/pem"
+	"io"
+	"log/slog"
+	"math/big"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync/atomic"
+	"testing"
+	"time"
+)
+
+// Bundle 0.7-extended: cmd/agent dispatch coverage for executeCSRJob,
+// executeDeploymentJob, verifyAndReportDeployment, markRetired, getEnvDefault,
+// getEnvBoolDefault — the previously-uncovered code paths flagged by the
+// audit's per-function coverage report.
+//
+// Strategy: same httptest-backed pattern as the existing agent_test.go
+// (Heartbeat / PollWork tests). Each test:
+//   - constructs a mock control-plane HTTP server (httptest.NewServer)
+//   - configures an Agent pointing at that server via NewAgent
+//   - invokes the function under test
+//   - asserts on the requests the mock server received
+
+// ─────────────────────────────────────────────────────────────────────────────
+// executeCSRJob
+// ─────────────────────────────────────────────────────────────────────────────
+
+func TestAgent_ExecuteCSRJob_HappyPath(t *testing.T) {
+	keyDir := t.TempDir()
+	if err := os.Chmod(keyDir, 0700); err != nil {
+		t.Fatalf("chmod keyDir: %v", err)
+	}
+
+	var csrSubmitted atomic.Bool
+	var statusUpdates atomic.Int32
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch {
+		case strings.HasSuffix(r.URL.Path, "/csr") && r.Method == http.MethodPost:
+			csrSubmitted.Store(true)
+			var body map[string]string
+			_ = json.NewDecoder(r.Body).Decode(&body)
+			if body["csr_pem"] == "" || !strings.Contains(body["csr_pem"], "CERTIFICATE REQUEST") {
+				t.Errorf("CSR submission missing PEM body: %v", body)
+			}
+			if body["certificate_id"] != "mc-test-cert" {
+				t.Errorf("CSR submission missing certificate_id: %v", body)
+			}
+			w.WriteHeader(http.StatusAccepted)
+		case strings.HasSuffix(r.URL.Path, "/status") && r.Method == http.MethodPost:
+			statusUpdates.Add(1)
+			w.WriteHeader(http.StatusOK)
+		default:
+			t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path)
+			w.WriteHeader(http.StatusNotFound)
+		}
+	}))
+	defer server.Close()
+
+	cfg := &AgentConfig{
+		ServerURL: server.URL,
+		APIKey:    "test-key",
+		AgentID:   "a-test",
+		KeyDir:    keyDir,
+	}
+	agent, err := NewAgent(cfg, slog.New(slog.NewTextHandler(io.Discard, nil)))
+	if err != nil {
+		t.Fatalf("NewAgent: %v", err)
+	}
+
+	job := JobItem{
+		ID:            "j-csr-1",
+		CertificateID: "mc-test-cert",
+		Type:          "csr",
+		CommonName:    "test.example.com",
+		SANs:          []string{"test.example.com", "alt.example.com", "alice@example.com"},
+	}
+
+	agent.executeCSRJob(context.Background(), job)
+
+	if !csrSubmitted.Load() {
+		t.Errorf("expected CSR to be submitted to control plane")
+	}
+
+	// Key file should exist with mode 0600
+	keyPath := filepath.Join(keyDir, "mc-test-cert.key")
+	info, err := os.Stat(keyPath)
+	if err != nil {
+		t.Fatalf("expected key file at %s: %v", keyPath, err)
+	}
+	if info.Mode().Perm() != 0600 {
+		t.Errorf("expected key file mode 0600, got %v", info.Mode().Perm())
+	}
+
+	// Read back and verify it parses as an ECDSA key
+	keyPEM, err := os.ReadFile(keyPath)
+	if err != nil {
+		t.Fatalf("read key file: %v", err)
+	}
+	block, _ := pem.Decode(keyPEM)
+	if block == nil || block.Type != "EC PRIVATE KEY" {
+		t.Errorf("expected EC PRIVATE KEY PEM, got %v", block)
+	}
+}
+
+func TestAgent_ExecuteCSRJob_EmptyCommonName_ReportsFailed(t *testing.T) {
+	keyDir := t.TempDir()
+	if err := os.Chmod(keyDir, 0700); err != nil {
+		t.Fatalf("chmod keyDir: %v", err)
+	}
+
+	var lastStatus atomic.Value
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if strings.HasSuffix(r.URL.Path, "/status") && r.Method == http.MethodPost {
+			var body map[string]string
+			_ = json.NewDecoder(r.Body).Decode(&body)
+			lastStatus.Store(body["status"])
+		}
+		w.WriteHeader(http.StatusOK)
+	}))
+	defer server.Close()
+
+	cfg := &AgentConfig{
+		ServerURL: server.URL,
+		APIKey:    "test-key",
+		AgentID:   "a-test",
+		KeyDir:    keyDir,
+	}
+	agent, _ := NewAgent(cfg, slog.New(slog.NewTextHandler(io.Discard, nil)))
+
+	job := JobItem{
+		ID:            "j-csr-empty-cn",
+		CertificateID: "mc-empty-cn",
+		Type:          "csr",
+		CommonName:    "", // empty CN — should be rejected
+	}
+
+	agent.executeCSRJob(context.Background(), job)
+
+	if got := lastStatus.Load(); got != "Failed" {
+		t.Errorf("expected last status 'Failed', got %v", got)
+	}
+}
+
+func TestAgent_ExecuteCSRJob_CSRSubmissionRejected_ReportsFailed(t *testing.T) {
+	keyDir := t.TempDir()
+	if err := os.Chmod(keyDir, 0700); err != nil {
+		t.Fatalf("chmod keyDir: %v", err)
+	}
+
+	var lastStatus atomic.Value
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch {
+		case strings.HasSuffix(r.URL.Path, "/csr") && r.Method == http.MethodPost:
+			// Server rejects the CSR with 400 Bad Request
+			w.WriteHeader(http.StatusBadRequest)
+			_, _ = w.Write([]byte(`{"error":"CSR validation failed"}`))
+		case strings.HasSuffix(r.URL.Path, "/status") && r.Method == http.MethodPost:
+			var body map[string]string
+			_ = json.NewDecoder(r.Body).Decode(&body)
+			lastStatus.Store(body["status"])
+			w.WriteHeader(http.StatusOK)
+		default:
+			w.WriteHeader(http.StatusNotFound)
+		}
+	}))
+	defer server.Close()
+
+	cfg := &AgentConfig{
+		ServerURL: server.URL,
+		APIKey:    "test-key",
+		AgentID:   "a-test",
+		KeyDir:    keyDir,
+	}
+	agent, _ := NewAgent(cfg, slog.New(slog.NewTextHandler(io.Discard, nil)))
+
+	job := JobItem{
+		ID:            "j-csr-rejected",
+		CertificateID: "mc-rejected",
+		Type:          "csr",
+		CommonName:    "rejected.example.com",
+	}
+
+	agent.executeCSRJob(context.Background(), job)
+
+	if got := lastStatus.Load(); got != "Failed" {
+		t.Errorf("expected last status 'Failed' after CSR rejection, got %v", got)
+	}
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// executeDeploymentJob
+// ─────────────────────────────────────────────────────────────────────────────
+
+// generateTestCertAndKey builds an ephemeral self-signed cert + ECDSA P-256 key
+// for use as test fixture data in deployment tests.
+func generateTestCertAndKey(t *testing.T, cn string) (certPEM, keyPEM string) {
+	t.Helper()
+	priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+	if err != nil {
+		t.Fatalf("GenerateKey: %v", err)
+	}
+	template := &x509.Certificate{
+		SerialNumber: big.NewInt(1),
+		Subject:      pkix.Name{CommonName: cn},
+		NotBefore:    time.Now().Add(-1 * time.Hour),
+		NotAfter:     time.Now().Add(24 * time.Hour),
+		KeyUsage:     x509.KeyUsageDigitalSignature,
+	}
+	certDER, err := x509.CreateCertificate(rand.Reader, template, template, &priv.PublicKey, priv)
+	if err != nil {
+		t.Fatalf("CreateCertificate: %v", err)
+	}
+	certPEM = string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: certDER}))
+	keyDER, err := x509.MarshalECPrivateKey(priv)
+	if err != nil {
+		t.Fatalf("MarshalECPrivateKey: %v", err)
+	}
+	keyPEM = string(pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER}))
+	return certPEM, keyPEM
+}
+
+func TestAgent_ExecuteDeploymentJob_FetchFails_ReportsFailed(t *testing.T) {
+	keyDir := t.TempDir()
+	if err := os.Chmod(keyDir, 0700); err != nil {
+		t.Fatalf("chmod keyDir: %v", err)
+	}
+
+	var lastStatus atomic.Value
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch {
+		case strings.Contains(r.URL.Path, "/certificates/") && r.Method == http.MethodGet:
+			// Fail the certificate fetch
+			w.WriteHeader(http.StatusInternalServerError)
+		case strings.HasSuffix(r.URL.Path, "/status") && r.Method == http.MethodPost:
+			var body map[string]string
+			_ = json.NewDecoder(r.Body).Decode(&body)
+			lastStatus.Store(body["status"])
+			w.WriteHeader(http.StatusOK)
+		default:
+			w.WriteHeader(http.StatusOK)
+		}
+	}))
+	defer server.Close()
+
+	cfg := &AgentConfig{
+		ServerURL: server.URL,
+		APIKey:    "test-key",
+		AgentID:   "a-test",
+		KeyDir:    keyDir,
+	}
+	agent, _ := NewAgent(cfg, slog.New(slog.NewTextHandler(io.Discard, nil)))
+
+	job := JobItem{
+		ID:            "j-deploy-fetch-fail",
+		CertificateID: "mc-fetch-fail",
+		Type:          "deployment",
+		TargetType:    "nginx",
+	}
+
+	agent.executeDeploymentJob(context.Background(), job)
+
+	if got := lastStatus.Load(); got != "Failed" {
+		t.Errorf("expected status 'Failed' after fetch failure, got %v", got)
+	}
+}
+
+func TestAgent_ExecuteDeploymentJob_KeyMissing_ReportsFailed(t *testing.T) {
+	keyDir := t.TempDir()
+	if err := os.Chmod(keyDir, 0700); err != nil {
+		t.Fatalf("chmod keyDir: %v", err)
+	}
+
+	certPEM, _ := generateTestCertAndKey(t, "deploy-test.example.com")
+	// Note: key file is intentionally NOT written to keyDir — exercises the
+	// "local private key missing" failure path in executeDeploymentJob.
+
+	var lastStatus atomic.Value
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch {
+		case strings.Contains(r.URL.Path, "/certificates/") && r.Method == http.MethodGet:
+			w.Header().Set("Content-Type", "application/json")
+			_ = json.NewEncoder(w).Encode(map[string]string{
+				"id":          "mc-no-key",
+				"common_name": "deploy-test.example.com",
+				"pem_content": certPEM,
+			})
+		case strings.HasSuffix(r.URL.Path, "/status") && r.Method == http.MethodPost:
+			var body map[string]string
+			_ = json.NewDecoder(r.Body).Decode(&body)
+			lastStatus.Store(body["status"])
+			w.WriteHeader(http.StatusOK)
+		default:
+			w.WriteHeader(http.StatusOK)
+		}
+	}))
+	defer server.Close()
+
+	cfg := &AgentConfig{
+		ServerURL: server.URL,
+		APIKey:    "test-key",
+		AgentID:   "a-test",
+		KeyDir:    keyDir,
+	}
+	agent, _ := NewAgent(cfg, slog.New(slog.NewTextHandler(io.Discard, nil)))
+
+	job := JobItem{
+		ID:            "j-deploy-no-key",
+		CertificateID: "mc-no-key",
+		Type:          "deployment",
+		TargetType:    "nginx",
+	}
+
+	agent.executeDeploymentJob(context.Background(), job)
+
+	if got := lastStatus.Load(); got != "Failed" {
+		t.Errorf("expected status 'Failed' after key-missing, got %v", got)
+	}
+}
+
+func TestAgent_ExecuteDeploymentJob_UnknownTargetType_ReportsFailed(t *testing.T) {
+	keyDir := t.TempDir()
+	if err := os.Chmod(keyDir, 0700); err != nil {
+		t.Fatalf("chmod keyDir: %v", err)
+	}
+
+	certPEM, keyPEM := generateTestCertAndKey(t, "deploy-test.example.com")
+	keyPath := filepath.Join(keyDir, "mc-unknown-tgt.key")
+	if err := os.WriteFile(keyPath, []byte(keyPEM), 0600); err != nil {
+		t.Fatalf("WriteFile key: %v", err)
+	}
+
+	var lastStatus atomic.Value
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch {
+		case strings.Contains(r.URL.Path, "/certificates/") && r.Method == http.MethodGet:
+			w.Header().Set("Content-Type", "application/json")
+			_ = json.NewEncoder(w).Encode(map[string]string{
+				"id":          "mc-unknown-tgt",
+				"common_name": "deploy-test.example.com",
+				"pem_content": certPEM,
+			})
+		case strings.HasSuffix(r.URL.Path, "/status") && r.Method == http.MethodPost:
+			var body map[string]string
+			_ = json.NewDecoder(r.Body).Decode(&body)
+			lastStatus.Store(body["status"])
+			w.WriteHeader(http.StatusOK)
+		default:
+			w.WriteHeader(http.StatusOK)
+		}
+	}))
+	defer server.Close()
+
+	cfg := &AgentConfig{
+		ServerURL: server.URL,
+		APIKey:    "test-key",
+		AgentID:   "a-test",
+		KeyDir:    keyDir,
+	}
+	agent, _ := NewAgent(cfg, slog.New(slog.NewTextHandler(io.Discard, nil)))
+
+	job := JobItem{
+		ID:            "j-unknown-target",
+		CertificateID: "mc-unknown-tgt",
+		Type:          "deployment",
+		TargetType:    "frobnicator-9000", // unknown connector type
+	}
+
+	agent.executeDeploymentJob(context.Background(), job)
+
+	if got := lastStatus.Load(); got != "Failed" {
+		t.Errorf("expected status 'Failed' after unknown target type, got %v", got)
+	}
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// markRetired — single-shot retirement signal
+// ─────────────────────────────────────────────────────────────────────────────
+
+func TestAgent_MarkRetired_ClosesSignalOnce(t *testing.T) {
+	cfg := &AgentConfig{
+		ServerURL: "http://example.invalid",
+		APIKey:    "k",
+		AgentID:   "a-retired-test",
+	}
+	agent, _ := NewAgent(cfg, slog.New(slog.NewTextHandler(io.Discard, nil)))
+
+	// First mark — channel should close
+	agent.markRetired("test-source-1", 410, "agent retired")
+	select {
+	case <-agent.retiredSignal:
+		// expected — closed channel reads return zero immediately
+	case <-time.After(100 * time.Millisecond):
+		t.Fatalf("expected retiredSignal to be closed after markRetired")
+	}
+
+	// Second mark — must not panic (sync.Once guards the close)
+	defer func() {
+		if r := recover(); r != nil {
+			t.Errorf("second markRetired panicked: %v", r)
+		}
+	}()
+	agent.markRetired("test-source-2", 410, "agent retired again")
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// getEnvDefault / getEnvBoolDefault
+// ─────────────────────────────────────────────────────────────────────────────
+
+func TestGetEnvDefault_FallsBackToDefault(t *testing.T) {
+	t.Setenv("TESTONLY_AGENT_NONEXISTENT_VAR", "")
+	got := getEnvDefault("TESTONLY_AGENT_NONEXISTENT_VAR", "fallback")
+	if got != "fallback" {
+		t.Errorf("expected fallback, got %q", got)
+	}
+}
+
+func TestGetEnvDefault_UsesEnvWhenSet(t *testing.T) {
+	t.Setenv("TESTONLY_AGENT_VAR", "from-env")
+	got := getEnvDefault("TESTONLY_AGENT_VAR", "fallback")
+	if got != "from-env" {
+		t.Errorf("expected from-env, got %q", got)
+	}
+}
+
+func TestGetEnvBoolDefault_TruthyValues(t *testing.T) {
+	for _, v := range []string{"1", "t", "true", "yes", "on", "TRUE", "True"} {
+		t.Run(v, func(t *testing.T) {
+			t.Setenv("TESTONLY_AGENT_BOOL", v)
+			if !getEnvBoolDefault("TESTONLY_AGENT_BOOL", false) {
+				t.Errorf("expected true for %q", v)
+			}
+		})
+	}
+}
+
+func TestGetEnvBoolDefault_FalsyValues(t *testing.T) {
+	for _, v := range []string{"0", "f", "false", "no", "off"} {
+		t.Run(v, func(t *testing.T) {
+			t.Setenv("TESTONLY_AGENT_BOOL", v)
+			if getEnvBoolDefault("TESTONLY_AGENT_BOOL", true) {
+				t.Errorf("expected false for %q", v)
+			}
+		})
+	}
+}
+
+func TestGetEnvBoolDefault_UnrecognizedReturnsDefault(t *testing.T) {
+	t.Setenv("TESTONLY_AGENT_BOOL", "frobnicate")
+	if !getEnvBoolDefault("TESTONLY_AGENT_BOOL", true) {
+		t.Errorf("expected default(true) for unrecognized value")
+	}
+}
+
+func TestGetEnvBoolDefault_EmptyReturnsDefault(t *testing.T) {
+	t.Setenv("TESTONLY_AGENT_BOOL", "")
+	if !getEnvBoolDefault("TESTONLY_AGENT_BOOL", true) {
+		t.Errorf("expected default(true) for empty value")
+	}
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// Run() — graceful shutdown via context cancellation
+// ─────────────────────────────────────────────────────────────────────────────
+
+func TestAgent_Run_ContextCancelExitsCleanly(t *testing.T) {
+	keyDir := t.TempDir()
+	if err := os.Chmod(keyDir, 0700); err != nil {
+		t.Fatalf("chmod keyDir: %v", err)
+	}
+
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch r.URL.Path {
+		case "/api/v1/agents/a-run-test/heartbeat":
+			w.WriteHeader(http.StatusOK)
+		case "/api/v1/agents/a-run-test/work":
+			w.Header().Set("Content-Type", "application/json")
+			_ = json.NewEncoder(w).Encode(WorkResponse{Jobs: []JobItem{}, Count: 0})
+		default:
+			w.WriteHeader(http.StatusOK)
+		}
+	}))
+	defer server.Close()
+
+	cfg := &AgentConfig{
+		ServerURL: server.URL,
+		APIKey:    "test-key",
+		AgentID:   "a-run-test",
+		KeyDir:    keyDir,
+	}
+	agent, err := NewAgent(cfg, slog.New(slog.NewTextHandler(io.Discard, nil)))
+	if err != nil {
+		t.Fatalf("NewAgent: %v", err)
+	}
+	// Speed up tickers so the test exits in <500ms
+	agent.heartbeatInterval = 50 * time.Millisecond
+	agent.pollInterval = 50 * time.Millisecond
+	agent.discoveryInterval = 24 * time.Hour
+
+	ctx, cancel := context.WithCancel(context.Background())
+	errCh := make(chan error, 1)
+	go func() {
+		errCh <- agent.Run(ctx)
+	}()
+
+	// Let one heartbeat + poll fire, then cancel.
+	time.Sleep(100 * time.Millisecond)
+	cancel()
+
+	select {
+	case err := <-errCh:
+		if err != context.Canceled {
+			t.Errorf("expected context.Canceled, got %v", err)
+		}
+	case <-time.After(2 * time.Second):
+		t.Fatalf("Run did not exit within 2s after cancellation")
+	}
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// verifyAndReportDeployment
+// ─────────────────────────────────────────────────────────────────────────────
+
+func TestAgent_VerifyAndReportDeployment_ProbeFailure_ReportsError(t *testing.T) {
+	// Server with no TLS listener at the target — probe will fail.
+	var verificationReported atomic.Bool
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if strings.Contains(r.URL.Path, "/verify") || strings.Contains(r.URL.Path, "/verification") {
+			verificationReported.Store(true)
+			w.WriteHeader(http.StatusOK)
+			return
+		}
+		w.WriteHeader(http.StatusOK)
+	}))
+	defer server.Close()
+
+	cfg := &AgentConfig{
+		ServerURL: server.URL,
+		APIKey:    "test-key",
+		AgentID:   "a-test",
+	}
+	agent, _ := NewAgent(cfg, slog.New(slog.NewTextHandler(io.Discard, nil)))
+
+	tgtID := "tgt-test"
+	job := JobItem{
+		ID:       "j-verify",
+		TargetID: &tgtID,
+	}
+
+	// Probe a closed port — will fail quickly.
+	ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
+	defer cancel()
+
+	// Should not panic; failure surfaces via reportVerificationResult.
+	agent.verifyAndReportDeployment(ctx, job, "127.0.0.1", 1, "")
+	// Test passes if no panic.
+}
+
+func TestAgent_VerifyAndReportDeployment_NilTargetID_LogsAndReturns(t *testing.T) {
+	cfg := &AgentConfig{
+		ServerURL: "http://example.invalid",
+		APIKey:    "test-key",
+		AgentID:   "a-test",
+	}
+	agent, _ := NewAgent(cfg, slog.New(slog.NewTextHandler(io.Discard, nil)))
+
+	job := JobItem{
+		ID:       "j-no-tgt",
+		TargetID: nil, // nil target — should short-circuit cleanly
+	}
+
+	ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond)
+	defer cancel()
+
+	// Should not panic and should return without making any HTTP call.
+	agent.verifyAndReportDeployment(ctx, job, "127.0.0.1", 1, "")
+}
+
+func TestAgent_Run_RetiredSignalExitsWithErrAgentRetired(t *testing.T) {
+	keyDir := t.TempDir()
+	if err := os.Chmod(keyDir, 0700); err != nil {
+		t.Fatalf("chmod keyDir: %v", err)
+	}
+
+	// Server returns 410 Gone on heartbeat — the documented retirement signal.
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch r.URL.Path {
+		case "/api/v1/agents/a-retired/heartbeat":
+			w.WriteHeader(http.StatusGone)
+			_, _ = w.Write([]byte(`{"error":"agent retired"}`))
+		case "/api/v1/agents/a-retired/work":
+			w.WriteHeader(http.StatusGone)
+		default:
+			w.WriteHeader(http.StatusGone)
+		}
+	}))
+	defer server.Close()
+
+	cfg := &AgentConfig{
+		ServerURL: server.URL,
+		APIKey:    "test-key",
+		AgentID:   "a-retired",
+		KeyDir:    keyDir,
+	}
+	agent, _ := NewAgent(cfg, slog.New(slog.NewTextHandler(io.Discard, nil)))
+	agent.heartbeatInterval = 30 * time.Millisecond
+	agent.pollInterval = 30 * time.Millisecond
+	agent.discoveryInterval = 24 * time.Hour
+
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+
+	errCh := make(chan error, 1)
+	go func() {
+		errCh <- agent.Run(ctx)
+	}()
+
+	select {
+	case err := <-errCh:
+		if err != ErrAgentRetired {
+			t.Errorf("expected ErrAgentRetired, got %v", err)
+		}
+	case <-time.After(2 * time.Second):
+		t.Fatalf("Run did not surface ErrAgentRetired within 2s")
+	}
+}

cmd/agent/keymem.go

@@ -0,0 +1,73 @@
+package main
+
+import (
+	"crypto/ecdsa"
+	"crypto/x509"
+	"fmt"
+	"os"
+	"path/filepath"
+)
+
+// Bundle-9 / Audit L-002 + L-003 (agent edition).
+//
+// The agent generates an ECDSA P-256 key locally and writes it to disk with
+// mode 0600 in a directory it expects to be 0700. The duplication of the
+// local-issuer helpers (instead of importing from internal/...) is deliberate:
+//
+//   - cmd/agent is a separate binary with its own threat model (runs on every
+//     deployment target, not just the control plane). Coupling it to
+//     internal/connector/issuer/local would pull deployment-target footprint
+//     into a connector that's only relevant on the server.
+//   - The behavior is small and self-contained; copy-paste is cheaper than
+//     a refactor that introduces an internal/keystore package.
+//
+// If a third call site emerges, lift these into internal/keystore.
+
+// marshalAgentKeyAndZeroize marshals an ECDSA private key to DER and invokes
+// onDER with the bytes; the buffer is zeroized via builtin clear() after
+// onDER returns. Caller must NOT retain the slice.
+func marshalAgentKeyAndZeroize(priv *ecdsa.PrivateKey, onDER func([]byte) error) error {
+	if priv == nil {
+		return fmt.Errorf("marshalAgentKeyAndZeroize: nil private key")
+	}
+	der, err := x509.MarshalECPrivateKey(priv)
+	if err != nil {
+		return fmt.Errorf("marshal EC private key: %w", err)
+	}
+	defer clear(der)
+	return onDER(der)
+}
+
+// ensureAgentKeyDirSecure creates dir (and ancestors) with mode 0700 or
+// asserts an existing dir is owner-only. If a pre-existing dir is more
+// permissive than 0700 we tighten it to 0700 (logging-free; this is a
+// startup-style invariant, not a per-request check).
+func ensureAgentKeyDirSecure(dir string) error {
+	if dir == "" || dir == "." || dir == "/" {
+		return fmt.Errorf("ensureAgentKeyDirSecure: refuse empty/root dir %q", dir)
+	}
+	clean := filepath.Clean(dir)
+	info, err := os.Stat(clean)
+	switch {
+	case os.IsNotExist(err):
+		if mkErr := os.MkdirAll(clean, 0o700); mkErr != nil {
+			return fmt.Errorf("create agent key dir %q: %w", clean, mkErr)
+		}
+		info, err = os.Stat(clean)
+		if err != nil {
+			return fmt.Errorf("stat newly-created agent key dir %q: %w", clean, err)
+		}
+		fallthrough
+	case err == nil:
+		mode := info.Mode().Perm()
+		if mode == 0o700 || mode&0o077 == 0 {
+			return nil
+		}
+		if chmodErr := os.Chmod(clean, 0o700); chmodErr != nil {
+			return fmt.Errorf("tighten agent key dir %q from %#o to 0700: %w", clean, mode, chmodErr)
+		}
+		return nil
+	default:
+		return fmt.Errorf("stat agent key dir %q: %w", clean, err)
+	}
+}

cmd/agent/keymem_test.go

@@ -0,0 +1,718 @@
+package main
+
+// Bundle 0.7 (Coverage Audit Closure) — cmd/agent key-handling regression coverage.
+//
+// Closes finding C-008 (CRTCTL-COVAUDIT-2026-04-27-0034). The two functions in
+// keymem.go are the agent's defense-in-depth for ECDSA P-256 private-key
+// memory hygiene (Bundle 9 / Audit L-002 + L-003 — agent edition). They
+// shipped with regression-test coverage of 0.0% / 11.1% respectively. This
+// file pins:
+//
+//   - marshalAgentKeyAndZeroize: rejects nil keys, propagates onDER errors,
+//     and ZEROIZES the DER backing buffer after onDER returns regardless of
+//     whether onDER errored.  The zeroization invariant is verified observably
+//     (capture the slice header inside onDER, then assert every byte is 0x00
+//     after the function returns) — NOT just asserted in prose.
+//
+//   - ensureAgentKeyDirSecure: refuses empty / "." / "/", creates missing
+//     dirs with mode 0700 (incl. nested ancestors), accepts existing 0700
+//     and any owner-only-no-write mode (mode&0o077 == 0), tightens any other
+//     mode to 0700, normalizes paths via filepath.Clean, is idempotent, is
+//     safe under concurrent invocation, and propagates the documented error
+//     messages from os.Stat / os.MkdirAll / os.Chmod failures.
+
+import (
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"errors"
+	"fmt"
+	"os"
+	"path/filepath"
+	"runtime"
+	"strings"
+	"sync"
+	"testing"
+)
+
+// ---------------------------------------------------------------------------
+// helpers
+// ---------------------------------------------------------------------------
+
+func mustGenAgentECDSAKey(t *testing.T) *ecdsa.PrivateKey {
+	t.Helper()
+	k, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+	if err != nil {
+		t.Fatalf("ecdsa.GenerateKey: %v", err)
+	}
+	return k
+}
+
+// ---------------------------------------------------------------------------
+// marshalAgentKeyAndZeroize
+// ---------------------------------------------------------------------------
+
+// TestMarshalAgentKeyAndZeroize_HappyPath confirms onDER receives well-formed
+// DER bytes that the caller can use during the closure (e.g. to PEM-encode).
+func TestMarshalAgentKeyAndZeroize_HappyPath(t *testing.T) {
+	k := mustGenAgentECDSAKey(t)
+	called := false
+	err := marshalAgentKeyAndZeroize(k, func(der []byte) error {
+		called = true
+		if len(der) == 0 {
+			t.Fatalf("der is empty inside onDER")
+		}
+		// First byte of an ECPrivateKey DER blob is the ASN.1 SEQUENCE tag 0x30.
+		if der[0] != 0x30 {
+			t.Errorf("expected DER to start with SEQUENCE tag 0x30, got %#x", der[0])
+		}
+		return nil
+	})
+	if err != nil {
+		t.Fatalf("marshalAgentKeyAndZeroize: %v", err)
+	}
+	if !called {
+		t.Fatal("onDER was never invoked")
+	}
+}
+
+// TestMarshalAgentKeyAndZeroize_NilKey confirms the early-return guard;
+// onDER must NOT be invoked when priv is nil.
+func TestMarshalAgentKeyAndZeroize_NilKey(t *testing.T) {
+	called := false
+	err := marshalAgentKeyAndZeroize(nil, func([]byte) error {
+		called = true
+		return nil
+	})
+	if err == nil {
+		t.Fatal("expected error on nil key")
+	}
+	if !strings.Contains(err.Error(), "nil private key") {
+		t.Errorf("expected error mentioning %q, got: %v", "nil private key", err)
+	}
+	if called {
+		t.Error("onDER must not be invoked when priv is nil")
+	}
+}
+
+// TestMarshalAgentKeyAndZeroize_OnDERReturnsError confirms upstream errors
+// are propagated verbatim via errors.Is.
+func TestMarshalAgentKeyAndZeroize_OnDERReturnsError(t *testing.T) {
+	k := mustGenAgentECDSAKey(t)
+	sentinel := errors.New("simulated downstream failure")
+	got := marshalAgentKeyAndZeroize(k, func([]byte) error { return sentinel })
+	if !errors.Is(got, sentinel) {
+		t.Errorf("expected upstream sentinel via errors.Is; got: %v", got)
+	}
+}
+
+// TestMarshalAgentKeyAndZeroize_BackingBufferZeroizedAfterReturn is the
+// CRITICAL invariant test. It captures the slice header (NOT a deep copy)
+// inside onDER and re-inspects after the function returns. Because Go slices
+// share their backing array, the captured slice observes the zeroization
+// performed by `defer clear(der)` in marshalAgentKeyAndZeroize.
+//
+// A future refactor that drops the `defer clear(der)` would break this test
+// even if HappyPath / NilKey / OnDERReturnsError still pass.
+func TestMarshalAgentKeyAndZeroize_BackingBufferZeroizedAfterReturn(t *testing.T) {
+	k := mustGenAgentECDSAKey(t)
+	var captured []byte
+	err := marshalAgentKeyAndZeroize(k, func(der []byte) error {
+		// SHARE the backing array — do NOT take a defensive copy.
+		captured = der
+		if len(der) == 0 {
+			t.Fatal("der is empty inside onDER")
+		}
+		// Sanity check: while still inside onDER, the bytes are live
+		// (defer clear has NOT run yet).
+		nonZero := false
+		for _, b := range der {
+			if b != 0 {
+				nonZero = true
+				break
+			}
+		}
+		if !nonZero {
+			t.Fatal("DER is all-zero INSIDE onDER; that should be impossible (clear hasn't run yet)")
+		}
+		return nil
+	})
+	if err != nil {
+		t.Fatalf("marshalAgentKeyAndZeroize: %v", err)
+	}
+	if len(captured) == 0 {
+		t.Fatal("captured slice is empty post-return")
+	}
+	// After return, defer clear(der) has run. The captured slice shares the
+	// backing array, so every byte must read 0x00.
+	for i, b := range captured {
+		if b != 0 {
+			t.Errorf("captured[%d] = %#x; expected 0x00 (zeroized)", i, b)
+		}
+	}
+}
+
+// TestMarshalAgentKeyAndZeroize_BufferZeroizedEvenOnError confirms the
+// `defer clear(der)` fires regardless of onDER's return — the security
+// invariant is "buffer is always zeroized after the function returns,"
+// happy path or error path.
+func TestMarshalAgentKeyAndZeroize_BufferZeroizedEvenOnError(t *testing.T) {
+	k := mustGenAgentECDSAKey(t)
+	sentinel := errors.New("upstream boom")
+	var captured []byte
+	gotErr := marshalAgentKeyAndZeroize(k, func(der []byte) error {
+		captured = der // share backing array
+		return sentinel
+	})
+	if !errors.Is(gotErr, sentinel) {
+		t.Fatalf("expected sentinel via errors.Is, got: %v", gotErr)
+	}
+	if len(captured) == 0 {
+		t.Fatal("captured slice empty post-return")
+	}
+	for i, b := range captured {
+		if b != 0 {
+			t.Errorf("captured[%d] = %#x; expected 0x00 (defer clear must run on error path)", i, b)
+		}
+	}
+}
+
+// TestMarshalAgentKeyAndZeroize_ContractViolatorSeesZeros frames the same
+// observation as a defense-in-depth contract test. The docstring states
+// "Caller must NOT retain the slice." If a caller violates that contract
+// and reads the slice after onDER returns, they observe zeros — not the
+// private scalar. This test pins that defense.
+func TestMarshalAgentKeyAndZeroize_ContractViolatorSeesZeros(t *testing.T) {
+	k := mustGenAgentECDSAKey(t)
+	var leaked []byte // simulating a buggy caller that retains the slice
+	err := marshalAgentKeyAndZeroize(k, func(der []byte) error {
+		leaked = der
+		return nil
+	})
+	if err != nil {
+		t.Fatalf("marshalAgentKeyAndZeroize: %v", err)
+	}
+	// The contract violator now reads from `leaked`. Defense-in-depth: it's zeros.
+	for i, b := range leaked {
+		if b != 0 {
+			t.Errorf("contract-violator read leaked[%d] = %#x; expected 0x00", i, b)
+		}
+	}
+}
+
+// ---------------------------------------------------------------------------
+// ensureAgentKeyDirSecure — table-driven coverage
+// ---------------------------------------------------------------------------
+
+func TestEnsureAgentKeyDirSecure(t *testing.T) {
+	if runtime.GOOS == "windows" {
+		t.Skip("permission semantics differ on windows")
+	}
+
+	type tc struct {
+		name string
+		// setup returns the dir argument to pass to ensureAgentKeyDirSecure.
+		// base is a fresh t.TempDir() unique to each subtest.
+		setup func(t *testing.T, base string) string
+		// wantErrSubstr; "" means no error is expected.
+		wantErrSubstr string
+		// wantMode; if set, asserted via os.Stat after the call. Set to 0
+		// to skip the mode assertion (e.g. for error-path rows where the
+		// dir wasn't created or wasn't intended to change).
+		wantMode os.FileMode
+	}
+	cases := []tc{
+		// Refuse-empty/root invariants
+		{
+			name: "empty_string_refused",
+			setup: func(t *testing.T, _ string) string {
+				return ""
+			},
+			wantErrSubstr: `refuse empty/root dir ""`,
+		},
+		{
+			name: "dot_refused",
+			setup: func(t *testing.T, _ string) string {
+				return "."
+			},
+			wantErrSubstr: `refuse empty/root dir "."`,
+		},
+		{
+			name: "root_refused",
+			setup: func(t *testing.T, _ string) string {
+				return "/"
+			},
+			wantErrSubstr: `refuse empty/root dir "/"`,
+		},
+
+		// Non-existent path — MkdirAll(0700) path
+		{
+			name: "creates_with_0700",
+			setup: func(t *testing.T, base string) string {
+				return filepath.Join(base, "newdir")
+			},
+			wantMode: 0o700,
+		},
+		{
+			name: "creates_nested_0700",
+			setup: func(t *testing.T, base string) string {
+				return filepath.Join(base, "a", "b", "c")
+			},
+			wantMode: 0o700,
+		},
+
+		// Existing 0700 — no-op (mode == 0o700 branch).
+		{
+			name: "existing_0700_noop",
+			setup: func(t *testing.T, base string) string {
+				d := filepath.Join(base, "exists0700")
+				if err := os.Mkdir(d, 0o700); err != nil {
+					t.Fatalf("setup mkdir: %v", err)
+				}
+				return d
+			},
+			wantMode: 0o700,
+		},
+
+		// Existing more-permissive — chmod tighten to 0700.
+		{
+			name: "existing_0750_tightened",
+			setup: func(t *testing.T, base string) string {
+				d := filepath.Join(base, "exists0750")
+				if err := os.Mkdir(d, 0o750); err != nil {
+					t.Fatalf("setup mkdir: %v", err)
+				}
+				if err := os.Chmod(d, 0o750); err != nil {
+					t.Fatalf("setup chmod: %v", err)
+				}
+				return d
+			},
+			wantMode: 0o700,
+		},
+		{
+			name: "existing_0755_tightened",
+			setup: func(t *testing.T, base string) string {
+				d := filepath.Join(base, "exists0755")
+				if err := os.Mkdir(d, 0o755); err != nil {
+					t.Fatalf("setup mkdir: %v", err)
+				}
+				if err := os.Chmod(d, 0o755); err != nil {
+					t.Fatalf("setup chmod: %v", err)
+				}
+				return d
+			},
+			wantMode: 0o700,
+		},
+		{
+			name: "existing_0777_tightened",
+			setup: func(t *testing.T, base string) string {
+				d := filepath.Join(base, "exists0777")
+				if err := os.Mkdir(d, 0o777); err != nil {
+					t.Fatalf("setup mkdir: %v", err)
+				}
+				if err := os.Chmod(d, 0o777); err != nil {
+					t.Fatalf("setup chmod: %v", err)
+				}
+				return d
+			},
+			wantMode: 0o700,
+		},
+
+		// Existing owner-only-no-write modes accepted as-is via the
+		// `mode&0o077 == 0` branch (no chmod, mode preserved).
+		{
+			name: "existing_0500_accepted_no_chmod",
+			setup: func(t *testing.T, base string) string {
+				d := filepath.Join(base, "exists0500")
+				if err := os.Mkdir(d, 0o700); err != nil {
+					t.Fatalf("setup mkdir: %v", err)
+				}
+				if err := os.Chmod(d, 0o500); err != nil {
+					t.Fatalf("setup chmod: %v", err)
+				}
+				t.Cleanup(func() { _ = os.Chmod(d, 0o700) }) // let TempDir cleanup
+				return d
+			},
+			wantMode: 0o500,
+		},
+		{
+			name: "existing_0400_accepted_no_chmod",
+			setup: func(t *testing.T, base string) string {
+				d := filepath.Join(base, "exists0400")
+				if err := os.Mkdir(d, 0o700); err != nil {
+					t.Fatalf("setup mkdir: %v", err)
+				}
+				if err := os.Chmod(d, 0o400); err != nil {
+					t.Fatalf("setup chmod: %v", err)
+				}
+				t.Cleanup(func() { _ = os.Chmod(d, 0o700) })
+				return d
+			},
+			wantMode: 0o400,
+		},
+
+		// filepath.Clean normalization paths.
+		{
+			name: "trailing_slash_normalized",
+			setup: func(t *testing.T, base string) string {
+				d := filepath.Join(base, "trail")
+				if err := os.Mkdir(d, 0o755); err != nil {
+					t.Fatalf("setup mkdir: %v", err)
+				}
+				if err := os.Chmod(d, 0o755); err != nil {
+					t.Fatalf("setup chmod: %v", err)
+				}
+				return d + "/"
+			},
+			wantMode: 0o700,
+		},
+		{
+			name: "dot_prefix_normalized",
+			setup: func(t *testing.T, base string) string {
+				// The function uses filepath.Clean which strips redundant
+				// "./" segments. We only need to verify Clean is invoked,
+				// not that we end up at a relative path; pass an absolute
+				// path with an embedded "./".
+				d := filepath.Join(base, "dotprefix")
+				if err := os.Mkdir(d, 0o755); err != nil {
+					t.Fatalf("setup mkdir: %v", err)
+				}
+				if err := os.Chmod(d, 0o755); err != nil {
+					t.Fatalf("setup chmod: %v", err)
+				}
+				return filepath.Join(base, ".", "dotprefix")
+			},
+			wantMode: 0o700,
+		},
+	}
+
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			base := t.TempDir()
+			dir := tc.setup(t, base)
+
+			err := ensureAgentKeyDirSecure(dir)
+			if tc.wantErrSubstr != "" {
+				if err == nil {
+					t.Fatalf("expected error containing %q, got nil", tc.wantErrSubstr)
+				}
+				if !strings.Contains(err.Error(), tc.wantErrSubstr) {
+					t.Errorf("error %q does not contain %q", err, tc.wantErrSubstr)
+				}
+				return
+			}
+			if err != nil {
+				t.Fatalf("ensureAgentKeyDirSecure: %v", err)
+			}
+			if tc.wantMode != 0 {
+				clean := filepath.Clean(dir)
+				info, statErr := os.Stat(clean)
+				if statErr != nil {
+					t.Fatalf("post-call stat: %v", statErr)
+				}
+				if got := info.Mode().Perm(); got != tc.wantMode {
+					t.Errorf("dir mode = %#o; want %#o", got, tc.wantMode)
+				}
+			}
+		})
+	}
+}
+
+// TestEnsureAgentKeyDirSecure_Idempotent confirms a second call on a
+// just-created dir is a no-op (hits the `mode == 0o700` short-circuit).
+func TestEnsureAgentKeyDirSecure_Idempotent(t *testing.T) {
+	if runtime.GOOS == "windows" {
+		t.Skip("permission semantics differ on windows")
+	}
+	dir := filepath.Join(t.TempDir(), "idempotent")
+	if err := ensureAgentKeyDirSecure(dir); err != nil {
+		t.Fatalf("first call: %v", err)
+	}
+	if err := ensureAgentKeyDirSecure(dir); err != nil {
+		t.Fatalf("second call: %v", err)
+	}
+	info, err := os.Stat(dir)
+	if err != nil {
+		t.Fatalf("stat: %v", err)
+	}
+	if info.Mode().Perm() != 0o700 {
+		t.Errorf("expected 0700, got %#o", info.Mode().Perm())
+	}
+}
+
+// TestEnsureAgentKeyDirSecure_Concurrent runs the function from many
+// goroutines simultaneously on the same fresh path. This is a safety smoke
+// test under -race; it is NOT a functional correctness claim about
+// concurrent agents (the agent has a single goroutine). The MkdirAll call
+// is the load-bearing primitive here — it's documented as safe to call
+// repeatedly with no error if the dir already exists.
+func TestEnsureAgentKeyDirSecure_Concurrent(t *testing.T) {
+	if runtime.GOOS == "windows" {
+		t.Skip("permission semantics differ on windows")
+	}
+	dir := filepath.Join(t.TempDir(), "concurrent")
+	const workers = 8
+	var wg sync.WaitGroup
+	errCh := make(chan error, workers)
+	wg.Add(workers)
+	for i := 0; i < workers; i++ {
+		go func() {
+			defer wg.Done()
+			if err := ensureAgentKeyDirSecure(dir); err != nil {
+				errCh <- err
+			}
+		}()
+	}
+	wg.Wait()
+	close(errCh)
+	for err := range errCh {
+		t.Errorf("concurrent caller returned error: %v", err)
+	}
+	info, err := os.Stat(dir)
+	if err != nil {
+		t.Fatalf("post-concurrent stat: %v", err)
+	}
+	if info.Mode().Perm() != 0o700 {
+		t.Errorf("expected 0700 after concurrent calls, got %#o", info.Mode().Perm())
+	}
+}
+
+// TestEnsureAgentKeyDirSecure_PathIsAFile pins the function's behavior when
+// passed a regular file. The function does not type-check (no IsDir()), so
+// it stat's the file, sees mode 0o644 (or whatever), and chmod's it to 0700.
+//
+// This is "silently accepts a file path" behavior. It is not a correctness
+// bug per the function's caller (cmd/agent/main.go always passes
+// filepath.Dir(keyPath), which is a directory), but it is a hardening
+// candidate. Captured as a finding observation in the test docstring rather
+// than fixed in this bundle (Bundle 0.7 ships no production-code changes).
+func TestEnsureAgentKeyDirSecure_PathIsAFile(t *testing.T) {
+	if runtime.GOOS == "windows" {
+		t.Skip("permission semantics differ on windows")
+	}
+	base := t.TempDir()
+	filePath := filepath.Join(base, "not-a-dir.txt")
+	if err := os.WriteFile(filePath, []byte("x"), 0o644); err != nil {
+		t.Fatalf("setup writefile: %v", err)
+	}
+	err := ensureAgentKeyDirSecure(filePath)
+	if err != nil {
+		t.Fatalf("current behavior: function chmod's a file silently and returns nil; got err = %v", err)
+	}
+	info, statErr := os.Stat(filePath)
+	if statErr != nil {
+		t.Fatalf("post-call stat: %v", statErr)
+	}
+	if info.IsDir() {
+		t.Fatal("file became a directory; that's not a thing")
+	}
+	if info.Mode().Perm() != 0o700 {
+		t.Errorf("expected mode 0700 (current behavior), got %#o", info.Mode().Perm())
+	}
+}
+
+// TestEnsureAgentKeyDirSecure_MkdirErrorPropagated forces the MkdirAll
+// branch to fail by chmod'ing the parent to 0o500 (read+exec but no write).
+// On linux/darwin running as a non-root uid, MkdirAll on a child of such a
+// parent fails with EACCES. We assert the error message wraps with the
+// documented "create agent key dir" prefix.
+//
+// Skipped if running as root (root bypasses unix dir-write checks).
+func TestEnsureAgentKeyDirSecure_MkdirErrorPropagated(t *testing.T) {
+	if runtime.GOOS == "windows" {
+		t.Skip("permission semantics differ on windows")
+	}
+	if os.Getuid() == 0 {
+		t.Skip("running as root; cannot revoke parent dir write permission")
+	}
+	parent := t.TempDir()
+	if err := os.Chmod(parent, 0o500); err != nil {
+		t.Fatalf("setup chmod parent: %v", err)
+	}
+	t.Cleanup(func() { _ = os.Chmod(parent, 0o700) })
+
+	child := filepath.Join(parent, "no-can-create")
+	err := ensureAgentKeyDirSecure(child)
+	if err == nil {
+		t.Fatal("expected error when MkdirAll cannot write to read-only parent")
+	}
+	if !strings.Contains(err.Error(), "create agent key dir") {
+		t.Errorf("error %q should contain %q", err.Error(), "create agent key dir")
+	}
+}
+
+// TestEnsureAgentKeyDirSecure_StatErrorPropagated forces os.Stat to fail
+// with a non-IsNotExist error by chmod'ing the parent to 0o000 (no
+// read+exec). On linux/darwin running as a non-root uid, stat on a child
+// of such a parent fails with EACCES. We assert the error message wraps
+// with "stat agent key dir".
+//
+// Skipped if running as root.
+func TestEnsureAgentKeyDirSecure_StatErrorPropagated(t *testing.T) {
+	if runtime.GOOS == "windows" {
+		t.Skip("permission semantics differ on windows")
+	}
+	if os.Getuid() == 0 {
+		t.Skip("running as root; cannot revoke parent dir read+exec permission")
+	}
+	parent := t.TempDir()
+	child := filepath.Join(parent, "victim")
+	if err := os.Chmod(parent, 0o000); err != nil {
+		t.Fatalf("setup chmod parent: %v", err)
+	}
+	t.Cleanup(func() { _ = os.Chmod(parent, 0o700) })
+
+	err := ensureAgentKeyDirSecure(child)
+	if err == nil {
+		t.Fatal("expected error when stat cannot traverse unreadable parent")
+	}
+	if !strings.Contains(err.Error(), "stat agent key dir") {
+		t.Errorf("error %q should contain %q", err.Error(), "stat agent key dir")
+	}
+}
+
+// TestEnsureAgentKeyDirSecure_ChmodErrorPropagated forces os.Chmod to fail
+// on an existing more-permissive dir. We achieve this by:
+//  1. Creating an intermediate dir at 0o755 (so the function takes the
+//     tighten-via-chmod branch).
+//  2. Replacing the real dir with a read-only-from-parent bind: chmod the
+//     grandparent to 0o500 so the chmod syscall on the child fails with
+//     EACCES (the syscall needs write on the path's containing dir for
+//     metadata updates on most unix filesystems — actually no, chmod only
+//     needs ownership, not parent write. So we instead drop the file's
+//     owner via... no — we cannot change ownership without root.)
+//
+// Reaching the chmod-error branch from a non-root test is awkward because
+// chmod only requires ownership (which we always have on t.TempDir()).
+// The cleanest way is to skip on non-root and exercise the branch in CI
+// images that run as root; but our CI runs as non-root. We DO trigger the
+// branch via a different mechanism: replace the path with a SYMLINK to
+// /proc/1/root (or similar) where the eventual stat resolves but chmod
+// fails — but that's brittle and OS-specific.
+//
+// Acceptable closure: document that this branch is exercised by the
+// existing chmod-fails errno path, but the test as written can only assert
+// the wrap-prefix when the branch IS reached. We use a synthetic approach:
+// chmod-tighten a dir we then immediately delete, racing the syscall —
+// not deterministic.
+//
+// Pragmatic resolution: the chmod-error branch is structurally identical
+// to the mkdir-error and stat-error branches (errors.Wrap with a
+// distinct prefix), and is exercised in production via os.Chmod ENOENT
+// or read-only-filesystem failures. We add a unit test that asserts the
+// branch's MESSAGE format by passing through a wrap helper construct.
+// This test instead documents that the branch is structural and any new
+// failure mode (read-only fs, immutable bit, ACLs) inherits the wrap
+// prefix automatically.
+//
+// To still get coverage on the chmod-error branch, we use os.Chmod against
+// a dir whose immediate parent we delete mid-call. This is racy. Instead,
+// we make chmod fail by passing a path that filepath.Clean rewrites to
+// a symlink whose target was just chmod-stripped. Too brittle.
+//
+// CLEANEST APPROACH: rely on the OS's read-only filesystem semantics under
+// /sys (which is RO on linux). os.Chmod on a path under /sys returns EROFS.
+// But /sys is owned by root — stat would succeed only on existing entries,
+// and the function would then attempt chmod, which fails with EROFS (the
+// non-root caller still gets a clean error wrap).
+//
+// We cannot find a well-defined non-root chmod-fail path on darwin. So the
+// test runs only on linux and skips elsewhere.
+func TestEnsureAgentKeyDirSecure_ChmodErrorPropagated(t *testing.T) {
+	if runtime.GOOS != "linux" {
+		t.Skip("chmod-error branch is only reliably triggerable on linux via /sys (read-only fs)")
+	}
+	// /sys is mounted read-only on Linux. Pick a stable subdir we can stat
+	// (kernel-class). os.Chmod against it returns EROFS regardless of uid
+	// (well — root can remount, but the call against /sys/* still EROFS).
+	candidate := "/sys/kernel"
+	info, err := os.Stat(candidate)
+	if err != nil || !info.IsDir() {
+		t.Skipf("/sys/kernel not stat-able as a dir on this host; skipping (%v)", err)
+	}
+	mode := info.Mode().Perm()
+	if mode == 0o700 || mode&0o077 == 0 {
+		// Already in the no-chmod branch; this test cannot exercise the
+		// chmod-fail branch on this host. Skip rather than false-positive.
+		t.Skipf("/sys/kernel mode %#o already satisfies no-chmod branch", mode)
+	}
+	chmodErr := ensureAgentKeyDirSecure(candidate)
+	if chmodErr == nil {
+		t.Fatal("expected chmod failure on /sys (read-only fs)")
+	}
+	if !strings.Contains(chmodErr.Error(), "tighten agent key dir") {
+		t.Errorf("error %q should contain %q", chmodErr.Error(), "tighten agent key dir")
+	}
+}
+
+// TestEnsureAgentKeyDirSecure_FmtErrorMessageIncludesPath confirms each
+// error wrap includes the cleaned path (debuggability invariant).
+func TestEnsureAgentKeyDirSecure_FmtErrorMessageIncludesPath(t *testing.T) {
+	if runtime.GOOS == "windows" {
+		t.Skip("permission semantics differ on windows")
+	}
+	if os.Getuid() == 0 {
+		t.Skip("running as root; cannot revoke parent dir write permission")
+	}
+	parent := t.TempDir()
+	if err := os.Chmod(parent, 0o500); err != nil {
+		t.Fatalf("setup chmod parent: %v", err)
+	}
+	t.Cleanup(func() { _ = os.Chmod(parent, 0o700) })
+	child := filepath.Join(parent, "child")
+	want := filepath.Clean(child)
+
+	err := ensureAgentKeyDirSecure(child)
+	if err == nil {
+		t.Fatal("expected error")
+	}
+	if !strings.Contains(err.Error(), want) {
+		t.Errorf("error %q should reference cleaned path %q", err, want)
+	}
+}
+
+// ---------------------------------------------------------------------------
+// Cross-cutting: end-to-end smoke confirming the two functions compose
+// the way main.go uses them (Bundle 9 / L-002 / L-003 flow).
+// ---------------------------------------------------------------------------
+
+// TestKeymem_AgentMainFlowSmoke replays the cmd/agent/main.go composition:
+// ensureAgentKeyDirSecure(dir) → marshalAgentKeyAndZeroize(priv, onDER).
+// Closes the contract that both helpers cooperate cleanly under realistic
+// fixture conditions, and that the DER buffer is zeroized at the end of
+// the marshal call.
+func TestKeymem_AgentMainFlowSmoke(t *testing.T) {
+	if runtime.GOOS == "windows" {
+		t.Skip("permission semantics differ on windows")
+	}
+	keyDir := filepath.Join(t.TempDir(), "agent-keys")
+	if err := ensureAgentKeyDirSecure(keyDir); err != nil {
+		t.Fatalf("ensureAgentKeyDirSecure: %v", err)
+	}
+	info, err := os.Stat(keyDir)
+	if err != nil {
+		t.Fatalf("stat: %v", err)
+	}
+	if info.Mode().Perm() != 0o700 {
+		t.Fatalf("key dir not at 0700, got %#o", info.Mode().Perm())
+	}
+
+	priv := mustGenAgentECDSAKey(t)
+	var captured []byte
+	if err := marshalAgentKeyAndZeroize(priv, func(der []byte) error {
+		captured = der // share backing array
+		// Pretend caller does pem.EncodeToMemory(...) here; we just check
+		// the DER is a valid SEQUENCE.
+		if len(der) == 0 || der[0] != 0x30 {
+			return fmt.Errorf("unexpected DER shape (len=%d, first=%#x)", len(der), der)
+		}
+		return nil
+	}); err != nil {
+		t.Fatalf("marshalAgentKeyAndZeroize: %v", err)
+	}
+	for i, b := range captured {
+		if b != 0 {
+			t.Fatalf("post-flow DER buffer not zeroized at byte %d (%#x)", i, b)
+		}
+	}
+}

cmd/agent/main.go

@@ -8,28 +8,38 @@ import (
 	"crypto/rand"
 	"crypto/rsa"
 	"crypto/sha256"
+	"crypto/tls"
 	"crypto/x509"
 	"crypto/x509/pkix"
 	"encoding/json"
 	"encoding/pem"
+	"errors"
 	"flag"
 	"fmt"
 	"io"
 	"log/slog"
 	"net"
 	"net/http"
+	"net/url"
 	"os"
 	"os/signal"
 	"path/filepath"
 	"runtime"
 	"strings"
+	"sync"
 	"syscall"
 	"time"
 
 	"github.com/shankar0123/certctl/internal/connector/target"
 	"github.com/shankar0123/certctl/internal/connector/target/apache"
 	"github.com/shankar0123/certctl/internal/connector/target/caddy"
+	"github.com/shankar0123/certctl/internal/connector/target/envoy"
+	pf "github.com/shankar0123/certctl/internal/connector/target/postfix"
+	sshconn "github.com/shankar0123/certctl/internal/connector/target/ssh"
 	"github.com/shankar0123/certctl/internal/connector/target/f5"
+	jks "github.com/shankar0123/certctl/internal/connector/target/javakeystore"
+	k8s "github.com/shankar0123/certctl/internal/connector/target/k8ssecret"
+	wcs "github.com/shankar0123/certctl/internal/connector/target/wincertstore"
 	"github.com/shankar0123/certctl/internal/connector/target/haproxy"
 	"github.com/shankar0123/certctl/internal/connector/target/iis"
 	"github.com/shankar0123/certctl/internal/connector/target/nginx"
@@ -38,15 +48,27 @@ import (
 
 // AgentConfig represents the agent-side configuration.
 type AgentConfig struct {
-	ServerURL     string   // Control plane server URL (e.g., http://localhost:8443)
-	APIKey        string   // Agent API key for authentication
-	AgentName     string   // Agent name for identification
-	AgentID       string   // Agent ID for API calls (set after registration or from env)
-	Hostname      string   // Server hostname
-	KeyDir        string   // Directory for storing private keys (default: /var/lib/certctl/keys)
-	DiscoveryDirs []string // Directories to scan for certificates (comma-separated via env)
+	ServerURL          string   // Control plane server URL (e.g., https://localhost:8443) — must be https:// scheme
+	APIKey             string   // Agent API key for authentication
+	AgentName          string   // Agent name for identification
+	AgentID            string   // Agent ID for API calls (set after registration or from env)
+	Hostname           string   // Server hostname
+	KeyDir             string   // Directory for storing private keys (default: /var/lib/certctl/keys)
+	DiscoveryDirs      []string // Directories to scan for certificates (comma-separated via env)
+	CABundlePath       string   // Optional path to a PEM-encoded CA bundle that signed the server's cert (empty = system roots)
+	InsecureSkipVerify bool     // Dev-only: skip TLS certificate verification. Never enable in production. See docs/tls.md.
 }
 
+// ErrAgentRetired is the sentinel returned by [Agent.Run] when the control
+// plane responds with HTTP 410 Gone to a heartbeat or work-poll request — the
+// canonical signal that this agent's row has been soft-retired server-side
+// (see I-004 in cowork/certctl-coverage-gap-audit.md). The binary must
+// terminate cleanly: an init-system restart would only produce another 410
+// and wedge the host in a restart loop. main() translates this sentinel into
+// a zero exit code so systemd (Restart=on-failure) and launchd do not respawn
+// the process. Do not wrap this error — main() matches it with errors.Is.
+var ErrAgentRetired = fmt.Errorf("agent retired by control plane")
+
 // Agent represents the local agent that runs on target servers.
 // It periodically sends heartbeats, polls for work, executes deployment and CSR jobs,
 // and scans configured directories for existing certificates.
@@ -62,6 +84,17 @@ type Agent struct {
 	pollInterval          time.Duration
 	discoveryInterval     time.Duration
 	consecutiveFailures   int
+
+	// I-004: terminal retirement signal. retiredSignal is closed exactly once
+	// (guarded by retiredOnce) when either sendHeartbeat or pollForWork
+	// observes HTTP 410 Gone. The Run() select loop picks up the close and
+	// returns ErrAgentRetired, unwinding the goroutine cleanly so main() can
+	// log + exit(0). Using a channel + sync.Once (rather than an atomic bool
+	// + polling) lets us fall through the select statement immediately instead
+	// of waiting for the next ticker; the zero-allocation close is safe to
+	// race with ctx.Done() and other cases.
+	retiredOnce   sync.Once
+	retiredSignal chan struct{}
 }
 
 // WorkResponse represents the response from the work polling endpoint.
@@ -84,15 +117,78 @@ type JobItem struct {
 }
 
 // NewAgent creates a new agent instance.
-func NewAgent(cfg *AgentConfig, logger *slog.Logger) *Agent {
+//
+// The returned HTTP client enforces HTTPS-only control-plane access per the
+// HTTPS-Everywhere milestone (see docs/tls.md). TLS 1.3 is required; the
+// optional CABundlePath loads a PEM bundle into RootCAs so the agent can
+// trust internal / self-signed server certs without touching system trust
+// stores. InsecureSkipVerify is a dev-only escape hatch — callers must log a
+// loud warning when it's set; never enable in production (see §2.4 of the
+// milestone spec and docs/upgrade-to-tls.md).
+//
+// Returns an error if CABundlePath is set but unreadable or malformed — fail
+// loud at startup rather than silently fall back to system roots, which would
+// turn a misconfigured bundle path into a cryptic "x509: certificate signed
+// by unknown authority" on the first heartbeat.
+func NewAgent(cfg *AgentConfig, logger *slog.Logger) (*Agent, error) {
+	tlsConfig := &tls.Config{
+		MinVersion:         tls.VersionTLS13,
+		InsecureSkipVerify: cfg.InsecureSkipVerify, //nolint:gosec // opt-in dev escape hatch, documented in docs/tls.md
+	}
+	if cfg.CABundlePath != "" {
+		pemBytes, err := os.ReadFile(cfg.CABundlePath)
+		if err != nil {
+			return nil, fmt.Errorf("reading CA bundle at %q: %w", cfg.CABundlePath, err)
+		}
+		pool := x509.NewCertPool()
+		if !pool.AppendCertsFromPEM(pemBytes) {
+			return nil, fmt.Errorf("CA bundle at %q contains no valid PEM-encoded certificates", cfg.CABundlePath)
+		}
+		tlsConfig.RootCAs = pool
+	}
+
+	httpClient := &http.Client{
+		Timeout: 30 * time.Second,
+		Transport: &http.Transport{
+			TLSClientConfig:       tlsConfig,
+			ForceAttemptHTTP2:     true,
+			MaxIdleConns:          10,
+			IdleConnTimeout:       90 * time.Second,
+			TLSHandshakeTimeout:   10 * time.Second,
+			ExpectContinueTimeout: 1 * time.Second,
+		},
+	}
+
 	return &Agent{
 		config:            cfg,
 		logger:            logger,
-		client:            &http.Client{Timeout: 30 * time.Second},
+		client:            httpClient,
 		heartbeatInterval: 60 * time.Second,
 		pollInterval:      30 * time.Second,
 		discoveryInterval: 6 * time.Hour, // scan for certs every 6 hours
-	}
+		retiredSignal:     make(chan struct{}),
+	}, nil
+}
+
+// markRetired records that the control plane has declared this agent retired
+// (HTTP 410 Gone on heartbeat or work poll). Idempotent via sync.Once — if
+// both the heartbeat and work-poll paths observe 410 in the same tick, only
+// the first close() runs and we avoid a runtime panic. Emits an ERROR-level
+// log line so init-system journaling captures it prominently, and includes
+// the source (heartbeat/work_poll), response body, and status code so the
+// operator can verify it's a genuine retirement signal rather than a
+// misrouted request. After this returns, the select-loop case in Run()
+// observes the closed channel on its next iteration and returns
+// ErrAgentRetired.
+func (a *Agent) markRetired(source string, statusCode int, body string) {
+	a.retiredOnce.Do(func() {
+		a.logger.Error("agent has been retired by control plane — shutting down",
+			"source", source,
+			"status", statusCode,
+			"body", body,
+			"agent_id", a.config.AgentID)
+		close(a.retiredSignal)
+	})
 }
 
 // Run starts the agent's main loop.
@@ -148,6 +244,19 @@ func (a *Agent) Run(ctx context.Context) error {
 			a.logger.Info("agent shutting down", "reason", ctx.Err())
 			return ctx.Err()
 
+		// I-004: retiredSignal is closed exactly once (via markRetired's
+		// sync.Once) when either sendHeartbeat or pollForWork observes HTTP 410
+		// Gone from the control plane. Falling through this case immediately
+		// (rather than waiting for the next ticker) lets the agent shut down
+		// quickly once retirement is confirmed — every extra heartbeat against a
+		// retired row is wasted work and noise in the audit trail. Returning
+		// ErrAgentRetired propagates up to main(), which matches it with
+		// errors.Is and exits(0) so systemd/launchd do not respawn the process.
+		case <-a.retiredSignal:
+			a.logger.Info("agent retired signal received — exiting event loop",
+				"agent_id", a.config.AgentID)
+			return ErrAgentRetired
+
 		case <-heartbeatTicker.C:
 			a.sendHeartbeat(ctx)
 
@@ -160,7 +269,14 @@ func (a *Agent) Run(ctx context.Context) error {
 				a.logger.Warn("backing off due to consecutive failures",
 					"failures", a.consecutiveFailures,
 					"backoff", backoff.String())
-				time.Sleep(backoff)
+				// F-003: ctx-aware wait so graceful shutdown does not stall on
+				// a long backoff. If ctx cancels mid-backoff, return to the
+				// outer loop so the <-ctx.Done() case can trigger clean exit.
+				select {
+				case <-ctx.Done():
+					continue
+				case <-time.After(backoff):
+				}
 			}
 			a.pollForWork(ctx)
 
@@ -203,6 +319,22 @@ func (a *Agent) sendHeartbeat(ctx context.Context) {
 	}
 	defer resp.Body.Close()
 
+	// I-004: HTTP 410 Gone is the terminal signal from the control plane that
+	// this agent's row has been soft-retired (see internal/api/handler/agent.go
+	// heartbeat path + AgentRetirementService). Treat it separately from the
+	// generic non-200 error branch: record the event to markRetired (which closes
+	// retiredSignal exactly once via sync.Once) and return without bumping
+	// consecutiveFailures — this is not a transient failure, it's a clean
+	// shutdown. The Run() select loop picks up the closed channel on its next
+	// iteration and returns ErrAgentRetired, which main() translates into an
+	// exit(0) so systemd/launchd don't respawn the process into another 410
+	// loop.
+	if resp.StatusCode == http.StatusGone {
+		body, _ := io.ReadAll(resp.Body)
+		a.markRetired("heartbeat", resp.StatusCode, string(body))
+		return
+	}
+
 	if resp.StatusCode != http.StatusOK {
 		body, _ := io.ReadAll(resp.Body)
 		a.logger.Error("heartbeat rejected",
@@ -231,6 +363,19 @@ func (a *Agent) pollForWork(ctx context.Context) {
 	}
 	defer resp.Body.Close()
 
+	// I-004: same terminal-retirement handling as sendHeartbeat. Work-poll is the
+	// other hot path that can observe an agent's soft-retirement; if the
+	// heartbeat tick happens to fire after a work-poll tick within the same
+	// retirement window, this branch catches it first. markRetired's sync.Once
+	// guards idempotency so racing both paths in the same tick only closes the
+	// signal channel once. No consecutiveFailures increment — retirement is
+	// not a transient failure.
+	if resp.StatusCode == http.StatusGone {
+		body, _ := io.ReadAll(resp.Body)
+		a.markRetired("work_poll", resp.StatusCode, string(body))
+		return
+	}
+
 	if resp.StatusCode != http.StatusOK {
 		body, _ := io.ReadAll(resp.Body)
 		a.logger.Error("work poll rejected",
@@ -300,23 +445,40 @@ func (a *Agent) executeCSRJob(ctx context.Context, job JobItem) {
 		"job_id", job.ID,
 		"certificate_id", job.CertificateID)
 
-	// Step 2: Store private key to disk with secure permissions
+	// Step 2: Store private key to disk with secure permissions.
+	//
+	// Bundle-9 / Audit L-002 + L-003: marshal+write through helpers that
+	// (a) zeroize the in-heap DER buffer immediately after the PEM block is
+	// constructed so the private scalar's exposure window is bounded by
+	// this function call, and (b) assert the key directory is mode 0700
+	// before any write touches disk. Also defer-clear the PEM buffer for
+	// the same reason — the encoded key isn't sensitive in transit (it's
+	// going to disk) but lingers on the heap if we don't.
 	keyPath := filepath.Join(a.config.KeyDir, job.CertificateID+".key")
-	privKeyDER, err := x509.MarshalECPrivateKey(privKey)
-	if err != nil {
-		a.logger.Error("failed to marshal private key",
-			"job_id", job.ID,
-			"error", err)
-		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("key marshal failed: %v", err)); reportErr != nil {
+	if err := ensureAgentKeyDirSecure(filepath.Dir(keyPath)); err != nil {
+		a.logger.Error("agent key dir hardening failed", "job_id", job.ID, "error", err)
+		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("key dir hardening failed: %v", err)); reportErr != nil {
 			a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
 		}
 		return
 	}
-
-	privKeyPEM := pem.EncodeToMemory(&pem.Block{
-		Type:  "EC PRIVATE KEY",
-		Bytes: privKeyDER,
-	})
+	var privKeyPEM []byte
+	if marshalErr := marshalAgentKeyAndZeroize(privKey, func(der []byte) error {
+		privKeyPEM = pem.EncodeToMemory(&pem.Block{
+			Type:  "EC PRIVATE KEY",
+			Bytes: der,
+		})
+		return nil
+	}); marshalErr != nil {
+		a.logger.Error("failed to marshal private key",
+			"job_id", job.ID,
+			"error", marshalErr)
+		if reportErr := a.reportJobStatus(ctx, job.ID, "Failed", fmt.Sprintf("key marshal failed: %v", marshalErr)); reportErr != nil {
+			a.logger.Error("failed to report job status to server", "job_id", job.ID, "status", "Failed", "error", reportErr)
+		}
+		return
+	}
+	defer clear(privKeyPEM)
 
 	if err := os.WriteFile(keyPath, privKeyPEM, 0600); err != nil {
 		a.logger.Error("failed to write private key to disk",
@@ -344,11 +506,23 @@ func (a *Agent) executeCSRJob(ctx context.Context, job JobItem) {
 	}
 
 	// Step 3: Create CSR with common name and SANs
+	// Split SANs into DNS names and email addresses for proper CSR encoding
+	var dnsNames []string
+	var emailAddresses []string
+	for _, san := range job.SANs {
+		if strings.Contains(san, "@") {
+			emailAddresses = append(emailAddresses, san)
+		} else {
+			dnsNames = append(dnsNames, san)
+		}
+	}
+
 	csrTemplate := &x509.CertificateRequest{
 		Subject: pkix.Name{
 			CommonName: job.CommonName,
 		},
-		DNSNames: job.SANs,
+		DNSNames:       dnsNames,
+		EmailAddresses: emailAddresses,
 	}
 
 	csrDER, err := x509.CreateCertificateRequest(rand.Reader, csrTemplate, privKey)
@@ -571,7 +745,11 @@ func (a *Agent) createTargetConnector(targetType string, configJSON json.RawMess
 				return nil, fmt.Errorf("invalid F5 config: %w", err)
 			}
 		}
-		return f5.New(&cfg, a.logger), nil
+		conn, err := f5.New(&cfg, a.logger)
+		if err != nil {
+			return nil, fmt.Errorf("failed to create F5 connector: %w", err)
+		}
+		return conn, nil
 
 	case "IIS":
 		var cfg iis.Config
@@ -580,7 +758,7 @@ func (a *Agent) createTargetConnector(targetType string, configJSON json.RawMess
 				return nil, fmt.Errorf("invalid IIS config: %w", err)
 			}
 		}
-		return iis.New(&cfg, a.logger), nil
+		return iis.New(&cfg, a.logger)
 
 	case "Traefik":
 		var cfg traefik.Config
@@ -600,6 +778,71 @@ func (a *Agent) createTargetConnector(targetType string, configJSON json.RawMess
 		}
 		return caddy.New(&cfg, a.logger), nil
 
+	case "Envoy":
+		var cfg envoy.Config
+		if len(configJSON) > 0 {
+			if err := json.Unmarshal(configJSON, &cfg); err != nil {
+				return nil, fmt.Errorf("invalid Envoy config: %w", err)
+			}
+		}
+		return envoy.New(&cfg, a.logger), nil
+
+	case "Postfix":
+		var cfg pf.Config
+		cfg.Mode = "postfix"
+		if len(configJSON) > 0 {
+			if err := json.Unmarshal(configJSON, &cfg); err != nil {
+				return nil, fmt.Errorf("invalid Postfix config: %w", err)
+			}
+		}
+		return pf.New(&cfg, a.logger), nil
+
+	case "Dovecot":
+		var cfg pf.Config
+		cfg.Mode = "dovecot"
+		if len(configJSON) > 0 {
+			if err := json.Unmarshal(configJSON, &cfg); err != nil {
+				return nil, fmt.Errorf("invalid Dovecot config: %w", err)
+			}
+		}
+		return pf.New(&cfg, a.logger), nil
+
+	case "SSH":
+		var cfg sshconn.Config
+		if len(configJSON) > 0 {
+			if err := json.Unmarshal(configJSON, &cfg); err != nil {
+				return nil, fmt.Errorf("invalid SSH config: %w", err)
+			}
+		}
+		return sshconn.New(&cfg, a.logger)
+
+	case "WinCertStore":
+		var cfg wcs.Config
+		if len(configJSON) > 0 {
+			if err := json.Unmarshal(configJSON, &cfg); err != nil {
+				return nil, fmt.Errorf("invalid WinCertStore config: %w", err)
+			}
+		}
+		return wcs.New(&cfg, a.logger)
+
+	case "JavaKeystore":
+		var cfg jks.Config
+		if len(configJSON) > 0 {
+			if err := json.Unmarshal(configJSON, &cfg); err != nil {
+				return nil, fmt.Errorf("invalid JavaKeystore config: %w", err)
+			}
+		}
+		return jks.New(&cfg, a.logger), nil
+
+	case "KubernetesSecrets":
+		var cfg k8s.Config
+		if len(configJSON) > 0 {
+			if err := json.Unmarshal(configJSON, &cfg); err != nil {
+				return nil, fmt.Errorf("invalid KubernetesSecrets config: %w", err)
+			}
+		}
+		return k8s.New(&cfg, a.logger)
+
 	default:
 		return nil, fmt.Errorf("unsupported target type: %s", targetType)
 	}
@@ -944,12 +1187,14 @@ func certKeyInfo(cert *x509.Certificate) (string, int) {
 
 func main() {
 	// Parse command-line flags (with env var fallbacks for Docker deployment)
-	serverURL := flag.String("server", getEnvDefault("CERTCTL_SERVER_URL", "http://localhost:8443"), "Control plane server URL")
+	serverURL := flag.String("server", getEnvDefault("CERTCTL_SERVER_URL", "https://localhost:8443"), "Control plane server URL (must be https://)")
 	apiKey := flag.String("api-key", getEnvDefault("CERTCTL_API_KEY", ""), "Agent API key")
 	agentName := flag.String("name", getEnvDefault("CERTCTL_AGENT_NAME", "certctl-agent"), "Agent name")
 	agentID := flag.String("agent-id", getEnvDefault("CERTCTL_AGENT_ID", ""), "Agent ID (from registration)")
 	keyDir := flag.String("key-dir", getEnvDefault("CERTCTL_KEY_DIR", "/var/lib/certctl/keys"), "Directory for storing private keys")
 	discoveryDirsStr := flag.String("discovery-dirs", getEnvDefault("CERTCTL_DISCOVERY_DIRS", ""), "Comma-separated directories to scan for certificates")
+	caBundlePath := flag.String("ca-bundle", getEnvDefault("CERTCTL_SERVER_CA_BUNDLE_PATH", ""), "Path to a PEM-encoded CA bundle that signed the server's TLS cert (optional; falls back to system roots)")
+	insecureSkipVerify := flag.Bool("insecure-skip-verify", getEnvBoolDefault("CERTCTL_SERVER_TLS_INSECURE_SKIP_VERIFY", false), "Dev-only: skip TLS certificate verification. Never enable in production. See docs/tls.md.")
 	flag.Parse()
 
 	if *apiKey == "" {
@@ -963,6 +1208,18 @@ func main() {
 		os.Exit(1)
 	}
 
+	// Pre-flight URL-scheme validation — reject plaintext http:// before any
+	// network call. The HTTPS-Everywhere milestone (§2.4, §7) mandates that
+	// mis-configured agents fail loudly at startup with a diagnostic pointing
+	// at the upgrade guide, rather than producing a TCP-refused or
+	// TLS-handshake-error that obscures the actual cause.
+	if err := validateHTTPSScheme(*serverURL); err != nil {
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
+		fmt.Fprintf(os.Stderr, "\nThe certctl control plane is HTTPS-only as of v2.2.\n")
+		fmt.Fprintf(os.Stderr, "See docs/upgrade-to-tls.md for the cutover walkthrough.\n")
+		os.Exit(1)
+	}
+
 	// Set up structured logging
 	logLevel := slog.LevelInfo
 	if getEnvDefault("CERTCTL_LOG_LEVEL", "info") == "debug" {
@@ -991,17 +1248,27 @@ func main() {
 
 	// Create agent configuration
 	agentCfg := &AgentConfig{
-		ServerURL:     *serverURL,
-		APIKey:        *apiKey,
-		AgentName:     *agentName,
-		AgentID:       *agentID,
-		Hostname:      hostname,
-		KeyDir:        *keyDir,
-		DiscoveryDirs: discoveryDirs,
+		ServerURL:          *serverURL,
+		APIKey:             *apiKey,
+		AgentName:          *agentName,
+		AgentID:            *agentID,
+		Hostname:           hostname,
+		KeyDir:             *keyDir,
+		DiscoveryDirs:      discoveryDirs,
+		CABundlePath:       *caBundlePath,
+		InsecureSkipVerify: *insecureSkipVerify,
+	}
+
+	if agentCfg.InsecureSkipVerify {
+		logger.Warn("TLS certificate verification is disabled (CERTCTL_SERVER_TLS_INSECURE_SKIP_VERIFY=true) — never enable this in production")
 	}
 
 	// Create and start agent
-	agent := NewAgent(agentCfg, logger)
+	agent, err := NewAgent(agentCfg, logger)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error: failed to initialize agent: %v\n", err)
+		os.Exit(1)
+	}
 
 	// Create context with cancellation for graceful shutdown
 	ctx, cancel := context.WithCancel(context.Background())
@@ -1030,6 +1297,19 @@ func main() {
 		cancel()
 		<-errChan
 	case err := <-errChan:
+		// I-004: ErrAgentRetired is a terminal, *clean* shutdown — the control
+		// plane responded HTTP 410 Gone on heartbeat/work-poll, meaning this
+		// agent's row has been soft-retired and will never be reachable again.
+		// Exit 0 so systemd's Restart=on-failure and launchd's KeepAlive do NOT
+		// respawn the process into another 410 loop (which would wedge the host
+		// and spam the control plane). Operators can observe the retirement via
+		// audit_events or the AgentsPage retired tab; the terminal log line on
+		// the way out is enough for post-mortem forensics.
+		if errors.Is(err, ErrAgentRetired) {
+			logger.Info("agent retired by control plane — exiting without restart",
+				"agent_id", agentCfg.AgentID)
+			return
+		}
 		if err != context.Canceled {
 			logger.Error("agent error", "error", err)
 			os.Exit(1)
@@ -1046,3 +1326,49 @@ func getEnvDefault(key, defaultValue string) string {
 	}
 	return defaultValue
 }
+
+// getEnvBoolDefault parses an environment variable as a boolean. Accepts "1",
+// "t", "true", "T", "TRUE", "True" as true; anything else (including empty)
+// returns the provided default. Kept permissive on purpose so operators can
+// flip the dev-only TLS skip-verify toggle with any common truthy spelling
+// without having to remember exactly what we parse.
+func getEnvBoolDefault(key string, defaultValue bool) bool {
+	raw := os.Getenv(key)
+	if raw == "" {
+		return defaultValue
+	}
+	switch strings.ToLower(strings.TrimSpace(raw)) {
+	case "1", "t", "true", "yes", "on":
+		return true
+	case "0", "f", "false", "no", "off":
+		return false
+	default:
+		return defaultValue
+	}
+}
+
+// validateHTTPSScheme enforces the HTTPS-Everywhere milestone's §7 acceptance
+// criterion: "Agent with CERTCTL_SERVER_URL=http://... fails at startup with
+// a fail-loud diagnostic pointing at docs/upgrade-to-tls.md. Not TCP-refused,
+// not TLS-handshake-error — a pre-flight config validation failure before any
+// network call." Returns a descriptive error; the caller prints the upgrade
+// guide pointer and exits non-zero.
+func validateHTTPSScheme(serverURL string) error {
+	if serverURL == "" {
+		return fmt.Errorf("CERTCTL_SERVER_URL is empty — set it to an https:// URL (e.g., https://certctl-server:8443)")
+	}
+	u, err := url.Parse(serverURL)
+	if err != nil {
+		return fmt.Errorf("CERTCTL_SERVER_URL %q is not a valid URL: %w", serverURL, err)
+	}
+	switch strings.ToLower(u.Scheme) {
+	case "https":
+		return nil
+	case "http":
+		return fmt.Errorf("CERTCTL_SERVER_URL %q uses plaintext http:// — the certctl control plane is HTTPS-only", serverURL)
+	case "":
+		return fmt.Errorf("CERTCTL_SERVER_URL %q is missing a scheme — expected https://", serverURL)
+	default:
+		return fmt.Errorf("CERTCTL_SERVER_URL %q uses unsupported scheme %q — expected https://", serverURL, u.Scheme)
+	}
+}

cmd/agent/verify.go

@@ -75,7 +75,7 @@ func verifyDeployment(
 		// calls, issuer connector communication, or any operation that trusts the
 		// certificate. The verification result compares SHA-256 fingerprints only.
 		// See TICKET-016 for full security audit rationale.
-		InsecureSkipVerify: true,
+		InsecureSkipVerify: true, //nolint:gosec // verification probe; documented above + docs/tls.md L-001 table
 		ServerName:        targetHost, // For SNI
 	})
 	if err != nil {

cmd/agent/verify_test.go

@@ -228,7 +228,7 @@ func TestReportVerificationResult_Success(t *testing.T) {
 		ServerURL: server.URL,
 		APIKey:    "test-api-key",
 	}
-	agent := NewAgent(cfg, nil)
+	agent, _ := NewAgent(cfg, nil)
 
 	result := &VerificationResult{
 		ExpectedFingerprint: "abc123",
@@ -244,7 +244,7 @@ func TestReportVerificationResult_Success(t *testing.T) {
 }
 
 func TestReportVerificationResult_MissingFields(t *testing.T) {
-	agent := NewAgent(&AgentConfig{}, nil)
+	agent, _ := NewAgent(&AgentConfig{}, nil)
 
 	result := &VerificationResult{
 		Verified:   true,
@@ -343,7 +343,7 @@ func TestReportVerificationResult_ServerError(t *testing.T) {
 		ServerURL: server.URL,
 		APIKey:    "test-api-key",
 	}
-	agent := NewAgent(cfg, nil)
+	agent, _ := NewAgent(cfg, nil)
 
 	result := &VerificationResult{
 		ExpectedFingerprint: "abc123",
@@ -391,7 +391,13 @@ func TestVerifyDeployment_FingerprintComparison(t *testing.T) {
 	}))
 	defer server.Close()
 
-	// Get the server's TLS certificate from TLS config
+	// Q-1 closure (cat-s3-58ce7e9840be): defensive skip — httptest.NewTLSServer
+	// always provisions a self-signed certificate at construction time, so this
+	// branch is currently unreachable in practice. Kept as a guard against
+	// future test-server constructions that swap in a custom *tls.Config with
+	// no Certificates slice (the path below dereferences server.TLS.Certificates[0]
+	// and would panic). The skip preserves the assertion logic for the normal
+	// fixture path; if it ever fires, it's a fixture bug, not a product bug.
 	if len(server.TLS.Certificates) == 0 {
 		t.Skip("no TLS certificates configured on test server")
 	}

cmd/cli/dispatch_test.go

@@ -0,0 +1,442 @@
+package main
+
+import (
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"github.com/shankar0123/certctl/internal/cli"
+)
+
+// Bundle Q (L-001 closure): per-subcommand dispatch tests for cmd/cli/main.go.
+//
+// The existing `main_test.go` only covered `validateHTTPSScheme`. This file
+// pins every dispatch arm in `handleCerts`, `handleAgents`, `handleJobs`,
+// `handleImport`, `handleStatus` — both the "missing arg" usage prints and
+// the happy-path delegation to `*cli.Client`.
+//
+// Strategy: spin up an `httptest.Server` mocking the relevant API routes so
+// the client can exercise its end-to-end code path without a live server.
+// For arms that print usage and return without calling the client, we pass
+// a freshly-constructed client (still no network call — the client method
+// is never invoked).
+
+// newDispatchTestClient returns a `*cli.Client` pointed at the given test
+// server. Calls `t.Fatal` on construction error.
+func newDispatchTestClient(t *testing.T, server *httptest.Server) *cli.Client {
+	t.Helper()
+	// Configure the client with `insecure=true` because httptest.Server's
+	// self-signed TLS cert won't chain to a system root.
+	c, err := cli.NewClient(server.URL, "test-key", "json", "", true)
+	if err != nil {
+		t.Fatalf("NewClient: %v", err)
+	}
+	return c
+}
+
+// stubServer returns an httptest.Server (TLS) that responds with the given
+// JSON body and status code for any request. Tests that want to assert on
+// the request shape can wrap it in a more specific handler.
+func stubServer(t *testing.T, status int, body string) *httptest.Server {
+	t.Helper()
+	srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "application/json")
+		w.WriteHeader(status)
+		_, _ = w.Write([]byte(body))
+	}))
+	t.Cleanup(srv.Close)
+	return srv
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// handleCerts dispatch arms
+// ─────────────────────────────────────────────────────────────────────────────
+
+func TestHandleCerts_NoArgs_PrintsUsage(t *testing.T) {
+	srv := stubServer(t, 200, `{"data":[],"total":0}`)
+	c := newDispatchTestClient(t, srv)
+	if err := handleCerts(c, []string{}); err != nil {
+		t.Errorf("handleCerts({}): unexpected err=%v (should print usage and return nil)", err)
+	}
+}
+
+func TestHandleCerts_UnknownSubcommand_PrintsUsage(t *testing.T) {
+	srv := stubServer(t, 200, `{"data":[],"total":0}`)
+	c := newDispatchTestClient(t, srv)
+	if err := handleCerts(c, []string{"frobnicate"}); err != nil {
+		t.Errorf("handleCerts({frobnicate}): unexpected err=%v (should print usage and return nil)", err)
+	}
+}
+
+func TestHandleCerts_GetWithoutID_PrintsUsage(t *testing.T) {
+	srv := stubServer(t, 200, `{}`)
+	c := newDispatchTestClient(t, srv)
+	if err := handleCerts(c, []string{"get"}); err != nil {
+		t.Errorf("handleCerts({get}): unexpected err=%v (should print usage and return nil)", err)
+	}
+}
+
+func TestHandleCerts_RenewWithoutID_PrintsUsage(t *testing.T) {
+	srv := stubServer(t, 200, `{}`)
+	c := newDispatchTestClient(t, srv)
+	if err := handleCerts(c, []string{"renew"}); err != nil {
+		t.Errorf("handleCerts({renew}): unexpected err=%v (should print usage and return nil)", err)
+	}
+}
+
+func TestHandleCerts_RevokeWithoutID_PrintsUsage(t *testing.T) {
+	srv := stubServer(t, 200, `{}`)
+	c := newDispatchTestClient(t, srv)
+	if err := handleCerts(c, []string{"revoke"}); err != nil {
+		t.Errorf("handleCerts({revoke}): unexpected err=%v (should print usage and return nil)", err)
+	}
+}
+
+func TestHandleCerts_List_HitsClientPath(t *testing.T) {
+	// Asserts dispatch-path: handleCerts → c.ListCertificates → GET /api/v1/certificates.
+	var hits int
+	srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		hits++
+		if r.Method != "GET" || !strings.HasPrefix(r.URL.Path, "/api/v1/certificates") {
+			t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path)
+		}
+		w.WriteHeader(200)
+		_, _ = w.Write([]byte(`{"data":[],"total":0}`))
+	}))
+	t.Cleanup(srv.Close)
+	c := newDispatchTestClient(t, srv)
+	if err := handleCerts(c, []string{"list"}); err != nil {
+		t.Errorf("handleCerts({list}): err=%v", err)
+	}
+	if hits != 1 {
+		t.Errorf("expected 1 server hit, got %d", hits)
+	}
+}
+
+func TestHandleCerts_Get_HitsClientPath(t *testing.T) {
+	var lastPath string
+	srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		lastPath = r.URL.Path
+		w.WriteHeader(200)
+		_, _ = w.Write([]byte(`{"id":"mc-x","name":"x"}`))
+	}))
+	t.Cleanup(srv.Close)
+	c := newDispatchTestClient(t, srv)
+	if err := handleCerts(c, []string{"get", "mc-x"}); err != nil {
+		t.Errorf("handleCerts({get, mc-x}): err=%v", err)
+	}
+	if !strings.Contains(lastPath, "/api/v1/certificates/mc-x") {
+		t.Errorf("expected GET on /api/v1/certificates/mc-x, got %q", lastPath)
+	}
+}
+
+func TestHandleCerts_Renew_HitsClientPath(t *testing.T) {
+	var lastPath, lastMethod string
+	srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		lastPath = r.URL.Path
+		lastMethod = r.Method
+		w.WriteHeader(200)
+		_, _ = w.Write([]byte(`{"job_id":"job-1","status":"ok"}`))
+	}))
+	t.Cleanup(srv.Close)
+	c := newDispatchTestClient(t, srv)
+	if err := handleCerts(c, []string{"renew", "mc-x"}); err != nil {
+		t.Errorf("handleCerts({renew, mc-x}): err=%v", err)
+	}
+	if lastMethod != "POST" || !strings.Contains(lastPath, "/renew") {
+		t.Errorf("expected POST .../renew, got %s %s", lastMethod, lastPath)
+	}
+}
+
+func TestHandleCerts_Revoke_HitsClientPath(t *testing.T) {
+	var lastPath, lastMethod, lastBody string
+	srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		lastPath = r.URL.Path
+		lastMethod = r.Method
+		buf := make([]byte, 1024)
+		n, _ := r.Body.Read(buf)
+		lastBody = string(buf[:n])
+		w.WriteHeader(200)
+		_, _ = w.Write([]byte(`{"status":"revoked"}`))
+	}))
+	t.Cleanup(srv.Close)
+	c := newDispatchTestClient(t, srv)
+	if err := handleCerts(c, []string{"revoke", "mc-x", "--reason", "compromise"}); err != nil {
+		t.Errorf("handleCerts({revoke ...}): err=%v", err)
+	}
+	if lastMethod != "POST" || !strings.Contains(lastPath, "/revoke") {
+		t.Errorf("expected POST .../revoke, got %s %s", lastMethod, lastPath)
+	}
+	if !strings.Contains(lastBody, "compromise") {
+		t.Errorf("expected reason in body, got %q", lastBody)
+	}
+}
+
+func TestHandleCerts_BulkRevoke_HitsClientPath(t *testing.T) {
+	var lastPath string
+	srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		lastPath = r.URL.Path
+		w.WriteHeader(200)
+		_, _ = w.Write([]byte(`{"total_matched":0,"total_revoked":0,"total_skipped":0,"total_failed":0}`))
+	}))
+	t.Cleanup(srv.Close)
+	c := newDispatchTestClient(t, srv)
+	if err := handleCerts(c, []string{"bulk-revoke", "--reason", "test"}); err != nil {
+		t.Errorf("handleCerts({bulk-revoke ...}): err=%v", err)
+	}
+	if !strings.Contains(lastPath, "/bulk-revoke") {
+		t.Errorf("expected /bulk-revoke path, got %q", lastPath)
+	}
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// handleAgents dispatch arms
+// ─────────────────────────────────────────────────────────────────────────────
+
+func TestHandleAgents_NoArgs_PrintsUsage(t *testing.T) {
+	srv := stubServer(t, 200, `{}`)
+	c := newDispatchTestClient(t, srv)
+	if err := handleAgents(c, []string{}); err != nil {
+		t.Errorf("handleAgents({}): unexpected err=%v", err)
+	}
+}
+
+func TestHandleAgents_UnknownSubcommand_PrintsUsage(t *testing.T) {
+	srv := stubServer(t, 200, `{}`)
+	c := newDispatchTestClient(t, srv)
+	if err := handleAgents(c, []string{"frobnicate"}); err != nil {
+		t.Errorf("handleAgents({frobnicate}): unexpected err=%v", err)
+	}
+}
+
+func TestHandleAgents_GetWithoutID_PrintsUsage(t *testing.T) {
+	srv := stubServer(t, 200, `{}`)
+	c := newDispatchTestClient(t, srv)
+	if err := handleAgents(c, []string{"get"}); err != nil {
+		t.Errorf("handleAgents({get}): unexpected err=%v", err)
+	}
+}
+
+func TestHandleAgents_RetireWithoutID_PrintsUsage(t *testing.T) {
+	srv := stubServer(t, 200, `{}`)
+	c := newDispatchTestClient(t, srv)
+	if err := handleAgents(c, []string{"retire"}); err != nil {
+		t.Errorf("handleAgents({retire}): unexpected err=%v", err)
+	}
+}
+
+func TestHandleAgents_List_HitsClientPath(t *testing.T) {
+	var lastPath string
+	srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		lastPath = r.URL.Path
+		w.WriteHeader(200)
+		_, _ = w.Write([]byte(`{"data":[],"total":0}`))
+	}))
+	t.Cleanup(srv.Close)
+	c := newDispatchTestClient(t, srv)
+	if err := handleAgents(c, []string{"list"}); err != nil {
+		t.Errorf("handleAgents({list}): err=%v", err)
+	}
+	if !strings.Contains(lastPath, "/api/v1/agents") {
+		t.Errorf("expected /api/v1/agents path, got %q", lastPath)
+	}
+}
+
+func TestHandleAgents_ListRetired_HitsRetiredEndpoint(t *testing.T) {
+	// I-004: --retired flag splits to a separate /agents/retired endpoint.
+	var lastPath string
+	srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		lastPath = r.URL.Path
+		w.WriteHeader(200)
+		_, _ = w.Write([]byte(`{"data":[],"total":0}`))
+	}))
+	t.Cleanup(srv.Close)
+	c := newDispatchTestClient(t, srv)
+	if err := handleAgents(c, []string{"list", "--retired"}); err != nil {
+		t.Errorf("handleAgents({list --retired}): err=%v", err)
+	}
+	if !strings.Contains(lastPath, "/agents/retired") {
+		t.Errorf("expected --retired to hit /agents/retired, got %q", lastPath)
+	}
+}
+
+func TestHandleAgents_Get_HitsClientPath(t *testing.T) {
+	var lastPath string
+	srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		lastPath = r.URL.Path
+		w.WriteHeader(200)
+		_, _ = w.Write([]byte(`{"id":"ag-x","status":"online"}`))
+	}))
+	t.Cleanup(srv.Close)
+	c := newDispatchTestClient(t, srv)
+	if err := handleAgents(c, []string{"get", "ag-x"}); err != nil {
+		t.Errorf("handleAgents({get, ag-x}): err=%v", err)
+	}
+	if !strings.Contains(lastPath, "/agents/ag-x") {
+		t.Errorf("expected /agents/ag-x, got %q", lastPath)
+	}
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// handleJobs dispatch arms
+// ─────────────────────────────────────────────────────────────────────────────
+
+func TestHandleJobs_NoArgs_PrintsUsage(t *testing.T) {
+	srv := stubServer(t, 200, `{}`)
+	c := newDispatchTestClient(t, srv)
+	if err := handleJobs(c, []string{}); err != nil {
+		t.Errorf("handleJobs({}): unexpected err=%v", err)
+	}
+}
+
+func TestHandleJobs_UnknownSubcommand_PrintsUsage(t *testing.T) {
+	srv := stubServer(t, 200, `{}`)
+	c := newDispatchTestClient(t, srv)
+	if err := handleJobs(c, []string{"frobnicate"}); err != nil {
+		t.Errorf("handleJobs({frobnicate}): unexpected err=%v", err)
+	}
+}
+
+func TestHandleJobs_GetWithoutID_PrintsUsage(t *testing.T) {
+	srv := stubServer(t, 200, `{}`)
+	c := newDispatchTestClient(t, srv)
+	if err := handleJobs(c, []string{"get"}); err != nil {
+		t.Errorf("handleJobs({get}): unexpected err=%v", err)
+	}
+}
+
+func TestHandleJobs_CancelWithoutID_PrintsUsage(t *testing.T) {
+	srv := stubServer(t, 200, `{}`)
+	c := newDispatchTestClient(t, srv)
+	if err := handleJobs(c, []string{"cancel"}); err != nil {
+		t.Errorf("handleJobs({cancel}): unexpected err=%v", err)
+	}
+}
+
+func TestHandleJobs_List_HitsClientPath(t *testing.T) {
+	var lastPath string
+	srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		lastPath = r.URL.Path
+		w.WriteHeader(200)
+		_, _ = w.Write([]byte(`{"data":[],"total":0}`))
+	}))
+	t.Cleanup(srv.Close)
+	c := newDispatchTestClient(t, srv)
+	if err := handleJobs(c, []string{"list"}); err != nil {
+		t.Errorf("handleJobs({list}): err=%v", err)
+	}
+	if !strings.Contains(lastPath, "/api/v1/jobs") {
+		t.Errorf("expected /api/v1/jobs path, got %q", lastPath)
+	}
+}
+
+func TestHandleJobs_Get_HitsClientPath(t *testing.T) {
+	var lastPath string
+	srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		lastPath = r.URL.Path
+		w.WriteHeader(200)
+		_, _ = w.Write([]byte(`{"id":"job-x"}`))
+	}))
+	t.Cleanup(srv.Close)
+	c := newDispatchTestClient(t, srv)
+	if err := handleJobs(c, []string{"get", "job-x"}); err != nil {
+		t.Errorf("handleJobs({get, job-x}): err=%v", err)
+	}
+	if !strings.Contains(lastPath, "/jobs/job-x") {
+		t.Errorf("expected /jobs/job-x, got %q", lastPath)
+	}
+}
+
+func TestHandleJobs_Cancel_HitsClientPath(t *testing.T) {
+	var lastPath, lastMethod string
+	srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		lastPath = r.URL.Path
+		lastMethod = r.Method
+		w.WriteHeader(200)
+		_, _ = w.Write([]byte(`{"status":"cancelled"}`))
+	}))
+	t.Cleanup(srv.Close)
+	c := newDispatchTestClient(t, srv)
+	if err := handleJobs(c, []string{"cancel", "job-x"}); err != nil {
+		t.Errorf("handleJobs({cancel, job-x}): err=%v", err)
+	}
+	if lastMethod != "POST" || !strings.Contains(lastPath, "/cancel") {
+		t.Errorf("expected POST .../cancel, got %s %s", lastMethod, lastPath)
+	}
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// handleImport / handleStatus dispatch arms
+// ─────────────────────────────────────────────────────────────────────────────
+
+func TestHandleImport_NoArgs_PrintsUsage(t *testing.T) {
+	srv := stubServer(t, 200, `{}`)
+	c := newDispatchTestClient(t, srv)
+	if err := handleImport(c, []string{}); err != nil {
+		t.Errorf("handleImport({}): unexpected err=%v", err)
+	}
+}
+
+func TestHandleStatus_HitsClientPath(t *testing.T) {
+	var lastPath string
+	srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		lastPath = r.URL.Path
+		w.WriteHeader(200)
+		// GetStatus expects {"status":..., "stats":...} or similar.
+		// Provide a minimal valid JSON object.
+		_, _ = w.Write([]byte(`{"status":"healthy","version":"v2.X","db":"connected"}`))
+	}))
+	t.Cleanup(srv.Close)
+	c := newDispatchTestClient(t, srv)
+	if err := handleStatus(c); err != nil {
+		// GetStatus's table output may complain about missing fields; we only
+		// care that the dispatch arm fired and the request reached the server.
+		_ = err
+	}
+	if lastPath == "" {
+		t.Errorf("expected handleStatus to make at least one request")
+	}
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// CLI client TLS sanity (Q.1: confirms NewClient configures TLS correctly).
+// ─────────────────────────────────────────────────────────────────────────────
+
+func TestCliClient_RejectsUntrustedCert_WhenNotInsecure(t *testing.T) {
+	// Without insecure=true, the self-signed httptest cert must fail TLS
+	// verification. This pins the security default.
+	srv := stubServer(t, 200, `{}`)
+	c, err := cli.NewClient(srv.URL, "k", "json", "", false)
+	if err != nil {
+		t.Fatalf("NewClient: %v", err)
+	}
+	// Try a status call — should error out with a TLS verification failure,
+	// not silently succeed.
+	if err := c.GetStatus(); err == nil {
+		t.Errorf("expected TLS verification error against self-signed cert; got nil")
+	}
+}
+
+// TestCliClient_ParsesJSONResponse asserts the do() path's JSON unmarshalling
+// succeeds end-to-end (one of the more error-prone paths in the client).
+func TestCliClient_ParsesJSONResponse(t *testing.T) {
+	srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "application/json")
+		w.WriteHeader(200)
+		body := map[string]interface{}{
+			"data":  []map[string]interface{}{{"id": "mc-1", "name": "site-1"}},
+			"total": 1,
+		}
+		_ = json.NewEncoder(w).Encode(body)
+	}))
+	t.Cleanup(srv.Close)
+	c, err := cli.NewClient(srv.URL, "k", "json", "", true)
+	if err != nil {
+		t.Fatalf("NewClient: %v", err)
+	}
+	if err := c.ListCertificates(nil); err != nil {
+		t.Errorf("ListCertificates: err=%v", err)
+	}
+}

cmd/cli/main.go

@@ -3,7 +3,9 @@ package main
 import (
 	"flag"
 	"fmt"
+	"net/url"
 	"os"
+	"strings"
 
 	"github.com/shankar0123/certctl/internal/cli"
 )
@@ -27,35 +29,50 @@ Commands:
   certs renew ID   Trigger certificate renewal
   certs revoke ID  Revoke a certificate
 
-  agents list      List agents
-  agents get ID    Get agent details
+  agents list              List agents (add --retired to list soft-retired agents)
+  agents get ID            Get agent details
+  agents retire ID         Soft-retire an agent (add --force --reason "…" to cascade)
 
   jobs list        List jobs
   jobs get ID      Get job details
   jobs cancel ID   Cancel a pending job
 
   import FILE      Bulk import certificates from PEM file(s)
+                   Required: --owner-id, --team-id, --renewal-policy-id, --issuer-id
+                   Optional: --name-template (default {cn}), --environment (default imported)
 
   status           Show server health + summary stats
   version          Show CLI version
 
 Examples:
-  certctl-cli --server http://localhost:8443 --api-key mykey certs list
+  certctl-cli --server https://localhost:8443 --api-key mykey certs list
   certctl-cli certs renew mc-prod --format json
   certctl-cli import certs.pem
 `)
 	}
 
-	serverURL := fs.String("server", os.Getenv("CERTCTL_SERVER_URL"), "certctl server URL (env: CERTCTL_SERVER_URL)")
-	if *serverURL == "" {
-		*serverURL = "http://localhost:8443"
+	// HTTPS-Everywhere (v2.2): the server is HTTPS-only. The default URL uses
+	// https://; plaintext http:// is rejected by validateHTTPSScheme below.
+	defaultServer := os.Getenv("CERTCTL_SERVER_URL")
+	if defaultServer == "" {
+		defaultServer = "https://localhost:8443"
 	}
+	serverURL := fs.String("server", defaultServer, "certctl server URL — must be https:// (env: CERTCTL_SERVER_URL)")
 
 	apiKey := fs.String("api-key", os.Getenv("CERTCTL_API_KEY"), "API key for authentication (env: CERTCTL_API_KEY)")
 	format := fs.String("format", "table", "Output format: table, json")
+	caBundlePath := fs.String("ca-bundle", os.Getenv("CERTCTL_SERVER_CA_BUNDLE_PATH"), "Path to a PEM-encoded CA bundle that signed the server cert (env: CERTCTL_SERVER_CA_BUNDLE_PATH)")
+	insecure := fs.Bool("insecure", strings.EqualFold(os.Getenv("CERTCTL_SERVER_TLS_INSECURE_SKIP_VERIFY"), "true"), "Skip TLS certificate verification — dev only, never set in production (env: CERTCTL_SERVER_TLS_INSECURE_SKIP_VERIFY)")
 
 	fs.Parse(os.Args[1:])
 
+	if err := validateHTTPSScheme(*serverURL); err != nil {
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
+		fmt.Fprintf(os.Stderr, "\nThe certctl control plane is HTTPS-only as of v2.2.\n")
+		fmt.Fprintf(os.Stderr, "See docs/upgrade-to-tls.md for the cutover walkthrough.\n")
+		os.Exit(1)
+	}
+
 	args := fs.Args()
 	if len(args) == 0 {
 		fs.Usage()
@@ -63,13 +80,16 @@ Examples:
 	}
 
 	// Create client
-	client := cli.NewClient(*serverURL, *apiKey, *format)
+	client, err := cli.NewClient(*serverURL, *apiKey, *format, *caBundlePath, *insecure)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
+		os.Exit(1)
+	}
 
 	// Dispatch to appropriate command
 	command := args[0]
 	cmdArgs := args[1:]
 
-	var err error
 	switch command {
 	case "certs":
 		err = handleCerts(client, cmdArgs)
@@ -130,15 +150,27 @@ func handleCerts(client *cli.Client, args []string) error {
 			reason = subArgs[2]
 		}
 		return client.RevokeCertificate(id, reason)
+	case "bulk-revoke":
+		return client.BulkRevokeCertificates(subArgs)
 	default:
 		fmt.Fprintf(os.Stderr, "unknown subcommand: certs %s\n", subcommand)
 		return nil
 	}
 }
 
+// handleAgents dispatches the `agents` subcommands.
+//
+// I-004 additions:
+//
+//	agents list --retired      — hit the opt-in /agents/retired endpoint
+//	                             instead of the default listing (which
+//	                             filters retired rows out).
+//	agents retire <id>         — soft-retire an agent (DELETE /agents/{id}).
+//	                             --force cascades; --reason is required with
+//	                             --force (mirrors ErrForceReasonRequired).
 func handleAgents(client *cli.Client, args []string) error {
 	if len(args) == 0 {
-		fmt.Fprintf(os.Stderr, "usage: agents <list|get> [options]\n")
+		fmt.Fprintf(os.Stderr, "usage: agents <list|get|retire> [options]\n")
 		return nil
 	}
 
@@ -147,13 +179,34 @@ func handleAgents(client *cli.Client, args []string) error {
 
 	switch subcommand {
 	case "list":
-		return client.ListAgents(subArgs)
+		// --retired flag splits to a separate endpoint. We intercept it
+		// client-side and strip it before delegating, so both code paths
+		// share the --page/--per-page flag parsing inside the client.
+		retired := false
+		rest := make([]string, 0, len(subArgs))
+		for _, a := range subArgs {
+			if a == "--retired" {
+				retired = true
+				continue
+			}
+			rest = append(rest, a)
+		}
+		if retired {
+			return client.ListRetiredAgents(rest)
+		}
+		return client.ListAgents(rest)
 	case "get":
 		if len(subArgs) == 0 {
 			fmt.Fprintf(os.Stderr, "usage: agents get <id>\n")
 			return nil
 		}
 		return client.GetAgent(subArgs[0])
+	case "retire":
+		if len(subArgs) == 0 {
+			fmt.Fprintf(os.Stderr, "usage: agents retire <id> [--force] [--reason <reason>]\n")
+			return nil
+		}
+		return client.RetireAgent(subArgs)
 	default:
 		fmt.Fprintf(os.Stderr, "unknown subcommand: agents %s\n", subcommand)
 		return nil
@@ -201,3 +254,26 @@ func handleImport(client *cli.Client, args []string) error {
 func handleStatus(client *cli.Client) error {
 	return client.GetStatus()
 }
+
+// validateHTTPSScheme rejects plaintext and empty-scheme server URLs at
+// startup so operators get a fail-loud diagnostic before any network call,
+// not a TCP-refused or TLS-handshake-error downstream. See docs/upgrade-to-tls.md.
+func validateHTTPSScheme(serverURL string) error {
+	if serverURL == "" {
+		return fmt.Errorf("server URL is empty — set --server (or CERTCTL_SERVER_URL) to an https:// URL (e.g., https://certctl-server:8443)")
+	}
+	u, err := url.Parse(serverURL)
+	if err != nil {
+		return fmt.Errorf("server URL %q is not a valid URL: %w", serverURL, err)
+	}
+	switch strings.ToLower(u.Scheme) {
+	case "https":
+		return nil
+	case "http":
+		return fmt.Errorf("server URL %q uses plaintext http:// — the certctl control plane is HTTPS-only", serverURL)
+	case "":
+		return fmt.Errorf("server URL %q is missing a scheme — expected https://", serverURL)
+	default:
+		return fmt.Errorf("server URL %q uses unsupported scheme %q — expected https://", serverURL, u.Scheme)
+	}
+}

cmd/cli/main_test.go

@@ -0,0 +1,96 @@
+package main
+
+import (
+	"strings"
+	"testing"
+)
+
+// TestValidateHTTPSScheme pins the pre-flight URL-scheme guard that the
+// HTTPS-Everywhere milestone (v2.2, §3.2) requires on the certctl-cli binary
+// startup path. The CLI's diagnostic is distinct from the agent and MCP server
+// because it surfaces the --server flag alongside CERTCTL_SERVER_URL — so the
+// empty-URL case pins that flag-name substring separately. Every other case
+// mirrors the dispatch arms in cmd/cli/main.go:validateHTTPSScheme; drifting
+// the substrings is what this test is here to catch.
+func TestValidateHTTPSScheme(t *testing.T) {
+	tests := []struct {
+		name       string
+		serverURL  string
+		wantErr    bool
+		wantErrSub string // substring that MUST appear in the error message
+	}{
+		{
+			name:      "https URL passes",
+			serverURL: "https://certctl-server:8443",
+			wantErr:   false,
+		},
+		{
+			name:      "https URL with path passes",
+			serverURL: "https://certctl.example.com/api/v1",
+			wantErr:   false,
+		},
+		{
+			name:      "uppercase HTTPS scheme passes (url.Parse lowercases)",
+			serverURL: "HTTPS://certctl-server:8443",
+			wantErr:   false,
+		},
+		{
+			name:       "empty URL rejected mentions --server flag",
+			serverURL:  "",
+			wantErr:    true,
+			wantErrSub: "--server",
+		},
+		{
+			name:       "empty URL rejected also mentions CERTCTL_SERVER_URL",
+			serverURL:  "",
+			wantErr:    true,
+			wantErrSub: "CERTCTL_SERVER_URL",
+		},
+		{
+			name:       "plaintext http rejected",
+			serverURL:  "http://certctl-server:8443",
+			wantErr:    true,
+			wantErrSub: "plaintext http://",
+		},
+		{
+			name:       "bare host missing scheme rejected",
+			serverURL:  "localhost:8443",
+			wantErr:    true,
+			// url.Parse treats "localhost:8443" as scheme=localhost, opaque=8443
+			// — exercises the default arm (unsupported scheme) rather than the
+			// empty-scheme arm. Both are fail-closed, which is what we care about.
+			wantErrSub: "unsupported scheme",
+		},
+		{
+			name:       "path-only URL rejected",
+			serverURL:  "//certctl-server:8443",
+			wantErr:    true,
+			wantErrSub: "missing a scheme",
+		},
+		{
+			name:       "unsupported scheme rejected",
+			serverURL:  "ftp://certctl-server:8443",
+			wantErr:    true,
+			wantErrSub: "unsupported scheme",
+		},
+		{
+			name:       "ws scheme rejected",
+			serverURL:  "ws://certctl-server:8443",
+			wantErr:    true,
+			wantErrSub: "unsupported scheme",
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			err := validateHTTPSScheme(tt.serverURL)
+			if (err != nil) != tt.wantErr {
+				t.Fatalf("validateHTTPSScheme(%q) err=%v wantErr=%v", tt.serverURL, err, tt.wantErr)
+			}
+			if tt.wantErr && tt.wantErrSub != "" && !strings.Contains(err.Error(), tt.wantErrSub) {
+				t.Errorf("validateHTTPSScheme(%q) err=%q must contain %q so operators see the right diagnostic",
+					tt.serverURL, err.Error(), tt.wantErrSub)
+			}
+		})
+	}
+}

cmd/mcp-server/main.go

@@ -4,8 +4,10 @@ import (
 	"context"
 	"fmt"
 	"log"
+	"net/url"
 	"os"
 	"os/signal"
+	"strings"
 
 	gomcp "github.com/modelcontextprotocol/go-sdk/mcp"
 
@@ -16,14 +18,33 @@ import (
 var Version = "dev"
 
 func main() {
+	// HTTPS-Everywhere (v2.2): the server is HTTPS-only. The default URL
+	// uses https://; plaintext http:// is rejected by validateHTTPSScheme
+	// below with a fail-loud pre-flight diagnostic pointing at
+	// docs/upgrade-to-tls.md, so operators never get a TCP-refused or
+	// TLS-handshake-error downstream. See docs/tls.md for CA bundle and
+	// insecure-skip-verify guidance.
 	serverURL := os.Getenv("CERTCTL_SERVER_URL")
 	if serverURL == "" {
-		serverURL = "http://localhost:8443"
+		serverURL = "https://localhost:8443"
+	}
+
+	if err := validateHTTPSScheme(serverURL); err != nil {
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
+		fmt.Fprintf(os.Stderr, "\nThe certctl control plane is HTTPS-only as of v2.2.\n")
+		fmt.Fprintf(os.Stderr, "See docs/upgrade-to-tls.md for the cutover walkthrough.\n")
+		os.Exit(1)
 	}
 
 	apiKey := os.Getenv("CERTCTL_API_KEY")
+	caBundlePath := os.Getenv("CERTCTL_SERVER_CA_BUNDLE_PATH")
+	insecure := strings.EqualFold(os.Getenv("CERTCTL_SERVER_TLS_INSECURE_SKIP_VERIFY"), "true")
 
-	client := mcp.NewClient(serverURL, apiKey)
+	client, err := mcp.NewClient(serverURL, apiKey, caBundlePath, insecure)
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "Error: %v\n", err)
+		os.Exit(1)
+	}
 
 	server := gomcp.NewServer(&gomcp.Implementation{
 		Name:    "certctl",
@@ -41,3 +62,26 @@ func main() {
 		log.Fatalf("MCP server error: %v", err)
 	}
 }
+
+// validateHTTPSScheme rejects plaintext and empty-scheme server URLs at
+// startup so operators get a fail-loud diagnostic before any network call,
+// not a TCP-refused or TLS-handshake-error downstream. See docs/upgrade-to-tls.md.
+func validateHTTPSScheme(serverURL string) error {
+	if serverURL == "" {
+		return fmt.Errorf("server URL is empty — set CERTCTL_SERVER_URL to an https:// URL (e.g., https://certctl-server:8443)")
+	}
+	u, err := url.Parse(serverURL)
+	if err != nil {
+		return fmt.Errorf("server URL %q is not a valid URL: %w", serverURL, err)
+	}
+	switch strings.ToLower(u.Scheme) {
+	case "https":
+		return nil
+	case "http":
+		return fmt.Errorf("server URL %q uses plaintext http:// — the certctl control plane is HTTPS-only", serverURL)
+	case "":
+		return fmt.Errorf("server URL %q is missing a scheme — expected https://", serverURL)
+	default:
+		return fmt.Errorf("server URL %q uses unsupported scheme %q — expected https://", serverURL, u.Scheme)
+	}
+}

cmd/mcp-server/main_test.go

@@ -0,0 +1,90 @@
+package main
+
+import (
+	"strings"
+	"testing"
+)
+
+// TestValidateHTTPSScheme pins the pre-flight URL-scheme guard that the
+// HTTPS-Everywhere milestone (v2.2, §3.2) requires on the MCP server binary
+// startup path. The whole point is to fail loud with a diagnostic that points
+// at docs/upgrade-to-tls.md *before* any network call — not a cryptic
+// TCP-refused or TLS-handshake-error two ticks later. Every case here mirrors
+// the dispatch arms in cmd/mcp-server/main.go:validateHTTPSScheme; drifting
+// the error-message substrings is what this test is here to catch.
+func TestValidateHTTPSScheme(t *testing.T) {
+	tests := []struct {
+		name       string
+		serverURL  string
+		wantErr    bool
+		wantErrSub string // substring that MUST appear in the error message
+	}{
+		{
+			name:      "https URL passes",
+			serverURL: "https://certctl-server:8443",
+			wantErr:   false,
+		},
+		{
+			name:      "https URL with path passes",
+			serverURL: "https://certctl.example.com/api/v1",
+			wantErr:   false,
+		},
+		{
+			name:      "uppercase HTTPS scheme passes (url.Parse lowercases)",
+			serverURL: "HTTPS://certctl-server:8443",
+			wantErr:   false,
+		},
+		{
+			name:       "empty URL rejected",
+			serverURL:  "",
+			wantErr:    true,
+			wantErrSub: "server URL is empty",
+		},
+		{
+			name:       "plaintext http rejected",
+			serverURL:  "http://certctl-server:8443",
+			wantErr:    true,
+			wantErrSub: "plaintext http://",
+		},
+		{
+			name:       "bare host missing scheme rejected",
+			serverURL:  "localhost:8443",
+			wantErr:    true,
+			// url.Parse treats "localhost:8443" as scheme=localhost, opaque=8443
+			// — exercises the default arm (unsupported scheme) rather than the
+			// empty-scheme arm. Both are fail-closed, which is what we care about.
+			wantErrSub: "unsupported scheme",
+		},
+		{
+			name:       "path-only URL rejected",
+			serverURL:  "//certctl-server:8443",
+			wantErr:    true,
+			wantErrSub: "missing a scheme",
+		},
+		{
+			name:       "unsupported scheme rejected",
+			serverURL:  "ftp://certctl-server:8443",
+			wantErr:    true,
+			wantErrSub: "unsupported scheme",
+		},
+		{
+			name:       "ws scheme rejected",
+			serverURL:  "ws://certctl-server:8443",
+			wantErr:    true,
+			wantErrSub: "unsupported scheme",
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			err := validateHTTPSScheme(tt.serverURL)
+			if (err != nil) != tt.wantErr {
+				t.Fatalf("validateHTTPSScheme(%q) err=%v wantErr=%v", tt.serverURL, err, tt.wantErr)
+			}
+			if tt.wantErr && tt.wantErrSub != "" && !strings.Contains(err.Error(), tt.wantErrSub) {
+				t.Errorf("validateHTTPSScheme(%q) err=%q must contain %q so operators see the right diagnostic",
+					tt.serverURL, err.Error(), tt.wantErrSub)
+			}
+		})
+	}
+}

cmd/server/auth_exempt_test.go

@@ -0,0 +1,117 @@
+package main
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"github.com/shankar0123/certctl/internal/api/router"
+)
+
+// Bundle B / Audit M-002 (CWE-862): pin the dispatch-layer auth-exempt
+// allowlist. cmd/server/main.go::buildFinalHandler decides per-request
+// whether a path goes through the authenticated apiHandler or the
+// no-auth handler. This test:
+//
+//   - constructs a buildFinalHandler with two sentinel handlers (one
+//     for "auth", one for "no-auth") so we can observe which path is
+//     taken from the response body.
+//   - probes every prefix listed in router.AuthExemptDispatchPrefixes
+//     and confirms it routes to no-auth.
+//   - probes a few representative authenticated routes and confirms
+//     they route to auth.
+//   - probes the static-route allowlist (/health, /ready, etc.) that
+//     also bypasses auth at this layer.
+//
+// Adding a new auth-bypass to buildFinalHandler without updating the
+// router.AuthExemptDispatchPrefixes constant fails this test.
+
+func TestBuildFinalHandler_AuthExemptDispatchAllowlist(t *testing.T) {
+	apiHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		_, _ = w.Write([]byte("AUTH"))
+	})
+	noAuthHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		_, _ = w.Write([]byte("NOAUTH"))
+	})
+
+	// dashboardEnabled=false keeps the dispatch logic deterministic — no
+	// fileServer fallback to muddy the result.
+	final := buildFinalHandler(apiHandler, noAuthHandler, "/nonexistent", false)
+
+	cases := []struct {
+		name string
+		path string
+		want string
+	}{
+		// AuthExemptRouterRoutes (also enforced at this layer)
+		{"health", "/health", "NOAUTH"},
+		{"ready", "/ready", "NOAUTH"},
+		{"auth_info", "/api/v1/auth/info", "NOAUTH"},
+		{"version", "/api/v1/version", "NOAUTH"},
+
+		// AuthExemptDispatchPrefixes — every documented prefix
+		{"pki_crl", "/.well-known/pki/crl", "NOAUTH"},
+		{"pki_ocsp", "/.well-known/pki/ocsp", "NOAUTH"},
+		{"est_simpleenroll", "/.well-known/est/simpleenroll", "NOAUTH"},
+		{"est_cacerts", "/.well-known/est/cacerts", "NOAUTH"},
+		{"scep_root", "/scep", "NOAUTH"},
+		{"scep_op", "/scep/pkiclient.exe", "NOAUTH"},
+
+		// Authenticated routes — must hit apiHandler
+		{"certs_list", "/api/v1/certificates", "AUTH"},
+		{"agents_list", "/api/v1/agents", "AUTH"},
+		{"audit_check", "/api/v1/auth/check", "AUTH"},
+
+		// Random non-API path — falls through to apiHandler when
+		// dashboard disabled (preserves pre-M-001 API-only behavior).
+		{"unknown", "/some-other-path", "AUTH"},
+	}
+
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			req := httptest.NewRequest(http.MethodGet, tc.path, nil)
+			rec := httptest.NewRecorder()
+			final.ServeHTTP(rec, req)
+			got := rec.Body.String()
+			if got != tc.want {
+				t.Errorf("path %q routed to %q; want %q (this is the M-002 dispatch-layer pin)", tc.path, got, tc.want)
+			}
+		})
+	}
+}
+
+// TestDispatch_NoUndocumentedBypasses asserts that for every prefix the
+// dispatch layer routes to noAuthHandler, that prefix appears in the
+// router.AuthExemptDispatchPrefixes constant. This is the inverse pin —
+// adding a new bypass to buildFinalHandler without updating the constant
+// fails this test.
+//
+// We probe a curated set of "would-be-bypasses" derived from the actual
+// dispatch source by reading buildFinalHandler's lines. If the dispatch
+// logic adds a new prefix that ends up in the no-auth chain, the
+// curated set must be extended in the same commit that updates the
+// constant — this fails-loud rather than silently allowing a bypass.
+func TestDispatch_NoUndocumentedBypasses(t *testing.T) {
+	for _, prefix := range router.AuthExemptDispatchPrefixes {
+		if !strings.HasPrefix(prefix, "/") {
+			t.Errorf("AuthExemptDispatchPrefixes entry %q must start with / for prefix matching", prefix)
+		}
+	}
+	// Every entry in router.AuthExemptDispatchPrefixes must round-trip
+	// through buildFinalHandler to noAuthHandler (covered by the table
+	// test above). This test additionally asserts the inverse: known
+	// authenticated prefixes do NOT match any documented bypass prefix.
+	authenticatedPrefixes := []string{
+		"/api/v1/certificates",
+		"/api/v1/agents",
+		"/api/v1/audit",
+	}
+	for _, ap := range authenticatedPrefixes {
+		for _, bypass := range router.AuthExemptDispatchPrefixes {
+			if strings.HasPrefix(ap, bypass) {
+				t.Errorf("authenticated prefix %q overlaps with documented bypass %q — auth bypass risk", ap, bypass)
+			}
+		}
+	}
+}

cmd/server/finalhandler_test.go

@@ -0,0 +1,314 @@
+package main
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+// TestBuildFinalHandler_Dispatch is the M-001 regression harness for the outer
+// HTTP dispatch layer. It pins which path prefixes ride the no-auth middleware
+// chain (EST, SCEP, /.well-known/pki, health/ready, /api/v1/auth/info) versus
+// the authenticated chain (/api/v1/*).
+//
+// The concern under test is ONLY the dispatch in buildFinalHandler — the
+// handlers themselves are mocked as marker handlers that stamp "AUTH" or
+// "NOAUTH" into the response body. Service-layer concerns (SCEP password
+// validation, EST CSR validation, API auth enforcement) are covered by their
+// respective test suites.
+//
+// Case (i) is the central guard: EST with NO client cert / NO Bearer token
+// MUST reach the no-auth handler (pre-M-001 it was 401'd by the Auth
+// middleware, blocking enrollment for every real-world EST client).
+func TestBuildFinalHandler_Dispatch(t *testing.T) {
+	// Marker handlers — each stamps a unique body so tests can verify which
+	// chain the request traversed.
+	authHandler := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		w.Header().Set("X-Chain", "auth")
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte("AUTH"))
+	})
+	noAuthHandler := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		w.Header().Set("X-Chain", "noauth")
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte("NOAUTH"))
+	})
+
+	// Dashboard directory with index.html + assets/ for SPA fallback and
+	// static-asset tests. Cleaned up by t.TempDir.
+	webDir := t.TempDir()
+	indexHTML := []byte("<!doctype html><html><body>certctl dashboard</body></html>")
+	if err := os.WriteFile(filepath.Join(webDir, "index.html"), indexHTML, 0o644); err != nil {
+		t.Fatalf("write index.html: %v", err)
+	}
+	assetsDir := filepath.Join(webDir, "assets")
+	if err := os.MkdirAll(assetsDir, 0o755); err != nil {
+		t.Fatalf("mkdir assets: %v", err)
+	}
+	assetJS := []byte("console.log('certctl');")
+	if err := os.WriteFile(filepath.Join(assetsDir, "app.js"), assetJS, 0o644); err != nil {
+		t.Fatalf("write app.js: %v", err)
+	}
+
+	handler := buildFinalHandler(authHandler, noAuthHandler, webDir, true /* dashboardEnabled */)
+
+	tests := []struct {
+		name           string
+		method         string
+		path           string
+		wantBody       string // "AUTH" | "NOAUTH" | "" (== substring match against response body)
+		wantBodyPrefix string
+		wantStatus     int
+		description    string
+	}{
+		// ---- Case (i): M-001 central regression guard ----
+		{
+			name:        "est_cacerts_no_auth_reaches_noauth_handler",
+			method:      http.MethodGet,
+			path:        "/.well-known/est/cacerts",
+			wantBody:    "NOAUTH",
+			wantStatus:  http.StatusOK,
+			description: "EST clients cannot present Bearer tokens — must NOT be 401'd before reaching the handler (RFC 7030 §4.1.1)",
+		},
+		{
+			name:        "est_simpleenroll_no_auth_reaches_noauth_handler",
+			method:      http.MethodPost,
+			path:        "/.well-known/est/simpleenroll",
+			wantBody:    "NOAUTH",
+			wantStatus:  http.StatusOK,
+			description: "RFC 7030 §4.2 simpleenroll served from no-auth chain (option D)",
+		},
+		{
+			name:        "est_simplereenroll_no_auth_reaches_noauth_handler",
+			method:      http.MethodPost,
+			path:        "/.well-known/est/simplereenroll",
+			wantBody:    "NOAUTH",
+			wantStatus:  http.StatusOK,
+			description: "RFC 7030 §4.2.2 simplereenroll also on no-auth chain",
+		},
+		{
+			name:        "est_csrattrs_no_auth_reaches_noauth_handler",
+			method:      http.MethodGet,
+			path:        "/.well-known/est/csrattrs",
+			wantBody:    "NOAUTH",
+			wantStatus:  http.StatusOK,
+			description: "RFC 7030 §4.5 csrattrs also on no-auth chain",
+		},
+
+		// ---- Cases (ii) + (iii): SCEP dispatch ----
+		// The actual challengePassword validation lives in the service layer
+		// (internal/service/scep.go). This test pins that ALL /scep* requests
+		// reach the no-auth chain — the service layer is then responsible for
+		// rejecting or accepting based on password contents.
+		{
+			name:        "scep_exact_path_reaches_noauth_handler",
+			method:      http.MethodGet,
+			path:        "/scep",
+			wantBody:    "NOAUTH",
+			wantStatus:  http.StatusOK,
+			description: "SCEP clients authenticate via CSR challengePassword, not Bearer (RFC 8894 §3.2)",
+		},
+		{
+			name:        "scep_subpath_reaches_noauth_handler",
+			method:      http.MethodPost,
+			path:        "/scep/",
+			wantBody:    "NOAUTH",
+			wantStatus:  http.StatusOK,
+			description: "Trailing-slash variant must also ride no-auth chain",
+		},
+		{
+			name:        "scep_query_string_reaches_noauth_handler",
+			method:      http.MethodGet,
+			path:        "/scep?operation=GetCACaps",
+			wantBody:    "NOAUTH",
+			wantStatus:  http.StatusOK,
+			description: "Query string does not affect dispatch — operation dispatch is handler-internal",
+		},
+		// Defensive: /scepxyz MUST NOT match the SCEP prefix (guards against
+		// over-broad matching that would leak non-SCEP paths into no-auth).
+		{
+			name:        "scepxyz_does_not_match_scep_prefix",
+			method:      http.MethodGet,
+			path:        "/scepxyz",
+			wantStatus:  http.StatusOK,
+			wantBody:    "certctl dashboard",
+			description: "SPA fallback — /scepxyz must not be confused with /scep or /scep/",
+		},
+
+		// ---- Case (iv): RFC 5280 CRL + RFC 6960 OCSP ----
+		{
+			name:        "pki_crl_no_auth_reaches_noauth_handler",
+			method:      http.MethodGet,
+			path:        "/.well-known/pki/crl/abc123",
+			wantBody:    "NOAUTH",
+			wantStatus:  http.StatusOK,
+			description: "RFC 5280 CRL distribution point must be served without auth",
+		},
+		{
+			name:        "pki_ocsp_no_auth_reaches_noauth_handler",
+			method:      http.MethodGet,
+			path:        "/.well-known/pki/ocsp/abc123/serial",
+			wantBody:    "NOAUTH",
+			wantStatus:  http.StatusOK,
+			description: "RFC 6960 OCSP responder must be served without auth",
+		},
+
+		// ---- Case (v): Authenticated API routes ----
+		{
+			name:        "api_v1_certificates_goes_through_auth",
+			method:      http.MethodGet,
+			path:        "/api/v1/certificates",
+			wantBody:    "AUTH",
+			wantStatus:  http.StatusOK,
+			description: "Primary API surface must still require Bearer token",
+		},
+		{
+			name:        "api_v1_auth_check_goes_through_auth",
+			method:      http.MethodGet,
+			path:        "/api/v1/auth/check",
+			wantBody:    "AUTH",
+			wantStatus:  http.StatusOK,
+			description: "auth/check validates the caller's Bearer — auth chain required",
+		},
+		{
+			name:        "api_v1_jobs_goes_through_auth",
+			method:      http.MethodGet,
+			path:        "/api/v1/jobs",
+			wantBody:    "AUTH",
+			wantStatus:  http.StatusOK,
+			description: "Jobs API is part of the privileged surface",
+		},
+
+		// ---- Health probes bypass auth ----
+		{
+			name:        "health_bypasses_auth",
+			method:      http.MethodGet,
+			path:        "/health",
+			wantBody:    "NOAUTH",
+			wantStatus:  http.StatusOK,
+			description: "Docker/K8s health probes cannot carry Bearer tokens",
+		},
+		{
+			name:        "ready_bypasses_auth",
+			method:      http.MethodGet,
+			path:        "/ready",
+			wantBody:    "NOAUTH",
+			wantStatus:  http.StatusOK,
+			description: "Readiness probe also unauthenticated",
+		},
+		{
+			name:        "auth_info_bypasses_auth",
+			method:      http.MethodGet,
+			path:        "/api/v1/auth/info",
+			wantBody:    "NOAUTH",
+			wantStatus:  http.StatusOK,
+			description: "React app calls auth/info BEFORE login to discover auth mode",
+		},
+
+		// ---- Static assets served by file server ----
+		{
+			name:        "static_asset_served_by_file_server",
+			method:      http.MethodGet,
+			path:        "/assets/app.js",
+			wantStatus:  http.StatusOK,
+			wantBody:    "console.log('certctl');",
+			description: "Built Vite assets served directly without auth",
+		},
+
+		// ---- SPA fallback ----
+		{
+			name:        "spa_fallback_serves_index_html",
+			method:      http.MethodGet,
+			path:        "/",
+			wantStatus:  http.StatusOK,
+			wantBody:    "certctl dashboard",
+			description: "Root path serves SPA entry point",
+		},
+		{
+			name:        "spa_fallback_for_unknown_route",
+			method:      http.MethodGet,
+			path:        "/certificates",
+			wantStatus:  http.StatusOK,
+			wantBody:    "certctl dashboard",
+			description: "React Router routes fall through to index.html",
+		},
+		{
+			name:        "spa_fallback_deep_route",
+			method:      http.MethodGet,
+			path:        "/certificates/mc-api-prod/detail",
+			wantStatus:  http.StatusOK,
+			wantBody:    "certctl dashboard",
+			description: "Deep React Router routes also fall through to SPA",
+		},
+	}
+
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			req := httptest.NewRequest(tc.method, tc.path, nil)
+			w := httptest.NewRecorder()
+			handler.ServeHTTP(w, req)
+
+			if w.Code != tc.wantStatus {
+				t.Errorf("status = %d, want %d (%s)", w.Code, tc.wantStatus, tc.description)
+			}
+			body := w.Body.String()
+			if tc.wantBody != "" && !strings.Contains(body, tc.wantBody) {
+				t.Errorf("body %q does not contain %q (%s)", body, tc.wantBody, tc.description)
+			}
+			if tc.wantBodyPrefix != "" && !strings.HasPrefix(body, tc.wantBodyPrefix) {
+				t.Errorf("body %q does not start with %q (%s)", body, tc.wantBodyPrefix, tc.description)
+			}
+		})
+	}
+}
+
+// TestBuildFinalHandler_NoDashboard pins the API-only (dashboard-absent)
+// dispatch behavior. When web/dist/index.html is missing, everything that's
+// not a no-auth bypass route falls through to the authenticated apiHandler
+// (pre-M-001 behavior for headless deployments). EST/SCEP/PKI still ride the
+// no-auth chain.
+func TestBuildFinalHandler_NoDashboard(t *testing.T) {
+	authHandler := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte("AUTH"))
+	})
+	noAuthHandler := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte("NOAUTH"))
+	})
+
+	handler := buildFinalHandler(authHandler, noAuthHandler, "/nonexistent", false /* dashboardEnabled */)
+
+	tests := []struct {
+		name     string
+		path     string
+		wantBody string
+	}{
+		{"est_still_no_auth", "/.well-known/est/cacerts", "NOAUTH"},
+		{"scep_still_no_auth", "/scep", "NOAUTH"},
+		{"pki_still_no_auth", "/.well-known/pki/crl/x", "NOAUTH"},
+		{"health_still_no_auth", "/health", "NOAUTH"},
+		{"api_still_auth", "/api/v1/certificates", "AUTH"},
+		// The difference: non-API, non-special paths go through auth chain when
+		// there's no dashboard to serve (preserves legacy headless behavior).
+		{"unknown_path_falls_through_to_auth", "/", "AUTH"},
+		{"unknown_deep_path_falls_through_to_auth", "/random/path", "AUTH"},
+	}
+
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			req := httptest.NewRequest(http.MethodGet, tc.path, nil)
+			w := httptest.NewRecorder()
+			handler.ServeHTTP(w, req)
+			if w.Code != http.StatusOK {
+				t.Errorf("status = %d, want 200", w.Code)
+			}
+			if got := w.Body.String(); !strings.Contains(got, tc.wantBody) {
+				t.Errorf("body = %q, want to contain %q", got, tc.wantBody)
+			}
+		})
+	}
+}

cmd/server/main_test.go

@@ -0,0 +1,646 @@
+package main
+
+import (
+	"context"
+	"fmt"
+	"log/slog"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"strings"
+	"testing"
+
+	"github.com/shankar0123/certctl/internal/api/middleware"
+	"github.com/shankar0123/certctl/internal/api/router"
+	"github.com/shankar0123/certctl/internal/config"
+	"github.com/shankar0123/certctl/internal/service"
+)
+
+// TestMain_HealthEndpointBypassesAuth verifies that health check endpoints
+// bypass auth middleware while protected API endpoints require auth.
+// This is the most critical test — it validates the core routing pattern used in main.go.
+func TestMain_HealthEndpointBypassesAuth(t *testing.T) {
+	// Simulate the finalHandler logic from main.go with minimal setup
+	// Create handler functions for health endpoints
+	healthHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		w.Write([]byte(`{"status":"ok"}`))
+	})
+
+	readyHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		w.Write([]byte(`{"status":"ready"}`))
+	})
+
+	authInfoHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		w.Write([]byte(`{"auth_type":"api-key"}`))
+	})
+
+	// Protected API endpoint
+	certHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		w.Write([]byte(`[]`))
+	})
+
+	// Build the handler chain the same way main.go does
+	authMiddleware := middleware.NewAuthWithNamedKeys([]middleware.NamedAPIKey{
+		{Name: "test", Key: "test-secret-key"},
+	})
+
+	// API handler with auth
+	authHandler := middleware.Chain(certHandler,
+		middleware.RequestID,
+		middleware.Recovery,
+		authMiddleware,
+	)
+
+	// Create finalHandler matching main.go logic
+	finalHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		path := r.URL.Path
+		switch path {
+		case "/health":
+			healthHandler.ServeHTTP(w, r)
+		case "/ready":
+			readyHandler.ServeHTTP(w, r)
+		case "/api/v1/auth/info":
+			authInfoHandler.ServeHTTP(w, r)
+		case "/api/v1/certificates":
+			authHandler.ServeHTTP(w, r)
+		default:
+			http.Error(w, "Not Found", http.StatusNotFound)
+		}
+	})
+
+	tests := []struct {
+		name           string
+		path           string
+		method         string
+		bypassesAuth   bool
+		expectedStatus int
+	}{
+		{
+			name:           "GET /health without auth",
+			path:           "/health",
+			method:         "GET",
+			bypassesAuth:   true,
+			expectedStatus: http.StatusOK,
+		},
+		{
+			name:           "GET /ready without auth",
+			path:           "/ready",
+			method:         "GET",
+			bypassesAuth:   true,
+			expectedStatus: http.StatusOK,
+		},
+		{
+			name:           "GET /api/v1/auth/info without auth",
+			path:           "/api/v1/auth/info",
+			method:         "GET",
+			bypassesAuth:   true,
+			expectedStatus: http.StatusOK,
+		},
+		{
+			name:           "GET /api/v1/certificates without auth (should fail)",
+			path:           "/api/v1/certificates",
+			method:         "GET",
+			bypassesAuth:   false,
+			expectedStatus: http.StatusUnauthorized,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			req := httptest.NewRequest(tt.method, tt.path, nil)
+			w := httptest.NewRecorder()
+
+			finalHandler.ServeHTTP(w, req)
+
+			if tt.bypassesAuth && w.Code != tt.expectedStatus {
+				t.Errorf("endpoint %s should bypass auth, got status %d, expected %d",
+					tt.path, w.Code, tt.expectedStatus)
+			}
+
+			if !tt.bypassesAuth && w.Code != tt.expectedStatus {
+				t.Logf("endpoint %s requires auth, got status %d, expected %d (auth middleware working)",
+					tt.path, w.Code, tt.expectedStatus)
+			}
+		})
+	}
+}
+
+// TestMain_HealthHandlersRespond verifies health endpoints return correct responses.
+func TestMain_HealthHandlersRespond(t *testing.T) {
+	healthHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		w.Write([]byte(`{"status":"ok"}`))
+	})
+
+	req := httptest.NewRequest("GET", "/health", nil)
+	w := httptest.NewRecorder()
+
+	healthHandler.ServeHTTP(w, req)
+
+	if w.Code != http.StatusOK {
+		t.Errorf("expected status 200, got %d", w.Code)
+	}
+
+	if body := w.Body.String(); body != `{"status":"ok"}` {
+		t.Errorf("expected body '{\"status\":\"ok\"}', got '%s'", body)
+	}
+}
+
+// TestMain_AuthMiddlewareRejectsUnauthorized verifies auth middleware works.
+func TestMain_AuthMiddlewareRejectsUnauthorized(t *testing.T) {
+	// Create a protected endpoint
+	protectedHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		w.Write([]byte(`{"data":"protected"}`))
+	})
+
+	// Wrap with auth middleware
+	authMiddleware := middleware.NewAuthWithNamedKeys([]middleware.NamedAPIKey{
+		{Name: "test", Key: "test-secret-key"},
+	})
+
+	chainedHandler := middleware.Chain(protectedHandler, authMiddleware)
+
+	// Request without auth should be rejected
+	req := httptest.NewRequest("GET", "/api/v1/protected", nil)
+	w := httptest.NewRecorder()
+
+	chainedHandler.ServeHTTP(w, req)
+
+	if w.Code != http.StatusUnauthorized {
+		t.Errorf("expected status 401 for unauthorized request, got %d", w.Code)
+	}
+}
+
+// TestMain_AuthMiddlewareAllowsWithValidKey verifies auth middleware allows valid keys.
+func TestMain_AuthMiddlewareAllowsWithValidKey(t *testing.T) {
+	testKey := "test-secret-key"
+
+	// Create a protected endpoint
+	protectedHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		w.Write([]byte(`{"data":"protected"}`))
+	})
+
+	// Wrap with auth middleware
+	authMiddleware := middleware.NewAuthWithNamedKeys([]middleware.NamedAPIKey{
+		{Name: "test", Key: testKey},
+	})
+
+	chainedHandler := middleware.Chain(protectedHandler, authMiddleware)
+
+	// Request with valid auth should be allowed
+	req := httptest.NewRequest("GET", "/api/v1/protected", nil)
+	req.Header.Set("Authorization", "Bearer "+testKey)
+	w := httptest.NewRecorder()
+
+	chainedHandler.ServeHTTP(w, req)
+
+	if w.Code != http.StatusOK {
+		t.Errorf("expected status 200 for authorized request, got %d", w.Code)
+	}
+}
+
+// TestMain_ServerConfigFromEnvironment verifies config.Load() reads env vars correctly.
+func TestMain_ServerConfigFromEnvironment(t *testing.T) {
+	// Save original env vars
+	oldAuthType := os.Getenv("CERTCTL_AUTH_TYPE")
+	oldServerHost := os.Getenv("CERTCTL_SERVER_HOST")
+	oldServerPort := os.Getenv("CERTCTL_SERVER_PORT")
+	oldTLSCert := os.Getenv("CERTCTL_SERVER_TLS_CERT_PATH")
+	oldTLSKey := os.Getenv("CERTCTL_SERVER_TLS_KEY_PATH")
+	defer func() {
+		if oldAuthType != "" {
+			os.Setenv("CERTCTL_AUTH_TYPE", oldAuthType)
+		} else {
+			os.Unsetenv("CERTCTL_AUTH_TYPE")
+		}
+		if oldServerHost != "" {
+			os.Setenv("CERTCTL_SERVER_HOST", oldServerHost)
+		} else {
+			os.Unsetenv("CERTCTL_SERVER_HOST")
+		}
+		if oldServerPort != "" {
+			os.Setenv("CERTCTL_SERVER_PORT", oldServerPort)
+		} else {
+			os.Unsetenv("CERTCTL_SERVER_PORT")
+		}
+		if oldTLSCert != "" {
+			os.Setenv("CERTCTL_SERVER_TLS_CERT_PATH", oldTLSCert)
+		} else {
+			os.Unsetenv("CERTCTL_SERVER_TLS_CERT_PATH")
+		}
+		if oldTLSKey != "" {
+			os.Setenv("CERTCTL_SERVER_TLS_KEY_PATH", oldTLSKey)
+		} else {
+			os.Unsetenv("CERTCTL_SERVER_TLS_KEY_PATH")
+		}
+	}()
+
+	// HTTPS-only control plane: Validate() refuses to pass without a readable
+	// cert/key pair on disk. Materialize a throwaway ECDSA P-256 pair using the
+	// same generator cmd/server/tls_test.go uses for the certHolder tests.
+	dir := t.TempDir()
+	certPath := dir + "/server.crt"
+	keyPath := dir + "/server.key"
+	generateTestCert(t, certPath, keyPath, "main-test-cn")
+
+	// Set test env vars
+	os.Setenv("CERTCTL_AUTH_TYPE", "none")
+	os.Setenv("CERTCTL_SERVER_HOST", "127.0.0.1")
+	os.Setenv("CERTCTL_SERVER_PORT", "8080")
+	os.Setenv("CERTCTL_SERVER_TLS_CERT_PATH", certPath)
+	os.Setenv("CERTCTL_SERVER_TLS_KEY_PATH", keyPath)
+
+	cfg, err := config.Load()
+	if err != nil {
+		t.Fatalf("Failed to load config from env vars: %v", err)
+	}
+
+	if cfg.Auth.Type != "none" {
+		t.Errorf("Expected auth type 'none', got '%s'", cfg.Auth.Type)
+	}
+
+	if cfg.Server.Host != "127.0.0.1" {
+		t.Errorf("Expected server host '127.0.0.1', got '%s'", cfg.Server.Host)
+	}
+
+	if cfg.Server.Port != 8080 {
+		t.Errorf("Expected server port 8080, got %d", cfg.Server.Port)
+	}
+}
+
+// TestMain_AuthTypeConfiguration verifies auth type is read from config.
+func TestMain_AuthTypeConfiguration(t *testing.T) {
+	// Save original env vars
+	oldAuthType := os.Getenv("CERTCTL_AUTH_TYPE")
+	oldAuthSecret := os.Getenv("CERTCTL_AUTH_SECRET")
+	oldTLSCert := os.Getenv("CERTCTL_SERVER_TLS_CERT_PATH")
+	oldTLSKey := os.Getenv("CERTCTL_SERVER_TLS_KEY_PATH")
+	defer func() {
+		if oldAuthType != "" {
+			os.Setenv("CERTCTL_AUTH_TYPE", oldAuthType)
+		} else {
+			os.Unsetenv("CERTCTL_AUTH_TYPE")
+		}
+		if oldAuthSecret != "" {
+			os.Setenv("CERTCTL_AUTH_SECRET", oldAuthSecret)
+		} else {
+			os.Unsetenv("CERTCTL_AUTH_SECRET")
+		}
+		if oldTLSCert != "" {
+			os.Setenv("CERTCTL_SERVER_TLS_CERT_PATH", oldTLSCert)
+		} else {
+			os.Unsetenv("CERTCTL_SERVER_TLS_CERT_PATH")
+		}
+		if oldTLSKey != "" {
+			os.Setenv("CERTCTL_SERVER_TLS_KEY_PATH", oldTLSKey)
+		} else {
+			os.Unsetenv("CERTCTL_SERVER_TLS_KEY_PATH")
+		}
+	}()
+
+	// HTTPS-only control plane: config.Load()→Validate() refuses to pass
+	// without a readable cert/key pair. Mint one throwaway pair for the whole
+	// sub-test cohort — auth type toggles don't care about the TLS surface.
+	dir := t.TempDir()
+	certPath := dir + "/server.crt"
+	keyPath := dir + "/server.key"
+	generateTestCert(t, certPath, keyPath, "main-test-cn")
+	os.Setenv("CERTCTL_SERVER_TLS_CERT_PATH", certPath)
+	os.Setenv("CERTCTL_SERVER_TLS_KEY_PATH", keyPath)
+
+	// Set auth secret for api-key mode
+	os.Setenv("CERTCTL_AUTH_SECRET", "test-secret")
+
+	testCases := []string{"api-key", "none"}
+
+	for _, authType := range testCases {
+		t.Run(fmt.Sprintf("auth_type_%s", authType), func(t *testing.T) {
+			os.Setenv("CERTCTL_AUTH_TYPE", authType)
+
+			cfg, err := config.Load()
+			if err != nil {
+				t.Fatalf("Failed to load config: %v", err)
+			}
+
+			if cfg.Auth.Type != authType {
+				t.Errorf("Expected auth type '%s', got '%s'", authType, cfg.Auth.Type)
+			}
+		})
+	}
+}
+
+// TestMain_MiddlewareChainConstruction tests that middleware can be properly chained.
+func TestMain_MiddlewareChainConstruction(t *testing.T) {
+	// Test that the middleware.Chain function works as expected
+	baseHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		w.Write([]byte("success"))
+	})
+
+	// Chain with RequestID and Recovery middleware
+	chainedHandler := middleware.Chain(baseHandler,
+		middleware.RequestID,
+		middleware.Recovery,
+	)
+
+	req := httptest.NewRequest("GET", "/test", nil)
+	w := httptest.NewRecorder()
+
+	chainedHandler.ServeHTTP(w, req)
+
+	if w.Code != http.StatusOK {
+		t.Errorf("expected status 200, got %d", w.Code)
+	}
+
+	if body := w.Body.String(); body != "success" {
+		t.Errorf("expected body 'success', got '%s'", body)
+	}
+}
+
+// TestMain_RequestIDMiddleware verifies RequestID is added to responses.
+func TestMain_RequestIDMiddleware(t *testing.T) {
+	baseHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+	})
+
+	// Wrap with RequestID middleware
+	chainedHandler := middleware.Chain(baseHandler, middleware.RequestID)
+
+	req := httptest.NewRequest("GET", "/test", nil)
+	w := httptest.NewRecorder()
+
+	chainedHandler.ServeHTTP(w, req)
+
+	// RequestID should be set in response header
+	if rid := w.Header().Get("X-Request-ID"); rid == "" {
+		t.Logf("X-Request-ID header not present (middleware may work differently)")
+	} else {
+		t.Logf("X-Request-ID header set: %s", rid)
+	}
+}
+
+// TestMain_RecoveryMiddlewareHandlesPanic verifies recovery middleware works.
+func TestMain_RecoveryMiddlewareHandlesPanic(t *testing.T) {
+	panicHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		panic("test panic")
+	})
+
+	// Wrap with recovery middleware
+	chainedHandler := middleware.Chain(panicHandler, middleware.Recovery)
+
+	req := httptest.NewRequest("GET", "/test", nil)
+	w := httptest.NewRecorder()
+
+	// Should not panic
+	chainedHandler.ServeHTTP(w, req)
+
+	// Should return 500 error
+	if w.Code != http.StatusInternalServerError {
+		t.Logf("Expected 500 for panicked handler, got %d", w.Code)
+	}
+}
+
+// TestMain_ServiceInitialization tests that services can be instantiated.
+// This validates the initialization pattern from main.go without needing a real DB.
+func TestMain_ServiceInitialization(t *testing.T) {
+	logger := slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{
+		Level: slog.LevelInfo,
+	}))
+
+	// Create test issuer registry (same as main.go does)
+	issuerRegistry := service.NewIssuerRegistry(logger)
+
+	if issuerRegistry == nil {
+		t.Fatal("issuer registry should not be nil")
+	}
+
+	// Verify the registry has a Len() method (used in main.go)
+	count := issuerRegistry.Len()
+	if count < 0 {
+		t.Errorf("issuer registry length should be >= 0, got %d", count)
+	}
+}
+
+// TestMain_CORSMiddlewareSetHeaders verifies CORS headers are set.
+func TestMain_CORSMiddlewareSetHeaders(t *testing.T) {
+	baseHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+	})
+
+	corsMiddleware := middleware.NewCORS(middleware.CORSConfig{
+		AllowedOrigins: []string{"http://example.com"},
+	})
+
+	chainedHandler := middleware.Chain(baseHandler, corsMiddleware)
+
+	req := httptest.NewRequest("GET", "/test", nil)
+	req.Header.Set("Origin", "http://example.com")
+	w := httptest.NewRecorder()
+
+	chainedHandler.ServeHTTP(w, req)
+
+	// CORS middleware should set access control headers
+	if acah := w.Header().Get("Access-Control-Allow-Origin"); acah == "" {
+		t.Logf("Access-Control-Allow-Origin not set (may be by design)")
+	}
+}
+
+// TestMain_AuthNoneMode verifies auth can be disabled.
+func TestMain_AuthNoneMode(t *testing.T) {
+	protectedHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		w.Write([]byte(`{"data":"protected"}`))
+	})
+
+	// Wrap with auth middleware in "none" mode
+	// auth=none equivalent: empty named-keys list is a no-op pass-through.
+	authMiddleware := middleware.NewAuthWithNamedKeys(nil)
+
+	chainedHandler := middleware.Chain(protectedHandler, authMiddleware)
+
+	// Request without auth should be allowed in "none" mode
+	req := httptest.NewRequest("GET", "/api/v1/protected", nil)
+	w := httptest.NewRecorder()
+
+	chainedHandler.ServeHTTP(w, req)
+
+	if w.Code != http.StatusOK {
+		t.Errorf("expected status 200 in 'none' auth mode, got %d", w.Code)
+	}
+}
+
+// TestMain_RouterRegistration tests that router registration works.
+func TestMain_RouterRegistration(t *testing.T) {
+	r := router.New()
+
+	// Register a test handler
+	r.RegisterFunc("GET /test", func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		w.Write([]byte("test"))
+	})
+
+	// Request the route
+	req := httptest.NewRequest("GET", "/test", nil)
+	w := httptest.NewRecorder()
+
+	r.ServeHTTP(w, req)
+
+	// Route should be registered and accessible
+	if w.Code == http.StatusNotFound {
+		t.Errorf("route not registered, got 404")
+	} else if w.Code == http.StatusOK {
+		t.Logf("route registered successfully")
+	}
+}
+
+// TestMain_RateLimiterIntegration tests rate limiter middleware works.
+func TestMain_RateLimiterIntegration(t *testing.T) {
+	baseHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+	})
+
+	// Create rate limiter with 10 RPS, 1 burst
+	rateLimiter := middleware.NewRateLimiter(middleware.RateLimitConfig{
+		RPS:       10,
+		BurstSize: 1,
+	})
+
+	chainedHandler := middleware.Chain(baseHandler, rateLimiter)
+
+	// First request should succeed
+	req := httptest.NewRequest("GET", "/test", nil)
+	w := httptest.NewRecorder()
+	chainedHandler.ServeHTTP(w, req)
+
+	if w.Code == http.StatusServiceUnavailable {
+		t.Logf("rate limiter is active")
+	} else {
+		t.Logf("rate limiter allowed request (status %d)", w.Code)
+	}
+}
+
+// TestMain_ContentTypeMiddleware verifies content type is set correctly.
+func TestMain_ContentTypeMiddleware(t *testing.T) {
+	baseHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		w.Write([]byte(`{"status":"ok"}`))
+	})
+
+	// Wrap with middleware that sets Content-Type
+	chainedHandler := middleware.Chain(baseHandler, middleware.ContentType)
+
+	req := httptest.NewRequest("GET", "/api/v1/test", nil)
+	w := httptest.NewRecorder()
+
+	chainedHandler.ServeHTTP(w, req)
+
+	// Verify response
+	if w.Code != http.StatusOK {
+		t.Errorf("expected status 200, got %d", w.Code)
+	}
+
+	// ContentType middleware should set header
+	if ct := w.Header().Get("Content-Type"); ct != "" {
+		t.Logf("Content-Type header set: %s", ct)
+	}
+}
+
+// TestMain_ContextPropagation verifies context is propagated through middleware.
+func TestMain_ContextPropagation(t *testing.T) {
+	type contextKey string
+	testKey := contextKey("test-key")
+	testValue := "test-value"
+
+	baseHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		val := r.Context().Value(testKey)
+		if val == testValue {
+			w.WriteHeader(http.StatusOK)
+		} else {
+			w.WriteHeader(http.StatusInternalServerError)
+		}
+	})
+
+	chainedHandler := middleware.Chain(baseHandler, middleware.RequestID)
+
+	req := httptest.NewRequest("GET", "/test", nil)
+	// Add context value before request
+	req = req.WithContext(context.WithValue(req.Context(), testKey, testValue))
+
+	w := httptest.NewRecorder()
+	chainedHandler.ServeHTTP(w, req)
+
+	if w.Code != http.StatusOK {
+		t.Logf("Context value may not be propagated (status %d), this may be expected", w.Code)
+	}
+}
+
+// TestPreflightSCEPChallengePassword is the H-2 regression guard for the
+// startup pre-flight check. The helper MUST return a non-nil error whenever
+// SCEP is enabled with an empty challenge password — that configuration
+// previously allowed unauthenticated certificate enrollment (CWE-306).
+// Disabled-SCEP and configured-password cases must pass cleanly.
+func TestPreflightSCEPChallengePassword(t *testing.T) {
+	tests := []struct {
+		name              string
+		enabled           bool
+		challengePassword string
+		wantErr           bool
+		wantErrSubstring  string
+	}{
+		{
+			name:              "disabled_empty_password_ok",
+			enabled:           false,
+			challengePassword: "",
+			wantErr:           false,
+		},
+		{
+			name:              "disabled_with_password_ok",
+			enabled:           false,
+			challengePassword: "leftover-value",
+			wantErr:           false,
+		},
+		{
+			name:              "enabled_empty_password_rejected",
+			enabled:           true,
+			challengePassword: "",
+			wantErr:           true,
+			wantErrSubstring:  "CERTCTL_SCEP_CHALLENGE_PASSWORD",
+		},
+		{
+			name:              "enabled_with_password_ok",
+			enabled:           true,
+			challengePassword: "hunter2",
+			wantErr:           false,
+		},
+		{
+			name:              "enabled_single_char_password_ok",
+			enabled:           true,
+			challengePassword: "x",
+			wantErr:           false,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			err := preflightSCEPChallengePassword(tt.enabled, tt.challengePassword)
+			if tt.wantErr {
+				if err == nil {
+					t.Fatalf("expected error, got nil")
+				}
+				if tt.wantErrSubstring != "" && !strings.Contains(err.Error(), tt.wantErrSubstring) {
+					t.Errorf("expected error to mention %q, got: %v", tt.wantErrSubstring, err)
+				}
+				if !strings.Contains(err.Error(), "CWE-306") {
+					t.Errorf("expected error to cite CWE-306 for traceability, got: %v", err)
+				}
+			} else if err != nil {
+				t.Errorf("expected no error, got: %v", err)
+			}
+		})
+	}
+}

cmd/server/preflight_test.go

@@ -0,0 +1,100 @@
+package main
+
+import (
+	"context"
+	"strings"
+	"testing"
+
+	"github.com/shankar0123/certctl/internal/service"
+)
+
+// fakeIssuerConn implements service.IssuerConnector enough for preflight tests.
+type fakeIssuerConn struct {
+	caCertPEM string
+	caCertErr error
+}
+
+func (f *fakeIssuerConn) IssueCertificate(ctx context.Context, commonName string, sans []string, csrPEM string, ekus []string, maxTTLSeconds int) (*service.IssuanceResult, error) {
+	return nil, nil
+}
+func (f *fakeIssuerConn) RenewCertificate(ctx context.Context, commonName string, sans []string, csrPEM string, ekus []string, maxTTLSeconds int) (*service.IssuanceResult, error) {
+	return nil, nil
+}
+func (f *fakeIssuerConn) RevokeCertificate(ctx context.Context, serial string, reason string) error {
+	return nil
+}
+func (f *fakeIssuerConn) GenerateCRL(ctx context.Context, revokedCerts []service.CRLEntry) ([]byte, error) {
+	return nil, nil
+}
+func (f *fakeIssuerConn) SignOCSPResponse(ctx context.Context, req service.OCSPSignRequest) ([]byte, error) {
+	return nil, nil
+}
+func (f *fakeIssuerConn) GetCACertPEM(ctx context.Context) (string, error) {
+	return f.caCertPEM, f.caCertErr
+}
+func (f *fakeIssuerConn) GetRenewalInfo(ctx context.Context, certPEM string) (*service.RenewalInfoResult, error) {
+	return nil, nil
+}
+
+// TestPreflightEnrollmentIssuer covers Bundle-4 / L-005 startup validation
+// for EST/SCEP issuer binding.
+func TestPreflightEnrollmentIssuer(t *testing.T) {
+	cases := []struct {
+		name        string
+		issuer      service.IssuerConnector
+		wantErr     bool
+		errContains string
+	}{
+		{
+			name:        "nil_connector_fails",
+			issuer:      nil,
+			wantErr:     true,
+			errContains: "connector is nil",
+		},
+		{
+			name: "issuer_returns_error_fails",
+			issuer: &fakeIssuerConn{
+				caCertErr: errStub("ACME issuers do not provide a static CA certificate"),
+			},
+			wantErr:     true,
+			errContains: "cannot serve CA certificate",
+		},
+		{
+			name: "issuer_returns_empty_pem_fails",
+			issuer: &fakeIssuerConn{
+				caCertPEM: "",
+				caCertErr: nil,
+			},
+			wantErr:     true,
+			errContains: "empty PEM",
+		},
+		{
+			name: "issuer_returns_valid_pem_succeeds",
+			issuer: &fakeIssuerConn{
+				caCertPEM: "-----BEGIN CERTIFICATE-----\nMIIB...\n-----END CERTIFICATE-----",
+				caCertErr: nil,
+			},
+			wantErr: false,
+		},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			err := preflightEnrollmentIssuer(context.Background(), "EST", "iss-test", tc.issuer)
+			if tc.wantErr && err == nil {
+				t.Fatalf("expected error, got nil")
+			}
+			if !tc.wantErr && err != nil {
+				t.Fatalf("unexpected error: %v", err)
+			}
+			if tc.wantErr && tc.errContains != "" && !strings.Contains(err.Error(), tc.errContains) {
+				t.Fatalf("error %q missing substring %q", err.Error(), tc.errContains)
+			}
+		})
+	}
+}
+
+// errStub is a tiny error wrapper so test cases can use string literals
+// without importing fmt in every test struct entry.
+type errStub string
+
+func (e errStub) Error() string { return string(e) }

cmd/server/tls.go

@@ -0,0 +1,164 @@
+package main
+
+import (
+	"crypto/tls"
+	"fmt"
+	"log/slog"
+	"os"
+	"os/signal"
+	"sync"
+	"syscall"
+)
+
+// certHolder stores the server's TLS certificate under a mutex so it can be
+// swapped atomically by a SIGHUP handler without restarting the server. A
+// *tls.Config that wires GetCertificate → (*certHolder).GetCertificate reads
+// through the holder on every ClientHello, so a successful reload takes
+// effect on the next new connection immediately and without dropping
+// in-flight requests.
+//
+// Concurrency: GetCertificate is invoked from crypto/tls handshake goroutines
+// on every new inbound connection; Reload is invoked from the SIGHUP watcher
+// goroutine. sync.Mutex is sufficient — TLS handshakes are not an inner-loop
+// hot path and the critical section is a single pointer read.
+type certHolder struct {
+	mu       sync.Mutex
+	cert     *tls.Certificate
+	certPath string
+	keyPath  string
+}
+
+// newCertHolder loads the initial cert+key pair from disk and returns a
+// holder ready to serve handshakes. Returns a non-nil error if either file
+// is missing, unreadable, or the pair does not round-trip through
+// tls.LoadX509KeyPair (for example the key does not sign the cert). The
+// caller is expected to treat a non-nil error as a fail-loud startup gate
+// and os.Exit(1) — the HTTPS-everywhere milestone (§3 locked decisions)
+// prohibits plaintext HTTP fallback.
+func newCertHolder(certPath, keyPath string) (*certHolder, error) {
+	cert, err := tls.LoadX509KeyPair(certPath, keyPath)
+	if err != nil {
+		return nil, fmt.Errorf("load TLS cert/key (cert=%q key=%q): %w", certPath, keyPath, err)
+	}
+	return &certHolder{
+		cert:     &cert,
+		certPath: certPath,
+		keyPath:  keyPath,
+	}, nil
+}
+
+// GetCertificate is the tls.Config.GetCertificate hook. Returns the current
+// cert under the holder's mutex. ClientHelloInfo is ignored — the control
+// plane does not multiplex by SNI.
+func (h *certHolder) GetCertificate(_ *tls.ClientHelloInfo) (*tls.Certificate, error) {
+	h.mu.Lock()
+	defer h.mu.Unlock()
+	return h.cert, nil
+}
+
+// Reload re-reads the cert+key pair from disk and swaps the holder
+// atomically on success. On failure the holder retains its previous cert
+// and the error is propagated to the caller — the SIGHUP watcher logs and
+// keeps serving the previous cert rather than crashing on a bad reload.
+// This is deliberately "fail-safe on reload, fail-loud on startup": an
+// operator rotating certs wants a recoverable error, not a restart loop.
+func (h *certHolder) Reload() error {
+	cert, err := tls.LoadX509KeyPair(h.certPath, h.keyPath)
+	if err != nil {
+		return fmt.Errorf("reload TLS cert/key (cert=%q key=%q): %w", h.certPath, h.keyPath, err)
+	}
+	h.mu.Lock()
+	h.cert = &cert
+	h.mu.Unlock()
+	return nil
+}
+
+// watchSIGHUP installs a signal handler that calls Reload() on each SIGHUP.
+// The returned stop function closes the internal done channel and stops
+// signal delivery so the goroutine can exit cleanly during shutdown. Errors
+// from Reload are logged but do not terminate the watcher — the operator
+// can fix the files and send another SIGHUP.
+//
+// Defensive design note: this deliberately does NOT panic on Reload error
+// even though HTTPS is mission-critical. A rotation that writes half-files
+// (operator overwrites cert.pem then key.pem as two separate copies) would
+// otherwise crash the server mid-rotation. Logging + retaining the old
+// cert gives the operator a bounded window to fix and re-SIGHUP.
+func (h *certHolder) watchSIGHUP(logger *slog.Logger) (stop func()) {
+	ch := make(chan os.Signal, 1)
+	signal.Notify(ch, syscall.SIGHUP)
+	done := make(chan struct{})
+	go func() {
+		for {
+			select {
+			case <-ch:
+				if err := h.Reload(); err != nil {
+					logger.Error("TLS cert reload failed; continuing with previous cert",
+						"error", err,
+						"cert_path", h.certPath,
+						"key_path", h.keyPath)
+					continue
+				}
+				logger.Info("TLS cert reloaded via SIGHUP",
+					"cert_path", h.certPath,
+					"key_path", h.keyPath)
+			case <-done:
+				signal.Stop(ch)
+				return
+			}
+		}
+	}()
+	return func() { close(done) }
+}
+
+// buildServerTLSConfig returns the TLS 1.3-only *tls.Config for the HTTPS
+// server. Pinned per HTTPS-everywhere milestone §2.1 + §3 locked decisions:
+//
+//   - MinVersion: TLS 1.3 (no TLS 1.2 escape hatch). Go 1.25's crypto/tls
+//     automatically rejects older versions.
+//   - CurvePreferences: explicit [X25519, P-256]. Explicit ordering keeps
+//     the handshake deterministic and documents the accepted curves.
+//   - No CipherSuites field: TLS 1.3 cipher suites are not negotiable in
+//     the handshake (all three mandatory suites — AES-128-GCM-SHA256,
+//     AES-256-GCM-SHA384, CHACHA20-POLY1305-SHA256 — are always offered).
+//     Go's crypto/tls ignores CipherSuites for TLS 1.3.
+//   - GetCertificate: reads through the holder so SIGHUP rotations take
+//     effect on the next new connection without a restart. Setting
+//     tls.Config.Certificates directly would pin the first-loaded cert
+//     and defeat SIGHUP reload.
+func buildServerTLSConfig(holder *certHolder) *tls.Config {
+	return &tls.Config{
+		MinVersion:       tls.VersionTLS13,
+		CurvePreferences: []tls.CurveID{tls.X25519, tls.CurveP256},
+		GetCertificate:   holder.GetCertificate,
+	}
+}
+
+// preflightServerTLS is the fail-loud startup gate for HTTPS. Returns a
+// non-nil error when the TLS configuration is missing or the cert+key pair
+// cannot be parsed, so the caller refuses to start the control plane
+// (HTTPS-everywhere §3 locked decisions: no plaintext HTTP fallback).
+//
+// Duplicates the emptiness + stat + parse checks in config.Validate() for
+// defense in depth, mirroring the pattern established by
+// preflightSCEPChallengePassword (which itself duplicates
+// config.Validate()'s SCEP check for CWE-306). Extracted into a separate
+// function so the gate is unit-testable without booting the full server.
+func preflightServerTLS(certPath, keyPath string) error {
+	if certPath == "" {
+		return fmt.Errorf("CERTCTL_SERVER_TLS_CERT_PATH is empty: HTTPS-only control plane refuses to start (see docs/tls.md)")
+	}
+	if keyPath == "" {
+		return fmt.Errorf("CERTCTL_SERVER_TLS_KEY_PATH is empty: HTTPS-only control plane refuses to start (see docs/tls.md)")
+	}
+	if _, err := os.Stat(certPath); err != nil {
+		return fmt.Errorf("TLS cert file %q unreadable: %w (see docs/tls.md)", certPath, err)
+	}
+	if _, err := os.Stat(keyPath); err != nil {
+		return fmt.Errorf("TLS key file %q unreadable: %w (see docs/tls.md)", keyPath, err)
+	}
+	if _, err := tls.LoadX509KeyPair(certPath, keyPath); err != nil {
+		return fmt.Errorf("TLS cert/key pair invalid (cert=%q key=%q): %w (see docs/tls.md)", certPath, keyPath, err)
+	}
+	return nil
+}

cmd/server/tls_test.go

@@ -0,0 +1,418 @@
+package main
+
+import (
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/tls"
+	"crypto/x509"
+	"crypto/x509/pkix"
+	"encoding/pem"
+	"errors"
+	"io"
+	"log/slog"
+	"math/big"
+	"net"
+	"os"
+	"path/filepath"
+	"sync"
+	"syscall"
+	"testing"
+	"time"
+)
+
+// generateTestCert writes a PEM-encoded self-signed leaf cert + ECDSA P-256
+// key pair to certPath/keyPath. The subject is derived from cn so tests can
+// tell reloaded certs apart from original certs by re-parsing the served
+// Certificate and comparing the CN.
+func generateTestCert(t *testing.T, certPath, keyPath, cn string) {
+	t.Helper()
+	priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+	if err != nil {
+		t.Fatalf("ecdsa.GenerateKey: %v", err)
+	}
+	tmpl := &x509.Certificate{
+		SerialNumber: big.NewInt(time.Now().UnixNano()),
+		Subject:      pkix.Name{CommonName: cn},
+		NotBefore:    time.Now().Add(-1 * time.Hour),
+		NotAfter:     time.Now().Add(24 * time.Hour),
+		KeyUsage:     x509.KeyUsageDigitalSignature,
+		ExtKeyUsage:  []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
+		DNSNames:     []string{"localhost"},
+		IPAddresses:  []net.IP{net.ParseIP("127.0.0.1"), net.ParseIP("::1")},
+	}
+	der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &priv.PublicKey, priv)
+	if err != nil {
+		t.Fatalf("x509.CreateCertificate: %v", err)
+	}
+	certPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
+	keyDER, err := x509.MarshalECPrivateKey(priv)
+	if err != nil {
+		t.Fatalf("MarshalECPrivateKey: %v", err)
+	}
+	keyPEM := pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER})
+	if err := os.WriteFile(certPath, certPEM, 0o600); err != nil {
+		t.Fatalf("write cert: %v", err)
+	}
+	if err := os.WriteFile(keyPath, keyPEM, 0o600); err != nil {
+		t.Fatalf("write key: %v", err)
+	}
+}
+
+// readCertCN returns the CommonName from the leaf cert currently held by the
+// holder, by exercising the same GetCertificate path the tls handshake would
+// take. Lets tests assert which generation of the cert is being served.
+func readCertCN(t *testing.T, h *certHolder) string {
+	t.Helper()
+	c, err := h.GetCertificate(&tls.ClientHelloInfo{})
+	if err != nil {
+		t.Fatalf("GetCertificate: %v", err)
+	}
+	leaf, err := x509.ParseCertificate(c.Certificate[0])
+	if err != nil {
+		t.Fatalf("ParseCertificate: %v", err)
+	}
+	return leaf.Subject.CommonName
+}
+
+func silentLogger() *slog.Logger {
+	return slog.New(slog.NewTextHandler(io.Discard, &slog.HandlerOptions{Level: slog.LevelError}))
+}
+
+func TestNewCertHolder_ValidPair_LoadsCert(t *testing.T) {
+	dir := t.TempDir()
+	certPath := filepath.Join(dir, "tls.crt")
+	keyPath := filepath.Join(dir, "tls.key")
+	generateTestCert(t, certPath, keyPath, "cn-initial")
+
+	h, err := newCertHolder(certPath, keyPath)
+	if err != nil {
+		t.Fatalf("newCertHolder: %v", err)
+	}
+	if got := readCertCN(t, h); got != "cn-initial" {
+		t.Fatalf("CN mismatch: got %q want %q", got, "cn-initial")
+	}
+}
+
+func TestNewCertHolder_MissingFile_Fails(t *testing.T) {
+	_, err := newCertHolder("/nonexistent/cert.pem", "/nonexistent/key.pem")
+	if err == nil {
+		t.Fatal("expected error for missing files, got nil")
+	}
+}
+
+func TestNewCertHolder_MalformedCert_Fails(t *testing.T) {
+	dir := t.TempDir()
+	certPath := filepath.Join(dir, "bad.crt")
+	keyPath := filepath.Join(dir, "bad.key")
+	if err := os.WriteFile(certPath, []byte("not a pem cert"), 0o600); err != nil {
+		t.Fatalf("write cert: %v", err)
+	}
+	if err := os.WriteFile(keyPath, []byte("not a pem key"), 0o600); err != nil {
+		t.Fatalf("write key: %v", err)
+	}
+	_, err := newCertHolder(certPath, keyPath)
+	if err == nil {
+		t.Fatal("expected error for malformed PEM, got nil")
+	}
+}
+
+func TestCertHolder_Reload_SwapsCert(t *testing.T) {
+	dir := t.TempDir()
+	certPath := filepath.Join(dir, "tls.crt")
+	keyPath := filepath.Join(dir, "tls.key")
+	generateTestCert(t, certPath, keyPath, "cn-v1")
+
+	h, err := newCertHolder(certPath, keyPath)
+	if err != nil {
+		t.Fatalf("newCertHolder: %v", err)
+	}
+	if got := readCertCN(t, h); got != "cn-v1" {
+		t.Fatalf("initial CN: got %q want cn-v1", got)
+	}
+
+	// Rotate on disk and reload.
+	generateTestCert(t, certPath, keyPath, "cn-v2")
+	if err := h.Reload(); err != nil {
+		t.Fatalf("Reload: %v", err)
+	}
+	if got := readCertCN(t, h); got != "cn-v2" {
+		t.Fatalf("post-reload CN: got %q want cn-v2", got)
+	}
+}
+
+func TestCertHolder_Reload_FailureRetainsPreviousCert(t *testing.T) {
+	dir := t.TempDir()
+	certPath := filepath.Join(dir, "tls.crt")
+	keyPath := filepath.Join(dir, "tls.key")
+	generateTestCert(t, certPath, keyPath, "cn-v1")
+
+	h, err := newCertHolder(certPath, keyPath)
+	if err != nil {
+		t.Fatalf("newCertHolder: %v", err)
+	}
+
+	// Corrupt the cert file and attempt reload.
+	if err := os.WriteFile(certPath, []byte("garbage"), 0o600); err != nil {
+		t.Fatalf("corrupt cert: %v", err)
+	}
+	if err := h.Reload(); err == nil {
+		t.Fatal("expected Reload error for corrupt file, got nil")
+	}
+	// Holder should still serve the v1 cert.
+	if got := readCertCN(t, h); got != "cn-v1" {
+		t.Fatalf("post-failed-reload CN: got %q want cn-v1 (reload must not clobber on failure)", got)
+	}
+}
+
+func TestCertHolder_GetCertificate_Concurrent(t *testing.T) {
+	dir := t.TempDir()
+	certPath := filepath.Join(dir, "tls.crt")
+	keyPath := filepath.Join(dir, "tls.key")
+	generateTestCert(t, certPath, keyPath, "cn-concurrent")
+
+	h, err := newCertHolder(certPath, keyPath)
+	if err != nil {
+		t.Fatalf("newCertHolder: %v", err)
+	}
+
+	// 64 readers + 1 rotator for 500ms. Race detector catches any unsynchronized
+	// swap of h.cert. Rotator writes fresh files + Reload, readers call
+	// GetCertificate in a tight loop.
+	var wg sync.WaitGroup
+	done := make(chan struct{})
+	const readers = 64
+	for i := 0; i < readers; i++ {
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			for {
+				select {
+				case <-done:
+					return
+				default:
+					if _, err := h.GetCertificate(&tls.ClientHelloInfo{}); err != nil {
+						t.Errorf("GetCertificate: %v", err)
+						return
+					}
+				}
+			}
+		}()
+	}
+	wg.Add(1)
+	go func() {
+		defer wg.Done()
+		for i := 0; i < 20; i++ {
+			generateTestCert(t, certPath, keyPath, "cn-concurrent")
+			_ = h.Reload()
+			time.Sleep(10 * time.Millisecond)
+		}
+	}()
+	time.Sleep(300 * time.Millisecond)
+	close(done)
+	wg.Wait()
+}
+
+func TestCertHolder_WatchSIGHUP_ReloadsOnSignal(t *testing.T) {
+	dir := t.TempDir()
+	certPath := filepath.Join(dir, "tls.crt")
+	keyPath := filepath.Join(dir, "tls.key")
+	generateTestCert(t, certPath, keyPath, "cn-before-sighup")
+
+	h, err := newCertHolder(certPath, keyPath)
+	if err != nil {
+		t.Fatalf("newCertHolder: %v", err)
+	}
+	stop := h.watchSIGHUP(silentLogger())
+	defer stop()
+
+	// Rotate on disk, then fire SIGHUP to our own process and poll for the swap.
+	generateTestCert(t, certPath, keyPath, "cn-after-sighup")
+	if err := syscall.Kill(syscall.Getpid(), syscall.SIGHUP); err != nil {
+		t.Fatalf("SIGHUP: %v", err)
+	}
+	deadline := time.Now().Add(2 * time.Second)
+	for time.Now().Before(deadline) {
+		if readCertCN(t, h) == "cn-after-sighup" {
+			return
+		}
+		time.Sleep(10 * time.Millisecond)
+	}
+	t.Fatalf("watcher did not reload cert within 2s (CN still %q)", readCertCN(t, h))
+}
+
+func TestCertHolder_WatchSIGHUP_StopExits(t *testing.T) {
+	dir := t.TempDir()
+	certPath := filepath.Join(dir, "tls.crt")
+	keyPath := filepath.Join(dir, "tls.key")
+	generateTestCert(t, certPath, keyPath, "cn-stop")
+
+	h, err := newCertHolder(certPath, keyPath)
+	if err != nil {
+		t.Fatalf("newCertHolder: %v", err)
+	}
+	stop := h.watchSIGHUP(silentLogger())
+
+	// Closing should be synchronous and safe; a subsequent SIGHUP must not
+	// cause a reload (the watcher goroutine is gone).
+	stop()
+	time.Sleep(50 * time.Millisecond) // let goroutine exit
+
+	// After stop, the signal may still be delivered to the process but the
+	// watcher has called signal.Stop so this channel is no longer receiving.
+	// Simply assert that calling stop() twice does not panic — the goroutine
+	// has already exited, so a second close would panic on the `done`
+	// channel; we do NOT call stop twice. Instead verify no regression in
+	// the held cert.
+	if got := readCertCN(t, h); got != "cn-stop" {
+		t.Fatalf("unexpected cert rotation after stop: got %q want cn-stop", got)
+	}
+}
+
+func TestBuildServerTLSConfig_IsTLS13Only(t *testing.T) {
+	dir := t.TempDir()
+	certPath := filepath.Join(dir, "tls.crt")
+	keyPath := filepath.Join(dir, "tls.key")
+	generateTestCert(t, certPath, keyPath, "cn-cfg")
+
+	h, err := newCertHolder(certPath, keyPath)
+	if err != nil {
+		t.Fatalf("newCertHolder: %v", err)
+	}
+	cfg := buildServerTLSConfig(h)
+	if cfg.MinVersion != tls.VersionTLS13 {
+		t.Fatalf("MinVersion: got %#x want %#x (TLS 1.3)", cfg.MinVersion, tls.VersionTLS13)
+	}
+	wantCurves := []tls.CurveID{tls.X25519, tls.CurveP256}
+	if len(cfg.CurvePreferences) != len(wantCurves) {
+		t.Fatalf("CurvePreferences length: got %d want %d", len(cfg.CurvePreferences), len(wantCurves))
+	}
+	for i, c := range cfg.CurvePreferences {
+		if c != wantCurves[i] {
+			t.Fatalf("CurvePreferences[%d]: got %v want %v", i, c, wantCurves[i])
+		}
+	}
+	if cfg.GetCertificate == nil {
+		t.Fatal("GetCertificate: nil (holder not wired; SIGHUP reload would be broken)")
+	}
+	if len(cfg.Certificates) != 0 {
+		t.Fatalf("Certificates: got %d want 0 (static cert would pin the first load and defeat reload)", len(cfg.Certificates))
+	}
+}
+
+func TestBuildServerTLSConfig_Handshake_TLS12Rejected(t *testing.T) {
+	dir := t.TempDir()
+	certPath := filepath.Join(dir, "tls.crt")
+	keyPath := filepath.Join(dir, "tls.key")
+	generateTestCert(t, certPath, keyPath, "cn-handshake")
+
+	h, err := newCertHolder(certPath, keyPath)
+	if err != nil {
+		t.Fatalf("newCertHolder: %v", err)
+	}
+	serverCfg := buildServerTLSConfig(h)
+
+	ln, err := tls.Listen("tcp", "127.0.0.1:0", serverCfg)
+	if err != nil {
+		t.Fatalf("tls.Listen: %v", err)
+	}
+	defer ln.Close()
+
+	// Server loop: accept and immediately close (we only care about the
+	// handshake outcome).
+	go func() {
+		for {
+			conn, err := ln.Accept()
+			if err != nil {
+				return
+			}
+			// Force handshake so the server-side error surfaces.
+			_ = conn.(*tls.Conn).Handshake()
+			conn.Close()
+		}
+	}()
+
+	// TLS 1.3 client — should succeed.
+	clientOK := &tls.Config{
+		MinVersion:         tls.VersionTLS13,
+		MaxVersion:         tls.VersionTLS13,
+		InsecureSkipVerify: true,
+	}
+	c, err := tls.Dial("tcp", ln.Addr().String(), clientOK)
+	if err != nil {
+		t.Fatalf("TLS 1.3 dial failed (expected success): %v", err)
+	}
+	if c.ConnectionState().Version != tls.VersionTLS13 {
+		t.Fatalf("negotiated version: got %#x want TLS 1.3 (%#x)", c.ConnectionState().Version, tls.VersionTLS13)
+	}
+	c.Close()
+
+	// TLS 1.2 client — must be rejected at handshake.
+	clientOld := &tls.Config{
+		MinVersion:         tls.VersionTLS12,
+		MaxVersion:         tls.VersionTLS12,
+		InsecureSkipVerify: true,
+	}
+	if _, err := tls.Dial("tcp", ln.Addr().String(), clientOld); err == nil {
+		t.Fatal("TLS 1.2 dial succeeded; HTTPS-everywhere requires server to refuse TLS 1.2")
+	}
+}
+
+func TestPreflightServerTLS_MissingCertPath(t *testing.T) {
+	err := preflightServerTLS("", "/any/key.pem")
+	if err == nil {
+		t.Fatal("expected error for empty cert path, got nil")
+	}
+}
+
+func TestPreflightServerTLS_MissingKeyPath(t *testing.T) {
+	dir := t.TempDir()
+	certPath := filepath.Join(dir, "tls.crt")
+	keyPath := filepath.Join(dir, "tls.key")
+	generateTestCert(t, certPath, keyPath, "cn-preflight")
+	err := preflightServerTLS(certPath, "")
+	if err == nil {
+		t.Fatal("expected error for empty key path, got nil")
+	}
+}
+
+func TestPreflightServerTLS_CertFileNotReadable(t *testing.T) {
+	dir := t.TempDir()
+	keyPath := filepath.Join(dir, "tls.key")
+	if err := os.WriteFile(keyPath, []byte("k"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	err := preflightServerTLS(filepath.Join(dir, "nope.crt"), keyPath)
+	if err == nil {
+		t.Fatal("expected error for unreadable cert path, got nil")
+	}
+	if !errors.Is(err, os.ErrNotExist) {
+		t.Fatalf("expected os.ErrNotExist wrapped in error chain, got: %v", err)
+	}
+}
+
+func TestPreflightServerTLS_InvalidKeyPair(t *testing.T) {
+	dir := t.TempDir()
+	certPath := filepath.Join(dir, "tls.crt")
+	keyPath := filepath.Join(dir, "tls.key")
+	// Pair of valid cert + garbage key — files are readable but the pair
+	// doesn't round-trip tls.LoadX509KeyPair.
+	generateTestCert(t, certPath, keyPath, "cn-bad-pair")
+	if err := os.WriteFile(keyPath, []byte("-----BEGIN EC PRIVATE KEY-----\nBAD\n-----END EC PRIVATE KEY-----\n"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	err := preflightServerTLS(certPath, keyPath)
+	if err == nil {
+		t.Fatal("expected error for invalid key pair, got nil")
+	}
+}
+
+func TestPreflightServerTLS_ValidPair_NoError(t *testing.T) {
+	dir := t.TempDir()
+	certPath := filepath.Join(dir, "tls.crt")
+	keyPath := filepath.Join(dir, "tls.key")
+	generateTestCert(t, certPath, keyPath, "cn-ok")
+	if err := preflightServerTLS(certPath, keyPath); err != nil {
+		t.Fatalf("unexpected error for valid pair: %v", err)
+	}
+}

deploy/ENVIRONMENTS.md

@@ -0,0 +1,525 @@
+# certctl Docker Compose Environments
+
+This guide walks through every Docker Compose file in the `deploy/` directory. Each section explains what the environment does, when to use it, every service and environment variable, and the commands to run it. If you've never used Docker before, start with the [Prerequisites](#prerequisites) section. If you're experienced, skip to the environment you need.
+
+## Contents
+
+1. [Prerequisites](#prerequisites)
+2. [How Docker Compose Works (30-Second Version)](#how-docker-compose-works)
+3. [Base Environment (docker-compose.yml)](#base-environment)
+4. [Demo Overlay (docker-compose.demo.yml)](#demo-overlay)
+5. [Development Overlay (docker-compose.dev.yml)](#development-overlay)
+6. [Test Environment (docker-compose.test.yml)](#test-environment)
+7. [Environment Variable Reference](#environment-variable-reference)
+8. [Common Operations](#common-operations)
+
+---
+
+## Prerequisites
+
+You need two things: **Docker** (the container runtime) and **Docker Compose** (an orchestration tool that ships with Docker Desktop).
+
+On macOS:
+```bash
+brew install --cask docker
+```
+
+On Linux (Ubuntu/Debian):
+```bash
+curl -fsSL https://get.docker.com | sh
+sudo usermod -aG docker $USER
+# Log out and back in for group changes to take effect
+```
+
+Verify the install:
+```bash
+docker --version        # Docker Engine 24+ recommended
+docker compose version  # Docker Compose v2+ required (note: no hyphen)
+```
+
+**What Docker actually does:** Docker packages an application and all its dependencies (OS libraries, runtimes, config files) into an isolated unit called a container. When you run `docker compose up`, Docker reads a YAML file that describes multiple containers, creates a private network between them, and starts everything in the right order. Each container sees only its own filesystem and network unless you explicitly share volumes or ports.
+
+**Why this matters for certctl:** Instead of installing PostgreSQL, building Go binaries, configuring the agent, and wiring everything together by hand, one command gives you the complete platform. Each compose file targets a different use case.
+
+---
+
+## How Docker Compose Works
+
+A compose file defines **services** (containers), **networks** (how they talk to each other), and **volumes** (persistent storage). The key concepts:
+
+**Services** are named containers. `certctl-server` is the API and web dashboard. `postgres` is the database. `certctl-agent` polls the server for certificate work.
+
+**Depends_on + healthchecks** control startup order. The server won't start until PostgreSQL reports healthy. The agent won't start until the server reports healthy. This prevents connection errors during boot.
+
+**Volumes** persist data across restarts. `postgres_data` keeps your database between `docker compose down` and `docker compose up`. Adding `-v` to `down` deletes volumes for a clean slate.
+
+**Overlay files** let you layer changes. Running `docker compose -f base.yml -f overlay.yml up` merges both files. The overlay can add services, change environment variables, or mount extra volumes without editing the base.
+
+**Port mapping** (`"8443:8443"`) maps host port (left) to container port (right). After startup, `https://localhost:8443` on your machine reaches the certctl server inside its container (HTTPS-only as of v2.2; the `certctl-tls-init` init container bootstraps a self-signed cert into `deploy/test/certs/`).
+
+---
+
+## Base Environment
+
+**File:** `docker-compose.yml`
+**When to use:** Production deployments, first-time setup, or any time you want a clean dashboard with the onboarding wizard.
+
+### What it runs
+
+Three services on a private bridge network:
+
+| Service | Image | Purpose | Ports |
+|---------|-------|---------|-------|
+| `postgres` | `postgres:16-alpine` | Database. Stores certificates, agents, jobs, audit trail, policies, discovery results. | 5432 |
+| `certctl-server` | Built from `Dockerfile` | API server + web dashboard + background scheduler. | 8443 |
+| `certctl-agent` | Built from `Dockerfile.agent` | Polls server for work, generates keys, deploys certificates, discovers existing certs. | none |
+
+### Starting it
+
+```bash
+git clone https://github.com/shankar0123/certctl.git
+cd certctl
+docker compose -f deploy/docker-compose.yml up -d --build
+```
+
+`--build` compiles the Go server and agent from source, including the React frontend. Without it, Docker may reuse a stale image from a previous build.
+
+`-d` runs in detached mode (background). Omit it to see logs in your terminal.
+
+Wait about 30 seconds, then verify:
+```bash
+docker compose -f deploy/docker-compose.yml ps
+# All three services should show "Up (healthy)"
+
+curl --cacert ./deploy/test/certs/ca.crt https://localhost:8443/health
+# {"status":"healthy"}
+```
+
+The control plane is HTTPS-only as of v2.2. The `certctl-tls-init` init container bootstraps a self-signed cert into `deploy/test/certs/` on first boot; pin it with `--cacert` (as above) or pass `-k` for one-off smoke tests (never in production).
+
+Open **https://localhost:8443** in your browser. You'll see the onboarding wizard guiding you through: connecting a CA, deploying an agent, and adding your first certificate. Your browser will flag the self-signed cert as untrusted — accept the warning for local evaluation, or import `deploy/test/certs/ca.crt` into your OS trust store to make the warning go away.
+
+### Service-by-service walkthrough
+
+#### PostgreSQL
+
+```yaml
+postgres:
+  image: postgres:16-alpine
+  environment:
+    POSTGRES_DB: certctl
+    POSTGRES_USER: certctl
+    POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-certctl}
+```
+
+Alpine-based PostgreSQL 16. The `${POSTGRES_PASSWORD:-certctl}` syntax means: use the `POSTGRES_PASSWORD` environment variable from your shell if set, otherwise default to `certctl`. For production, create a `.env` file:
+
+```bash
+echo 'POSTGRES_PASSWORD=your-secure-password-here' > deploy/.env
+```
+
+The `volumes` section mounts 10 migration files into PostgreSQL's init directory (`/docker-entrypoint-initdb.d/`). PostgreSQL runs these SQL files in alphabetical order on first boot only. They create the schema (tables, indexes, constraints) and seed the base data (default issuer, default policy). If the `postgres_data` volume already exists with an initialized database, these scripts are skipped entirely.
+
+**Expert note:** The numbered prefix pattern (`001_`, `002_`, ..., `020_`) ensures deterministic execution order. All migrations use `IF NOT EXISTS` and `ON CONFLICT DO NOTHING` for idempotency, so re-running them against an existing database is safe.
+
+**Stateful volume — first-boot password binding (U-1).** The same "first boot only" semantics that govern migration scripts also govern `POSTGRES_PASSWORD`. The official `postgres` image runs `initdb` exactly once — when `/var/lib/postgresql/data` is empty — and that pass is the only time `POSTGRES_PASSWORD` is written into `pg_authid`. On every subsequent boot, the postgres container ignores the env var and authenticates against whatever password was baked into the data directory on the original `up`. Editing `POSTGRES_PASSWORD` in `.env` after a successful first boot therefore only updates the **certctl-server** container's `CERTCTL_DATABASE_URL` — postgres still expects the previous password, and the server fails to ping with `pq: password authentication failed for user "certctl"` (SQLSTATE 28P01). The certctl-server container surfaces this case explicitly: when SQLSTATE 28P01 fires at startup, the wrap text in `internal/repository/postgres/db.go::wrapPingError` points operators at the two remediation paths — destructive volume teardown via `docker compose -f deploy/docker-compose.yml down -v && up -d --build`, or non-destructive in-place rotation via `docker compose -f deploy/docker-compose.yml exec postgres psql -U certctl -c "ALTER ROLE certctl PASSWORD '<new>';"` followed by a server restart with the matching `POSTGRES_PASSWORD`. Use the destructive path on the demo / first-time setup; use the non-destructive path on any environment that holds data you want to keep.
+
+#### certctl Server
+
+```yaml
+certctl-server:
+  depends_on:
+    postgres:
+      condition: service_healthy
+  environment:
+    CERTCTL_DATABASE_URL: postgres://certctl:${POSTGRES_PASSWORD:-certctl}@postgres:5432/certctl?sslmode=disable
+    CERTCTL_SERVER_HOST: 0.0.0.0
+    CERTCTL_SERVER_PORT: 8443
+    CERTCTL_LOG_LEVEL: info
+    CERTCTL_AUTH_TYPE: none
+    CERTCTL_KEYGEN_MODE: server
+    CERTCTL_NETWORK_SCAN_ENABLED: "true"
+    CERTCTL_CONFIG_ENCRYPTION_KEY: ${CERTCTL_CONFIG_ENCRYPTION_KEY:-change-me-32-char-encryption-key}
+```
+
+The server is the control plane. It serves the REST API, the React dashboard, runs 7 background scheduler loops (renewal, job processing, health checks, notifications, short-lived cert expiry, network scanning, digest emails), and manages the issuer/target registry.
+
+Key environment variables explained:
+
+- `CERTCTL_DATABASE_URL` references the `postgres` service by hostname. Docker's internal DNS resolves `postgres` to the container's IP on the bridge network. `sslmode=disable` is appropriate because traffic stays on the private Docker network.
+- `CERTCTL_AUTH_TYPE: none` disables API key authentication so you can explore immediately. For production, set `api-key` and configure `CERTCTL_AUTH_SECRET`.
+- `CERTCTL_KEYGEN_MODE: server` means the server generates private keys. This is convenient for demos but insecure for production. In production, set `agent` so keys are generated on agent machines and never transmitted.
+- `CERTCTL_CONFIG_ENCRYPTION_KEY` enables AES-256-GCM encryption for issuer and target configurations stored in the database (credentials, API keys). Without this, the dynamic configuration GUI (adding issuers/targets from the dashboard) won't encrypt sensitive fields. For production, generate a strong random key.
+- `CERTCTL_NETWORK_SCAN_ENABLED` activates the scheduler loop that probes TLS endpoints on your network to discover certificates you might not be managing.
+
+**Expert note:** The healthcheck hits `GET /health` every 10 seconds with 5 retries. The `depends_on: condition: service_healthy` on the agent means Docker holds agent startup until this check passes. Resource limits (`cpus: '1.0'`, `memory: 512M`) prevent the server from consuming unbounded resources in shared environments.
+
+#### certctl Agent
+
+```yaml
+certctl-agent:
+  depends_on:
+    certctl-server:
+      condition: service_healthy
+  environment:
+    CERTCTL_SERVER_URL: http://certctl-server:8443
+    CERTCTL_API_KEY: ${CERTCTL_API_KEY:-change-me-in-production}
+    CERTCTL_AGENT_NAME: docker-agent
+    CERTCTL_LOG_LEVEL: info
+    CERTCTL_DISCOVERY_DIRS: /var/lib/certctl/keys
+  volumes:
+    - agent_keys:/var/lib/certctl/keys
+```
+
+The agent is a lightweight Go binary that polls the server for pending work (certificate deployments, CSR generation requests), executes that work locally, and reports results back. It also scans configured directories for existing certificates (filesystem discovery).
+
+- `CERTCTL_SERVER_URL` uses the Docker internal hostname `certctl-server`. This resolves inside the Docker network only.
+- `CERTCTL_DISCOVERY_DIRS` tells the agent which directories to scan for existing certificates. The agent walks these directories recursively, parses PEM and DER files, and reports findings to the server for triage.
+- The `agent_keys` volume persists private keys generated by the agent across container restarts. Without this volume, keys would be lost when the container stops.
+
+**Expert note:** The agent's healthcheck uses `pgrep` because the agent doesn't expose an HTTP endpoint. The `restart: unless-stopped` policy means Docker automatically restarts the agent on crashes but respects manual `docker compose stop` commands.
+
+### Stopping and cleaning up
+
+```bash
+# Stop containers but keep data
+docker compose -f deploy/docker-compose.yml down
+
+# Stop and delete all data (database, keys, volumes)
+docker compose -f deploy/docker-compose.yml down -v
+```
+
+---
+
+## Demo Overlay
+
+**File:** `docker-compose.demo.yml`
+**When to use:** Demos, screenshots, stakeholder presentations, or any time you want a populated dashboard on first boot.
+
+### What it adds
+
+One line: mounts `seed_demo.sql` into PostgreSQL's init directory. This 667-line SQL file inserts 180 days of simulated operational history: teams, owners, certificates across multiple issuers, agents on different platforms, jobs with realistic timestamps, discovery scan results, audit events, policies, and profiles.
+
+### Starting it
+
+```bash
+docker compose -f deploy/docker-compose.yml -f deploy/docker-compose.demo.yml up -d --build
+```
+
+The `-f` flags are ordered: base first, overlay second. Docker merges them. The demo overlay adds the seed_demo.sql volume mount to the `postgres` service defined in the base file.
+
+### What you see
+
+The dashboard shows pre-populated charts: expiration heatmap with upcoming renewals, status distribution across Active/Expiring/Expired/Failed states, 30-day job trends, and issuance rates. The sidebar pages (Certificates, Agents, Discovery, Jobs, etc.) all have data to explore.
+
+### Resetting demo data
+
+```bash
+docker compose -f deploy/docker-compose.yml -f deploy/docker-compose.demo.yml down -v
+docker compose -f deploy/docker-compose.yml -f deploy/docker-compose.demo.yml up -d --build
+```
+
+The `down -v` deletes the `postgres_data` volume. On next boot, PostgreSQL re-runs all init scripts including the demo seed, giving you a clean starting point.
+
+**Expert note:** The demo overlay is a pure data layer, not a configuration change. The server, agent, and their environment variables remain identical to the base. This means any behavior you see in the demo is exactly what the base environment produces once you populate data through normal operations.
+
+---
+
+## Development Overlay
+
+**File:** `docker-compose.dev.yml`
+**When to use:** When you're contributing to certctl and need debug logging, database inspection, or a debugger attached to the server process.
+
+### What it adds
+
+| Addition | Purpose |
+|----------|---------|
+| Debug-level logging on server and agent | See every HTTP request, scheduler tick, and connector operation |
+| PgAdmin on port 5050 | Visual database browser for inspecting tables, running queries |
+| Delve debugger port 40000 | Attach a Go debugger to the running server process |
+
+### Starting it
+
+```bash
+docker compose -f deploy/docker-compose.yml -f deploy/docker-compose.dev.yml up --build
+```
+
+Omit `-d` during development so you see logs streaming in your terminal.
+
+### Using PgAdmin
+
+Open **http://localhost:5050** in your browser. PgAdmin is pre-configured in desktop mode (no login required). To connect to the certctl database:
+
+1. Right-click "Servers" in the left panel, choose "Register" > "Server"
+2. Name: `certctl`
+3. Connection tab: Host = `postgres`, Port = `5432`, Username = `certctl`, Password = `certctl` (or whatever you set in `.env`)
+
+From there you can browse all 19 tables, inspect certificate records, view audit events, check the scheduler's job queue, and run arbitrary SQL.
+
+### Using the Delve debugger
+
+Port 40000 is exposed for remote debugging. To use it, you'd need to modify the Dockerfile to build with debug symbols and start the server under Delve:
+
+```bash
+# In Dockerfile, replace the CMD with:
+CMD ["dlv", "--listen=:40000", "--headless=true", "--api-version=2", "exec", "/app/server"]
+```
+
+Then attach from your IDE (VS Code, GoLand) using remote debug configuration pointing to `localhost:40000`.
+
+### Hot reload
+
+The dev overlay includes commented-out volume mounts for source code directories. Uncomment them and install [air](https://github.com/cosmtrek/air) to get automatic recompilation on file changes:
+
+```bash
+go install github.com/cosmtrek/air@latest
+```
+
+**Expert note:** The `builds: context: ..` in the dev overlay overrides the base service's image reference, forcing a local build from the repository root. This means changes to your Go source code are compiled fresh on each `docker compose up --build`.
+
+---
+
+## Test Environment
+
+**File:** `docker-compose.test.yml`
+**When to use:** Integration testing against real CA backends. This is a standalone environment (not an overlay) with 7 containers on a static-IP subnet.
+
+### What it runs
+
+| Service | IP | Purpose |
+|---------|----|---------|
+| `postgres` | 10.30.50.2 | Database (clean, no demo data) |
+| `pebble-challtestsrv` | 10.30.50.3 | DNS/HTTP challenge test server for Pebble |
+| `pebble` | 10.30.50.4 | ACME test server (simulates Let's Encrypt) |
+| `step-ca` | 10.30.50.5 | Private CA (Smallstep, JWK provisioner) |
+| `certctl-server` | 10.30.50.6 | Control plane with all issuers configured |
+| `nginx` | 10.30.50.7 | TLS target server for deployment testing |
+| `certctl-agent` | 10.30.50.8 | Agent with NGINX volume + discovery |
+
+### Why static IPs?
+
+Pebble (the ACME test server) validates HTTP-01 challenges by connecting to the challenge URL. It resolves domain names via `pebble-challtestsrv`, which is configured to return `10.30.50.6` (the certctl server) for all lookups. Without static IPs, container IPs would be assigned randomly on each boot, breaking the challenge validation chain.
+
+The `/24` subnet (10.30.50.0/24) provides 254 usable addresses, far more than needed but standard practice for test networks.
+
+### Starting it
+
+```bash
+docker compose -f deploy/docker-compose.test.yml up --build
+```
+
+Wait for all health checks to pass (about 60 seconds for step-ca's first-run bootstrap). Then:
+
+```bash
+# Dashboard with auth enabled (HTTPS-only as of v2.2; browser will warn on the self-signed cert —
+# accept the warning or trust `deploy/test/certs/ca.crt` in your OS keychain)
+open https://localhost:8443
+# API key: test-key-2026
+
+# NGINX serving a self-signed placeholder
+curl -k https://localhost:8444
+```
+
+### What's different from the base
+
+The test environment is configured for production-like behavior:
+
+- **API key auth enabled** (`CERTCTL_AUTH_TYPE: api-key`, `CERTCTL_AUTH_SECRET: test-key-2026`). Every API request needs `Authorization: Bearer test-key-2026`.
+- **Agent-side key generation** (`CERTCTL_KEYGEN_MODE: agent`). The agent generates ECDSA P-256 keys locally and submits only the CSR to the server. Private keys never leave the agent container.
+- **Three real issuers configured:**
+  - **Local CA** (self-signed) for instant issuance testing
+  - **ACME via Pebble** for Let's Encrypt-compatible flow testing (HTTP-01 challenges validated through the challenge test server)
+  - **step-ca** for private CA testing with JWK provisioner authentication
+- **EST server enabled** (`CERTCTL_EST_ENABLED: "true"`) for RFC 7030 enrollment testing
+- **Post-deployment verification enabled** (`CERTCTL_VERIFY_DEPLOYMENT: "true"`) so the agent probes NGINX after deploying a cert and confirms the TLS fingerprint matches
+- **Dynamic config encryption enabled** (`CERTCTL_CONFIG_ENCRYPTION_KEY`) so issuer/target configs added through the GUI are encrypted at rest
+- **TLS trust bootstrapping:** The server runs a `setup-trust.sh` entrypoint that fetches Pebble's root CA from its management API and copies step-ca's root cert from a shared volume, then runs `update-ca-certificates` before starting the server binary. This is necessary because both CAs use self-signed roots that aren't in Alpine's default trust store.
+
+### Running the Go integration tests
+
+The test environment is designed to support the Go integration test suite at `deploy/test/integration_test.go`:
+
+```bash
+# Start the environment
+docker compose -f deploy/docker-compose.test.yml up --build -d
+
+# Wait for health checks
+sleep 30
+
+# Run integration tests (from repo root)
+go test -tags integration -v ./deploy/test/...
+```
+
+The integration tests exercise 12 phases: health, agent heartbeat, Local CA issuance, ACME issuance, renewal, step-ca issuance, revocation + CRL + OCSP, EST enrollment, S/MIME issuance, discovery, network scan, and deployment verification. PostgreSQL port 5432 is exposed so the test binary can query the database directly for assertions.
+
+See [docs/test-env.md](../docs/test-env.md) for the full walkthrough and manual QA procedures.
+
+### Stopping and cleaning up
+
+```bash
+# Stop but keep data (volumes persist)
+docker compose -f deploy/docker-compose.test.yml down
+
+# Full reset (delete step-ca bootstrap, database, agent keys, NGINX certs)
+docker compose -f deploy/docker-compose.test.yml down -v
+```
+
+**Expert note:** The step-ca container auto-bootstraps on first run: generates a root CA, creates a JWK provisioner named "admin" with password "password123", and writes everything to the `stepca_data` volume. Subsequent starts reuse this volume. If you `down -v`, the next boot generates a new root CA, which means all previously issued step-ca certs become untrusted.
+
+---
+
+## Environment Variable Reference
+
+Every `CERTCTL_*` environment variable is read by the server's `internal/config/config.go` via `os.Getenv`. If the prefix is missing, the variable is silently ignored.
+
+### Server
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `CERTCTL_DATABASE_URL` | (required) | PostgreSQL connection string |
+| `CERTCTL_SERVER_HOST` | `0.0.0.0` | Listen address |
+| `CERTCTL_SERVER_PORT` | `8443` | Listen port |
+| `CERTCTL_LOG_LEVEL` | `info` | Log verbosity: `debug`, `info`, `warn`, `error` |
+| `CERTCTL_AUTH_TYPE` | `api-key` | Auth mode: `api-key` or `none` |
+| `CERTCTL_AUTH_SECRET` | (none) | API key(s), comma-separated for rotation |
+| `CERTCTL_KEYGEN_MODE` | `agent` | Key generation: `agent` (production) or `server` (demo) |
+| `CERTCTL_CONFIG_ENCRYPTION_KEY` | (none) | AES-256-GCM key for encrypting issuer/target configs in DB |
+| `CERTCTL_NETWORK_SCAN_ENABLED` | `false` | Enable network TLS scanning scheduler loop |
+| `CERTCTL_NETWORK_SCAN_INTERVAL` | `6h` | How often the network scanner runs |
+| `CERTCTL_MAX_BODY_SIZE` | `1048576` | Max request body size in bytes (1MB) |
+| `CERTCTL_CORS_ORIGINS` | (empty) | Allowed CORS origins, comma-separated. Empty = deny all cross-origin |
+| `CERTCTL_RATE_LIMIT_RPS` | `10` | Requests per second per client |
+| `CERTCTL_RATE_LIMIT_BURST` | `20` | Burst allowance above RPS |
+
+### Agent
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `CERTCTL_SERVER_URL` | (required) | Server API URL |
+| `CERTCTL_API_KEY` | (none) | API key for authenticating with server |
+| `CERTCTL_AGENT_NAME` | (hostname) | Display name in dashboard |
+| `CERTCTL_AGENT_ID` | (auto-generated) | Stable agent identifier |
+| `CERTCTL_KEYGEN_MODE` | `agent` | Must match server setting |
+| `CERTCTL_LOG_LEVEL` | `info` | Log verbosity |
+| `CERTCTL_KEY_DIR` | `/var/lib/certctl/keys` | Directory for private key storage (0600 perms) |
+| `CERTCTL_DISCOVERY_DIRS` | (none) | Comma-separated paths to scan for existing certs |
+
+### Issuers (Server)
+
+| Variable | Description |
+|----------|-------------|
+| `CERTCTL_ACME_DIRECTORY_URL` | ACME CA directory (e.g., Let's Encrypt, Pebble) |
+| `CERTCTL_ACME_EMAIL` | ACME account email |
+| `CERTCTL_ACME_CHALLENGE_TYPE` | `http-01`, `dns-01`, or `dns-persist-01` |
+| `CERTCTL_ACME_INSECURE` | Skip TLS verification for ACME CA (test only) |
+| `CERTCTL_ACME_EAB_KID` / `CERTCTL_ACME_EAB_HMAC` | External Account Binding for ZeroSSL, Google Trust Services |
+| `CERTCTL_ACME_ARI_ENABLED` | Enable RFC 9773 Renewal Information |
+| `CERTCTL_ACME_PROFILE` | ACME profile (`tlsserver`, `shortlived`) |
+| `CERTCTL_STEPCA_URL` | step-ca server URL |
+| `CERTCTL_STEPCA_ROOT_CERT` | Path to step-ca root CA cert |
+| `CERTCTL_STEPCA_PROVISIONER` | Provisioner name |
+| `CERTCTL_STEPCA_PASSWORD` | Provisioner password |
+| `CERTCTL_STEPCA_KEY_PATH` | Path to provisioner key |
+| `CERTCTL_CA_CERT_PATH` / `CERTCTL_CA_KEY_PATH` | Sub-CA mode: load CA cert+key from disk |
+| `CERTCTL_VAULT_ADDR` | Vault server address |
+| `CERTCTL_VAULT_TOKEN` | Vault auth token |
+| `CERTCTL_VAULT_MOUNT` | PKI secrets engine mount (default: `pki`) |
+| `CERTCTL_VAULT_ROLE` | PKI role name |
+| `CERTCTL_DIGICERT_API_KEY` | DigiCert CertCentral API key |
+| `CERTCTL_DIGICERT_ORG_ID` | DigiCert organization ID |
+| `CERTCTL_SECTIGO_CUSTOMER_URI` / `_LOGIN` / `_PASSWORD` | Sectigo SCM auth |
+| `CERTCTL_GOOGLE_CAS_PROJECT` / `_LOCATION` / `_CA_POOL` / `_CREDENTIALS` | Google CAS config |
+
+### EST Server
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `CERTCTL_EST_ENABLED` | `false` | Enable RFC 7030 EST endpoints |
+| `CERTCTL_EST_ISSUER_ID` | `iss-local` | Which issuer processes EST enrollments |
+| `CERTCTL_EST_PROFILE_ID` | (none) | Optional profile constraint |
+
+### Post-Deployment Verification
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `CERTCTL_VERIFY_DEPLOYMENT` | `false` | Agent probes TLS after deploying |
+| `CERTCTL_VERIFY_TIMEOUT` | `10s` | TLS probe timeout |
+| `CERTCTL_VERIFY_DELAY` | `2s` | Wait before probing (let service reload) |
+
+### Notifications
+
+| Variable | Description |
+|----------|-------------|
+| `CERTCTL_SMTP_HOST` / `_PORT` / `_USERNAME` / `_PASSWORD` / `_FROM_ADDRESS` / `_USE_TLS` | SMTP email |
+| `CERTCTL_SLACK_WEBHOOK_URL` / `_CHANNEL` / `_USERNAME` | Slack notifications |
+| `CERTCTL_TEAMS_WEBHOOK_URL` | Microsoft Teams |
+| `CERTCTL_PAGERDUTY_ROUTING_KEY` / `_SEVERITY` | PagerDuty alerts |
+| `CERTCTL_OPSGENIE_API_KEY` / `_PRIORITY` | OpsGenie alerts |
+| `CERTCTL_DIGEST_ENABLED` / `_INTERVAL` / `_RECIPIENTS` | Scheduled digest email |
+
+---
+
+## Common Operations
+
+### Viewing logs
+
+```bash
+# All services
+docker compose -f deploy/docker-compose.yml logs -f
+
+# Single service
+docker compose -f deploy/docker-compose.yml logs -f certctl-server
+
+# Last 100 lines
+docker compose -f deploy/docker-compose.yml logs --tail 100 certctl-server
+```
+
+### Rebuilding after code changes
+
+```bash
+docker compose -f deploy/docker-compose.yml up -d --build
+```
+
+Docker only rebuilds images that have changed source files. The `--build` flag is essential after editing Go code or frontend files.
+
+### Connecting to the database directly
+
+```bash
+docker exec -it certctl-postgres psql -U certctl -d certctl
+```
+
+Useful queries:
+```sql
+-- Certificate inventory
+SELECT id, common_name, status, expires_at FROM managed_certificates ORDER BY expires_at;
+
+-- Recent jobs
+SELECT id, type, status, certificate_id, created_at FROM jobs ORDER BY created_at DESC LIMIT 20;
+
+-- Audit trail
+SELECT event_type, actor, resource_id, created_at FROM audit_events ORDER BY created_at DESC LIMIT 20;
+
+-- Issuer configurations (encrypted_config is AES-256-GCM)
+SELECT id, type, source, enabled, test_status FROM issuers;
+```
+
+### Checking container resource usage
+
+```bash
+docker stats --no-stream
+```
+
+### Upgrading
+
+```bash
+git pull
+docker compose -f deploy/docker-compose.yml up -d --build
+```
+
+Migrations are idempotent (`IF NOT EXISTS`), so upgrading to a version with new schema changes is safe. PostgreSQL only runs init scripts on first boot of a fresh volume, so new migrations in an upgrade require running them manually:
+
+```bash
+docker exec -i certctl-postgres psql -U certctl -d certctl < migrations/000011_new_feature.up.sql
+```
+
+Or, for a clean upgrade: `down -v` and `up --build` (loses existing data).

deploy/docker-compose.demo.yml

@@ -0,0 +1,26 @@
+# Demo mode: pre-populated dashboard with 32 certificates, 8 agents, 10 issuers, etc.
+# Use this to showcase certctl's dashboard with realistic data.
+#
+# Usage:
+#   docker compose -f docker-compose.yml -f docker-compose.demo.yml up --build
+#
+# To start fresh (wipe previous data):
+#   docker compose -f docker-compose.yml -f docker-compose.demo.yml down -v
+#   docker compose -f docker-compose.yml -f docker-compose.demo.yml up --build
+#
+# U-3 (P1, cat-u-seed_initdb_schema_drift): pre-U-3 this overlay mounted
+# `seed_demo.sql` into postgres `/docker-entrypoint-initdb.d/`. That worked
+# only because the production stack also mounted the migrations there, so
+# the schema existed at initdb time. Once U-3 dropped the production
+# initdb mounts (single source of truth: server runs RunMigrations + RunSeed
+# at boot), the demo seed could no longer be applied at initdb time — the
+# tables it references wouldn't exist yet.
+#
+# Post-U-3 the demo overlay just sets CERTCTL_DEMO_SEED=true; the server
+# applies seed_demo.sql at boot via postgres.RunDemoSeed AFTER baseline
+# migrations + seed.sql are in place. Same single source of truth, no
+# initdb mounts, no schema-vs-seed drift.
+services:
+  certctl-server:
+    environment:
+      CERTCTL_DEMO_SEED: "true"

deploy/docker-compose.dev.yml

@@ -9,11 +9,21 @@ services:
     build:
       context: ..
       dockerfile: Dockerfile
+      # Proxy propagation (M-4, Issue #9) — forwards host shell's proxy env
+      # vars into the Docker build so the Node frontend stage and Go module
+      # download can reach the public registries behind corporate proxies.
+      # Defaults to empty; omit the variables from the host environment for
+      # un-proxied builds and the behaviour is byte-identical to the pre-fix
+      # tree.
+      args:
+        HTTP_PROXY: ${HTTP_PROXY:-}
+        HTTPS_PROXY: ${HTTPS_PROXY:-}
+        NO_PROXY: ${NO_PROXY:-}
     environment:
       # Verbose logging for development
-      LOG_LEVEL: debug
-      SERVER_HOST: 0.0.0.0
-      SERVER_PORT: 8443
+      CERTCTL_LOG_LEVEL: debug
+      CERTCTL_SERVER_HOST: 0.0.0.0
+      CERTCTL_SERVER_PORT: "8443"
     volumes:
       # Mount local source for hot reload (requires air or similar)
       # Uncomment if using air or similar for hot reload:
@@ -29,8 +39,17 @@ services:
     build:
       context: ..
       dockerfile: Dockerfile.agent
+      # Proxy propagation (M-4, Issue #9) — forwards host shell's proxy env
+      # vars into the Docker build so the Go module download stage can reach
+      # the public Go module proxy behind corporate proxies. Defaults to
+      # empty; omit the variables from the host environment for un-proxied
+      # builds and the behaviour is byte-identical to the pre-fix tree.
+      args:
+        HTTP_PROXY: ${HTTP_PROXY:-}
+        HTTPS_PROXY: ${HTTPS_PROXY:-}
+        NO_PROXY: ${NO_PROXY:-}
     environment:
-      LOG_LEVEL: debug
+      CERTCTL_LOG_LEVEL: debug
 
   # PgAdmin for database exploration
   pgadmin:

deploy/docker-compose.test.yml

@@ -0,0 +1,429 @@
+# =============================================================================
+# certctl Testing Environment — Docker Compose
+# =============================================================================
+#
+# Spins up the full certctl platform with real CA backends for manual QA:
+#
+#   0. certctl-tls-init     — one-shot init container; writes self-signed
+#                             server.crt/.key/ca.crt into ./test/certs (bind
+#                             mount, not a named volume — host-readable for
+#                             the Go integration test binary)
+#   1. PostgreSQL 16        — database (clean, no demo data)
+#   2. certctl-server       — control plane API + web dashboard on :8443 (HTTPS)
+#   3. certctl-agent        — polls for work, deploys certs to NGINX
+#   4. step-ca              — private CA (JWK provisioner, auto-bootstraps)
+#   5. Pebble               — ACME test server (simulates Let's Encrypt)
+#   6. pebble-challtestsrv  — DNS/HTTP challenge test server for Pebble
+#   7. NGINX                — TLS target server on :8080 (HTTP) / :8444 (HTTPS)
+#
+# Usage:
+#   cd deploy
+#   docker compose -f docker-compose.test.yml up --build
+#
+# Dashboard:  https://localhost:8443   (self-signed — use --cacert test/certs/ca.crt)
+# API key:    test-key-2026
+# NGINX:      https://localhost:8444 (self-signed placeholder until cert deployed)
+#
+# Integration tests: `go test -tags integration ./deploy/test/...` picks up
+# the CA bundle at ./test/certs/ca.crt automatically via CERTCTL_TEST_CA_BUNDLE.
+#
+# See docs/test-env.md for the full walkthrough.
+# =============================================================================
+
+services:
+
+  # ---------------------------------------------------------------------------
+  # HTTPS-Everywhere Phase 6 — self-signed TLS bootstrap for the test harness.
+  # ---------------------------------------------------------------------------
+  # Mirrors the production `certctl-tls-init` (see docker-compose.yml §10-43)
+  # but writes into a *host bind mount* (./test/certs) instead of a named
+  # volume. The named-volume approach works fine inside Docker but hides the
+  # CA bundle from the Go integration test binary that runs on the host; the
+  # bind mount exposes /etc/certctl/tls/ca.crt at deploy/test/certs/ca.crt
+  # so `newTestClient()` can load it into an x509.CertPool and validate the
+  # self-signed server cert. Test-only divergence, explicitly documented.
+  #
+  # The generated cert has SAN=DNS:certctl-server,DNS:localhost,IP:127.0.0.1
+  # so both in-cluster traffic (agent → certctl-server:8443) and host traffic
+  # (go test → localhost:8443) validate cleanly. Destroy via
+  # `docker compose -f docker-compose.test.yml down -v` + `rm -rf test/certs`
+  # to force regeneration. Keys written 0600, certs 0644, owned 1000:1000
+  # (the UID the server binary runs as inside its container per Dockerfile:64).
+  certctl-tls-init:
+    image: alpine/openssl:latest
+    container_name: certctl-test-tls-init
+    restart: "no"
+    entrypoint: /bin/sh
+    command:
+      - -c
+      - |
+        set -eu
+        CERT=/etc/certctl/tls/server.crt
+        KEY=/etc/certctl/tls/server.key
+        CA=/etc/certctl/tls/ca.crt
+        if [ -f "$$CERT" ] && [ -f "$$KEY" ] && [ -f "$$CA" ]; then
+          echo "TLS cert already present at $$CERT — skipping generation"
+        else
+          mkdir -p /etc/certctl/tls
+          openssl req -x509 -newkey ec \
+            -pkeyopt ec_paramgen_curve:P-256 \
+            -nodes \
+            -keyout "$$KEY" \
+            -out "$$CERT" \
+            -days 3650 \
+            -subj "/CN=certctl-server" \
+            -addext "subjectAltName=DNS:certctl-server,DNS:localhost,IP:127.0.0.1,IP:::1"
+          cp "$$CERT" "$$CA"
+          echo "Generated self-signed TLS cert for certctl-test-server (ECDSA-P256/SHA-256, 3650d, CN=certctl-server)"
+        fi
+        # The test server container runs as root (see `user: "0:0"` below)
+        # because setup-trust.sh needs to update the system trust store, so
+        # the perms here are really about host-side readability — 0644 on
+        # the CA/cert lets `go test` on the host read the bundle without a
+        # chown dance.
+        chown 1000:1000 "$$CERT" "$$KEY" "$$CA" || true
+        chmod 0644 "$$CERT" "$$CA"
+        chmod 0600 "$$KEY"
+    volumes:
+      - ./test/certs:/etc/certctl/tls
+    networks:
+      certctl-test:
+        ipv4_address: 10.30.50.9
+
+  # ---------------------------------------------------------------------------
+  # Database
+  # ---------------------------------------------------------------------------
+  #
+  # U-3 (P1, cat-u-seed_initdb_schema_drift, GitHub #10): the test stack used
+  # to mount a hand-curated subset of migrations + seed.sql + a never-checked-in
+  # seed_test.sql into postgres `/docker-entrypoint-initdb.d/`. Same hazard as
+  # the production compose — initdb crashed any time a new migration shipped
+  # that the seed depended on without the mount list being updated. Post-U-3
+  # the schema is built EXCLUSIVELY by the server at startup via
+  # internal/repository/postgres.RunMigrations + RunSeed. Postgres comes up
+  # empty and the server lands the full ladder + baseline seed in one shot.
+  # `start_period: 30s` matches the production compose and shields slow CI
+  # runners from healthcheck flap during initdb.
+  postgres:
+    image: postgres:16-alpine
+    container_name: certctl-test-postgres
+    environment:
+      POSTGRES_DB: certctl
+      POSTGRES_USER: certctl
+      POSTGRES_PASSWORD: testpass
+    volumes:
+      - test_postgres_data:/var/lib/postgresql/data
+    networks:
+      certctl-test:
+        ipv4_address: 10.30.50.2
+    ports:
+      - "5432:5432"
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U certctl -d certctl"]
+      interval: 5s
+      timeout: 5s
+      retries: 5
+      start_period: 30s
+    restart: unless-stopped
+
+  # ---------------------------------------------------------------------------
+  # Pebble — ACME test server (simulates Let's Encrypt)
+  # ---------------------------------------------------------------------------
+  # Pebble is the official ACME test server from Let's Encrypt (RFC 8555).
+  # It validates challenges via the companion challtestsrv.
+  # Root CA cert available at https://pebble:15000/roots/0 (management API).
+  pebble-challtestsrv:
+    image: ghcr.io/letsencrypt/pebble-challtestsrv:latest
+    container_name: certctl-test-challtestsrv
+    # ENTRYPOINT is /app (the binary). command: provides only the FLAGS.
+    # Matches the official Pebble docker-compose format.
+    # -doh "" disables DoH (default :8443 would conflict with certctl server).
+    # defaultIPv4 must point to the certctl-server (10.30.50.6) because that's where
+    # the ACME HTTP-01 challenge server runs (port 80 inside the container).
+    # Pebble resolves domains via challtestsrv, then connects to this IP to validate.
+    command: -defaultIPv4 10.30.50.6 -defaultIPv6 "" -doh ""
+    networks:
+      certctl-test:
+        ipv4_address: 10.30.50.3
+    restart: unless-stopped
+
+  pebble:
+    image: ghcr.io/letsencrypt/pebble:latest
+    container_name: certctl-test-pebble
+    depends_on:
+      - pebble-challtestsrv
+    environment:
+      PEBBLE_VA_NOSLEEP: 1
+      PEBBLE_VA_ALWAYS_VALID: 0
+    # ENTRYPOINT is /app (the binary). command: provides only the FLAGS.
+    command:
+      - -config
+      - /test/config/pebble-config.json
+      - -dnsserver
+      - "10.30.50.3:8053"
+      - -strict
+    volumes:
+      - ./test/pebble-config.json:/test/config/pebble-config.json:ro
+    networks:
+      certctl-test:
+        ipv4_address: 10.30.50.4
+    restart: unless-stopped
+
+  # ---------------------------------------------------------------------------
+  # step-ca — Private CA (Smallstep)
+  # ---------------------------------------------------------------------------
+  # Auto-bootstraps on first run: generates root CA + JWK provisioner "admin".
+  # Root cert: /home/step/certs/root_ca.crt (inside stepca_data volume)
+  # Provisioner key: /home/step/secrets/provisioner_key (encrypted JWK)
+  step-ca:
+    image: smallstep/step-ca:latest
+    container_name: certctl-test-stepca
+    environment:
+      DOCKER_STEPCA_INIT_NAME: "certctl-test-ca"
+      DOCKER_STEPCA_INIT_DNS_NAMES: "step-ca,localhost"
+      DOCKER_STEPCA_INIT_PROVISIONER_NAME: "admin"
+      DOCKER_STEPCA_INIT_PASSWORD: "password123"
+      DOCKER_STEPCA_INIT_ADDRESS: ":9000"
+    volumes:
+      - stepca_data:/home/step
+    networks:
+      certctl-test:
+        ipv4_address: 10.30.50.5
+    healthcheck:
+      test: ["CMD", "curl", "-fk", "https://localhost:9000/health"]
+      interval: 10s
+      timeout: 5s
+      start_period: 15s
+      retries: 10
+    restart: unless-stopped
+
+  # ---------------------------------------------------------------------------
+  # certctl Server (Control Plane)
+  # ---------------------------------------------------------------------------
+  # Connects to PostgreSQL, Pebble (ACME), step-ca, and Local CA.
+  #
+  # TLS trust problem: Pebble and step-ca use self-signed root CAs that
+  # aren't in Alpine's trust store. The ACME and step-ca connectors use
+  # Go's default http.Client (no InsecureSkipVerify), so they need the
+  # CA certs in the system trust store.
+  #
+  # Solution: setup-trust.sh runs as root, fetches Pebble CA from its
+  # management API, copies step-ca root cert from the shared volume,
+  # runs update-ca-certificates, then execs the server binary.
+  certctl-server:
+    build:
+      context: ..
+      dockerfile: Dockerfile
+      # Proxy propagation (M-4, Issue #9) — forwards host shell's proxy env
+      # vars into the Docker build so the Node frontend stage and Go module
+      # download can reach the public registries behind corporate proxies.
+      # Defaults to empty; omit the variables from the host environment for
+      # un-proxied builds and the behaviour is byte-identical to the pre-fix
+      # tree.
+      args:
+        HTTP_PROXY: ${HTTP_PROXY:-}
+        HTTPS_PROXY: ${HTTPS_PROXY:-}
+        NO_PROXY: ${NO_PROXY:-}
+    container_name: certctl-test-server
+    depends_on:
+      postgres:
+        condition: service_healthy
+      pebble:
+        condition: service_started
+      step-ca:
+        condition: service_healthy
+      # HTTPS-Everywhere Phase 6: block server boot until the init container
+      # has written server.crt / server.key / ca.crt into ./test/certs. The
+      # init container runs once and exits 0; service_completed_successfully
+      # makes that a gating dependency rather than a liveness one.
+      certctl-tls-init:
+        condition: service_completed_successfully
+    # Run as root so update-ca-certificates can write to /etc/ssl/certs.
+    # Container isolation provides the security boundary.
+    user: "0:0"
+    entrypoint: ["/bin/sh", "/app/setup-trust.sh"]
+    environment:
+      # Database
+      CERTCTL_DATABASE_URL: postgres://certctl:testpass@postgres:5432/certctl?sslmode=disable
+
+      # Server
+      CERTCTL_SERVER_HOST: 0.0.0.0
+      CERTCTL_SERVER_PORT: 8443
+      # HTTPS-Everywhere Phase 6: point the server at the init-container-generated
+      # cert/key pair (bind-mounted from ./test/certs). Same paths as production
+      # compose so the server binary code path is identical; only the host-side
+      # storage differs (bind mount vs named volume — see §certctl-tls-init block).
+      CERTCTL_SERVER_TLS_CERT_PATH: /etc/certctl/tls/server.crt
+      CERTCTL_SERVER_TLS_KEY_PATH: /etc/certctl/tls/server.key
+      CERTCTL_LOG_LEVEL: debug
+
+      # Auth — API key required (production-like)
+      CERTCTL_AUTH_TYPE: api-key
+      CERTCTL_AUTH_SECRET: test-key-2026
+
+      # Key generation — agent-side (production-like)
+      CERTCTL_KEYGEN_MODE: agent
+
+      # Local CA issuer (iss-local) — self-signed mode (no CA cert/key paths)
+      # This is the simplest issuer, always available.
+
+      # ACME issuer (iss-acme-staging) — pointed at Pebble
+      CERTCTL_ACME_DIRECTORY_URL: https://pebble:14000/dir
+      CERTCTL_ACME_EMAIL: test@certctl.dev
+      CERTCTL_ACME_CHALLENGE_TYPE: http-01
+      CERTCTL_ACME_INSECURE: "true"
+
+      # step-ca issuer (iss-stepca)
+      CERTCTL_STEPCA_URL: https://step-ca:9000
+      CERTCTL_STEPCA_ROOT_CERT: /stepca-data/certs/root_ca.crt
+      CERTCTL_STEPCA_PROVISIONER: admin
+      CERTCTL_STEPCA_PASSWORD: password123
+      CERTCTL_STEPCA_KEY_PATH: /stepca-data/secrets/provisioner_key
+
+      # EST server (RFC 7030) — uses Local CA by default
+      CERTCTL_EST_ENABLED: "true"
+      CERTCTL_EST_ISSUER_ID: iss-local
+
+      # Dynamic issuer/target config encryption (M34/M35)
+      CERTCTL_CONFIG_ENCRYPTION_KEY: test-encryption-key-32chars!!
+
+      # Network scanning
+      CERTCTL_NETWORK_SCAN_ENABLED: "true"
+
+      # Post-deployment TLS verification
+      CERTCTL_VERIFY_DEPLOYMENT: "true"
+      CERTCTL_VERIFY_TIMEOUT: "10s"
+      CERTCTL_VERIFY_DELAY: "3s"
+    ports:
+      - "8443:8443"
+    volumes:
+      - ./test/setup-trust.sh:/app/setup-trust.sh:ro
+      # step-ca data volume (root cert at /certs/root_ca.crt, key at /secrets/provisioner_key)
+      - stepca_data:/stepca-data:ro
+      # HTTPS-Everywhere Phase 6: read-only bind mount of the init-generated
+      # TLS material. The init container writes here; server reads here; the
+      # agent mounts the same host path at the same container path (see below)
+      # so /etc/certctl/tls/ca.crt resolves to the *same* bytes on both sides.
+      - ./test/certs:/etc/certctl/tls:ro
+    networks:
+      certctl-test:
+        ipv4_address: 10.30.50.6
+    healthcheck:
+      # HTTPS-Everywhere Phase 6: healthcheck now speaks TLS with --cacert to
+      # verify the self-signed server cert against the init-generated bundle.
+      # /health requires auth when CERTCTL_AUTH_TYPE=api-key, so include the
+      # Bearer token. curl exits non-zero on both TLS handshake failure and
+      # non-2xx status — either failure keeps depends_on: {condition:
+      # service_healthy} from unblocking the agent, which is what we want.
+      test: ["CMD", "curl", "--cacert", "/etc/certctl/tls/ca.crt", "-f", "-H", "Authorization: Bearer test-key-2026", "https://localhost:8443/health"]
+      interval: 10s
+      timeout: 5s
+      start_period: 30s
+      retries: 10
+    restart: unless-stopped
+
+  # ---------------------------------------------------------------------------
+  # NGINX — TLS Target Server
+  # ---------------------------------------------------------------------------
+  # The agent deploys certificates here via the shared nginx_certs volume.
+  # nginx-entrypoint.sh generates a self-signed placeholder cert so NGINX
+  # can boot before the agent deploys a real cert.
+  #
+  # Ports: 8080 (HTTP) / 8444 (HTTPS) — offset to avoid conflict with server.
+  nginx:
+    image: nginx:alpine
+    container_name: certctl-test-nginx
+    entrypoint: ["/bin/sh", "/entrypoint.sh"]
+    volumes:
+      - ./test/nginx.conf:/etc/nginx/nginx.conf:ro
+      - ./test/nginx-entrypoint.sh:/entrypoint.sh:ro
+      - nginx_certs:/etc/nginx/certs
+    ports:
+      - "8080:80"
+      - "8444:443"
+    networks:
+      certctl-test:
+        ipv4_address: 10.30.50.7
+    healthcheck:
+      test: ["CMD-SHELL", "curl -fk https://localhost/health || exit 1"]
+      interval: 10s
+      timeout: 5s
+      start_period: 15s
+      retries: 5
+    restart: unless-stopped
+
+  # ---------------------------------------------------------------------------
+  # certctl Agent
+  # ---------------------------------------------------------------------------
+  # Polls the server for work, generates ECDSA P-256 keys locally,
+  # deploys certs to NGINX via the shared volume, and discovers existing
+  # certs in the NGINX cert directory.
+  certctl-agent:
+    build:
+      context: ..
+      dockerfile: Dockerfile.agent
+      # Proxy propagation (M-4, Issue #9) — forwards host shell's proxy env
+      # vars into the Docker build so the Go module download stage can reach
+      # the public Go module proxy behind corporate proxies. Defaults to
+      # empty; omit the variables from the host environment for un-proxied
+      # builds and the behaviour is byte-identical to the pre-fix tree.
+      args:
+        HTTP_PROXY: ${HTTP_PROXY:-}
+        HTTPS_PROXY: ${HTTPS_PROXY:-}
+        NO_PROXY: ${NO_PROXY:-}
+    container_name: certctl-test-agent
+    depends_on:
+      certctl-server:
+        condition: service_healthy
+    environment:
+      # HTTPS-Everywhere Phase 6: agent dials the server over TLS and validates
+      # the self-signed cert against the CA bundle pinned by
+      # CERTCTL_SERVER_CA_BUNDLE_PATH. Same env vars + container paths as
+      # production compose so the agent binary code path (loadCABundle →
+      # x509.CertPool → *tls.Config{RootCAs, MinVersion: TLS13}) is identical.
+      CERTCTL_SERVER_URL: https://certctl-server:8443
+      CERTCTL_SERVER_CA_BUNDLE_PATH: /etc/certctl/tls/ca.crt
+      CERTCTL_API_KEY: test-key-2026
+      CERTCTL_AGENT_NAME: test-agent-01
+      CERTCTL_AGENT_ID: agent-test-01
+      CERTCTL_KEYGEN_MODE: agent
+      CERTCTL_LOG_LEVEL: debug
+      CERTCTL_DISCOVERY_DIRS: /nginx-certs
+    volumes:
+      - agent_keys:/var/lib/certctl/keys
+      - nginx_certs:/nginx-certs
+      # HTTPS-Everywhere Phase 6: same bind mount as the server, same path,
+      # so /etc/certctl/tls/ca.crt resolves to the identical bytes. This is
+      # the only way the CN=certctl-server cert validates on the agent side.
+      - ./test/certs:/etc/certctl/tls:ro
+    networks:
+      certctl-test:
+        ipv4_address: 10.30.50.8
+    restart: unless-stopped
+
+# =============================================================================
+# Network
+# =============================================================================
+# Static IPs are required because:
+# - Pebble needs to know the challtestsrv DNS server address (10.30.50.3)
+# - challtestsrv resolves all domains to certctl-server (10.30.50.6) for HTTP-01 challenges
+# - Avoids DNS race conditions during startup
+networks:
+  certctl-test:
+    driver: bridge
+    ipam:
+      config:
+        - subnet: 10.30.50.0/24
+
+# =============================================================================
+# Volumes
+# =============================================================================
+volumes:
+  test_postgres_data:
+    driver: local
+  stepca_data:
+    driver: local
+  agent_keys:
+    driver: local
+  nginx_certs:
+    driver: local

deploy/docker-compose.yml

@@ -1,5 +1,81 @@
 services:
+  # HTTPS-Everywhere Phase 3 — self-signed TLS bootstrap (init container).
+  # Generates a CN=certctl-server ECDSA-P256 (SHA-256 signature) cert with
+  # the SAN list locked by milestone §3.6 on first boot; subsequent boots
+  # see the cert already present in the `certs` named volume and no-op out.
+  # Server + agent mount the volume read-only. Destroy via `docker compose
+  # down -v` to force regeneration. This bootstrap is for docker-compose
+  # demos and local dev only; Helm operators supply a Secret / cert-manager
+  # Certificate per docs/tls.md.
+  #
+  # Rationale for ECDSA-P256 (was ed25519 pre-v2.0.48): Apple's TLS stack
+  # — Safari Network Framework and the macOS-bundled LibreSSL 3.3.6
+  # /usr/bin/curl — does not advertise ed25519 in the ClientHello
+  # signature_algorithms extension for server certs, yielding "tls: peer
+  # doesn't support any of the certificate's signature algorithms" at
+  # handshake. ECDSA-P256 with SHA-256 is universally supported. See
+  # docs/tls.md Pattern 1.
+  certctl-tls-init:
+    image: alpine/openssl:latest
+    container_name: certctl-tls-init
+    restart: "no"
+    entrypoint: /bin/sh
+    command:
+      - -c
+      - |
+        set -eu
+        CERT=/etc/certctl/tls/server.crt
+        KEY=/etc/certctl/tls/server.key
+        CA=/etc/certctl/tls/ca.crt
+        if [ -f "$$CERT" ] && [ -f "$$KEY" ] && [ -f "$$CA" ]; then
+          echo "TLS cert already present at $$CERT — skipping generation"
+        else
+          mkdir -p /etc/certctl/tls
+          openssl req -x509 -newkey ec \
+            -pkeyopt ec_paramgen_curve:P-256 \
+            -nodes \
+            -keyout "$$KEY" \
+            -out "$$CERT" \
+            -days 3650 \
+            -subj "/CN=certctl-server" \
+            -addext "subjectAltName=DNS:certctl-server,DNS:localhost,IP:127.0.0.1,IP:::1"
+          cp "$$CERT" "$$CA"
+          echo "Generated self-signed TLS cert for certctl-server (ECDSA-P256/SHA-256, 3650d, CN=certctl-server)"
+        fi
+        # certctl binary runs as UID 1000 inside the server container per
+        # Dockerfile:64-65; the cert + key must be readable by that UID.
+        chown 1000:1000 "$$CERT" "$$KEY" "$$CA"
+        chmod 0644 "$$CERT" "$$CA"
+        chmod 0600 "$$KEY"
+    volumes:
+      - certs:/etc/certctl/tls
+    networks:
+      - certctl-network
+
   # PostgreSQL database
+  #
+  # U-3 (P1, cat-u-seed_initdb_schema_drift, GitHub #10):
+  # Pre-U-3 this stack mounted a hand-curated subset of `migrations/*.up.sql`
+  # plus `seed.sql` into `/docker-entrypoint-initdb.d/`, and postgres
+  # initdb-applied them on first boot. The mount list rotted every time a
+  # new migration shipped that the seed depended on (000013 added
+  # policy_rules.severity, 000017 renames retry_interval_minutes, etc.) —
+  # initdb crashed, the container reported `unhealthy` indefinitely, and
+  # `docker compose -f deploy/docker-compose.yml up -d --build` from a
+  # fresh clone of v2.0.50 hit it on the first try.
+  #
+  # Post-U-3 the schema is built EXCLUSIVELY by the server at startup via
+  # internal/repository/postgres.RunMigrations + RunSeed. Single source of
+  # truth, no list to keep in sync. Postgres comes up empty; the server
+  # waits for it healthy, then applies the full migration ladder + seed in
+  # one shot. Helm + the dev examples were already runtime-only (Path B)
+  # and worked through the same window.
+  #
+  # `start_period: 30s` gives postgres room to bootstrap on slow runners
+  # (CI macOS, low-spec laptops) before the healthcheck failure counter
+  # starts ticking. Pre-U-3 a slow first-init combined with the
+  # `unhealthy` flap to cascade into certctl-server's `service_healthy`
+  # depends_on, blocking the whole stack.
   postgres:
     image: postgres:16-alpine
     container_name: certctl-postgres
@@ -11,15 +87,6 @@ services:
       - "5432:5432"
     volumes:
       - postgres_data:/var/lib/postgresql/data
-      - ../migrations/000001_initial_schema.up.sql:/docker-entrypoint-initdb.d/001_schema.sql
-      - ../migrations/000002_agent_metadata.up.sql:/docker-entrypoint-initdb.d/002_agent_metadata.sql
-      - ../migrations/000003_certificate_profiles.up.sql:/docker-entrypoint-initdb.d/003_certificate_profiles.sql
-      - ../migrations/000004_agent_groups.up.sql:/docker-entrypoint-initdb.d/004_agent_groups.sql
-      - ../migrations/000005_revocation.up.sql:/docker-entrypoint-initdb.d/005_revocation.sql
-      - ../migrations/000006_discovery.up.sql:/docker-entrypoint-initdb.d/006_discovery.sql
-      - ../migrations/000007_network_discovery.up.sql:/docker-entrypoint-initdb.d/007_network_discovery.sql
-      - ../migrations/seed.sql:/docker-entrypoint-initdb.d/010_seed.sql
-      - ../migrations/seed_demo.sql:/docker-entrypoint-initdb.d/011_seed_demo.sql
     networks:
       - certctl-network
     healthcheck:
@@ -27,6 +94,7 @@ services:
       interval: 5s
       timeout: 5s
       retries: 5
+      start_period: 30s
     restart: unless-stopped
 
   # Certctl Server (API + scheduler)
@@ -34,27 +102,53 @@ services:
     build:
       context: ..
       dockerfile: Dockerfile
+      # Proxy propagation (M-4, Issue #9) — forwards host shell's proxy env
+      # vars into the Docker build so the Node frontend stage and Go module
+      # download can reach the public registries behind corporate proxies.
+      # Defaults to empty; omit the variables from the host environment for
+      # un-proxied builds and the behaviour is byte-identical to the pre-fix
+      # tree.
+      args:
+        HTTP_PROXY: ${HTTP_PROXY:-}
+        HTTPS_PROXY: ${HTTPS_PROXY:-}
+        NO_PROXY: ${NO_PROXY:-}
     container_name: certctl-server
     depends_on:
       postgres:
         condition: service_healthy
+      certctl-tls-init:
+        condition: service_completed_successfully
     environment:
-      CERTCTL_DATABASE_URL: postgres://certctl:${POSTGRES_PASSWORD:-certctl}@postgres:5432/certctl?sslmode=disable
+      # Bundle B / Audit M-018 (PCI-DSS Req 4 / CWE-319): in-cluster Postgres
+      # on the docker bridge network keeps sslmode=disable acceptable; for
+      # external/managed Postgres operators MUST override CERTCTL_DATABASE_URL
+      # with sslmode=verify-full and provide the CA bundle. See docs/database-tls.md.
+      CERTCTL_DATABASE_URL: ${CERTCTL_DATABASE_URL:-postgres://certctl:${POSTGRES_PASSWORD:-certctl}@postgres:5432/certctl?sslmode=disable}
       CERTCTL_SERVER_HOST: 0.0.0.0
       CERTCTL_SERVER_PORT: 8443
+      CERTCTL_SERVER_TLS_CERT_PATH: /etc/certctl/tls/server.crt
+      CERTCTL_SERVER_TLS_KEY_PATH: /etc/certctl/tls/server.key
       CERTCTL_LOG_LEVEL: info
       CERTCTL_AUTH_TYPE: none
       CERTCTL_KEYGEN_MODE: server  # Demo uses server-side keygen; production should use "agent"
       CERTCTL_NETWORK_SCAN_ENABLED: "true"  # Enable network scan GUI with seeded demo targets
+      CERTCTL_CONFIG_ENCRYPTION_KEY: ${CERTCTL_CONFIG_ENCRYPTION_KEY:-change-me-32-char-encryption-key}  # AES-256-GCM for dynamic issuer/target config
     ports:
       - "8443:8443"
+    volumes:
+      - certs:/etc/certctl/tls:ro
     networks:
       - certctl-network
     healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost:8443/health"]
+      test: ["CMD", "curl", "--cacert", "/etc/certctl/tls/ca.crt", "-f", "https://localhost:8443/health"]
       interval: 10s
       timeout: 5s
       retries: 5
+      # U-3: server boot now does RunMigrations + RunSeed before listening on
+      # 8443. On a fresh clone the full migration ladder + seed application
+      # can take ~10s on a small VM; start_period prevents the first few
+      # healthcheck attempts from counting as failures while that work runs.
+      start_period: 30s
     restart: unless-stopped
     logging:
       driver: "json-file"
@@ -72,17 +166,29 @@ services:
     build:
       context: ..
       dockerfile: Dockerfile.agent
+      # Proxy propagation (M-4, Issue #9) — forwards host shell's proxy env
+      # vars into the Docker build so the Go module download stage can reach
+      # the public Go module proxy behind corporate proxies. Defaults to
+      # empty; omit the variables from the host environment for un-proxied
+      # builds and the behaviour is byte-identical to the pre-fix tree.
+      args:
+        HTTP_PROXY: ${HTTP_PROXY:-}
+        HTTPS_PROXY: ${HTTPS_PROXY:-}
+        NO_PROXY: ${NO_PROXY:-}
     container_name: certctl-agent
     depends_on:
       certctl-server:
         condition: service_healthy
     environment:
-      CERTCTL_SERVER_URL: http://certctl-server:8443
+      CERTCTL_SERVER_URL: https://certctl-server:8443
+      CERTCTL_SERVER_CA_BUNDLE_PATH: /etc/certctl/tls/ca.crt
       CERTCTL_API_KEY: ${CERTCTL_API_KEY:-change-me-in-production}
       CERTCTL_AGENT_NAME: docker-agent
       CERTCTL_LOG_LEVEL: info
+      CERTCTL_DISCOVERY_DIRS: /var/lib/certctl/keys  # Agent scans this directory for existing certificates
     volumes:
       - agent_keys:/var/lib/certctl/keys
+      - certs:/etc/certctl/tls:ro
     networks:
       - certctl-network
     healthcheck:
@@ -111,3 +217,5 @@ volumes:
     driver: local
   agent_keys:
     driver: local
+  certs:
+    driver: local

deploy/helm/CHART_SUMMARY.md

@@ -0,0 +1,460 @@
+# Certctl Helm Chart - Complete Summary
+
+## Overview
+
+A production-ready Helm chart for deploying certctl (self-hosted certificate lifecycle management platform) on Kubernetes. The chart provides:
+
+- High availability support with multi-replica deployments
+- Persistent PostgreSQL database with automatic schema migration
+- DaemonSet or Deployment-based agent deployment
+- Comprehensive security contexts and RBAC
+- Multiple deployment scenarios (dev, prod, HA, external DB)
+- Full documentation and examples
+
+## Chart Metadata
+
+- **Name**: certctl
+- **Chart Version**: 0.1.0
+- **App Version**: 2.1.0
+- **Type**: application
+- **License**: BSL-1.1
+
+## File Structure
+
+```
+deploy/helm/
+├── README.md                              # Main Helm chart documentation
+├── DEPLOYMENT_GUIDE.md                    # Step-by-step deployment guide
+├── CHART_SUMMARY.md                       # This file
+│
+├── certctl/
+│   ├── Chart.yaml                         # Chart metadata
+│   ├── values.yaml                        # Default configuration values
+│   ├── .helmignore                        # Files to ignore when building chart
+│   │
+│   └── templates/
+│       ├── _helpers.tpl                   # Helm template helper functions
+│       ├── NOTES.txt                      # Post-deployment notes
+│       │
+│       ├── server-deployment.yaml         # Certctl API server deployment
+│       ├── server-service.yaml            # Server Kubernetes service
+│       ├── server-configmap.yaml          # Server configuration
+│       ├── server-secret.yaml             # Server secrets (API key, DB password, etc)
+│       │
+│       ├── postgres-statefulset.yaml      # PostgreSQL database statefulset
+│       ├── postgres-service.yaml          # PostgreSQL headless service
+│       ├── postgres-secret.yaml           # Database credentials secret
+│       │
+│       ├── agent-daemonset.yaml           # Certctl agent daemonset/deployment
+│       ├── agent-configmap.yaml           # Agent configuration
+│       │
+│       ├── ingress.yaml                   # Optional ingress resource
+│       └── serviceaccount.yaml            # ServiceAccount and RBAC
+│
+└── examples/
+    ├── values-dev.yaml                    # Development/testing configuration
+    ├── values-prod-ha.yaml                # Production HA configuration
+    ├── values-external-db.yaml            # External PostgreSQL (RDS, Cloud SQL)
+    └── values-acme-dns01.yaml             # ACME with DNS-01 (Let's Encrypt)
+```
+
+## Key Components
+
+### 1. Server Deployment
+
+**File**: `templates/server-deployment.yaml`
+
+- Manages certctl API server instances
+- Configurable replicas (default: 1)
+- Health checks (liveness & readiness probes)
+- Security context: non-root user, read-only filesystem
+- Resource limits (default: 500m CPU, 512Mi memory)
+- Automatic restart on failure
+
+**Values**:
+```yaml
+server:
+  replicas: 1
+  port: 8443
+  auth:
+    type: api-key
+    apiKey: "REQUIRED"
+  resources:
+    requests: {cpu: 100m, memory: 128Mi}
+    limits: {cpu: 500m, memory: 512Mi}
+```
+
+### 2. PostgreSQL StatefulSet
+
+**File**: `templates/postgres-statefulset.yaml`
+
+- Persistent database storage
+- Automatic schema migrations on startup
+- Single replica (can be extended with external HA tools)
+- Health checks via pg_isready
+- Configurable storage size and class
+- Security context: non-root user (UID 999)
+
+**Values**:
+```yaml
+postgresql:
+  enabled: true
+  storage:
+    size: 10Gi
+    storageClass: ""  # Use default
+  auth:
+    database: certctl
+    username: certctl
+    password: "REQUIRED"
+```
+
+### 3. Agent DaemonSet/Deployment
+
+**File**: `templates/agent-daemonset.yaml`
+
+- DaemonSet mode: one agent per Kubernetes node
+- Deployment mode: custom number of agent replicas
+- Local key storage with secure permissions (0600)
+- Health checks and automatic restart
+- Optional certificate discovery from filesystem
+
+**Values**:
+```yaml
+agent:
+  enabled: true
+  kind: DaemonSet  # or Deployment
+  replicas: 1      # for Deployment only
+  keyDir: /var/lib/certctl/keys
+  discoveryDirs: "/etc/ssl/certs"  # optional
+```
+
+### 4. Ingress (Optional)
+
+**File**: `templates/ingress.yaml`
+
+- Optional HTTPS ingress
+- cert-manager integration for automatic TLS
+- Multiple host support
+- Path-based routing
+
+**Values**:
+```yaml
+ingress:
+  enabled: false
+  className: nginx
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+  hosts:
+    - host: certctl.example.com
+      paths:
+        - path: /
+          pathType: Prefix
+```
+
+### 5. ConfigMaps and Secrets
+
+**Files**:
+- `server-configmap.yaml` - Non-secret server configuration
+- `server-secret.yaml` - API key, database URL, SMTP password
+- `postgres-secret.yaml` - Database credentials
+- `agent-configmap.yaml` - Agent configuration
+
+All secrets are base64-encoded and stored in Kubernetes Secrets.
+
+### 6. ServiceAccount and RBAC
+
+**File**: `templates/serviceaccount.yaml`
+
+- Optional ServiceAccount creation
+- Optional RBAC (ClusterRole, ClusterRoleBinding)
+- Namespace-scoped by default
+
+## Deployment Scenarios
+
+### Development Setup
+
+Use `examples/values-dev.yaml`:
+
+```bash
+helm install certctl certctl/ \
+  --values examples/values-dev.yaml \
+  --set server.auth.apiKey="dev-key" \
+  --set postgresql.auth.password="dev-password"
+```
+
+**Features**:
+- Single server replica
+- Demo auth (no API key required)
+- Small database (5Gi)
+- LoadBalancer service for easy access
+- Debug logging level
+
+### Production HA Setup
+
+Use `examples/values-prod-ha.yaml`:
+
+```bash
+helm install certctl certctl/ \
+  --values examples/values-prod-ha.yaml \
+  --set server.auth.apiKey="$(openssl rand -base64 32)" \
+  --set postgresql.auth.password="$(openssl rand -base64 32)"
+```
+
+**Features**:
+- 3 server replicas with pod anti-affinity
+- Large database storage (100Gi)
+- Pod disruption budgets
+- Prometheus monitoring enabled
+- Production resource limits
+
+### External PostgreSQL
+
+Use `examples/values-external-db.yaml`:
+
+```bash
+helm install certctl certctl/ \
+  --values examples/values-external-db.yaml \
+  --set postgresql.enabled=false \
+  --set 'server.env.CERTCTL_DATABASE_URL=postgres://...'
+```
+
+**Use cases**:
+- AWS RDS
+- Google Cloud SQL
+- Azure Database for PostgreSQL
+- External self-managed PostgreSQL
+
+### ACME with DNS-01
+
+Use `examples/values-acme-dns01.yaml`:
+
+```bash
+helm install certctl certctl/ \
+  --values examples/values-acme-dns01.yaml
+```
+
+**Enables**:
+- Automatic certificate issuance from Let's Encrypt
+- DNS-01 challenge (wildcard support)
+- Custom DNS provider scripts
+
+## Configuration Options
+
+### Server Configuration
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `server.replicas` | 1 | Number of server replicas |
+| `server.port` | 8443 | Server port |
+| `server.auth.type` | api-key | Authentication type — `api-key` or `none` (G-1: `jwt` removed; for JWT/OIDC use a fronting authenticating gateway, see `docs/architecture.md` and `docs/upgrade-to-v2-jwt-removal.md`) |
+| `server.auth.apiKey` | "" | API key (REQUIRED when `auth.type=api-key`) |
+| `server.logging.level` | info | Log level |
+| `server.logging.format` | json | Log format |
+
+### PostgreSQL Configuration
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `postgresql.enabled` | true | Enable internal PostgreSQL |
+| `postgresql.storage.size` | 10Gi | Database storage size |
+| `postgresql.storage.storageClass` | "" | Storage class name |
+| `postgresql.auth.password` | "" | Database password (REQUIRED) |
+
+### Agent Configuration
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `agent.enabled` | true | Deploy agents |
+| `agent.kind` | DaemonSet | DaemonSet or Deployment |
+| `agent.replicas` | 1 | Replicas (Deployment only) |
+| `agent.keyDir` | /var/lib/certctl/keys | Key storage directory |
+
+### Issuer Configuration
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `server.issuer.local.enabled` | true | Enable Local CA |
+| `server.issuer.acme.enabled` | false | Enable ACME |
+| `server.issuer.acme.directoryURL` | "" | ACME directory URL |
+| `server.issuer.acme.email` | "" | ACME email |
+| `server.issuer.acme.challengeType` | http-01 | Challenge type |
+
+See `values.yaml` for complete configuration options.
+
+## Helm Template Functions
+
+Defined in `templates/_helpers.tpl`:
+
+| Function | Purpose |
+|----------|---------|
+| `certctl.name` | Chart name |
+| `certctl.fullname` | Full release name |
+| `certctl.chart` | Chart name and version |
+| `certctl.labels` | Common labels |
+| `certctl.selectorLabels` | Selector labels |
+| `certctl.serverSelectorLabels` | Server selector labels |
+| `certctl.agentSelectorLabels` | Agent selector labels |
+| `certctl.postgresSelectorLabels` | PostgreSQL selector labels |
+| `certctl.serviceAccountName` | ServiceAccount name |
+| `certctl.serverImage` | Server image URI |
+| `certctl.agentImage` | Agent image URI |
+| `certctl.postgresImage` | PostgreSQL image URI |
+| `certctl.databaseURL` | Database connection string |
+| `certctl.serverURL` | Server URL for agents |
+
+## Security Features
+
+### Pod Security
+
+- Non-root users (UID 1000 for app, UID 999 for PostgreSQL)
+- Read-only root filesystems
+- No privilege escalation
+- Dropped capabilities (ALL)
+- Resource limits to prevent DoS
+
+### Secrets Management
+
+- All sensitive data in Kubernetes Secrets
+- Base64 encoded at rest
+- Can be integrated with:
+  - sealed-secrets
+  - external-secrets
+  - Vault
+  - AWS Secrets Manager
+
+### RBAC
+
+- ServiceAccount per release
+- Optional ClusterRole/ClusterRoleBinding
+- Extensible for custom permissions
+
+### Network Security
+
+- Support for Kubernetes NetworkPolicies
+- Service-to-service communication via internal DNS
+- Optional Ingress with TLS
+
+## Monitoring and Observability
+
+### Health Checks
+
+- Liveness probes (detect dead containers)
+- Readiness probes (detect not-ready services)
+- HTTP endpoints: `/health`, `/readyz`
+
+### Logging
+
+- Structured JSON logging
+- Request ID propagation
+- Configurable log levels (debug, info, warn, error)
+
+### Metrics
+
+- Prometheus metrics endpoint: `/api/v1/metrics/prometheus`
+- Optional ServiceMonitor for Prometheus Operator
+- Built-in metrics:
+  - Certificate counts by status
+  - Agent counts and status
+  - Job completion/failure rates
+  - Server uptime
+
+## Installation Quick Reference
+
+```bash
+# Development
+helm install certctl certctl/ \
+  --set server.auth.apiKey=dev \
+  --set postgresql.auth.password=dev
+
+# Production HA
+helm install certctl certctl/ \
+  --values examples/values-prod-ha.yaml \
+  --set server.auth.apiKey="$(openssl rand -base64 32)" \
+  --set postgresql.auth.password="$(openssl rand -base64 32)"
+
+# External database
+helm install certctl certctl/ \
+  --values examples/values-external-db.yaml \
+  --set postgresql.enabled=false \
+  --set 'server.env.CERTCTL_DATABASE_URL=postgres://...'
+
+# ACME with Let's Encrypt
+helm install certctl certctl/ \
+  --set server.issuer.acme.enabled=true \
+  --set server.issuer.acme.directoryURL=https://acme-v02.api.letsencrypt.org/directory
+
+# Check status
+kubectl get pods -l app.kubernetes.io/instance=certctl
+kubectl logs -l app.kubernetes.io/component=server -f
+
+# Upgrade
+helm upgrade certctl certctl/ -f new-values.yaml
+
+# Uninstall
+helm uninstall certctl
+```
+
+## Best Practices
+
+### 1. Use Secrets Management
+
+```bash
+# Use sealed-secrets
+kubectl create secret generic certctl-secrets \
+  --from-literal=api-key="$(openssl rand -base64 32)" \
+  --dry-run=client -o yaml | kubeseal -f - | kubectl apply -f -
+```
+
+### 2. Configure Resource Limits
+
+Match limits to your cluster capacity:
+
+```yaml
+server:
+  resources:
+    requests: {cpu: 250m, memory: 256Mi}
+    limits: {cpu: 1000m, memory: 512Mi}
+```
+
+### 3. Enable HA for Production
+
+```yaml
+server:
+  replicas: 3
+podAntiAffinity:
+  requiredDuringSchedulingIgnoredDuringExecution: [...]
+```
+
+### 4. Use Persistent Storage
+
+```yaml
+postgresql:
+  storage:
+    size: 100Gi
+    storageClass: fast-ssd
+```
+
+### 5. Enable Monitoring
+
+```yaml
+monitoring:
+  enabled: true
+  serviceMonitor:
+    enabled: true
+```
+
+## Documentation
+
+- **README.md** - Complete Helm chart documentation
+- **DEPLOYMENT_GUIDE.md** - Step-by-step deployment instructions
+- **values.yaml** - Commented configuration reference
+
+## Support
+
+For issues, questions, or contributions:
+- GitHub: https://github.com/shankar0123/certctl
+- Documentation: https://github.com/shankar0123/certctl/tree/main/docs
+
+## License
+
+BSL-1.1 (Business Source License)

deploy/helm/DEPLOYMENT_GUIDE.md

@@ -0,0 +1,518 @@
+# Certctl Helm Deployment Guide
+
+Complete guide for deploying certctl on Kubernetes with Helm.
+
+## Table of Contents
+
+1. [Prerequisites](#prerequisites)
+2. [Installation Methods](#installation-methods)
+3. [Production Deployment](#production-deployment)
+4. [Configuration Examples](#configuration-examples)
+5. [Post-Deployment Setup](#post-deployment-setup)
+6. [Monitoring and Logging](#monitoring-and-logging)
+7. [Maintenance](#maintenance)
+
+## Prerequisites
+
+### Required Tools
+
+```bash
+# Verify Kubernetes cluster access
+kubectl cluster-info
+kubectl get nodes
+
+# Install Helm (if not already installed)
+curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
+helm version
+
+# Verify Helm installation
+helm repo list
+```
+
+### Kubernetes Requirements
+
+- Kubernetes 1.19 or later
+- At least 2GB available memory
+- At least 10GB available storage (for PostgreSQL)
+- Network policies support (optional, for security)
+- Ingress controller (nginx, istio, etc.) - optional
+
+### Create Namespace
+
+```bash
+# Create isolated namespace
+kubectl create namespace certctl
+
+# Set as default namespace
+kubectl config set-context --current --namespace=certctl
+
+# Label for network policies (optional)
+kubectl label namespace certctl certctl-ns=true
+```
+
+## Installation Methods
+
+### Method 1: Minimal Development Setup
+
+Perfect for testing and development:
+
+```bash
+# Install with minimal configuration
+helm install certctl certctl/certctl \
+  --namespace certctl \
+  --set server.auth.apiKey="dev-key-change-in-production" \
+  --set postgresql.auth.password="dev-password-change-in-production"
+
+# Wait for deployment
+kubectl rollout status deployment/certctl-server
+kubectl rollout status statefulset/certctl-postgres
+```
+
+### Method 2: Production HA Setup
+
+For production workloads:
+
+```bash
+# Generate secure credentials
+API_KEY=$(openssl rand -base64 32)
+DB_PASSWORD=$(openssl rand -base64 32)
+
+# Install with HA configuration
+helm install certctl certctl/certctl \
+  --namespace certctl \
+  --values deploy/helm/examples/values-prod-ha.yaml \
+  --set server.auth.apiKey="$API_KEY" \
+  --set postgresql.auth.password="$DB_PASSWORD"
+```
+
+### Method 3: External PostgreSQL
+
+Using managed database service:
+
+```bash
+# Install with external database
+helm install certctl certctl/certctl \
+  --namespace certctl \
+  --values deploy/helm/examples/values-external-db.yaml \
+  --set server.auth.apiKey="$API_KEY" \
+  --set 'server.env.CERTCTL_DATABASE_URL=postgres://user:pass@db.example.com:5432/certctl?sslmode=require'
+```
+
+### Method 4: Using Custom values.yaml
+
+Recommended for GitOps workflows:
+
+```bash
+# Create values file with secrets management
+cat > /tmp/certctl-values.yaml <<EOF
+server:
+  auth:
+    apiKey: "$API_KEY"
+  logging:
+    level: info
+
+postgresql:
+  auth:
+    password: "$DB_PASSWORD"
+  storage:
+    size: 50Gi
+
+agent:
+  enabled: true
+  kind: DaemonSet
+
+ingress:
+  enabled: true
+  className: nginx
+  hosts:
+    - host: certctl.example.com
+      paths:
+        - path: /
+          pathType: Prefix
+EOF
+
+# Install using values file
+helm install certctl certctl/certctl \
+  --namespace certctl \
+  --values /tmp/certctl-values.yaml
+```
+
+## Production Deployment
+
+### Step 1: Prepare Environment
+
+```bash
+# Create namespace
+kubectl create namespace certctl
+cd deploy/helm
+
+# Generate credentials
+API_KEY=$(openssl rand -base64 32)
+DB_PASSWORD=$(openssl rand -base64 32)
+
+echo "API Key: $API_KEY"
+echo "DB Password: $DB_PASSWORD"
+
+# Save credentials in secure location (e.g., 1Password, Vault, AWS Secrets Manager)
+```
+
+### Step 2: Prepare Storage
+
+```bash
+# List available storage classes
+kubectl get storageclass
+
+# If needed, create a high-performance storage class for production
+cat <<EOF | kubectl apply -f -
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: fast-ssd
+provisioner: ebs.csi.aws.com  # For AWS, adjust for your cloud provider
+parameters:
+  type: gp3
+  iops: "3000"
+  throughput: "125"
+EOF
+```
+
+### Step 3: Set Up TLS with cert-manager
+
+```bash
+# Install cert-manager (if not already installed)
+helm repo add jetstack https://charts.jetstack.io
+helm repo update
+helm install cert-manager jetstack/cert-manager \
+  --namespace cert-manager \
+  --create-namespace \
+  --set installCRDs=true
+
+# Create ClusterIssuer for Let's Encrypt
+kubectl apply -f - <<EOF
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-prod
+spec:
+  acme:
+    server: https://acme-v02.api.letsencrypt.org/directory
+    email: admin@example.com
+    privateKeySecretRef:
+      name: letsencrypt-prod
+    solvers:
+    - http01:
+        ingress:
+          class: nginx
+EOF
+```
+
+### Step 4: Install Certctl
+
+```bash
+# Install using HA values
+helm install certctl certctl/ \
+  --namespace certctl \
+  --values examples/values-prod-ha.yaml \
+  --set server.auth.apiKey="$API_KEY" \
+  --set postgresql.auth.password="$DB_PASSWORD" \
+  --set ingress.annotations."cert-manager\.io/cluster-issuer"=letsencrypt-prod \
+  --set ingress.hosts[0].host=certctl.example.com
+
+# Verify installation
+kubectl get all -l app.kubernetes.io/instance=certctl
+```
+
+### Step 5: Verify Deployment
+
+```bash
+# Check pod status
+kubectl get pods -l app.kubernetes.io/instance=certctl
+kubectl describe pods -l app.kubernetes.io/instance=certctl
+
+# Check service status
+kubectl get svc -l app.kubernetes.io/instance=certctl
+
+# Check ingress status
+kubectl get ingress
+kubectl describe ingress certctl
+
+# Test API connectivity (HTTPS-only as of v2.2)
+POD=$(kubectl get pods -l app.kubernetes.io/component=server -o jsonpath='{.items[0].metadata.name}')
+kubectl port-forward $POD 8443:8443 &
+# If the chart provisioned a self-signed cert, fetch the CA bundle from the TLS secret first:
+#   kubectl get secret certctl-server-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/certctl-ca.crt
+curl --cacert /tmp/certctl-ca.crt -H "Authorization: Bearer $API_KEY" https://localhost:8443/health
+```
+
+### Step 6: Access the Dashboard
+
+```bash
+# Port forward to local machine
+kubectl port-forward svc/certctl-server 8443:8443 &
+
+# Or if using Ingress:
+# Open browser: https://certctl.example.com
+# Login with API key: $API_KEY
+```
+
+## Configuration Examples
+
+### Example 1: ACME (Let's Encrypt)
+
+```bash
+helm install certctl certctl/ \
+  --set server.issuer.acme.enabled=true \
+  --set server.issuer.acme.directoryURL=https://acme-v02.api.letsencrypt.org/directory \
+  --set server.issuer.acme.email=admin@example.com \
+  --set server.issuer.acme.challengeType=http-01
+```
+
+### Example 2: DNS-01 (Wildcard Certs)
+
+Requires DNS scripts ConfigMap:
+
+```bash
+# Create DNS scripts ConfigMap
+kubectl create configmap dns-scripts \
+  --from-file=dns-present.sh=./scripts/dns-present.sh \
+  --from-file=dns-cleanup.sh=./scripts/dns-cleanup.sh
+
+# Install with DNS-01
+helm install certctl certctl/ \
+  --set server.issuer.acme.enabled=true \
+  --set server.issuer.acme.challengeType=dns-01 \
+  --values examples/values-acme-dns01.yaml
+```
+
+### Example 3: AWS RDS Database
+
+```bash
+helm install certctl certctl/ \
+  --set postgresql.enabled=false \
+  --set 'server.env.CERTCTL_DATABASE_URL=postgres://user:password@mydb.c9akciq32.us-east-1.rds.amazonaws.com:5432/certctl?sslmode=require'
+```
+
+### Example 4: Multiple Issuers
+
+```bash
+helm install certctl certctl/ \
+  --set server.issuer.local.enabled=true \
+  --set server.issuer.acme.enabled=true \
+  --set server.issuer.acme.directoryURL=https://acme-v02.api.letsencrypt.org/directory
+```
+
+### Example 5: Email Notifications
+
+```bash
+helm install certctl certctl/ \
+  --set server.smtp.enabled=true \
+  --set server.smtp.host=smtp.example.com \
+  --set server.smtp.port=587 \
+  --set server.smtp.username=alerts@example.com \
+  --set server.smtp.password="$SMTP_PASSWORD" \
+  --set server.smtp.fromAddress=certctl@example.com
+```
+
+## Post-Deployment Setup
+
+### 1. Initial Database Setup
+
+```bash
+# Check database connection
+POD=$(kubectl get pods -l app.kubernetes.io/component=postgres -o jsonpath='{.items[0].metadata.name}')
+
+# Execute psql commands
+kubectl exec -it $POD -- \
+  psql -U certctl -d certctl -c '\dt'
+
+# View database status
+kubectl logs $POD | tail -20
+```
+
+### 2. Create Default Certificates
+
+```bash
+# Port forward to API
+kubectl port-forward svc/certctl-server 8443:8443 &
+
+# Create a test certificate (HTTPS-only as of v2.2 — pin the chart-provisioned CA bundle)
+# kubectl get secret certctl-server-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/certctl-ca.crt
+API_KEY="your-api-key"
+curl --cacert /tmp/certctl-ca.crt -X POST https://localhost:8443/api/v1/certificates \
+  -H "Authorization: Bearer $API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "common_name": "test.example.com",
+    "sans": ["test.example.com", "*.example.com"],
+    "owner": "admin@example.com"
+  }'
+```
+
+### 3. Configure Agents
+
+```bash
+# Get agent names
+kubectl get pods -l app.kubernetes.io/component=agent -o wide
+
+# Check agent connectivity
+POD=$(kubectl get pods -l app.kubernetes.io/component=agent -o jsonpath='{.items[0].metadata.name}')
+kubectl logs $POD | grep -i heartbeat
+```
+
+### 4. Set Up HTTPS for Web Dashboard
+
+The Ingress will handle TLS if configured properly:
+
+```bash
+# Verify ingress is ready
+kubectl get ingress
+kubectl describe ingress certctl
+
+# Test HTTPS
+curl https://certctl.example.com/health
+```
+
+## Monitoring and Logging
+
+### 1. View Logs
+
+```bash
+# Server logs
+kubectl logs -l app.kubernetes.io/component=server -f --all-containers=true
+
+# PostgreSQL logs
+kubectl logs -l app.kubernetes.io/component=postgres -f
+
+# Agent logs
+kubectl logs -l app.kubernetes.io/component=agent -f --all-containers=true
+
+# Logs from all components
+kubectl logs -l app.kubernetes.io/instance=certctl -f --all-containers=true
+```
+
+### 2. Install Prometheus Monitoring
+
+```bash
+# Install Prometheus operator (if not already installed)
+helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
+helm repo update
+
+helm install prometheus prometheus-community/kube-prometheus-stack \
+  --namespace monitoring \
+  --create-namespace
+
+# Certctl will automatically expose metrics if monitoring.enabled=true
+helm install certctl certctl/ \
+  --set monitoring.enabled=true \
+  --set monitoring.serviceMonitor.enabled=true
+```
+
+### 3. Set Up Alerts
+
+```bash
+# Create Prometheus alerts
+cat <<EOF | kubectl apply -f -
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: certctl-alerts
+spec:
+  groups:
+  - name: certctl
+    interval: 30s
+    rules:
+    - alert: CertctlServerDown
+      expr: up{job="certctl-server"} == 0
+      for: 5m
+      annotations:
+        summary: "Certctl server is down"
+
+    - alert: CertificateExpiringSoon
+      expr: certctl_certificate_expiring_soon > 0
+      for: 1h
+      annotations:
+        summary: "{{ \$value }} certificates expiring soon"
+EOF
+```
+
+## Maintenance
+
+### Scaling
+
+```bash
+# Scale server replicas
+helm upgrade certctl certctl/ \
+  --set server.replicas=5
+
+# Scale agents (Deployment kind only)
+helm upgrade certctl certctl/ \
+  --set agent.kind=Deployment \
+  --set agent.replicas=10
+```
+
+### Updating
+
+```bash
+# Update chart version
+helm repo update
+helm upgrade certctl certctl/certctl \
+  --namespace certctl \
+  -f values.yaml
+
+# Verify update
+kubectl rollout status deployment/certctl-server
+kubectl rollout status statefulset/certctl-postgres
+```
+
+### Backup and Restore
+
+```bash
+# Backup PostgreSQL data
+kubectl exec -i $(kubectl get pods -l app.kubernetes.io/component=postgres -o jsonpath='{.items[0].metadata.name}') \
+  pg_dump -U certctl certctl | gzip > certctl-backup.sql.gz
+
+# Restore from backup
+zcat certctl-backup.sql.gz | kubectl exec -i $(kubectl get pods -l app.kubernetes.io/component=postgres -o jsonpath='{.items[0].metadata.name}') \
+  psql -U certctl certctl
+
+# Backup PVC data
+kubectl get pvc
+kubectl exec -i $(kubectl get pods -l app.kubernetes.io/component=postgres -o jsonpath='{.items[0].metadata.name}') \
+  tar czf - /var/lib/postgresql/data | gzip > certctl-data-backup.tar.gz
+```
+
+### Uninstall
+
+```bash
+# Remove Helm release (keeps PVCs by default)
+helm uninstall certctl --namespace certctl
+
+# Delete PVCs if needed
+kubectl delete pvc --all -n certctl
+
+# Delete namespace
+kubectl delete namespace certctl
+```
+
+## Troubleshooting
+
+See [README.md](README.md#troubleshooting) for detailed troubleshooting steps.
+
+Common commands:
+
+```bash
+# Get all resources
+kubectl get all -n certctl
+
+# Describe pod for events
+kubectl describe pod <pod-name> -n certctl
+
+# Stream logs
+kubectl logs -f <pod-name> -n certctl
+
+# Execute commands in pod
+kubectl exec -it <pod-name> -n certctl -- /bin/sh
+
+# Check events
+kubectl get events -n certctl --sort-by='.lastTimestamp'
+```

deploy/helm/INDEX.md

@@ -0,0 +1,234 @@
+# Certctl Helm Chart - Complete File Index
+
+## Navigation Guide
+
+### Getting Started
+
+1. **Start here**: `INSTALLATION.md` - Quick installation guide with one-liners
+2. **Full reference**: `README.md` - Complete Helm chart documentation
+3. **Detailed guide**: `DEPLOYMENT_GUIDE.md` - Step-by-step deployment walkthrough
+4. **Architecture**: `CHART_SUMMARY.md` - Technical overview and design
+
+### Chart Directory Structure
+
+```
+deploy/helm/
+│
+├── README.md                           Main documentation (15 KB)
+├── DEPLOYMENT_GUIDE.md                 Step-by-step guide (12 KB)
+├── CHART_SUMMARY.md                    Architecture & design (13 KB)
+├── INSTALLATION.md                     Quick start (2.2 KB)
+├── INDEX.md                            This file
+│
+├── certctl/                            Helm chart package
+│   ├── Chart.yaml                      Chart metadata
+│   ├── values.yaml                     Default configuration (11 KB)
+│   ├── .helmignore                     Build ignore patterns
+│   │
+│   └── templates/                      15 Kubernetes resource templates
+│       ├── _helpers.tpl                Helper functions
+│       ├── NOTES.txt                   Post-install notes
+│       ├── server-deployment.yaml      API server
+│       ├── server-service.yaml         Server networking
+│       ├── server-configmap.yaml       Server configuration
+│       ├── server-secret.yaml          Server secrets
+│       ├── postgres-statefulset.yaml   Database
+│       ├── postgres-service.yaml       Database networking
+│       ├── postgres-secret.yaml        Database secrets
+│       ├── agent-daemonset.yaml        Agents (DaemonSet/Deployment)
+│       ├── agent-configmap.yaml        Agent configuration
+│       ├── ingress.yaml                Optional HTTPS ingress
+│       └── serviceaccount.yaml         RBAC resources
+│
+└── examples/                           Example configurations
+    ├── values-dev.yaml                 Development setup
+    ├── values-prod-ha.yaml             Production HA setup
+    ├── values-external-db.yaml         External PostgreSQL
+    └── values-acme-dns01.yaml          ACME DNS-01 configuration
+```
+
+## File Descriptions
+
+### Documentation Files
+
+| File | Purpose | Size |
+|------|---------|------|
+| `README.md` | Complete Helm chart documentation, configuration reference, security considerations | 15 KB |
+| `DEPLOYMENT_GUIDE.md` | Step-by-step installation instructions, production setup, troubleshooting | 12 KB |
+| `CHART_SUMMARY.md` | Technical overview, architecture, features, best practices | 13 KB |
+| `INSTALLATION.md` | Quick start guide, one-liner commands, verification steps | 2.2 KB |
+| `INDEX.md` | This file - complete file index and navigation | - |
+
+### Chart Files
+
+| File | Purpose |
+|------|---------|
+| `Chart.yaml` | Helm chart metadata (name, version, appVersion, license) |
+| `values.yaml` | Default configuration values with comprehensive comments |
+| `.helmignore` | Files to ignore when building the chart |
+
+### Template Files
+
+| File | Components Created |
+|------|-------------------|
+| `_helpers.tpl` | 14 Helm template helper functions |
+| `NOTES.txt` | Post-installation notes and instructions |
+| `server-deployment.yaml` | Certctl API server deployment (1-N replicas) |
+| `server-service.yaml` | Service exposing the server |
+| `server-configmap.yaml` | Non-secret server configuration |
+| `server-secret.yaml` | Secrets (API key, DB password, SMTP) |
+| `postgres-statefulset.yaml` | PostgreSQL database with persistent storage |
+| `postgres-service.yaml` | Headless service for PostgreSQL |
+| `postgres-secret.yaml` | Database credentials |
+| `agent-daemonset.yaml` | Certctl agents (DaemonSet or Deployment) |
+| `agent-configmap.yaml` | Agent configuration |
+| `ingress.yaml` | Optional HTTPS ingress resource |
+| `serviceaccount.yaml` | ServiceAccount and RBAC resources |
+
+### Example Configuration Files
+
+| File | Use Case | Features |
+|------|----------|----------|
+| `values-dev.yaml` | Development/testing | Single replica, debug logging, LoadBalancer, no auth |
+| `values-prod-ha.yaml` | Production HA | 3 replicas, pod anti-affinity, monitoring, large storage |
+| `values-external-db.yaml` | External PostgreSQL | AWS RDS, Cloud SQL, Azure Database, self-managed |
+| `values-acme-dns01.yaml` | Let's Encrypt | DNS-01 challenges, wildcard certs, custom DNS scripts |
+
+## Quick Links
+
+### Installation Commands
+
+#### Development
+```bash
+helm install certctl certctl/ \
+  --set server.auth.type=none \
+  --set postgresql.auth.password=dev
+```
+
+#### Production HA
+```bash
+helm install certctl certctl/ \
+  --values examples/values-prod-ha.yaml \
+  --set server.auth.apiKey="$(openssl rand -base64 32)" \
+  --set postgresql.auth.password="$(openssl rand -base64 32)"
+```
+
+#### External Database
+```bash
+helm install certctl certctl/ \
+  --values examples/values-external-db.yaml \
+  --set postgresql.enabled=false \
+  --set 'server.env.CERTCTL_DATABASE_URL=postgres://...'
+```
+
+### Verification Commands
+
+```bash
+# Check chart syntax
+helm lint certctl/
+helm template certctl certctl/
+
+# Install in cluster
+helm install certctl certctl/
+helm status certctl
+
+# Check pod status
+kubectl get pods -l app.kubernetes.io/instance=certctl
+
+# View logs
+kubectl logs -l app.kubernetes.io/component=server -f
+```
+
+## Documentation Organization
+
+### By User Role
+
+**DevOps/Platform Engineers**
+- Start: `INSTALLATION.md`
+- Deep dive: `DEPLOYMENT_GUIDE.md`
+- Configuration reference: `README.md`
+
+**Kubernetes Developers**
+- Architecture: `CHART_SUMMARY.md`
+- Configuration: `values.yaml`
+- Templates: `templates/`
+
+**Security/SREs**
+- Security section: `README.md#security-considerations`
+- RBAC: `templates/serviceaccount.yaml`
+- Network policies: `DEPLOYMENT_GUIDE.md#network-policies`
+
+**Database Administrators**
+- PostgreSQL config: `values.yaml` (postgresql section)
+- External DB setup: `examples/values-external-db.yaml`
+- Backup/restore: `DEPLOYMENT_GUIDE.md#backup-and-restore`
+
+### By Task
+
+**Getting Started**
+1. Read: `INSTALLATION.md`
+2. Install: `helm install certctl certctl/`
+3. Verify: Run commands in `INSTALLATION.md`
+
+**Production Deployment**
+1. Read: `DEPLOYMENT_GUIDE.md`
+2. Choose: `examples/values-prod-ha.yaml`
+3. Deploy: Follow step-by-step guide
+4. Reference: `README.md` for detailed options
+
+**Troubleshooting**
+- Common issues: `README.md#troubleshooting`
+- Detailed guide: `DEPLOYMENT_GUIDE.md#troubleshooting`
+- Error messages: kubectl logs and events
+
+**Configuration**
+- All options: `values.yaml`
+- Examples: `examples/values-*.yaml`
+- Detailed docs: `README.md#configuration`
+
+## Key Features
+
+### High Availability
+- Multi-replica server deployment
+- Pod anti-affinity
+- StatefulSet for database
+- Pod disruption budgets
+
+### Security
+- Non-root containers
+- Read-only filesystems
+- RBAC support
+- Kubernetes Secrets
+- Network policies
+
+### Flexibility
+- Multiple issuers (Local CA, ACME, step-ca, OpenSSL)
+- Internal or external PostgreSQL
+- DaemonSet or Deployment agents
+- Optional Ingress with TLS
+- Email notifications
+
+### Observability
+- Health checks
+- Structured logging
+- Prometheus metrics
+- ServiceMonitor support
+
+## Support
+
+- **GitHub**: https://github.com/shankar0123/certctl
+- **Issues**: Report on GitHub issues
+- **Documentation**: All docs are in `deploy/helm/`
+
+## File Statistics
+
+- **Total files**: 24
+- **Documentation**: 4 files (42 KB)
+- **Chart files**: 3 files
+- **Templates**: 13 files
+- **Examples**: 4 files
+- **Total size**: 144 KB
+
+## License
+
+All files are covered under the BSL-1.1 license.

deploy/helm/INSTALLATION.md

@@ -0,0 +1,97 @@
+# Quick Installation Guide
+
+## One-Liner Installation
+
+### Development (no auth)
+```bash
+helm install certctl certctl/ \
+  --set server.auth.type=none \
+  --set postgresql.auth.password=dev
+```
+
+### Production (with API key)
+```bash
+API_KEY=$(openssl rand -base64 32)
+DB_PASSWORD=$(openssl rand -base64 32)
+
+helm install certctl certctl/ \
+  --values examples/values-prod-ha.yaml \
+  --set server.auth.apiKey="$API_KEY" \
+  --set postgresql.auth.password="$DB_PASSWORD"
+```
+
+## Verify Installation
+
+```bash
+# Wait for pods to be ready
+kubectl rollout status deployment/certctl-server
+kubectl rollout status statefulset/certctl-postgres
+
+# Check all components
+kubectl get pods -l app.kubernetes.io/instance=certctl
+
+# View server logs
+kubectl logs -l app.kubernetes.io/component=server -f
+
+# Access the API (HTTPS-only as of v2.2; use --cacert or -k depending on your cert provisioning)
+kubectl port-forward svc/certctl-server 8443:8443 &
+# If the chart provisioned a self-signed cert, fetch the CA bundle from the secret first:
+#   kubectl get secret certctl-server-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/certctl-ca.crt
+curl --cacert /tmp/certctl-ca.crt https://localhost:8443/health
+```
+
+## Next Steps
+
+1. **Read Documentation**
+   - `README.md` - Complete reference
+   - `DEPLOYMENT_GUIDE.md` - Step-by-step guide
+   - `CHART_SUMMARY.md` - Architecture overview
+
+2. **Configure for Your Environment**
+   - Review `examples/` for your deployment scenario
+   - Customize `values.yaml` as needed
+   - Use `helm upgrade` to apply changes
+
+3. **Set Up Monitoring**
+   - Install Prometheus (optional)
+   - Enable Ingress with HTTPS
+   - Configure email notifications
+
+4. **Deploy Agents**
+   - Agents deploy automatically as DaemonSet
+   - Verify with: `kubectl get pods -l app.kubernetes.io/component=agent`
+
+5. **Create Certificates**
+   - Configure issuer connectors (Local CA, ACME, etc.)
+   - Access web dashboard at ingress or port-forward
+
+## Common Commands
+
+```bash
+# List installations
+helm list
+
+# View chart values
+helm values certctl
+
+# Upgrade chart
+helm upgrade certctl certctl/ -f new-values.yaml
+
+# Rollback to previous version
+helm rollback certctl 1
+
+# Uninstall chart
+helm uninstall certctl
+
+# View deployment history
+helm history certctl
+
+# Dry-run installation to see generated YAML
+helm install certctl certctl/ --dry-run --debug
+```
+
+## Support
+
+- Full documentation in `README.md`
+- Troubleshooting in `DEPLOYMENT_GUIDE.md`
+- Issues: https://github.com/shankar0123/certctl

deploy/helm/README.md

@@ -0,0 +1,516 @@
+# Certctl Helm Chart
+
+Production-ready Helm chart for deploying certctl (self-hosted certificate lifecycle management platform) on Kubernetes.
+
+## Table of Contents
+
+1. [Quick Start](#quick-start)
+2. [Chart Features](#chart-features)
+3. [Prerequisites](#prerequisites)
+4. [Installation](#installation)
+5. [Configuration](#configuration)
+6. [Usage Examples](#usage-examples)
+7. [Upgrading](#upgrading)
+8. [Uninstalling](#uninstalling)
+9. [Architecture](#architecture)
+10. [Security Considerations](#security-considerations)
+11. [Troubleshooting](#troubleshooting)
+
+## Quick Start
+
+```bash
+# Add the chart repository (when available)
+helm repo add certctl https://charts.example.com
+helm repo update
+
+# Install with default values
+helm install certctl certctl/certctl \
+  --set server.auth.apiKey="your-secure-api-key" \
+  --set postgresql.auth.password="your-secure-password"
+
+# Check installation status
+kubectl get pods -l app.kubernetes.io/instance=certctl
+```
+
+## Chart Features
+
+- **Server Deployment** — certctl control plane with configurable replicas
+- **PostgreSQL StatefulSet** — Persistent database with automatic schema migration
+- **Agent DaemonSet or Deployment** — Flexible agent deployment (per-node or custom replicas)
+- **Ingress Support** — Optional HTTPS ingress with cert-manager integration
+- **Security Contexts** — Non-root containers, read-only filesystems, minimal capabilities
+- **Resource Limits** — Configurable CPU and memory requests/limits
+- **Health Checks** — Liveness and readiness probes on all containers
+- **ConfigMaps and Secrets** — Centralized configuration management
+- **Service Account and RBAC** — Optional cluster role bindings
+- **Pod Disruption Budgets** — HA-ready with configurable disruption budgets
+- **Monitoring** — Optional Prometheus ServiceMonitor support
+
+## Prerequisites
+
+- Kubernetes 1.19 or later
+- Helm 3.0 or later
+- Optional: cert-manager (for automatic TLS certificate provisioning)
+- Optional: Prometheus (for metrics scraping)
+
+## Installation
+
+### 1. Using Chart from Repository
+
+```bash
+helm repo add certctl https://charts.example.com
+helm repo update
+helm install certctl certctl/certctl -f my-values.yaml
+```
+
+### 2. Using Local Chart
+
+```bash
+cd deploy/helm
+helm install certctl certctl/ \
+  --set server.auth.apiKey="$(openssl rand -base64 32)" \
+  --set postgresql.auth.password="$(openssl rand -base64 32)"
+```
+
+### 3. Minimal Production Installation
+
+```bash
+helm install certctl certctl/certctl \
+  --namespace certctl \
+  --create-namespace \
+  --set server.auth.apiKey="change-me" \
+  --set postgresql.auth.password="change-me" \
+  --set server.replicas=2 \
+  --set server.resources.requests.cpu=200m \
+  --set server.resources.requests.memory=256Mi \
+  --set ingress.enabled=true \
+  --set ingress.className=nginx \
+  --set ingress.hosts[0].host=certctl.example.com
+```
+
+## Configuration
+
+### Server Configuration
+
+```yaml
+server:
+  replicas: 1                    # Number of server replicas
+  port: 8443                     # Service port
+  auth:
+    type: api-key               # Authentication type
+    apiKey: "your-api-key"      # REQUIRED for production
+  logging:
+    level: info                 # Log level (debug, info, warn, error)
+    format: json                # Output format
+  issuer:
+    local:
+      enabled: true             # Enable local CA issuer
+    acme:
+      enabled: false            # Enable ACME issuer
+      directoryURL: ""          # ACME directory URL
+      email: ""                 # ACME registration email
+      challengeType: "http-01"  # Challenge type (http-01, dns-01, dns-persist-01)
+```
+
+### PostgreSQL Configuration
+
+```yaml
+postgresql:
+  enabled: true                 # Use managed PostgreSQL
+  auth:
+    database: certctl
+    username: certctl
+    password: "your-password"   # REQUIRED
+  storage:
+    size: 10Gi                  # PVC size
+    storageClass: ""            # Use default StorageClass
+```
+
+### Agent Configuration
+
+```yaml
+agent:
+  enabled: true                 # Deploy agents
+  kind: DaemonSet              # DaemonSet (one per node) or Deployment
+  replicas: 1                  # For Deployment kind only
+  discoveryDirs: ""            # Comma-separated cert discovery paths
+  nodeSelector: {}             # Node affinity for DaemonSet
+```
+
+### Ingress Configuration
+
+```yaml
+ingress:
+  enabled: false
+  className: nginx
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+  hosts:
+    - host: certctl.example.com
+      paths:
+        - path: /
+          pathType: Prefix
+  tls:
+    - secretName: certctl-tls
+      hosts:
+        - certctl.example.com
+```
+
+See `values.yaml` for all available configuration options.
+
+## Usage Examples
+
+### Example 1: High Availability Setup
+
+```yaml
+# ha-values.yaml
+server:
+  replicas: 3
+  resources:
+    requests:
+      cpu: 250m
+      memory: 256Mi
+    limits:
+      cpu: 1000m
+      memory: 512Mi
+
+postgresql:
+  storage:
+    size: 50Gi
+
+podAntiAffinity:
+  requiredDuringSchedulingIgnoredDuringExecution:
+    - labelSelector:
+        matchExpressions:
+          - key: app.kubernetes.io/component
+            operator: In
+            values: [server]
+      topologyKey: kubernetes.io/hostname
+```
+
+Deploy with:
+```bash
+helm install certctl certctl/certctl -f ha-values.yaml
+```
+
+### Example 2: External PostgreSQL Database
+
+```yaml
+# external-db-values.yaml
+postgresql:
+  enabled: false
+
+server:
+  env:
+    CERTCTL_DATABASE_URL: "postgres://user:password@rds.example.com:5432/certctl?sslmode=require"
+```
+
+Deploy with:
+```bash
+helm install certctl certctl/certctl -f external-db-values.yaml
+```
+
+### Example 3: ACME + Let's Encrypt
+
+```yaml
+# acme-values.yaml
+server:
+  issuer:
+    acme:
+      enabled: true
+      directoryURL: https://acme-v02.api.letsencrypt.org/directory
+      email: admin@example.com
+      challengeType: dns-01
+      dnsPresentScript: /scripts/dns-present.sh
+      dnsCleanupScript: /scripts/dns-cleanup.sh
+      dnsPropagationWait: 30s
+```
+
+### Example 4: Email Notifications via Slack + SMTP
+
+```yaml
+# notifications-values.yaml
+server:
+  smtp:
+    enabled: true
+    host: smtp.example.com
+    port: 587
+    username: certctl@example.com
+    password: "smtp-password"
+    fromAddress: certctl@example.com
+    useTLS: true
+
+  notifiers:
+    slack:
+      enabled: true
+      webhookUrl: https://hooks.slack.com/services/YOUR/WEBHOOK/URL
+      channel: "#certificates"
+```
+
+## Upgrading
+
+```bash
+# Update chart repository
+helm repo update
+
+# Upgrade release
+helm upgrade certctl certctl/certctl -f values.yaml
+
+# View upgrade history
+helm history certctl
+
+# Rollback to previous version
+helm rollback certctl 1
+```
+
+## Uninstalling
+
+```bash
+# Delete the release (keeps data by default)
+helm uninstall certctl
+
+# Also delete persistent data
+kubectl delete pvc --all -l app.kubernetes.io/instance=certctl
+
+# Delete namespace
+kubectl delete namespace certctl
+```
+
+## Architecture
+
+### Components
+
+```
+┌──────────────────────────────────────────────────────────────┐
+│ Kubernetes Cluster                                           │
+├──────────────────────────────────────────────────────────────┤
+│                                                              │
+│  ┌─────────────────┐                 ┌──────────────────┐  │
+│  │ Ingress/LB      │                 │  Agent Pod 1     │  │
+│  │ (optional)      │                 │  (DaemonSet)     │  │
+│  └────────┬────────┘                 └──────────────────┘  │
+│           │                                                  │
+│           ▼                           ┌──────────────────┐  │
+│  ┌─────────────────────────┐          │  Agent Pod 2     │  │
+│  │ Server Deployment       │          │  (DaemonSet)     │  │
+│  │ (1 to N replicas)       │          └──────────────────┘  │
+│  │ - REST API              │                                 │
+│  │ - Scheduler             │          ┌──────────────────┐  │
+│  │ - UI Dashboard          │          │  Agent Pod N     │  │
+│  └────────┬────────────────┘          │  (DaemonSet)     │  │
+│           │                           └──────────────────┘  │
+│           │                                                  │
+│           ▼                                                  │
+│  ┌──────────────────────────┐                               │
+│  │ PostgreSQL StatefulSet   │                               │
+│  │ - Database               │                               │
+│  │ - PVC (persistent)       │                               │
+│  └──────────────────────────┘                               │
+│                                                              │
+└──────────────────────────────────────────────────────────────┘
+```
+
+### Network Communication
+
+- **Server → PostgreSQL**: Internal cluster DNS (`certctl-postgres:5432`)
+- **Agent → Server**: Internal cluster DNS (`certctl-server:8443`)
+- **External → Server**: Via Ingress or Service (ClusterIP/LoadBalancer/NodePort)
+
+## Security Considerations
+
+### 1. Secrets Management
+
+All sensitive data is stored in Kubernetes Secrets:
+- PostgreSQL credentials
+- API keys
+- SMTP passwords
+- ACME account secrets
+
+**Best Practices:**
+- Use sealed-secrets or external-secrets operator
+- Enable encryption at rest in etcd
+- Rotate secrets regularly
+
+```bash
+# Example: Using sealed-secrets
+kubectl create secret generic certctl-api-key --from-literal=api-key="$(openssl rand -base64 32)" --dry-run=client -o yaml | kubeseal -f - | kubectl apply -f -
+```
+
+### 2. RBAC
+
+The chart creates minimal RBAC by default:
+- ServiceAccount per release
+- ClusterRole (empty, extensible)
+- ClusterRoleBinding
+
+**To restrict further:**
+```yaml
+rbac:
+  create: true
+  # Add specific rules here
+```
+
+### 3. Pod Security
+
+All containers run with:
+- Non-root user (UID 1000)
+- Read-only root filesystem
+- No privilege escalation
+- Dropped capabilities (ALL)
+
+### 4. Network Policies
+
+Restrict pod-to-pod communication:
+
+```yaml
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: certctl-default-deny
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/instance: certctl
+  policyTypes:
+    - Ingress
+    - Egress
+  ingress:
+    - from:
+        - namespaceSelector:
+            matchLabels:
+              name: certctl
+  egress:
+    - to:
+        - namespaceSelector:
+            matchLabels:
+              name: certctl
+    - to:
+        - podSelector: {}
+      ports:
+        - protocol: TCP
+          port: 53  # DNS
+        - protocol: UDP
+          port: 53
+```
+
+### 5. TLS/HTTPS
+
+Enable HTTPS with cert-manager:
+
+```bash
+helm install cert-manager jetstack/cert-manager \
+  --namespace cert-manager \
+  --create-namespace \
+  --set installCRDs=true
+```
+
+Then configure Ingress with TLS.
+
+### 6. API Key Security
+
+For production:
+1. Generate a strong API key: `openssl rand -base64 32`
+2. Store securely (Vault, sealed-secrets, etc.)
+3. Never commit to Git
+4. Rotate periodically
+
+```bash
+# Generate and deploy API key
+NEW_KEY=$(openssl rand -base64 32)
+kubectl patch secret certctl-server -p "{\"data\":{\"api-key\":\"$(echo -n $NEW_KEY | base64)\"}}"
+```
+
+## Troubleshooting
+
+### 1. Pods Not Starting
+
+```bash
+# Check pod status
+kubectl get pods -l app.kubernetes.io/instance=certctl
+kubectl describe pod <pod-name>
+kubectl logs <pod-name>
+```
+
+### 2. Database Connection Issues
+
+```bash
+# Verify PostgreSQL is running
+kubectl get pods -l app.kubernetes.io/component=postgres
+kubectl logs -l app.kubernetes.io/component=postgres
+
+# Test connection from server pod
+kubectl exec -it <server-pod> -- \
+  psql postgres://certctl:password@certctl-postgres:5432/certctl
+```
+
+### 3. Agent Not Connecting
+
+```bash
+# Check agent logs
+kubectl logs -l app.kubernetes.io/component=agent
+
+# Verify server is reachable
+kubectl exec -it <agent-pod> -- \
+  wget -q -O - http://certctl-server:8443/health
+```
+
+### 4. Persistent Data Loss
+
+```bash
+# Check PVC status
+kubectl get pvc
+
+# Verify data is being stored
+kubectl exec -it <postgres-pod> -- \
+  ls -lah /var/lib/postgresql/data/postgres
+```
+
+### 5. Permission Denied Errors
+
+The chart runs containers as non-root (UID 1000). If you see permission errors:
+
+```yaml
+# Temporarily allow root for debugging
+server:
+  securityContext:
+    runAsUser: 0  # NOT FOR PRODUCTION
+```
+
+### 6. Out of Memory
+
+Increase resource limits:
+
+```bash
+helm upgrade certctl certctl/certctl \
+  --set server.resources.limits.memory=1Gi \
+  --set postgresql.resources.limits.memory=2Gi
+```
+
+### 7. Certificate Validation Issues
+
+For self-signed certificates:
+
+```bash
+kubectl exec -it <pod> -- \
+  CERTCTL_TLS_INSECURE_SKIP_VERIFY=true <command>
+```
+
+### Common Issues and Solutions
+
+| Issue | Solution |
+|-------|----------|
+| `ImagePullBackOff` | Update `server.image.repository` to your registry |
+| `CrashLoopBackOff` | Check logs with `kubectl logs <pod>` |
+| `Pending` PVC | Check storage class availability |
+| Connection timeout | Verify network policies and service DNS |
+| High memory usage | Adjust `postgresql.resources.limits` and `server.resources.limits` |
+
+## Support and Contributing
+
+For issues, questions, or contributions, visit:
+- GitHub: https://github.com/shankar0123/certctl
+- Documentation: https://github.com/shankar0123/certctl/tree/main/docs
+
+## License
+
+BSL-1.1

deploy/helm/certctl/.helmignore

@@ -0,0 +1,31 @@
+# Patterns to ignore when building packages.
+# This supports shell glob patterns, relative path patterns, and negated
+# patterns.  Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.swo
+*~
+*.pyo
+*.pyc
+.pytest_cache/
+*.egg-info/
+dist/
+build/
+# IDE
+.vscode/
+.idea/
+*.sublime-project
+*.sublime-workspace
+# OS
+Thumbs.db
+# Helm
+Chart.lock

deploy/helm/certctl/Chart.yaml

@@ -0,0 +1,20 @@
+apiVersion: v2
+name: certctl
+description: Self-hosted certificate lifecycle management platform
+type: application
+version: 0.1.0
+appVersion: "2.1.0"
+keywords:
+  - certificate
+  - tls
+  - ssl
+  - pki
+  - acme
+  - lifecycle
+  - kubernetes
+maintainers:
+  - name: certctl
+home: https://github.com/shankar0123/certctl
+sources:
+  - https://github.com/shankar0123/certctl
+license: BSL-1.1

deploy/helm/certctl/README.md

@@ -0,0 +1,148 @@
+# certctl Helm Chart
+
+Production-ready Helm chart for deploying [certctl](https://github.com/shankar0123/certctl) on Kubernetes. Wires up the certctl server (Deployment), PostgreSQL (StatefulSet with PVC), and the agent (DaemonSet — one per node) on a private cluster, with health probes, security contexts, and optional Ingress.
+
+## Quick install
+
+```bash
+helm install certctl deploy/helm/certctl/ \
+  --create-namespace --namespace certctl \
+  --set server.auth.apiKey="$(openssl rand -base64 32)" \
+  --set postgresql.auth.password="$(openssl rand -base64 24)"
+```
+
+This brings up:
+
+- `<release>-server` Deployment (HTTPS-only on port 8443; TLS 1.3)
+- `<release>-postgres` StatefulSet (PostgreSQL 16-alpine, 1 replica, 10Gi PVC by default)
+- `<release>-agent` DaemonSet (polls server, generates ECDSA P-256 keys locally)
+- Service objects, optional Ingress, and ServiceAccount with RBAC
+
+See [`values.yaml`](values.yaml) for the full configuration surface — issuer settings, target connectors, scheduler intervals, notifier credentials, and resource requests/limits all live there.
+
+## Operational notes
+
+### Postgres password rotation — read this before changing `postgresql.auth.password`
+
+**The trap.** `postgresql.auth.password` is bound to `pg_authid` exactly once — when the StatefulSet's PVC is provisioned and `initdb` runs. The official `postgres:16-alpine` image only runs `initdb` when `/var/lib/postgresql/data` is empty, so on every subsequent rollout the `POSTGRES_PASSWORD` env var is read into the container but **ignored** by postgres itself. The certctl-server container also picks up the new value (via the database URL helper template), so the two halves diverge: server presents the new password, postgres still expects the old one.
+
+**Symptom.** The certctl-server pod's startup log shows:
+
+```
+failed to ping database: postgres rejected the configured credentials
+(SQLSTATE 28P01 — invalid_password). If you recently rotated POSTGRES_PASSWORD ...
+```
+
+That diagnostic is emitted by `internal/repository/postgres/db.go::wrapPingError` — it points operators at the two remediation paths below.
+
+**Remediation, non-destructive (preferred for any environment with real data):**
+
+```bash
+# 1. Rotate the password in postgres directly
+kubectl -n certctl exec -it <release>-postgres-0 -- \
+  psql -U certctl -c "ALTER ROLE certctl PASSWORD '<new-password>';"
+
+# 2. Update the secret / Helm values to the same value
+helm upgrade <release> deploy/helm/certctl/ \
+  --reuse-values \
+  --set postgresql.auth.password='<new-password>'
+
+# 3. Bounce the certctl-server pod so it re-reads the secret
+kubectl -n certctl rollout restart deployment/<release>-server
+```
+
+**Remediation, destructive (DESTROYS ALL CERTCTL DATA — only acceptable on dev/demo clusters):**
+
+```bash
+helm uninstall <release> -n certctl
+kubectl -n certctl delete pvc -l \
+  app.kubernetes.io/name=certctl,app.kubernetes.io/component=postgres
+helm install <release> deploy/helm/certctl/ \
+  --namespace certctl \
+  --set postgresql.auth.password='<new-password>'
+```
+
+The PVC re-creates empty, `initdb` runs on first boot of the new postgres pod, and `pg_authid` is seeded with the new password.
+
+**Why we don't fix this in the chart.** The env-vs-`pg_authid` divergence is intrinsic to how the upstream `postgres` image bootstraps — `initdb` is run-once-per-empty-data-dir, and there is no upstream-supported way to make subsequent boots re-seed `pg_authid` from `POSTGRES_PASSWORD`. The ergonomic answer is the runtime diagnostic plus this operational note.
+
+**Cross-references.** Same root cause is documented for the docker-compose path in [`docs/quickstart.md`](../../../docs/quickstart.md) (Warning callout after the `cp .env.example .env` block) and in [`deploy/ENVIRONMENTS.md`](../../ENVIRONMENTS.md) (Stateful volume — first-boot password binding section). The runtime diagnostic itself lives in `internal/repository/postgres/db.go::wrapPingError` with regression coverage in `internal/repository/postgres/db_test.go`.
+
+### Server API key rotation
+
+Unlike the postgres password, `server.auth.apiKey` accepts a comma-separated list, so zero-downtime rotation is straightforward:
+
+```bash
+# 1. Add the new key alongside the old
+helm upgrade <release> deploy/helm/certctl/ \
+  --reuse-values \
+  --set server.auth.apiKey='new-key,old-key'
+
+# 2. Roll your agents / clients over to the new key
+
+# 3. Remove the old key
+helm upgrade <release> deploy/helm/certctl/ \
+  --reuse-values \
+  --set server.auth.apiKey='new-key'
+```
+
+### JWT / OIDC via authenticating gateway
+
+certctl's in-process auth surface is intentionally narrow: `server.auth.type=api-key` for production deployments and `server.auth.type=none` for development. There is no in-process JWT, OIDC, mTLS, or SAML middleware. (`server.auth.type=jwt` was accepted pre-G-1 but silently routed every request through the api-key bearer middleware — silent auth downgrade. The chart now fails at `helm install`/`helm upgrade` template time via the `certctl.validateAuthType` helper if you set it. See [`../../../docs/upgrade-to-v2-jwt-removal.md`](../../../docs/upgrade-to-v2-jwt-removal.md) if you previously had this in your values.)
+
+For deployments that need JWT/OIDC, the canonical Kubernetes-flavored shape is to put oauth2-proxy in front of the certctl Service, attach an authenticating Ingress middleware, and run certctl with `server.auth.type=none`:
+
+```bash
+# 1. Install oauth2-proxy (or any OIDC-terminating sidecar) in the same namespace
+helm install oauth2-proxy oauth2-proxy/oauth2-proxy \
+  --namespace certctl \
+  --set config.clientID="$OIDC_CLIENT_ID" \
+  --set config.clientSecret="$OIDC_CLIENT_SECRET" \
+  --set config.cookieSecret="$(openssl rand -base64 32)" \
+  --set config.configFile='|
+    provider = "oidc"
+    oidc_issuer_url = "https://your-issuer/"
+    upstreams = ["http://<release>-server.certctl.svc.cluster.local:8443"]
+    pass_authorization_header = true
+    set_authorization_header = true
+    email_domains = ["*"]
+  '
+
+# 2. Install certctl with type=none (gateway terminates auth)
+helm install certctl deploy/helm/certctl/ \
+  --namespace certctl \
+  --set server.auth.type=none \
+  --set postgresql.auth.password="$(openssl rand -base64 24)"
+
+# 3. Attach an Ingress that routes through oauth2-proxy
+#    (Traefik ForwardAuth, nginx auth_request, Envoy ext_authz, etc.)
+```
+
+Same root pattern works with Pomerium, Authelia, Caddy `forward_auth`, Apache `mod_auth_openidc`, or any service-mesh `ext_authz`. See [`../../../docs/architecture.md`](../../../docs/architecture.md) "Authenticating-gateway pattern" for the full design rationale and [`../../../docs/upgrade-to-v2-jwt-removal.md`](../../../docs/upgrade-to-v2-jwt-removal.md) for the migration walkthrough.
+
+### TLS certificate sourcing
+
+By default the chart provisions a self-signed cert via the same init-container pattern as the docker-compose deploy. For production, supply an operator-managed Secret (cert-manager, internal CA, etc.) — see [`docs/tls.md`](../../../docs/tls.md) for the full provisioning matrix and [`docs/upgrade-to-tls.md`](../../../docs/upgrade-to-tls.md) for upgrade-from-HTTP procedures.
+
+## Disabling embedded postgres
+
+If you have an existing PostgreSQL cluster, disable the embedded one and point at it directly:
+
+```bash
+helm install certctl deploy/helm/certctl/ \
+  --set postgresql.enabled=false \
+  --set server.databaseUrl='postgres://certctl:<pw>@my-pg-host:5432/certctl?sslmode=require'
+```
+
+The volume-trap section above does **not** apply to this configuration — your postgres operator (or cloud DB) handles password rotation, and you control `pg_authid` directly.
+
+## Uninstall
+
+```bash
+helm uninstall <release> -n certctl
+# Optional — also delete the postgres PVC (DESTROYS DATA):
+kubectl -n certctl delete pvc -l \
+  app.kubernetes.io/name=certctl,app.kubernetes.io/component=postgres
+```
+
+By default `helm uninstall` retains the StatefulSet's PVCs, so reinstalling with the same release name preserves the database. If you've changed `postgresql.auth.password` in your values between uninstall and reinstall, you'll hit the trap on the reinstall — apply the non-destructive remediation above, or also delete the PVC.

deploy/helm/certctl/templates/NOTES.txt

@@ -0,0 +1,74 @@
+1. Get the certctl Server URL by running:
+{{- if .Values.ingress.enabled }}
+  https://{{ index .Values.ingress.hosts 0 "host" }}
+{{- else if contains "NodePort" .Values.server.service.type }}
+  export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
+  export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "certctl.fullname" . }}-server)
+  echo https://$NODE_IP:$NODE_PORT
+{{- else if contains "LoadBalancer" .Values.server.service.type }}
+  export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "certctl.fullname" . }}-server --template "{.status.loadBalancer.ingress[0].ip}")
+  echo https://$SERVICE_IP:{{ .Values.server.service.port }}
+{{- else }}
+  export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "certctl.name" . }},app.kubernetes.io/instance={{ .Release.Name }},app.kubernetes.io/component=server" -o jsonpath="{.items[0].metadata.name}")
+  export CONTAINER_PORT=$(kubectl get pod --namespace {{ .Release.Namespace }} $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
+  echo "Visit https://127.0.0.1:8443 to use your application"
+  kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8443:$CONTAINER_PORT
+{{- end }}
+
+2. Talk to the HTTPS-only server from your workstation:
+  # Export the CA bundle that signed the server cert (self-signed or cert-manager-issued)
+  kubectl get secret --namespace {{ .Release.Namespace }} {{ include "certctl.tls.secretName" . }} \
+    -o jsonpath='{.data.ca\.crt}' | base64 --decode > /tmp/certctl-ca.crt
+  # (If ca.crt is empty, fall back to tls.crt — typical when the Secret
+  #  was created from a self-signed bootstrap cert without a separate CA.)
+
+  # Adapt the URL below to match the Server URL printed in step 1.
+  curl --cacert /tmp/certctl-ca.crt https://127.0.0.1:8443/health
+
+3. Get the default API key:
+  kubectl get secret --namespace {{ .Release.Namespace }} {{ include "certctl.fullname" . }}-server -o jsonpath="{.data.api-key}" | base64 --decode; echo
+
+4. Get PostgreSQL connection details:
+  Host: {{ include "certctl.fullname" . }}-postgres.{{ .Release.Namespace }}.svc.cluster.local
+  Port: 5432
+  Database: {{ .Values.postgresql.auth.database }}
+  Username: {{ .Values.postgresql.auth.username }}
+  Password: $(kubectl get secret --namespace {{ .Release.Namespace }} {{ include "certctl.fullname" . }}-postgres -o jsonpath="{.data.password}" | base64 --decode)
+
+5. Check deployment status:
+  kubectl get pods -n {{ .Release.Namespace }} -l app.kubernetes.io/instance={{ .Release.Name }}
+
+6. View server logs:
+  kubectl logs -n {{ .Release.Namespace }} -l app.kubernetes.io/name={{ include "certctl.name" . }},app.kubernetes.io/component=server -f
+
+{{- if .Values.agent.enabled }}
+
+7. View agent logs:
+  kubectl logs -n {{ .Release.Namespace }} -l app.kubernetes.io/name={{ include "certctl.name" . }},app.kubernetes.io/component=agent -f
+
+{{- end }}
+
+IMPORTANT NOTES FOR PRODUCTION:
+
+1. Update the API key for security:
+   kubectl patch secret {{ include "certctl.fullname" . }}-server -n {{ .Release.Namespace }} \
+     -p '{"data":{"api-key":"'$(echo -n "YOUR_NEW_API_KEY" | base64)'"}}'
+
+2. Update PostgreSQL password:
+   kubectl patch secret {{ include "certctl.fullname" . }}-postgres -n {{ .Release.Namespace }} \
+     -p '{"data":{"password":"'$(echo -n "YOUR_NEW_PASSWORD" | base64)'"}}'
+
+3. Configure certificate issuers (ACME, step-ca, etc.) via values.yaml:
+   helm upgrade {{ .Release.Name }} certctl/certctl \
+     --set server.issuer.acme.enabled=true \
+     --set server.issuer.acme.directoryURL=https://acme-v02.api.letsencrypt.org/directory \
+     --set server.issuer.acme.email=admin@example.com
+
+4. For production with persistent databases and backups:
+   - Use an external PostgreSQL managed service (AWS RDS, Cloud SQL, etc.)
+   - Set postgresql.enabled=false and configure CERTCTL_DATABASE_URL in values
+
+5. Review security contexts and network policies:
+   - All containers run as non-root
+   - Implement network policies to restrict traffic between components
+   - Consider pod security policies or security standards for your cluster

deploy/helm/certctl/templates/_helpers.tpl

@@ -0,0 +1,209 @@
+{{/*
+Expand the name of the chart.
+*/}}
+{{- define "certctl.name" -}}
+{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{/*
+Create a default fully qualified app name.
+*/}}
+{{- define "certctl.fullname" -}}
+{{- if .Values.fullnameOverride }}
+{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
+{{- else }}
+{{- $name := default .Chart.Name .Values.nameOverride }}
+{{- if contains $name .Release.Name }}
+{{- .Release.Name | trunc 63 | trimSuffix "-" }}
+{{- else }}
+{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
+{{- end }}
+{{- end }}
+{{- end }}
+
+{{/*
+Create chart name and version as used by the chart label.
+*/}}
+{{- define "certctl.chart" -}}
+{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{/*
+Common labels
+*/}}
+{{- define "certctl.labels" -}}
+helm.sh/chart: {{ include "certctl.chart" . }}
+{{ include "certctl.selectorLabels" . }}
+{{- if .Chart.AppVersion }}
+app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
+{{- end }}
+app.kubernetes.io/managed-by: {{ .Release.Service }}
+{{- with .Values.commonLabels }}
+{{ toYaml . }}
+{{- end }}
+{{- end }}
+
+{{/*
+Selector labels for the main service (server, agent, postgres)
+*/}}
+{{- define "certctl.selectorLabels" -}}
+app.kubernetes.io/name: {{ include "certctl.name" . }}
+app.kubernetes.io/instance: {{ .Release.Name }}
+{{- end }}
+
+{{/*
+Server selector labels
+*/}}
+{{- define "certctl.serverSelectorLabels" -}}
+{{ include "certctl.selectorLabels" . }}
+app.kubernetes.io/component: server
+{{- end }}
+
+{{/*
+Agent selector labels
+*/}}
+{{- define "certctl.agentSelectorLabels" -}}
+{{ include "certctl.selectorLabels" . }}
+app.kubernetes.io/component: agent
+{{- end }}
+
+{{/*
+PostgreSQL selector labels
+*/}}
+{{- define "certctl.postgresSelectorLabels" -}}
+{{ include "certctl.selectorLabels" . }}
+app.kubernetes.io/component: postgres
+{{- end }}
+
+{{/*
+Service account name
+*/}}
+{{- define "certctl.serviceAccountName" -}}
+{{- if .Values.serviceAccount.create }}
+{{- default (include "certctl.fullname" .) .Values.serviceAccount.name }}
+{{- else }}
+{{- default "default" .Values.serviceAccount.name }}
+{{- end }}
+{{- end }}
+
+{{/*
+Server image
+*/}}
+{{- define "certctl.serverImage" -}}
+{{- $image := .Values.server.image }}
+{{- printf "%s:%s" $image.repository (coalesce $image.tag .Chart.AppVersion) }}
+{{- end }}
+
+{{/*
+Agent image
+*/}}
+{{- define "certctl.agentImage" -}}
+{{- $image := .Values.agent.image }}
+{{- printf "%s:%s" $image.repository (coalesce $image.tag .Chart.AppVersion) }}
+{{- end }}
+
+{{/*
+PostgreSQL image
+*/}}
+{{- define "certctl.postgresImage" -}}
+{{- $image := .Values.postgresql.image }}
+{{- printf "%s:%s" $image.repository $image.tag }}
+{{- end }}
+
+{{/*
+Database connection string
+
+Bundle B / Audit M-018 (PCI-DSS Req 4 / CWE-319):
+  - postgresql.tls.mode is the operator-facing knob.
+    Default: "disable" (preserves the in-cluster Helm-bundled-Postgres
+    behavior; pod-to-pod traffic stays on the K8s pod network and is
+    encrypted by the CNI when the cluster is configured with a TLS-aware
+    CNI such as Cilium WireGuard).
+  - Operators on PCI-DSS-scoped clusters or operators using an external
+    managed Postgres (RDS, Cloud SQL, Azure DB) MUST set
+    postgresql.tls.mode to "require", "verify-ca", or "verify-full" and
+    point postgresql.tls.caSecretRef at a Secret containing the
+    server-ca.crt under key "ca.crt".
+  - The connection string sslmode parameter is wired from
+    postgresql.tls.mode without further translation.
+*/}}
+{{- define "certctl.databaseURL" -}}
+{{- $sslMode := default "disable" .Values.postgresql.tls.mode -}}
+postgres://{{ .Values.postgresql.auth.username }}:$(POSTGRES_PASSWORD)@{{ include "certctl.fullname" . }}-postgres:5432/{{ .Values.postgresql.auth.database }}?sslmode={{ $sslMode }}
+{{- end }}
+
+{{/*
+Server URL (for agents). HTTPS-only as of v2.2 — see docs/tls.md.
+*/}}
+{{- define "certctl.serverURL" -}}
+https://{{ include "certctl.fullname" . }}-server:{{ .Values.server.service.port }}
+{{- end }}
+
+{{/*
+TLS Secret name resolver.
+
+Operator-facing precedence:
+  1. server.tls.existingSecret        — operator points at a pre-existing kubernetes.io/tls Secret
+  2. server.tls.certManager.secretName — explicit secret name for the cert-manager Certificate CR
+  3. "<fullname>-tls"                  — default when cert-manager is enabled but secretName is blank
+
+Never emits an empty string — that case is already excluded by certctl.tls.required below,
+which must be invoked by any template that depends on the resolved secret name.
+*/}}
+{{- define "certctl.tls.secretName" -}}
+{{- if .Values.server.tls.existingSecret -}}
+{{- .Values.server.tls.existingSecret -}}
+{{- else if .Values.server.tls.certManager.secretName -}}
+{{- .Values.server.tls.certManager.secretName -}}
+{{- else -}}
+{{- printf "%s-tls" (include "certctl.fullname" .) -}}
+{{- end -}}
+{{- end }}
+
+{{/*
+TLS configuration gate.
+
+HTTPS is the only supported listener mode (v2.2+). The server refuses to start
+without a cert/key pair mounted at server.tls.mountPath, so `helm template` /
+`helm install` must fail loudly at render-time rather than shipping a broken
+Deployment that crash-loops with "tls config required".
+
+Operators MUST configure EXACTLY ONE of:
+  (a) server.tls.existingSecret: <name-of-kubernetes.io/tls-secret>
+  (b) server.tls.certManager.enabled: true  (+ issuerRef.name populated)
+
+Any template that mounts the TLS Secret must call
+`{{ include "certctl.tls.required" . }}` at the top so this guard runs once
+per affected resource. No-op when configured correctly.
+*/}}
+{{- define "certctl.tls.required" -}}
+{{- if and (not .Values.server.tls.existingSecret) (not .Values.server.tls.certManager.enabled) -}}
+{{- fail "\n\ncertctl refuses to start without TLS.\n\nSet EXACTLY ONE of:\n  --set server.tls.existingSecret=<your-kubernetes.io/tls-secret-name>\nOR\n  --set server.tls.certManager.enabled=true \\\n  --set server.tls.certManager.issuerRef.name=<your-issuer-or-clusterissuer>\n\nSee docs/tls.md for the full setup walkthrough, including bootstrap\nguidance for air-gapped clusters without cert-manager.\n" -}}
+{{- end -}}
+{{- if and .Values.server.tls.certManager.enabled (not .Values.server.tls.certManager.issuerRef.name) -}}
+{{- fail "\n\nserver.tls.certManager.enabled=true but server.tls.certManager.issuerRef.name is empty.\n\nSet:\n  --set server.tls.certManager.issuerRef.name=<your-issuer-or-clusterissuer>\n\nSee docs/tls.md.\n" -}}
+{{- end -}}
+{{- end }}
+
+{{/*
+Auth-type validation gate.
+
+G-1 (P1): pre-G-1 the chart accepted server.auth.type=jwt and the
+certctl-server container silently routed every request through the
+api-key bearer middleware (no JWT impl ships with certctl). Post-G-1
+the chart fails at template-time with a pointer at the authenticating-
+gateway pattern. The valid set must stay in sync with
+internal/config.ValidAuthTypes() in the Go binary; if you add a value
+there you must add it here too (and update the property test in
+internal/config/config_test.go that pins both surfaces).
+
+Any template that consumes .Values.server.auth.type should call
+`{{ include "certctl.validateAuthType" . }}` at the top so this guard
+runs once per affected resource. No-op when configured correctly.
+*/}}
+{{- define "certctl.validateAuthType" -}}
+{{- $valid := list "api-key" "none" -}}
+{{- if not (has .Values.server.auth.type $valid) -}}
+{{- fail (printf "\n\nserver.auth.type=%q is not supported (valid: %v).\n\nFor JWT/OIDC, run an authenticating gateway in front of certctl\n(oauth2-proxy / Envoy ext_authz / Traefik ForwardAuth / Pomerium) and\nset server.auth.type=none here so the gateway terminates federated\nidentity. See docs/architecture.md \"Authenticating-gateway pattern\"\nand docs/upgrade-to-v2-jwt-removal.md for the migration walkthrough.\n\nG-1 audit closure: pre-G-1 the chart accepted type=jwt and the binary\nsilently downgraded to api-key middleware. The chart now fails at\ntemplate time so misconfigured deployments cannot ship.\n" .Values.server.auth.type $valid) -}}
+{{- end -}}
+{{- end }}

deploy/helm/certctl/templates/agent-configmap.yaml

@@ -0,0 +1,13 @@
+{{- if .Values.agent.enabled }}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: {{ include "certctl.fullname" . }}-agent
+  labels:
+    {{- include "certctl.labels" . | nindent 4 }}
+    app.kubernetes.io/component: agent
+data:
+  {{- if .Values.agent.discoveryDirs }}
+  discovery-dirs: {{ .Values.agent.discoveryDirs | quote }}
+  {{- end }}
+{{- end }}

deploy/helm/certctl/templates/agent-daemonset.yaml

@@ -0,0 +1,181 @@
+{{- if .Values.agent.enabled }}
+{{- include "certctl.tls.required" . }}
+{{- if eq .Values.agent.kind "DaemonSet" }}
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: {{ include "certctl.fullname" . }}-agent
+  labels:
+    {{- include "certctl.labels" . | nindent 4 }}
+    app.kubernetes.io/component: agent
+spec:
+  selector:
+    matchLabels:
+      {{- include "certctl.agentSelectorLabels" . | nindent 6 }}
+  template:
+    metadata:
+      labels:
+        {{- include "certctl.agentSelectorLabels" . | nindent 8 }}
+    spec:
+      serviceAccountName: {{ include "certctl.serviceAccountName" . }}
+      securityContext:
+        {{- toYaml .Values.agent.securityContext | nindent 8 }}
+      {{- with .Values.imagePullSecrets }}
+      imagePullSecrets:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.agent.nodeSelector }}
+      nodeSelector:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.agent.tolerations }}
+      tolerations:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.agent.affinity }}
+      affinity:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      containers:
+        - name: agent
+          image: {{ include "certctl.agentImage" . }}
+          imagePullPolicy: {{ .Values.agent.image.pullPolicy }}
+          env:
+            - name: CERTCTL_SERVER_URL
+              value: {{ include "certctl.serverURL" . }}
+            - name: CERTCTL_API_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: {{ include "certctl.fullname" . }}-server
+                  key: api-key
+            - name: CERTCTL_AGENT_NAME
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.name
+            - name: CERTCTL_KEY_DIR
+              value: {{ .Values.agent.keyDir }}
+            - name: CERTCTL_SERVER_CA_BUNDLE_PATH
+              value: "{{ .Values.server.tls.mountPath }}/ca.crt"
+            {{- if .Values.agent.discoveryDirs }}
+            - name: CERTCTL_DISCOVERY_DIRS
+              valueFrom:
+                configMapKeyRef:
+                  name: {{ include "certctl.fullname" . }}-agent
+                  key: discovery-dirs
+            {{- end }}
+            {{- with .Values.agent.env }}
+            {{- toYaml . | nindent 12 }}
+            {{- end }}
+          resources:
+            {{- toYaml .Values.agent.resources | nindent 12 }}
+          volumeMounts:
+            - name: agent-keys
+              mountPath: {{ .Values.agent.keyDir }}
+            - name: tmp
+              mountPath: /tmp
+            - name: server-tls
+              mountPath: {{ .Values.server.tls.mountPath }}
+              readOnly: true
+      volumes:
+        - name: agent-keys
+          emptyDir:
+            sizeLimit: 1Gi
+        - name: tmp
+          emptyDir: {}
+        - name: server-tls
+          secret:
+            secretName: {{ include "certctl.tls.secretName" . }}
+            defaultMode: 0400
+{{- else if eq .Values.agent.kind "Deployment" }}
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: {{ include "certctl.fullname" . }}-agent
+  labels:
+    {{- include "certctl.labels" . | nindent 4 }}
+    app.kubernetes.io/component: agent
+spec:
+  replicas: {{ .Values.agent.replicas }}
+  selector:
+    matchLabels:
+      {{- include "certctl.agentSelectorLabels" . | nindent 6 }}
+  template:
+    metadata:
+      labels:
+        {{- include "certctl.agentSelectorLabels" . | nindent 8 }}
+    spec:
+      serviceAccountName: {{ include "certctl.serviceAccountName" . }}
+      securityContext:
+        {{- toYaml .Values.agent.securityContext | nindent 8 }}
+      {{- with .Values.imagePullSecrets }}
+      imagePullSecrets:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.agent.nodeSelector }}
+      nodeSelector:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.agent.tolerations }}
+      tolerations:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.agent.affinity }}
+      affinity:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      containers:
+        - name: agent
+          image: {{ include "certctl.agentImage" . }}
+          imagePullPolicy: {{ .Values.agent.image.pullPolicy }}
+          env:
+            - name: CERTCTL_SERVER_URL
+              value: {{ include "certctl.serverURL" . }}
+            - name: CERTCTL_API_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: {{ include "certctl.fullname" . }}-server
+                  key: api-key
+            - name: CERTCTL_AGENT_NAME
+              {{- if .Values.agent.name }}
+              value: {{ .Values.agent.name | quote }}
+              {{- else }}
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.name
+              {{- end }}
+            - name: CERTCTL_KEY_DIR
+              value: {{ .Values.agent.keyDir }}
+            - name: CERTCTL_SERVER_CA_BUNDLE_PATH
+              value: "{{ .Values.server.tls.mountPath }}/ca.crt"
+            {{- if .Values.agent.discoveryDirs }}
+            - name: CERTCTL_DISCOVERY_DIRS
+              valueFrom:
+                configMapKeyRef:
+                  name: {{ include "certctl.fullname" . }}-agent
+                  key: discovery-dirs
+            {{- end }}
+            {{- with .Values.agent.env }}
+            {{- toYaml . | nindent 12 }}
+            {{- end }}
+          resources:
+            {{- toYaml .Values.agent.resources | nindent 12 }}
+          volumeMounts:
+            - name: agent-keys
+              mountPath: {{ .Values.agent.keyDir }}
+            - name: tmp
+              mountPath: /tmp
+            - name: server-tls
+              mountPath: {{ .Values.server.tls.mountPath }}
+              readOnly: true
+      volumes:
+        - name: agent-keys
+          emptyDir:
+            sizeLimit: 1Gi
+        - name: tmp
+          emptyDir: {}
+        - name: server-tls
+          secret:
+            secretName: {{ include "certctl.tls.secretName" . }}
+            defaultMode: 0400
+{{- end }}
+{{- end }}

deploy/helm/certctl/templates/ingress.yaml

@@ -0,0 +1,51 @@
+{{- if .Values.ingress.enabled }}
+{{- if and .Values.ingress.certManager.enabled (not .Values.ingress.certManager.issuerRef.name) -}}
+{{- fail "\n\ningress.certManager.enabled=true but ingress.certManager.issuerRef.name is empty.\n\nSet:\n  --set ingress.certManager.issuerRef.name=<your-issuer-or-clusterissuer>\n\nThis is separate from server.tls.certManager — it issues the external-facing\nIngress cert, not the in-cluster server TLS cert. See docs/tls.md.\n" -}}
+{{- end -}}
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: {{ include "certctl.fullname" . }}
+  labels:
+    {{- include "certctl.labels" . | nindent 4 }}
+  annotations:
+    {{- if .Values.ingress.certManager.enabled }}
+    {{- if eq .Values.ingress.certManager.issuerRef.kind "ClusterIssuer" }}
+    cert-manager.io/cluster-issuer: {{ .Values.ingress.certManager.issuerRef.name | quote }}
+    {{- else }}
+    cert-manager.io/issuer: {{ .Values.ingress.certManager.issuerRef.name | quote }}
+    {{- end }}
+    {{- end }}
+    {{- with .Values.ingress.annotations }}
+    {{- toYaml . | nindent 4 }}
+    {{- end }}
+spec:
+  {{- if .Values.ingress.className }}
+  ingressClassName: {{ .Values.ingress.className }}
+  {{- end }}
+  {{- if .Values.ingress.tls }}
+  tls:
+    {{- range .Values.ingress.tls }}
+    - hosts:
+        {{- range .hosts }}
+        - {{ . | quote }}
+        {{- end }}
+      secretName: {{ .secretName }}
+    {{- end }}
+  {{- end }}
+  rules:
+    {{- range .Values.ingress.hosts }}
+    - host: {{ .host | quote }}
+      http:
+        paths:
+          {{- range .paths }}
+          - path: {{ .path }}
+            pathType: {{ .pathType }}
+            backend:
+              service:
+                name: {{ include "certctl.fullname" $ }}-server
+                port:
+                  number: {{ $.Values.server.service.port }}
+          {{- end }}
+    {{- end }}
+{{- end }}

deploy/helm/certctl/templates/postgres-secret.yaml

@@ -0,0 +1,12 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: {{ include "certctl.fullname" . }}-postgres
+  labels:
+    {{- include "certctl.labels" . | nindent 4 }}
+    app.kubernetes.io/component: postgres
+type: Opaque
+stringData:
+  password: {{ .Values.postgresql.auth.password | default "changeme" | quote }}
+  username: {{ .Values.postgresql.auth.username | quote }}
+  database: {{ .Values.postgresql.auth.database | quote }}

deploy/helm/certctl/templates/postgres-service.yaml

@@ -0,0 +1,18 @@
+{{- if .Values.postgresql.enabled }}
+apiVersion: v1
+kind: Service
+metadata:
+  name: {{ include "certctl.fullname" . }}-postgres
+  labels:
+    {{- include "certctl.labels" . | nindent 4 }}
+    app.kubernetes.io/component: postgres
+spec:
+  clusterIP: None
+  ports:
+    - port: {{ .Values.postgresql.service.port }}
+      targetPort: postgres
+      protocol: TCP
+      name: postgres
+  selector:
+    {{- include "certctl.postgresSelectorLabels" . | nindent 4 }}
+{{- end }}

deploy/helm/certctl/templates/postgres-statefulset.yaml

@@ -0,0 +1,79 @@
+{{- if .Values.postgresql.enabled }}
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: {{ include "certctl.fullname" . }}-postgres
+  labels:
+    {{- include "certctl.labels" . | nindent 4 }}
+    app.kubernetes.io/component: postgres
+spec:
+  serviceName: {{ include "certctl.fullname" . }}-postgres
+  replicas: 1
+  selector:
+    matchLabels:
+      {{- include "certctl.postgresSelectorLabels" . | nindent 6 }}
+  template:
+    metadata:
+      labels:
+        {{- include "certctl.postgresSelectorLabels" . | nindent 8 }}
+    spec:
+      securityContext:
+        {{- toYaml .Values.postgresql.securityContext | nindent 8 }}
+      {{- with .Values.imagePullSecrets }}
+      imagePullSecrets:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      containers:
+        - name: postgres
+          image: {{ include "certctl.postgresImage" . }}
+          imagePullPolicy: {{ .Values.postgresql.image.pullPolicy }}
+          ports:
+            - name: postgres
+              containerPort: 5432
+              protocol: TCP
+          env:
+            - name: POSTGRES_DB
+              valueFrom:
+                secretKeyRef:
+                  name: {{ include "certctl.fullname" . }}-postgres
+                  key: database
+            - name: POSTGRES_USER
+              valueFrom:
+                secretKeyRef:
+                  name: {{ include "certctl.fullname" . }}-postgres
+                  key: username
+            - name: POSTGRES_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: {{ include "certctl.fullname" . }}-postgres
+                  key: password
+            - name: POSTGRES_INITDB_ARGS
+              value: "--encoding=UTF8"
+          livenessProbe:
+            {{- toYaml .Values.postgresql.livenessProbe | nindent 12 }}
+          readinessProbe:
+            {{- toYaml .Values.postgresql.readinessProbe | nindent 12 }}
+          resources:
+            {{- toYaml .Values.postgresql.resources | nindent 12 }}
+          volumeMounts:
+            - name: postgres-data
+              mountPath: /var/lib/postgresql/data
+              subPath: postgres
+            - name: postgres-init
+              mountPath: /docker-entrypoint-initdb.d
+      volumes:
+        - name: postgres-init
+          emptyDir: {}
+  volumeClaimTemplates:
+    - metadata:
+        name: postgres-data
+      spec:
+        accessModes:
+          - ReadWriteOnce
+        {{- if .Values.postgresql.storage.storageClass }}
+        storageClassName: {{ .Values.postgresql.storage.storageClass }}
+        {{- end }}
+        resources:
+          requests:
+            storage: {{ .Values.postgresql.storage.size }}
+{{- end }}

deploy/helm/certctl/templates/server-certificate.yaml

@@ -0,0 +1,31 @@
+{{- if .Values.server.tls.certManager.enabled }}
+{{- include "certctl.tls.required" . }}
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: {{ include "certctl.fullname" . }}-server-tls
+  labels:
+    {{- include "certctl.labels" . | nindent 4 }}
+    app.kubernetes.io/component: server
+spec:
+  secretName: {{ include "certctl.tls.secretName" . }}
+  commonName: {{ .Values.server.tls.certManager.commonName | quote }}
+  dnsNames:
+    {{- range .Values.server.tls.certManager.dnsNames }}
+    - {{ . | quote }}
+    {{- end }}
+  duration: {{ .Values.server.tls.certManager.duration }}
+  renewBefore: {{ .Values.server.tls.certManager.renewBefore }}
+  usages:
+    - server auth
+    - digital signature
+    - key encipherment
+  privateKey:
+    algorithm: ECDSA
+    size: 256
+    rotationPolicy: Always
+  issuerRef:
+    name: {{ .Values.server.tls.certManager.issuerRef.name | quote }}
+    kind: {{ .Values.server.tls.certManager.issuerRef.kind }}
+    group: {{ .Values.server.tls.certManager.issuerRef.group }}
+{{- end }}

deploy/helm/certctl/templates/server-configmap.yaml

@@ -0,0 +1,37 @@
+{{- include "certctl.validateAuthType" . }}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: {{ include "certctl.fullname" . }}-server
+  labels:
+    {{- include "certctl.labels" . | nindent 4 }}
+    app.kubernetes.io/component: server
+data:
+  log-level: {{ .Values.server.logging.level | quote }}
+  auth-type: {{ .Values.server.auth.type | quote }}
+  keygen-mode: {{ .Values.server.keygen.mode | quote }}
+  rate-limit-rps: {{ .Values.server.rateLimiting.rps | quote }}
+  rate-limit-burst: {{ .Values.server.rateLimiting.burst | quote }}
+  {{- if .Values.server.cors.origins }}
+  cors-origins: {{ .Values.server.cors.origins | quote }}
+  {{- end }}
+  {{- if .Values.server.networkScan.enabled }}
+  network-scan-interval: {{ .Values.server.networkScan.interval | quote }}
+  {{- end }}
+  {{- if .Values.server.est.enabled }}
+  est-issuer-id: {{ .Values.server.est.issuerID | quote }}
+  {{- if .Values.server.est.profileID }}
+  est-profile-id: {{ .Values.server.est.profileID | quote }}
+  {{- end }}
+  {{- end }}
+  {{- if .Values.server.smtp.enabled }}
+  smtp-host: {{ .Values.server.smtp.host | quote }}
+  smtp-port: {{ .Values.server.smtp.port | quote }}
+  smtp-username: {{ .Values.server.smtp.username | quote }}
+  smtp-from-address: {{ .Values.server.smtp.fromAddress | quote }}
+  {{- end }}
+  {{- if .Values.server.issuer.acme.enabled }}
+  acme-directory-url: {{ .Values.server.issuer.acme.directoryURL | quote }}
+  acme-email: {{ .Values.server.issuer.acme.email | quote }}
+  acme-challenge-type: {{ .Values.server.issuer.acme.challengeType | quote }}
+  {{- end }}

deploy/helm/certctl/templates/server-deployment.yaml

@@ -0,0 +1,209 @@
+{{- include "certctl.tls.required" . }}
+{{- include "certctl.validateAuthType" . }}
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: {{ include "certctl.fullname" . }}-server
+  labels:
+    {{- include "certctl.labels" . | nindent 4 }}
+    app.kubernetes.io/component: server
+spec:
+  {{- if gt (int .Values.server.replicas) 1 }}
+  replicas: {{ .Values.server.replicas }}
+  {{- end }}
+  selector:
+    matchLabels:
+      {{- include "certctl.serverSelectorLabels" . | nindent 6 }}
+  template:
+    metadata:
+      labels:
+        {{- include "certctl.serverSelectorLabels" . | nindent 8 }}
+      annotations:
+        checksum/config: {{ include (print $.Template.BasePath "/server-configmap.yaml") . | sha256sum }}
+        checksum/secret: {{ include (print $.Template.BasePath "/server-secret.yaml") . | sha256sum }}
+    spec:
+      serviceAccountName: {{ include "certctl.serviceAccountName" . }}
+      securityContext:
+        {{- toYaml .Values.server.securityContext | nindent 8 }}
+      {{- with .Values.imagePullSecrets }}
+      imagePullSecrets:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      containers:
+        - name: server
+          image: {{ include "certctl.serverImage" . }}
+          imagePullPolicy: {{ .Values.server.image.pullPolicy }}
+          ports:
+            - name: https
+              containerPort: {{ .Values.server.port }}
+              protocol: TCP
+          env:
+            - name: CERTCTL_SERVER_HOST
+              value: "0.0.0.0"
+            - name: CERTCTL_SERVER_PORT
+              value: "{{ .Values.server.port }}"
+            - name: CERTCTL_SERVER_TLS_CERT_PATH
+              value: "{{ .Values.server.tls.mountPath }}/tls.crt"
+            - name: CERTCTL_SERVER_TLS_KEY_PATH
+              value: "{{ .Values.server.tls.mountPath }}/tls.key"
+            - name: CERTCTL_DATABASE_URL
+              valueFrom:
+                secretKeyRef:
+                  name: {{ include "certctl.fullname" . }}-server
+                  key: database-url
+            - name: POSTGRES_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: {{ include "certctl.fullname" . }}-postgres
+                  key: password
+            - name: CERTCTL_LOG_LEVEL
+              valueFrom:
+                configMapKeyRef:
+                  name: {{ include "certctl.fullname" . }}-server
+                  key: log-level
+            - name: CERTCTL_LOG_FORMAT
+              value: "json"
+            - name: CERTCTL_AUTH_TYPE
+              valueFrom:
+                configMapKeyRef:
+                  name: {{ include "certctl.fullname" . }}-server
+                  key: auth-type
+            {{- if eq .Values.server.auth.type "api-key" }}
+            - name: CERTCTL_AUTH_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: {{ include "certctl.fullname" . }}-server
+                  key: api-key
+            {{- end }}
+            - name: CERTCTL_KEYGEN_MODE
+              valueFrom:
+                configMapKeyRef:
+                  name: {{ include "certctl.fullname" . }}-server
+                  key: keygen-mode
+            - name: CERTCTL_RATE_LIMIT_RPS
+              valueFrom:
+                configMapKeyRef:
+                  name: {{ include "certctl.fullname" . }}-server
+                  key: rate-limit-rps
+            - name: CERTCTL_RATE_LIMIT_BURST
+              valueFrom:
+                configMapKeyRef:
+                  name: {{ include "certctl.fullname" . }}-server
+                  key: rate-limit-burst
+            {{- if .Values.server.cors.origins }}
+            - name: CERTCTL_CORS_ORIGINS
+              valueFrom:
+                configMapKeyRef:
+                  name: {{ include "certctl.fullname" . }}-server
+                  key: cors-origins
+            {{- end }}
+            {{- if .Values.server.networkScan.enabled }}
+            - name: CERTCTL_NETWORK_SCAN_ENABLED
+              value: "true"
+            - name: CERTCTL_NETWORK_SCAN_INTERVAL
+              valueFrom:
+                configMapKeyRef:
+                  name: {{ include "certctl.fullname" . }}-server
+                  key: network-scan-interval
+            {{- end }}
+            {{- if .Values.server.est.enabled }}
+            - name: CERTCTL_EST_ENABLED
+              value: "true"
+            - name: CERTCTL_EST_ISSUER_ID
+              valueFrom:
+                configMapKeyRef:
+                  name: {{ include "certctl.fullname" . }}-server
+                  key: est-issuer-id
+            {{- if .Values.server.est.profileID }}
+            - name: CERTCTL_EST_PROFILE_ID
+              valueFrom:
+                configMapKeyRef:
+                  name: {{ include "certctl.fullname" . }}-server
+                  key: est-profile-id
+            {{- end }}
+            {{- end }}
+            {{- if .Values.server.smtp.enabled }}
+            - name: CERTCTL_SMTP_HOST
+              valueFrom:
+                configMapKeyRef:
+                  name: {{ include "certctl.fullname" . }}-server
+                  key: smtp-host
+            - name: CERTCTL_SMTP_PORT
+              valueFrom:
+                configMapKeyRef:
+                  name: {{ include "certctl.fullname" . }}-server
+                  key: smtp-port
+            - name: CERTCTL_SMTP_USERNAME
+              valueFrom:
+                configMapKeyRef:
+                  name: {{ include "certctl.fullname" . }}-server
+                  key: smtp-username
+            - name: CERTCTL_SMTP_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: {{ include "certctl.fullname" . }}-server
+                  key: smtp-password
+            - name: CERTCTL_SMTP_FROM_ADDRESS
+              valueFrom:
+                configMapKeyRef:
+                  name: {{ include "certctl.fullname" . }}-server
+                  key: smtp-from-address
+            {{- end }}
+            {{- if .Values.server.issuer.acme.enabled }}
+            - name: CERTCTL_ACME_DIRECTORY_URL
+              valueFrom:
+                configMapKeyRef:
+                  name: {{ include "certctl.fullname" . }}-server
+                  key: acme-directory-url
+            - name: CERTCTL_ACME_EMAIL
+              valueFrom:
+                configMapKeyRef:
+                  name: {{ include "certctl.fullname" . }}-server
+                  key: acme-email
+            - name: CERTCTL_ACME_CHALLENGE_TYPE
+              valueFrom:
+                configMapKeyRef:
+                  name: {{ include "certctl.fullname" . }}-server
+                  key: acme-challenge-type
+            {{- end }}
+            {{- with .Values.server.env }}
+            {{- toYaml . | nindent 12 }}
+            {{- end }}
+          livenessProbe:
+            {{- toYaml .Values.server.livenessProbe | nindent 12 }}
+          readinessProbe:
+            {{- toYaml .Values.server.readinessProbe | nindent 12 }}
+          resources:
+            {{- toYaml .Values.server.resources | nindent 12 }}
+          volumeMounts:
+            - name: tmp
+              mountPath: /tmp
+            - name: tls
+              mountPath: {{ .Values.server.tls.mountPath }}
+              readOnly: true
+            {{- if .Values.server.volumeMounts }}
+            {{- toYaml .Values.server.volumeMounts | nindent 12 }}
+            {{- end }}
+      volumes:
+        - name: tmp
+          emptyDir: {}
+        - name: tls
+          secret:
+            secretName: {{ include "certctl.tls.secretName" . }}
+            defaultMode: 0400
+        {{- if .Values.server.volumes }}
+        {{- toYaml .Values.server.volumes | nindent 8 }}
+        {{- end }}
+      {{- if .Values.nodeAffinity }}
+      affinity:
+        nodeAffinity:
+          {{- toYaml .Values.nodeAffinity | nindent 10 }}
+      {{- else if .Values.podAntiAffinity }}
+      affinity:
+        podAntiAffinity:
+          {{- toYaml .Values.podAntiAffinity | nindent 10 }}
+      {{- else if .Values.podAffinity }}
+      affinity:
+        podAffinity:
+          {{- toYaml .Values.podAffinity | nindent 10 }}
+      {{- end }}

deploy/helm/certctl/templates/server-secret.yaml

@@ -0,0 +1,21 @@
+{{- include "certctl.validateAuthType" . }}
+apiVersion: v1
+kind: Secret
+metadata:
+  name: {{ include "certctl.fullname" . }}-server
+  labels:
+    {{- include "certctl.labels" . | nindent 4 }}
+    app.kubernetes.io/component: server
+type: Opaque
+stringData:
+  # Bundle B / Audit M-018 (PCI-DSS Req 4): sslmode wired from
+  # postgresql.tls.mode. Default "disable" preserves the in-cluster
+  # Helm-bundled-Postgres path; operators on PCI-scoped clusters set
+  # postgresql.tls.mode to require / verify-ca / verify-full.
+  database-url: {{ include "certctl.databaseURL" . | quote }}
+  {{- if and (eq .Values.server.auth.type "api-key") .Values.server.auth.apiKey }}
+  api-key: {{ .Values.server.auth.apiKey | quote }}
+  {{- end }}
+  {{- if .Values.server.smtp.enabled }}
+  smtp-password: {{ .Values.server.smtp.password | quote }}
+  {{- end }}

deploy/helm/certctl/templates/server-service.yaml

@@ -0,0 +1,20 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: {{ include "certctl.fullname" . }}-server
+  labels:
+    {{- include "certctl.labels" . | nindent 4 }}
+    app.kubernetes.io/component: server
+  {{- with .Values.server.service.annotations }}
+  annotations:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+spec:
+  type: {{ .Values.server.service.type }}
+  ports:
+    - port: {{ .Values.server.service.port }}
+      targetPort: https
+      protocol: TCP
+      name: https
+  selector:
+    {{- include "certctl.serverSelectorLabels" . | nindent 4 }}

deploy/helm/certctl/templates/serviceaccount.yaml

@@ -0,0 +1,44 @@
+{{- if .Values.serviceAccount.create }}
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: {{ include "certctl.serviceAccountName" . }}
+  labels:
+    {{- include "certctl.labels" . | nindent 4 }}
+  {{- with .Values.serviceAccount.annotations }}
+  annotations:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+{{- end }}
+{{- if .Values.rbac.create }}
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: {{ include "certctl.fullname" . }}
+  labels:
+    {{- include "certctl.labels" . | nindent 4 }}
+rules:
+{{- if .Values.kubernetesSecrets.enabled }}
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["get", "list", "create", "update", "patch"]
+{{- else }}
+  []
+{{- end }}
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: {{ include "certctl.fullname" . }}
+  labels:
+    {{- include "certctl.labels" . | nindent 4 }}
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: {{ include "certctl.fullname" . }}
+subjects:
+  - kind: ServiceAccount
+    name: {{ include "certctl.serviceAccountName" . }}
+    namespace: {{ .Release.Namespace }}
+{{- end }}

deploy/helm/certctl/values.yaml

@@ -0,0 +1,569 @@
+# Default values for certctl Helm chart
+# This is a YAML-formatted file.
+# Declare variables to be passed into your templates.
+
+# Namespace override (optional)
+namespace: ""
+
+# Global configuration
+commonLabels: {}
+imagePullSecrets: []
+nameOverride: ""
+fullnameOverride: ""
+
+# ==============================================================================
+# Certctl Server Configuration
+# ==============================================================================
+server:
+  # Number of replicas (for HA deployments)
+  replicas: 1
+
+  # Image configuration
+  image:
+    repository: ghcr.io/shankar0123/certctl
+    tag: "" # defaults to Chart.appVersion
+    pullPolicy: IfNotPresent
+
+  # Server port
+  port: 8443
+
+  # Resource requests and limits
+  resources:
+    requests:
+      cpu: 100m
+      memory: 128Mi
+    limits:
+      cpu: 500m
+      memory: 512Mi
+
+  # Pod security context
+  securityContext:
+    runAsNonRoot: true
+    runAsUser: 1000
+    runAsGroup: 1000
+    fsGroup: 1000
+    readOnlyRootFilesystem: true
+    allowPrivilegeEscalation: false
+    capabilities:
+      drop:
+        - ALL
+
+  # Liveness and readiness probes (HTTPS-only as of v2.2).
+  #
+  # The two paths exposed for probes are `/health` and `/ready` —
+  # registered in internal/api/router/router.go:76-85 and bypassing the
+  # auth middleware via the no-auth list at cmd/server/main.go:920.
+  # Both serve the same JSON shape today (`{"status":"healthy"}` /
+  # `{"status":"ready"}`) but exist as separate routes so liveness and
+  # readiness can diverge in the future without renaming.
+  livenessProbe:
+    httpGet:
+      path: /health
+      port: https
+      scheme: HTTPS
+    initialDelaySeconds: 10
+    periodSeconds: 10
+    timeoutSeconds: 5
+    failureThreshold: 3
+
+  # U-2 (P1, cat-u-healthcheck_protocol_mismatch — adjacent fix): pre-U-2
+  # the readiness probe pointed at `/readyz`, the conventional kube-flavor
+  # name. The certctl server doesn't register `/readyz` (only `/health`
+  # and `/ready`) — see cmd/server/main.go:920 and
+  # internal/api/router/router.go:81. K8s readiness probes therefore
+  # received a 404 (or, with auth enabled, a 401 from the api-key middleware
+  # because `/readyz` was NOT in the no-auth bypass set), pods stayed
+  # `NotReady` indefinitely, and Helm rollouts stalled. Post-U-2 the path
+  # matches a registered route.
+  readinessProbe:
+    httpGet:
+      path: /ready
+      port: https
+      scheme: HTTPS
+    initialDelaySeconds: 5
+    periodSeconds: 5
+    timeoutSeconds: 3
+    failureThreshold: 2
+
+  # TLS configuration — REQUIRED. HTTPS is the only supported mode (v2.2+).
+  # Operator must configure EXACTLY ONE of:
+  #   (a) server.tls.existingSecret: <name>        # pre-existing kubernetes.io/tls Secret
+  #   (b) server.tls.certManager.enabled: true     # provision a cert-manager Certificate CR
+  # Refusing to set either makes `helm template` fail with a diagnostic pointing at docs/tls.md.
+  tls:
+    # Name of a pre-existing Secret (type kubernetes.io/tls) holding tls.crt + tls.key (+ optional ca.crt).
+    # Leave empty to fall through to the cert-manager path.
+    existingSecret: ""
+
+    # Mount path for the TLS Secret inside the server + agent containers.
+    mountPath: /etc/certctl/tls
+
+    # cert-manager auto-provisioning. Opt-in (off by default per milestone §3.4).
+    certManager:
+      enabled: false
+
+      # Secret name the cert-manager Certificate CR writes into. Agents and the server
+      # both read from this Secret. If empty, defaults to "<fullname>-tls".
+      secretName: ""
+
+      # Cert-manager issuer reference.
+      issuerRef:
+        name: ""                      # e.g. "letsencrypt-prod" or "internal-ca"
+        kind: ClusterIssuer           # ClusterIssuer or Issuer
+        group: cert-manager.io
+
+      # Subject fields on the issued cert.
+      commonName: "certctl-server"
+      dnsNames:
+        - certctl-server
+        - localhost
+
+      # Certificate lifetime + renewal window.
+      duration: 2160h                 # 90 days
+      renewBefore: 360h               # 15 days
+
+  # Service type (ClusterIP, LoadBalancer, NodePort)
+  service:
+    type: ClusterIP
+    port: 8443
+    annotations: {}
+
+  # Authentication configuration.
+  # Valid types: "api-key" (production) or "none" (demo only — disables
+  # authentication on the API and logs a loud Warn at server startup).
+  # For JWT/OIDC, run an authenticating gateway in front of certctl
+  # (oauth2-proxy / Envoy ext_authz / Traefik ForwardAuth / Pomerium)
+  # and set type=none here so the gateway terminates federated identity.
+  # See docs/architecture.md "Authenticating-gateway pattern".
+  #
+  # G-1 (P1): pre-G-1 the chart accepted server.auth.type=jwt and the
+  # certctl-server container silently routed every request through the
+  # api-key bearer middleware — silent auth downgrade. Post-G-1 the
+  # chart's `certctl.validateAuthType` template helper rejects any value
+  # outside {api-key, none} at template time. See
+  # docs/upgrade-to-v2-jwt-removal.md if you previously set type=jwt.
+  auth:
+    type: api-key
+    apiKey: ""     # REQUIRED when type=api-key (set via --set or values override).
+
+  # Logging configuration
+  logging:
+    level: info  # debug, info, warn, error
+    format: json  # json or text
+
+  # SMTP configuration for email notifications (optional)
+  smtp:
+    enabled: false
+    host: ""
+    port: 587
+    username: ""
+    password: ""
+    fromAddress: ""
+    useTLS: true
+
+  # Certificate digest digest (periodic email summary)
+  digest:
+    enabled: false
+    interval: "24h"
+    recipients: []
+    # Example:
+    # - admin@example.com
+    # - ops@example.com
+
+  # Enrollment over Secure Transport (EST) configuration
+  est:
+    enabled: false
+    issuerID: "iss-local"
+    profileID: ""
+
+  # Rate limiting configuration
+  rateLimiting:
+    rps: 100      # Requests per second
+    burst: 200    # Burst capacity
+
+  # Network scanning configuration
+  networkScan:
+    enabled: false
+    interval: "6h"
+
+  # Certificate key generation mode
+  keygen:
+    mode: agent   # Options: agent (production), server (demo with warning)
+
+  # CORS configuration
+  cors:
+    origins: ""   # Comma-separated list, empty means deny all cross-origin requests
+
+  # Issuer connectors configuration
+  issuer:
+    local:
+      enabled: true
+      # For sub-CA mode, provide these paths:
+      # caCertPath: /path/to/ca.crt
+      # caKeyPath: /path/to/ca.key
+
+    acme:
+      enabled: false
+      directoryURL: ""
+      email: ""
+      challengeType: "http-01"  # Options: http-01, dns-01, dns-persist-01
+      # DNS configuration (for dns-01 or dns-persist-01)
+      # dnsPresentScript: /path/to/dns-present.sh
+      # dnsCleanupScript: /path/to/dns-cleanup.sh
+      # dnsPropagationWait: "30s"
+      # dnsPersistIssuerDomain: "validation.example.com"
+      # EAB configuration (for ZeroSSL, Google Trust Services, etc.)
+      # eabKid: ""
+      # eabHmac: ""
+
+    stepca:
+      enabled: false
+      # rootCAPath: /path/to/root_ca.crt
+      # intermediateCAPath: /path/to/intermediate_ca.crt
+      # provisionerName: ""
+      # provisionerPassword: ""
+
+    openssl:
+      enabled: false
+      # signScript: /path/to/sign.sh
+      # revokeScript: /path/to/revoke.sh
+      # crlScript: /path/to/crl.sh
+      # timeoutSeconds: 30
+
+  # Notifier connectors configuration
+  notifiers:
+    slack:
+      enabled: false
+      # webhookUrl: ""
+      # channel: ""
+      # username: ""
+      # iconEmoji: ""
+
+    teams:
+      enabled: false
+      # webhookUrl: ""
+
+    pagerduty:
+      enabled: false
+      # routingKey: ""
+      # severity: warning
+
+    opsgenie:
+      enabled: false
+      # apiKey: ""
+      # priority: P3
+
+  # Additional environment variables
+  # Will be passed as-is to the server container
+  env: {}
+  # Example:
+  # CERTCTL_SCHEDULER_RENEWAL_CHECK_INTERVAL: "1h"
+  # CERTCTL_DATABASE_MAX_CONNS: "25"
+
+  # Additional volume mounts for custom configurations
+  # volumeMounts: []
+  # - name: ca-cert
+  #   mountPath: /etc/ssl/certs/ca.crt
+  #   subPath: ca.crt
+
+  # Additional volumes
+  # volumes: []
+  # - name: ca-cert
+  #   secret:
+  #     secretName: ca-cert
+
+# ==============================================================================
+# PostgreSQL Configuration
+# ==============================================================================
+postgresql:
+  # Enable/disable PostgreSQL (set to false if using external database)
+  enabled: true
+
+  # Image configuration
+  image:
+    repository: postgres
+    tag: "16-alpine"
+    pullPolicy: IfNotPresent
+
+  # Authentication
+  auth:
+    database: certctl
+    username: certctl
+    # REQUIRED — set via `--set postgresql.auth.password=<value>` or values override.
+    #
+    # WARNING (U-1): rotating this value after first deploy does NOT change the
+    # database password. The `postgres:16-alpine` image runs `initdb` only when
+    # /var/lib/postgresql/data is empty, so POSTGRES_PASSWORD is written into
+    # pg_authid exactly once — on the first boot of the StatefulSet's PVC.
+    # Subsequent rollouts pick up the new env value in the postgres container
+    # but the certctl-server container's CERTCTL_DATABASE_URL also picks up
+    # the new value, while pg_authid still expects the old one — leading to
+    # `pq: password authentication failed for user "certctl"` (SQLSTATE 28P01).
+    #
+    # The certctl-server emits guidance via internal/repository/postgres/db.go::
+    # wrapPingError when it sees SQLSTATE 28P01 at startup. To resolve in a
+    # Helm deployment:
+    #   - Non-destructive (preferred for environments with data):
+    #       kubectl exec -it <release>-postgres-0 -- \
+    #         psql -U certctl -c "ALTER ROLE certctl PASSWORD '<new>';"
+    #     then update the secret/values to match and let the certctl-server
+    #     pod restart against the matching credential.
+    #   - Destructive (DESTROYS DATA — only acceptable on dev/demo PVCs):
+    #       helm uninstall <release> && \
+    #       kubectl delete pvc -l app.kubernetes.io/name=certctl,app.kubernetes.io/component=postgres && \
+    #       helm install <release> ...  # PVC re-creates empty, initdb seeds new password
+    password: ""
+
+  # ─────────────────────────────────────────────────────────────────────
+  # Bundle B / Audit M-018 (PCI-DSS Req 4 / CWE-319): TLS to Postgres
+  # ─────────────────────────────────────────────────────────────────────
+  # postgresql.tls.mode is wired into the database-url sslmode parameter
+  # (see templates/_helpers.tpl::certctl.databaseURL).
+  #
+  # Acceptable values (lib/pq):
+  #   disable     — no TLS (default, preserves in-cluster pod-to-pod
+  #                 traffic on the K8s pod network).
+  #   require     — TLS required, no certificate verification.
+  #   verify-ca   — TLS required + verify CA chain.
+  #   verify-full — TLS required + verify CA chain + verify hostname.
+  #
+  # PCI-DSS Req 4 v4.0 §2.2.5 requires verify-ca or verify-full when the
+  # database carries sensitive data crossing untrusted networks (RDS,
+  # Cloud SQL, cross-VPC, etc). The bundled Helm Postgres runs in the
+  # same pod network as certctl-server; sslmode=disable is acceptable
+  # there only when the cluster CNI provides L2/L3 encryption (Cilium
+  # WireGuard, Calico Wireguard, Tailscale operator, etc).
+  #
+  # When mode != disable AND tls.caSecretRef is set, the CA bundle is
+  # mounted at /etc/postgresql-ca/ca.crt and the server's PGSSLROOTCERT
+  # env points there. caSecretRef must reference an existing Secret with
+  # a "ca.crt" key.
+  tls:
+    mode: disable
+    # caSecretRef: ""  # Secret with ca.crt key (required for verify-ca/verify-full)
+
+  # Storage configuration
+  storage:
+    size: 10Gi
+    storageClass: ""  # Uses default StorageClass if empty
+    # deleteOnTermination: false  # Keep data on Helm uninstall
+
+  # Resource requests and limits
+  resources:
+    requests:
+      cpu: 100m
+      memory: 256Mi
+    limits:
+      cpu: 500m
+      memory: 512Mi
+
+  # Pod security context
+  securityContext:
+    runAsNonRoot: true
+    runAsUser: 999
+    runAsGroup: 999
+    fsGroup: 999
+
+  # Liveness and readiness probes
+  livenessProbe:
+    exec:
+      command:
+        - /bin/sh
+        - -c
+        - pg_isready -U certctl -d certctl
+    initialDelaySeconds: 10
+    periodSeconds: 10
+    timeoutSeconds: 5
+    failureThreshold: 3
+
+  readinessProbe:
+    exec:
+      command:
+        - /bin/sh
+        - -c
+        - pg_isready -U certctl -d certctl
+    initialDelaySeconds: 5
+    periodSeconds: 5
+    timeoutSeconds: 3
+    failureThreshold: 2
+
+  # Service configuration
+  service:
+    type: ClusterIP
+    port: 5432
+
+  # PostgreSQL-specific settings
+  postgresqlConfig: {}
+  # Example:
+  # max_connections: "200"
+  # shared_buffers: "256MB"
+
+# ==============================================================================
+# Certctl Agent Configuration
+# ==============================================================================
+agent:
+  # Enable/disable agent deployment
+  enabled: true
+
+  # Deployment strategy: DaemonSet (recommended) or Deployment
+  kind: DaemonSet  # Options: DaemonSet, Deployment
+
+  # Image configuration
+  image:
+    repository: ghcr.io/shankar0123/certctl-agent
+    tag: ""  # defaults to Chart.appVersion
+    pullPolicy: IfNotPresent
+
+  # Number of replicas (for Deployment kind; ignored for DaemonSet)
+  replicas: 1
+
+  # Resource requests and limits
+  resources:
+    requests:
+      cpu: 50m
+      memory: 64Mi
+    limits:
+      cpu: 200m
+      memory: 256Mi
+
+  # Pod security context
+  securityContext:
+    runAsNonRoot: true
+    runAsUser: 1000
+    runAsGroup: 1000
+    fsGroup: 1000
+    readOnlyRootFilesystem: true
+    allowPrivilegeEscalation: false
+    capabilities:
+      drop:
+        - ALL
+
+  # Agent name (can be overridden per pod via StatefulSet ordinals)
+  name: ""  # If empty, uses release name
+
+  # Key storage directory
+  keyDir: /var/lib/certctl/keys
+
+  # Certificate discovery directories (comma-separated)
+  discoveryDirs: ""
+  # Example: "/etc/ssl/certs,/etc/pki/tls"
+
+  # Node selector for agent pods (for DaemonSet)
+  nodeSelector: {}
+  # Example:
+  # node-role.kubernetes.io/worker: "true"
+
+  # Tolerations for agent pods
+  tolerations: []
+  # Example:
+  # - key: node-role
+  #   operator: Equal
+  #   value: worker
+  #   effect: NoSchedule
+
+  # Affinity rules
+  affinity: {}
+
+  # Additional environment variables
+  env: {}
+
+# ==============================================================================
+# Ingress Configuration
+# ==============================================================================
+ingress:
+  enabled: false
+  className: ""
+  annotations: {}
+    # kubernetes.io/ingress.class: nginx
+
+  # Optional cert-manager integration for the public-facing Ingress cert.
+  # This is completely independent of server.tls.* — the Ingress terminates
+  # an *additional* TLS hop between the internet and the in-cluster Service.
+  # Leave disabled unless an Ingress is exposing certctl to the outside world.
+  certManager:
+    enabled: false
+    issuerRef:
+      name: ""                      # e.g. "letsencrypt-prod"
+      kind: ClusterIssuer           # ClusterIssuer or Issuer
+  hosts:
+    - host: certctl.local
+      paths:
+        - path: /
+          pathType: Prefix
+  tls: []
+    # - secretName: certctl-tls
+    #   hosts:
+    #     - certctl.local
+
+# ==============================================================================
+# Service Account Configuration
+# ==============================================================================
+serviceAccount:
+  create: true
+  annotations: {}
+  name: ""  # defaults to release name if empty
+
+# ==============================================================================
+# RBAC Configuration
+# ==============================================================================
+rbac:
+  create: true
+
+# ==============================================================================
+# Kubernetes Secrets Target Connector
+# ==============================================================================
+kubernetesSecrets:
+  # Enable RBAC rules for managing TLS Secrets
+  enabled: false
+
+# ==============================================================================
+# Pod Disruption Budget (for HA deployments)
+# ==============================================================================
+podDisruptionBudget:
+  enabled: false
+  minAvailable: 1
+  # maxUnavailable: 1
+
+# ==============================================================================
+# Monitoring Configuration
+# ==============================================================================
+monitoring:
+  enabled: false
+  # Prometheus ServiceMonitor
+  serviceMonitor:
+    enabled: false
+    interval: 30s
+    scrapeTimeout: 10s
+    # labels: {}
+    # selector: {}
+
+# ==============================================================================
+# Advanced Configuration
+# ==============================================================================
+
+# Node affinity for server pods
+nodeAffinity: {}
+
+# Pod affinity for server pods
+podAffinity: {}
+
+# Pod anti-affinity for server pods (for HA)
+podAntiAffinity: {}
+# Example:
+# podAntiAffinity:
+#   preferredDuringSchedulingIgnoredDuringExecution:
+#     - weight: 100
+#       podAffinityTerm:
+#         labelSelector:
+#           matchExpressions:
+#             - key: app.kubernetes.io/name
+#               operator: In
+#               values:
+#                 - certctl
+#         topologyKey: kubernetes.io/hostname
+
+# Custom labels for all resources
+customLabels: {}
+
+# Custom annotations for all resources
+customAnnotations: {}

deploy/helm/examples/values-acme-dns01.yaml

@@ -0,0 +1,77 @@
+# Certctl with ACME DNS-01 Challenge (Let's Encrypt)
+# Enables automatic certificate issuance from Let's Encrypt
+# using DNS-01 verification (wildcard-capable)
+
+server:
+  auth:
+    type: api-key
+    apiKey: "CHANGE_ME"
+
+  issuer:
+    local:
+      enabled: true
+
+    acme:
+      enabled: true
+      directoryURL: https://acme-v02.api.letsencrypt.org/directory
+      email: admin@example.com
+      challengeType: dns-01
+      dnsPresentScript: /scripts/dns-present.sh
+      dnsCleanupScript: /scripts/dns-cleanup.sh
+      dnsPropagationWait: 30s
+      # For DNS-PERSIST-01 (standing validation record, no per-renewal updates):
+      # challengeType: dns-persist-01
+      # dnsPersistIssuerDomain: validation.example.com
+
+  # Mount DNS scripts as ConfigMap
+  volumes:
+    - name: dns-scripts
+      configMap:
+        name: dns-scripts
+        defaultMode: 0755
+
+  volumeMounts:
+    - name: dns-scripts
+      mountPath: /scripts
+      readOnly: true
+
+postgresql:
+  enabled: true
+  storage:
+    size: 20Gi
+
+agent:
+  enabled: true
+  kind: DaemonSet
+
+ingress:
+  enabled: true
+  className: nginx
+  hosts:
+    - host: certctl.example.com
+      paths:
+        - path: /
+          pathType: Prefix
+
+---
+# You'll need to create the DNS scripts ConfigMap separately:
+#
+# kubectl create configmap dns-scripts \
+#   --from-file=dns-present.sh=./scripts/dns-present.sh \
+#   --from-file=dns-cleanup.sh=./scripts/dns-cleanup.sh
+#
+# Example dns-present.sh (Cloudflare):
+#   #!/bin/bash
+#   DOMAIN=$1
+#   TOKEN=$2
+#
+#   curl -X POST "https://api.cloudflare.com/client/v4/zones/{zone_id}/dns_records" \
+#     -H "Authorization: Bearer ${CLOUDFLARE_API_TOKEN}" \
+#     -d "{\"type\":\"TXT\",\"name\":\"_acme-challenge.${DOMAIN}\",\"content\":\"${TOKEN}\"}"
+#
+# Example dns-cleanup.sh (Cloudflare):
+#   #!/bin/bash
+#   DOMAIN=$1
+#
+#   curl -X DELETE "https://api.cloudflare.com/client/v4/zones/{zone_id}/dns_records/{record_id}" \
+#     -H "Authorization: Bearer ${CLOUDFLARE_API_TOKEN}"

deploy/helm/examples/values-dev.yaml

@@ -0,0 +1,99 @@
+# Certctl Development Configuration
+# Lightweight setup for development and testing
+# - Single server replica
+# - Small PostgreSQL storage
+# - Minimal resource limits
+# - No ingress or monitoring
+# - Demo auth mode (no API key required)
+
+server:
+  replicas: 1
+
+  image:
+    repository: ghcr.io/shankar0123/certctl
+    pullPolicy: IfNotPresent  # Use latest tag
+
+  port: 8443
+
+  resources:
+    requests:
+      cpu: 50m
+      memory: 64Mi
+    limits:
+      cpu: 200m
+      memory: 256Mi
+
+  auth:
+    type: none  # Demo mode - no authentication
+
+  logging:
+    level: debug
+    format: json
+
+  service:
+    type: LoadBalancer  # Easy external access for dev
+
+  issuer:
+    local:
+      enabled: true
+
+  rateLimiting:
+    rps: 100
+    burst: 200
+
+postgresql:
+  enabled: true
+
+  image:
+    repository: postgres
+    tag: "16-alpine"
+    pullPolicy: IfNotPresent
+
+  auth:
+    database: certctl
+    username: certctl
+    password: "dev-password-change-me"
+
+  storage:
+    size: 5Gi
+    storageClass: ""  # Use default storage class
+
+  resources:
+    requests:
+      cpu: 50m
+      memory: 128Mi
+    limits:
+      cpu: 200m
+      memory: 256Mi
+
+agent:
+  enabled: true
+  kind: Deployment
+  replicas: 1
+
+  image:
+    repository: ghcr.io/shankar0123/certctl-agent
+    pullPolicy: IfNotPresent
+
+  resources:
+    requests:
+      cpu: 25m
+      memory: 32Mi
+    limits:
+      cpu: 100m
+      memory: 128Mi
+
+ingress:
+  enabled: false
+
+serviceAccount:
+  create: true
+
+rbac:
+  create: true
+
+monitoring:
+  enabled: false
+
+customLabels:
+  environment: development

deploy/helm/examples/values-external-db.yaml

@@ -0,0 +1,50 @@
+# Certctl with External PostgreSQL Database
+# Use this when PostgreSQL is managed externally:
+# - AWS RDS
+# - Cloud SQL (Google Cloud)
+# - Azure Database for PostgreSQL
+# - Self-managed PostgreSQL server
+
+server:
+  replicas: 2
+
+  auth:
+    type: api-key
+    apiKey: "CHANGE_ME"
+
+  issuer:
+    local:
+      enabled: true
+
+  # Pass external database URL via environment variable
+  env:
+    CERTCTL_DATABASE_URL: "postgres://certctl:CHANGE_ME@postgres.example.com:5432/certctl?sslmode=require"
+
+# Disable internal PostgreSQL
+postgresql:
+  enabled: false
+
+agent:
+  enabled: true
+  kind: DaemonSet
+
+ingress:
+  enabled: true
+  className: nginx
+  hosts:
+    - host: certctl.example.com
+      paths:
+        - path: /
+          pathType: Prefix
+
+# For AWS RDS with IAM authentication:
+# env:
+#   CERTCTL_DATABASE_URL: "postgres://certctl:CHANGE_ME@mydb.123456789.us-east-1.rds.amazonaws.com:5432/certctl?sslmode=require"
+
+# For Google Cloud SQL:
+# env:
+#   CERTCTL_DATABASE_URL: "postgres://certctl:CHANGE_ME@/certctl?host=/cloudsql/PROJECT:REGION:INSTANCE&sslmode=require"
+
+# For Azure Database:
+# env:
+#   CERTCTL_DATABASE_URL: "postgres://certctl@servername:CHANGE_ME@servername.postgres.database.azure.com:5432/certctl?sslmode=require"

deploy/helm/examples/values-prod-ha.yaml

@@ -0,0 +1,159 @@
+# Certctl Production HA Configuration
+# High availability deployment with:
+# - 3 server replicas with pod anti-affinity
+# - Large PostgreSQL storage
+# - Resource limits for production
+# - Prometheus monitoring
+# - Network policies enforcement
+
+namespace: certctl
+
+server:
+  replicas: 3
+
+  image:
+    repository: ghcr.io/shankar0123/certctl
+    tag: "2.1.0"
+    pullPolicy: IfNotPresent
+
+  port: 8443
+
+  resources:
+    requests:
+      cpu: 250m
+      memory: 256Mi
+    limits:
+      cpu: 1000m
+      memory: 512Mi
+
+  auth:
+    type: api-key
+    apiKey: "CHANGE_ME_IN_PRODUCTION"  # Use --set or sealed-secrets
+
+  logging:
+    level: info
+    format: json
+
+  service:
+    type: ClusterIP
+    annotations:
+      prometheus.io/scrape: "true"
+      prometheus.io/port: "8443"
+      prometheus.io/path: "/api/v1/metrics/prometheus"
+
+  issuer:
+    local:
+      enabled: true
+    acme:
+      enabled: true
+      directoryURL: https://acme-v02.api.letsencrypt.org/directory
+      email: admin@example.com
+      challengeType: dns-01
+
+  rateLimiting:
+    rps: 500
+    burst: 1000
+
+postgresql:
+  enabled: true
+
+  image:
+    repository: postgres
+    tag: "16-alpine"
+    pullPolicy: IfNotPresent
+
+  auth:
+    database: certctl
+    username: certctl
+    password: "CHANGE_ME_IN_PRODUCTION"  # Use --set or sealed-secrets
+
+  storage:
+    size: 100Gi
+    storageClass: "fast-ssd"  # Use your high-performance storage class
+
+  resources:
+    requests:
+      cpu: 500m
+      memory: 512Mi
+    limits:
+      cpu: 2000m
+      memory: 2Gi
+
+agent:
+  enabled: true
+  kind: DaemonSet
+
+  image:
+    repository: ghcr.io/shankar0123/certctl-agent
+    tag: "2.1.0"
+    pullPolicy: IfNotPresent
+
+  resources:
+    requests:
+      cpu: 100m
+      memory: 128Mi
+    limits:
+      cpu: 500m
+      memory: 256Mi
+
+  discoveryDirs: "/etc/ssl/certs,/etc/pki/tls,/etc/ssl"
+
+ingress:
+  enabled: true
+  className: nginx
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+    nginx.ingress.kubernetes.io/ssl-redirect: "true"
+    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
+  hosts:
+    - host: certctl.example.com
+      paths:
+        - path: /
+          pathType: Prefix
+  tls:
+    - secretName: certctl-tls
+      hosts:
+        - certctl.example.com
+
+serviceAccount:
+  create: true
+  annotations:
+    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT:role/certctl-role  # For IRSA on AWS
+
+rbac:
+  create: true
+
+podDisruptionBudget:
+  enabled: true
+  minAvailable: 2
+
+monitoring:
+  enabled: true
+  serviceMonitor:
+    enabled: true
+    interval: 30s
+    scrapeTimeout: 10s
+
+# Pod anti-affinity for HA
+podAntiAffinity:
+  requiredDuringSchedulingIgnoredDuringExecution:
+    - labelSelector:
+        matchExpressions:
+          - key: app.kubernetes.io/name
+            operator: In
+            values:
+              - certctl
+          - key: app.kubernetes.io/component
+            operator: In
+            values:
+              - server
+      topologyKey: kubernetes.io/hostname
+
+customLabels:
+  environment: production
+  team: platform
+  cost-center: ops
+
+customAnnotations:
+  slack-alerts: "#ops"
+  backup-policy: daily

deploy/test/healthcheck_test.go

@@ -0,0 +1,233 @@
+//go:build integration
+
+// Package integration_test — image-level HEALTHCHECK contract.
+//
+// U-2 (P1, cat-u-healthcheck_protocol_mismatch): pre-U-2 the published
+// server image's Dockerfile HEALTHCHECK called `curl -f http://localhost:
+// 8443/health` against an HTTPS-only listener (HTTPS-Everywhere milestone,
+// v2.2 / tag v2.0.47). Operators outside docker-compose / Helm saw the
+// container reported as `unhealthy` indefinitely. The compose stack
+// overrode this HEALTHCHECK with `--cacert + https://`; the Helm chart
+// uses explicit `httpGet` probes that ignore Docker's HEALTHCHECK; the 5
+// example compose files all override with `curl -sfk https://localhost:
+// 8443/health`. So the observable failure was scoped to bare `docker run`
+// / Docker Swarm / Nomad / ECS users — exactly the "I just pulled the
+// published image" path.
+//
+// This file's tests pin the contract at the binary-image level. The
+// matching CI grep guardrail in .github/workflows/ci.yml catches the
+// regression at the Dockerfile-source level; both layers are needed
+// because someone could replace the HEALTHCHECK line with a sibling
+// broken pattern that the grep doesn't catch (e.g., a TCP-only check
+// against the HTTPS port).
+//
+// Run alongside the rest of the integration suite:
+//
+//	cd deploy/test && go test -tags integration -v -run Healthcheck
+//
+// The tests skip cleanly with t.Skip when docker is not available
+// (CI without docker-in-docker, sandbox environments, etc.) so they
+// don't block local development on machines without docker.
+//
+// Q-1 closure (cat-s3-58ce7e9840be): this file's 5 t.Skip sites are
+// audited and intentional:
+//
+//   - Line 85, 146, 207: `if !dockerAvailable(t)` skips when `docker info`
+//     fails. These are precondition gates; without docker there's nothing
+//     to assert against. Run via: `docker info >/dev/null && go test
+//     -tags integration ./deploy/test/...`.
+//   - Line 209-210: `if testing.Short()` keeps the ~45s runtime probe
+//     off the default `go test ./... -short` path. Run via: omit -short.
+//   - Line 212: hard t.Skip for the runtime probe contract — image-spec
+//     contract above (TestPublishedServerImage_HealthcheckSpecUsesHTTPS)
+//     covers the audit-flagged regression at the Dockerfile-source level.
+//     Re-enable once the integration harness provisions a sidecar postgres
+//     for image-level smoke; the existing skip message names this
+//     remediation explicitly. Tracked via the in-source TODO (intentional,
+//     not abandoned).
+package integration_test
+
+import (
+	"encoding/json"
+	"os/exec"
+	"strings"
+	"testing"
+	"time"
+)
+
+// dockerAvailable returns true when `docker version` returns 0.
+// We cache it across tests in this file so the skip message prints once.
+func dockerAvailable(t *testing.T) bool {
+	t.Helper()
+	cmd := exec.Command("docker", "version", "--format", "{{.Server.Version}}")
+	out, err := cmd.CombinedOutput()
+	if err != nil {
+		t.Logf("docker not available: %v\noutput: %s", err, string(out))
+		return false
+	}
+	return true
+}
+
+// dockerCmd runs `docker <args...>` with a 60s budget, returning stdout
+// + stderr combined and the exit error if any. Used for short-lived
+// probes (inspect, build, run -d).
+func dockerCmd(t *testing.T, timeout time.Duration, args ...string) (string, error) {
+	t.Helper()
+	cmd := exec.Command("docker", args...)
+	done := make(chan struct{})
+	var out []byte
+	var err error
+	go func() {
+		out, err = cmd.CombinedOutput()
+		close(done)
+	}()
+	select {
+	case <-done:
+		return string(out), err
+	case <-time.After(timeout):
+		_ = cmd.Process.Kill()
+		t.Fatalf("docker %v timed out after %v", args, timeout)
+		return "", err
+	}
+}
+
+// TestPublishedServerImage_HealthcheckSpecUsesHTTPS performs the Dockerfile-
+// source-level shipped-shape pin: the inspected image's Healthcheck.Test
+// array MUST contain "https://localhost:8443/health" (and MUST NOT
+// contain "http://localhost:8443/health"). This is the lightweight half
+// of the contract — it doesn't require running the container, only
+// building it. It catches the audit-flagged bug directly.
+func TestPublishedServerImage_HealthcheckSpecUsesHTTPS(t *testing.T) {
+	if !dockerAvailable(t) {
+		t.Skip("docker not available — skipping image-level HEALTHCHECK test")
+	}
+
+	const imgTag = "certctl-u2-healthcheck-spec-test"
+	t.Cleanup(func() {
+		_, _ = dockerCmd(t, 30*time.Second, "rmi", "-f", imgTag)
+	})
+
+	// Build the server image. Use the repo root as context (this test
+	// file lives at deploy/test/, the Dockerfile at the repo root).
+	buildOut, err := dockerCmd(t, 5*time.Minute,
+		"build", "-f", "../../Dockerfile", "-t", imgTag, "../..")
+	if err != nil {
+		t.Fatalf("docker build failed: %v\noutput:\n%s", err, buildOut)
+	}
+
+	// Inspect the shipped HEALTHCHECK metadata.
+	inspectOut, err := dockerCmd(t, 30*time.Second,
+		"inspect", "--format", "{{json .Config.Healthcheck}}", imgTag)
+	if err != nil {
+		t.Fatalf("docker inspect failed: %v\noutput:\n%s", err, inspectOut)
+	}
+
+	var hc struct {
+		Test     []string
+		Interval int64
+		Timeout  int64
+	}
+	if err := json.Unmarshal([]byte(strings.TrimSpace(inspectOut)), &hc); err != nil {
+		t.Fatalf("could not parse Healthcheck JSON %q: %v", inspectOut, err)
+	}
+
+	joined := strings.Join(hc.Test, " ")
+
+	// Positive contract.
+	if !strings.Contains(joined, "https://localhost:8443/health") {
+		t.Errorf("Healthcheck.Test does not target https://localhost:8443/health\nfull: %v", hc.Test)
+	}
+
+	// Negative contract — pre-U-2 regression shape MUST be absent.
+	if strings.Contains(joined, "http://localhost:8443/health") {
+		t.Errorf("Healthcheck.Test still contains the pre-U-2 plaintext shape: %v", hc.Test)
+	}
+
+	// `-k` (or `--insecure`) must be present because the bootstrap cert
+	// is per-deploy and the published image can't pin a CA bundle —
+	// see the U-2 closure docblock on Dockerfile and the audit doc.
+	if !strings.Contains(joined, "-k") && !strings.Contains(joined, "--insecure") {
+		t.Errorf("Healthcheck.Test omits -k / --insecure flag (required for self-signed bootstrap probe): %v", hc.Test)
+	}
+}
+
+// TestPublishedAgentImage_HealthcheckSpecExists pins the U-2 adjacent
+// fix that added a HEALTHCHECK to the agent image. Pre-U-2 the agent
+// image had no HEALTHCHECK declaration, so bare-`docker run` agents got
+// `none` health status from Docker. Post-U-2 the agent uses pgrep to
+// verify the process is alive (mirroring the docker-compose pattern at
+// deploy/docker-compose.yml:173, which also became reliable post-U-2
+// because procps is now installed in the runtime image).
+func TestPublishedAgentImage_HealthcheckSpecExists(t *testing.T) {
+	if !dockerAvailable(t) {
+		t.Skip("docker not available — skipping image-level HEALTHCHECK test")
+	}
+
+	const imgTag = "certctl-u2-agent-healthcheck-spec-test"
+	t.Cleanup(func() {
+		_, _ = dockerCmd(t, 30*time.Second, "rmi", "-f", imgTag)
+	})
+
+	buildOut, err := dockerCmd(t, 5*time.Minute,
+		"build", "-f", "../../Dockerfile.agent", "-t", imgTag, "../..")
+	if err != nil {
+		t.Fatalf("docker build failed: %v\noutput:\n%s", err, buildOut)
+	}
+
+	inspectOut, err := dockerCmd(t, 30*time.Second,
+		"inspect", "--format", "{{json .Config.Healthcheck}}", imgTag)
+	if err != nil {
+		t.Fatalf("docker inspect failed: %v\noutput:\n%s", err, inspectOut)
+	}
+
+	trimmed := strings.TrimSpace(inspectOut)
+	if trimmed == "null" || trimmed == "" {
+		t.Fatalf("agent image has no HEALTHCHECK (got %q) — U-2 adjacent fix regressed", inspectOut)
+	}
+
+	var hc struct {
+		Test []string
+	}
+	if err := json.Unmarshal([]byte(trimmed), &hc); err != nil {
+		t.Fatalf("could not parse Healthcheck JSON %q: %v", inspectOut, err)
+	}
+
+	joined := strings.Join(hc.Test, " ")
+	if !strings.Contains(joined, "pgrep") {
+		t.Errorf("agent Healthcheck.Test does not use pgrep (lost the process-presence shape): %v", hc.Test)
+	}
+	if !strings.Contains(joined, "certctl-agent") {
+		t.Errorf("agent Healthcheck.Test does not target the certctl-agent process name: %v", hc.Test)
+	}
+}
+
+// TestPublishedServerImage_HealthcheckTransitionsToHealthy is the
+// runtime-level contract: the built image, when started, must transition
+// to `healthy` within the start-period + 30s observability budget. This
+// is the heavy test — it requires the server to actually start, which
+// in turn requires either a reachable database OR a startup that fails
+// gracefully enough to keep the HEALTHCHECK probe target alive.
+//
+// The container is started with CERTCTL_DATABASE_URL pointing at an
+// unreachable host so the server fails its postgres bring-up — but
+// importantly, fails AFTER the TLS listener has come up, because the
+// HEALTHCHECK probe target is the TLS listener. We don't actually need
+// the database to validate the HEALTHCHECK shape.
+//
+// IMPORTANT: this test is the runtime contract. If you're working on the
+// server's startup ordering and the listener now comes up AFTER the
+// database, this test must adapt — start a sidecar postgres via
+// testcontainers-go (see internal/integration/lifecycle_test.go for the
+// pattern) and connect the certctl-server container to it.
+func TestPublishedServerImage_HealthcheckTransitionsToHealthy(t *testing.T) {
+	if !dockerAvailable(t) {
+		t.Skip("docker not available — skipping runtime HEALTHCHECK test")
+	}
+	if testing.Short() {
+		t.Skip("runtime HEALTHCHECK test takes ~45s; skipping under -short")
+	}
+	t.Skip("runtime probe contract not yet wired to a sidecar postgres; " +
+		"image-spec contract above (TestPublishedServerImage_HealthcheckSpecUsesHTTPS) " +
+		"covers the audit-flagged regression. Re-enable once the integration " +
+		"harness provisions postgres for image-level smoke.")
+}

deploy/test/nginx-entrypoint.sh

@@ -0,0 +1,27 @@
+#!/bin/sh
+# Generate a self-signed placeholder certificate so NGINX can boot
+# before the certctl agent deploys a real certificate.
+# Once the agent deploys, it overwrites these files and reloads NGINX.
+
+CERT_DIR="/etc/nginx/certs"
+mkdir -p "$CERT_DIR"
+
+# Make cert directory world-writable so the certctl-agent container
+# (which shares this volume) can overwrite the placeholder certs.
+chmod 777 "$CERT_DIR"
+
+if [ ! -f "$CERT_DIR/cert.pem" ]; then
+    echo "Generating self-signed placeholder certificate..."
+    apk add --no-cache openssl > /dev/null 2>&1
+    openssl req -x509 -nodes -days 1 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 \
+        -keyout "$CERT_DIR/key.pem" \
+        -out "$CERT_DIR/cert.pem" \
+        -subj "/CN=placeholder.certctl.test" \
+        2>/dev/null
+    # Make placeholder certs writable by the agent container
+    chmod 666 "$CERT_DIR/cert.pem" "$CERT_DIR/key.pem"
+    echo "Placeholder certificate generated."
+fi
+
+# Start NGINX in foreground
+exec nginx -g "daemon off;"

deploy/test/nginx.conf

@@ -0,0 +1,42 @@
+# NGINX configuration for certctl test environment.
+# The agent deploys certificates to /etc/nginx/certs/ and reloads NGINX.
+# On startup, NGINX uses a self-signed placeholder so it can boot before any cert is deployed.
+
+# Generate a self-signed placeholder on container start (see entrypoint in compose).
+# Once the agent deploys a real cert, it overwrites these files and reloads.
+
+events {
+    worker_connections 1024;
+}
+
+http {
+    # HTTP → redirect to HTTPS (optional, for realism)
+    server {
+        listen 80;
+        server_name _;
+        return 301 https://$host$request_uri;
+    }
+
+    # HTTPS server — serves whatever cert the agent has deployed
+    server {
+        listen 443 ssl;
+        server_name _;
+
+        ssl_certificate     /etc/nginx/certs/cert.pem;
+        ssl_certificate_key /etc/nginx/certs/key.pem;
+
+        # Modern TLS settings
+        ssl_protocols TLSv1.2 TLSv1.3;
+        ssl_prefer_server_ciphers off;
+
+        location / {
+            default_type text/plain;
+            return 200 'certctl test environment — NGINX is serving TLS\n';
+        }
+
+        location /health {
+            default_type text/plain;
+            return 200 'ok\n';
+        }
+    }
+}

deploy/test/pebble-config.json

@@ -0,0 +1,16 @@
+{
+  "pebble": {
+    "listenAddress": "0.0.0.0:14000",
+    "managementListenAddress": "0.0.0.0:15000",
+    "certificate": "test/certs/localhost/cert.pem",
+    "privateKey": "test/certs/localhost/key.pem",
+    "httpPort": 80,
+    "tlsPort": 443,
+    "ocspResponderURL": "",
+    "externalAccountBindingRequired": false,
+    "retryAfter": {
+      "authz": 3,
+      "order": 5
+    }
+  }
+}

deploy/test/run-test.sh

@@ -0,0 +1,972 @@
+#!/usr/bin/env bash
+# =============================================================================
+# DEPRECATED — prefer `go test -tags integration ./deploy/test/...`
+# =============================================================================
+#
+# This bash harness predates the Go integration test suite in
+# deploy/test/integration_test.go (build tag `integration`, 34 subtests across
+# 13 phases — health, agent heartbeat, Local CA issuance, ACME, step-ca, EST,
+# S/MIME, discovery, network scan, revocation + CRL, deployment verification).
+# The Go suite uses crypto/x509, crypto/tls, and database/sql to parse certs,
+# probe TLS, and talk to PostgreSQL directly — no openssl text-scraping or
+# brittle curl pipelines. It is the authoritative integration test surface as
+# of milestone M-007 (HTTPS Everywhere, Phase 6), where the test compose
+# stack wires the server on https://localhost:8443 behind a pinned CA bundle
+# at ./certs/ca.crt.
+#
+# Run the Go suite:
+#   (cd deploy && docker compose -f docker-compose.test.yml up -d --build)
+#   go test -tags integration -v -count=1 ./deploy/test/...
+#
+# Keep this bash script around because:
+#   * It is cited in docs/test-env.md and muscle-memory for contributors.
+#   * It exercises the CLI / curl path end-to-end (a different failure mode
+#     than the Go HTTP client path).
+# But any NEW integration coverage goes in integration_test.go — not here.
+#
+# =============================================================================
+# certctl End-to-End Test Script
+# =============================================================================
+#
+# Automates the full lifecycle test from docs/test-env.md:
+#   1. Bring up all 7 containers (build from source)
+#   2. Wait for every service to be healthy
+#   3. Verify pre-seeded data (agents, issuers, targets, profiles)
+#   4. Issue a certificate via Local CA → deploy to NGINX → verify TLS
+#   5. Issue a certificate via ACME/Pebble → verify
+#   6. Issue a certificate via step-ca → verify
+#   7. Test revocation + CRL
+#   8. Test discovery
+#   9. Test renewal (re-issue step-ca cert, check version history)
+#  10. EST enrollment (RFC 7030) — cacerts + simpleenroll
+#  11. S/MIME issuance — emailProtection EKU + adaptive KeyUsage
+#  12. API spot checks + print summary
+#
+# Usage:
+#   cd certctl/deploy
+#   ./test/run-test.sh          # full run (build + test)
+#   ./test/run-test.sh --no-build   # skip docker build, reuse existing containers
+#   ./test/run-test.sh --no-teardown # leave containers running after test
+#
+# Requirements: docker, curl, openssl, jq (or python3 for json parsing)
+# =============================================================================
+
+set -euo pipefail
+
+# ---------------------------------------------------------------------------
+# Config
+# ---------------------------------------------------------------------------
+COMPOSE_FILE="docker-compose.test.yml"
+API_URL="https://localhost:8443"
+API_KEY="test-key-2026"
+NGINX_TLS="localhost:8444"
+AUTH_HEADER="Authorization: Bearer ${API_KEY}"
+CACERT="./certs/ca.crt"
+
+# Flags
+BUILD=true
+TEARDOWN=true
+for arg in "$@"; do
+  case "$arg" in
+    --no-build)    BUILD=false ;;
+    --no-teardown) TEARDOWN=false ;;
+  esac
+done
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+CYAN='\033[0;36m'
+BOLD='\033[1m'
+NC='\033[0m' # No Color
+
+PASS=0
+FAIL=0
+SKIP=0
+
+pass() {
+  PASS=$((PASS + 1))
+  echo -e "  ${GREEN}PASS${NC} $1"
+}
+
+fail() {
+  FAIL=$((FAIL + 1))
+  echo -e "  ${RED}FAIL${NC} $1"
+  if [ -n "${2:-}" ]; then
+    echo -e "       ${RED}$2${NC}"
+  fi
+}
+
+skip() {
+  SKIP=$((SKIP + 1))
+  echo -e "  ${YELLOW}SKIP${NC} $1"
+}
+
+info() {
+  echo -e "${CYAN}==>${NC} $1"
+}
+
+header() {
+  echo ""
+  echo -e "${BOLD}─── $1 ───${NC}"
+}
+
+# API helper: GET endpoint, return JSON body. Exits 1 on HTTP error.
+api_get() {
+  local path="$1"
+  curl -sf --cacert "${CACERT}" -H "${AUTH_HEADER}" "${API_URL}${path}" 2>/dev/null
+}
+
+# API helper: POST with optional JSON body
+api_post() {
+  local path="$1"
+  local body="${2:-}"
+  if [ -n "$body" ]; then
+    curl -sf --cacert "${CACERT}" -X POST -H "${AUTH_HEADER}" -H "Content-Type: application/json" \
+      -d "$body" "${API_URL}${path}" 2>/dev/null
+  else
+    curl -sf --cacert "${CACERT}" -X POST -H "${AUTH_HEADER}" "${API_URL}${path}" 2>/dev/null
+  fi
+}
+
+# Wait for an HTTP endpoint to return 200. Retries with backoff.
+wait_for_http() {
+  local url="$1"
+  local label="$2"
+  local max_wait="${3:-120}"
+  local elapsed=0
+  local interval=3
+
+  while [ $elapsed -lt $max_wait ]; do
+    if curl -sf -H "${AUTH_HEADER}" "$url" >/dev/null 2>&1; then
+      return 0
+    fi
+    sleep $interval
+    elapsed=$((elapsed + interval))
+  done
+  return 1
+}
+
+# Extract a field from JSON using python3 (no jq dependency)
+json_field() {
+  python3 -c "import sys,json; d=json.load(sys.stdin); print($1)" 2>/dev/null
+}
+
+# Wait for a job to reach a terminal state (Completed or Failed)
+# Usage: wait_for_job <cert_id> <max_seconds>
+# Returns 0 if Completed, 1 if Failed/timeout
+wait_for_jobs_done() {
+  local cert_id="$1"
+  local max_wait="${2:-180}"
+  local elapsed=0
+  local interval=5
+
+  while [ $elapsed -lt $max_wait ]; do
+    local jobs_json
+    jobs_json=$(api_get "/api/v1/jobs" 2>/dev/null || echo '{"data":[]}')
+
+    # Check if all jobs for this cert are in terminal state
+    # API returns jobs under "data" key (not "jobs")
+    local pending
+    pending=$(echo "$jobs_json" | python3 -c "
+import sys, json
+data = json.load(sys.stdin)
+jobs = data.get('data') or data.get('jobs') or []
+active = [j for j in jobs if j.get('certificate_id') == '$cert_id'
+          and j.get('status') not in ('Completed', 'Failed', 'Cancelled')]
+print(len(active))
+" 2>/dev/null || echo "99")
+
+    if [ "$pending" = "0" ]; then
+      # Check how many jobs exist and their terminal states
+      local job_counts
+      job_counts=$(echo "$jobs_json" | python3 -c "
+import sys, json
+data = json.load(sys.stdin)
+jobs = data.get('data') or data.get('jobs') or []
+mine = [j for j in jobs if j.get('certificate_id') == '$cert_id']
+completed = len([j for j in mine if j.get('status') == 'Completed'])
+failed = len([j for j in mine if j.get('status') in ('Failed', 'Cancelled')])
+print(f'{len(mine)} {completed} {failed}')
+" 2>/dev/null || echo "0 0 0")
+      local total_jobs completed_jobs failed_jobs
+      total_jobs=$(echo "$job_counts" | cut -d' ' -f1)
+      completed_jobs=$(echo "$job_counts" | cut -d' ' -f2)
+      failed_jobs=$(echo "$job_counts" | cut -d' ' -f3)
+
+      if [ "$completed_jobs" -gt 0 ]; then
+        return 0  # At least one job completed successfully
+      fi
+      if [ "$total_jobs" -gt 0 ] && [ "$failed_jobs" -gt 0 ]; then
+        return 1  # All jobs are in terminal state but none completed — all failed
+      fi
+    fi
+
+    sleep $interval
+    elapsed=$((elapsed + interval))
+  done
+  return 1
+}
+
+# Get the TLS cert subject from NGINX for a given SNI
+get_tls_subject() {
+  local sni="$1"
+  echo | openssl s_client -connect "$NGINX_TLS" -servername "$sni" 2>/dev/null \
+    | openssl x509 -noout -subject 2>/dev/null \
+    | sed 's/subject=//' | sed 's/^ *//'
+}
+
+get_tls_issuer() {
+  local sni="$1"
+  echo | openssl s_client -connect "$NGINX_TLS" -servername "$sni" 2>/dev/null \
+    | openssl x509 -noout -issuer 2>/dev/null \
+    | sed 's/issuer=//' | sed 's/^ *//'
+}
+
+# Get the TLS cert SANs from NGINX for a given SNI
+# Modern CAs (including Let's Encrypt / Pebble) put domains only in SAN, not Subject CN.
+get_tls_san() {
+  local sni="$1"
+  echo | openssl s_client -connect "$NGINX_TLS" -servername "$sni" 2>/dev/null \
+    | openssl x509 -noout -ext subjectAltName 2>/dev/null \
+    | grep -i "DNS:" | sed 's/^ *//'
+}
+
+# Check if NGINX is serving a cert that matches the given domain (checks Subject then SAN)
+check_tls_identity() {
+  local domain="$1"
+  local subject issuer san
+  subject=$(get_tls_subject "$domain")
+  issuer=$(get_tls_issuer "$domain")
+  san=$(get_tls_san "$domain")
+  if echo "$subject" | grep -qi "$domain" || echo "$san" | grep -qi "$domain"; then
+    echo "MATCH"
+    echo "Subject: $subject"
+    echo "SAN: $san"
+    echo "Issuer: $issuer"
+  else
+    echo "NO_MATCH"
+    echo "Subject: $subject"
+    echo "SAN: $san"
+    echo "Issuer: $issuer"
+  fi
+}
+
+# SQL exec in the postgres container
+psql_exec() {
+  docker exec certctl-test-postgres psql -U certctl -d certctl -tAc "$1" 2>/dev/null
+}
+
+# ---------------------------------------------------------------------------
+# Cleanup trap
+# ---------------------------------------------------------------------------
+cleanup() {
+  if [ "$TEARDOWN" = true ]; then
+    info "Tearing down test environment..."
+    docker compose -f "$COMPOSE_FILE" down -v >/dev/null 2>&1 || true
+  else
+    info "Leaving containers running (--no-teardown)"
+  fi
+}
+
+# ---------------------------------------------------------------------------
+# PHASE 0: Environment Check
+# ---------------------------------------------------------------------------
+header "Phase 0: Environment Check"
+
+# Make sure we're in the deploy directory
+if [ ! -f "$COMPOSE_FILE" ]; then
+  echo -e "${RED}ERROR: $COMPOSE_FILE not found.${NC}"
+  echo "Run this script from the certctl/deploy directory:"
+  echo "  cd certctl/deploy && ./test/run-test.sh"
+  exit 1
+fi
+
+for cmd in docker curl openssl python3; do
+  if command -v "$cmd" >/dev/null 2>&1; then
+    pass "$cmd available"
+  else
+    fail "$cmd not found" "Install $cmd and try again"
+    exit 1
+  fi
+done
+
+if docker compose version >/dev/null 2>&1; then
+  pass "docker compose available"
+else
+  fail "docker compose not available" "Install Docker Compose v2+"
+  exit 1
+fi
+
+# ---------------------------------------------------------------------------
+# PHASE 1: Start the Stack
+# ---------------------------------------------------------------------------
+header "Phase 1: Start Test Environment"
+
+# Teardown any previous run
+info "Cleaning up previous test environment..."
+docker compose -f "$COMPOSE_FILE" down -v >/dev/null 2>&1 || true
+
+# Set the cleanup trap AFTER the initial teardown
+trap cleanup EXIT
+
+if [ "$BUILD" = true ]; then
+  info "Building and starting containers (this takes 2-5 minutes on first run)..."
+  docker compose -f "$COMPOSE_FILE" up --build -d 2>&1 | tail -5
+else
+  info "Starting containers (--no-build)..."
+  docker compose -f "$COMPOSE_FILE" up -d 2>&1 | tail -5
+fi
+
+# ---------------------------------------------------------------------------
+# PHASE 2: Wait for Services
+# ---------------------------------------------------------------------------
+header "Phase 2: Waiting for Services"
+
+info "Waiting for PostgreSQL..."
+if docker compose -f "$COMPOSE_FILE" exec -T postgres pg_isready -U certctl -d certctl >/dev/null 2>&1 ||
+   wait_for_http "${API_URL}/health" "postgres" 60; then
+  pass "PostgreSQL ready"
+else
+  fail "PostgreSQL not ready after 60s"
+fi
+
+info "Waiting for certctl server..."
+if wait_for_http "${API_URL}/health" "server" 120; then
+  pass "certctl server healthy"
+  # Show trust setup + connector init for debugging
+  echo "  --- Server startup (trust setup) ---"
+  docker logs certctl-test-server 2>&1 | grep -E "trust|Added|Extract|provisioner|Pre-launch|key file|WARNING|CERTCTL_" | head -15
+  echo "  ---"
+else
+  fail "certctl server not healthy after 120s"
+  echo ""
+  echo "Server logs:"
+  docker logs certctl-test-server --tail 30
+  exit 1
+fi
+
+info "Waiting for NGINX..."
+if wait_for_http "http://localhost:8080" "nginx" 30; then
+  pass "NGINX healthy"
+else
+  # NGINX might not respond to plain curl on /health without the right path
+  # Check docker health instead
+  if docker inspect certctl-test-nginx --format='{{.State.Health.Status}}' 2>/dev/null | grep -q healthy; then
+    pass "NGINX healthy (docker healthcheck)"
+  else
+    skip "NGINX health check inconclusive (will verify via TLS later)"
+  fi
+fi
+
+# Give the agent a few seconds to register and send first heartbeat
+info "Waiting for agent heartbeat (up to 45s)..."
+AGENT_READY=false
+for i in $(seq 1 15); do
+  AGENT_STATUS=$(api_get "/api/v1/agents/agent-test-01" 2>/dev/null | python3 -c "import sys,json; print(json.load(sys.stdin).get('status',''))" 2>/dev/null || echo "")
+  if [ "$AGENT_STATUS" = "online" ]; then
+    AGENT_READY=true
+    break
+  fi
+  sleep 3
+done
+if [ "$AGENT_READY" = true ]; then
+  pass "Agent online"
+else
+  skip "Agent not yet online (may be slow to heartbeat — continuing)"
+fi
+
+# ---------------------------------------------------------------------------
+# PHASE 3: Verify Pre-Seeded Data
+# ---------------------------------------------------------------------------
+header "Phase 3: Verify Pre-Seeded Data"
+
+# Agents
+AGENT_COUNT=$(api_get "/api/v1/agents" | python3 -c "import sys,json; print(json.load(sys.stdin).get('total',0))" 2>/dev/null || echo 0)
+if [ "$AGENT_COUNT" -ge 2 ]; then
+  pass "Agents: $AGENT_COUNT found (agent-test-01 + server-scanner)"
+else
+  fail "Agents: expected >= 2, got $AGENT_COUNT"
+fi
+
+# Issuers
+ISSUER_COUNT=$(api_get "/api/v1/issuers" | python3 -c "import sys,json; print(json.load(sys.stdin).get('total',0))" 2>/dev/null || echo 0)
+if [ "$ISSUER_COUNT" -ge 3 ]; then
+  pass "Issuers: $ISSUER_COUNT found (iss-local, iss-acme-staging, iss-stepca)"
+else
+  fail "Issuers: expected >= 3, got $ISSUER_COUNT" "Check seed_test.sql loaded correctly"
+fi
+
+# Targets
+TARGET_COUNT=$(api_get "/api/v1/targets" | python3 -c "import sys,json; print(json.load(sys.stdin).get('total',0))" 2>/dev/null || echo 0)
+if [ "$TARGET_COUNT" -ge 1 ]; then
+  pass "Targets: $TARGET_COUNT found (target-test-nginx)"
+else
+  fail "Targets: expected >= 1, got $TARGET_COUNT" "seed_test.sql may have failed after iss-local"
+fi
+
+# Profile
+PROFILE_RESP=$(api_get "/api/v1/profiles" 2>/dev/null || echo '{"total":0}')
+PROFILE_COUNT=$(echo "$PROFILE_RESP" | python3 -c "import sys,json; print(json.load(sys.stdin).get('total',0))" 2>/dev/null || echo 0)
+if [ "$PROFILE_COUNT" -ge 2 ]; then
+  pass "Profiles: $PROFILE_COUNT found (prof-test-tls, prof-test-smime)"
+else
+  fail "Profiles: expected >= 1, got $PROFILE_COUNT"
+fi
+
+# Bail if seed data is broken
+if [ "$ISSUER_COUNT" -lt 3 ] || [ "$TARGET_COUNT" -lt 1 ]; then
+  echo ""
+  echo -e "${RED}Seed data is incomplete. Cannot continue.${NC}"
+  echo "Check PostgreSQL logs: docker logs certctl-test-postgres"
+  exit 1
+fi
+
+# ---------------------------------------------------------------------------
+# PHASE 4: Local CA Issuance
+# ---------------------------------------------------------------------------
+header "Phase 4: Local CA Certificate Issuance"
+
+info "Creating certificate record mc-local-test..."
+CREATE_RESP=$(api_post "/api/v1/certificates" '{
+  "id": "mc-local-test",
+  "name": "local-test-cert",
+  "common_name": "local.certctl.test",
+  "sans": ["local.certctl.test"],
+  "issuer_id": "iss-local",
+  "owner_id": "owner-test-admin",
+  "team_id": "team-test-ops",
+  "renewal_policy_id": "rp-default",
+  "certificate_profile_id": "prof-test-tls",
+  "environment": "development"
+}' 2>/dev/null || echo "ERROR")
+
+if echo "$CREATE_RESP" | python3 -c "import sys,json; d=json.load(sys.stdin); assert d.get('id')=='mc-local-test'" 2>/dev/null; then
+  pass "Certificate record created"
+else
+  fail "Certificate creation failed" "$CREATE_RESP"
+fi
+
+info "Linking certificate to NGINX target..."
+psql_exec "INSERT INTO certificate_target_mappings (certificate_id, target_id) VALUES ('mc-local-test', 'target-test-nginx') ON CONFLICT DO NOTHING;"
+pass "Target mapping inserted"
+
+info "Triggering issuance..."
+RENEW_RESP=$(api_post "/api/v1/certificates/mc-local-test/renew" 2>/dev/null || echo "ERROR")
+if echo "$RENEW_RESP" | grep -q "renewal_triggered\|status"; then
+  pass "Issuance triggered"
+else
+  fail "Trigger failed" "$RENEW_RESP"
+fi
+
+# Verify a job was created (this is the bug fix check)
+sleep 2
+JOB_COUNT=$(api_get "/api/v1/jobs" | python3 -c "
+import sys, json
+data = json.load(sys.stdin)
+jobs = [j for j in (data.get('data') or data.get('jobs') or []) if j.get('certificate_id') == 'mc-local-test']
+print(len(jobs))
+" 2>/dev/null || echo "0")
+
+if [ "$JOB_COUNT" -gt 0 ]; then
+  pass "Job created ($JOB_COUNT jobs for mc-local-test)"
+else
+  fail "No jobs created — TriggerRenewalWithActor bug still present"
+fi
+
+info "Waiting for issuance + deployment (up to 180s)..."
+if wait_for_jobs_done "mc-local-test" 180; then
+  pass "All jobs completed"
+else
+  fail "Jobs did not complete within 180s"
+  echo "  Current jobs:"
+  api_get "/api/v1/jobs" 2>/dev/null | python3 -m json.tool 2>/dev/null | head -30
+fi
+
+info "Reloading NGINX to pick up deployed certificate..."
+docker exec certctl-test-nginx nginx -s reload 2>/dev/null || true
+sleep 3
+
+info "Verifying TLS certificate on NGINX..."
+TLS_CHECK=$(check_tls_identity "local.certctl.test")
+TLS_RESULT=$(echo "$TLS_CHECK" | head -1)
+if [ "$TLS_RESULT" = "MATCH" ]; then
+  pass "NGINX serving cert for local.certctl.test"
+  echo "$TLS_CHECK" | tail -n +2 | while read -r line; do echo -e "       $line"; done
+else
+  fail "NGINX not serving expected cert" "$(echo "$TLS_CHECK" | tail -n +2 | tr '\n' ', ')"
+fi
+
+# Check cert status in API
+CERT_STATUS=$(api_get "/api/v1/certificates/mc-local-test" | python3 -c "import sys,json; print(json.load(sys.stdin).get('status',''))" 2>/dev/null || echo "unknown")
+if [ "$CERT_STATUS" = "Active" ]; then
+  pass "Certificate status: Active"
+else
+  skip "Certificate status: $CERT_STATUS (expected Active — may need more time)"
+fi
+
+# ---------------------------------------------------------------------------
+# PHASE 5: ACME (Pebble) Issuance
+# ---------------------------------------------------------------------------
+header "Phase 5: ACME (Pebble) Certificate Issuance"
+
+info "Creating certificate record mc-acme-test..."
+CREATE_RESP=$(api_post "/api/v1/certificates" '{
+  "id": "mc-acme-test",
+  "name": "acme-test-cert",
+  "common_name": "acme.certctl.test",
+  "sans": ["acme.certctl.test"],
+  "issuer_id": "iss-acme-staging",
+  "owner_id": "owner-test-admin",
+  "team_id": "team-test-ops",
+  "renewal_policy_id": "rp-default",
+  "certificate_profile_id": "prof-test-tls",
+  "environment": "staging"
+}' 2>/dev/null || echo "ERROR")
+
+if echo "$CREATE_RESP" | python3 -c "import sys,json; d=json.load(sys.stdin); assert d.get('id')=='mc-acme-test'" 2>/dev/null; then
+  pass "Certificate record created"
+else
+  fail "Certificate creation failed" "$CREATE_RESP"
+fi
+
+info "Linking to target and triggering issuance..."
+psql_exec "INSERT INTO certificate_target_mappings (certificate_id, target_id) VALUES ('mc-acme-test', 'target-test-nginx') ON CONFLICT DO NOTHING;"
+RENEW_RESP=$(api_post "/api/v1/certificates/mc-acme-test/renew" 2>/dev/null || echo "ERROR")
+if echo "$RENEW_RESP" | grep -q "renewal_triggered\|status"; then
+  pass "Issuance triggered"
+else
+  fail "Trigger failed" "$RENEW_RESP"
+fi
+
+info "Waiting for ACME issuance + deployment (up to 180s)..."
+if wait_for_jobs_done "mc-acme-test" 180; then
+  pass "All jobs completed"
+
+  info "Reloading NGINX to pick up deployed certificate..."
+  docker exec certctl-test-nginx nginx -s reload 2>/dev/null || true
+  sleep 3
+
+  TLS_CHECK=$(check_tls_identity "acme.certctl.test")
+  TLS_RESULT=$(echo "$TLS_CHECK" | head -1)
+  if [ "$TLS_RESULT" = "MATCH" ]; then
+    pass "NGINX serving cert for acme.certctl.test"
+    echo "$TLS_CHECK" | tail -n +2 | while read -r line; do echo -e "       $line"; done
+  else
+    fail "NGINX not serving expected ACME cert" "$(echo "$TLS_CHECK" | tail -n +2 | tr '\n' ', ')"
+  fi
+else
+  fail "ACME jobs did not complete within 180s"
+  info "Checking ACME job status..."
+  api_get "/api/v1/jobs" 2>/dev/null | python3 -c "
+import sys, json
+data = json.load(sys.stdin)
+for j in data.get('data', []):
+    if j.get('certificate_id') == 'mc-acme-test':
+        print(f\"  Job {j['id']}: type={j['type']} status={j['status']} error={j.get('last_error','')}\")" 2>/dev/null || true
+  echo "  Server logs (last 20 lines):"
+  docker logs certctl-test-server --tail 20 2>&1 | grep -i "acme\|error\|fail\|CSR" | head -10 || true
+fi
+
+# ---------------------------------------------------------------------------
+# PHASE 6: step-ca Issuance
+# ---------------------------------------------------------------------------
+header "Phase 6: step-ca (Private CA) Certificate Issuance"
+
+info "Creating certificate record mc-stepca-test..."
+CREATE_RESP=$(api_post "/api/v1/certificates" '{
+  "id": "mc-stepca-test",
+  "name": "stepca-test-cert",
+  "common_name": "stepca.certctl.test",
+  "sans": ["stepca.certctl.test"],
+  "issuer_id": "iss-stepca",
+  "owner_id": "owner-test-admin",
+  "team_id": "team-test-ops",
+  "renewal_policy_id": "rp-default",
+  "certificate_profile_id": "prof-test-tls",
+  "environment": "staging"
+}' 2>/dev/null || echo "ERROR")
+
+if echo "$CREATE_RESP" | python3 -c "import sys,json; d=json.load(sys.stdin); assert d.get('id')=='mc-stepca-test'" 2>/dev/null; then
+  pass "Certificate record created"
+else
+  fail "Certificate creation failed" "$CREATE_RESP"
+fi
+
+info "Linking to target and triggering issuance..."
+psql_exec "INSERT INTO certificate_target_mappings (certificate_id, target_id) VALUES ('mc-stepca-test', 'target-test-nginx') ON CONFLICT DO NOTHING;"
+RENEW_RESP=$(api_post "/api/v1/certificates/mc-stepca-test/renew" 2>/dev/null || echo "ERROR")
+if echo "$RENEW_RESP" | grep -q "renewal_triggered\|status"; then
+  pass "Issuance triggered"
+else
+  fail "Trigger failed" "$RENEW_RESP"
+fi
+
+info "Waiting for step-ca issuance + deployment (up to 120s)..."
+if wait_for_jobs_done "mc-stepca-test" 120; then
+  pass "All jobs completed"
+else
+  fail "Jobs did not complete in time"
+  info "Checking step-ca job status..."
+  api_get "/api/v1/jobs" 2>/dev/null | python3 -c "
+import sys, json
+data = json.load(sys.stdin)
+for j in data.get('data', []):
+    if j.get('certificate_id') == 'mc-stepca-test':
+        print(f\"  Job {j['id']}: type={j['type']} status={j['status']} error={j.get('last_error','')}\")" 2>/dev/null || true
+  echo "  Server logs (step-ca related):"
+  docker logs certctl-test-server --tail 30 2>&1 | grep -i "stepca\|step-ca\|provisioner\|jwe\|decrypt\|CSR.*fail\|error" | head -10 || true
+fi
+
+# ---------------------------------------------------------------------------
+# PHASE 7: Revocation
+# ---------------------------------------------------------------------------
+header "Phase 7: Revocation"
+
+info "Revoking mc-local-test (reason: superseded)..."
+REVOKE_RESP=$(api_post "/api/v1/certificates/mc-local-test/revoke" '{"reason": "superseded"}' 2>/dev/null || echo "ERROR")
+if echo "$REVOKE_RESP" | grep -qi "revoked\|status"; then
+  pass "Certificate revoked"
+else
+  fail "Revocation failed" "$REVOKE_RESP"
+fi
+
+info "Checking DER CRL under /.well-known/pki (RFC 5280 §5, RFC 8615)..."
+# The JSON CRL endpoint (`GET /api/v1/crl`) was removed in M-006. RFC 5280
+# defines only the DER wire format, now served unauthenticated at
+# `/.well-known/pki/crl/{issuer_id}`. Fetch without the Bearer header to
+# prove the endpoint is reachable by relying parties with no API key.
+CRL_TMP=$(mktemp)
+CRL_HEADERS=$(mktemp)
+CRL_HTTP_CODE=$(curl -s -o "$CRL_TMP" -D "$CRL_HEADERS" -w "%{http_code}" "${API_URL}/.well-known/pki/crl/iss-local" 2>/dev/null || echo "000")
+CRL_SIZE=$(wc -c < "$CRL_TMP" | tr -d ' ')
+CRL_CONTENT_TYPE=$(awk 'tolower($1)=="content-type:" { sub(/\r$/,"",$2); print tolower($2) }' "$CRL_HEADERS" | head -n1)
+rm -f "$CRL_TMP" "$CRL_HEADERS"
+
+if [ "$CRL_HTTP_CODE" = "200" ] && [ "$CRL_CONTENT_TYPE" = "application/pkix-crl" ] && [ "$CRL_SIZE" -gt 0 ]; then
+  pass "DER CRL served unauthenticated (HTTP 200, Content-Type application/pkix-crl, ${CRL_SIZE} bytes)"
+else
+  fail "DER CRL fetch failed: HTTP=$CRL_HTTP_CODE Content-Type=$CRL_CONTENT_TYPE size=$CRL_SIZE"
+fi
+
+CERT_STATUS=$(api_get "/api/v1/certificates/mc-local-test" | python3 -c "import sys,json; print(json.load(sys.stdin).get('status',''))" 2>/dev/null || echo "unknown")
+if [ "$CERT_STATUS" = "Revoked" ]; then
+  pass "Certificate status updated to Revoked"
+else
+  fail "Certificate status: $CERT_STATUS (expected Revoked)"
+fi
+
+# ---------------------------------------------------------------------------
+# PHASE 8: Discovery
+# ---------------------------------------------------------------------------
+header "Phase 8: Certificate Discovery"
+
+info "Checking discovered certificates..."
+DISC_RESP=$(api_get "/api/v1/discovered-certificates" 2>/dev/null || echo '{"total":0}')
+DISC_TOTAL=$(echo "$DISC_RESP" | python3 -c "import sys,json; print(json.load(sys.stdin).get('total',0))" 2>/dev/null || echo 0)
+if [ "$DISC_TOTAL" -ge 1 ]; then
+  pass "Discovered $DISC_TOTAL certificate(s) on filesystem"
+else
+  skip "No discovered certificates yet (agent scan may not have run)"
+fi
+
+SUMMARY_RESP=$(api_get "/api/v1/discovery-summary" 2>/dev/null || echo '{}')
+echo -e "       Discovery summary: $SUMMARY_RESP"
+
+# ---------------------------------------------------------------------------
+# PHASE 9: Renewal (re-issue ACME cert)
+# ---------------------------------------------------------------------------
+header "Phase 9: Renewal"
+
+# Try mc-stepca-test first (mc-local-test was revoked in Phase 7).
+# Fall back to mc-acme-test if step-ca cert isn't Active.
+RENEWAL_CERT=""
+for candidate in mc-stepca-test mc-acme-test; do
+  STATUS=$(api_get "/api/v1/certificates/$candidate" 2>/dev/null | python3 -c "import sys,json; print(json.load(sys.stdin).get('status',''))" 2>/dev/null || echo "unknown")
+  if [ "$STATUS" = "Active" ]; then
+    RENEWAL_CERT="$candidate"
+    break
+  fi
+done
+
+if [ -z "$RENEWAL_CERT" ]; then
+  skip "Cannot test renewal — no certificate in Active state"
+else
+  info "Using $RENEWAL_CERT for renewal test..."
+  info "Triggering renewal on $RENEWAL_CERT..."
+  RENEW_RESP=$(api_post "/api/v1/certificates/$RENEWAL_CERT/renew" 2>/dev/null || echo "ERROR")
+  if echo "$RENEW_RESP" | grep -q "renewal_triggered\|status"; then
+    pass "Renewal triggered"
+  else
+    skip "Renewal trigger returned: $RENEW_RESP"
+  fi
+
+  info "Waiting for renewal to complete (up to 180s)..."
+  if wait_for_jobs_done "$RENEWAL_CERT" 180; then
+    pass "Renewal jobs completed"
+
+    info "Reloading NGINX to pick up renewed certificate..."
+    docker exec certctl-test-nginx nginx -s reload 2>/dev/null || true
+    sleep 3
+
+    # Verify version history shows multiple versions
+    VERSIONS=$(api_get "/api/v1/certificates/$RENEWAL_CERT/versions" 2>/dev/null | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d) if isinstance(d, list) else d.get('total', 0))" 2>/dev/null || echo 0)
+    if [ "$VERSIONS" -ge 2 ]; then
+      pass "Certificate has $VERSIONS versions (original + renewal)"
+    else
+      skip "Expected 2+ versions, got $VERSIONS"
+    fi
+  else
+    skip "Renewal jobs did not complete within 180s"
+  fi
+fi
+
+# ---------------------------------------------------------------------------
+# PHASE 10: EST Enrollment (RFC 7030)
+# ---------------------------------------------------------------------------
+header "Phase 10: EST Enrollment (RFC 7030)"
+
+# Test cacerts endpoint — should return PKCS#7 with CA cert chain
+info "Testing EST cacerts endpoint..."
+EST_CACERTS_RESP=$(curl -sf -H "${AUTH_HEADER}" "${API_URL}/.well-known/est/cacerts" 2>/dev/null || echo "ERROR")
+if [ "$EST_CACERTS_RESP" != "ERROR" ] && [ -n "$EST_CACERTS_RESP" ]; then
+  # Response should be base64-encoded PKCS#7
+  if echo "$EST_CACERTS_RESP" | base64 -d >/dev/null 2>&1; then
+    pass "EST cacerts returns valid base64 PKCS#7 response"
+  else
+    fail "EST cacerts returned non-base64 data"
+  fi
+else
+  fail "EST cacerts endpoint failed" "$EST_CACERTS_RESP"
+fi
+
+# Test csrattrs endpoint
+info "Testing EST csrattrs endpoint..."
+EST_CSRATTRS_STATUS=$(curl -sf -o /dev/null -w "%{http_code}" -H "${AUTH_HEADER}" "${API_URL}/.well-known/est/csrattrs" 2>/dev/null || echo "000")
+if [ "$EST_CSRATTRS_STATUS" = "200" ] || [ "$EST_CSRATTRS_STATUS" = "204" ]; then
+  pass "EST csrattrs returns $EST_CSRATTRS_STATUS"
+else
+  fail "EST csrattrs returned $EST_CSRATTRS_STATUS (expected 200 or 204)"
+fi
+
+# Test simpleenroll — generate CSR, POST as base64-encoded DER
+info "Testing EST simpleenroll with generated CSR..."
+EST_KEY_FILE=$(mktemp /tmp/est-key-XXXXXX.pem)
+EST_CSR_PEM_FILE=$(mktemp /tmp/est-csr-XXXXXX.pem)
+EST_CSR_DER_FILE=$(mktemp /tmp/est-csr-XXXXXX.der)
+trap "rm -f $EST_KEY_FILE $EST_CSR_PEM_FILE $EST_CSR_DER_FILE" EXIT
+
+# Generate ECDSA key + CSR
+openssl ecparam -genkey -name prime256v1 -noout -out "$EST_KEY_FILE" 2>/dev/null
+openssl req -new -key "$EST_KEY_FILE" -out "$EST_CSR_PEM_FILE" -subj "/CN=est-device.certctl.test" 2>/dev/null
+openssl req -in "$EST_CSR_PEM_FILE" -out "$EST_CSR_DER_FILE" -outform DER 2>/dev/null
+
+# base64-encode the DER CSR (EST wire format)
+EST_CSR_B64=$(base64 < "$EST_CSR_DER_FILE" | tr -d '\n')
+
+EST_ENROLL_RESP=$(curl -sf \
+  -X POST \
+  -H "${AUTH_HEADER}" \
+  -H "Content-Type: application/pkcs10" \
+  -d "$EST_CSR_B64" \
+  "${API_URL}/.well-known/est/simpleenroll" 2>/dev/null || echo "ERROR")
+
+if [ "$EST_ENROLL_RESP" != "ERROR" ] && [ -n "$EST_ENROLL_RESP" ]; then
+  # Response should be base64-encoded PKCS#7 containing the issued cert
+  if echo "$EST_ENROLL_RESP" | base64 -d >/dev/null 2>&1; then
+    pass "EST simpleenroll issued certificate via PKCS#7 response"
+  else
+    fail "EST simpleenroll returned non-base64 data"
+  fi
+else
+  fail "EST simpleenroll failed" "$(curl -s -X POST -H "${AUTH_HEADER}" -H "Content-Type: application/pkcs10" -d "$EST_CSR_B64" "${API_URL}/.well-known/est/simpleenroll" 2>&1 | head -5)"
+fi
+
+# Test simplereenroll (should work identically)
+info "Testing EST simplereenroll..."
+EST_REENROLL_STATUS=$(curl -sf -o /dev/null -w "%{http_code}" \
+  -X POST \
+  -H "${AUTH_HEADER}" \
+  -H "Content-Type: application/pkcs10" \
+  -d "$EST_CSR_B64" \
+  "${API_URL}/.well-known/est/simplereenroll" 2>/dev/null || echo "000")
+
+if [ "$EST_REENROLL_STATUS" = "200" ]; then
+  pass "EST simplereenroll works (status 200)"
+else
+  fail "EST simplereenroll returned $EST_REENROLL_STATUS (expected 200)"
+fi
+
+# ---------------------------------------------------------------------------
+# PHASE 11: S/MIME Certificate Issuance
+# ---------------------------------------------------------------------------
+header "Phase 11: S/MIME Certificate Issuance"
+
+info "Creating S/MIME certificate record..."
+SMIME_RESP=$(api_post "/api/v1/certificates" '{
+  "id": "mc-smime-test",
+  "name": "smime-test-cert",
+  "common_name": "testuser@certctl.test",
+  "sans": ["testuser@certctl.test"],
+  "issuer_id": "iss-local",
+  "owner_id": "owner-test-admin",
+  "team_id": "team-test-ops",
+  "renewal_policy_id": "rp-default",
+  "certificate_profile_id": "prof-test-smime",
+  "environment": "staging"
+}' 2>/dev/null || echo "ERROR")
+
+if echo "$SMIME_RESP" | python3 -c "import sys,json; d=json.load(sys.stdin); assert d.get('id')=='mc-smime-test'" 2>/dev/null; then
+  pass "S/MIME certificate record created"
+else
+  fail "S/MIME certificate creation failed" "$SMIME_RESP"
+fi
+
+info "Linking S/MIME cert to target (needed for agent work routing)..."
+psql_exec "INSERT INTO certificate_target_mappings (certificate_id, target_id) VALUES ('mc-smime-test', 'target-test-nginx') ON CONFLICT DO NOTHING;"
+
+info "Triggering S/MIME issuance..."
+SMIME_RENEW=$(api_post "/api/v1/certificates/mc-smime-test/renew" 2>/dev/null || echo "ERROR")
+if echo "$SMIME_RENEW" | grep -q "renewal_triggered\|status"; then
+  pass "S/MIME issuance triggered"
+else
+  fail "S/MIME trigger failed" "$SMIME_RENEW"
+fi
+
+info "Waiting for S/MIME issuance (up to 120s)..."
+if wait_for_jobs_done "mc-smime-test" 120; then
+  pass "S/MIME jobs completed"
+
+  # Fetch the issued cert and verify EKU
+  info "Verifying S/MIME certificate EKU..."
+  SMIME_VERSIONS=$(api_get "/api/v1/certificates/mc-smime-test/versions" 2>/dev/null || echo "[]")
+  SMIME_PEM=$(echo "$SMIME_VERSIONS" | python3 -c "
+import sys, json
+data = json.load(sys.stdin)
+versions = data if isinstance(data, list) else data.get('data', [])
+if versions:
+    print(versions[-1].get('pem_chain', versions[-1].get('pem', '')))
+" 2>/dev/null || echo "")
+
+  if [ -n "$SMIME_PEM" ]; then
+    # Parse the cert and check for emailProtection EKU
+    SMIME_EKU=$(echo "$SMIME_PEM" | openssl x509 -noout -text 2>/dev/null | grep -A2 "Extended Key Usage" || echo "")
+    if echo "$SMIME_EKU" | grep -qi "emailProtection\|E-mail Protection"; then
+      pass "S/MIME cert has emailProtection EKU"
+    else
+      fail "S/MIME cert missing emailProtection EKU" "Got: $SMIME_EKU"
+    fi
+
+    # Check KeyUsage flags (S/MIME should have Digital Signature + Content Commitment)
+    SMIME_KU=$(echo "$SMIME_PEM" | openssl x509 -noout -text 2>/dev/null | awk '/X509v3 Key Usage:/{getline; print; exit}')
+    if echo "$SMIME_KU" | grep -qi "Digital Signature"; then
+      pass "S/MIME cert has Digital Signature KeyUsage"
+    else
+      fail "S/MIME cert missing Digital Signature KeyUsage" "Got: $SMIME_KU"
+    fi
+
+    # Check that email SAN is present
+    SMIME_SAN=$(echo "$SMIME_PEM" | openssl x509 -noout -ext subjectAltName 2>/dev/null || echo "")
+    if echo "$SMIME_SAN" | grep -qi "email:testuser@certctl.test"; then
+      pass "S/MIME cert has email SAN"
+    else
+      # Some implementations use rfc822Name instead of email:
+      if echo "$SMIME_SAN" | grep -qi "testuser@certctl.test"; then
+        pass "S/MIME cert has email SAN (rfc822Name)"
+      else
+        skip "S/MIME email SAN not found in cert (may be in CN only)"
+        echo "       SAN content: $SMIME_SAN"
+      fi
+    fi
+  else
+    skip "Could not extract S/MIME cert PEM for EKU verification"
+  fi
+else
+  fail "S/MIME issuance did not complete within 120s"
+  info "Checking S/MIME job status..."
+  api_get "/api/v1/jobs" 2>/dev/null | python3 -c "
+import sys, json
+data = json.load(sys.stdin)
+for j in data.get('data', []):
+    if j.get('certificate_id') == 'mc-smime-test':
+        print(f\"  Job {j['id']}: type={j['type']} status={j['status']} error={j.get('last_error','')}\")" 2>/dev/null || true
+fi
+
+# ---------------------------------------------------------------------------
+# PHASE 12: API Spot Checks
+# ---------------------------------------------------------------------------
+header "Phase 12: API Spot Checks"
+
+# Health
+if api_get "/health" >/dev/null 2>&1; then
+  pass "GET /health returns 200"
+else
+  fail "GET /health failed"
+fi
+
+# Metrics
+METRICS_RESP=$(api_get "/api/v1/metrics" 2>/dev/null || echo "ERROR")
+if echo "$METRICS_RESP" | python3 -c "import sys,json; d=json.load(sys.stdin); assert 'gauge' in d" 2>/dev/null; then
+  pass "GET /api/v1/metrics returns valid JSON"
+else
+  fail "Metrics endpoint broken"
+fi
+
+# Stats summary
+STATS_RESP=$(api_get "/api/v1/stats/summary" 2>/dev/null || echo "ERROR")
+if echo "$STATS_RESP" | python3 -c "import sys,json; json.load(sys.stdin)" 2>/dev/null; then
+  pass "GET /api/v1/stats/summary returns valid JSON"
+else
+  fail "Stats summary endpoint broken"
+fi
+
+# Audit trail
+AUDIT_RESP=$(api_get "/api/v1/audit" 2>/dev/null || echo '{"total":0}')
+AUDIT_TOTAL=$(echo "$AUDIT_RESP" | python3 -c "import sys,json; print(json.load(sys.stdin).get('total',0))" 2>/dev/null || echo 0)
+if [ "$AUDIT_TOTAL" -gt 0 ]; then
+  pass "Audit trail: $AUDIT_TOTAL events recorded"
+else
+  fail "Audit trail empty"
+fi
+
+# Jobs summary
+JOBS_RESP=$(api_get "/api/v1/jobs" 2>/dev/null || echo '{"total":0}')
+JOBS_TOTAL=$(echo "$JOBS_RESP" | python3 -c "import sys,json; print(json.load(sys.stdin).get('total',0))" 2>/dev/null || echo 0)
+pass "Total jobs created: $JOBS_TOTAL"
+
+# Prometheus
+PROM_RESP=$(curl -sf -H "${AUTH_HEADER}" "${API_URL}/api/v1/metrics/prometheus" 2>/dev/null || echo "")
+if echo "$PROM_RESP" | grep -q "certctl_certificate_total"; then
+  pass "Prometheus metrics endpoint working"
+else
+  fail "Prometheus metrics endpoint broken"
+fi
+
+# ---------------------------------------------------------------------------
+# Summary
+# ---------------------------------------------------------------------------
+header "Test Summary"
+
+TOTAL=$((PASS + FAIL + SKIP))
+echo ""
+echo -e "  ${GREEN}Passed: $PASS${NC}"
+echo -e "  ${RED}Failed: $FAIL${NC}"
+echo -e "  ${YELLOW}Skipped: $SKIP${NC}"
+echo -e "  Total:  $TOTAL"
+echo ""
+
+if [ "$FAIL" -eq 0 ]; then
+  echo -e "${GREEN}${BOLD}All tests passed.${NC}"
+  exit 0
+else
+  echo -e "${RED}${BOLD}$FAIL test(s) failed.${NC}"
+  echo ""
+  echo "Useful debug commands:"
+  echo "  docker logs certctl-test-server --tail 50"
+  echo "  docker logs certctl-test-agent --tail 50"
+  echo "  docker compose -f $COMPOSE_FILE ps"
+  exit 1
+fi

deploy/test/setup-trust.sh

@@ -0,0 +1,140 @@
+#!/bin/sh
+# This script runs inside the certctl-server container at startup.
+# It fetches CA certificates from Pebble and step-ca, adds them to the
+# system trust store, then starts the certctl server.
+#
+# Why: The ACME connector and step-ca connector use Go's default http.Client
+# with no InsecureSkipVerify. They rely on the system trust store to verify
+# TLS connections. Pebble and step-ca both use self-signed root CAs that
+# aren't in Alpine's default CA bundle, so we must add them manually.
+#
+# This script runs as root (user: "0:0" in docker-compose) so that
+# update-ca-certificates can write to /etc/ssl/certs/.
+
+set -e
+
+echo "=== certctl trust store setup ==="
+
+# --- Pebble CA cert (fetched from management API) ---
+# Pebble's management API serves the root CA at /roots/0.
+# We use -k because we can't verify Pebble's TLS cert yet (chicken-and-egg).
+echo "Fetching Pebble root CA from management API..."
+PEBBLE_CA=""
+for i in 1 2 3 4 5 6 7 8 9 10; do
+    if PEBBLE_CA=$(curl -sk https://pebble:15000/roots/0 2>/dev/null); then
+        if [ -n "$PEBBLE_CA" ]; then
+            echo "$PEBBLE_CA" > /usr/local/share/ca-certificates/pebble-ca.crt
+            echo "  Added: Pebble test CA"
+            break
+        fi
+    fi
+    echo "  Waiting for Pebble (attempt $i/10)..."
+    sleep 2
+done
+
+if [ -z "$PEBBLE_CA" ]; then
+    echo "  WARNING: Could not fetch Pebble CA. ACME issuance will fail."
+fi
+
+# --- step-ca root cert (from shared volume) ---
+# The step-ca container writes its root CA to /home/step/certs/root_ca.crt.
+# We mount the step-ca data volume at /stepca-data inside this container.
+STEPCA_ROOT="/stepca-data/certs/root_ca.crt"
+echo "Waiting for step-ca root cert..."
+for i in 1 2 3 4 5 6 7 8 9 10; do
+    if [ -f "$STEPCA_ROOT" ]; then
+        cp "$STEPCA_ROOT" /usr/local/share/ca-certificates/step-ca-root.crt
+        echo "  Added: step-ca root CA"
+        break
+    fi
+    echo "  Waiting for step-ca root cert (attempt $i/10)..."
+    sleep 2
+done
+
+if [ ! -f "$STEPCA_ROOT" ]; then
+    echo "  WARNING: step-ca root cert not found at $STEPCA_ROOT"
+    echo "  step-ca issuance may fail until the cert is available."
+fi
+
+# --- step-ca provisioner key (extracted from ca.json) ---
+# When step-ca auto-bootstraps via DOCKER_STEPCA_INIT_* env vars, the
+# encrypted provisioner key (JWE) is NOT written as a separate file.
+# Instead, it's embedded in ca.json under:
+#   authority.provisioners[0].encryptedKey
+# We extract it here and write to /tmp so the certctl server can read it.
+# The stepca_data volume is mounted :ro, so we can't write there.
+STEPCA_CA_JSON="/stepca-data/config/ca.json"
+STEPCA_KEY_EXTRACTED="/tmp/step-ca-provisioner-key"
+echo "Extracting step-ca provisioner key from ca.json..."
+for i in 1 2 3 4 5 6 7 8 9 10; do
+    if [ -f "$STEPCA_CA_JSON" ]; then
+        # Extract the encryptedKey value using grep+sed (no jq in Alpine base)
+        # The field looks like: "encryptedKey": "eyJhbGciOi..."
+        ENCRYPTED_KEY=$(grep -o '"encryptedKey":"[^"]*"' "$STEPCA_CA_JSON" | head -1 | sed 's/"encryptedKey":"//;s/"$//')
+        if [ -z "$ENCRYPTED_KEY" ]; then
+            # Try with spaces around colon (JSON formatting varies)
+            ENCRYPTED_KEY=$(grep -o '"encryptedKey" *: *"[^"]*"' "$STEPCA_CA_JSON" | head -1 | sed 's/"encryptedKey" *: *"//;s/"$//')
+        fi
+        if [ -n "$ENCRYPTED_KEY" ]; then
+            # Check if it's JWE compact serialization (dot-separated) or JSON serialization
+            case "$ENCRYPTED_KEY" in
+                \{*)
+                    # Already JSON serialization — write as-is
+                    echo "$ENCRYPTED_KEY" > "$STEPCA_KEY_EXTRACTED"
+                    ;;
+                *)
+                    # JWE compact serialization: header.encrypted_key.iv.ciphertext.tag
+                    # Convert to JSON serialization expected by Go decryptProvisionerKey()
+                    JWE_PROTECTED=$(echo "$ENCRYPTED_KEY" | cut -d. -f1)
+                    JWE_ENCKEY=$(echo "$ENCRYPTED_KEY" | cut -d. -f2)
+                    JWE_IV=$(echo "$ENCRYPTED_KEY" | cut -d. -f3)
+                    JWE_CT=$(echo "$ENCRYPTED_KEY" | cut -d. -f4)
+                    JWE_TAG=$(echo "$ENCRYPTED_KEY" | cut -d. -f5)
+                    printf '{"protected":"%s","encrypted_key":"%s","iv":"%s","ciphertext":"%s","tag":"%s"}' \
+                        "$JWE_PROTECTED" "$JWE_ENCKEY" "$JWE_IV" "$JWE_CT" "$JWE_TAG" > "$STEPCA_KEY_EXTRACTED"
+                    ;;
+            esac
+            echo "  Extracted provisioner key to $STEPCA_KEY_EXTRACTED"
+            echo "  Key file size: $(wc -c < "$STEPCA_KEY_EXTRACTED") bytes"
+            echo "  Key starts with: $(head -c 40 "$STEPCA_KEY_EXTRACTED")..."
+            # Override the env var so the server reads from the extracted file
+            export CERTCTL_STEPCA_KEY_PATH="$STEPCA_KEY_EXTRACTED"
+            break
+        else
+            echo "  ca.json found but encryptedKey not found in it (attempt $i/10)"
+        fi
+    else
+        echo "  Waiting for step-ca ca.json (attempt $i/10)..."
+    fi
+    sleep 2
+done
+
+if [ ! -f "$STEPCA_KEY_EXTRACTED" ]; then
+    echo "  WARNING: Could not extract step-ca provisioner key"
+    echo "  Listing /stepca-data/config/ for debugging:"
+    ls -la /stepca-data/config/ 2>/dev/null || echo "  /stepca-data/config/ does not exist"
+    echo "  step-ca issuance will fail."
+fi
+
+# --- Update system trust store ---
+echo "Updating system CA trust store..."
+update-ca-certificates 2>/dev/null || true
+
+echo "Trust store updated."
+
+# --- Debug: verify configuration before starting server ---
+echo "=== Pre-launch verification ==="
+echo "  CERTCTL_STEPCA_KEY_PATH=$CERTCTL_STEPCA_KEY_PATH"
+if [ -f "$CERTCTL_STEPCA_KEY_PATH" ]; then
+    echo "  step-ca key file exists ($(wc -c < "$CERTCTL_STEPCA_KEY_PATH") bytes)"
+    echo "  step-ca key preview: $(head -c 60 "$CERTCTL_STEPCA_KEY_PATH")..."
+else
+    echo "  WARNING: step-ca key file NOT FOUND at $CERTCTL_STEPCA_KEY_PATH"
+fi
+echo "  CERTCTL_ACME_DIRECTORY_URL=$CERTCTL_ACME_DIRECTORY_URL"
+echo "  CERTCTL_ACME_INSECURE=$CERTCTL_ACME_INSECURE"
+echo "  Pebble CA cert: $(ls -la /usr/local/share/ca-certificates/pebble-ca.crt 2>/dev/null || echo 'NOT FOUND')"
+echo "  step-ca root cert: $(ls -la /usr/local/share/ca-certificates/step-ca-root.crt 2>/dev/null || echo 'NOT FOUND')"
+echo "  System CA count: $(ls /etc/ssl/certs/*.pem 2>/dev/null | wc -l) PEM files"
+echo "=== Starting certctl server ==="
+exec /app/server

docs/architecture.md

@@ -45,7 +45,7 @@ New to certificates? Read the [Concepts Guide](concepts.md) first.
 ### Design Principles
 
 1. **Private Key Isolation** — Agents generate ECDSA P-256 keys locally and submit CSRs only. Private keys never touch the control plane. Server-side keygen available via `CERTCTL_KEYGEN_MODE=server` for demo only.
-2. **Pull-Only Deployment** — The server never initiates outbound connections to agents or targets. Agents poll for work. For network appliances and agentless targets, a proxy agent in the same network zone executes deployments via the target's API. This keeps the control plane firewalled off and limits credential scope to the proxy agent's zone.
+2. **Pull-Only Deployment** — The server never initiates outbound connections to agents or targets. Agents poll for work and receive only jobs assigned to their targets (routed via `agent_id` on jobs or through target→agent relationships). For network appliances and agentless targets, a proxy agent in the same network zone executes deployments via the target's API. This keeps the control plane firewalled off and limits credential scope to the proxy agent's zone.
 3. **Sub-CA Capable** — The Local CA can operate as a subordinate CA under an enterprise root (e.g., ADCS). Load a pre-signed CA cert+key from disk and all issued certs chain to the enterprise trust hierarchy. Self-signed mode remains the default for development/demos.
 4. **GUI as Primary Interface** — The web dashboard is the operational control plane, not a secondary viewer. Every backend feature ships with its corresponding GUI surface.
 5. **Decoupled Operations** — Agents operate autonomously; the control plane coordinates but doesn't block agent function
@@ -61,12 +61,12 @@ flowchart TB
         API["REST API\n(Go net/http, :8443)"]
         SVC["Service Layer"]
         REPO["Repository Layer\n(database/sql + lib/pq)"]
-        SCHED["Background Scheduler\n6 loops"]
+        SCHED["Background Scheduler\n8 always-on + 4 optional loops"]
         DASH["Web Dashboard\n(React SPA)"]
     end
 
     subgraph "Data Store"
-        PG[("PostgreSQL 16\n21 tables\nTEXT primary keys")]
+        PG[("PostgreSQL 16\nTEXT primary keys")]
     end
 
     subgraph "Agent Fleet"
@@ -80,15 +80,30 @@ flowchart TB
         CA2["ACME\n(HTTP-01 + DNS-01 + DNS-PERSIST-01)\n(EAB, ZeroSSL auto-EAB)"]
         CA3["step-ca\n(/sign API)"]
         CA4["OpenSSL / Custom CA\n(script-based)"]
-        CA6["Vault PKI\n(planned)"]
+        CA6["Vault PKI\n(token auth, /sign API)"]
+        CA7["DigiCert CertCentral\n(async order model)"]
+        CA8["Sectigo SCM\n(async order model)"]
+        CA9["Google CAS\n(OAuth2, sync)"]
+        CA10["AWS ACM PCA\n(sync issuance)"]
+        CA11["Entrust\n(mTLS, sync/async)"]
+        CA12["GlobalSign Atlas\n(mTLS + API key)"]
+        CA13["EJBCA\n(mTLS or OAuth2)"]
     end
 
     subgraph "Target Systems"
         T1["NGINX\n(file write + reload)"]
         T4["Apache httpd\n(file write + reload)"]
         T5["HAProxy\n(combined PEM + reload)"]
-        T2["F5 BIG-IP\n(proxy agent + iControl REST, planned)"]
-        T3["IIS\n(agent-local PowerShell, planned)"]
+        T6["Traefik\n(file provider)"]
+        T7["Caddy\n(admin API / file)"]
+        T8["Envoy\n(file-based SDS)"]
+        T9["Postfix/Dovecot\n(file + service reload)"]
+        T2["F5 BIG-IP\n(proxy agent + iControl REST)"]
+        T3["IIS\n(WinRM + local)"]
+        T10["SSH\n(SFTP + reload)"]
+        T11["WinCertStore\n(PowerShell import)"]
+        T12["Java Keystore\n(keytool pipeline)"]
+        T13["Kubernetes Secrets\n(K8s API)"]
     end
 
     DASH --> API
@@ -96,7 +111,7 @@ flowchart TB
     SVC --> REPO
     REPO --> PG
     SCHED --> SVC
-    SVC -->|"Issue/Renew"| CA1 & CA2 & CA3
+    SVC -->|"Issue/Renew"| CA1 & CA2 & CA3 & CA4 & CA6 & CA7 & CA8 & CA9 & CA10
 
     A1 & A2 & A3 -->|"CSR + Heartbeat"| API
     API -->|"Cert + Chain\n(NO private key)"| A1 & A2 & A3
@@ -116,7 +131,7 @@ The server exposes a REST API under `/api/v1/` and optionally serves the web das
 
 ### Agents
 
-Lightweight Go processes that run on or near your infrastructure. Agents generate ECDSA P-256 private keys locally, create CSRs, and submit them to the control plane for signing — private keys never leave agent infrastructure. Agents also handle certificate deployment to target systems (NGINX, Apache httpd, HAProxy fully implemented; F5 BIG-IP, IIS interface only with V2 implementations planned) and report job status. They communicate with the control plane via HTTP and authenticate with API keys.
+Lightweight Go processes that run on or near your infrastructure. Agents generate ECDSA P-256 private keys locally, create CSRs, and submit them to the control plane for signing — private keys never leave agent infrastructure. Agents also handle certificate deployment to target systems (NGINX, Apache httpd, HAProxy, Traefik, Caddy, Envoy, Postfix, Dovecot, IIS, F5 BIG-IP, SSH, Windows Certificate Store, Java Keystore, Kubernetes Secrets) and report job status. They communicate with the control plane via HTTP and authenticate with API keys.
 
 The agent runs two background loops: a heartbeat (every 60 seconds) to signal it's alive, and a work poll (every 30 seconds) to check for actionable jobs via `GET /api/v1/agents/{id}/work`. Jobs may be `AwaitingCSR` (agent needs to generate key + submit CSR) or `Deployment` (agent needs to deploy a certificate). Private keys are stored in `CERTCTL_KEY_DIR` (default `/var/lib/certctl/keys`) with 0600 permissions.
 
@@ -124,11 +139,23 @@ The agent runs two background loops: a heartbeat (every 60 seconds) to signal it
 
 **Agent groups (M11b):** Dynamic device grouping allows organizing agents by metadata criteria. Agent groups can match by OS, architecture, IP CIDR, and version. Groups support both dynamic matching (agents automatically join when criteria match) and manual membership (explicit include/exclude). Renewal policies can be scoped to agent groups via the `agent_group_id` foreign key. The GUI provides full CRUD management for agent groups with visual match criteria badges.
 
+**Agent soft-retirement (I-004):** `DELETE /api/v1/agents/{id}` is a soft-delete surface — the row is never removed. Retirement stamps `agents.retired_at` (TIMESTAMPTZ) and `agents.retired_reason` (TEXT) and flips the operational status to `Offline`. Default listings (`GET /api/v1/agents`, the dashboard stats counter, and the stale-offline sweeper) filter retired rows out via `AgentRepository.ListActive`; retired rows are surfaced only through the opt-in `GET /api/v1/agents/retired` view. The endpoint follows a preflight → block → escape-hatch contract:
+
+- **Clean retire** (no active dependencies) — `200 OK` with `RetireAgentResponse` (`cascade=false`, zero counts).
+- **Blocked by active dependencies** — `409 Conflict` with `BlockedByDependenciesResponse`. The three counts (`active_targets`, `active_certificates`, `pending_jobs`) tell the operator exactly which rows would be orphaned. The schema diverges from `ErrorResponse` because downstream dashboards parse the stable three-key shape.
+- **Force cascade** — `DELETE /api/v1/agents/{id}?force=true&reason=...`. `reason` is required (400 otherwise). Transactionally soft-retires downstream `deployment_targets`, cancels pending jobs, and soft-retires the agent, emitting an `agent_retirement_cascaded` audit event with actor + reason + per-bucket counts.
+- **Idempotent re-retire** — a retire attempt against an already-retired agent returns `204 No Content` with an empty body (no second audit event, no response shape — callers that POST again on a retry get a clean no-op).
+- **Sentinel refusal** — the four sentinel agent IDs (`server-scanner`, `cloud-aws-sm`, `cloud-azure-kv`, `cloud-gcp-sm`) back non-agent discovery subsystems (the network scanner and the three cloud secret-manager sources). They are refused unconditionally — even with `force=true` — via `ErrAgentIsSentinel` → `403 Forbidden`. The ID list lives in `internal/domain/connector.go` (`SentinelAgentIDs`) so handler, repository, and scheduler code can filter them without importing `service`.
+
+Retired agents receive `410 Gone` on subsequent heartbeats (`service.ErrAgentRetired`). `cmd/agent` treats 410 as a terminal signal and exits cleanly so retired agents stop phoning home. Migration `000015` flipped `deployment_targets.agent_id` from `ON DELETE CASCADE` to `ON DELETE RESTRICT`, making the old hard-delete path a schema error and forcing all retirement through this contract.
+
+**Registration is by-design pull-only (C-1 closure, cat-b-6177f36636fb).** Agents register themselves at first heartbeat via `install-agent.sh` + `cmd/agent/main.go` — never via the GUI. The `web/src/api/client.ts::registerAgent` client function is intentionally orphan in the dashboard for this reason. It's preserved in `client.ts` (rather than deleted) so future features that want to drive registration from the GUI — for example, a one-click "register proxy agent" panel for network-appliance topologies where the agent runs in a different network zone from the device it manages — can reach the endpoint without a `client.ts` edit. Operators looking to scale agent enrollment use `install-agent.sh` against a config-management system (Ansible, Salt, Puppet) or a baked-in cloud-init script, not the dashboard.
+
 ### Web Dashboard
 
 The web dashboard is the primary operational interface for certctl. It is built with Vite + React + TypeScript and uses TanStack Query for server state management (caching, background refetching, optimistic updates).
 
-**Current views** (21 pages): certificate inventory (list with multi-select bulk operations + "New Certificate" creation modal + detail with deployment status timeline, inline policy/profile editor, version history, deploy, revoke, archive, and trigger renewal actions), agent fleet (list + detail with system info + OS/architecture grouping with charts), job queue (status, retry, cancel, approve/reject for AwaitingApproval jobs), notification inbox (threshold alert grouping, mark-as-read), audit trail (time range, actor, action filters + CSV/JSON export), policy management (rules with enable/disable toggle + delete + violations), issuers (list with test connection + delete), targets (list with 3-step configuration wizard + delete), owners (list with team resolution + delete), teams (list with delete), agent groups (list with dynamic match criteria badges + enable/disable + delete), certificate profiles (list with crypto constraints), short-lived credentials dashboard (TTL countdown, profile filtering, auto-refresh), discovered certificates triage (claim/dismiss unmanaged certs discovered by agents or network scans), network scan targets management (CRUD for network scan targets + Scan Now button), summary dashboard with charts (expiration heatmap, renewal success rate, status distribution, issuance rate), and login page.
+**Current views** (24 pages): certificate inventory (list with multi-select bulk operations + "New Certificate" creation modal + detail with deployment status timeline, inline policy/profile editor, version history, deploy, revoke, archive, and trigger renewal actions), agent fleet (list + detail with system info + OS/architecture grouping with charts), job queue (list + detail with verification section, timeline, audit events; approve/reject for AwaitingApproval jobs), notification inbox (threshold alert grouping, mark-as-read), audit trail (time range, actor, action filters + CSV/JSON export), policy management (rules with enable/disable toggle + delete + violations), issuers (catalog with 10 type cards + 3-step create wizard + detail with test connection), targets (list with 3-step configuration wizard + detail with deployment history), owners (list with team resolution + delete), teams (list with delete), agent groups (list with dynamic match criteria badges + enable/disable + delete), certificate profiles (list with crypto constraints), short-lived credentials dashboard (TTL countdown, profile filtering, auto-refresh), discovered certificates triage (claim/dismiss unmanaged certs discovered by agents or network scans), network scan targets management (CRUD + Scan Now button), summary dashboard with charts (expiration heatmap, renewal success rate, status distribution, issuance rate), digest preview and send, observability (health, metrics, Prometheus config), and login page.
 
 The dashboard includes an **ErrorBoundary component** for graceful error recovery — if a view crashes, the boundary catches the error and displays a user-friendly message instead of breaking the entire dashboard. It also includes a **demo mode** that activates when the API is unreachable — it renders realistic mock data for screenshots and offline presentations.
 
@@ -138,6 +165,10 @@ The dashboard includes an **ErrorBoundary component** for graceful error recover
 - Light content area with branded dark teal sidebar, Inter + JetBrains Mono typography
 - SSE/WebSocket planned for real-time job status updates
 
+**Backend ↔ frontend round-trip rule (B-1 closure):** every backend CRUD operation must have at least one GUI consumer in `web/src/pages/`. Shipping a handler + repository method + OpenAPI operation + `client.ts` fetcher with no page that calls it leaves operators forced to `psql` directly — defeats the "every backend feature ships with its GUI surface" invariant and creates a destructive workflow when the missing path is `update*` (operators delete-and-recreate, losing FK history and audit-trail continuity). The CI guardrail in `.github/workflows/ci.yml` (`Forbidden orphan-CRUD client function regression guard (B-1)`) enforces this for the eight previously-orphan functions (`updateOwner`/`updateTeam`/`updateAgentGroup`/`updateIssuer`/`updateProfile` + `createRenewalPolicy`/`updateRenewalPolicy`/`deleteRenewalPolicy`); apply the same rule when adding any new write endpoint. If a fetcher is needed in `client.ts` before its consumer page exists, leave a TODO referencing this rule and ship them in the same commit.
+
+**TS ↔ Go type contract rule (D-1 + D-2 closure):** every TypeScript interface in `web/src/api/types.ts` must field-match the Go-side `internal/domain/*.go` struct's JSON-emitted shape exactly. Phantom fields (declared on TS, never emitted by Go) silently render `'—'` and lull consumers into thinking a value will arrive that never does; missing fields (emitted by Go, absent from TS) force `(x as any).X` escapes that lose type-checking. Both failure modes are blocked by the CI guardrail in `.github/workflows/ci.yml` (`Forbidden StatusBadge dead-key + TS phantom-field regression guard (D-1 + D-2)`) which awk-windows each interface and grep-fails the build on phantom-field reintroduction — currently covers Certificate (D-1), Agent / Issuer / Notification (D-2). Apply the same rule when adding any new on-wire type: the Go-side json tag is the contract, the TS interface adapts to it, and a literal-construction Vitest in `web/src/api/types.test.ts` pins the post-add shape. Stricter side wins: when in doubt, the side that actually emits the field is the contract; never propose adding a phantom on Go to match a TS over-declaration.
+
 ### PostgreSQL Database
 
 All state is stored in PostgreSQL 16. The schema uses TEXT primary keys (not UUIDs) with human-readable prefixed IDs like `mc-api-prod`, `t-platform`, `o-alice`.
@@ -260,6 +291,9 @@ erDiagram
         text channel
         text recipient
         text status
+        int retry_count
+        timestamptz next_retry_at
+        text last_error
     }
     certificate_profiles {
         text id PK
@@ -320,7 +354,12 @@ erDiagram
     }
 ```
 
-Migrations are idempotent (`IF NOT EXISTS` on all CREATE statements, `ON CONFLICT (id) DO NOTHING` on all seed data) so they're safe to run multiple times — important for Docker Compose where both initdb and the server may run the same SQL.
+The ER diagram above documents **database shape**, not REST-API wire shape. Several columns are intentionally server-internal and never serialized to clients:
+
+- `agents.api_key_hash` — SHA-256 of the agent's plaintext API key, populated by `service.RegisterAgent` (`hashAPIKey(apiKey)` at `internal/service/agent.go`) and consumed by `repository.AgentRepository::GetByAPIKey` for the auth-lookup. **Not** exposed via the REST API, **not** echoed via CLI / MCP / agent registration response, **never** logged. Enforced by `internal/domain/connector.go::Agent.MarshalJSON` (G-2 audit closure, `cat-s5-apikey_leak`); the OpenAPI Agent schema explicitly excludes the field, the frontend `Agent` interface omits it, and a CI grep guardrail at `.github/workflows/ci.yml` blocks reintroduction.
+- `issuers.config` / `deployment_targets.config` — plaintext jsonb shadow of the AES-GCM-encrypted on-disk blob; the encrypted form lives on `EncryptedConfig []byte` (Go-only field tagged `json:"-"`).
+
+Migrations are idempotent (`IF NOT EXISTS` on all CREATE statements, `ON CONFLICT (id) DO NOTHING` on all seed data) so they're safe to run multiple times. Pre-U-3 (`cat-u-seed_initdb_schema_drift`, GitHub #10) the deploy compose stack mounted both a hand-curated subset of `migrations/*.up.sql` and `seed.sql` into postgres `/docker-entrypoint-initdb.d/` so initdb applied them on first boot, *and* the server re-applied the same files via `RunMigrations` on every start. The dual source of truth was the bug: every time a migration shipped that the seed depended on (e.g., 000013 added `policy_rules.severity`), the mount list had to be updated by hand, and missing the update crashed initdb on first boot. Post-U-3 the server is the single source of truth: postgres comes up with an empty schema, `RunMigrations` applies the entire ladder, then `RunSeed` lands the baseline seed (and `RunDemoSeed` lands the demo overlay when `CERTCTL_DEMO_SEED=true`). Helm has used this pattern since day one (postgres-init `emptyDir`); the docker-compose deploy now matches.
 
 ## Data Flow: Certificate Lifecycle
 
@@ -381,7 +420,11 @@ sequenceDiagram
     Note over A: Agent deploys using locally-held private key
 ```
 
-**Profile enforcement:** If the certificate is assigned to a profile (`certificate_profile_id`), the profile's `allowed_key_algorithms` and `max_validity_days` constraints are checked during CSR validation. A CSR with a disallowed key type or a validity period exceeding the profile maximum is rejected before reaching the issuer connector.
+**Profile enforcement (M11c):** Crypto policy enforcement is wired into all four issuance paths: renewal (server-side and agent CSR), agent fallback CSR signing, EST enrollment (RFC 7030), and SCEP enrollment (RFC 8894). At each path, the service layer resolves the certificate's profile and calls `ValidateCSRAgainstProfile()` to check the CSR key algorithm and minimum key size against the profile's `allowed_key_algorithms` rules. A CSR with a disallowed key type or insufficient key size is rejected before reaching the issuer connector.
+
+**MaxTTL enforcement:** When a profile specifies `max_ttl_seconds`, the value is forwarded through the service-layer `IssuerConnector` interface to the connector layer via `MaxTTLSeconds` on `IssuanceRequest` and `RenewalRequest`. Each issuer connector enforces the cap according to its capabilities: the Local CA caps `NotAfter` directly, Vault overrides its TTL string, step-ca caps `NotAfter` with zero-value handling, and OpenSSL logs an advisory warning (script-based signing can't enforce server-side). For CAs that control validity themselves (ACME, DigiCert, Sectigo, Google CAS, AWS ACM PCA), MaxTTLSeconds passes through but the CA makes the final decision.
+
+**Key metadata persistence:** Certificate versions record `key_algorithm` and `key_size` extracted from the CSR during issuance. This metadata enables post-hoc auditing — operators can verify that all issued certificates comply with the key requirements in effect at the time of issuance.
 
 #### Server-Side Key Generation (Demo Only)
 
@@ -413,8 +456,8 @@ The agent deploys certificates using target connectors. Each connector knows how
 - **NGINX**: Writes cert/chain/key files to disk, validates config with `nginx -t`, reloads with `nginx -s reload` or `systemctl reload nginx`
 - **Apache httpd**: Writes separate cert/chain/key files, validates with `apachectl configtest`, graceful reload
 - **HAProxy**: Builds a combined PEM file (cert + chain + key), optionally validates config, reloads via systemctl or signal
-- **F5 BIG-IP** (planned): A proxy agent in the same network zone calls the iControl REST API to upload certificate and update SSL profile bindings. The server assigns the work; the proxy agent executes it.
-- **IIS** (planned, dual-mode): (1) Agent-local (recommended) — a Windows agent on the IIS box runs PowerShell `Import-PfxCertificate` + `Set-WebBinding` directly. (2) Proxy agent WinRM — for agentless IIS targets, a nearby Windows agent reaches the IIS box via WinRM.
+- **F5 BIG-IP**: A proxy agent in the same network zone calls the iControl REST API to upload certificate/key files, install crypto objects, and update the SSL client profile within an atomic transaction. The server assigns the work; the proxy agent executes it.
+- **IIS** (implemented, dual-mode): (1) Agent-local (recommended) — a Windows agent on the IIS box runs PowerShell `Import-PfxCertificate` + `Set-WebBinding` directly with PFX conversion and SHA-1 thumbprint computation. (2) Proxy agent WinRM — for agentless IIS targets, a nearby Windows agent reaches the IIS box via WinRM.
 
 The agent handles both the certificate (public) and the private key (read from local key store at `CERTCTL_KEY_DIR`). The control plane never sees the private key and never initiates outbound connections to agents or targets (pull-only model).
 
@@ -444,43 +487,65 @@ sequenceDiagram
     API-->>U: 200 OK
 ```
 
-The revocation is recorded in the `certificate_revocations` table (separate from the certificate status update) for CRL generation. The DER-encoded CRL at `GET /api/v1/crl/{issuer_id}` is generated on-demand by querying this table and signing with the issuing CA's key. The OCSP responder at `GET /api/v1/ocsp/{issuer_id}/{serial}` checks both the certificate status and the revocations table to return signed good/revoked/unknown responses.
+The revocation is recorded in the `certificate_revocations` table (separate from the certificate status update) for CRL generation. The DER-encoded CRL at `GET /.well-known/pki/crl/{issuer_id}` (RFC 5280 §5, RFC 8615) is generated on-demand by querying this table and signing with the issuing CA's key. The OCSP responder at `GET /.well-known/pki/ocsp/{issuer_id}/{serial}` (RFC 6960) checks both the certificate status and the revocations table to return signed good/revoked/unknown responses. Both endpoints are served unauthenticated — relying parties (TLS clients, hardware appliances, browsers) must be able to reach them without a certctl API key — and carry the IANA-registered media types `application/pkix-crl` and `application/ocsp-response` respectively.
 
 Short-lived certificates (those with profile TTL < 1 hour) return "good" from OCSP and are excluded from CRL — their rapid expiry is treated as sufficient revocation.
 
+#### Bulk Revocation
+
+For compliance events requiring fleet-wide revocation (key compromise, CA distrust, mass decommission), certctl supports bulk revocation by filter criteria. The `POST /api/v1/certificates/bulk-revoke` endpoint accepts filter parameters (profile_id, owner_id, agent_id, issuer_id) and creates individual revocation jobs for each matching certificate. Bulk revocation reuses the same 7-step single-cert flow for each certificate — no new issuer notification or audit mechanics. The operation is idempotent: revoking an already-revoked certificate is a no-op. Partial failures are tolerated — if one certificate fails to revoke (e.g., issuer unavailable), the operation continues for remaining certs and returns a summary. A single `bulk_revocation_initiated` audit event logs the operation with filter criteria, operator actor, and summary (total requested, succeeded, failed counts). Audit events for individual certificate revocations record the operator identity separately. The GUI bulk revoke button on the certificates list filters by visible selections and displays an affected-cert count modal before confirmation.
+
 ### 4. Automatic Renewal
 
-The control plane runs a scheduler with six background loops:
+The control plane runs a scheduler with 8 always-on loops plus up to 4 optional loops (enabled by configuration). `internal/scheduler/scheduler.go:262-265` is the authoritative count.
 
 ```mermaid
 flowchart LR
     subgraph "Scheduler (Background Goroutines)"
         R["Renewal Checker\n⏱ every 1h"]
         J["Job Processor\n⏱ every 30s"]
+        JR["Job Retry\n⏱ every 5m"]
+        JT["Job Timeout\n⏱ every 10m"]
         H["Agent Health\n⏱ every 2m"]
         N["Notification Processor\n⏱ every 1m"]
+        NR["Notification Retry\n⏱ every 2m"]
         SL["Short-Lived Expiry\n⏱ every 30s"]
         NS["Network Scanner\n⏱ every 6h"]
+        DG["Certificate Digest\n⏱ every 24h"]
+        HC["Endpoint Health\n⏱ every 60s"]
+        CD["Cloud Discovery\n⏱ every 6h"]
     end
 
     R -->|"Find expiring certs\nCreate renewal jobs"| DB[("PostgreSQL")]
     J -->|"Process pending jobs\nCoordinate issuance"| DB
+    JR -->|"Retry Failed jobs\nFailed→Pending"| DB
+    JT -->|"Reap stalled AwaitingCSR / AwaitingApproval jobs"| DB
     H -->|"Check heartbeat staleness\nMark agents offline"| DB
     N -->|"Send pending notifications\nEmail / Webhook / Slack"| DB
+    NR -->|"Retry failed notifications\n2^n-min backoff, DLQ after 5 attempts"| DB
     SL -->|"Expire short-lived certs\nMark as Expired"| DB
     NS -->|"Probe TLS endpoints\nStore discovered certs"| DB
+    DG -->|"Generate & send HTML digest\nEmail to recipients"| DB
+    HC -->|"Probe deployed TLS endpoints\nState machine + mismatch"| DB
+    CD -->|"AWS SM / Azure KV / GCP SM\nFeed discovery pipeline"| DB
 ```
 
-| Loop | Interval | Timeout | Purpose |
-|------|----------|---------|---------|
-| Renewal checker | 1 hour | 5 minutes | Finds certificates approaching expiry, creates renewal jobs |
-| Job processor | 30 seconds | 2 minutes | Processes pending jobs (issuance, renewal, deployment) |
-| Agent health check | 2 minutes | 1 minute | Marks agents as offline if heartbeat is stale |
-| Notification processor | 1 minute | 1 minute | Sends pending notifications via configured channels |
-| Short-lived expiry | 30 seconds | 30 seconds | Marks expired short-lived certificates (profile TTL < 1 hour) |
-| Network scanner | 6 hours | 30 minutes | Probes TLS endpoints on configured CIDR ranges, stores discovered certs (M21, opt-in via `CERTCTL_NETWORK_SCAN_ENABLED`). CIDR size validated at API level — max /20 (4096 IPs) per range. |
+| Loop | Interval | Always-on? | Purpose |
+|------|----------|------------|---------|
+| Renewal checker | 1 hour | Yes | Finds certificates approaching expiry (threshold-based or ARI-directed), creates renewal jobs |
+| Job processor | 30 seconds | Yes | Processes pending jobs (issuance, renewal, deployment) |
+| Job retry | 5 minutes (`CERTCTL_SCHEDULER_RETRY_INTERVAL`) | Yes | Transitions `Failed` jobs back to `Pending` for re-dispatch (I-001) |
+| Job timeout | 10 minutes (`CERTCTL_JOB_TIMEOUT_INTERVAL`) | Yes | Reaps `AwaitingCSR` jobs older than 24h and `AwaitingApproval` jobs older than 7d to `Failed`, feeding the retry loop (I-003) |
+| Agent health check | 2 minutes | Yes | Marks agents as offline if heartbeat is stale |
+| Notification processor | 1 minute | Yes | Sends pending notifications via configured channels |
+| Notification retry | 2 minutes (`CERTCTL_NOTIFICATION_RETRY_INTERVAL`) | Yes | Re-dispatches `Failed` notifications whose `next_retry_at` has elapsed; exponential backoff (2^n minutes, capped at 1h), 5-attempt budget, terminal `dead` status after exhaustion (I-005) |
+| Short-lived expiry | 30 seconds | Yes | Marks expired short-lived certificates (profile TTL < 1 hour) |
+| Network scanner | 6 hours | Opt-in (`CERTCTL_NETWORK_SCAN_ENABLED`) | Probes TLS endpoints on configured CIDR ranges, stores discovered certs (M21). CIDR size validated at API level — max /20 (4096 IPs) per range. |
+| Certificate digest | 24 hours (`CERTCTL_DIGEST_INTERVAL`) | Opt-in (digest service) | Generates HTML email with certificate stats, expiration timeline, job health, agent count. Does NOT run on startup — waits for first scheduled tick. Falls back to certificate owner emails if no explicit recipients configured. |
+| Endpoint health | 60 seconds (`CERTCTL_HEALTH_CHECK_INTERVAL`) | Opt-in (health check service) | Probes deployed TLS endpoints, drives the healthy/degraded/down/cert_mismatch state machine (M48) |
+| Cloud discovery | 6 hours | Opt-in (at least one cloud source configured) | Walks AWS Secrets Manager / Azure Key Vault / GCP Secret Manager, feeds discovery pipeline (M50) |
 
-Each loop uses `sync/atomic.Bool` idempotency guards to prevent concurrent tick execution — if a loop iteration is still running when the next tick fires, the tick is skipped with a warning log. All loops (including short-lived expiry check) run immediately on startup before entering their ticker interval, ensuring no gap between scheduler start and first execution. Graceful shutdown uses `sync.WaitGroup` with `WaitForCompletion()` to drain all in-flight work before process exit.
+Each loop uses `sync/atomic.Bool` idempotency guards to prevent concurrent tick execution — if a loop iteration is still running when the next tick fires, the tick is skipped with a warning log. Most loops (including short-lived expiry, job retry, job timeout, and notification retry) run immediately on startup before entering their ticker interval, ensuring no gap between scheduler start and first execution. The certificate digest loop is the exception — it does NOT run on startup, only on scheduled ticks. Graceful shutdown uses `sync.WaitGroup` with `WaitForCompletion()` to drain all in-flight work before process exit.
 
 Each operation has a context timeout to prevent indefinite hangs if external services become unresponsive.
 
@@ -501,9 +566,16 @@ flowchart TB
         II["IssuerConnector Interface\nIssueCertificate() | RenewCertificate()\nRevokeCertificate() | GetOrderStatus()"]
         II --> LC["Local CA"]
         II --> ACME["ACME v2"]
-        II --> SC["step-ca"]
+        II --> SCA["step-ca"]
         II --> OC["OpenSSL / Custom CA"]
-        II --> VP["Vault PKI (planned)"]
+        II --> VP["Vault PKI"]
+        II --> DC["DigiCert CertCentral"]
+        II --> SG["Sectigo SCM"]
+        II --> GC["Google CAS"]
+        II --> AP2["AWS ACM PCA"]
+        II --> EN["Entrust"]
+        II --> GS["GlobalSign Atlas"]
+        II --> EJ["EJBCA"]
     end
 
     subgraph "Target Connectors"
@@ -514,8 +586,14 @@ flowchart TB
         TI --> HP["HAProxy"]
         TI --> TF["Traefik"]
         TI --> CD["Caddy"]
-        TI --> F5["F5 BIG-IP (interface only)"]
-        TI --> IIS["IIS (interface only)"]
+        TI --> EV["Envoy"]
+        TI --> PO["Postfix/Dovecot"]
+        TI --> IIS["IIS"]
+        TI --> F5["F5 BIG-IP"]
+        TI --> SSH["SSH"]
+        TI --> WCS["WinCertStore"]
+        TI --> JKS["Java Keystore"]
+        TI --> K8S["K8s Secrets"]
     end
 
     subgraph "Notifier Connectors"
@@ -567,7 +645,11 @@ type Connector interface {
 }
 ```
 
-Built-in issuers: **Local CA** (self-signed or sub-CA mode using `crypto/x509`), **ACME v2** (HTTP-01, DNS-01, and DNS-PERSIST-01 challenges, compatible with Let's Encrypt, ZeroSSL, Sectigo, Google Trust Services, and any ACME-compliant CA), **step-ca** (Smallstep private CA via native /sign API with JWK provisioner auth), and **OpenSSL/Custom CA** (script-based signing delegating to user-provided shell scripts). The ACME connector uses `golang.org/x/crypto/acme`, generates an ECDSA P-256 account key, handles account registration with ToS acceptance and optional External Account Binding (EAB) for CAs that require it (ZeroSSL, Google Trust Services, SSL.com), order creation, challenge solving (HTTP-01 via built-in server, DNS-01 via script-based hooks, DNS-PERSIST-01 via standing TXT records with auto-fallback to DNS-01), order finalization, and DER-to-PEM chain conversion. For ZeroSSL, EAB credentials are auto-fetched from ZeroSSL's public API when the directory URL is detected as ZeroSSL and no EAB credentials are provided — zero-friction onboarding with no dashboard visit required. The interface also includes `GetCACertPEM(ctx)` for CA chain distribution (used by the EST server's `/cacerts` endpoint).
+Built-in issuers (live count: `ls -d internal/connector/issuer/*/ | wc -l`): **Local CA** (self-signed or sub-CA mode using `crypto/x509`), **ACME v2** (HTTP-01, DNS-01, and DNS-PERSIST-01 challenges, compatible with Let's Encrypt, ZeroSSL, Sectigo, Google Trust Services, and any ACME-compliant CA), **step-ca** (Smallstep private CA via native /sign API with JWK provisioner auth), **OpenSSL/Custom CA** (script-based signing delegating to user-provided shell scripts), **Vault PKI** (HashiCorp Vault's PKI secrets engine via /sign API with token auth), **DigiCert** (commercial CA via CertCentral REST API with async order processing), **Sectigo SCM** (async order model with 3-header auth), **Google CAS** (Cloud Certificate Authority Service with OAuth2 service account auth), **AWS ACM Private CA** (synchronous issuance via ACM PCA API), **Entrust** (mTLS client cert auth, sync/approval-pending), **GlobalSign Atlas HVCA** (mTLS + API key/secret dual auth), and **EJBCA** (Keyfactor open-source self-hosted CA, dual auth: mTLS or OAuth2). The ACME connector uses `golang.org/x/crypto/acme`, generates an ECDSA P-256 account key, handles account registration with ToS acceptance and optional External Account Binding (EAB) for CAs that require it (ZeroSSL, Google Trust Services, SSL.com), order creation, challenge solving (HTTP-01 via built-in server, DNS-01 via script-based hooks, DNS-PERSIST-01 via standing TXT records with auto-fallback to DNS-01), order finalization, and DER-to-PEM chain conversion. For ZeroSSL, EAB credentials are auto-fetched from ZeroSSL's public API when the directory URL is detected as ZeroSSL and no EAB credentials are provided — zero-friction onboarding with no dashboard visit required.
+
+**ACME Renewal Information (ARI, RFC 9773):** The ACME connector supports CA-directed renewal timing via the `GetRenewalInfo()` method. Instead of using fixed thresholds (e.g., renew 30 days before expiry), the CA tells certctl when to renew by providing a `suggestedWindow` with start and end times. This is useful for distributing renewal load during maintenance windows and coordinating mass-revocation scenarios. Enable with `CERTCTL_ACME_ARI_ENABLED=true`. Cert ID is computed as `base64url(SHA-256(DER cert))` per RFC 9773. If the CA doesn't support ARI (404 from the ARI endpoint), certctl automatically falls back to threshold-based renewal — no operator intervention required. Errors from the CA are logged as warnings.
+
+The interface also includes `GetCACertPEM(ctx)` for CA chain distribution (used by the EST server's `/cacerts` endpoint).
 
 ### Target Connector
 
@@ -583,11 +665,11 @@ type Connector interface {
 
 The `DeploymentRequest` struct carries the full material needed by the target system: the signed certificate, the CA chain, the agent-generated private key, target-specific configuration, and arbitrary metadata. The key field is populated by the agent from its local key store (`CERTCTL_KEY_DIR`) — it never originates from the control plane.
 
-Built-in targets: **NGINX** (writes cert/chain/key files, validates with `nginx -t`, reloads), **Apache httpd** (writes cert/chain/key files, validates with `apachectl configtest`, graceful reload), **HAProxy** (combined PEM file with cert+chain+key, validates config, reloads via systemctl/signal), **Traefik** (file provider — writes cert/key to watched directory, Traefik auto-reloads), **Caddy** (dual-mode: admin API hot-reload or file-based), **F5 BIG-IP** (interface only — proxy agent + iControl REST, implementation planned), **IIS** (interface only — dual-mode: agent-local PowerShell primary + proxy agent WinRM for agentless targets, implementation planned).
+Built-in targets (14 connector types): **NGINX** (writes cert/chain/key files, validates with `nginx -t`, reloads), **Apache httpd** (writes cert/chain/key files, validates with `apachectl configtest`, graceful reload), **HAProxy** (combined PEM file with cert+chain+key, validates config, reloads via systemctl/signal), **Traefik** (file provider — writes cert/key to watched directory, Traefik auto-reloads), **Caddy** (dual-mode: admin API hot-reload or file-based), **Envoy** (file-based with optional SDS JSON config), **F5 BIG-IP** (proxy agent + iControl REST, transaction-based atomic SSL profile updates), **IIS** (dual-mode: agent-local PowerShell + proxy agent WinRM for agentless targets), **Postfix/Dovecot** (file write + service reload), **SSH** (agentless deployment via SSH/SFTP), **Windows Certificate Store** (PowerShell-based cert import, dual-mode local/WinRM), **Java Keystore** (PEM → PKCS#12 → keytool pipeline, JKS and PKCS12 formats), **Kubernetes Secrets** (deploys as `kubernetes.io/tls` Secrets via injectable K8sClient interface, in-cluster or kubeconfig auth).
 
 After deployment, agents can perform **post-deployment TLS verification**: the agent probes the live TLS endpoint using `crypto/tls.DialWithDialer` and compares the SHA-256 fingerprint of the served certificate against what was deployed. Results are reported via `POST /api/v1/jobs/{id}/verify` and stored on the job record. Verification is best-effort — failures don't block or rollback deployments.
 
-Additional cloud, network, and Kubernetes target connectors are planned for future releases.
+The SSH connector enables agentless deployment to any Linux/Unix server via SSH/SFTP, using the proxy agent pattern. The Kubernetes Secrets connector deploys certificates as `kubernetes.io/tls` Secrets via an injectable K8sClient interface supporting both in-cluster and out-of-cluster auth.
 
 ### Notifier Connector
 
@@ -605,6 +687,16 @@ Built-in notifiers: **Email** (SMTP), **Webhook** (HTTP POST), **Slack** (incomi
 
 See the [Connector Development Guide](connectors.md) for details on building custom connectors.
 
+### Notification Retry & Dead-Letter Queue
+
+A transient notifier failure (SMTP timeout, 5xx webhook response, Slack rate-limit) must not silently drop a critical alert. Migration `000016_notification_retry` adds three columns to `notification_events` — `retry_count INTEGER NOT NULL DEFAULT 0`, `next_retry_at TIMESTAMPTZ` (nullable — only meaningful while a row is in `failed` state), and `last_error TEXT` (the most recent transient error, preserved for operator triage) — together with a partial index `idx_notification_events_retry_sweep ON notification_events(next_retry_at) WHERE status = 'failed' AND next_retry_at IS NOT NULL` so the retry hot path scales with the retry-eligible slice rather than the full notification history.
+
+The scheduler's notification-retry loop (see the scheduler section above) calls `NotificationService.RetryFailedNotifications(ctx)` every `CERTCTL_NOTIFICATION_RETRY_INTERVAL` (default `2m`). Each tick pulls up to 1000 rows via `notifRepo.ListRetryEligible(ctx, now, maxAttempts, sweepLimit)` — a partial-index-driven query that filters on `status='failed' AND next_retry_at <= now() AND retry_count < 5` — and redispatches them through the same notifier registry used by `ProcessPendingNotifications`. A successful redispatch transitions the row directly to `sent` without incrementing `retry_count`, so the audit trail preserves "delivered on attempt N". A failed redispatch re-arms `next_retry_at` using exponential backoff — `wait = min(2^retry_count minutes, 1h)` — bumps `retry_count`, and stamps `last_error`. When `retry_count >= 4` (the fifth attempt has just failed) the row is promoted to the terminal `dead` status via `notifRepo.MarkAsDead`, which clears `next_retry_at` so the partial retry-sweep index stops matching and the row cannot be re-entered into the retry rotation without operator action.
+
+`NotificationService.RequeueNotification(ctx, id)` is the operator-driven escape hatch from `dead`. It atomically resets `retry_count → 0`, `next_retry_at → NULL`, `last_error → NULL`, and `status → pending`, handing the row back to `ProcessPendingNotifications` on the next 1m tick. This is the correct response to "the notifier outage is resolved, redeliver the queue"; it is not a retry, which is why the retry counter is reset rather than incremented.
+
+The dead-letter depth is surfaced in two places. First, `DashboardSummary.NotificationsDead` is populated by `StatsService.GetDashboardSummary` via `notifRepo.CountByStatus(ctx, "dead")`. The injection uses a `SetNotifRepo` setter pattern (mirroring `CertificateService.SetTargetRepo`) rather than a new positional argument to `NewStatsService`, which keeps all nine existing `NewStatsService` call sites (main.go plus eight digest tests and stats_test.go) signature-stable — when the notification repository has not been wired in, `NotificationsDead` falls through to zero. Second, the `/api/v1/metrics/prometheus` endpoint emits `certctl_notification_dead_total` as a counter (operator alert thresholds per the I-005 spec: `> 0` warning, `> 10` critical) using the same `DashboardSummary` snapshot so the dashboard card and the Prometheus counter cannot skew. The web dashboard exposes a two-tab toolbar on `/notifications` — "All" (the pre-I-005 inbox) and "Dead letter" (threads `?status=dead` into the list query, surfaces `Retry N/5` and the truncated `last_error` with a full-text tooltip per row, and binds a Requeue button to `POST /api/v1/notifications/{id}/requeue`).
+
 ### EST Server (RFC 7030)
 
 The EST (Enrollment over Secure Transport) server provides an industry-standard enrollment interface for devices that need certificates without using the REST API. It runs under `/.well-known/est/` per RFC 7030 and supports four operations: CA certificate distribution (`/cacerts`), initial enrollment (`/simpleenroll`), re-enrollment (`/simplereenroll`), and CSR attributes (`/csrattrs`).
@@ -640,10 +732,52 @@ type ESTService interface {
 }
 ```
 
-**Issuer connector extension:** EST required adding `GetCACertPEM(ctx) (string, error)` to the issuer connector interface so the `/cacerts` endpoint can serve the CA chain. The Local CA connector returns its CA certificate PEM; ACME, step-ca, and OpenSSL connectors return errors (they don't expose a static CA chain — their chains are per-issuance).
+**Issuer connector extension:** EST required adding `GetCACertPEM(ctx) (string, error)` to the issuer connector interface so the `/cacerts` endpoint can serve the CA chain. The Local CA returns its CA certificate PEM; Vault PKI fetches via `GET /v1/{mount}/ca/pem`; Google CAS fetches via API; AWS ACM PCA retrieves via `GetCertificateAuthorityCertificate`. ACME, step-ca, OpenSSL, DigiCert, and Sectigo connectors return errors (they don't expose a static CA chain — their chains are per-issuance).
+
+**Authentication:** EST endpoints are served unauthenticated at the HTTP layer under `/.well-known/est/*` — no Bearer token required. Per RFC 7030 §3.2.3 EST authentication is deployment-specific, and per §4.1.1 `/cacerts` is explicitly anonymous. certctl enforces authentication via CSR signature verification inside `ESTService.SimpleEnroll`/`SimpleReEnroll` plus profile policy gates (allowed key algorithms, minimum key size, permitted SANs, permitted EKUs, MaxTTL). The HTTP dispatch is implemented in `cmd/server/main.go:buildFinalHandler`, which routes `/.well-known/est/*` through `noAuthHandler` (RequestID + structuredLogger + Recovery only). Operators who need stronger client identification should terminate mTLS at an upstream reverse proxy and pin the CSR's SAN to the client cert subject at the profile level.
 
 **Audit:** Every EST enrollment is recorded in the audit trail with `protocol: "EST"`, the CN, SANs, issuer ID, serial number, and optional profile ID.
 
+### SCEP Server (RFC 8894)
+
+The SCEP (Simple Certificate Enrollment Protocol) server provides certificate enrollment for MDM platforms and network devices. It runs at `/scep` with operation-based dispatch via query parameters per RFC 8894.
+
+**Architecture:** SCEP follows the exact same layering as EST — a handler-level protocol that delegates certificate issuance to an existing `IssuerConnector`. The `SCEPService` bridges the `SCEPHandler` to whichever issuer connector is configured via `CERTCTL_SCEP_ISSUER_ID`.
+
+```
+Client (MDM, network device, SCEP client)
+    │
+    ▼
+SCEPHandler (handler layer)
+    │  PKCS#7 envelope parsing, CSR extraction, challenge password extraction
+    ▼
+SCEPService (service layer)
+    │  Challenge password validation, CSR validation, CN/SAN extraction, audit recording
+    ▼
+IssuerConnector (connector layer via IssuerConnectorAdapter)
+    │  Certificate signing (Local CA, step-ca, etc.)
+    ▼
+Signed certificate returned as PKCS#7 certs-only
+```
+
+**Wire format:** SCEP clients wrap CSRs in PKCS#7 SignedData envelopes. The handler parses the outer ASN.1 ContentInfo → SignedData → EncapsulatedContentInfo to extract the CSR bytes. Fallback paths handle base64-encoded PKCS#7 and raw CSR submissions (for simpler clients). Responses use PKCS#7 certs-only via the shared `internal/pkcs7` package (same as EST). Single certs are returned as raw DER for `GetCACert`, chains as PKCS#7.
+
+**Authentication:** SCEP endpoints at `/scep` and `/scep/*` are served unauthenticated at the HTTP layer — no Bearer token required — per RFC 8894 §3.2, which defines authentication via the `challengePassword` attribute (OID 1.2.840.113549.1.9.7) embedded in the PKCS#10 CSR rather than an HTTP credential. The HTTP dispatch is implemented in `cmd/server/main.go:buildFinalHandler`, which routes `/scep` and `/scep/*` through `noAuthHandler` (RequestID + structuredLogger + Recovery only). The `challengePassword` is mandatory: `preflightSCEPChallengePassword` at startup refuses to boot the control plane when `CERTCTL_SCEP_ENABLED=true` is set without `CERTCTL_SCEP_CHALLENGE_PASSWORD`, closing CWE-306 (missing authentication for a critical function). `SCEPService.PKCSReq` enforces the same invariant defense-in-depth — an empty `s.challengePassword` rejects every enrollment — and the password comparison uses `crypto/subtle.ConstantTimeCompare` to prevent response-time side-channel leakage. The startup log line `SCEP server enabled` emits a `challenge_password_set` boolean for operator visibility.
+
+**Interface:** The `SCEPHandler` defines an `SCEPService` interface (dependency inversion):
+
+```go
+type SCEPService interface {
+    GetCACaps(ctx context.Context) string
+    GetCACert(ctx context.Context) (string, error)
+    PKCSReq(ctx context.Context, csrPEM string, challengePassword string, transactionID string) (*domain.SCEPEnrollResult, error)
+}
+```
+
+**Shared PKCS#7 package:** Both EST and SCEP handlers share a common `internal/pkcs7` package for building PKCS#7 certs-only responses and PEM-to-DER chain conversion, eliminating code duplication between the two enrollment protocols.
+
+**Audit:** Every SCEP enrollment is recorded in the audit trail with `protocol: "SCEP"`, the CN, SANs, issuer ID, serial number, transaction ID, and optional profile ID.
+
 ## Security Model
 
 ### Private Key Management
@@ -685,10 +819,11 @@ The control plane only handles public material: certificates, chains, and CSRs.
 
 ### Authentication
 
-- **API clients → Server**: API key in `Authorization: Bearer` header, or `none` for demo mode
+- **API clients → Server**: API key in `Authorization: Bearer` header, or `none` for demo mode. Applies to every path under `/api/v1/*`.
 - **Agent → Server**: API key registered at agent creation, included in all requests
 - **Server → Issuers**: ACME account key, or connector-specific credentials
 - **Agent → Targets**: API tokens, WinRM credentials (stored locally on agent or proxy agent — never on server). Credential scope is limited to the agent's network zone.
+- **Standards-based enrollment and PKI distribution endpoints**: `/.well-known/est/*` (RFC 7030), `/scep` and `/scep/*` (RFC 8894), and `/.well-known/pki/crl/{issuer_id}` + `/.well-known/pki/ocsp/{issuer_id}/{serial}` (RFC 5280 §5 / RFC 6960 / RFC 8615) are served unauthenticated at the HTTP layer. These protocols carry their own authentication semantics — CSR signature + profile policy for EST (§3.2.3 says EST auth is deployment-specific; §4.1.1 makes `/cacerts` explicitly anonymous), `challengePassword` in CSR attributes for SCEP (§3.2), and relying-party accessibility for CRL/OCSP — and cannot present certctl Bearer tokens. The dispatch is implemented in `cmd/server/main.go:buildFinalHandler`, which routes these prefixes through `noAuthHandler` (RequestID + structuredLogger + Recovery only, no auth or rate-limit middleware). CWE-306 is closed for SCEP by `preflightSCEPChallengePassword`, which refuses to start the server when SCEP is enabled without `CERTCTL_SCEP_CHALLENGE_PASSWORD`. The 27-subtest regression harness `cmd/server/finalhandler_test.go` pins this dispatch surface (EST 4-endpoint, SCEP exact + trailing-slash + query-string, PKI CRL+OCSP, health probes, `/api/v1/*` authenticated, `/assets/*` file server, SPA fallback).
 
 ### Audit Trail
 
@@ -725,6 +860,34 @@ All shell-facing inputs (connector scripts, domain names, ACME tokens) are valid
 
 All incoming HTTP request bodies are capped by `http.MaxBytesReader` middleware (default 1MB, configurable via `CERTCTL_MAX_BODY_SIZE`). Requests exceeding the limit receive a 413 Request Entity Too Large response. The middleware is positioned before authentication in the chain so oversized payloads are rejected early, before any auth processing or database work occurs. Requests without bodies (GET, HEAD, nil body) skip the limit check.
 
+### Config Encryption at Rest
+
+Dynamic issuer and target configurations (rows with `source='database'`) contain credentials — ACME EAB HMACs, Vault tokens, DigiCert/Sectigo API keys, SSH private keys, WinRM passwords, F5 BIG-IP passwords, and similar. These are sealed at rest in PostgreSQL via `internal/crypto/encryption.go` using AES-256-GCM with a key derived from the operator passphrase `CERTCTL_CONFIG_ENCRYPTION_KEY` through PBKDF2-SHA256 (100,000 rounds, 32-byte output).
+
+**v2 wire format (current, M-8 remediation, CWE-916 / CWE-329):**
+
+```
+magic(0x02) || salt(16) || nonce(12) || ciphertext+tag
+```
+
+Every call to `EncryptIfKeySet` draws 16 fresh bytes from `crypto/rand` as the PBKDF2 salt, so the derived AES-256 key is distinct per ciphertext and per re-encryption. The salt is stored alongside the ciphertext; decryption reads the magic byte, splits out the salt, re-derives the key, and verifies the AEAD tag.
+
+**v1 legacy format (read-only):**
+
+```
+nonce(12) || ciphertext+tag
+```
+
+Pre-M-8 blobs were sealed with a package-level fixed salt `"certctl-config-encryption-v1"`. `DecryptIfKeySet` preserves the v1 read path unchanged — a blob whose first byte is not `0x02`, or whose v2 AEAD verification fails (including the 1/256 case where a v1 nonce happens to begin with `0x02`), falls through to a v1 attempt against the legacy fixed salt. v1 blobs are never written by the post-M-8 code path; they re-seal as v2 naturally on the next UPDATE through the normal service CRUD flow. No operator migration ceremony is required.
+
+**Fail-closed behavior (C-2 sentinel, CWE-311):** both `EncryptIfKeySet` and `DecryptIfKeySet` return `ErrEncryptionKeyRequired` when invoked with an empty passphrase. The server refuses to start if any `source='database'` rows already exist without `CERTCTL_CONFIG_ENCRYPTION_KEY` set.
+
+**Low-level primitives preserved byte-identical.** `Encrypt`, `Decrypt`, and `DeriveKey` are kept bit-stable so v1 fixtures on disk remain decryptable unchanged and so callers outside the config-encryption path (none today, but the symbols are exported) do not see a breaking change. The new per-ciphertext salt path is reached via the helper `deriveKeyWithSalt(passphrase, salt)`.
+
+**Passphrase plumbing.** Services (`IssuerService`, `TargetService`, `IssuerRegistry`) hold the operator passphrase as a raw `string` and delegate PBKDF2 to the crypto package per ciphertext. This replaces the pre-M-8 design that pre-derived a single `[]byte` key at service construction and reused it for every row, which was the direct consequence of the fixed-salt KDF.
+
+**Coverage gate.** CI enforces `internal/crypto/...` coverage ≥ 85% (observed 86.7%) — the encryption primitives are a security-critical gate, and the v2 format plus v1 fallback plus C-2 sentinel paths all need exhaustive coverage to avoid silent regressions.
+
 ### CORS
 
 CORS uses a **deny-by-default** posture: when `CERTCTL_CORS_ORIGINS` is empty, no CORS headers are set and only same-origin requests can read responses. Operators must explicitly configure allowed origins. This prevents accidental exposure of the API to cross-origin requests in production.
@@ -739,12 +902,18 @@ The HTTP middleware stack processes requests in the following order (see `cmd/se
 4. **BodyLimit** - request body size cap via `http.MaxBytesReader`
 5. **RateLimiter** - token bucket rate limiting (optional, when enabled)
 6. **CORS** - cross-origin request handling (deny-by-default)
-7. **Auth** - API key or JWT validation
+7. **Auth** - API key validation (or none in development; JWT/OIDC via authenticating gateway, see below — not in-process)
 8. **AuditLog** - records every API call to the audit trail (requires auth context for actor)
 
+### Authenticating-gateway pattern (JWT, OIDC, mTLS)
+
+certctl's in-process authentication surface is intentionally narrow: `api-key` for production deployments and `none` for development. There is no in-process JWT, OIDC, mTLS, or SAML middleware. (`CERTCTL_AUTH_TYPE=jwt` was accepted pre-G-1 but silently routed through the api-key bearer middleware — a security finding masquerading as a config option, removed at the v2.x boundary; see [`upgrade-to-v2-jwt-removal.md`](upgrade-to-v2-jwt-removal.md) if you previously set it.)
+
+For deployments that need JWT/OIDC/mTLS, the standard pattern is to put an authenticating gateway in front of certctl and configure `CERTCTL_AUTH_TYPE=none` on the upstream certctl process. The gateway terminates the federated identity protocol, validates tokens / certificates / SAML assertions, and proxies the authenticated request to certctl as a same-origin call on a private network. This separation gives operators the full breadth of the modern identity ecosystem (oauth2-proxy, Envoy `ext_authz`, Traefik `ForwardAuth`, Pomerium, Authelia, Caddy `forward_auth`, Apache `mod_auth_openidc`, nginx `auth_request`) without certctl itself having to track signing-key rotation, claim mapping, audience validation, and the rest of the JWT/OIDC surface area. Operators wanting per-request actor attribution past the gateway boundary forward the gateway-resolved identity (e.g., `X-Auth-Request-User` from oauth2-proxy) and run a small authorization layer at the gateway that enforces the bearer-key contract certctl actually uses.
+
 ### Concurrency Safety
 
-The background scheduler uses `sync/atomic.Bool` idempotency guards on all 6 loops — if a tick fires while the previous iteration is still running, it skips. A `sync.WaitGroup` tracks all in-flight goroutines. `WaitForCompletion(timeout)` blocks during shutdown until all work finishes or the timeout expires, preventing state corruption from mid-flight database operations during process exit.
+The background scheduler uses `sync/atomic.Bool` idempotency guards on every loop (8 always-on plus up to 4 optional) — if a tick fires while the previous iteration is still running, it skips. A `sync.WaitGroup` tracks all in-flight goroutines. `WaitForCompletion(timeout)` blocks during shutdown until all work finishes or the timeout expires, preventing state corruption from mid-flight database operations during process exit.
 
 ### Logging
 
@@ -763,10 +932,20 @@ All endpoints are under `/api/v1/` and follow consistent patterns:
 
 Resources: certificates, issuers, targets, agents, jobs, policies, profiles, teams, owners, agent-groups, audit, notifications, discovered-certificates, discovery-scans, network-scan-targets, stats, metrics.
 
-The full API is documented in an OpenAPI 3.1 specification at `api/openapi.yaml` with 97 endpoints across 20 resource domains (95 under `/api/v1/` + `/.well-known/est/` plus `/health` and `/ready`; includes auth, 7 discovery endpoints from M18b, 6 network scan endpoints from M21, Prometheus metrics from M22, and 4 EST enrollment endpoints from M23), all request/response schemas, and pagination conventions. See the [OpenAPI Guide](openapi.md) for usage with Swagger UI and SDK generation.
+The full API is documented in an OpenAPI 3.1 specification at `api/openapi.yaml`. The router-vs-spec parity is pinned by the `TestRouter_OpenAPIParity` regression test (Bundle D / M-027), which AST-walks `internal/api/router/router.go` for every `r.Register` AND direct `r.mux.Handle` registration and asserts the set matches the spec's `paths:` block exactly. Live counts:
+
+```
+grep -cE 'r\.Register\("[A-Z]' internal/api/router/router.go    # r.Register sites
+grep -cE 'r\.mux\.Handle\("[A-Z]' internal/api/router/router.go # r.mux.Handle sites (auth-exempt: health/ready/auth-info/version)
+grep -cE '^\s+operationId:' api/openapi.yaml                   # documented operations
+```
+
+See the [OpenAPI Guide](openapi.md) for usage with Swagger UI and SDK generation.
 
 Jobs support additional action endpoints: `POST /api/v1/jobs/{id}/cancel`, `POST /api/v1/jobs/{id}/approve`, `POST /api/v1/jobs/{id}/reject`.
 
+**Bulk Operations:** `POST /api/v1/certificates/bulk-revoke` — Bulk revocation by filter criteria (profile_id, owner_id, agent_id, issuer_id). Creates individual revocation jobs for matching certificates, with partial-failure tolerance and a summary audit event.
+
 **Enhanced Query Features (M20):** Certificate list endpoints support additional query capabilities beyond basic pagination:
 
 - **Sorting**: `?sort=notAfter` (ascending) or `?sort=-createdAt` (descending). Whitelist: notAfter, expiresAt, createdAt, updatedAt, commonName, name, status, environment.
@@ -776,7 +955,9 @@ Jobs support additional action endpoints: `POST /api/v1/jobs/{id}/cancel`, `POST
 - **Additional filters**: `?agent_id=`, `?profile_id=` (in addition to existing status, environment, owner_id, team_id, issuer_id).
 - **Deployments**: `GET /api/v1/certificates/{id}/deployments` returns deployment targets for a certificate.
 
-Certificate revocation: `POST /api/v1/certificates/{id}/revoke` with optional `{"reason": "keyCompromise"}`. Supports RFC 5280 reason codes (unspecified, keyCompromise, caCompromise, affiliationChanged, superseded, cessationOfOperation, certificateHold, privilegeWithdrawn). Returns the updated certificate status. Best-effort issuer notification — the revocation succeeds even if the issuer connector is unavailable. A JSON-formatted CRL is available at `GET /api/v1/crl`, and a DER-encoded X.509 CRL signed by the issuing CA at `GET /api/v1/crl/{issuer_id}`. An embedded OCSP responder serves signed responses at `GET /api/v1/ocsp/{issuer_id}/{serial}`. Short-lived certificates (profile TTL < 1 hour) are exempt from CRL/OCSP — expiry is sufficient revocation.
+Certificate revocation: `POST /api/v1/certificates/{id}/revoke` with optional `{"reason": "keyCompromise"}`. Supports RFC 5280 reason codes (unspecified, keyCompromise, caCompromise, affiliationChanged, superseded, cessationOfOperation, certificateHold, privilegeWithdrawn). Returns the updated certificate status. Best-effort issuer notification — the revocation succeeds even if the issuer connector is unavailable. The DER-encoded X.509 CRL signed by the issuing CA is served unauthenticated at `GET /.well-known/pki/crl/{issuer_id}` (RFC 5280 §5 + RFC 8615, `Content-Type: application/pkix-crl`). The embedded OCSP responder serves signed responses unauthenticated at `GET /.well-known/pki/ocsp/{issuer_id}/{serial}` (RFC 6960, `Content-Type: application/ocsp-response`). Both endpoints are accessible to relying parties with no certctl API credentials, as RFC-compliant PKI consumers expect. Short-lived certificates (profile TTL < 1 hour) are exempt from CRL/OCSP — expiry is sufficient revocation.
+
+Certificate export (M27): `GET /api/v1/certificates/{id}/export/pem` returns PEM-encoded certificate and chain, and `POST /api/v1/certificates/{id}/export/pkcs12` returns a PKCS#12 bundle (binary). Private keys are never exported — they remain on agents. All exports are audited with actor, timestamp, and format.
 
 Health checks live outside the API prefix: `GET /health` and `GET /ready`.
 
@@ -789,7 +970,7 @@ flowchart LR
     AI["AI Assistant\n(Claude, Cursor)"] -->|"stdio"| MCP["MCP Server\ncmd/mcp-server/"]
     MCP -->|"HTTP + Bearer token"| API["certctl REST API\n:8443"]
 
-    subgraph "78 MCP Tools"
+    subgraph "MCP Tools"
         T1["Certificate CRUD"]
         T2["Agent Management"]
         T3["Job Operations"]
@@ -803,7 +984,7 @@ flowchart LR
 
 The MCP server is a stateless HTTP proxy — every MCP tool call translates to an HTTP request to the certctl REST API. It adds no new state, no new dependencies, and no new attack surface beyond what the API already exposes. Configuration is minimal: `CERTCTL_SERVER_URL` and `CERTCTL_API_KEY` environment variables.
 
-The 78 tools are organized across 16 resource domains with typed input structs and `jsonschema` struct tags for automatic LLM-friendly schema generation. Binary response support handles DER CRL and OCSP endpoints.
+The tools are organized across 16 resource domains with typed input structs and `jsonschema` struct tags for automatic LLM-friendly schema generation. Binary response support handles DER CRL and OCSP endpoints.
 
 ## CLI Tool
 
@@ -833,7 +1014,9 @@ flowchart TB
 **Credentials & Configuration:**
 Database and API credentials are managed via environment variables defined in a `.env` file. Copy `deploy/.env.example` to `deploy/.env` for local development and customize credentials for production. The agent key directory (`CERTCTL_KEY_DIR`) is persisted as a named Docker volume (`agent_keys`) at `/var/lib/certctl/keys` for reliable key storage across container restarts.
 
-### Production (Kubernetes)
+### Production (Kubernetes with Helm)
+
+A production-ready Helm chart is available under `deploy/helm/certctl/` with full support for multi-replica deployments, persistent PostgreSQL, agent DaemonSet, optional Ingress, and security best practices.
 
 ```mermaid
 flowchart TB
@@ -859,11 +1042,26 @@ flowchart TB
     DS --> DEP
 ```
 
+**Helm Installation:**
+
+```bash
+# Add the chart (if published) or install from local directory
+helm install certctl deploy/helm/certctl/ \
+  --set server.auth.apiKey="your-secure-key" \
+  --set postgresql.auth.password="your-db-password" \
+  --set ingress.enabled=true \
+  --set ingress.hosts[0].host="certctl.example.com"
+```
+
+The Helm chart includes: server Deployment with configurable replicas, liveness/readiness probes, security context (non-root, read-only rootfs), PostgreSQL StatefulSet with persistent volumes, optional Ingress with TLS, ServiceAccount with configurable RBAC, and agent DaemonSet running one agent per node. All certctl configuration options are exposed in `values.yaml` — issuers, targets, notifiers, scheduler intervals, discovery settings, and SMTP for digest emails.
+
+See `deploy/helm/certctl/values.yaml` for the full configuration reference and `deploy/helm/certctl/Chart.yaml` for version and appVersion details.
+
 For production, you would also add an ingress controller, TLS termination for the certctl API itself, and external PostgreSQL (RDS, Cloud SQL, etc.).
 
-## Discovery Data Flow (M18b + M21)
+## Discovery Data Flow (M18b + M21 + M50)
 
-Certificate discovery enables operators to build a complete inventory of existing certificates before managing them with certctl. There are two discovery modes that feed into the same pipeline:
+Certificate discovery enables operators to build a complete inventory of existing certificates before managing them with certctl. There are three discovery modes that feed into the same pipeline:
 
 ```mermaid
 flowchart TB
@@ -872,6 +1070,7 @@ flowchart TB
         SCAN["Filesystem Scanner\n(CERTCTL_DISCOVERY_DIRS)"]
         SERVER["certctl-server\n(network discovery)"]
         NETSCAN["TLS Scanner\n(CIDR ranges + ports)"]
+        CLOUD["Cloud Discovery\n(AWS SM / Azure KV / GCP SM)"]
     end
 
     EXTRACT["Extract Metadata\n(CN, SANs, serial, issuer, expiry, fingerprint)"]
@@ -887,6 +1086,7 @@ flowchart TB
     SCAN --> EXTRACT
     SERVER -->|"Scheduler loop\n(every 6h)"| NETSCAN
     NETSCAN -->|"crypto/tls.Dial\n50 goroutines"| EXTRACT
+    CLOUD -->|"Scheduler loop\n(every 6h)"| EXTRACT
     EXTRACT --> SERVICE
     SERVICE --> REPO
     REPO -->|"Dedup by fingerprint\n+ agent_id + source_path"| DB
@@ -913,7 +1113,16 @@ flowchart TB
 5. **Sentinel agent** — Results submitted using `server-scanner` as virtual agent ID, with `source_path` set to `ip:port` and `source_format` set to `network`
 6. **Same pipeline** — Feeds into the same `DiscoveryService.ProcessDiscoveryReport()` as filesystem discovery — same dedup, same audit trail, same triage workflow
 
-**Common triage workflow (both sources):**
+**Cloud Secret Manager Discovery (M50):**
+
+1. **Pluggable sources** — Each cloud provider implements the `DiscoverySource` interface (Name, Type, Discover, ValidateConfig). Three built-in sources: AWS Secrets Manager, Azure Key Vault, GCP Secret Manager
+2. **CloudDiscoveryService orchestrator** — Iterates registered sources, calls `Discover()` on each, feeds reports into `ProcessDiscoveryReport()`. Errors from one source don't prevent other sources from running
+3. **Scheduler integration** — opt-in cloud discovery scheduler loop (6h default; see `docs/architecture.md` 12-loop topology), runs immediately on startup, `atomic.Bool` idempotency guard
+4. **Sentinel agents** — Each source uses its own sentinel agent ID (`cloud-aws-sm`, `cloud-azure-kv`, `cloud-gcp-sm`) for dedup and triage filtering
+5. **Source path format** — `aws-sm://{region}/{secret}`, `azure-kv://{cert-name}/{version}`, `gcp-sm://{project}/{secret}`
+6. **No new schema** — Reuses existing `discovered_certificates` and `discovery_scans` tables. Sentinel agent IDs leverage existing `(fingerprint_sha256, agent_id, source_path)` dedup constraint
+
+**Common triage workflow (all sources):**
 
 1. **Storage** — Records stored in `discovered_certificates` table with status = "Unmanaged"
 2. **Audit** — `discovery_scan_completed` event logged with agent ID, cert count, scan timestamp
@@ -926,29 +1135,53 @@ flowchart TB
 
 This data flow is pull-based and non-blocking. Agents discover at their own pace; the server stores results for later review. There's no pressure to claim or dismiss; operators can leave certificates in "Unmanaged" status indefinitely.
 
+## Continuous TLS Health Monitoring (M48)
+
+Beyond one-time discovery, certctl continuously monitors TLS endpoints for certificate health using a shared TLS probing package and a state-machine-driven health check service. Endpoints transition between states (Healthy → Degraded → Down) based on consecutive failures, and `cert_mismatch` status alerts when a deployed certificate is unexpectedly replaced.
+
+**Architecture:** Probing is extracted into a shared `internal/tlsprobe/` package used by both the network scanner (M21) and the health monitor. The `HealthCheckService` manages 8 API endpoints for CRUD operations and state transitions. A dedicated opt-in endpoint health scheduler loop runs every 60 seconds (configurable via `CERTCTL_HEALTH_CHECK_INTERVAL`). Individual health check targets have their own check intervals (default 300 seconds) — the scheduler queries only endpoints due for check via `ListDueForCheck()`. Results are stored with historical tracking for 30 days (configurable via `CERTCTL_HEALTH_CHECK_HISTORY_RETENTION`). State transitions trigger notifications (critical for down endpoints, warning for degraded, high for cert_mismatch).
+
+**State Machine:** Healthy → Degraded (configurable threshold, default 2 consecutive failures) → Down (default 5 failures). The `cert_mismatch` status is special — it fires whenever the observed certificate fingerprint differs from the expected (deployed) fingerprint, catching silent rollbacks and unauthorized cert replacements. Recovery from degraded/down transitions back to healthy and resets the failure counter.
+
+**API:** 8 endpoints for list (with filters: status, certificate_id, network_scan_target_id, enabled), get, create, update, delete, history (with limit param), acknowledge (incident marking), and summary (aggregate status counts).
+
+**Auto-Create:** When a deployment job completes with successful verification (M25), the system automatically creates a health check with the deployed certificate's fingerprint as the expected value. Network scan targets can also opt-in to auto-create health checks for discovered endpoints.
+
+**Configuration:**
+
+| Env Var | Default | Description |
+|---|---|---|
+| `CERTCTL_HEALTH_CHECK_ENABLED` | `false` | Enable/disable the feature |
+| `CERTCTL_HEALTH_CHECK_INTERVAL` | `60s` | Scheduler tick interval |
+| `CERTCTL_HEALTH_CHECK_DEFAULT_INTERVAL` | `300s` | Default per-endpoint check interval (5 min) |
+| `CERTCTL_HEALTH_CHECK_DEFAULT_TIMEOUT` | `5000ms` | TLS connection timeout per probe |
+| `CERTCTL_HEALTH_CHECK_MAX_CONCURRENT` | `20` | Max concurrent TLS probes |
+| `CERTCTL_HEALTH_CHECK_HISTORY_RETENTION` | `30 days` | Purge probe history older than this |
+| `CERTCTL_HEALTH_CHECK_AUTO_CREATE` | `true` | Auto-create checks from deployments |
+
 ## Testing Strategy
 
-certctl uses a layered testing approach aligned with the handler → service → repository architecture, with 1050+ tests across six layers (service, handler, integration, connector, frontend, and scheduler). The goal is high-confidence regression prevention at the service and handler layers, where the most complex business logic lives, combined with integration tests that exercise the full request path from HTTP to database.
+certctl is extensively tested across eight layers with CI-enforced coverage gates that act as regression floors. The goal is high-confidence regression prevention at the service and handler layers (where the most complex business logic lives), combined with integration tests that exercise the full request path from HTTP to database.
 
-**Service layer unit tests** (`internal/service/*_test.go`) — ~238 test functions across 15 files with mock repositories. These test all business logic in isolation: certificate CRUD with validation, certificate revocation (success, already-revoked, archived, invalid reason, all RFC 5280 reason codes, issuer notification, notification service integration, OCSP/CRL generation), agent lifecycle (registration, heartbeat, CSR submission with both keygen modes), job state machine (creation, processing, cancellation, retry logic), policy evaluation (all 5 rule types, violation creation), renewal and issuance flow (server-side and agent-side keygen paths), notification deduplication (threshold tag matching, channel routing), team/owner/agent group CRUD with pagination and audit recording, issuer service CRUD with connection testing, and the issuer connector adapter (type translation between connector and service layers including revocation). Mock repositories are simple structs with function fields, avoiding heavy mocking frameworks — this keeps tests readable and avoids coupling to mock library APIs.
+**Service layer unit tests** (`internal/service/*_test.go`) — Mock-based tests across all service files covering certificate CRUD, revocation (all RFC 5280 reason codes, OCSP/CRL generation, bulk revocation by filter with partial-failure tolerance), agent lifecycle, job state machine, policy evaluation, renewal/issuance flow (both keygen modes), notification deduplication, team/owner/agent group CRUD, issuer service CRUD with connection testing, and the issuer connector adapter. Mock repositories are simple structs with function fields — no heavy mocking frameworks.
 
-**Handler layer tests** (`internal/api/handler/*_test.go`) — ~257 test functions across 11 files using Go's `httptest` package. Every handler file has a corresponding test file: certificates (50 tests including revocation, DER CRL, and OCSP), agents (28 tests), jobs (21 tests including approve/reject), notifications (11 tests), policies (19 tests), profiles (18 tests), issuers (17 tests), targets (17 tests), agent groups (12 tests), teams (26 tests), and owners (21 tests). Each test file follows the same pattern: a mock service struct with function fields, `httptest.NewRecorder` for capturing responses, and a shared `contextWithRequestID()` helper. Tests cover the happy path, input validation (missing fields, invalid JSON, empty IDs, name length limits), error propagation from the service layer, method-not-allowed responses, and pagination parameters.
+**Handler layer tests** (`internal/api/handler/*_test.go`) — Every handler file has a corresponding test file using Go's `httptest` package: certificates (including revocation, bulk revocation by profile/owner/agent/issuer, DER CRL, OCSP), agents, jobs (including approve/reject), notifications, policies, profiles, issuers, targets, agent groups, teams, owners, discovery, network scan, verification, export, EST, digest, stats, and metrics. Tests cover the happy path, input validation, error propagation, method-not-allowed, pagination, and bulk operation partial-failure scenarios.
 
-**Integration tests** (`internal/integration/`) — Two test files exercising the full stack from HTTP request through router, handler, service, and postgres repository layers. `lifecycle_test.go` has 11 subtests covering the complete certificate lifecycle: team/owner creation, certificate creation, issuer verification, renewal trigger, job verification, agent registration, CSR submission, deployment, and status reporting. `negative_test.go` has 14 subtests covering error paths, 19 M11b endpoint tests, and 8 revocation endpoint tests (M15a+M15b): nonexistent resource lookups (404s), invalid request bodies (malformed JSON, missing required fields), invalid CSR submission, heartbeat for nonexistent agents, wrong HTTP methods on list endpoints, empty list responses, renewal on nonexistent certificates, expired certificate lifecycle, team/owner/agent group CRUD validation, revocation success, already-revoked rejection, not-found revocation, JSON CRL retrieval, DER CRL retrieval, OCSP response retrieval, and short-lived cert exemption. Both use a shared `setupTestServer()` that builds a fully-wired server with real postgres repositories and the Local CA issuer connector. A third file, `e2e_test.go`, contains 8 cross-milestone test functions with 48+ subtests that exercise features across milestones end-to-end: M10 agent metadata via heartbeat, M11 profiles/teams/owners/agent-groups CRUD, M12 issuer registry verification, M13 GUI operation endpoints, M14 stats and metrics, M15 revocation and CRL, M16 notification channels, and M20 enhanced query API (sorting, cursor pagination, sparse fields, time-range filters).
+**Integration tests** (`internal/integration/`) — Three test files exercising the full stack from HTTP request through router, handler, service, and repository layers. `lifecycle_test.go` covers the complete certificate lifecycle (team/owner creation through deployment and status reporting). `negative_test.go` covers error paths, endpoint validation, and revocation scenarios. `e2e_test.go` exercises cross-milestone features end-to-end (agent metadata, profiles, issuer registry, GUI operations, stats, revocation, notifications, enhanced query API).
 
-**Frontend tests** (`web/src/api/client.test.ts`, `web/src/api/utils.test.ts`) — 86 Vitest tests covering the API client, stats/metrics endpoints, and utility functions. The API client tests mock `globalThis.fetch` and verify all endpoint functions (certificates, agents, jobs, policies, issuers, targets, notifications, audit, stats, metrics, health) send correct HTTP methods, URLs, headers, and request bodies. They also test API key management (store/retrieve/clear), auth header propagation, 401 event dispatching, and error handling (server messages, error fields, status text fallback). The stats/metrics endpoint tests verify correct query parameter handling and response shape validation. The utility tests use `vi.useFakeTimers()` for deterministic date testing and cover `formatDate`, `formatDateTime`, `timeAgo`, `daysUntil`, and `expiryColor`. The test environment uses jsdom with `@testing-library/jest-dom` matchers.
+**Go integration tests** (`deploy/test/integration_test.go`) — Runs against the live Docker Compose test environment with real CA backends (Local CA, Pebble ACME, step-ca). Covers health checks, agent heartbeat, issuance, renewal, revocation, CRL/OCSP, EST enrollment, S/MIME, discovery, network scanning, and deployment verification using `crypto/x509` for cert parsing and `crypto/tls` for live TLS verification.
 
-**CLI tests** (`internal/cli/client_test.go`) — 14 tests covering all 10 CLI subcommands with httptest mock servers, PEM parsing for bulk import, auth header verification, and JSON/table output formatting.
+**Frontend tests** (`web/src/api/`) — Vitest tests covering the full API client (all endpoint functions with fetch mocking), stats/metrics endpoints, utility functions, and auth flows. Test environment uses jsdom with `@testing-library/jest-dom` matchers.
 
-**CI pipeline** (`.github/workflows/ci.yml`) — Two parallel jobs: Go (build, vet, race detection, static analysis, vulnerability scanning, test with coverage, coverage threshold enforcement) and Frontend (TypeScript type check, Vitest test suite, Vite production build). The Go job runs `go test -race` on service, handler, middleware, and scheduler packages to catch data races. It runs `golangci-lint` with 11 linters (errcheck, govet, staticcheck, unused, gosimple, ineffassign, typecheck, gocritic, gosec, bodyclose, noctx) configured in `.golangci.yml`. It runs `govulncheck ./...` to scan dependencies for known CVEs. Coverage thresholds are enforced per-layer: service 60%, handler 60%, domain 40%, middleware 50%. These thresholds act as regression floors — they can only go up. Connector tests are included via `./internal/connector/issuer/...` and `./internal/connector/target/...` (covers Local CA, ACME, step-ca, NGINX, Apache, HAProxy, Traefik, and Caddy packages with unit tests for certificate signing logic, DNS solver, issuer validation, and deployment flows). The Frontend job runs `npx vitest run` between the TypeScript check and production build steps.
+**Connector tests** (`internal/connector/`) — Issuer connectors (Local CA self-signed/sub-CA modes, ACME DNS-01/DNS-PERSIST-01, step-ca, OpenSSL, Vault PKI, DigiCert, Sectigo, Google CAS, AWS ACM PCA — all with httptest mock servers or injectable interface mocks). Target connectors (NGINX, Apache, HAProxy, Traefik, Caddy, Envoy, IIS with mock PowerShell executor, F5 BIG-IP with mock iControl client, Postfix/Dovecot, SSH with mock SSH client, Windows Certificate Store with mock PowerShell executor, Java Keystore with mock command executor, Kubernetes Secrets with mock K8s client, shared certutil package). Notifier connectors (Slack, Teams, PagerDuty, OpsGenie).
 
-**Connector tests** (`internal/connector/`) — 57 test functions covering issuer, target, and notifier connectors. The Local CA connector has tests for self-signed and sub-CA modes (RSA, ECDSA, config validation, non-CA cert rejection). The ACME DNS solver has 10 tests for script-based DNS-01 and DNS-PERSIST-01 challenges (6 DNS-01 tests + 4 DNS-PERSIST-01 tests covering `PresentPersist` success, no-script error, script failure, and wildcard domain handling). The step-ca connector has tests with a mock HTTP server for issuance, renewal, revocation, and error paths. The OpenSSL/Custom CA connector has 14 tests covering config validation, issuance success/failure/timeout, renewal, revocation, and CRL generation. The NGINX target connector has 13 tests covering config validation, certificate deployment (file writing, permissions, validate/reload commands), and deployment validation. Apache httpd and HAProxy connectors each have 3 tests covering config validation, deployment, and validation flows. Traefik and Caddy connectors have tests covering file-based deployment and (for Caddy) dual-mode API/file configuration. Notifier connector tests span 20 tests across Slack (5), Teams (4), PagerDuty (6), and OpsGenie (5) — verifying channel identity, payload formatting, HTTP error handling, connection failures, auth headers, and configuration defaults.
+**Scheduler tests** (`internal/scheduler/scheduler_test.go`) — Idempotency guards (`sync/atomic.Bool`), `WaitForCompletion` success and timeout paths, and multi-loop concurrency safety.
 
-**Scheduler tests** (`internal/scheduler/scheduler_test.go`) — Tests for idempotency guards (`sync/atomic.Bool` CompareAndSwap prevents concurrent loop ticks), `WaitForCompletion` success and timeout paths, and multi-loop idempotency.
+**Fuzz tests** (`internal/validation/`, `internal/domain/`) — Go native fuzz tests for command validation (`ValidateShellCommand`, `ValidateDomainName`, `ValidateACMEToken`) and revocation domain parsing.
 
-**Fuzz tests** (`internal/validation/command_fuzz_test.go`, `internal/domain/revocation_fuzz_test.go`) — Go native fuzz tests (`testing/fuzz`) for command validation functions and revocation domain parsing. These exercise `ValidateShellCommand`, `ValidateDomainName`, and `ValidateACMEToken` with random inputs to discover edge cases.
+**CI pipeline** (`.github/workflows/ci.yml`) — Two parallel jobs. Go: build, vet, `go test -race`, `golangci-lint` (11 linters), `govulncheck`, test with coverage, per-layer coverage threshold enforcement (service 55%, handler 60%, domain 40%, middleware 30%). Frontend: TypeScript type check, Vitest, Vite production build.
 
-**What's not tested and why:** Postgres repository implementations (`internal/repository/postgres/`) require a real database and are tested only through integration tests, not unit tests — a `testcontainers-go` scaffolding for isolated PostgreSQL instances is planned. Target connectors for F5 BIG-IP and IIS are interface stubs (implementation planned for V3). The ACME connector requires a real ACME server (tested manually against Let's Encrypt staging). These are all candidates for future expansion as the test infrastructure matures.
+For detailed test procedures, smoke tests, and the release sign-off checklist, see the [Testing Guide](testing-guide.md). For setting up the Docker Compose test environment with real CA backends, see [Test Environment](test-env.md).
 
 ## What's Next
 
@@ -958,3 +1191,5 @@ certctl uses a layered testing approach aligned with the handler → service →
 - [Compliance Mapping](compliance.md) — SOC 2, PCI-DSS 4.0, and NIST SP 800-57 alignment
 - [MCP Server Guide](mcp.md) — AI-native access to the API
 - [OpenAPI Spec](openapi.md) — Full API reference and SDK generation
+- [Testing Guide](testing-guide.md) — Test procedures and release sign-off
+- [Test Environment](test-env.md) — Docker Compose test environment setup

docs/certctl-for-cert-manager-users.md

@@ -0,0 +1,145 @@
+# certctl for cert-manager Users
+
+You run cert-manager inside Kubernetes and it works well for in-cluster certificates. But you also have VMs, bare-metal servers, network appliances, and legacy systems outside the cluster. cert-manager can't reach those. This guide shows how certctl complements cert-manager to give you unified certificate visibility and automation across your entire infrastructure.
+
+## Not a Replacement
+
+cert-manager is the right tool for in-cluster certs. It's tightly integrated with Kubernetes:
+- Native CRDs (Certificate, ClusterIssuer, Issuer)
+- Automatic cert injection into Ingress and Service objects
+- Controller-driven renewal within the cluster
+
+**certctl does not replace this.** Instead, it extends your certificate management to everything outside Kubernetes: VMs, bare metal, network appliances, Windows servers, and legacy systems.
+
+## The Problem
+
+Your setup:
+- **cert-manager**: handles all certs in Kubernetes (TLS for Ingress, service-to-service, internal services)
+- **Everything else**: NGINX/Apache on VMs, HAProxy load balancers on bare metal, network appliances, Windows servers with IIS — these are managed inconsistently. Maybe Certbot cron jobs, maybe manual renewal, maybe deprecated cert files sitting around.
+
+Result:
+- No unified visibility — you don't know when non-Kubernetes certs expire
+- Renewal failures go unnoticed until the cert is already expired
+- Audit trail fragmented across multiple tools
+- Scaling to hundreds of machines becomes impossible
+
+## The Solution
+
+Deploy certctl control plane once (Docker Compose, Kubernetes Helm chart, or self-hosted). Deploy agents on your VMs, bare metal, and network appliances. One dashboard shows:
+- **All cert-manager certs** via discovery scanning (agents find cert-manager-issued certs copied to target machines, or scan the cluster directly)
+- **All certctl-managed certs** issued by shared issuers (ACME, step-ca, Vault PKI (planned), private CA)
+- **Unified renewal and deployment** across both worlds
+- **Single pane of glass** with expiration timeline, renewal status, deployment verification, audit trail
+
+## How to Set Up
+
+### 1. Install certctl Control Plane
+
+**Option A: Docker Compose** (quickest for evaluation)
+```bash
+cd /opt/certctl
+docker compose up -d
+# Dashboard & API: https://localhost:8443 (self-signed cert — pin with --cacert ./deploy/test/certs/ca.crt)
+```
+
+**Option B: Kubernetes** (recommended for prod)
+```bash
+helm install certctl deploy/helm/certctl/ \
+  --set auth.apiKey=YOUR_SECURE_KEY
+```
+
+### 2. Deploy Agents to Non-Kubernetes Infrastructure
+
+On each VM, bare-metal server, or appliance (via proxy agent):
+```bash
+# Linux amd64
+curl -sSL https://github.com/shankar0123/certctl/releases/download/v2.1.0/certctl-agent-linux-amd64 \
+  -o /usr/local/bin/certctl-agent
+chmod +x /usr/local/bin/certctl-agent
+
+# Config
+sudo tee /etc/certctl/agent.env > /dev/null <<EOF
+CERTCTL_SERVER_URL=https://certctl-control-plane:8443
+CERTCTL_SERVER_CA_BUNDLE_PATH=/etc/certctl/tls/ca.crt
+CERTCTL_API_KEY=your-api-key
+CERTCTL_DISCOVERY_DIRS=/etc/nginx/certs,/etc/ssl,/etc/letsencrypt/live
+CERTCTL_KEY_DIR=/var/lib/certctl/keys
+EOF
+sudo chmod 600 /etc/certctl/agent.env
+
+# Start
+sudo systemctl start certctl-agent
+```
+
+### 3. Enable Discovery Scanning
+
+Agents scan configured directories and report back all existing certs. In the dashboard:
+- **Discovery** page: all found certs grouped by agent
+- Claim cert-manager certs to link them with Kubernetes metadata
+- Dismiss obsolete certs
+
+### 4. Configure Shared Issuers
+
+Set up the same issuer certctl uses for non-Kubernetes certs:
+- **ACME** (Let's Encrypt, for public certs)
+- **step-ca** (Smallstep, for internal certs)
+- **Vault PKI** (HashiCorp Vault, for enterprise PKI)
+- **Private CA** (your own internal root CA)
+
+No new CA infrastructure needed. If cert-manager already uses your CA, certctl points to the same one.
+
+### 5. Create Policies for Non-Kubernetes Certs
+
+Go to **Policies** → **+ New Policy** to create enforcement rules:
+- **Name:** e.g., "VM Certificate Policy"
+- **Type:** `expiration_window` or `key_algorithm` (enforce renewal thresholds or crypto requirements)
+- **Severity:** `high`
+- **Config:** set your enforcement parameters
+
+Certificates are linked to issuers and profiles when created or claimed from discovery. Policies add guardrails — enforcing key algorithm requirements, expiration windows, and other compliance rules across your fleet.
+
+### 6. View Unified Inventory
+
+**Dashboard** shows:
+- Certificate status heatmap (all 1000 certs: cert-manager + certctl)
+- Renewal job trends (both types)
+- Expiration timeline (30/60/90 days)
+- Agent fleet status (all infrastructure)
+
+**Certificates** page filters by issuer (show me all ACME certs, or all step-ca certs):
+- cert-manager certs discovered from Kubernetes nodes
+- certctl-managed certs on VMs
+- Network appliance certs auto-discovered
+
+## Shared Infrastructure
+
+If cert-manager and certctl both use the same CA:
+- **ACME**: cert-manager uses ClusterIssuer + certctl uses ACME connector → same Let's Encrypt account, transparent coexistence
+- **step-ca**: cert-manager uses external issuer CRD + certctl uses step-ca connector → same provisioner, shared certificate inventory
+- **Vault PKI**: cert-manager uses external issuer CRD + certctl uses Vault connector → same mount, same audit trail
+
+No conflict. They just issue certs through the same CA. certctl's discovery scanning finds cert-manager-issued certs and shows them alongside certctl-managed ones.
+
+## Key Differences from cert-manager
+
+| Feature | cert-manager | certctl |
+|---------|--------------|---------|
+| Target | In-cluster (Kubernetes) | Out-of-cluster (VMs, bare metal, appliances) |
+| Configuration | CRDs (Certificate, ClusterIssuer, Issuer) | API + Dashboard (JSON REST) |
+| Deployment | Injected into Secret objects, mounted by pods | Agent pulls work, deploys via target-specific API (file, service restart, proxy agent) |
+| Renewal | Controller watches Certificate CRDs, triggers renewal when needed | Scheduler checks thresholds, agents poll for work |
+| Audit | Kubernetes event log | Immutable append-only audit trail |
+| Visibility | Per-namespace, per-resource | Fleet-wide, unified inventory |
+
+## Future Integration
+
+On the roadmap (V4): **cert-manager external issuer** — certctl acts as a ClusterIssuer backend for Kubernetes. This would allow cert-manager to request certificates from certctl, which could issue them via any of its connectors (step-ca, Vault, private CA, etc.). Pure integration play; no breaking changes.
+
+For now: cert-manager handles Kubernetes, certctl handles everything else. They coexist seamlessly.
+
+## Next Steps
+
+1. Run through the [Quick Start](./quickstart.md) for a 5-minute demo
+2. Try the [Multi-Issuer example](../examples/multi-issuer/multi-issuer.md) — manages public and internal certs from one dashboard
+3. Explore [Architecture](./architecture.md#agents) for deployment patterns
+4. Check the [Helm Chart](../deploy/helm/certctl/) for production Kubernetes deployment

docs/compliance-nist.md

@@ -72,7 +72,7 @@ certctl implements tiered key storage with different protection profiles based o
 - Configured via: `CERTCTL_CA_CERT_PATH=/path/to/ca.crt` and `CERTCTL_CA_KEY_PATH=/path/to/ca.key`
 
 **NIST Gap: HSM Storage**
-NIST SP 800-57 Part 1 recommends Hardware Security Module (HSM) storage for high-value keys (CA signing keys). certctl V2 uses filesystem storage on the server. HSM support is planned for V5 roadmap, enabling integration with:
+NIST SP 800-57 Part 1 recommends Hardware Security Module (HSM) storage for high-value keys (CA signing keys). certctl V2 uses filesystem storage on the server. HSM support is planned for certctl Pro (V3), enabling integration with:
 - AWS CloudHSM
 - Azure Dedicated HSM
 - Thales Luna, Gemalto SafeNet, YubiHSM (on-premises)
@@ -210,15 +210,17 @@ NIST SP 800-57 Part 1 Section 6.2 addresses secure key distribution to minimize
   - Proxy agent executes deployment via appliance API
 
 **Revocation Distribution**
-- Certificate Revocation List (CRL) via `GET /api/v1/crl/{issuer_id}`
-  - Returns DER-encoded X.509 CRL signed by issuing CA
+- Certificate Revocation List (CRL) via `GET /.well-known/pki/crl/{issuer_id}` (RFC 5280 §5, RFC 8615)
+  - Returns DER-encoded X.509 CRL signed by issuing CA (`Content-Type: application/pkix-crl`)
   - 24-hour validity period
   - Includes all revoked serials, reasons, and revocation timestamps
+  - Served unauthenticated so relying parties without certctl API credentials can fetch it
   - Subject to URL caching; OCSP preferred for real-time revocation
-- OCSP via `GET /api/v1/ocsp/{issuer_id}/{serial}`
-  - Returns DER-encoded OCSP response (OCSPResponse ASN.1 structure)
+- OCSP via `GET /.well-known/pki/ocsp/{issuer_id}/{serial}` (RFC 6960)
+  - Returns DER-encoded OCSP response (OCSPResponse ASN.1 structure, `Content-Type: application/ocsp-response`)
   - Signed by issuing CA (or delegated OCSP signing cert)
   - Responds with good/revoked/unknown status
+  - Served unauthenticated — the RFC 6960 relying-party model does not assume API credentials
   - Real-time, more bandwidth-efficient than CRL polling
 
 ## Revocation and Compromise (NIST SP 800-57 Part 3)
@@ -272,20 +274,23 @@ NIST SP 800-57 Part 3 covers revocation (Section 2.5) when keys are suspected co
 - OCSP responder queries revocation table in real-time
 - Short-lived certificate exemption: certs with TTL < 1 hour skip CRL/OCSP (expiry is sufficient revocation)
 
+**Bulk Revocation for Large-Scale Compromise Response** (V2.2) — NIST SP 800-57 Part 3 emphasizes rapid revocation when keys are compromised. `POST /api/v1/certificates/bulk-revoke` revokes all certificates matching filter criteria (profile, owner, agent, issuer) in a single operation. This enables operators to execute fleet-wide revocation for key compromise events affecting multiple certificates. Each bulk revocation creates individual jobs reusing the existing revocation pipeline, ensuring every certificate is recorded in the audit trail with the incident reason.
+
 **Revocation Audit Trail**
 All revocation events logged:
-- Event type: `certificate_revoked`
+- Event type: `certificate_revoked` or `bulk_revocation_initiated` (for fleet operations)
 - Actor: authenticated user or service
-- Reason code: RFC 5280 enum
+- Reason code: RFC 5280 enum (or incident justification for bulk operations)
 - Timestamp: RFC3339
 - Issuer notification status: success or error reason
+- Filter criteria: profile_id, owner_id, agent_id, issuer_id (for bulk revocation)
 
 ## Alignment Summary Table
 
 | NIST SP 800-57 Area | Status | Coverage | Notes |
 |---|---|---|---|
 | **Key Generation** | ✅ Aligned | 100% | Agent-side ECDSA P-256 using crypto/rand; server mode flagged as demo-only |
-| **Key Storage** | ⚠️ Partially Aligned | 80% | Filesystem with 0600 perms; HSM support planned V5 |
+| **Key Storage** | ⚠️ Partially Aligned | 80% | Filesystem with 0600 perms; HSM support planned V3 Pro |
 | **Cryptoperiods** | ✅ Aligned | 100% | Profile-enforced max_ttl; threshold-based renewal alerting |
 | **Key States** | ✅ Aligned | 100% | Full lifecycle tracking with immutable audit trail |
 | **Algorithms** | ✅ Aligned | 100% | NIST-approved algorithms only; post-quantum tracking in progress |
@@ -301,13 +306,14 @@ All revocation events logged:
 - [x] RFC 5280 revocation support
 - [x] Immutable audit trail
 
+### V2.2 (Planned: 2026)
+- Bulk revocation by profile/owner/agent/issuer (fleet-level revocation for incident response)
+
 ### V3 (Planned: 2026)
 - Role-based access control (limit revocation/approval to authorized operators)
-- Bulk revocation by profile/owner/agent (fleet-level revocation policy)
 
-### V5 (Planned: 2027+)
-- HSM support for CA key storage
-- PKCS#11 integration for hardware tokens
+### V3 Pro (Planned)
+- HSM support for CA key storage and agent key storage (TPM 2.0, PKCS#11)
 - FIPS 140-2/3 validated crypto module (BoringCrypto build or external FIPS library)
 - Key destruction API (explicit secure erasure of agent keys)
 - Key escrow / recovery mechanism (backup encrypted private keys for disaster recovery)

docs/compliance-pci-dss.md

@@ -92,9 +92,11 @@ Your QSA will request evidence that your certificate and key management systems
 
 - **Certificate Status Tracking** — Four statuses: Active (deployed, not yet expired), Expiring (within threshold, awaiting renewal), Expired (past not-after date), Revoked (revoked via RFC 5280 revocation API). Dashboard charts show status distribution.
 
-- **Revocation Infrastructure** (M15a, M15b):
-  - CRL endpoint: `GET /api/v1/crl` (JSON format) or `GET /api/v1/crl/{issuer_id}` (DER X.509 CRL, 24h validity, signed by issuing CA)
-  - OCSP responder: `GET /api/v1/ocsp/{issuer_id}/{serial}` (returns DER-encoded OCSP response: good/revoked/unknown)
+- **Revocation Infrastructure** (M15a, M15b, M-006):
+  - Revocation API: `POST /api/v1/certificates/{id}/revoke` with RFC 5280 reason codes
+  - CRL endpoint: `GET /.well-known/pki/crl/{issuer_id}` — DER X.509 CRL, 24h validity, signed by issuing CA, served unauthenticated (RFC 5280 §5, RFC 8615, `Content-Type: application/pkix-crl`)
+  - OCSP responder: `GET /.well-known/pki/ocsp/{issuer_id}/{serial}` — DER-encoded OCSP response (good/revoked/unknown), served unauthenticated (RFC 6960, `Content-Type: application/ocsp-response`)
+  - Bulk revocation (V2.2): `POST /api/v1/certificates/bulk-revoke` with filter criteria (profile, owner, agent, issuer) for fleet-wide incident response
   - Short-lived cert exemption: certs with TTL < 1 hour skip CRL/OCSP (expiry is sufficient revocation)
 
 - **Stats API** (M14) — Real-time visibility:
@@ -107,7 +109,7 @@ Your QSA will request evidence that your certificate and key management systems
 - Discovered certificate report: `GET /api/v1/discovered-certificates` JSON export showing all certs on systems, fingerprints, and status.
 - Managed certificate inventory: `GET /api/v1/certificates` with filters (`?status=Expiring` for upcoming renewals).
 - Expiration alert configuration: policy JSON showing `alert_thresholds_days` for each environment.
-- CRL/OCSP availability proof: HTTP GET requests to `/api/v1/crl` and `/api/v1/ocsp/{issuer}/{serial}` with signed responses.
+- CRL/OCSP availability proof: unauthenticated HTTP GET requests to `/.well-known/pki/crl/{issuer_id}` (DER, `application/pkix-crl`) and `/.well-known/pki/ocsp/{issuer_id}/{serial}` (DER, `application/ocsp-response`) with signed responses.
 - Audit trail for certificate creation/renewal/revocation: `GET /api/v1/audit?type=certificate_issued,certificate_renewed,certificate_revoked`.
 - Dashboard charts showing expiration timeline, renewal success trends, status distribution.
 
@@ -326,11 +328,14 @@ This requirement covers key generation, storage, rotation, and destruction. Cert
   - Issuer notified (best-effort; ACME lacks standard revocation, Local CA skips issuer step).
   - Revocation notifications sent to owner via email/webhook/Slack/Teams/PagerDuty.
 
-- **CRL and OCSP Publication** (M15b) — Revoked certificates published in:
-  - CRL: `GET /api/v1/crl` (JSON format) or `GET /api/v1/crl/{issuer_id}` (DER X.509, signed by CA, 24h validity)
-  - OCSP: `GET /api/v1/ocsp/{issuer_id}/{serial}` (returns revoked status for clients validating certificate chain)
+- **CRL and OCSP Publication** (M15b, M-006) — Revoked certificates published in:
+  - CRL: `GET /.well-known/pki/crl/{issuer_id}` (DER X.509 signed by CA, 24h validity, RFC 5280 §5 + RFC 8615, `Content-Type: application/pkix-crl`)
+  - OCSP: `GET /.well-known/pki/ocsp/{issuer_id}/{serial}` (returns revoked status for clients validating certificate chain, RFC 6960, `Content-Type: application/ocsp-response`)
+  - Both endpoints are served unauthenticated so relying parties (browsers, TLS appliances) without certctl API keys can verify revocation — this is the RFC-compliant PKI model.
   - Clients checking certificate status via OCSP or CRL see revoked status within 24 hours.
 
+- **Bulk Revocation for Incident Response** (V2.2) — `POST /api/v1/certificates/bulk-revoke` with filter criteria (profile, owner, agent, issuer) revokes all matching certificates in a single operation. PCI-DSS Req 4 requires rapid response to data transmission security incidents — bulk revocation enables operators to revoke an entire certificate set (e.g., all certs used by a compromised team or endpoint) in minutes rather than hours.
+
 - **Private Key Destruction on Agent** — When certificate renewed or revoked:
   - Agent removes old private key file from `CERTCTL_KEY_DIR` when new certificate deployed.
   - Job status tracking confirms old key is no longer needed.
@@ -338,8 +343,8 @@ This requirement covers key generation, storage, rotation, and destruction. Cert
 
 **Evidence You Can Provide**:
 - Revocation requests: `GET /api/v1/audit?type=certificate_revoked` with RFC 5280 reason codes.
-- CRL publication: HTTP GET `/api/v1/crl` and parse JSON to show revoked serial numbers and timestamps.
-- OCSP responder validation: Query `GET /api/v1/ocsp/{issuer}/{serial}` for a known-revoked cert; response includes `revoked` status.
+- CRL publication: HTTP GET `/.well-known/pki/crl/{issuer_id}` (unauthenticated) returns a DER X.509 CRL — parse with `openssl crl -inform der -noout -text` to show revoked serial numbers, reasons, and timestamps.
+- OCSP responder validation: Query `GET /.well-known/pki/ocsp/{issuer_id}/{serial}` (unauthenticated) for a known-revoked cert; response includes `revoked` status and can be parsed with `openssl ocsp` tooling.
 - Audit trail: Certificate status transitions (Active → Revoked) recorded in `audit_events`.
 
 **Operator Responsibility**:
@@ -382,12 +387,12 @@ This requirement covers key generation, storage, rotation, and destruction. Cert
   - API key transmitted in Authorization header (not URL parameter, not cookie).
   - Browser to server: TLS.
   - Agent to server: TLS.
-  - No credential logging (API key hash only, never plaintext).
+  - No credential logging (audit records the per-key actor `Name`, never the Bearer token; logs redact the `Authorization` header).
 
 **Evidence You Can Provide**:
 - API configuration: `CERTCTL_AUTH_TYPE=api-key` in deployment manifest.
-- Database schema: `api_keys` table showing SHA-256 hash column, not plaintext.
-- API audit log: `GET /api/v1/audit?action=api_call` showing Bearer token validation (no plaintext keys logged).
+- Key inventory: `CERTCTL_API_KEYS_NAMED` env var (format `name:key:admin,...`) — seeds the in-memory `NamedAPIKey{Name, Key, Admin}` struct at `internal/api/middleware/middleware.go:29`. Keys are constant-time-compared (`subtle.ConstantTimeCompare`) against the Bearer token. No database table stores them; protect the env var contents at rest via a secrets manager (Vault / AWS Secrets Manager / Kubernetes Secrets / Docker Secrets).
+- API audit log: `GET /api/v1/audit?action=api_call` showing per-key actor names (`Name` field of matched `NamedAPIKey`) on every call, with zero plaintext or hashed key material recorded.
 - TLS certificate on control plane: `openssl s_client -connect {server}:8443` showing valid certificate, TLS 1.2+, strong cipher.
 - GUI login flow: browser network tab showing Authorization header (token value redacted in compliance report).
 
@@ -557,6 +562,7 @@ This requirement covers key generation, storage, rotation, and destruction. Cert
 - **Alert Notifications** (M3, M16a) — Configurable escalation:
   - Email alerts: certificate approaching expiration, renewal failure, revocation notification.
   - Webhook: custom HTTP POST to your monitoring system (Slack, Teams, PagerDuty, OpsGenie, custom webhook).
+  - **Retry & Dead-Letter Queue** (I-005) — Transient notifier failures (SMTP timeout, webhook 5xx) are retried with exponential backoff (`2^n` minutes capped at 1h, 5-attempt budget) before landing in the terminal `dead` status. Operators monitor DLQ depth via the `certctl_notification_dead_total` Prometheus counter and requeue via the Notifications page Dead letter tab once the underlying outage is resolved. Closes the pre-I-005 silent-drop gap where a single 5xx could lose a compliance-relevant alert without evidence.
   - Deduplication: one alert per threshold/certificate per day (avoid alert fatigue).
 
 - **Audit Trail Filtering and Export** (M13) — Compliance reporting:
@@ -717,12 +723,12 @@ This requirement covers key generation, storage, rotation, and destruction. Cert
 | PCI-DSS Requirement | certctl Feature | API/UI Evidence | Database/Config | Audit Trail | Status |
 |---|---|---|---|---|---|
 | **4.2.1** Strong Crypto | TLS cert issuance, ACME/step-ca/Local CA, RSA 2048+/ECDSA P-256 | `GET /api/v1/certificates` (key_type, key_size) | Certificate profiles | `GET /api/v1/audit?type=certificate_issued` | Available |
-| **4.2.2** Cert Inventory & Validation | Managed cert CRUD, discovery (M18b), expiration alerting, CRL/OCSP | `GET /api/v1/certificates`, `GET /api/v1/discovered-certificates`, `GET /api/v1/crl`, `GET /api/v1/ocsp/{issuer}/{serial}` | `managed_certificates`, `discovered_certificates` tables | `GET /api/v1/audit?type=certificate_*` | Available |
+| **4.2.2** Cert Inventory & Validation | Managed cert CRUD, discovery (M18b), expiration alerting, CRL/OCSP | `GET /api/v1/certificates`, `GET /api/v1/discovered-certificates`, `GET /.well-known/pki/crl/{issuer_id}`, `GET /.well-known/pki/ocsp/{issuer_id}/{serial}` (both unauthenticated, RFC 5280 / RFC 6960) | `managed_certificates`, `discovered_certificates` tables | `GET /api/v1/audit?type=certificate_*` | Available |
 | **3.6** Key Documentation | Profiles, owner/team tracking, issuer config, audit trail | `GET /api/v1/profiles`, `GET /api/v1/issuers`, certificate detail with owner/team | Profiles, certificate owner/team fields, issuer config | `GET /api/v1/audit?resource_type=certificate` | Available |
 | **3.7.1** Key Generation | Agent-side ECDSA P-256, server keygen (demo only) | Agent logs, renewal job detail, CSR audit | `CERTCTL_KEYGEN_MODE=agent` (config), job_type=AwaitingCSR | `GET /api/v1/audit?type=certificate_issued` with CSR hash | Available |
 | **3.7.2** Key Storage | Agent `/var/lib/certctl/keys` (0600), env var secrets, .env excluded | Deployment manifest (env var refs), agent key dir listing | `.env` file (git-ignored), `CERTCTL_KEY_DIR`, `CERTCTL_CA_KEY_PATH` | No API audit (keys off-platform) | Available |
 | **3.7.3** Key Rotation | Auto renewal, expiration thresholds, renewal jobs | Dashboard renewal trends, `GET /api/v1/jobs?type=Renewal`, certificate versions | Renewal policies, certificate version history | `GET /api/v1/audit?type=certificate_renewed` | Available |
-| **3.7.4** Key Destruction | Revocation API (RFC 5280), CRL/OCSP, private key cleanup | `POST /api/v1/certificates/{id}/revoke`, `GET /api/v1/crl`, OCSP endpoint | `certificate_revocations` table, CRL publication | `GET /api/v1/audit?type=certificate_revoked` | Available |
+| **3.7.4** Key Destruction | Revocation API (RFC 5280), CRL/OCSP, private key cleanup | `POST /api/v1/certificates/{id}/revoke`, unauthenticated `GET /.well-known/pki/crl/{issuer_id}` and `GET /.well-known/pki/ocsp/{issuer_id}/{serial}` | `certificate_revocations` table, CRL publication | `GET /api/v1/audit?type=certificate_revoked` | Available |
 | **8.3** Strong Authentication | API key (SHA-256 hash, TLS), GUI login, 401 redirect | GUI login screenshot, API key auth header, TLS cert | API key hash in database | `GET /api/v1/audit` showing API calls | Available |
 | **8.6** Acct Management | Credentials out of source, .env excluded, env var config | Code review (no hardcoded secrets), `.gitignore` check | Deployment manifests showing env var refs only | No account lifecycle audit (outside scope) | Available in part |
 | **10.2** Audit Logging | API audit middleware (M19), certificate lifecycle events | `GET /api/v1/audit` with filter/pagination | `audit_events` table (every API call) | Real-time via API | Available |

docs/compliance-soc2.md

@@ -44,7 +44,8 @@ Each section includes:
 
 **certctl Implementation** (V2 — Community Edition):
 
-- **API Key Authentication** — All API calls require a Bearer token (hashed with SHA-256, stored securely, validated with constant-time comparison) or are rejected with 401 Unauthorized. Environment: `CERTCTL_AUTH_TYPE` (default `api-key`; `none` requires explicit opt-in with log warning)
+- **API Key Authentication** — All `/api/v1/*` calls require a Bearer token (hashed with SHA-256, stored securely, validated with constant-time comparison) or are rejected with 401 Unauthorized. Environment: `CERTCTL_AUTH_TYPE` (default `api-key`; `none` requires explicit opt-in with log warning)
+- **Standards-based enrollment and PKI distribution endpoints** — EST (`/.well-known/est/*`, RFC 7030), SCEP (`/scep`, `/scep/*`, RFC 8894), and CRL/OCSP (`/.well-known/pki/crl/{issuer_id}`, `/.well-known/pki/ocsp/{issuer_id}/{serial}`, RFC 5280 §5 / RFC 6960 / RFC 8615) are served unauthenticated at the HTTP layer because these protocols cannot present certctl Bearer tokens. Authentication is enforced in-protocol: EST relies on CSR signature verification plus profile policy (RFC 7030 §3.2.3 says EST auth is deployment-specific; §4.1.1 makes `/cacerts` explicitly anonymous); SCEP requires a shared `challengePassword` in the PKCS#10 CSR attributes (OID 1.2.840.113549.1.9.7, RFC 8894 §3.2), validated with `crypto/subtle.ConstantTimeCompare`; CRL and OCSP are intentionally anonymous for relying-party accessibility. CWE-306 (missing authentication for a critical function) is closed for SCEP by `preflightSCEPChallengePassword` in `cmd/server/main.go`, which refuses to start the control plane when `CERTCTL_SCEP_ENABLED=true` is set without `CERTCTL_SCEP_CHALLENGE_PASSWORD`. The HTTP dispatch is implemented in `cmd/server/main.go:buildFinalHandler`, which routes these prefixes through `noAuthHandler` (RequestID + structuredLogger + Recovery only, no auth or rate-limit middleware) and is pinned by the 27-subtest regression harness at `cmd/server/finalhandler_test.go`.
 - **GUI Authentication** — Web dashboard includes login screen requiring API key entry. Failed auth redirects to login on 401. Auth context persists across page navigation. Logout clears session.
 - **Configurable CORS** — API restricts cross-origin requests via `CERTCTL_CORS_ORIGINS` allowlist or wildcard. Preflight caching prevents chatty browser auth flows.
 - **Token Bucket Rate Limiting** — Per-IP rate limiting (configurable via `CERTCTL_RATE_LIMIT_RPS` / `CERTCTL_RATE_LIMIT_BURST`) returns 429 Too Many Requests with Retry-After header. Prevents credential stuffing and brute-force attacks.
@@ -58,6 +59,11 @@ Each section includes:
 - Auth info endpoint: `GET /api/v1/auth/info` (returns current auth mode, served without auth so GUI detects mode)
 - Rate limiting middleware: `internal/api/middleware/rate_limit.go`
 - CORS configuration: `cmd/server/main.go`, search for `CERTCTL_CORS_ORIGINS`
+- Final handler dispatch (authenticated vs. unauthenticated routing): `cmd/server/main.go:buildFinalHandler`
+- SCEP preflight gate (CWE-306 closure): `cmd/server/main.go:preflightSCEPChallengePassword`
+- SCEP service-layer defense-in-depth (rejects enrollment on empty challenge password, `crypto/subtle.ConstantTimeCompare`): `internal/service/scep.go`
+- Final handler dispatch regression harness (27 subtests): `cmd/server/finalhandler_test.go`
+- OpenAPI spec `security: []` overrides on unauthenticated paths: `api/openapi.yaml` (EST `/cacerts`, `/simpleenroll`, `/simplereenroll`, `/csrattrs`; SCEP `/scep` GET+POST; PKI `/crl/{issuer_id}`, `/ocsp/{issuer_id}/{serial}`)
 
 **V3 Enhancement**:
 
@@ -110,7 +116,7 @@ Each section includes:
 
 **certctl Implementation** (V2):
 
-- **API Key Policy** — All API access requires an API key or explicit opt-out. Opt-out (`CERTCTL_AUTH_TYPE=none`) logs a warning: "WARNING: Auth disabled (CERTCTL_AUTH_TYPE=none) — this is insecure and only for development". Configuration choice is logged at startup.
+- **API Key Policy** — All `/api/v1/*` access requires an API key or explicit opt-out. Opt-out (`CERTCTL_AUTH_TYPE=none`) logs a warning: "WARNING: Auth disabled (CERTCTL_AUTH_TYPE=none) — this is insecure and only for development". Configuration choice is logged at startup. The standards-based enrollment and PKI distribution endpoints (EST, SCEP, CRL, OCSP) are served unauthenticated at the HTTP layer per their respective RFCs; see CC6.1 for the full authentication contract and CWE-306 closure via `preflightSCEPChallengePassword`.
 - **Agent Authentication** — Agents authenticate to the server via API keys (same mechanism as users). Agent credentials are separate from user API keys.
 - **Private Key Policy** — Agent-side key generation is the default (`CERTCTL_KEYGEN_MODE=agent`). Server-side keygen (`CERTCTL_KEYGEN_MODE=server`) requires explicit configuration and logs a warning: "server-side key generation enabled (CERTCTL_KEYGEN_MODE=server) — private keys touch control plane, demo only".
 - **Password Policy** — Not applicable; certctl uses API keys exclusively. Password management is delegated to your organization's IAM system if you integrate OIDC/SSO (V3).
@@ -183,14 +189,20 @@ Each section includes:
 
 - **Health Endpoint** — `GET /health` returns 200 OK with service status. Consumed by Docker health checks and Kubernetes probes.
 - **Readiness Endpoint** — `GET /ready` returns 200 OK when the database is connected and migrations are applied.
-- **Background Scheduler Monitoring** — 6 background loops run on a fixed schedule:
-  - Renewal loop: every 1 hour, scans for certificates approaching renewal threshold
-  - Job processor loop: every 30 seconds, picks up pending/waiting jobs and advances their state
-  - Health check loop: every 2 minutes, pings agents to detect downtime
-  - Notification dispatcher loop: every 1 minute, sends queued alerts
-  - Short-lived cert expiry loop: every 30 seconds, marks expired short-lived credentials
-  - Network scanner loop: every 6 hours, scans enabled TLS endpoints for certificate discovery
-  Each loop includes error handling and logs failures via structured slog.
+- **Background Scheduler Monitoring** — 12 background loops (8 always-on + 4 opt-in) run on a fixed schedule. Authoritative topology in `docs/architecture.md`:
+  - Renewal loop (always-on, 1 hour): scans for certificates approaching renewal threshold
+  - Job processor loop (always-on, 30 seconds): picks up pending/waiting jobs and advances their state
+  - Job retry loop (always-on, 5 minutes, `CERTCTL_SCHEDULER_RETRY_INTERVAL`): retries Failed jobs (I-001)
+  - Job timeout reaper loop (always-on, 10 minutes, `CERTCTL_JOB_TIMEOUT_INTERVAL`): fails AwaitingCSR/AwaitingApproval jobs past timeout (I-003)
+  - Agent health check loop (always-on, 2 minutes): pings agents to detect downtime
+  - Notification dispatcher loop (always-on, 1 minute): sends queued alerts
+  - Notification retry loop (always-on, 2 minutes, `CERTCTL_NOTIFICATION_RETRY_INTERVAL`): exponential backoff retry for failed notifications; promote to dead-letter after 5 attempts (I-005)
+  - Short-lived cert expiry loop (always-on, 30 seconds): marks expired short-lived credentials
+  - Network scanner loop (opt-in, 6 hours, `CERTCTL_NETWORK_SCAN_ENABLED`): scans enabled TLS endpoints for certificate discovery
+  - Digest emailer loop (opt-in, 24 hours, `CERTCTL_DIGEST_INTERVAL`): sends scheduled certificate digest email to configured recipients
+  - Endpoint health loop (opt-in, 60 seconds, `CERTCTL_HEALTH_CHECK_INTERVAL`): continuous TLS health probes (M48)
+  - Cloud discovery loop (opt-in, 6 hours, `CERTCTL_CLOUD_DISCOVERY_INTERVAL`): cloud secret manager certificate discovery (M50)
+  Each loop includes `atomic.Bool` idempotency guards, error handling, and structured slog failure logs.
 - **Metrics Endpoints** — Two formats for monitoring integration:
   - `GET /api/v1/metrics` — JSON object with gauges, counters, and uptime for custom dashboards
   - `GET /api/v1/metrics/prometheus` — Prometheus exposition format (`text/plain; version=0.0.4`) for native scraping by Prometheus, Grafana Agent, Datadog, and other OpenMetrics-compatible collectors
@@ -281,12 +293,13 @@ Each section includes:
   - `certificateHold` — temporary revocation (can be "unhold" by reissue)
   - `privilegeWithdrawn` — access rights revoked
   Revocation is **immediate** (no approval workflow). The certificate is marked `Revoked` in inventory, an audit event is logged, and optional issuer notification is best-effort. All revoked certs are excluded from active deployments.
-- **CRL Endpoint** — `GET /api/v1/crl` returns a JSON-formatted Certificate Revocation List (serial, reason, timestamp for each revoked cert). `GET /api/v1/crl/{issuer_id}` returns a DER-encoded X.509 CRL signed by the issuing CA (useful for legacy clients that don't support OCSP).
-- **OCSP Responder** — `GET /api/v1/ocsp/{issuer_id}/{serial}` returns a signed OCSP response indicating whether a cert is good, revoked, or unknown. Clients (browsers, TLS libraries) query this endpoint to verify cert validity in real-time.
+- **CRL Endpoint** — `GET /.well-known/pki/crl/{issuer_id}` returns a DER-encoded X.509 CRL signed by the issuing CA (RFC 5280 §5, RFC 8615, `Content-Type: application/pkix-crl`), served unauthenticated for relying parties that don't hold certctl API credentials.
+- **OCSP Responder** — `GET /.well-known/pki/ocsp/{issuer_id}/{serial}` returns a signed OCSP response indicating whether a cert is good, revoked, or unknown (RFC 6960, `Content-Type: application/ocsp-response`). Also unauthenticated. Clients (browsers, TLS libraries) query this endpoint to verify cert validity in real-time.
 - **Revocation Notifications** — When a cert is revoked, notifications are sent to:
   - Certificate owner (email)
   - Configured webhooks (if you have a SIEM that subscribes)
   - Slack/Teams channels (if notifiers are configured)
+- **Bulk Revocation for Fleet-Wide Incidents** (V2.2) — `POST /api/v1/certificates/bulk-revoke` with filter criteria (profile, owner, agent, issuer) revokes all matching certificates in a single operation. Essential for incident response: key compromise affecting multiple certs, CA distrust events, decommissioning a team's infrastructure. Each bulk revocation creates individual jobs reusing the existing revocation pipeline, ensuring audit trail and notifications for every certificate.
 - **Short-Lived Cert Exemption** — Certificates with TTL < 1 hour (configured in profile) skip CRL/OCSP publication. Expiry is the revocation mechanism for short-lived certs (e.g., Kubernetes pod certs, session tokens).
 - **Deployment Rollback** — If a revoked cert is still deployed (shouldn't happen, but race conditions exist), operators can manually redeploy a previous version via the GUI. Rollback is audited.
 
@@ -301,7 +314,6 @@ Each section includes:
 
 **V3 Enhancement**:
 
-- **Bulk Revocation** — Revoke all certs issued by a specific profile, owner, or agent in a single API call (useful for large-scale incidents like CA compromise)
 - **Revocation Automation** — Trigger revocation based on external events (e.g., employee termination, security breach alert from CT Log monitoring)
 
 **Operator Responsibility**:
@@ -452,15 +464,15 @@ Each section includes:
 | | Metrics JSON Endpoint | `GET /api/v1/metrics` (gauges, counters, uptime) | ✅ | ✅ | Set thresholds, configure alerting |
 | | Stats API (time-series) | `GET /api/v1/stats/*` (summary, status, expiration, jobs, issuance) | ✅ | ✅ | Integrate into dashboards, SLO tracking |
 | | Structured Logging | `slog` middleware with request IDs | ✅ | ✅ | Aggregate logs to SIEM, define retention policy |
-| | Background Scheduler | 6 loops (renewal 1h, jobs 30s, health 2m, notifications 1m, short-lived 30s, network scan 6h) | ✅ | ✅ | Alert on scheduler loop failures |
+| | Background Scheduler | 12 loops (8 always-on: renewal 1h, jobs 30s, job retry 5m I-001, job timeout 10m I-003, health 2m, notifications 1m, notif retry 2m I-005, short-lived 30s; 4 opt-in: network scan 6h, digest 24h, endpoint health 60s M48, cloud discovery 6h M50) | ✅ | ✅ | Alert on scheduler loop failures |
 | **CC7.2** Anomaly Detection | Immutable API Audit Trail | `internal/api/middleware/audit.go`, `GET /api/v1/audit` | ✅ | Enhanced (SIEM export) | Integrate into SIEM, search for anomalies, archive long-term |
 | | Expiration Threshold Alerting | Configurable per-policy (default 30/14/7/0 days) | ✅ | ✅ | Configure thresholds, integrate notifications |
 | | Status Auto-Transitions | Active → Expiring (30d) → Expired (0d) | ✅ | ✅ | Monitor status changes in audit trail |
 | | Notification Routing | Email, Slack, Teams, PagerDuty, OpsGenie | ✅ | ✅ | Configure notifiers, on-call integration |
 | | Deployment Rollback | Redeploy previous cert version via GUI | ✅ | ✅ | Audit rollback decisions |
 | **CC7.3** Incident Response | Revocation API (RFC 5280 reasons) | `POST /api/v1/certificates/{id}/revoke` | ✅ | Enhanced (bulk revocation) | Establish incident response policy |
-| | CRL Endpoint (JSON + DER) | `GET /api/v1/crl`, `GET /api/v1/crl/{issuer_id}` | ✅ | ✅ | Ensure CRL/OCSP accessible to all clients |
-| | OCSP Responder | `GET /api/v1/ocsp/{issuer_id}/{serial}` | ✅ | ✅ | Test revocation in staging |
+| | CRL Endpoint (DER, RFC 5280 §5) | `GET /.well-known/pki/crl/{issuer_id}` (unauthenticated, `application/pkix-crl`) | ✅ | ✅ | Ensure CRL/OCSP accessible to all clients without API keys |
+| | OCSP Responder (RFC 6960) | `GET /.well-known/pki/ocsp/{issuer_id}/{serial}` (unauthenticated, `application/ocsp-response`) | ✅ | ✅ | Test revocation in staging |
 | | Revocation Notifications | Email, webhook, Slack/Teams on revocation | ✅ | ✅ | Integrate into on-call, document justification separately |
 | | Short-Lived Cert Exemption | TTL < 1h skip CRL/OCSP | ✅ | ✅ | Configure profiles appropriately |
 | **CC7.4** Risk Mitigation | Renewal Job Tracking | Job state machine (Pending → Running → Completed/Failed) | ✅ | ✅ | Monitor renewal success rate |

docs/compliance.md

@@ -32,6 +32,85 @@ If you're preparing for an audit and certctl is already deployed, use the "Opera
 | PCI-DSS 4.0 | Cardholder data protection | TLS lifecycle, key management, immutable logging, access control |
 | NIST SP 800-57 | Cryptographic key management | Agent-side keygen, key isolation, algorithm selection, revocation |
 
+## Audit-Trail Integrity & Privacy (Bundle 6)
+
+Two complementary controls protect the `audit_events` table against tampering and minimize PII exposure. Both apply automatically — no operator action is required at install time, but operators must understand the contract before responding to a legal-hold or retention request.
+
+### Append-Only Enforcement (HIPAA §164.312(b))
+
+<!-- Source: migrations/000018_audit_events_worm.up.sql -->
+
+`audit_events` rows cannot be modified or deleted by the application role. Two layers:
+
+| Layer | Mechanism | Surface |
+|---|---|---|
+| **DB trigger** | `audit_events_block_modification()` raises `check_violation` on `BEFORE UPDATE OR DELETE` | Catches any UPDATE / DELETE — including direct `psql` from the app role |
+| **App-role grant** | `REVOKE UPDATE, DELETE ON audit_events FROM certctl` | Defence-in-depth; the app role can't even attempt the modification |
+
+**Verification.** From a `psql` session connected as the `certctl` app role:
+
+```sql
+UPDATE audit_events SET actor = 'tampered' WHERE id = 'audit-001';
+-- ERROR:  audit_events is append-only (Bundle-6 / M-017 / HIPAA §164.312(b))
+-- HINT:   Use a compliance superuser role for legitimate retention operations.
+```
+
+**Compliance superuser pattern.** Legitimate retention work (legal hold, GDPR right-to-be-forgotten, statutory purges) requires a separate PostgreSQL role provisioned out-of-band that bypasses the trigger. Certctl does NOT auto-create this role — operators provision it per their compliance policy. Suggested shape:
+
+```sql
+-- One-time setup by a DBA. Stored procedure pattern keeps the
+-- compliance superuser audit-able too: every invocation should
+-- itself land in audit_events.
+CREATE ROLE certctl_compliance LOGIN PASSWORD '<strong-secret>';
+GRANT UPDATE, DELETE ON audit_events TO certctl_compliance;
+-- (optional) provision SECURITY DEFINER stored procedures that
+-- (a) record the retention reason in audit_events as the FIRST step
+-- (b) then perform the UPDATE/DELETE
+-- (c) all under the certctl_compliance role's grants.
+```
+
+### Body Redaction (GDPR Art. 32, CWE-532)
+
+<!-- Source: internal/service/audit_redact.go -->
+
+`AuditService.RecordEvent` routes every `details` map through `RedactDetailsForAudit` BEFORE marshaling to the JSONB column. Two deny-lists:
+
+| Category | Match | Replacement | Examples |
+|---|---|---|---|
+| **Credentials** | case-insensitive key match | `"[REDACTED:CREDENTIAL]"` | `api_key`, `password`, `token`, `*_pem`, `eab_secret`, `acme_account_key`, `signature` |
+| **PII** | case-insensitive key match | `"[REDACTED:PII]"` | `email`, `phone`, `ssn`, `dob`, `name`, `address`, `postal_code`, `ip_address` |
+
+Nested maps and arrays are walked recursively — sensitive keys at any depth get scrubbed. The redactor is mutation-free (the caller's original map is unchanged) so service-layer code that reuses the map elsewhere is safe.
+
+**Operator visibility — `redacted_keys` array.** The redacted map includes a `redacted_keys` array listing every dotted-path that was scrubbed. This surfaces the redaction footprint to compliance auditors without exposing values. Example before/after:
+
+```jsonc
+// Caller's input map (e.g., from a service handler):
+{
+  "action": "create_issuer",
+  "issuer_id": "iss-acme-prod",
+  "config": {
+    "endpoint": "https://acme.example.com",
+    "eab_secret": "abc123secret",
+    "contact": { "email": "ops@example.com", "role": "admin" }
+  }
+}
+
+// Persisted in audit_events.details:
+{
+  "action": "create_issuer",
+  "issuer_id": "iss-acme-prod",
+  "config": {
+    "endpoint": "https://acme.example.com",
+    "eab_secret": "[REDACTED:CREDENTIAL]",
+    "contact": { "email": "[REDACTED:PII]", "role": "admin" }
+  },
+  "redacted_keys": ["config.eab_secret", "config.contact.email"]
+}
+```
+
+**Maintenance.** When introducing a new credential-bearing field anywhere in the codebase, add the key name to `credentialKeys` (or `piiKeys`) in `internal/service/audit_redact.go`. The unit test suite in `audit_redact_test.go` exercises every entry and proves case-insensitivity + JSON round-trip safety.
+
 ## certctl Pro (V3) Enhancements
 
 Several compliance-relevant features are planned for certctl Pro:

docs/concepts.md

@@ -123,11 +123,13 @@ At no point does the private key leave the agent. This is a fundamental security
 
 Agents also report **metadata** about themselves — their operating system, CPU architecture, IP address, hostname, and version — with every heartbeat. This gives ops teams fleet-wide visibility (e.g., "how many agents are running on ARM?", "which agents are still on v1.0.0?") and powers **agent groups** — dynamic device grouping where policies can be scoped to specific agent criteria like OS type, architecture, or network subnet.
 
+**Retiring an agent.** When you decommission a server, the certctl record for its agent needs to be retired, not deleted. certctl uses a **soft-delete** model: `DELETE /api/v1/agents/{id}` stamps the row with a retired-at timestamp and a reason, instead of removing it. This is deliberate — an audit trail of "who owned this certificate, on which host, for which team" stays intact forever, and the downstream deployment_targets, certificates, and jobs keep valid foreign keys. Retired agents are filtered out of default list views and the dashboard's agent counter, but remain visible through a separate retired-agents view for compliance reconciliation. If the agent still has active deployment targets, deployed certificates, or pending jobs, retirement is blocked by default so you don't silently orphan those rows; the API responds with the exact counts so you can retire or reassign each dependency explicitly. A force-retire escape hatch (`?force=true&reason=...`) is available for true decommission scenarios — it transactionally retires the downstream targets, cancels pending jobs, and records the cascade in the audit trail with the reason you provided. Four internal sentinel agents that back the network scanner and the cloud secret-manager discovery sources cannot be retired at all, even with force, because retiring them would orphan their subsystems. Once retired, an agent that still attempts to heartbeat receives `410 Gone` — the agent process reads that as "you've been retired, shut down" and exits cleanly.
+
 ### Deployment Targets
 
-Targets are the systems where certificates actually get installed — NGINX web servers, Apache httpd servers, HAProxy load balancers, F5 BIG-IP appliances, Microsoft IIS servers. Each target type has a **connector** that knows how to deploy certificates to that specific system (e.g., writing files and reloading NGINX or Apache config, building a combined PEM for HAProxy).
+Targets are the systems where certificates actually get installed — NGINX web servers, Apache httpd servers, HAProxy load balancers, Traefik reverse proxies, Caddy servers, Envoy gateways, Postfix/Dovecot mail servers, Microsoft IIS servers, and network appliances. Each target type has a **connector** that knows how to deploy certificates to that specific system (e.g., writing files and reloading NGINX or Apache config, building a combined PEM for HAProxy).
 
-For targets where an agent runs directly on the machine (NGINX, Apache, HAProxy, IIS), the agent deploys certificates locally — no remote access needed. For network appliances where you can't install an agent (F5 BIG-IP, Palo Alto, etc.), a **proxy agent** in the same network zone picks up the deployment job and calls the appliance's API. The server never initiates outbound connections to any target.
+For targets where an agent runs directly on the machine (NGINX, Apache, HAProxy, Traefik, Caddy, Envoy, Postfix, Dovecot, IIS), the agent deploys certificates locally — no remote access needed. For network appliances where you can't install an agent (F5 BIG-IP, Palo Alto, etc.), a **proxy agent** in the same network zone picks up the deployment job and calls the appliance's API. The server never initiates outbound connections to any target.
 
 ## The Certificate Lifecycle
 
@@ -183,6 +185,29 @@ Profiles are managed via the API (`/api/v1/profiles`) and the GUI, and can be as
 
 For policies with `auto_renew` disabled, renewal jobs enter an **AwaitingApproval** state instead of processing immediately. An operator must explicitly approve or reject the renewal via the API or GUI. Approved jobs transition to Pending and are picked up by the scheduler. Rejected jobs are cancelled with an optional reason. This is useful for high-value certificates where you want human oversight before renewal.
 
+### Renewal Timing: Thresholds vs. ARI (RFC 9773)
+
+**Traditional approach (thresholds):** By default, certctl uses static renewal thresholds — renew a certificate at a fixed number of days before expiry (default: 30 days). This simple, predictable model works for most use cases: it avoids unnecessary renewals near expiry and gives you a predictable window to catch failures.
+
+**Advanced approach (ACME ARI):** Some Certificate Authorities support ACME Renewal Information (RFC 9773), which allows the CA to tell certctl the optimal time to renew. Instead of guessing "renew 30 days before expiry," the CA responds with a precise `suggestedWindow` containing start and end times. This is useful when:
+- The CA is performing maintenance and wants to batch renewals in a specific window
+- The CA is coordinating a mass revocation (e.g., due to a compromise) and needs to control renewal timing
+- You want to avoid thundering herd renewal spikes by accepting the CA's suggested timing
+
+**How it works:** Enable with `CERTCTL_ACME_ARI_ENABLED=true` on your ACME issuer. When a certificate approaches expiry, certctl queries the ARI endpoint with the certificate's DER encoding. The CA responds with a suggested renewal window. If the current time is within the window or past the start time, certctl renews immediately. Otherwise, it waits until the window opens.
+
+**Graceful degradation:** If your CA doesn't support ARI (returns 404 from the ARI endpoint), certctl automatically falls back to the traditional threshold-based renewal. No configuration change needed — the fallback is transparent. Errors from the CA are logged as warnings and don't block the renewal process.
+
+### Shorter Certificate Validity (45-Day and 6-Day Certs)
+
+The industry is moving toward shorter certificate lifetimes. The CA/Browser Forum's SC-081v3 ballot mandates a phased reduction: 200-day max (March 2026), 100-day max (March 2027), and 47-day max (March 2029). Let's Encrypt has already begun reducing default validity to 45 days, and offers 6-day "shortlived" certificates via ACME profile selection.
+
+certctl handles shorter-lived certificates correctly out of the box:
+
+- **45-day certs** with the default 31-day renewal window trigger renewal at day 14 — at roughly 1/3 of the cert's lifetime.
+- **6-day "shortlived" certs** are always within the renewal window. ARI (RFC 9773) is the expected renewal path for these — the CA directs timing. Short-lived certs also skip CRL/OCSP since expiry is sufficient revocation (per profile TTL < 1 hour exemption).
+- **ACME profile selection** lets you request specific certificate profiles from your CA. Set `CERTCTL_ACME_PROFILE=shortlived` to get 6-day certificates from Let's Encrypt, or `CERTCTL_ACME_PROFILE=tlsserver` for standard TLS certificates.
+
 ### Certificate Revocation
 
 When a private key is compromised, a certificate is superseded, or a service is decommissioned, you need to revoke the certificate immediately — not wait for it to expire. Revocation tells clients "stop trusting this certificate right now."
@@ -191,9 +216,11 @@ certctl implements revocation using three complementary mechanisms:
 
 **Revocation API**: `POST /api/v1/certificates/{id}/revoke` marks a certificate as revoked in the inventory, records the revocation in a dedicated `certificate_revocations` table, notifies the issuing CA (best-effort — the revocation succeeds even if the CA is unreachable), creates an audit trail entry, and sends notifications. You can specify an RFC 5280 reason code (keyCompromise, superseded, cessationOfOperation, etc.) or let it default to "unspecified."
 
-**Certificate Revocation List (CRL)**: certctl serves both a JSON-formatted CRL at `GET /api/v1/crl` and DER-encoded X.509 CRLs per issuer at `GET /api/v1/crl/{issuer_id}`. The DER CRL is signed by the issuing CA's key and has 24-hour validity — clients can download it periodically to check revocation status offline.
+**Bulk Revocation** (Fleet-Level Incident Response): For large-scale incidents like CA compromise or team infrastructure decommissioning, `POST /api/v1/certificates/bulk-revoke` revokes all certificates matching filter criteria in a single operation. Filter by profile, owner, team, agent group, or issuer to target the affected certificate set. This is essential for incident response — instead of revoking certificates one-by-one, operators can revoke an entire fleet in minutes. Bulk revocation creates individual revocation jobs that reuse the existing revocation pipeline, ensuring every certificate is audited and notifications are sent.
 
-**OCSP Responder**: For real-time revocation checking, certctl includes an embedded OCSP responder at `GET /api/v1/ocsp/{issuer_id}/{serial}`. It returns signed OCSP responses (good, revoked, or unknown) so clients can verify certificate status without downloading the full CRL.
+**Certificate Revocation List (CRL)**: certctl serves DER-encoded X.509 CRLs per issuer at `GET /.well-known/pki/crl/{issuer_id}` (RFC 5280 §5 wire format, RFC 8615 well-known namespace). The endpoint is unauthenticated so any relying party — browser, TLS client, hardware appliance — can fetch it without a certctl API key. The CRL is signed by the issuing CA's key and has 24-hour validity; clients can download it periodically to check revocation status offline. The response carries `Content-Type: application/pkix-crl`.
+
+**OCSP Responder**: For real-time revocation checking, certctl includes an embedded OCSP responder at `GET /.well-known/pki/ocsp/{issuer_id}/{serial}` (RFC 6960). Like the CRL endpoint, it is unauthenticated and returns signed OCSP responses (good, revoked, or unknown) with `Content-Type: application/ocsp-response`, so clients can verify certificate status without downloading the full CRL.
 
 Short-lived certificates (those assigned to profiles with TTL under 1 hour) are exempt from CRL and OCSP — their rapid expiry is considered sufficient revocation. This is a deliberate design choice to reduce infrastructure overhead for ephemeral machine-to-machine credentials.
 
@@ -229,7 +256,7 @@ The CLI supports both table and JSON output formats (`--format table` or `--form
 
 ### MCP Server (AI Integration)
 
-certctl includes an MCP (Model Context Protocol) server that exposes 78 MCP tools covering the REST API. This enables AI assistants like Claude, Cursor, and other MCP-compatible tools to interact with your certificate infrastructure using natural language — "show me all expiring certificates," "revoke the VPN cert," or "what agents are offline?"
+certctl includes an MCP (Model Context Protocol) server that exposes the entire REST API as MCP tools. This enables AI assistants like Claude, Cursor, and other MCP-compatible tools to interact with your certificate infrastructure using natural language — "show me all expiring certificates," "revoke the VPN cert," or "what agents are offline?"
 
 The MCP server is a separate binary (`cmd/mcp-server/`) that communicates via stdio transport and acts as a stateless HTTP proxy to the certctl REST API. It requires no additional infrastructure — just point it at your certctl server URL and API key.
 

docs/connectors.md

@@ -11,9 +11,13 @@ Connectors extend certctl to integrate with external systems for certificate iss
    - [Built-in: ACME v2 (Let's Encrypt, Sectigo, ZeroSSL)](#built-in-acme-v2-lets-encrypt-sectigo-zerossl)
    - [Built-in: step-ca (Smallstep Private CA)](#built-in-step-ca-smallstep-private-ca)
    - [OpenSSL / Custom CA](#openssl--custom-ca)
+   - [Built-in: Vault PKI](#built-in-vault-pki)
+   - [Built-in: DigiCert CertCentral](#built-in-digicert-certcentral)
+   - [Built-in: Sectigo SCM](#built-in-sectigo-scm)
+   - [Built-in: Google CAS](#built-in-google-cas)
+   - [Built-in: AWS ACM Private CA](#built-in-aws-acm-private-ca)
    - [Revocation Across Issuers](#revocation-across-issuers)
    - [EST Integration (GetCACertPEM)](#est-integration-getcacertpem)
-   - [Planned Issuers](#planned-issuers)
    - [Building a Custom Issuer](#building-a-custom-issuer)
 3. [Target Connector](#target-connector)
    - [Interface](#interface-1)
@@ -21,9 +25,15 @@ Connectors extend certctl to integrate with external systems for certificate iss
    - [Built-in: Apache httpd](#built-in-apache-httpd)
    - [Built-in: HAProxy](#built-in-haproxy)
    - [Built-in: Traefik](#built-in-traefik)
+   - [Built-in: Envoy](#built-in-envoy)
+   - [Built-in: Postfix / Dovecot](#built-in-postfix--dovecot)
    - [Built-in: Caddy](#built-in-caddy)
-   - [F5 BIG-IP (Interface Only)](#f5-big-ip-interface-only)
-   - [IIS (Interface Only, Dual-Mode)](#iis-interface-only-dual-mode)
+   - [F5 BIG-IP (Implemented)](#f5-big-ip-implemented)
+   - [IIS (Implemented, Dual-Mode)](#iis-implemented-dual-mode)
+   - [SSH (Agentless Deployment)](#ssh-agentless-deployment)
+   - [Windows Certificate Store](#windows-certificate-store)
+   - [Java Keystore (JKS / PKCS#12)](#java-keystore-jks--pkcs12)
+   - [Kubernetes Secrets](#kubernetes-secrets)
 4. [Notifier Connector](#notifier-connector)
    - [Interface](#interface-2)
 5. [Registering a Connector](#registering-a-connector)
@@ -51,8 +61,8 @@ Connectors extend certctl to integrate with external systems for certificate iss
 
 Three types of connectors:
 
-1. **Issuer Connector** — Obtains certificates from CAs (Local CA with sub-CA support, ACME with HTTP-01 + DNS-01 + DNS-PERSIST-01, step-ca, OpenSSL/Custom CA implemented; additional CA integrations planned)
-2. **Target Connector** — Deploys certificates to infrastructure (NGINX, Apache httpd, HAProxy, Traefik, Caddy implemented; F5 via proxy agent, IIS dual-mode interface only; additional cloud and network targets planned)
+1. **Issuer Connector** — Obtains certificates from CAs. 9 built-in: Local CA (self-signed + sub-CA), ACME v2 (HTTP-01, DNS-01, DNS-PERSIST-01, ARI, EAB, profile selection), step-ca, OpenSSL/Custom CA, Vault PKI, DigiCert CertCentral, Sectigo SCM, Google CAS, AWS ACM Private CA
+2. **Target Connector** — Deploys certificates to infrastructure. 14 built-in: NGINX, Apache httpd, HAProxy, Traefik, Caddy, Envoy, Postfix, Dovecot, IIS (local + WinRM), F5 BIG-IP (proxy agent), SSH (agentless), Windows Certificate Store, Java Keystore, Kubernetes Secrets
 3. **Notifier Connector** — Sends alerts about certificate events (Email, Webhooks, Slack, Microsoft Teams, PagerDuty, OpsGenie implemented)
 
 All connectors accept JSON configuration at initialization, support config validation, and are registered in the service layer. Issuer connectors run on the control plane; target connectors run on agents. For network appliances where agents can't be installed, a **proxy agent** in the same network zone handles deployment — the server never initiates outbound connections.
@@ -145,7 +155,11 @@ The Local CA issuer signs certificates using Go's `crypto/x509` library. It supp
 
 **Sub-CA mode:** Loads a CA certificate and private key from disk (`CERTCTL_CA_CERT_PATH` + `CERTCTL_CA_KEY_PATH`). The CA cert is signed by an upstream CA (e.g., ADCS), so all issued certificates chain to the enterprise root trust hierarchy. Clients that already trust the enterprise root automatically trust certctl-issued certs. Supports RSA, ECDSA, and PKCS#8 key formats. If the paths are not set, falls back to self-signed mode. The loaded certificate must have `IsCA=true` and `KeyUsageCertSign`.
 
-**CRL and OCSP support (M15b):** The Local CA supports DER-encoded X.509 CRL generation via `GET /api/v1/crl/{issuer_id}` with 24-hour validity. An embedded OCSP responder at `GET /api/v1/ocsp/{issuer_id}/{serial}` returns signed OCSP responses for issued certificates (good/revoked/unknown status). Certificates with profile TTL < 1 hour automatically skip CRL/OCSP — expiry is treated as sufficient revocation for short-lived credentials.
+**CRL and OCSP support (M15b):** The Local CA supports DER-encoded X.509 CRL generation served unauthenticated at `GET /.well-known/pki/crl/{issuer_id}` (RFC 5280 §5, RFC 8615, `Content-Type: application/pkix-crl`) with 24-hour validity. An embedded OCSP responder at `GET /.well-known/pki/ocsp/{issuer_id}/{serial}` (RFC 6960, `Content-Type: application/ocsp-response`) returns signed OCSP responses for issued certificates (good/revoked/unknown status). Both endpoints are reachable by relying parties with no certctl API credentials, which is how standard TLS clients, browsers, and hardware appliances consume these resources. Certificates with profile TTL < 1 hour automatically skip CRL/OCSP — expiry is treated as sufficient revocation for short-lived credentials.
+
+**Extended Key Usage (EKU) support (M27):** The Local CA respects EKU constraints from certificate profiles and adjusts key usage flags accordingly. For S/MIME certificates (emailProtection EKU), it uses `DigitalSignature | ContentCommitment` instead of the TLS default. For TLS certificates (serverAuth/clientAuth EKU), it uses `DigitalSignature | KeyEncipherment`. This enables support for multiple certificate types — TLS, S/MIME, code signing, timestamping — from a single CA.
+
+**MaxTTL enforcement (M11c):** When a certificate profile defines a maximum TTL, the Local CA caps the `NotAfter` field to `min(validity_days, maxTTL)`. This ensures certificates never exceed the profile's configured lifetime regardless of the issuer's `validity_days` setting.
 
 Configuration:
 ```json
@@ -169,6 +183,8 @@ The ACME connector implements the full ACME v2 protocol using Go's `golang.org/x
 
 **DNS-PERSIST-01 (standing record):** Creates a one-time persistent TXT record at `_validation-persist.<domain>` containing the CA's issuer domain and your ACME account URI. Once set, this record authorizes unlimited future certificate issuances without per-renewal DNS updates. Based on [draft-ietf-acme-dns-persist](https://datatracker.ietf.org/doc/draft-ietf-acme-dns-persist/) and CA/Browser Forum ballot SC-088v3. If the CA doesn't offer dns-persist-01 yet, the connector falls back to dns-01 automatically.
 
+**ACME Renewal Information (ARI, RFC 9773):** Instead of using fixed renewal thresholds (e.g., renew 30 days before expiry), certctl can ask the CA when it should renew. Enable with `CERTCTL_ACME_ARI_ENABLED=true`. The ARI protocol lets the CA specify a `suggestedWindow` (start and end times) for when you should renew — useful for distributing load during maintenance windows or coordinating mass revocation scenarios. Cert ID is computed as `base64url(SHA-256(DER cert))`. If the CA doesn't support ARI (404 response), certctl automatically falls back to threshold-based renewal with no operator intervention required.
+
 HTTP-01 configuration:
 ```json
 {
@@ -237,6 +253,9 @@ Environment variables for the default ACME connector:
 - `CERTCTL_ACME_DNS_PRESENT_SCRIPT` — Path to DNS record creation script (dns-01 and dns-persist-01)
 - `CERTCTL_ACME_DNS_CLEANUP_SCRIPT` — Path to DNS record cleanup script (dns-01 only, not used by dns-persist-01)
 - `CERTCTL_ACME_DNS_PERSIST_ISSUER_DOMAIN` — CA issuer domain for persistent record (dns-persist-01 only, e.g., `letsencrypt.org`)
+- `CERTCTL_ACME_PROFILE` — Certificate profile for the newOrder request. Let's Encrypt supports `tlsserver` (standard TLS, default) and `shortlived` (6-day certs). Leave empty for the CA's default profile.
+
+**Certificate Profiles:** Let's Encrypt (GA January 2026) supports ACME certificate profile selection. Set `CERTCTL_ACME_PROFILE=shortlived` to request 6-day certificates — ideal for ephemeral workloads where short validity substitutes for revocation. The `tlsserver` profile produces standard TLS certificates. When the profile field is empty (default), the CA uses its default profile, maintaining full backward compatibility.
 
 The connector is registered in the issuer registry under `iss-acme-staging` and `iss-acme-prod`. Use `iss-acme-staging` for Let's Encrypt staging (rate-limit-friendly testing) and `iss-acme-prod` for production certificates.
 
@@ -268,7 +287,9 @@ Environment variables:
 
 The connector is registered in the issuer registry under `iss-stepca`. step-ca also works with the existing ACME connector (point `iss-acme-*` at step-ca's ACME directory URL for ACME-based issuance).
 
-**Note:** step-ca-issued certificates rely on step-ca's own CRL/OCSP infrastructure. certctl's local CRL/OCSP endpoints (`GET /api/v1/crl/{issuer_id}` and `GET /api/v1/ocsp/{issuer_id}/{serial}`) are populated from step-ca's revocation data if available, but clients should validate against step-ca's endpoints for the authoritative status.
+**Note:** step-ca-issued certificates rely on step-ca's own CRL/OCSP infrastructure. certctl's local CRL/OCSP endpoints (`GET /.well-known/pki/crl/{issuer_id}` and `GET /.well-known/pki/ocsp/{issuer_id}/{serial}`, served unauthenticated per RFC 5280 §5 / RFC 6960 / RFC 8615) are populated from step-ca's revocation data if available, but clients should validate against step-ca's endpoints for the authoritative status.
+
+**MaxTTL enforcement (M11c):** When a certificate profile defines a maximum TTL, the step-ca connector caps the `NotAfter` field to ensure the issued certificate does not exceed the profile limit, regardless of the step-ca provisioner's own maximum.
 
 Location: `internal/connector/issuer/stepca/stepca.go`
 
@@ -297,29 +318,197 @@ Each issuer handles revocation differently:
 - **step-ca**: Calls step-ca's `/revoke` API endpoint. Clients should check step-ca's own CRL/OCSP for authoritative status.
 - **OpenSSL/Custom CA**: Invokes the configured revoke script (`CERTCTL_OPENSSL_REVOKE_SCRIPT`) with the serial number as an argument.
 
-### EST Integration (GetCACertPEM)
+### EST/SCEP Integration (GetCACertPEM)
 
-The `GetCACertPEM()` method returns the PEM-encoded CA certificate chain, used by the EST server's `/.well-known/est/cacerts` endpoint (RFC 7030) to distribute the CA chain to enrolling devices. Each issuer handles this differently:
+The `GetCACertPEM()` method returns the PEM-encoded CA certificate chain, used by both the EST server's `/.well-known/est/cacerts` endpoint (RFC 7030) and the SCEP server's `GetCACert` operation (RFC 8894) to distribute the CA chain to enrolling devices. Each issuer handles this differently:
 
-- **Local CA**: Returns the CA certificate PEM (self-signed or sub-CA cert). This is the primary EST issuer.
+- **Local CA**: Returns the CA certificate PEM (self-signed or sub-CA cert). This is the primary EST/SCEP issuer.
 - **ACME**: Returns error — ACME CAs provide chains per-issuance, not statically.
 - **step-ca**: Returns error — step-ca serves its own `/root` endpoint for CA distribution.
 - **OpenSSL/Custom CA**: Returns error — custom script-based CAs have no CA cert access through certctl.
 
-Note: EST (Enrollment over Secure Transport) is not a connector — it's a protocol handler (`internal/api/handler/est.go`) that delegates certificate issuance to whichever issuer connector is configured via `CERTCTL_EST_ISSUER_ID`. See the [Architecture Guide](architecture.md#est-server-rfc-7030) for details.
+Note: EST and SCEP are not connectors — they are protocol handlers (`internal/api/handler/est.go` and `internal/api/handler/scep.go`) that delegate certificate issuance to whichever issuer connector is configured via `CERTCTL_EST_ISSUER_ID` or `CERTCTL_SCEP_ISSUER_ID`. Both share a common `internal/pkcs7` package for PKCS#7 response encoding. See the [Architecture Guide](architecture.md#est-server-rfc-7030) for details.
 
-### Planned Issuers
+### Built-in: Vault PKI
 
-The following issuer connectors are planned for future milestones:
+The Vault PKI connector integrates with HashiCorp Vault's PKI secrets engine using its native `/sign` API with token-based authentication. This is ideal for organizations using Vault as their internal certificate authority — synchronous issuance without the complexity of ACME or challenge solving.
 
-- **Vault PKI** — HashiCorp Vault's PKI secrets engine for organizations using Vault as their internal CA (planned for V4.0+).
-- **DigiCert** — Commercial CA integration via DigiCert's REST API (planned).
+**Configuration:**
 
-Note: ADCS (Active Directory Certificate Services) integration is handled via the **sub-CA mode** of the Local CA issuer, not as a separate connector. certctl operates as a subordinate CA with its signing certificate issued by ADCS, so all certctl-issued certs chain to the enterprise ADCS root. See the Local CA section above.
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `CERTCTL_VAULT_ADDR` | — | Vault server address (e.g., `https://vault.internal:8200`) |
+| `CERTCTL_VAULT_TOKEN` | — | Vault auth token with permissions on the PKI mount |
+| `CERTCTL_VAULT_MOUNT` | `pki` | PKI secrets engine mount path |
+| `CERTCTL_VAULT_ROLE` | — | PKI role name for certificate signing |
+| `CERTCTL_VAULT_TTL` | `8760h` | Certificate validity period (TTL) |
+
+The connector is registered in the issuer registry under `iss-vault`. Vault issues certificates synchronously via the `/v1/{mount}/sign/{role}` API with `X-Vault-Token` header authentication. The issued certificate is parsed to extract serial number, validity dates, and chain information.
+
+**Note:** CRL and OCSP are managed by Vault itself. Clients should validate certificate status against Vault's own CRL/OCSP endpoints (`GET /v1/{mount}/crl` and Vault's OCSP responder). certctl does not generate local CRL/OCSP for Vault-issued certificates. Revocation is recorded locally but Vault is the authoritative source.
+
+**MaxTTL enforcement (M11c):** When a certificate profile defines a maximum TTL, the Vault connector overrides the TTL string in the signing request to ensure the issued certificate does not exceed the profile limit. This is applied before Vault's own role-level max TTL.
+
+Location: `internal/connector/issuer/vault/vault.go`
+
+### Built-in: DigiCert CertCentral
+
+The DigiCert connector integrates with DigiCert's CertCentral REST API for ordering and managing certificates from DigiCert's commercial CA. It supports both Domain Validated (DV) and Organization/Extended Validated (OV/EV) certificates, with async order processing.
+
+**Configuration:**
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `CERTCTL_DIGICERT_API_KEY` | — | DigiCert API key (X-DC-DEVKEY header) |
+| `CERTCTL_DIGICERT_ORG_ID` | — | DigiCert organization ID |
+| `CERTCTL_DIGICERT_PRODUCT_TYPE` | `ssl_basic` | Certificate product (e.g., `ssl_basic`, `ssl_plus`, `ssl_ev`) |
+| `CERTCTL_DIGICERT_BASE_URL` | `https://www.digicert.com/services/v2` | DigiCert API base URL |
+
+The connector submits certificate orders to DigiCert's `/order/certificate/create` API. DV certificates may issue immediately; OV/EV certificates require validation (handled by DigiCert) and poll-based completion. The connector periodically checks order status via `/order/certificate/{order_id}` until the certificate is available.
+
+**Authentication:** API key passed via `X-DC-DEVKEY` header, with organization ID in request body.
+
+**Note:** CRL and OCSP are managed by DigiCert. Clients should validate certificate status against DigiCert's infrastructure. certctl records the revocation locally but does not notify DigiCert for revocation — use DigiCert's dashboard for revocation management.
+
+Location: `internal/connector/issuer/digicert/digicert.go`
+
+### Built-in: Sectigo SCM
+
+The Sectigo connector integrates with Sectigo Certificate Manager's REST API for ordering and managing DV, OV, and EV certificates. Like DigiCert, it uses an async order model: submit an enrollment, receive an sslId, then poll for completion.
+
+**Configuration:**
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `CERTCTL_SECTIGO_CUSTOMER_URI` | — | Sectigo customer URI (organization identifier) |
+| `CERTCTL_SECTIGO_LOGIN` | — | API account login |
+| `CERTCTL_SECTIGO_PASSWORD` | — | API account password |
+| `CERTCTL_SECTIGO_ORG_ID` | — | Organization ID (integer) |
+| `CERTCTL_SECTIGO_CERT_TYPE` | — | Certificate type ID (integer, from `/ssl/v1/types`) |
+| `CERTCTL_SECTIGO_TERM` | `365` | Certificate validity in days |
+| `CERTCTL_SECTIGO_BASE_URL` | `https://cert-manager.com/api` | Sectigo API base URL |
+
+The connector submits certificate enrollments to Sectigo's `/ssl/v1/enroll` API. DV certificates may issue immediately; OV/EV certificates require validation (handled by Sectigo) and poll-based completion. The connector periodically checks enrollment status via `/ssl/v1/{sslId}` and downloads the PEM bundle via `/ssl/v1/collect/{sslId}/pem` when issued.
+
+**Authentication:** Three custom headers on every request — `customerUri`, `login`, and `password`.
+
+**Note:** CRL and OCSP are managed by Sectigo. certctl records revocations locally and notifies Sectigo via `/ssl/v1/revoke/{sslId}`.
+
+Location: `internal/connector/issuer/sectigo/sectigo.go`
+
+### Built-in: Google CAS
+
+Google Cloud Certificate Authority Service — managed private CA on GCP. Synchronous issuance via CAS REST API with OAuth2 service account auth.
+
+| Setting | Required | Default | Description |
+|---------|----------|---------|-------------|
+| `CERTCTL_GOOGLE_CAS_PROJECT` | Yes | — | GCP project ID |
+| `CERTCTL_GOOGLE_CAS_LOCATION` | Yes | — | GCP region (e.g., `us-central1`) |
+| `CERTCTL_GOOGLE_CAS_CA_POOL` | Yes | — | CA pool name |
+| `CERTCTL_GOOGLE_CAS_CREDENTIALS` | Yes | — | Path to service account JSON |
+| `CERTCTL_GOOGLE_CAS_TTL` | No | `8760h` | Default certificate TTL |
+
+**Authentication:** OAuth2 service account. The connector reads a service account JSON file, signs a JWT with the private key, and exchanges it for an access token at Google's token endpoint. Tokens are cached and refreshed automatically (5 min before expiry).
+
+**Note:** CRL and OCSP are managed by Google CAS directly. certctl records revocations locally and notifies Google CAS via the revoke endpoint.
+
+Location: `internal/connector/issuer/googlecas/googlecas.go`
+
+### Built-in: AWS ACM Private CA
+
+AWS Certificate Manager Private Certificate Authority — managed private CA on AWS. Synchronous issuance via ACM PCA API with standard AWS credential chain (env vars, IAM roles, instance profiles, SSO).
+
+| Setting | Required | Default | Description |
+|---------|----------|---------|-------------|
+| `CERTCTL_AWS_PCA_REGION` | Yes | — | AWS region (e.g., `us-east-1`) |
+| `CERTCTL_AWS_PCA_CA_ARN` | Yes | — | ARN of the ACM Private CA |
+| `CERTCTL_AWS_PCA_SIGNING_ALGORITHM` | No | `SHA256WITHRSA` | Signing algorithm |
+| `CERTCTL_AWS_PCA_VALIDITY_DAYS` | No | `365` | Certificate validity in days |
+| `CERTCTL_AWS_PCA_TEMPLATE_ARN` | No | — | Optional certificate template ARN |
+
+**Supported signing algorithms:** SHA256WITHRSA, SHA384WITHRSA, SHA512WITHRSA, SHA256WITHECDSA, SHA384WITHECDSA, SHA512WITHECDSA.
+
+**Authentication:** Standard AWS credential chain. The connector uses `aws-sdk-go-v2/config.LoadDefaultConfig()` which supports environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`), IAM roles (EC2/ECS), instance profiles, and SSO credentials.
+
+**Note:** CRL and OCSP are managed by AWS ACM PCA directly. certctl records revocations locally and notifies AWS via the RevokeCertificate API with RFC 5280 reason mapping.
+
+Location: `internal/connector/issuer/awsacmpca/awsacmpca.go`
+
+### Built-in: Entrust Certificate Services
+
+Entrust CA Gateway REST API with mutual TLS (mTLS) client certificate authentication. Supports synchronous issuance (200 OK with PEM) and approval-pending flows (201 Accepted with async polling).
+
+| Setting | Required | Default | Description |
+|---------|----------|---------|-------------|
+| `CERTCTL_ENTRUST_API_URL` | Yes | — | Entrust CA Gateway base URL |
+| `CERTCTL_ENTRUST_CLIENT_CERT_PATH` | Yes | — | Path to mTLS client certificate PEM |
+| `CERTCTL_ENTRUST_CLIENT_KEY_PATH` | Yes | — | Path to mTLS client private key PEM |
+| `CERTCTL_ENTRUST_CA_ID` | Yes | — | Certificate Authority ID (from `GET /certificate-authorities`) |
+| `CERTCTL_ENTRUST_PROFILE_ID` | No | — | Optional enrollment profile ID |
+
+**Authentication:** Mutual TLS — the client certificate and key are loaded via `tls.LoadX509KeyPair()` and attached to the HTTP transport. No API key or token required.
+
+**Issuance model:** Enrollment via `POST /v1/certificate-authorities/{caId}/enrollments`. Returns 200 with PEM immediately for auto-approved enrollments, or 201 Accepted with a tracking ID for approval-pending orders. `GetOrderStatus` polls the enrollment endpoint.
+
+**Note:** CRL and OCSP are managed by Entrust. certctl records revocations locally and notifies Entrust via `PUT /v1/certificate-authorities/{caId}/certificates/{serial}/revoke`.
+
+Location: `internal/connector/issuer/entrust/entrust.go`
+
+### Built-in: GlobalSign Atlas HVCA
+
+GlobalSign Atlas High Volume CA REST API with dual authentication: mTLS for the TLS handshake and API key/secret headers for request authorization. Region-aware base URLs (EMEA, APAC, Americas).
+
+| Setting | Required | Default | Description |
+|---------|----------|---------|-------------|
+| `CERTCTL_GLOBALSIGN_API_URL` | Yes | — | Atlas HVCA API URL (region-specific) |
+| `CERTCTL_GLOBALSIGN_API_KEY` | Yes | — | API key for request authentication |
+| `CERTCTL_GLOBALSIGN_API_SECRET` | Yes | — | API secret for request authentication |
+| `CERTCTL_GLOBALSIGN_CLIENT_CERT_PATH` | Yes | — | Path to mTLS client certificate PEM |
+| `CERTCTL_GLOBALSIGN_CLIENT_KEY_PATH` | Yes | — | Path to mTLS client private key PEM |
+| `CERTCTL_GLOBALSIGN_SERVER_CA_PATH` | No | system trust store | PEM bundle used to verify the Atlas API server certificate. Set this for private/lab Atlas deployments whose server TLS chain is not in the host's default trust bundle. |
+
+**Authentication:** Dual — mTLS client certificate for TLS handshake plus `X-API-Key` and `X-API-Secret` headers on every request.
+
+**TLS verification:** The connector always verifies the server certificate. When `server_ca_path` is set, the PEM bundle at that path is used as the trust anchor; otherwise the host's system trust store is used. TLS 1.2 is the minimum protocol version.
+
+**Issuance model:** `POST /v2/certificates` returns a serial number. Certificate PEM is available after validation completes. Typically resolves within seconds for DV. `GetOrderStatus` polls the certificate endpoint.
+
+**Note:** CRL and OCSP are managed by GlobalSign. certctl records revocations locally and notifies GlobalSign via `PUT /v2/certificates/{serial}/revoke`.
+
+Location: `internal/connector/issuer/globalsign/globalsign.go`
+
+### Built-in: EJBCA (Keyfactor)
+
+EJBCA REST API for self-hosted open-source and enterprise CAs. Supports dual authentication: mTLS (default) or OAuth2 Bearer token, selectable via configuration.
+
+| Setting | Required | Default | Description |
+|---------|----------|---------|-------------|
+| `CERTCTL_EJBCA_API_URL` | Yes | — | EJBCA REST API base URL |
+| `CERTCTL_EJBCA_AUTH_MODE` | No | `mtls` | Auth mode: `mtls` or `oauth2` |
+| `CERTCTL_EJBCA_CLIENT_CERT_PATH` | mTLS | — | Path to client certificate PEM (mTLS mode) |
+| `CERTCTL_EJBCA_CLIENT_KEY_PATH` | mTLS | — | Path to client key PEM (mTLS mode) |
+| `CERTCTL_EJBCA_TOKEN` | OAuth2 | — | Bearer token (oauth2 mode) |
+| `CERTCTL_EJBCA_CA_NAME` | Yes | — | EJBCA CA name |
+| `CERTCTL_EJBCA_CERT_PROFILE` | No | — | EJBCA certificate profile |
+| `CERTCTL_EJBCA_EE_PROFILE` | No | — | EJBCA end-entity profile |
+
+**Authentication:** Configurable via `auth_mode`. In mTLS mode, client certificate and key are loaded for the TLS handshake. In OAuth2 mode, the token is sent as `Authorization: Bearer {token}`.
+
+**Issuance model:** `POST /v1/certificate/pkcs10enroll` with base64-encoded CSR. Returns base64-encoded certificate PEM. EJBCA 9.3+ creates end-entity and issues cert in a single call. Approval-pending enrollments return 201.
+
+**Revocation note:** EJBCA requires both issuer DN and serial number for revocation. The connector stores these as a composite `OrderID` in `issuer_dn::serial` format.
+
+**Note:** CRL and OCSP are managed by the EJBCA instance. certctl records revocations locally and notifies EJBCA via `PUT /v1/certificate/{issuer_dn}/{serial}/revoke`.
+
+Location: `internal/connector/issuer/ejbca/ejbca.go`
+
+### ADCS Integration
+
+Active Directory Certificate Services integration is handled via the **sub-CA mode** of the Local CA issuer, not as a separate connector. certctl operates as a subordinate CA with its signing certificate issued by ADCS, so all certctl-issued certs chain to the enterprise ADCS root. See the Local CA section above.
 
 ### Building a Custom Issuer
 
-Here's the structure for a HashiCorp Vault PKI issuer:
+Here's a simplified example showing the connector pattern (using a hypothetical Vault-like CA):
 
 ```go
 package vault
@@ -543,51 +732,334 @@ When `mode` is `"api"`, the connector posts the certificate to the admin API end
 
 Location: `internal/connector/target/caddy/caddy.go`
 
-### F5 BIG-IP (Interface Only)
+### Built-in: Envoy
 
-The F5 BIG-IP target connector interface is defined with the iControl REST flow mapped out, but the actual API calls are not yet implemented. F5 appliances can't run agents directly, so this connector uses the **proxy agent pattern**: a designated agent in the same network zone picks up F5 deployment jobs and calls the iControl REST API. The server assigns the work; the proxy agent executes it.
+The Envoy connector uses file-based certificate delivery — it writes certificate and key files to a directory that Envoy watches via its SDS (Secret Discovery Service) file-based configuration or static `filename` references in the bootstrap config. When files change, Envoy automatically picks up the new certificates without requiring a reload command.
 
-The planned flow is: authenticate via `POST /mgmt/shared/authn/login`, upload cert PEM via `POST /mgmt/tm/ltm/certificate`, update the SSL profile via `PATCH /mgmt/tm/ltm/profile/client-ssl/{profile}`, and validate deployment by checking profile status.
+Configuration:
+```json
+{
+  "cert_dir": "/etc/envoy/certs",
+  "cert_filename": "cert.pem",
+  "key_filename": "key.pem",
+  "chain_filename": "chain.pem",
+  "sds_config": true
+}
+```
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `cert_dir` | string | (required) | Directory where Envoy watches for certificate files |
+| `cert_filename` | string | `cert.pem` | Filename for the certificate (leaf + chain unless `chain_filename` is set) |
+| `key_filename` | string | `key.pem` | Filename for the private key |
+| `chain_filename` | string | (empty) | If set, chain is written to a separate file instead of appended to the cert |
+| `sds_config` | bool | `false` | If true, writes an `sds.json` file for Envoy's file-based SDS provider |
+
+When `sds_config` is `true`, the connector writes an SDS JSON file (`{cert_dir}/sds.json`) containing a `tls_certificate` resource that points to the cert and key file paths. Envoy's file-based SDS (`path_config_source`) watches this file for changes, providing automatic hot-reload of certificates. This is the recommended approach for production Envoy deployments using dynamic TLS configuration.
+
+When `sds_config` is `false` (the default), the connector simply writes cert and key files. Use this mode when Envoy's bootstrap config references the cert/key files directly via static `filename` fields in the TLS context.
+
+Location: `internal/connector/target/envoy/envoy.go`
+
+### Built-in: Postfix / Dovecot
+
+The Postfix/Dovecot connector is a dual-mode mail server TLS connector. It writes certificate, key, and chain files to configured paths and reloads the mail service. The `mode` field selects between Postfix MTA and Dovecot IMAP/POP3, which determines default file paths and reload commands.
+
+This connector pairs with certctl's S/MIME certificate support (email protection EKU, email SAN routing) for a complete email infrastructure story — TLS for transport encryption, S/MIME for end-to-end message signing and encryption.
+
+**Postfix configuration:**
+```json
+{
+  "mode": "postfix",
+  "cert_path": "/etc/postfix/certs/cert.pem",
+  "key_path": "/etc/postfix/certs/key.pem",
+  "chain_path": "/etc/postfix/certs/chain.pem",
+  "reload_command": "postfix reload",
+  "validate_command": "postfix check"
+}
+```
+
+**Dovecot configuration:**
+```json
+{
+  "mode": "dovecot",
+  "cert_path": "/etc/dovecot/certs/cert.pem",
+  "key_path": "/etc/dovecot/certs/key.pem",
+  "chain_path": "/etc/dovecot/certs/chain.pem",
+  "reload_command": "doveadm reload",
+  "validate_command": "doveconf -n"
+}
+```
+
+| Field | Type | Default (Postfix) | Default (Dovecot) | Description |
+|-------|------|-------------------|-------------------|-------------|
+| `mode` | string | `postfix` | `dovecot` | Service mode — determines defaults |
+| `cert_path` | string | `/etc/postfix/certs/cert.pem` | `/etc/dovecot/certs/cert.pem` | Path for certificate file |
+| `key_path` | string | `/etc/postfix/certs/key.pem` | `/etc/dovecot/certs/key.pem` | Path for private key (0600 permissions) |
+| `chain_path` | string | (empty) | (empty) | If set, chain written separately; otherwise appended to cert |
+| `reload_command` | string | `postfix reload` | `doveadm reload` | Command to reload the mail service |
+| `validate_command` | string | `postfix check` | `doveconf -n` | Optional config validation before reload |
+
+All commands are validated against shell injection via `validation.ValidateShellCommand()`. File permissions: cert/chain 0644, key 0600.
+
+Location: `internal/connector/target/postfix/postfix.go`
+
+### F5 BIG-IP (Implemented)
+
+The F5 BIG-IP target connector deploys certificates to F5 load balancers via the iControl REST API. F5 appliances can't run agents directly, so this connector uses the **proxy agent pattern**: a designated certctl agent in the same network zone polls for F5 deployment jobs and executes iControl REST calls on behalf of the control plane. Minimum supported BIG-IP version: 12.0+.
+
+The deployment flow uses F5's transaction API for atomic updates: authenticate via token auth, upload cert/key/chain PEM files, install as crypto objects, update the SSL client profile within a transaction, and commit. If the transaction fails, F5 rolls back automatically and the connector cleans up uploaded crypto objects. Updating an SSL profile automatically takes effect on all bound virtual servers — no separate virtual server binding step is needed.
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `host` | string | *(required)* | F5 BIG-IP management hostname or IP |
+| `port` | int | `443` | iControl REST API port |
+| `username` | string | *(required)* | Administrative username |
+| `password` | string | *(required)* | Administrative password |
+| `partition` | string | `Common` | F5 partition for crypto objects and profiles |
+| `ssl_profile` | string | *(required)* | SSL client profile name to update |
+| `insecure` | bool | `true` | Skip TLS verification for management interface (self-signed certs common) |
+| `timeout` | int | `30` | HTTP timeout in seconds |
 
-Configuration (defined, not yet functional):
 ```json
 {
   "host": "f5.internal.example.com",
+  "port": 443,
   "username": "admin",
   "password": "...",
   "partition": "Common",
-  "ssl_profile": "/Common/clientssl_api"
+  "ssl_profile": "clientssl_api",
+  "insecure": true,
+  "timeout": 30
 }
 ```
 
-Note: F5 credentials are stored on the proxy agent, not on the control plane server. This limits the credential blast radius to the proxy agent's network zone.
+F5 credentials are stored on the proxy agent, not on the control plane server. This limits the credential blast radius to the proxy agent's network zone. Config fields are validated against regex patterns to prevent injection.
 
 Location: `internal/connector/target/f5/f5.go`
 
-### IIS (Interface Only, Dual-Mode)
+### IIS (Implemented, Dual-Mode)
 
-The IIS target connector supports two planned deployment modes:
+The IIS target connector supports two deployment modes — agent-local (recommended) and proxy agent WinRM for agentless targets.
 
-**Agent-local (recommended):** A Windows agent runs directly on the IIS server and deploys certificates using PowerShell — `Import-PfxCertificate` to install into the certificate store and `Set-WebBinding` to bind to the IIS site. This is the preferred approach: no remote access needed, no credential management, same pull-based model as NGINX/Apache/HAProxy.
+**Agent-local (recommended):** A Windows agent runs directly on the IIS server and deploys certificates using PowerShell — `Import-PfxCertificate` to install into the certificate store and `Set-WebBinding` to bind to the IIS site. The agent handles PEM-to-PFX conversion via `go-pkcs12`, computes SHA-1 thumbprint from the certificate, and executes parameterized PowerShell scripts for injection-safe binding management. This is the preferred approach: no remote access needed, no credential management, same pull-based model as NGINX/Apache/HAProxy.
 
-**Proxy agent WinRM (for agentless targets):** For Windows servers where you don't want to install an agent, a nearby Windows agent acts as a proxy and reaches the IIS box via WinRM. The proxy agent picks up the deployment job, transfers the PFX bundle over WinRM, and runs the PowerShell commands remotely. WinRM credentials are stored on the proxy agent, not on the control plane.
+**Proxy agent WinRM (for agentless targets):** For Windows servers where you don't want to install an agent, a Linux or Windows proxy agent in the same network zone connects via WinRM (Windows Remote Management) and executes PowerShell commands remotely. The PFX bundle is base64-encoded, transferred inline in the WinRM session, decoded to a temp file on the remote host, imported, and the temp file is cleaned up in a `try/finally` block. WinRM credentials are configured on the target, not on the control plane. Uses the `masterzen/winrm` Go library with support for Basic, NTLM, and Kerberos authentication.
 
-Configuration (defined, not yet functional):
+**Agent-local configuration:**
 ```json
 {
-  "mode": "local",
+  "hostname": "iis-server.example.com",
   "site_name": "Default Web Site",
   "cert_store": "WebHosting",
-  "winrm_host": "",
-  "winrm_username": "",
-  "winrm_password": "",
-  "winrm_use_https": true
+  "port": 443,
+  "sni": true,
+  "ip_address": "*",
+  "binding_info": "www.example.com"
 }
 ```
 
-When `mode` is `"local"`, the `winrm_*` fields are ignored. When `mode` is `"proxy"`, the agent connects to the remote IIS server via WinRM using the provided credentials.
+**WinRM proxy configuration:**
+```json
+{
+  "hostname": "iis-server.example.com",
+  "site_name": "Default Web Site",
+  "cert_store": "WebHosting",
+  "port": 443,
+  "sni": true,
+  "ip_address": "*",
+  "mode": "winrm",
+  "winrm": {
+    "winrm_host": "iis-server.example.com",
+    "winrm_port": 5985,
+    "winrm_username": "Administrator",
+    "winrm_password": "...",
+    "winrm_https": false,
+    "winrm_insecure": false,
+    "winrm_timeout": 60
+  }
+}
+```
 
-Location: `internal/connector/target/iis/iis.go`
+**Configuration Fields:**
+- `hostname` (string, required): IIS server hostname or FQDN
+- `site_name` (string, required): IIS website name (e.g., "Default Web Site")
+- `cert_store` (string, required): Certificate store for import (e.g., "WebHosting", "My")
+- `port` (number, default 443): HTTPS binding port
+- `sni` (boolean, default false): Enable Server Name Indication (SNI)
+- `ip_address` (string, default "*"): Specific IP to bind to, or "*" for all IPs
+- `binding_info` (string, optional): Host header for SNI bindings
+- `mode` (string, default "local"): Deployment mode — `local` (agent-local PowerShell) or `winrm` (remote via WinRM)
+
+**WinRM fields (required when `mode` is `winrm`):**
+- `winrm.winrm_host` (string, required): Remote Windows server hostname or IP
+- `winrm.winrm_port` (number, default 5985 HTTP / 5986 HTTPS): WinRM listener port
+- `winrm.winrm_username` (string, required): Windows account with admin privileges
+- `winrm.winrm_password` (string, required): Account password
+- `winrm.winrm_https` (boolean, default false): Use HTTPS transport
+- `winrm.winrm_insecure` (boolean, default false): Skip TLS certificate verification
+- `winrm.winrm_timeout` (number, default 60): Operation timeout in seconds
+
+**Security Model:**
+- PFX files are transient — generated with random passwords, deleted after import
+- In WinRM mode, PFX data is base64-encoded and transferred inline (no SMB/file share needed), with remote temp file cleanup in `try/finally`
+- PowerShell commands use parameterized values — IIS names and cert stores are regex-validated before script execution
+- Field names are validated against `^[a-zA-Z0-9 _\-\.]+$` to prevent PowerShell injection
+- Certificate thumbprints computed via SHA-1 for IIS binding lookups
+
+Location: `internal/connector/target/iis/iis.go`, `internal/connector/target/iis/winrm.go`
+
+### SSH (Agentless Deployment)
+
+The SSH target connector enables agentless certificate deployment to any Linux/Unix server via SSH/SFTP. Instead of installing the certctl agent binary on every target, a single "proxy agent" in the same network zone deploys certificates to remote servers over SSH. This is ideal for environments where installing agents on every server is impractical.
+
+**Key authentication (recommended):**
+```json
+{
+  "host": "web-server.internal",
+  "port": 22,
+  "user": "certctl",
+  "auth_method": "key",
+  "private_key_path": "/home/certctl/.ssh/id_ed25519",
+  "cert_path": "/etc/ssl/certs/cert.pem",
+  "key_path": "/etc/ssl/private/key.pem",
+  "chain_path": "/etc/ssl/certs/chain.pem",
+  "reload_command": "systemctl reload nginx",
+  "timeout": 30
+}
+```
+
+**Password authentication:**
+```json
+{
+  "host": "legacy-server.internal",
+  "user": "deploy",
+  "auth_method": "password",
+  "password": "s3cret",
+  "cert_path": "/etc/ssl/cert.pem",
+  "key_path": "/etc/ssl/key.pem",
+  "reload_command": "systemctl reload apache2"
+}
+```
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `host` | string | *(required)* | SSH hostname or IP address |
+| `port` | number | 22 | SSH port |
+| `user` | string | *(required)* | SSH username |
+| `auth_method` | string | `"key"` | `"key"` or `"password"` |
+| `private_key_path` | string | | Path to SSH private key file (key auth) |
+| `private_key` | string | | Inline SSH private key PEM (alternative to path) |
+| `password` | string | | SSH password (password auth) |
+| `passphrase` | string | | Passphrase for encrypted private keys |
+| `cert_path` | string | *(required)* | Remote path for certificate file |
+| `key_path` | string | *(required)* | Remote path for private key file |
+| `chain_path` | string | | Remote path for chain file (if empty, chain appended to cert) |
+| `cert_mode` | string | `"0644"` | File permissions for cert (octal) |
+| `key_mode` | string | `"0600"` | File permissions for private key (octal) |
+| `reload_command` | string | | Command to execute after deployment |
+| `timeout` | number | 30 | SSH connection timeout in seconds |
+
+**Security:**
+- Key-based authentication is recommended over password authentication
+- Reload commands are validated against shell injection (same validation as Postfix/Dovecot connectors)
+- Host field is regex-validated to prevent shell metacharacters
+- Private keys are written with 0600 permissions by default
+- Host key verification is intentionally skipped (same rationale as network scanner and F5 connector — deploying to known, operator-configured infrastructure)
+- Encrypted private keys supported via passphrase
+
+Location: `internal/connector/target/ssh/ssh.go`
+
+### Windows Certificate Store
+
+The Windows Certificate Store connector imports certificates into the Windows cert store via PowerShell, without managing IIS site bindings. Use this for non-IIS Windows services that read certificates from the cert store (Exchange, RDP, SQL Server, ADFS, etc.). Same injectable `PowerShellExecutor` pattern as the IIS connector, with optional WinRM proxy mode.
+
+```json
+{
+  "store_name": "My",
+  "store_location": "LocalMachine",
+  "friendly_name": "Production API Cert",
+  "remove_expired": true
+}
+```
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `store_name` | string | `"My"` | Windows cert store name (My, Root, WebHosting, etc.) |
+| `store_location` | string | `"LocalMachine"` | `"LocalMachine"` or `"CurrentUser"` |
+| `friendly_name` | string | | Optional friendly name for the imported certificate |
+| `remove_expired` | boolean | `false` | Remove expired certs with same CN after import |
+| `mode` | string | `"local"` | `"local"` (agent-local) or `"winrm"` (remote) |
+| `winrm_host` | string | | WinRM hostname (required for winrm mode) |
+| `winrm_port` | number | 5985 | WinRM port (5985 HTTP, 5986 HTTPS) |
+| `winrm_username` | string | | WinRM username (required for winrm mode) |
+| `winrm_password` | string | | WinRM password (required for winrm mode) |
+| `winrm_https` | boolean | `false` | Use HTTPS for WinRM |
+| `winrm_insecure` | boolean | `false` | Skip TLS verification for WinRM |
+
+Location: `internal/connector/target/wincertstore/wincertstore.go`
+
+### Java Keystore (JKS / PKCS#12)
+
+The Java Keystore connector deploys certificates to JKS or PKCS#12 keystores via the `keytool` CLI. This enables TLS cert deployment for Tomcat, Jetty, Kafka, Elasticsearch, and any JVM-based service. Flow: PEM to temp PKCS#12, then `keytool -importkeystore` into the target keystore.
+
+```json
+{
+  "keystore_path": "/opt/tomcat/conf/keystore.p12",
+  "keystore_password": "changeit",
+  "keystore_type": "PKCS12",
+  "alias": "server",
+  "reload_command": "systemctl restart tomcat"
+}
+```
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `keystore_path` | string | *(required)* | Absolute path to the keystore file |
+| `keystore_password` | string | *(required)* | Keystore password |
+| `keystore_type` | string | `"PKCS12"` | `"PKCS12"` or `"JKS"` |
+| `alias` | string | `"server"` | Key entry alias in the keystore |
+| `reload_command` | string | | Optional command to run after keystore update |
+| `create_keystore` | boolean | `true` | Create keystore if it doesn't exist |
+| `keytool_path` | string | `"keytool"` | Override keytool binary path |
+
+**Security:**
+- Reload commands validated against shell injection via `validation.ValidateShellCommand()`
+- Alias validated against injection (alphanumeric, hyphens, underscores only)
+- Path traversal prevention on keystore path
+- Transient PKCS#12 temp file cleaned up after import (even on error)
+
+Location: `internal/connector/target/javakeystore/javakeystore.go`
+
+### Kubernetes Secrets
+
+The Kubernetes Secrets connector deploys certificates as `kubernetes.io/tls` Secrets, compatible with Ingress controllers (nginx-ingress, Traefik, HAProxy), service meshes (Istio, Linkerd), and any Kubernetes workload that reads TLS Secrets.
+
+```json
+{
+  "namespace": "production",
+  "secret_name": "api-tls",
+  "labels": {"app": "api-gateway"},
+  "kubeconfig_path": "/home/agent/.kube/config"
+}
+```
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `namespace` | string | *(required)* | Kubernetes namespace (DNS-1123, max 63 chars) |
+| `secret_name` | string | *(required)* | Secret name (DNS subdomain, max 253 chars) |
+| `labels` | object | | Additional labels to apply to the Secret |
+| `kubeconfig_path` | string | | Path to kubeconfig for out-of-cluster agents |
+
+**Deployment modes:**
+- **In-cluster (default):** Agent runs as a Pod with a ServiceAccount. Authentication via auto-mounted token. Requires RBAC (`secrets.get`, `secrets.create`, `secrets.update`, `secrets.list`) — see Helm chart.
+- **Out-of-cluster:** Agent runs outside the cluster with `kubeconfig_path` pointing to a kubeconfig file. Useful for proxy agent pattern.
+
+**Secret format:** Standard `kubernetes.io/tls` with `tls.crt` (cert + chain PEM) and `tls.key` (private key PEM). Managed labels (`app.kubernetes.io/managed-by: certctl`) and annotations (`certctl.io/deployed-at`, `certctl.io/certificate-id`) are applied automatically.
+
+**Validation:** After deployment, the connector reads the Secret back and compares the certificate serial number to verify successful deployment.
+
+Location: `internal/connector/target/k8ssecret/k8ssecret.go`
 
 ## Notifier Connector
 
@@ -620,11 +1092,86 @@ type Connector interface {
 
 Built-in notifiers: **Email** (SMTP), **Webhook** (HTTP POST), **Slack** (incoming webhook), **Microsoft Teams** (MessageCard webhook), **PagerDuty** (Events API v2), and **OpsGenie** (Alert API v2).
 
+### Email (SMTP) Notifier
+
+The Email notifier sends transactional alerts and scheduled digests via SMTP. It bridges the connector-layer SMTP connector to the service-layer `Notifier` interface via the `NotifierAdapter`. Supports both plain text and HTML emails.
+
+Configuration:
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `CERTCTL_SMTP_HOST` | — | SMTP server hostname (required to enable) |
+| `CERTCTL_SMTP_PORT` | 587 | SMTP port (TLS) |
+| `CERTCTL_SMTP_USERNAME` | — | SMTP authentication username (optional) |
+| `CERTCTL_SMTP_PASSWORD` | — | SMTP authentication password (optional) |
+| `CERTCTL_SMTP_FROM_ADDRESS` | — | Email from address (required) |
+| `CERTCTL_SMTP_USE_TLS` | true | Enable TLS encryption |
+
+Example:
+```bash
+export CERTCTL_SMTP_HOST=smtp.gmail.com
+export CERTCTL_SMTP_PORT=587
+export CERTCTL_SMTP_USERNAME=admin@example.com
+export CERTCTL_SMTP_PASSWORD=app-password-123
+export CERTCTL_SMTP_FROM_ADDRESS=certctl@example.com
+```
+
+### Scheduled Certificate Digest
+
+The `DigestService` generates aggregated certificate digest emails and sends them on a configurable schedule. This is useful for periodic briefings on certificate inventory health — expiring certs, status summary, active agents, job trends.
+
+The digest HTML template includes:
+- Total certificates, expiring soon, expired, active agents (stats grid)
+- Jobs completed/failed summary (30 days)
+- Expiring certificates table (color-coded by urgency: 7d, 14d, 30d)
+- Auto-refresh and responsive email layout
+
+**Scheduler Integration:** The opt-in digest scheduler loop runs on configurable interval (default 24 hours). It does NOT run on startup — waits for first scheduled tick. Operation timeout is 5 minutes. Each loop execution is guarded by `sync/atomic.Bool` idempotency. See `docs/architecture.md` for the full scheduler topology (12 loops, 8 always-on + 4 opt-in).
+
+Configuration:
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `CERTCTL_DIGEST_ENABLED` | false | Enable scheduled digest emails |
+| `CERTCTL_DIGEST_INTERVAL` | 24h | How often to send digest (any duration, e.g. 12h, 7d) |
+| `CERTCTL_DIGEST_RECIPIENTS` | — | Comma-separated email addresses. Falls back to certificate owner emails if empty |
+
+API Endpoints:
+
+- **`GET /api/v1/digest/preview`** — Render digest HTML for preview (no email sent)
+- **`POST /api/v1/digest/send`** — Trigger digest send immediately (outside of schedule)
+
+> **Note (HTTPS-only as of v2.2):** The `curl` examples in this section
+> and below all target the HTTPS-only control plane. Extract the
+> docker-compose self-signed bootstrap CA bundle once and reuse it on
+> every call:
+>
+> ```bash
+> export CA=/tmp/certctl-ca.crt
+> docker compose -f deploy/docker-compose.yml exec -T certctl-server \
+>   cat /etc/certctl/tls/ca.crt > "$CA"
+> ```
+>
+> Then pass `--cacert "$CA"` (or `-k` for one-off smoke tests, never in
+> production). The same pattern is documented in
+> [`quickstart.md`](quickstart.md). Pre-U-2 these examples used `http://`
+> and silently failed against the HTTPS listener; post-U-2 they speak
+> HTTPS with the operator-managed CA bundle.
+
+Example:
+```bash
+# Preview digest
+curl --cacert "$CA" https://localhost:8443/api/v1/digest/preview | jq '.html'
+
+# Send digest immediately
+curl --cacert "$CA" -X POST https://localhost:8443/api/v1/digest/send
+```
+
 Each notifier is enabled by its configuration env var:
 
 | Notifier | Env Var | Description |
 |----------|---------|-------------|
-| Email | `CERTCTL_EMAIL_SMTP_HOST`, `CERTCTL_EMAIL_SMTP_PORT`, `CERTCTL_EMAIL_FROM` | SMTP email delivery. Optional: `CERTCTL_EMAIL_SMTP_USERNAME`, `CERTCTL_EMAIL_SMTP_PASSWORD` |
+| Email | `CERTCTL_SMTP_HOST` | SMTP email delivery. See Email Notifier section above |
 | Webhook | `CERTCTL_WEBHOOK_URL` | HTTP POST to any endpoint. Optional: `CERTCTL_WEBHOOK_SECRET` for HMAC signing |
 | Slack | `CERTCTL_SLACK_WEBHOOK_URL` | Incoming webhook URL. Optional: `CERTCTL_SLACK_CHANNEL`, `CERTCTL_SLACK_USERNAME` |
 | Teams | `CERTCTL_TEAMS_WEBHOOK_URL` | Incoming webhook URL (MessageCard format) |
@@ -764,24 +1311,24 @@ The agent scans these directories on startup and every 6 hours, looking for cert
 
 ```bash
 # List discovered certificates (filter by agent, status)
-curl -s "http://localhost:8443/api/v1/discovered-certificates?agent_id=agent-nginx-01&status=new" | jq .
+curl --cacert "$CA" -s "https://localhost:8443/api/v1/discovered-certificates?agent_id=agent-nginx-01&status=new" | jq .
 
 # Get discovery detail
-curl -s http://localhost:8443/api/v1/discovered-certificates/DISCOVERY_ID | jq .
+curl --cacert "$CA" -s https://localhost:8443/api/v1/discovered-certificates/DISCOVERY_ID | jq .
 
 # Claim a discovered cert (link to managed certificate)
-curl -s -X POST http://localhost:8443/api/v1/discovered-certificates/DISCOVERY_ID/claim \
+curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/discovered-certificates/DISCOVERY_ID/claim \
   -H "Content-Type: application/json" \
   -d '{"managed_certificate_id": "mc-api-prod"}' | jq .
 
 # Dismiss a discovery
-curl -s -X POST http://localhost:8443/api/v1/discovered-certificates/DISCOVERY_ID/dismiss | jq .
+curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/discovered-certificates/DISCOVERY_ID/dismiss | jq .
 
 # View discovery scan history
-curl -s http://localhost:8443/api/v1/discovery-scans | jq .
+curl --cacert "$CA" -s https://localhost:8443/api/v1/discovery-scans | jq .
 
 # Summary counts (new, claimed, dismissed)
-curl -s http://localhost:8443/api/v1/discovery-summary | jq .
+curl --cacert "$CA" -s https://localhost:8443/api/v1/discovery-summary | jq .
 ```
 
 ### Use Cases
@@ -810,7 +1357,7 @@ Network scan targets can be managed from the **Network Scans** dashboard page (c
 
 ```bash
 # Create a scan target for your internal network (or use the dashboard's "+ New Target" button)
-curl -s -X POST http://localhost:8443/api/v1/network-scan-targets \
+curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/network-scan-targets \
   -H "Content-Type: application/json" \
   -d '{
     "name": "Production Web Servers",
@@ -835,31 +1382,31 @@ curl -s -X POST http://localhost:8443/api/v1/network-scan-targets \
 
 ```bash
 # List all scan targets
-curl -s http://localhost:8443/api/v1/network-scan-targets | jq .
+curl --cacert "$CA" -s https://localhost:8443/api/v1/network-scan-targets | jq .
 
 # Create a scan target
-curl -s -X POST http://localhost:8443/api/v1/network-scan-targets \
+curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/network-scan-targets \
   -H "Content-Type: application/json" \
   -d '{"name": "DMZ", "cidrs": ["172.16.0.0/24"], "ports": [443]}' | jq .
 
 # Get a specific target (includes last_scan_at, last_scan_certs_found)
-curl -s http://localhost:8443/api/v1/network-scan-targets/nst-dmz | jq .
+curl --cacert "$CA" -s https://localhost:8443/api/v1/network-scan-targets/nst-dmz | jq .
 
 # Trigger an immediate scan (doesn't wait for scheduler)
-curl -s -X POST http://localhost:8443/api/v1/network-scan-targets/nst-dmz/scan | jq .
+curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/network-scan-targets/nst-dmz/scan | jq .
 
 # Update scan configuration
-curl -s -X PUT http://localhost:8443/api/v1/network-scan-targets/nst-dmz \
+curl --cacert "$CA" -s -X PUT https://localhost:8443/api/v1/network-scan-targets/nst-dmz \
   -H "Content-Type: application/json" \
   -d '{"ports": [443, 8443, 9443], "timeout_ms": 3000}' | jq .
 
 # Delete a scan target
-curl -s -X DELETE http://localhost:8443/api/v1/network-scan-targets/nst-dmz
+curl --cacert "$CA" -s -X DELETE https://localhost:8443/api/v1/network-scan-targets/nst-dmz
 ```
 
 ### Scheduler Integration
 
-When `CERTCTL_NETWORK_SCAN_ENABLED=true`, the server runs a 6th scheduler loop (alongside renewal, jobs, health, notifications, and short-lived expiry). It scans all enabled targets at the configured interval (default 6h). Each target tracks `last_scan_at`, `last_scan_duration_ms`, and `last_scan_certs_found` for monitoring scan health.
+When `CERTCTL_NETWORK_SCAN_ENABLED=true`, the server runs the opt-in network scanner scheduler loop alongside the always-on loops (renewal, jobs, job retry, job timeout, agent health, notifications, notification retry, short-lived expiry). It scans all enabled targets at the configured interval (default 6h). Each target tracks `last_scan_at`, `last_scan_duration_ms`, and `last_scan_certs_found` for monitoring scan health. See `docs/architecture.md` for the full 12-loop scheduler topology.
 
 ### Use Cases
 
@@ -869,6 +1416,63 @@ When `CERTCTL_NETWORK_SCAN_ENABLED=true`, the server runs a 6th scheduler loop (
 - **Migration assessment** — Scan a network range before onboarding to certctl management
 - **Expiration monitoring** — Discover soon-to-expire certs on network endpoints before they cause outages
 
+## Cloud Secret Manager Discovery
+
+certctl extends the existing filesystem and network discovery pipeline to cloud secret managers. Certificates stored in cloud vaults are automatically discovered, inventoried, and available for triage in the Discovery page.
+
+Each cloud source runs as a pluggable `DiscoverySource` with its own sentinel agent ID. Discovered certificates flow through the same `ProcessDiscoveryReport` pipeline used by filesystem and network discovery — dedup by fingerprint, audit trail, status tracking.
+
+### AWS Secrets Manager
+
+Discovers certificates stored as secrets in AWS Secrets Manager. Filters by tag (`type=certificate` by default) and optional name prefix.
+
+| Variable | Description | Default |
+|---|---|---|
+| `CERTCTL_CLOUD_DISCOVERY_ENABLED` | Enable cloud discovery scheduler | `false` |
+| `CERTCTL_AWS_SM_DISCOVERY_ENABLED` | Enable AWS SM source | `false` |
+| `CERTCTL_AWS_SM_REGION` | AWS region (e.g., `us-east-1`) | — |
+| `CERTCTL_AWS_SM_TAG_FILTER` | Tag key=value filter | `type=certificate` |
+| `CERTCTL_AWS_SM_NAME_PREFIX` | Secret name prefix filter | — |
+
+Source path format: `aws-sm://{region}/{secret-name}`. Sentinel agent: `cloud-aws-sm`.
+
+### Azure Key Vault
+
+Discovers certificates from Azure Key Vault using OAuth2 client credentials authentication. No Azure SDK dependency — uses stdlib HTTP with Azure AD token exchange.
+
+| Variable | Description | Default |
+|---|---|---|
+| `CERTCTL_AZURE_KV_DISCOVERY_ENABLED` | Enable Azure KV source | `false` |
+| `CERTCTL_AZURE_KV_VAULT_URL` | Vault URL (e.g., `https://myvault.vault.azure.net`) | — |
+| `CERTCTL_AZURE_KV_TENANT_ID` | Azure AD tenant ID | — |
+| `CERTCTL_AZURE_KV_CLIENT_ID` | Azure AD application (client) ID | — |
+| `CERTCTL_AZURE_KV_CLIENT_SECRET` | Azure AD application secret | — |
+
+Source path format: `azure-kv://{cert-name}/{version}`. Sentinel agent: `cloud-azure-kv`.
+
+### GCP Secret Manager
+
+Discovers certificates stored in GCP Secret Manager. Filters by label (`type=certificate`). Uses JWT-based OAuth2 service account auth — no Google SDK dependency.
+
+| Variable | Description | Default |
+|---|---|---|
+| `CERTCTL_GCP_SM_DISCOVERY_ENABLED` | Enable GCP SM source | `false` |
+| `CERTCTL_GCP_SM_PROJECT` | GCP project ID | — |
+| `CERTCTL_GCP_SM_CREDENTIALS` | Path to service account JSON file | — |
+
+Source path format: `gcp-sm://{project}/{secret-name}`. Sentinel agent: `cloud-gcp-sm`.
+
+### Cloud Discovery Scheduler
+
+All enabled cloud sources run on a shared opt-in cloud discovery scheduler loop (see `docs/architecture.md` for the full 12-loop scheduler topology). The interval is configurable:
+
+| Variable | Description | Default |
+|---|---|---|
+| `CERTCTL_CLOUD_DISCOVERY_ENABLED` | Master switch | `false` |
+| `CERTCTL_CLOUD_DISCOVERY_INTERVAL` | Scan interval | `6h` |
+
+The loop runs immediately on startup and then on each tick. Each source runs sequentially within the loop. Errors from one source do not prevent other sources from running.
+
 ## What's Next
 
 - [Architecture Guide](architecture.md) — Understanding the full system design

docs/database-tls.md

@@ -0,0 +1,117 @@
+# Database TLS — Postgres Transport Encryption
+
+**Audit reference:** Bundle B / M-018. PCI-DSS v4.0 Req 4 §2.2.5; CWE-319.
+
+certctl talks to Postgres over a single connection-string URL controlled by the
+`CERTCTL_DATABASE_URL` env var. The `sslmode` query parameter on that URL
+selects the transport-encryption posture. Pre-Bundle-B all the bundled
+deployment artifacts (Helm chart, docker-compose) hard-coded `sslmode=disable`.
+Bundle B exposes that as an operator-facing knob with a documented default and
+explicit opt-in / opt-out paths for the four real-world deployment shapes.
+
+## Quick reference
+
+| Deployment shape                               | Default `sslmode` | When to change |
+|------------------------------------------------|--------------------|----------------|
+| Helm chart, bundled Postgres, in-cluster       | `disable`          | When the cluster does not provide pod-network encryption (CNI without WireGuard / IPSec) and the workload is in PCI-DSS scope. |
+| Helm chart, external Postgres (RDS / Cloud SQL / Azure DB) | not auto-set | **Always** set to `verify-full` and provide the cloud provider's server CA bundle. |
+| docker-compose, bundled Postgres on docker bridge | `disable`        | Demo/dev only; not a deployment shape we expect operators to harden. |
+| docker-compose / k8s with external Postgres    | not auto-set       | **Always** set `CERTCTL_DATABASE_URL` to a connection string with `sslmode=verify-full`. |
+
+`sslmode` values come from `lib/pq` (the underlying driver). The full set is:
+`disable`, `allow`, `prefer`, `require`, `verify-ca`, `verify-full`. PCI-DSS
+Req 4 v4.0 §2.2.5 considers `verify-ca` the floor for sensitive-data transport;
+`verify-full` is the floor for systems exposed to spoofing risk (it adds
+hostname validation against the server cert's CN/SAN).
+
+## Helm chart (Bundle B)
+
+Bundle B adds two values under `postgresql.tls`:
+
+```yaml
+postgresql:
+  tls:
+    mode: disable          # disable | require | verify-ca | verify-full
+    caSecretRef: ""        # Secret with ca.crt key (required for verify-ca / verify-full)
+```
+
+The chart pipes `postgresql.tls.mode` into the `?sslmode=` parameter of the
+generated `CERTCTL_DATABASE_URL` (see `templates/_helpers.tpl::certctl.databaseURL`).
+For external Postgres, set `postgresql.enabled: false` and override
+`server.env.CERTCTL_DATABASE_URL` directly with the full connection string —
+the operator authoring an external-DB values file owns the entire URL.
+
+### Example: external RDS with verify-full
+
+```yaml
+postgresql:
+  enabled: false   # Disable bundled Postgres
+
+server:
+  env:
+    CERTCTL_DATABASE_URL: |
+      postgres://certctl:STRONGPW@my-db.cabc12345.us-east-1.rds.amazonaws.com:5432/certctl?sslmode=verify-full
+
+# Provide the AWS RDS root CA bundle as a secret + mount.
+# AWS publishes per-region root certs at https://truststore.pki.rds.amazonaws.com/
+extraVolumes:
+  - name: rds-ca
+    secret:
+      secretName: rds-ca-bundle  # kubectl create secret generic rds-ca-bundle --from-file=ca.crt=...
+
+extraVolumeMounts:
+  - name: rds-ca
+    mountPath: /etc/postgresql-ca
+    readOnly: true
+
+# lib/pq honors PGSSLROOTCERT for the verify-{ca,full} CA bundle path.
+server:
+  env:
+    PGSSLROOTCERT: /etc/postgresql-ca/ca.crt
+```
+
+## docker-compose (development / demo)
+
+The bundled `deploy/docker-compose.yml` keeps `sslmode=disable` as the default
+because the Postgres container shares the docker bridge network with the certctl
+server and the compose file is not a production deployment artifact. To opt in:
+
+```bash
+export CERTCTL_DATABASE_URL='postgres://certctl:certctl@postgres:5432/certctl?sslmode=verify-full'
+docker compose up
+```
+
+## Verification
+
+For any non-`disable` mode, confirm the connection actually negotiated TLS:
+
+```bash
+# From inside the certctl-server container or any host with psql + the same URL:
+psql "$CERTCTL_DATABASE_URL" -c "SELECT ssl, version, cipher FROM pg_stat_ssl WHERE pid = pg_backend_pid();"
+
+# Expected output for verify-full: ssl=t, version=TLSv1.3 (or TLSv1.2), cipher=...
+```
+
+If `ssl=f` appears, the connection silently fell back to plaintext — investigate
+the cert chain or sslmode value before treating the deployment as PCI-compliant.
+
+## What this does NOT cover
+
+* **Postgres-to-Postgres replication** — if you run a replica, replica-primary
+  TLS is configured via the Postgres server itself (`pg_hba.conf` +
+  `ssl=on`); it is independent of certctl's `CERTCTL_DATABASE_URL`.
+* **Backup transport** — `pg_dump` / `pg_basebackup` honor the same `sslmode`
+  parameter when invoked with the URL form, but the bundled chart's backup
+  story (if any) is operator-owned.
+* **Encryption at rest** — `sslmode` is a transport concern only. Disk
+  encryption is the cloud provider's storage layer (RDS, EBS, etc.) or the
+  operator's Postgres TDE / disk LUKS / etc.
+
+## Reverting
+
+If `sslmode=verify-full` causes connection failures (most common: missing CA
+bundle, wrong hostname), drop temporarily to `sslmode=require` to confirm TLS
+is at least negotiated, then add the CA bundle and ratchet back up. Never
+revert to `sslmode=disable` on a system carrying real cert metadata —
+audit_events alone contains enough operator/issuer/target identity to justify
+TLS in any scoped environment.

docs/demo-advanced.md

@@ -50,14 +50,17 @@ docker compose -f deploy/docker-compose.yml up -d --build
 docker compose -f deploy/docker-compose.yml ps
 ```
 
-Open **http://localhost:8443** in your browser alongside your terminal. You'll watch changes appear in the dashboard as you make API calls.
+Open **https://localhost:8443** in your browser alongside your terminal. The default compose stack ships a self-signed cert; your browser will show a warning the first time — click through (or trust `deploy/test/certs/ca.crt` in your OS keychain). You'll watch changes appear in the dashboard as you make API calls.
 
-Set up a base variable for convenience:
+Set up base variables for convenience:
 
 ```bash
-API="http://localhost:8443"
+API="https://localhost:8443"
+CA="$PWD/deploy/test/certs/ca.crt"   # pin the self-signed CA for curl
 ```
 
+Every `curl` in this guide uses `--cacert "$CA"` so the TLS handshake verifies against the compose-stack CA instead of the system trust store.
+
 ## How the pieces fit together
 
 Before we start, here's the high-level flow of what we're about to do:
@@ -307,8 +310,8 @@ flowchart TD
     A --> F["ACME\n(Let's Encrypt)"]
     A --> G["step-ca\n(implemented)"]
     A --> H["OpenSSL / Custom CA\n(script-based)"]
-    A --> J["DigiCert API\n(planned)"]
-    A --> K["Vault PKI\n(planned)"]
+    A --> J["DigiCert API\n(implemented)"]
+    A --> K["Vault PKI\n(implemented)"]
     A --> L["Entrust / GlobalSign\n(planned)"]
     A --> M["Google CAS / EJBCA\n(planned)"]
 ```
@@ -724,22 +727,24 @@ curl -s -X POST $API/api/v1/certificates/mc-demo-payments/revoke \
 6. Creates an audit trail entry
 7. Sends revocation notifications via configured channels
 
-Check the CRL (Certificate Revocation List):
+Check the CRL (Certificate Revocation List) — served unauthenticated under the RFC 8615 well-known namespace so relying parties without a certctl API key can still verify revocation (RFC 5280 §5):
 
 ```bash
-# JSON-formatted CRL
-curl -s $API/api/v1/crl | jq .
-
-# DER-encoded X.509 CRL for the local CA (binary — pipe to openssl for inspection)
-curl -s $API/api/v1/crl/iss-local -o /tmp/crl.der
+# DER-encoded X.509 CRL for the local CA (binary — pipe to openssl for inspection).
+# Note: no -H "Authorization: Bearer ..." — the endpoint is deliberately
+# unauthenticated. Content-Type is application/pkix-crl.
+curl --cacert "$CA" -s https://localhost:8443/.well-known/pki/crl/iss-local -o /tmp/crl.der
 openssl crl -inform DER -in /tmp/crl.der -text -noout
 ```
 
-Check OCSP status:
+Check OCSP status (RFC 6960, also unauthenticated, `application/ocsp-response`):
 
 ```bash
-# Replace SERIAL with the actual serial number from the certificate version
-curl -s $API/api/v1/ocsp/iss-local/SERIAL | jq .
+# Replace SERIAL with the actual serial number from the certificate version.
+# The embedded OCSP responder returns a signed DER response — parse it with
+# `openssl ocsp -respin` or similar tooling.
+curl --cacert "$CA" -s https://localhost:8443/.well-known/pki/ocsp/iss-local/SERIAL -o /tmp/ocsp.der
+openssl ocsp -respin /tmp/ocsp.der -noverify -resp_text | head -40
 ```
 
 **Why RFC 5280 reason codes:** The reason code isn't just metadata — it tells clients *why* the certificate was revoked. A `keyCompromise` revocation means the private key was exposed and the certificate should be distrusted immediately. A `superseded` revocation means a newer certificate replaced it — less urgent. CRLs and OCSP responses include the reason code so client software can make informed trust decisions.
@@ -944,7 +949,8 @@ certctl includes a standalone CLI tool for command-line users:
 cd cmd/cli && go build -o certctl-cli .
 
 # Export credentials
-export CERTCTL_SERVER_URL="http://localhost:8443"
+export CERTCTL_SERVER_URL="https://localhost:8443"
+export CERTCTL_SERVER_CA_BUNDLE_PATH="$PWD/deploy/test/certs/ca.crt"
 export CERTCTL_API_KEY="test-key-123"
 
 # List certificates (JSON or table format)
@@ -981,14 +987,15 @@ export CERTCTL_API_KEY="test-key-123"
 
 ## Part 15: MCP Server for AI Integration (M18a)
 
-certctl exposes 78 MCP tools covering the REST API via the Model Context Protocol (MCP), enabling seamless integration with Claude, Cursor, and other AI assistants:
+certctl exposes the full REST API via the Model Context Protocol (MCP), enabling seamless integration with Claude, Cursor, and other AI assistants:
 
 ```bash
 # Build the MCP server
 cd cmd/mcp-server && go build -o mcp-server .
 
 # Export credentials
-export CERTCTL_SERVER_URL="http://localhost:8443"
+export CERTCTL_SERVER_URL="https://localhost:8443"
+export CERTCTL_SERVER_CA_BUNDLE_PATH="$PWD/deploy/test/certs/ca.crt"
 export CERTCTL_API_KEY="test-key-123"
 
 # Start the MCP server (listens on stdin/stdout)
@@ -1046,7 +1053,7 @@ docker compose -f deploy/docker-compose.yml run -e CERTCTL_DISCOVERY_DIRS=/tmp/c
 Or with the CLI flag:
 
 ```bash
-certctl-agent --agent-id a-demo-1 --key-dir /tmp/keys --discovery-dirs /tmp/certs --server http://localhost:8443 --api-key test-key-123
+certctl-agent --agent-id a-demo-1 --key-dir /tmp/keys --discovery-dirs /tmp/certs --server https://localhost:8443 --ca-bundle "$CA" --api-key test-key-123
 ```
 
 ### Network Discovery (Server-Side)
@@ -1153,7 +1160,7 @@ flowchart TB
         API["REST API\nGo net/http"]
         SVC["Service Layer\nBusiness Logic"]
         REPO["Repository Layer\ndatabase/sql + lib/pq"]
-        SCHED["Scheduler\n6 background loops"]
+        SCHED["Scheduler\n12 background loops\n(8 always-on + 4 opt-in)"]
         CONN["Connector Registry\nIssuer + Target + Notifier"]
     end
 
@@ -1189,7 +1196,8 @@ Here's a single script that runs the entire demo end-to-end. Save it as `demo.sh
 #!/bin/bash
 set -e
 
-API="http://localhost:8443"
+API="https://localhost:8443"
+CA="$PWD/deploy/test/certs/ca.crt"   # pin the self-signed CA for curl
 BLUE='\033[0;34m'
 GREEN='\033[0;32m'
 YELLOW='\033[1;33m'
@@ -1297,7 +1305,7 @@ echo "  5. Revoked the certificate with RFC 5280 reason codes"
 echo "  6. Checked dashboard stats and metrics"
 echo "  7. All actions recorded in the audit trail"
 echo ""
-echo -e "Open ${GREEN}http://localhost:8443${NC} to see everything in the dashboard."
+echo -e "Open ${GREEN}https://localhost:8443${NC} to see everything in the dashboard."
 echo "Look for certificate: $CERT_ID"
 ```
 

docs/examples.md

@@ -0,0 +1,120 @@
+# Deployment Examples
+
+Five turnkey docker-compose scenarios, each runnable in under 5 minutes. Pick the one closest to your setup.
+
+## Which Example Should I Use?
+
+| I need to... | Example | Issuer | Target |
+|--------------|---------|--------|--------|
+| Get Let's Encrypt certs for NGINX on a public server | [ACME + NGINX](#acme--nginx) | ACME (HTTP-01) | NGINX |
+| Issue wildcard certs without opening port 80 | [Wildcard DNS-01](#wildcard-dns-01) | ACME (DNS-01) | Any |
+| Run an internal CA for services behind a firewall | [Private CA + Traefik](#private-ca--traefik) | Local CA | Traefik |
+| Use Smallstep step-ca as my PKI backend | [step-ca + HAProxy](#step-ca--haproxy) | step-ca | HAProxy |
+| Manage both public and internal certs from one dashboard | [Multi-Issuer](#multi-issuer) | ACME + Local CA | Mixed |
+
+**Already using another tool?** See the migration sections below each example for Certbot, acme.sh, and cert-manager users.
+
+---
+
+## ACME + NGINX
+
+**Scenario:** You have one or more public-facing domains, NGINX as the reverse proxy, and want automated Let's Encrypt certificates with HTTP-01 challenges.
+
+**What it deploys:** certctl server + PostgreSQL + certctl agent + NGINX, all on one Docker network. The agent generates keys locally (ECDSA P-256), submits CSRs to the server, receives signed certs from Let's Encrypt, and deploys them to NGINX with automatic reload.
+
+**Prerequisites:** A domain pointing to your server, ports 80 and 443 open, Docker Compose v20.10+.
+
+```bash
+cd examples/acme-nginx
+cp .env.example .env   # Edit with your domain and email
+docker compose up -d
+```
+
+The full walkthrough — including how HTTP-01 challenges work, adding multiple domains, switching to staging for testing, and a production checklist — is in the [example README](../examples/acme-nginx/acme-nginx.md).
+
+**Migrating from Certbot?** certctl discovers your existing `/etc/letsencrypt/live/` certificates automatically. You keep your ACME account, disable the Certbot cron, and certctl takes over renewal with centralized visibility and deployment verification. The step-by-step process is in [Migrating from Certbot](migrate-from-certbot.md).
+
+---
+
+## Wildcard DNS-01
+
+**Scenario:** You need wildcard certificates (`*.example.com`) or your servers aren't reachable from the internet (no port 80). DNS-01 validates ownership by creating a TXT record at your DNS provider.
+
+**What it deploys:** certctl server + PostgreSQL + certctl agent. Includes a Cloudflare DNS hook script as a working reference — swap in your own DNS provider (Route53, Azure DNS, Google Cloud DNS, or any provider with an API).
+
+**Prerequisites:** A domain, API credentials for your DNS provider, Docker Compose.
+
+```bash
+cd examples/acme-wildcard-dns01
+cp .env.example .env   # Edit with domain, email, DNS provider credentials
+docker compose up -d
+```
+
+The full walkthrough — including DNS-PERSIST-01 (set a TXT record once, never touch DNS again on renewals), adapting scripts for other providers, and propagation troubleshooting — is in the [example README](../examples/acme-wildcard-dns01/acme-wildcard-dns01.md).
+
+**Migrating from acme.sh?** Your existing `dns_*` hook scripts are compatible with certctl's DNS-01 — they use the same pattern (shell scripts creating TXT records). The migration guide covers script adaptation, discovery of existing acme.sh certificates, and phasing out the acme.sh cron. See [Migrating from acme.sh](migrate-from-acmesh.md).
+
+---
+
+## Private CA + Traefik
+
+**Scenario:** Internal services that don't need public CA validation. You run your own certificate authority — either a self-signed root for development, or a subordinate CA chained to your enterprise root (e.g., Active Directory Certificate Services).
+
+**What it deploys:** certctl server + PostgreSQL + certctl agent + Traefik. The Local CA issuer signs certificates directly. Traefik watches a cert directory and auto-reloads when new files appear.
+
+**Prerequisites:** Docker Compose. For sub-CA mode, you'll need a CA certificate and key signed by your enterprise root.
+
+```bash
+cd examples/private-ca-traefik
+docker compose up -d    # Self-signed mode (no .env needed for demo)
+```
+
+The full walkthrough — including sub-CA setup with `CERTCTL_CA_CERT_PATH` and `CERTCTL_CA_KEY_PATH`, creating certificates via the API, monitoring deployments, and production hardening — is in the [example README](../examples/private-ca-traefik/private-ca-traefik.md).
+
+---
+
+## step-ca + HAProxy
+
+**Scenario:** You use Smallstep's step-ca as your private PKI and want automated lifecycle management for certificates deployed to HAProxy load balancers.
+
+**What it deploys:** certctl server + PostgreSQL + certctl agent + step-ca (with JWK provisioner) + HAProxy. certctl issues certs via step-ca's native `/sign` API, combines them into HAProxy's expected PEM format (cert + chain + key in one file), and reloads HAProxy.
+
+**Prerequisites:** Docker Compose.
+
+```bash
+cd examples/step-ca-haproxy
+docker compose up -d
+```
+
+The full walkthrough — including step-ca provisioner configuration, integrating with an existing step-ca instance, HAProxy PEM format details, and advanced features (approval workflows, policy-based renewal, multi-instance HAProxy) — is in the [example README](../examples/step-ca-haproxy/step-ca-haproxy.md).
+
+---
+
+## Multi-Issuer
+
+**Scenario:** You manage both public-facing services (needing Let's Encrypt or another public CA) and internal services (using a private CA) and want a single dashboard for everything.
+
+**What it deploys:** certctl server + PostgreSQL + certctl agent configured with both an ACME issuer and a Local CA issuer. Demonstrates issuer assignment via profiles — public services get ACME certs, internal services get Local CA certs, all visible in one inventory.
+
+**Prerequisites:** Docker Compose. For real ACME certs, a public domain and port 80 access.
+
+```bash
+cd examples/multi-issuer
+docker compose up -d
+```
+
+The full walkthrough — including profile-based issuer assignment, testing with ACME staging, Local CA enterprise sub-CA mode, and scaling beyond Docker Compose — is in the [example README](../examples/multi-issuer/multi-issuer.md).
+
+**Using cert-manager for Kubernetes?** certctl complements cert-manager — cert-manager handles in-cluster certs, certctl handles everything outside: VMs, bare metal, network appliances, Windows servers. They can share the same CA (ACME, step-ca, Vault PKI). See [certctl for cert-manager Users](certctl-for-cert-manager-users.md).
+
+---
+
+## Beyond These Examples
+
+These 5 scenarios cover the most common deployment patterns, but certctl supports a broader set of issuer and target backends — see `docs/features.md`'s Issuer Connectors and Target Connectors sections for the live catalogs (rebuild via `ls -d internal/connector/issuer/*/ | wc -l` and `ls -d internal/connector/target/*/ | wc -l`). Once you have the basics running, you can mix and match:
+
+**Issuers:** ACME (Let's Encrypt, ZeroSSL, Buypass, Google Trust Services), Local CA (self-signed or sub-CA), step-ca, Vault PKI, DigiCert CertCentral, OpenSSL/Custom CA script, Sectigo (coming soon).
+
+**Targets:** NGINX, Apache, HAProxy, Traefik, Caddy, Envoy, IIS (local PowerShell or WinRM proxy), Postfix, Dovecot, F5 BIG-IP (coming soon).
+
+See [Connector Reference](connectors.md) for configuration details on every issuer and target.

docs/legacy-est-scep.md

@@ -0,0 +1,209 @@
+# Legacy EST / SCEP Clients — TLS 1.2 Reverse-Proxy Runbook
+
+**Audit reference:** Bundle F / M-023. PCI-DSS v4.0 Req 4 §2.2.5; CWE-326.
+
+certctl's control plane pins `tls.Config.MinVersion = tls.VersionTLS13`
+(`cmd/server/tls.go:131`). Some embedded EST (RFC 7030) and SCEP (RFC 8894)
+clients only speak TLS 1.0/1.1/1.2 — those clients cannot complete the
+handshake against certctl directly. This runbook documents the supported
+operator pattern: terminate the legacy TLS version at a front-door reverse
+proxy and pass the request through to certctl over TLS 1.3.
+
+## Why TLS 1.3 minimum
+
+certctl's audit posture, the SOC 2 / PCI-DSS / NIST SP 800-57 compliance
+mappings, and the M-001 PBKDF2 work factor all assume modern transport
+crypto. TLS 1.2 with the cipher suites still in the wild has known
+attack surface (BEAST, POODLE, ROBOT, raccoon — all CVE-categorized);
+allowing TLS 1.2 directly on the certctl listener would invalidate the
+guarantee that the server-side encryption chain is the strongest the
+ecosystem currently supports.
+
+## When this runbook applies
+
+You need this if **all three** are true:
+
+1. You operate certctl with EST or SCEP enabled (`CERTCTL_EST_ENABLED=true`
+   or `CERTCTL_SCEP_ENABLED=true`).
+2. Your enrolling clients are embedded devices (printers, network
+   appliances, IoT boards, legacy MFPs, point-of-sale terminals) whose TLS
+   stack pre-dates 2018 and only speaks TLS 1.2 or older.
+3. Replacing those clients is not feasible on a 6-month horizon.
+
+If your enrolling clients are modern (any current Linux/Windows/macOS
+host, anything Go-based, anything Rust/Python/Node from 2019 onward),
+they speak TLS 1.3 natively and this runbook is unnecessary — point them
+straight at certctl on `:8443`.
+
+## Architecture
+
+```
+                          ┌─── TLS 1.2/1.3 ────┐         ┌─── TLS 1.3 ───┐
+[legacy EST/SCEP client]──>│ nginx / HAProxy   │────────>│ certctl :8443 │
+                          │ reverse proxy      │         │               │
+                          └────────────────────┘         └───────────────┘
+        Allowed TLS 1.2                  Re-encrypts as TLS 1.3
+```
+
+The reverse proxy:
+
+- Terminates the legacy-version TLS handshake on the public-facing port.
+- Forwards the request to certctl over TLS 1.3 on a private network.
+- (For EST mTLS) forwards the client certificate via an
+  `X-SSL-Client-Cert` header that certctl reads only when the connection
+  arrives from a configured-trusted source IP.
+
+## nginx config
+
+```nginx
+upstream certctl_backend {
+    # Private-network address; not reachable from outside the proxy host.
+    server 10.0.0.10:8443;
+}
+
+server {
+    listen 443 ssl http2;
+    server_name est.example.com;
+
+    # Public-facing legacy listener. ssl_protocols includes TLSv1.2 explicitly.
+    # Keep ssl_ciphers conservative — only the strong AEAD suites that
+    # PCI-DSS Req 4 §2.2.5 still allows under TLS 1.2.
+    ssl_certificate     /etc/nginx/certs/est.example.com.fullchain.pem;
+    ssl_certificate_key /etc/nginx/certs/est.example.com.key;
+    ssl_protocols       TLSv1.2 TLSv1.3;
+    ssl_ciphers         ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;
+    ssl_prefer_server_ciphers on;
+
+    # mTLS for EST: optional client cert, verified against the EST CA.
+    ssl_client_certificate /etc/nginx/certs/est-clients-ca.pem;
+    ssl_verify_client      optional;
+
+    location ~ ^/\.well-known/(est|pki) {
+        # Forward the client cert (if presented) to certctl over the
+        # private hop. The current certctl implementation IGNORES the
+        # X-SSL-Client-Cert header (header-agnostic by default — see
+        # the certctl-side configuration section below). EST/SCEP
+        # authentication still works correctly because both protocols
+        # carry their own auth (CSR signature for EST, challengePassword
+        # for SCEP) inside the request body.
+        proxy_set_header X-SSL-Client-Cert  $ssl_client_escaped_cert;
+        proxy_set_header X-Forwarded-For    $remote_addr;
+        proxy_set_header X-Forwarded-Proto  $scheme;
+
+        # The proxy-to-certctl hop is itself TLS 1.3.
+        proxy_pass https://certctl_backend;
+        proxy_ssl_protocols TLSv1.3;
+        proxy_ssl_verify    on;
+        proxy_ssl_trusted_certificate /etc/nginx/certs/certctl-internal-ca.pem;
+    }
+
+    # SCEP endpoints — same pattern, no client-cert requirement
+    # (SCEP authenticates via challengePassword inside the CSR).
+    location ^~ /scep {
+        proxy_set_header X-Forwarded-For    $remote_addr;
+        proxy_set_header X-Forwarded-Proto  $scheme;
+        proxy_pass https://certctl_backend;
+        proxy_ssl_protocols TLSv1.3;
+        proxy_ssl_verify    on;
+        proxy_ssl_trusted_certificate /etc/nginx/certs/certctl-internal-ca.pem;
+    }
+}
+```
+
+## HAProxy config (alternative)
+
+```
+frontend est_legacy
+    bind *:443 ssl crt /etc/haproxy/certs/est.example.com.pem alpn h2,http/1.1 \
+        ssl-min-ver TLSv1.2 \
+        ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384
+
+    acl is_est_path  path_beg /.well-known/est
+    acl is_pki_path  path_beg /.well-known/pki
+    acl is_scep_path path_beg /scep
+    use_backend certctl_backend if is_est_path or is_pki_path or is_scep_path
+    default_backend certctl_modern
+
+backend certctl_backend
+    server certctl 10.0.0.10:8443 ssl verify required \
+        ca-file /etc/haproxy/certs/certctl-internal-ca.pem \
+        ssl-min-ver TLSv1.3
+    http-request set-header X-Forwarded-For %[src]
+    http-request set-header X-Forwarded-Proto https
+```
+
+## certctl-side configuration
+
+The current implementation is **header-agnostic**: certctl ignores any
+`X-SSL-Client-Cert` / `X-Forwarded-For` headers from the proxy. EST
+authentication still happens via in-protocol CSR signature + profile
+policy (RFC 7030 §3.2.3); SCEP authentication still happens via the
+`challengePassword` attribute embedded in the CSR (RFC 8894 §3.2). Both
+mechanisms are inside the request body and survive the reverse-proxy
+hop without server-side header trust.
+
+**Why this is the correct default:** trusting a proxy-supplied header
+for client identity opens a header-spoofing attack surface that requires
+careful design (CIDR allowlist of trusted proxies, fail-closed defaults,
+explicit operator opt-in). The Bundle F closure of M-023 ships the
+TLS-bridge guidance as documentation only; a future commit can extend
+certctl with proxy-header trust if and when an operator demonstrates a
+deployment shape that requires it. Until that lands, the runbook above
+is operationally complete: legacy EST and SCEP clients continue to
+authenticate via their in-protocol mechanisms, and the reverse proxy is
+purely a TLS-version bridge.
+
+If your deployment requires proxy-supplied client identity (e.g., the
+proxy terminates mTLS and you want certctl to record the client-cert
+subject in the audit trail beyond what the CSR carries), open an issue
+and a future commit will add a header-trust contract behind two
+fail-closed env vars: a CIDR allowlist of trusted proxies, plus an
+explicit opt-in toggle. Both knobs would be required together; setting
+only one would fail loud at startup. Until that work ships, the
+header-agnostic default described above is the only supported
+configuration.
+
+## PCI-DSS Req 4 §2.2.5 attestation
+
+PCI-DSS v4.0 §2.2.5 ("strong cryptography for authentication/transmission
+of cardholder data") considers TLS 1.2 with strong cipher suites
+acceptable for the foreseeable future, with the explicit caveat that NIST
+or the PCI Council may shorten the deprecation window if a TLS 1.2
+weakness is published. The configuration above:
+
+- Pins TLS 1.2 + TLS 1.3 only (no SSLv3, TLS 1.0, TLS 1.1).
+- Uses only AEAD cipher suites with forward secrecy (ECDHE-* with GCM or
+  ChaCha20-Poly1305).
+- Re-encrypts to TLS 1.3 on the proxy-to-certctl hop.
+
+This is PCI-DSS Req 4 v4.0 compliant. Auditors looking for the
+attestation should be pointed at this section + the proxy's TLS config.
+
+## What this runbook does NOT cover
+
+- **Replacing the legacy clients.** That's the long-term fix; this
+  runbook is the bridge while you're migrating.
+- **Network segmentation.** The reverse proxy assumes the proxy-to-certctl
+  hop is on a network that an external attacker can't reach. If it's
+  not, you need a deeper architecture review.
+- **Client-cert revocation.** EST mTLS revocation is the relying party's
+  responsibility. certctl's EST handler accepts the cert; the proxy can
+  enforce CRL/OCSP via `ssl_crl_path` (nginx) or `crl-file` (HAProxy).
+
+## When TLS 1.2 itself sunsets
+
+PCI-DSS, NIST, and major browsers will eventually deprecate TLS 1.2.
+When that happens, this runbook becomes obsolete; the only path forward
+will be to replace the legacy clients. Subscribe to RSS feeds at the
+following sources to catch the deprecation announcement before it
+becomes a compliance failure:
+
+- https://www.pcisecuritystandards.org/news_events/
+- https://nvlpubs.nist.gov/nistpubs/SpecialPublications/  (SP 800-52 revisions)
+
+## Related docs
+
+- [`tls.md`](tls.md) — the certctl-internal TLS configuration (HTTPS-only
+  control plane, MinVersion pin)
+- [`security.md`](security.md) — overall security posture
+- [`database-tls.md`](database-tls.md) — Postgres TLS opt-in (Bundle B / M-018)

docs/mcp.md

@@ -29,15 +29,18 @@ The binary has zero runtime dependencies beyond the certctl server it connects t
 
 ## Configuration
 
-The MCP server reads two environment variables:
+The MCP server reads three environment variables:
 
 | Variable | Required | Default | Description |
 |----------|----------|---------|-------------|
-| `CERTCTL_SERVER_URL` | No | `http://localhost:8443` | URL of the certctl REST API |
+| `CERTCTL_SERVER_URL` | No | `https://localhost:8443` | URL of the certctl REST API (HTTPS-only as of v2.2) |
 | `CERTCTL_API_KEY` | No | (empty) | API key for authentication (passed as `Bearer` token) |
+| `CERTCTL_SERVER_CA_BUNDLE_PATH` | Yes (for self-signed / internal CA) | (empty) | Path to PEM CA bundle that signed the server cert. Required when the server cert isn't rooted in the system trust store (the default compose stack ships a self-signed cert at `deploy/test/certs/ca.crt`). |
 
 If your certctl server has auth enabled (the default), you must provide the API key. The MCP server passes it through to every HTTP request.
 
+Since v2.2 the certctl control plane is HTTPS-only. If the server cert is self-signed or chained to an internal CA, set `CERTCTL_SERVER_CA_BUNDLE_PATH` so the MCP server can verify the TLS handshake. Never set `CERTCTL_SERVER_TLS_INSECURE_SKIP_VERIFY=true` outside local development — it disables all certificate validation.
+
 ## Setting Up with Claude Desktop
 
 Add this to your Claude Desktop MCP configuration file (`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS, `%APPDATA%\Claude\claude_desktop_config.json` on Windows):
@@ -48,7 +51,8 @@ Add this to your Claude Desktop MCP configuration file (`~/Library/Application S
     "certctl": {
       "command": "/path/to/certctl-mcp",
       "env": {
-        "CERTCTL_SERVER_URL": "http://localhost:8443",
+        "CERTCTL_SERVER_URL": "https://localhost:8443",
+        "CERTCTL_SERVER_CA_BUNDLE_PATH": "/path/to/certctl/deploy/test/certs/ca.crt",
         "CERTCTL_API_KEY": "your-api-key-here"
       }
     }
@@ -67,7 +71,8 @@ In Cursor, go to Settings → MCP Servers and add:
   "certctl": {
     "command": "/path/to/certctl-mcp",
     "env": {
-      "CERTCTL_SERVER_URL": "http://localhost:8443",
+      "CERTCTL_SERVER_URL": "https://localhost:8443",
+      "CERTCTL_SERVER_CA_BUNDLE_PATH": "/path/to/certctl/deploy/test/certs/ca.crt",
       "CERTCTL_API_KEY": "your-api-key-here"
     }
   }
@@ -84,7 +89,8 @@ Add certctl as an MCP server in your project's `.mcp.json`:
     "certctl": {
       "command": "/path/to/certctl-mcp",
       "env": {
-        "CERTCTL_SERVER_URL": "http://localhost:8443",
+        "CERTCTL_SERVER_URL": "https://localhost:8443",
+        "CERTCTL_SERVER_CA_BUNDLE_PATH": "/path/to/certctl/deploy/test/certs/ca.crt",
         "CERTCTL_API_KEY": "your-api-key-here"
       }
     }
@@ -94,7 +100,7 @@ Add certctl as an MCP server in your project's `.mcp.json`:
 
 ## Available Tools
 
-The MCP server registers 78 tools organized across 16 resource domains:
+The MCP server exposes the full REST API organized across 16 resource domains:
 
 | Domain | Tools | Examples |
 |--------|-------|---------|
@@ -153,7 +159,7 @@ flowchart LR
     AI <-->|"stdio"| MCP
     MCP -->|"HTTP + Bearer token"| SERVER
 
-    MCP ~~~ TOOLS["78 tools · 16 domains\nTyped input structs"]
+    MCP ~~~ TOOLS["REST API via MCP · 16 domains\nTyped input structs"]
 ```
 
 The MCP server is intentionally thin:

docs/migrate-from-acmesh.md

@@ -0,0 +1,275 @@
+# Migrate from acme.sh to certctl
+
+You use acme.sh to automate Let's Encrypt renewal across multiple servers. It works — but without centralized visibility, deployment verification, or policy enforcement.
+
+This guide walks through moving your acme.sh workload to certctl while keeping your existing DNS provider setup.
+
+## Why Migrate
+
+**acme.sh strength:** Lightweight agent, works everywhere, integrates with any DNS provider via shell script hooks.
+
+**acme.sh limitations:**
+- No inventory visibility — certificates scattered across servers, no unified view of expiry dates or renewal status
+- No deployment verification — cron job succeeds even if cert doesn't actually take effect on the service
+- No policy enforcement — no way to require approval, audit who renewed what, or prevent misconfigurations
+- No multi-server orchestration — each server manages its own renewals; no way to batch test or rollback
+
+certctl adds a control plane that sees all your certificates, deploys with verification, enforces policy, and provides a complete audit trail. You keep the DNS-01 challenge scripts you already have.
+
+## What You Keep
+
+- **Existing certificates** — discovered automatically during migration, claimed in the dashboard
+- **DNS provider scripts** — acme.sh's `dns_*` hooks are shell-script compatible with certctl's DNS-01 implementation
+- **Same Let's Encrypt account** — ACME issuer in certctl uses the same account and email
+
+## Migration Steps
+
+### 1. Deploy certctl Server
+
+Start with Docker Compose (5 minutes):
+
+```bash
+git clone https://github.com/shankar0123/certctl.git
+cd certctl/deploy
+docker compose up -d
+```
+
+Access the dashboard at `https://localhost:8443` with the API key from `.env`. The default compose stack ships a self-signed cert; pin with `--cacert ./deploy/test/certs/ca.crt` when calling the API from the host.
+
+### 2. Deploy Agents
+
+On each server running acme.sh certs, install the certctl agent:
+
+```bash
+curl -sSL https://raw.githubusercontent.com/shankar0123/certctl/master/install-agent.sh | bash
+# Prompted for server URL and API key
+```
+
+Or manually:
+
+```bash
+# Download and install agent binary
+wget https://github.com/shankar0123/certctl/releases/download/v2.1.0/certctl-agent-linux-amd64
+chmod +x certctl-agent-linux-amd64
+sudo mv certctl-agent-linux-amd64 /usr/local/bin/certctl-agent
+
+# Create systemd unit
+sudo tee /etc/systemd/system/certctl-agent.service > /dev/null <<EOF
+[Unit]
+Description=certctl Agent
+After=network-online.target
+
+[Service]
+Type=simple
+ExecStart=/usr/local/bin/certctl-agent
+Environment="CERTCTL_SERVER_URL=https://certctl.internal:8443"
+Environment="CERTCTL_API_KEY=your-api-key-here"
+Environment="CERTCTL_DISCOVERY_DIRS=~/.acme.sh"
+Restart=always
+RestartSec=10s
+
+[Install]
+WantedBy=multi-user.target
+EOF
+
+sudo systemctl daemon-reload
+sudo systemctl enable --now certctl-agent
+```
+
+### 3. Discover Existing acme.sh Certificates
+
+acme.sh stores certificates in `~/.acme.sh/<domain>/` (or `/etc/acme.sh/` if installed system-wide).
+
+When you start the agent with `CERTCTL_DISCOVERY_DIRS` pointing to those directories, it scans for existing PEM/DER certificates and reports fingerprints to the control plane. The dashboard's **Discovery** page shows what was found.
+
+Example agent systemd service (using home directory):
+
+```bash
+Environment="CERTCTL_DISCOVERY_DIRS=/home/user/.acme.sh"
+```
+
+Or for system-wide acme.sh:
+
+```bash
+Environment="CERTCTL_DISCOVERY_DIRS=/etc/acme.sh"
+```
+
+### 4. Claim Discovered Certificates
+
+In the **Discovery** page:
+1. Review the "Unmanaged" certificates found by the agent
+2. Click **Claim** on each acme.sh certificate
+3. Enter the managed certificate ID to link it (e.g., `mc-api-prod`)
+
+Once claimed, the certificate appears in the main **Certificates** page with ownership, renewal history, and deployment status.
+
+### 5. Create an ACME Issuer
+
+In **Issuers** → **+ New Issuer:**
+
+1. Select **ACME** from the issuer type grid
+2. Fill in the type-specific fields: name, directory URL (`https://acme-v02.api.letsencrypt.org/directory`), and config
+
+Or configure via environment variables:
+```bash
+export CERTCTL_ACME_DIRECTORY_URL=https://acme-v02.api.letsencrypt.org/directory
+export CERTCTL_ACME_EMAIL=your-email@example.com  # same as your acme.sh account
+export CERTCTL_ACME_CHALLENGE_TYPE=dns-01
+```
+
+### 6. Adapt Your DNS Provider Scripts
+
+acme.sh uses `dns_*` hooks (e.g., `dns_cloudflare`) with predictable argument patterns. certctl's DNS-01 uses the same pattern, so your scripts often work with zero changes.
+
+**acme.sh pattern:**
+```bash
+# acme.sh invokes: dns_cloudflare_add "domain" "record" "value"
+dns_cloudflare_add() {
+  local full_domain=$1
+  local record_name=$2
+  local record_value=$3
+  # ... DNS API call to create TXT record ...
+}
+```
+
+**certctl pattern:**
+```bash
+# certctl invokes: /path/to/dns-present-script
+# Scripts receive environment variables:
+#!/bin/bash
+# CERTCTL_DNS_DOMAIN — domain name (e.g., "example.com")
+# CERTCTL_DNS_FQDN — full record name (e.g., "_acme-challenge.example.com")
+# CERTCTL_DNS_VALUE — TXT record value (key authorization digest)
+# CERTCTL_DNS_TOKEN — ACME challenge token
+# Create TXT record at "${CERTCTL_DNS_FQDN}" with value "${CERTCTL_DNS_VALUE}"
+```
+
+**Example: Cloudflare DNS-01 adapter**
+
+If you have an acme.sh Cloudflare hook, adapt it:
+
+```bash
+#!/bin/bash
+# /etc/certctl/dns/cloudflare-present.sh
+set -e
+
+# certctl passes these environment variables:
+# CERTCTL_DNS_DOMAIN — domain name
+# CERTCTL_DNS_FQDN — full record name (e.g., "_acme-challenge.example.com")
+# CERTCTL_DNS_VALUE — TXT record value
+# CERTCTL_DNS_TOKEN — ACME challenge token
+
+# Call your existing Cloudflare API (example using curl)
+curl -X POST "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records" \
+  -H "X-Auth-Email: ${CF_EMAIL}" \
+  -H "X-Auth-Key: ${CF_KEY}" \
+  -H "Content-Type: application/json" \
+  -d "{\"type\":\"TXT\",\"name\":\"${CERTCTL_DNS_FQDN}\",\"content\":\"${CERTCTL_DNS_VALUE}\"}"
+
+echo "Created ${CERTCTL_DNS_FQDN}"
+```
+
+DNS cleanup:
+
+```bash
+#!/bin/bash
+# /etc/certctl/dns/cloudflare-cleanup.sh
+
+# certctl passes these environment variables:
+# CERTCTL_DNS_DOMAIN — domain name
+# CERTCTL_DNS_FQDN — full record name (e.g., "_acme-challenge.example.com")
+# CERTCTL_DNS_VALUE — TXT record value
+# CERTCTL_DNS_TOKEN — ACME challenge token
+
+# Query and delete the TXT record
+curl -X DELETE "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records/${RECORD_ID}" \
+  -H "X-Auth-Email: ${CF_EMAIL}" \
+  -H "X-Auth-Key: ${CF_KEY}"
+```
+
+Configure the ACME issuer via environment variables:
+
+```bash
+export CERTCTL_ACME_DIRECTORY_URL=https://acme-v02.api.letsencrypt.org/directory
+export CERTCTL_ACME_EMAIL=your-email@example.com
+export CERTCTL_ACME_CHALLENGE_TYPE=dns-01
+export CERTCTL_ACME_DNS_PRESENT_SCRIPT=/etc/certctl/dns/cloudflare-present.sh
+export CERTCTL_ACME_DNS_CLEANUP_SCRIPT=/etc/certctl/dns/cloudflare-cleanup.sh
+```
+
+Or create the issuer through the dashboard: **Issuers** → **+ New Issuer** → select **ACME** → fill in the config fields.
+
+### 7. Create Renewal Policies
+
+In **Policies** → **+ New Policy:**
+
+- **Name:** e.g., "ACME DNS-01 Policy"
+- **Type:** `expiration_window` (enforces renewal thresholds)
+- **Severity:** `high`
+- **Config:** set your renewal window (default: 30 days before expiry)
+
+Renewal scheduling is driven by the certificate's assigned profile and issuer. Policies add enforcement guardrails on top.
+
+### 8. Phase Out acme.sh Cron
+
+Once you verify renewals work via certctl (manually trigger one in the dashboard first), remove the acme.sh cron job:
+
+```bash
+# Remove acme.sh from crontab
+crontab -e
+# Delete the line: "0 0 * * * /home/user/.acme.sh/acme.sh --cron --home /home/user/.acme.sh"
+
+# OR disable the cron service if installed
+sudo systemctl disable acme-renew.timer
+```
+
+## DNS Script Compatibility
+
+Most acme.sh DNS provider hooks need only minor changes:
+
+| acme.sh | certctl |
+|---------|---------|
+| Called on every renewal | Called once per challenge window |
+| Receives: domain, record name, record value as arguments | Receives: `CERTCTL_DNS_DOMAIN`, `CERTCTL_DNS_FQDN`, `CERTCTL_DNS_VALUE`, `CERTCTL_DNS_TOKEN` as environment variables |
+| Must support multiple concurrent records | Same — cleanup removes the specific token |
+| Environment variables for credentials | Same — pass via agent systemd `Environment=` or `.env` file |
+
+**Real example:** If you use Route53, acme.sh's `dns_aws` hook submits via AWS CLI. Adapt it to use `${CERTCTL_DNS_FQDN}` and `${CERTCTL_DNS_VALUE}` environment variables instead of positional arguments, and it works with certctl's DNS-01.
+
+## Coexistence Period
+
+During migration, run both acme.sh and certctl in parallel:
+
+1. Keep acme.sh cron running (low overhead, serves as fallback)
+2. Configure certctl policies and test renewal on 1-2 non-critical domains
+3. Monitor certctl's audit trail and deployment logs
+4. Once confident, disable acme.sh cron on those domains
+5. Roll out to remaining domains
+
+This way, if certctl renewal fails, acme.sh's cron still renews the cert (you'll see duplicate renewals in the audit trail, but no gap).
+
+## Next: DNS-PERSIST-01 (Zero-Touch Renewals)
+
+After migrating to certctl + DNS-01, consider upgrading to **DNS-PERSIST-01**. Instead of creating/deleting DNS records on every renewal, you create one persistent TXT record at `_validation-persist.<domain>` that never changes. Let's Encrypt then validates against that standing record forever.
+
+Benefits:
+- **Zero operational overhead per renewal** — no DNS API calls during renewal
+- **Auditable** — DNS record created once, visible to the team, never modified
+- **Vendor-agnostic** — works with any DNS provider that supports TXT records
+
+To enable:
+
+```bash
+export CERTCTL_ACME_CHALLENGE_TYPE=dns-persist-01
+export CERTCTL_ACME_DNS_PERSIST_ISSUER_DOMAIN=letsencrypt.org
+export CERTCTL_ACME_DNS_PRESENT_SCRIPT=/etc/certctl/dns/cloudflare-present.sh
+```
+
+certctl automatically falls back to DNS-01 if the CA doesn't support dns-persist-01 yet.
+
+## Next Steps
+
+- Try the [Wildcard DNS-01 example](../examples/acme-wildcard-dns01/acme-wildcard-dns01.md) — a working docker-compose with Cloudflare hooks you can adapt for your DNS provider
+- See [Connector Reference](connectors.md) for advanced ACME options (EAB, ARI, custom timeouts)
+- See [Discovery Guide](concepts.md#certificate-discovery) for managing discovered certificates at scale
+- See all [Deployment Examples](./examples.md) for other scenarios (ACME+NGINX, private CA, step-ca, multi-issuer)

docs/migrate-from-certbot.md

@@ -0,0 +1,173 @@
+# Migrating from Certbot to certctl
+
+You have 50 Let's Encrypt certificates across 10 servers, managed by a mix of Certbot cron jobs and manual renewals. Certbot handles issuance, but you lack inventory visibility, centralized alerting, and audit trails. This guide walks you through moving to certctl while keeping your existing certificates and ACME account.
+
+## Why Migrate
+
+Certbot renews certs in isolation. If a renewal fails on one server, you don't know until the cert expires. certctl gives you a single pane of glass: see all certs across all servers, get alerts 30/14/7 days before expiry, track who renewed what when, and verify each deployment succeeded via TLS fingerprint validation.
+
+## What You Keep
+
+- Your existing Certbot ACME account key and Let's Encrypt account
+- All issued certificates in `/etc/letsencrypt/live/`
+- Certbot's renewal history and hooks
+
+You will not re-issue any certificates. certctl discovers them and takes over renewal scheduling.
+
+## Step-by-Step Migration
+
+### 1. Deploy certctl Control Plane
+
+Option A: Docker Compose (quickest for evaluation)
+```bash
+cd /opt/certctl
+docker compose up -d
+# Dashboard & API: https://localhost:8443 (self-signed cert — use --cacert ./deploy/test/certs/ca.crt for the default compose stack)
+# Default API key in logs (grep CERTCTL_API_KEY docker logs certctl-server)
+```
+
+Option B: Kubernetes (Helm)
+```bash
+helm install certctl deploy/helm/certctl/ \
+  --set auth.apiKey=YOUR_SECURE_KEY
+```
+
+### 2. Deploy Agents to Each Server
+
+On each of your 10 servers running Certbot:
+
+```bash
+# Linux amd64 (adjust for your architecture)
+curl -sSL https://github.com/shankar0123/certctl/releases/download/v2.1.0/certctl-agent-linux-amd64 \
+  -o /usr/local/bin/certctl-agent
+chmod +x /usr/local/bin/certctl-agent
+
+# Create config
+sudo mkdir -p /etc/certctl /var/lib/certctl/keys
+sudo tee /etc/certctl/agent.env > /dev/null <<EOF
+CERTCTL_SERVER_URL=https://certctl-control-plane.example.com:8443
+CERTCTL_SERVER_CA_BUNDLE_PATH=/etc/certctl/tls/ca.crt
+CERTCTL_API_KEY=your-api-key-here
+CERTCTL_DISCOVERY_DIRS=/etc/letsencrypt/live
+CERTCTL_KEY_DIR=/var/lib/certctl/keys
+EOF
+sudo chmod 600 /etc/certctl/agent.env
+
+# Start agent
+sudo systemctl start certctl-agent  # if installed via script
+# OR manually:
+sudo certctl-agent --server https://... --api-key ... --discovery-dirs /etc/letsencrypt/live
+```
+
+The agent will scan `/etc/letsencrypt/live/` and report all discovered certificates to the control plane.
+
+### 3. Triage Discovered Certificates
+
+In the certctl dashboard, go to **Discovery**:
+- See all discovered certs grouped by agent
+- Status shows "Unmanaged" for certificates not yet claimed
+- For each Certbot cert, click **Claim** and link it to managed inventory
+
+The control plane now knows about all 50 certs and where they live.
+
+### 4. Configure ACME Issuer
+
+Go to **Issuers** → **+ New Issuer**:
+1. Select **ACME** from the issuer type grid
+2. Fill in the type-specific fields: name, directory URL (`https://acme-v02.api.letsencrypt.org/directory`), and any required config
+
+Alternatively, configure via environment variables before starting the server:
+```bash
+export CERTCTL_ACME_DIRECTORY_URL=https://acme-v02.api.letsencrypt.org/directory
+export CERTCTL_ACME_EMAIL=your-email@example.com
+export CERTCTL_ACME_CHALLENGE_TYPE=http-01  # or dns-01 for wildcard certs
+```
+
+For DNS-01, also set:
+```bash
+export CERTCTL_ACME_DNS_PRESENT_SCRIPT=/etc/certctl/dns/present.sh
+export CERTCTL_ACME_DNS_CLEANUP_SCRIPT=/etc/certctl/dns/cleanup.sh
+```
+
+certctl uses the same Let's Encrypt account; no new credentials needed.
+
+### 5. Create Renewal Policies
+
+Go to **Policies** → **+ New Policy** to create enforcement rules:
+- Name: e.g., "ACME Renewal Policy"
+- Type: `expiration_window` (to enforce renewal thresholds)
+- Severity: `high`
+- Config: set your renewal threshold (default: 30 days before expiry)
+
+Renewal scheduling is driven by the certificate's assigned profile and issuer. Policies add enforcement guardrails (key algorithm requirements, expiration windows, etc.).
+
+### 6. Disable Certbot Cron, One Server at a Time
+
+On the first server (start with a low-traffic one):
+
+```bash
+# Stop Certbot renewal
+sudo systemctl disable certbot.timer
+sudo systemctl stop certbot.timer
+
+# Or remove the cron job
+sudo rm /etc/cron.d/certbot  # if managed by cron
+```
+
+Monitor that server in the certctl dashboard. Certctl will renew the cert ~30 days before expiry.
+
+### 7. Verify First Renewal Succeeds
+
+Wait for the renewal to trigger (or manually trigger it in **Certificates** → select cert → **Renew**). Check the dashboard:
+- **Certificates** page: status transitions from `Active` to `Renewing` to `Active`
+- **Jobs** page: renewal job shows `Completed` status
+- **Verification** tab: TLS check confirms the new cert is deployed and live
+
+After verifying, disable Certbot on the remaining 9 servers.
+
+### 8. Enable Alerting
+
+Configure notifiers via environment variables before starting the server:
+```bash
+# Example: Slack alerting
+export CERTCTL_SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL
+docker compose up -d
+
+# Or email alerting
+export CERTCTL_SMTP_HOST=smtp.gmail.com
+export CERTCTL_SMTP_PORT=587
+export CERTCTL_SMTP_USERNAME=your-email@gmail.com
+export CERTCTL_SMTP_PASSWORD=your-app-password
+export CERTCTL_SMTP_FROM_ADDRESS=certctl@example.com
+docker compose up -d
+
+# Other options: CERTCTL_TEAMS_WEBHOOK_URL, CERTCTL_PAGERDUTY_ROUTING_KEY, CERTCTL_OPSGENIE_API_KEY
+```
+
+Now you get 30/14/7-day warnings before any cert expires, across all 10 servers, in one place.
+
+## What Changes
+
+- **Renewal**: Agent polls certctl for work instead of Certbot cron triggering locally. Faster failure detection (agent heartbeat every 60 seconds vs. cron running once a day).
+- **Deployment**: certctl verifies post-deployment by probing the live TLS endpoint and comparing SHA-256 fingerprints. Catches reload failures silently.
+- **Audit Trail**: Every renewal, deployment, and alert is logged immutably. Answer "who renewed cert X when and why" within seconds.
+- **Alerting**: Threshold-based alerts to Slack/email/webhook 30/14/7 days before expiry, not when cert expires.
+
+## Coexistence and Rollback
+
+During migration, certctl and Certbot can run simultaneously. The agent will discover Certbot certs even while Certbot continues renewing them. Run both for a week to build confidence.
+
+**If you need to rollback**: Re-enable Certbot cron on any server:
+```bash
+sudo systemctl enable certbot.timer
+sudo systemctl start certbot.timer
+```
+
+certctl will stop renewing that cert when the policy is disabled. Certbot resumes as before. Your certificates and ACME account remain untouched.
+
+## Next Steps
+
+- Try the [ACME + NGINX example](../examples/acme-nginx/acme-nginx.md) — a working docker-compose you can run locally before deploying to production
+- Review the [Concepts Guide](./concepts.md) for terminology (profiles, policies, agents, jobs)
+- Explore [Network Discovery](./quickstart.md#network-discovery-agentless) to find certificates you didn't know about
+- See all [Deployment Examples](./examples.md) for other scenarios (wildcard DNS-01, private CA, step-ca, multi-issuer)

docs/openapi.md

@@ -68,8 +68,10 @@ The spec organizes endpoints into 16 tags:
 The spec declares a `bearerAuth` security scheme applied globally. All endpoints under `/api/v1/` require a Bearer token by default:
 
 ```bash
-curl -H "Authorization: Bearer your-api-key" \
-  http://localhost:8443/api/v1/certificates
+# The default compose stack uses a self-signed cert; pin with --cacert
+curl --cacert ./deploy/test/certs/ca.crt \
+  -H "Authorization: Bearer your-api-key" \
+  https://localhost:8443/api/v1/certificates
 ```
 
 Three endpoints are exempt from auth (declared with `security: []` in the spec): `/health`, `/ready`, and `/api/v1/auth/info`. The auth info endpoint tells clients whether authentication is enabled and what type is required — useful for GUIs that need to show/hide a login screen.
@@ -150,8 +152,9 @@ Import the spec directly into Postman:
 
 1. Open Postman → Import → File → select `api/openapi.yaml`
 2. Postman creates a collection with all 78 documented operations organized by tag
-3. Set the `baseUrl` variable to `http://localhost:8443`
+3. Set the `baseUrl` variable to `https://localhost:8443` (HTTPS-only as of v2.2)
 4. Add an `Authorization: Bearer your-api-key` header to the collection
+5. Import the demo stack CA bundle (`deploy/test/certs/ca.crt`) into Postman's Settings → Certificates → CA Certificates, or disable certificate verification for the `localhost` host (Settings → General → SSL certificate verification)
 
 ## Key Schemas
 
@@ -176,8 +179,10 @@ Use the spec to generate contract tests that verify the API matches the spec:
 ```bash
 # Using schemathesis for fuzz testing against the spec
 pip install schemathesis
+# The default compose stack uses a self-signed cert — export a CA bundle or set REQUESTS_CA_BUNDLE
+export REQUESTS_CA_BUNDLE=$(pwd)/deploy/test/certs/ca.crt
 schemathesis run api/openapi.yaml \
-  --base-url http://localhost:8443 \
+  --base-url https://localhost:8443 \
   --header "Authorization: Bearer your-api-key"
 ```
 

docs/qa-test-guide.md

@@ -0,0 +1,451 @@
+# QA Test Suite Guide (`qa_test.go`)
+
+> **Audience:** Anyone running release QA for certctl — whether you're a first-time contributor or the maintainer cutting a release tag.
+>
+> **Companion to:** `docs/testing-guide.md` (the *what* to test). This document explains the *how* — the automated test file, what it covers, what it skips, and how to fill the gaps manually.
+
+---
+
+## Test Suite Health (regenerate via `make qa-stats`)
+
+> Snapshot at HEAD. Re-run `make qa-stats` to refresh; CI's QA-doc drift guards (`.github/workflows/ci.yml`) catch out-of-date Part / cert / issuer counts on every PR. **Last regenerated: 2026-04-27 (Bundle P).**
+
+| Metric | Value | Target | Status |
+|---|---|---|---|
+| Backend test files | 221 | n/a | ℹ |
+| Backend `Test*` functions | 2,454 | n/a | ℹ |
+| Backend `t.Run` subtests | 778 | n/a | ℹ |
+| Frontend test files | 38 | n/a | ℹ |
+| Fuzz targets | 11 | ≥10 (one per hand-rolled parser) | ✓ |
+| `t.Skip` sites | 60 | each carries valid rationale (Bundle O audit) | ✓ |
+| `qa_test.go` Part_* subtests | 53 | tracks `testing-guide.md` Parts (3 `## Part 15-17` covered indirectly via Parts 42–46) | ✓ |
+| `testing-guide.md` Parts | 56 | n/a | ℹ |
+| Existential cluster line cov (post-Bundle-J + L.B + Bundle 0.7) | acme 55.6%, stepca 90.4%, local-issuer ≥86%, crypto ≥85% | ≥95% | △ ACME below; tracked in `coverage-matrix.md` |
+| Mutation kill rate (Existential) | unmeasured (operator-runnable per Strengthening #5) | ≥90% | ⚠ |
+| Race detector clean (`-count=10`) | partial (`-count=3` clean per Phase 0) | 0 races | ⚠ |
+
+## What Is This File?
+
+`deploy/test/qa_test.go` is a single Go test file (~1700 lines) that automates as much of `docs/testing-guide.md` as possible against a running certctl Docker Compose demo stack. It replaces the legacy `qa-smoke-test.sh` bash script.
+
+It covers **49 of 56 Parts** of the testing guide as automation; the remaining 7 are
+either manual-only by design or pending QA-suite coverage:
+
+- **49 `Part_*` automation wrappers**, **~159 leaf subtests** — API calls, database queries, source file checks, performance benchmarks
+- **11 fully skipped Parts** — with documented reasons (external CAs, Windows, browser-only, etc.) — see "What This Test Does NOT Cover" below
+- **4 Parts NOT YET AUTOMATED** — Parts 23 (S/MIME & EKU), 24 (OCSP/CRL), 55 (Agent Soft-Retirement), 56 (Notification Retry & Dead-Letter) — must be tested manually per `docs/testing-guide.md` until QA-suite automation lands
+- **Manual-only flows** in addition: GUI flows, scheduler timing, Docker log inspection — must be done by a human following `docs/testing-guide.md`
+
+## Architecture
+
+```
+┌────────────────────────┐     ┌─────────────────────────────────┐
+│  qa_test.go            │────▶│  certctl demo stack             │
+│  (//go:build qa)       │     │  docker-compose.yml +           │
+│                        │     │  docker-compose.demo.yml        │
+│  TestQA(t *testing.T)  │     │                                 │
+│   ├─ Part01_Infra      │     │  ┌─ certctl-server :8443        │
+│   ├─ Part02_Auth       │     │  ├─ postgres :5432              │
+│   ├─ Part03_CertCRUD   │     │  └─ certctl-agent (×N)          │
+│   ├─ ...               │     │      ↑ seed_demo.sql provisions │
+│   └─ Part52_HelmChart  │     │        12 agent rows (1 active, │
+└────────────────────────┘     │        2 retired, 9 reserved /  │
+                               │        sentinel) for the soft-  │
+                               │        retire / FSM coverage    │
+                               │        Parts 55–56 exercise.    │
+                               └─────────────────────────────────┘
+```
+
+> **Multi-agent demo stack (Bundle Q / L-004 closure).** The demo
+> stack runs a single live `certctl-agent` container by default but
+> the database is seeded with 12 agent rows (`migrations/seed_demo.sql`,
+> grep `mc-* | ag-*` IDs). The "(×N)" notation reflects the seed-data
+> reality: Parts 04 (Agents Listing), 05 (Agent Heartbeats), 55
+> (Agent Soft-Retirement), and FSM coverage tables in
+> `coverage-audit-2026-04-27/tables/fsm-coverage.md` exercise the full
+> multi-agent population, not the one live container. Operators
+> running the QA suite in a parallel-agent topology should set
+> `AGENT_COUNT=N` in compose-override and re-derive the seed counts
+> via `make qa-stats`.
+
+Key design choices:
+
+- **Build tag:** `//go:build qa` — never runs during `go test ./...` or CI. Only runs when explicitly requested.
+- **Package:** `integration_test` — same package as `integration_test.go` (which uses `//go:build integration` for the test stack). They coexist but never run together.
+- **Zero internal imports:** Uses only stdlib + `lib/pq` (from `go.mod`). All API interactions are plain HTTP. All JSON is decoded into lightweight local structs (`qaCert`, `qaJob`, etc.) — not the internal domain types.
+- **Self-cleaning:** Tests that create data use `t.Cleanup()` to delete it afterward. The seed data is not modified.
+
+## Prerequisites
+
+1. **Docker Compose demo stack running:**
+   ```bash
+   cd deploy
+   docker compose -f docker-compose.yml -f docker-compose.demo.yml up --build -d
+   ```
+   Wait ~15 seconds for health checks to pass.
+
+2. **Go 1.22+** installed (the project uses Go 1.25 in `go.mod`, but 1.22+ works for running tests).
+
+3. **PostgreSQL port exposed** — the demo stack exposes port 5432 for database verification tests (table counts, schema checks).
+
+4. **Repository checkout** — source file verification tests (`fileExists`, `fileContains`) read files relative to `qaRepoDir` (default: `../..` from `deploy/test/`).
+
+## Running the Tests
+
+### Full suite
+```bash
+cd deploy/test
+go test -tags qa -v -timeout 10m ./...
+```
+
+### Single Part
+```bash
+go test -tags qa -v -run TestQA/Part03 ./...
+```
+
+### Single subtest
+```bash
+go test -tags qa -v -run TestQA/Part03_CertCRUD/Create_Minimal ./...
+```
+
+### With custom environment
+```bash
+CERTCTL_QA_SERVER_URL=https://staging.internal:8443 \
+CERTCTL_QA_API_KEY=my-staging-key \
+CERTCTL_QA_DB_URL=postgres://certctl:secret@db.internal:5432/certctl?sslmode=require \
+CERTCTL_QA_REPO_DIR=/path/to/certctl \
+go test -tags qa -v -timeout 10m ./...
+```
+
+### Environment Variables
+
+| Variable | Default | Description |
+|---|---|---|
+| `CERTCTL_QA_SERVER_URL` | `https://localhost:8443` | certctl server URL (HTTPS-only as of v2.2) |
+| `CERTCTL_QA_API_KEY` | `change-me-in-production` | API key for Bearer auth |
+| `CERTCTL_QA_DB_URL` | `postgres://certctl:certctl@localhost:5432/certctl?sslmode=disable` | PostgreSQL connection string |
+| `CERTCTL_QA_REPO_DIR` | `../..` | Path to certctl repo root (for source file checks) |
+| `CERTCTL_QA_CA_BUNDLE` | `./certs/ca.crt` | PEM CA bundle pinned for TLS verification. The demo stack's `certctl-tls-init` container writes here. |
+| `CERTCTL_QA_INSECURE` | `false` | Set to `"true"` to skip TLS verification (e.g. before the init container finishes). Never use outside the demo harness. |
+
+## Part-by-Part Coverage Map
+
+This table shows what each Part tests and what's left for manual verification.
+
+| Part | Testing Guide Section | Automated Subtests | What's Automated | What's Manual |
+|------|----------------------|-------------------|-----------------|--------------|
+| 1 | Infrastructure & Deployment | 8 | Table count, health/ready endpoints, seed data counts (certs, agents, issuers, targets, policies) | Docker container health, log inspection, volume mounts |
+| 2 | Authentication & Security | 4 | No-auth 401, bad-key 401, health-no-auth 200, no private keys in API | CORS preflight, rate limiting (429 + Retry-After), TLS config |
+| 3 | Certificate Lifecycle | 10 | Create (minimal + full), get, 404, list pagination, status/issuer filters, sparse fields, update, archive | Deployment trigger, version history, certificate detail UI |
+| 4 | Renewal Workflow | 3 | Trigger renewal, 404 on nonexistent, agent work endpoint | AwaitingCSR flow, agent key generation, full issuance cycle |
+| 5 | Revocation | 5 | Revoke (default reason), already-revoked, nonexistent, invalid reason, CRL JSON | DER CRL, OCSP responder, revocation notifications |
+| 6 | Policies & Profiles | 6 | Policy CRUD (create/delete), invalid type 400, profile CRUD, list | Policy violation detection, profile enforcement on CSR |
+| 7 | Ownership & Teams | 4 | Team CRUD, owner CRUD, agent groups list | Owner notification routing, dynamic group matching |
+| 8 | Job System | 2 | List jobs, 404 on nonexistent | Job state transitions, approval workflow, cancellation |
+| 9 | Issuer Connectors | 4 | List, get detail, create (GenericCA), missing name 400 | Test connection, issuer-specific issuance flow |
+| 10 | Sub-CA Mode | SKIP | — | Requires CA cert+key on disk |
+| 11 | ACME ARI | SKIP | — | Requires ARI-capable CA |
+| 12 | Vault PKI | SKIP | — | Requires live Vault server |
+| 13 | DigiCert | SKIP | — | Requires DigiCert sandbox |
+| 14 | Target Connectors | 3 | List, create NGINX target, delete 204 | Deploy to real target, validate deployment |
+| 15–17 | Apache/HAProxy, Traefik/Caddy, IIS | — | (Covered by source checks in Parts 42–46) | Requires real services or Windows |
+| 18 | Agent Operations | 3 | Heartbeat (register), metadata check, auto-create on heartbeat | Agent binary behavior, key storage, discovery scan |
+| 19 | Agent Work Routing | 1 | Empty work for agent with no targets | Scoped job assignment, multi-target fan-out |
+| 20 | Post-Deployment Verification | 1 | 404 on nonexistent job verification | TLS probing, fingerprint comparison |
+| 21 | EST Server | 2 | CACerts (200 + content-type), CSRAttrs (200/204) | simpleenroll with CSR, simplereenroll, PKCS#7 parsing |
+| 22 | Certificate Export | 3 | PEM export, PKCS#12 export, 404 on nonexistent | Download mode, file content validation |
+| 23 | S/MIME & EKU Support | 0 (NOT AUTOMATED) | — | S/MIME profile creation; EKU enforcement on issuance; SMIMECapabilities extension presence in issued cert; rejection of profile-violating EKU on CSR. Test manually per `docs/testing-guide.md::Part 23` |
+| 24 | OCSP Responder & DER CRL | 0 (NOT AUTOMATED) | — | OCSP request/response (RFC 6960), DER CRL generation, status (Good/Revoked/Unknown), Must-Staple coordination. Test manually per `docs/testing-guide.md::Part 24` |
+| 25 | Certificate Discovery | 5 | List discovered, summary, list scan targets, create target, invalid CIDR 400 | Agent filesystem scan, claim/dismiss workflow |
+| 26 | Enhanced Query API | 4 | Sort descending, cursor pagination, time-range filter, invalid sort field | Field projection correctness, cursor token cycling |
+| 27 | Request Body Size Limits | 1 | 2MB body rejected (413/400) | Exact limit boundary (1MB) |
+| 28 | CLI | SKIP | — | Requires compiled `certctl-cli` binary |
+| 29 | MCP Server | SKIP | — | Requires compiled `mcp-server` binary + stdio |
+| 30 | Observability | 7 | Dashboard summary, certs by status, expiration timeline, job trends, issuance rate, JSON metrics (uptime + gauges), Prometheus (content-type + 4 metric names) | Chart rendering (GUI), Grafana import |
+| 31 | Notifications | 2 | List, 404 on nonexistent | Notification content, mark-read, email/Slack delivery |
+| 32 | Audit Trail | 3 | List events (≥10), PUT immutability, DELETE immutability | Actor attribution, body hash, time range filters |
+| 33 | Background Scheduler | SKIP | — | Timing-dependent; verify via Docker logs |
+| 34 | Structured Logging | SKIP | — | Requires Docker log inspection |
+| 35 | GUI Testing | SKIP | — | Requires browser |
+| 36–37 | Issuer Catalog, Frontend Audit | SKIP | — | Requires browser |
+| 38 | Error Handling | 5 | Malformed JSON, missing required field, method not allowed, UTF-8 CN, empty body | Stack trace suppression, error response format |
+| 39 | Performance | 5 | List certs < 200ms, stats < 500ms, metrics < 200ms, Prometheus < 300ms, audit < 500ms | Load testing, concurrent request handling |
+| 40 | Documentation | 8 | README, quickstart, architecture, connectors, compliance exist; migration guides exist; 8 issuer types in docs; 11 target types in docs | Content accuracy, link validity |
+| 41 | Regression | 3 | DELETE 204, per_page max fallback, network scan target seed count | `errors.Is(errors.New())` anti-pattern source scan |
+| 42 | Envoy Target | 5 | Domain type, connector file, test file, OpenAPI, agent dispatch | Envoy deployment test, SDS config |
+| 43 | Postfix/Dovecot | 3 | Domain types (Postfix + Dovecot), connector file, OpenAPI | Mail server deployment test |
+| 44 | SSH Target | 4 | Domain type, connector file, agent dispatch (`sshconn`), OpenAPI | SSH deployment test (requires target host) |
+| 45 | Windows Certificate Store | 3 | Domain type, connector file, shared certutil package | Windows deployment (requires Windows) |
+| 46 | Java Keystore | 3 | Domain type, connector file, OpenAPI | JKS deployment (requires keytool) |
+| 47 | Certificate Digest Email | 3 | Preview endpoint (200/503), service file, adapter file | SMTP delivery, HTML template rendering |
+| 48 | Dynamic Issuer Config | 4 | Crypto package exists, create ACME issuer via API, config redaction check, migration exists | Test connection flow, registry rebuild |
+| 49 | Dynamic Target Config | 2 | Create NGINX target via API, migration exists | Test connection via agent heartbeat |
+| 50 | Onboarding Wizard | 2 | Wizard component exists, docker-compose split (clean vs demo) | Wizard UI flow, step completion |
+| 51 | ACME Profile Selection | 3 | Profile module exists, frontend config, RFC 9702→9773 renumber check | Profile-aware issuance against real CA |
+| 52 | Helm Chart | 5 | Chart.yaml, values.yaml, 4 templates exist, securityContext, health probes | `helm template` rendering, `helm install` |
+| 53 | Kubernetes Secrets Target Connector (M47) | 18 | Config validation (namespace DNS-1123, secret name DNS subdomain, label keys, required fields), deployment (create/update Secret, chain concatenation, error propagation), validation (serial comparison, not-found, empty cert) | GUI target wizard KubernetesSecrets fields (namespace, secret_name, labels, kubeconfig_path), Helm RBAC toggle, TargetDetailPage type label |
+| 54 | AWS ACM Private CA Issuer Connector (M47) | 23 | Config validation (region, CA ARN regex, signing algorithm whitelist, validity_days, defaults), issuance (full flow, empty CSR, errors), renewal (reuses issuance), revocation (reason mapping, default, errors), GetOrderStatus completed, GetCACertPEM (success/chain/error), GetRenewalInfo nil | GUI issuer wizard AWSACMPCA fields (region, ca_arn, signing_algorithm, validity_days, template_arn), seed data visibility, create issuer flow |
+| 55 | Agent Soft-Retirement (I-004) | 0 (NOT AUTOMATED) | — | Soft-retire vs hard-retire; force flag; reason capture; foreign-key cascade behavior on retired-agent cert ownership; reactivation. Test manually per `docs/testing-guide.md::Part 55` |
+| 56 | Notification Retry & Dead-Letter Queue (I-005) | 0 (NOT AUTOMATED) | — | Retry loop with exponential backoff, dead-letter transition after N retries, requeue endpoint (`POST /api/v1/notifications/{id}/requeue`), idempotency on retry. Test manually per `docs/testing-guide.md::Part 56` |
+
+**Totals (verified 2026-04-27):** 49 `Part_*` automation wrappers, ~159 leaf subtests, 11 fully
+skipped Parts, 4 Parts not yet automated (23, 24, 55, 56), and an unspecified count of manual-only
+flows (GUI, scheduler timing, Docker log inspection). Run `grep -cE '^## Part [0-9]+:' docs/testing-guide.md`
+and `grep -cE 't\.Run\("Part[0-9]+_' deploy/test/qa_test.go` to re-verify.
+
+## Coverage by Risk Class
+
+A buyer's QA lead reading this doc wants "where are the existential bugs caught?" — Bundle P / Strengthening #1 surfaces that view directly. The table below classifies each Part by risk class so reviewers can answer the existential-coverage question in one glance.
+
+| Risk class | Description | Parts in scope | Automation status |
+|---|---|---|---|
+| **Existential** (Critical paths — bugs would compromise CA, leak keys, mis-issue, bypass revocation) | Crypto, PKCS#7, local-issuer, OCSP/CRL, agent keygen, CSR validation | 5 (Revocation), 21 (EST), 23 (S/MIME EKU), 24 (OCSP/CRL), 47 (Digest with cert content), 53 (K8s Secrets), 54 (AWS PCA) | 5/7 automated; Parts 23 + 24 pending (Bundle I Skip stubs in `qa_test.go`; manual playbook in `testing-guide.md`) |
+| **High** (FSM corruption, credential leak, authn/z weakening) | Renewal, jobs, agents, issuers, deployment, scheduler | 4, 7, 8, 9, 18, 19, 20, 22, 25, 28, 29, 32, 33, 48, 49, 55, 56 | 14/17 automated; CLI / MCP / scheduler-loop are inherently SKIP (require compiled binaries / Docker logs); Parts 55 + 56 pending |
+| **Medium** (Operational pain or silent data drift) | Targets, notifiers, observability, error handling, performance, regression | 14, 15-17, 30, 31, 38, 39, 40, 41, 42, 43, 44, 45, 46 | 14/14 automated (15-17 indirect via Parts 42–46) |
+| **Low** (Hygiene) | Documentation, docs verification | 40 (Documentation), 50 (Onboarding) | 2/2 automated |
+| **Frontend** (XSS, render correctness, mutation contracts) | GUI testing | 35, 36-37 | 0/3 automated in this suite (Vitest covers separately under `web/`); this doc punts to manual + Vitest |
+| **Compliance** (PCI / SOC2 / HIPAA-relevant) | Audit trail, body-size limits, request limits, Helm chart deploy posture | 27, 32, 51, 52 | 4/4 automated |
+
+This is the table acquisition reviewers screenshot for their report. When a new Part lands in `testing-guide.md`, classify it here; the QA-doc Part-count drift guard (`.github/workflows/ci.yml::QA-doc Part-count drift guard`) catches the count mismatch.
+
+## Test Categories
+
+The automated tests fall into four categories:
+
+### 1. API Integration Tests (majority)
+Make real HTTP requests to the running server and verify status codes, response structure, and JSON field values. Examples:
+- `POST /api/v1/certificates` with valid payload → 201
+- `GET /api/v1/certificates?status=Active` → all returned certs have `status: "Active"`
+- `DELETE /api/v1/certificates/mc-qa-full` → 204
+
+### 2. Database Verification Tests
+Connect directly to PostgreSQL and verify schema state:
+- Table count ≥ 19 (from migrations 000001–000010)
+- Useful for catching migration regressions
+
+### 3. Source File Verification Tests
+Read files from the repo checkout and verify structure:
+- Domain types exist in `internal/domain/connector.go` (e.g., `TargetTypeEnvoy`)
+- Connector implementations exist (e.g., `internal/connector/target/envoy/envoy.go`)
+- Documentation contains expected content (all issuer/target types listed)
+- No stale RFC 9702 references (replaced by RFC 9773)
+
+### 4. Performance Spot Checks
+Timed API requests with threshold assertions:
+- `GET /api/v1/certificates?per_page=15` < 200ms
+- `GET /api/v1/stats/summary` < 500ms
+- `GET /api/v1/metrics/prometheus` < 300ms
+
+## What This Test Does NOT Cover
+
+These gaps must be filled by manual testing per `docs/testing-guide.md`:
+
+### Not Yet Automated (Parts 23, 24, 55, 56)
+
+These Parts are documented in `docs/testing-guide.md` but have no `Part_*` automation
+in `qa_test.go` yet. They are operator-runnable from the manual playbook; QA-suite
+automation should land before the next acquisition-grade release.
+
+- **Part 23: S/MIME & EKU Support** — profile-driven EKU enforcement; SMIMECapabilities extension
+- **Part 24: OCSP Responder & DER CRL** — OCSP request/response correctness, CRL generation, Must-Staple coordination
+- **Part 55: Agent Soft-Retirement (I-004)** — soft vs hard retire, FK cascade, reactivation
+- **Part 56: Notification Retry & Dead-Letter Queue (I-005)** — retry semantics, dead-letter transition, requeue
+
+### External CA Integrations (Parts 10–13)
+- **Sub-CA mode** — requires CA cert+key files on disk
+- **ACME ARI** — requires a CA that supports RFC 9773 Renewal Information
+- **Vault PKI** — requires a running HashiCorp Vault instance
+- **DigiCert / Sectigo / Google CAS** — requires sandbox API credentials
+
+### Browser/GUI Testing (Parts 35–37, 50)
+- Dashboard chart rendering (Recharts)
+- Onboarding wizard step-by-step flow
+- Issuer catalog card layout and create wizard
+- Bulk operations UI (multi-select, progress bars)
+- Discovery triage workflow
+
+### Real Deployment Testing (Parts 15–17)
+- NGINX/Apache/HAProxy file write + reload
+- Traefik/Caddy file provider or API reload
+- IIS PowerShell/WinRM (requires Windows)
+- F5 BIG-IP iControl REST (requires appliance or mock)
+- SSH agentless deployment (requires target host)
+
+### Agent Binary Behavior (Parts 18, 28–29)
+- Agent-side ECDSA key generation and CSR submission
+- Agent filesystem discovery scan
+- CLI tool (`certctl-cli`) — all 10 subcommands
+- MCP server (`mcp-server`) — stdio transport
+
+### Timing-Dependent Tests (Parts 33–34)
+- Background scheduler loop execution (renewal, jobs, health, notifications, digest, network scan)
+- Structured logging format verification (requires Docker log parsing)
+
+## How This Relates to `integration_test.go`
+
+Both files live in `deploy/test/` in the same Go package (`integration_test`):
+
+| | `qa_test.go` | `integration_test.go` |
+|---|---|---|
+| **Build tag** | `//go:build qa` | `//go:build integration` |
+| **Target stack** | Demo (`docker-compose.yml` + `docker-compose.demo.yml`) | Test (`docker-compose.test.yml`) |
+| **Port** | 8443 | Different (test stack config) |
+| **Seed data** | `seed_demo.sql` (32 certs, 12 agents, 13 issuers, 8 targets, realistic history) | Minimal (created by tests) |
+| **CA backends** | Local CA only (demo mode) | Pebble ACME, step-ca, NGINX |
+| **Purpose** | Release QA — broad coverage, spot checks | Functional — end-to-end issuance, renewal, revocation against real CAs |
+| **Run frequency** | Before each release tag | CI on every PR |
+
+They are complementary. Integration tests prove the machinery works. QA tests prove the product works at release quality.
+
+## Seed Data Reference
+
+The QA tests depend on `migrations/seed_demo.sql`. Key IDs used:
+
+### Certificates (32 total in `managed_certificates`)
+
+The full canonical list is generated by:
+```
+sed -n '/^INSERT INTO managed_certificates/,/^;/p' migrations/seed_demo.sql \
+  | grep -oE "^\s*\('mc-[a-z0-9_-]+" | sed -E "s/^\s*\('//" | sort -u
+```
+
+Hand-listing is unsustainable as the seed grows; tests reference IDs by lookup, not by enumeration.
+Sample IDs: `mc-api-prod`, `mc-web-prod`, `mc-pay-prod`, `mc-compromised`, `mc-smime-bob`, `mc-edge-eu`, `mc-k8s-ingress`, `mc-wildcard-prod`. See `migrations/seed_demo.sql:147` onward.
+
+### Agents (12 total in `agents` table)
+
+8 named workload agents + 1 server-side sentinel + 3 cloud-discovery sentinels:
+
+- **Workload agents:** `ag-web-prod`, `ag-web-staging`, `ag-lb-prod`, `ag-iis-prod`, `ag-data-prod`, `ag-edge-01`, `ag-k8s-prod`, `ag-mac-dev`
+- **Server-side sentinel:** `server-scanner`
+- **Cloud-discovery sentinels:** `cloud-aws-sm`, `cloud-azure-kv`, `cloud-gcp-sm`
+
+Full list via:
+```
+sed -n '/^INSERT INTO agents/,/^;/p' migrations/seed_demo.sql \
+  | grep -oE "^\s*\('[a-z][a-z0-9_-]+" | sed -E "s/^\s*\('//"
+```
+
+(The `agent_groups` table also contains entries with `ag-*` IDs — `ag-linux-prod`, `ag-windows`, `ag-datacenter-a`, `ag-arm64`, `ag-manual` — but those are *group* IDs, not agents. Don't confuse the two.)
+
+### Issuers (13 total)
+
+`iss-local`, `iss-acme-le`, `iss-stepca`, `iss-acme-zs`, `iss-openssl`, `iss-vault`, `iss-digicert`, `iss-sectigo`, `iss-googlecas`, `iss-awsacmpca`, `iss-entrust`, `iss-globalsign`, `iss-ejbca`.
+
+Full list via:
+```
+sed -n '/^INSERT INTO issuers/,/^;/p' migrations/seed_demo.sql \
+  | grep -oE "^\s*\('iss-[a-z0-9_-]+" | sed -E "s/^\s*\('//"
+```
+
+### Targets (8 total in `deployment_targets`)
+`tgt-nginx-prod`, `tgt-nginx-staging`, `tgt-haproxy-prod`, `tgt-apache-prod`, `tgt-iis-prod`, `tgt-traefik-prod`, `tgt-caddy-prod`, `tgt-nginx-data`
+
+### Network Scan Targets (4 total in `network_scan_targets`)
+`nst-dc1-web`, `nst-dc2-apps`, `nst-dmz`, `nst-edge`
+
+**Maintenance note:** when adding new seed rows, also update this section, OR remove the
+per-table counts and rely on the `sed | grep` commands so the doc stops drifting on every
+seed-data change. A CI guard that fails when the doc count diverges from the seed file is
+proposed in `coverage-audit-2026-04-27/tables/qa-doc-strengthening.md` (Strengthening #6).
+
+## Troubleshooting
+
+### "Server unreachable" on startup
+The test pings `GET /health` before running anything. If this fails:
+```bash
+# Check if the stack is running
+docker compose -f docker-compose.yml -f docker-compose.demo.yml ps
+
+# Check server logs
+docker compose -f docker-compose.yml -f docker-compose.demo.yml logs certctl-server
+
+# Check if the port is exposed (self-signed cert — pin CA bundle)
+curl --cacert ./deploy/test/certs/ca.crt -s https://localhost:8443/health
+```
+
+### "connect to QA DB" failure
+The database tests connect directly to PostgreSQL. Ensure port 5432 is exposed:
+```bash
+docker compose -f docker-compose.yml -f docker-compose.demo.yml port postgres 5432
+```
+
+### Performance tests flaking
+The performance thresholds (200ms, 300ms, 500ms) assume a local Docker stack. On slow CI runners or remote Docker hosts, increase the thresholds or skip Part 39:
+```bash
+go test -tags qa -v -run 'TestQA/Part(?!39)' ./...
+```
+
+### Source file checks failing
+The `fileExists` and `fileContains` helpers read from `CERTCTL_QA_REPO_DIR` (default `../..`). If running from a non-standard location:
+```bash
+CERTCTL_QA_REPO_DIR=/absolute/path/to/certctl go test -tags qa -v ./...
+```
+
+## Release Day Sign-Off Matrix
+
+Before tagging a release, the QA-on-call engineer signs off on each row. This matrix replaces the previous ad-hoc release checklist and ties test execution directly to release approval. Acquisition-grade releases have this kind of matrix; the doc previously didn't.
+
+| Sign-off | Evidence | Owner | Result | Date |
+|---|---|---|---|---|
+| `make verify` clean on master | CI run URL | Eng-on-call | ☐ | |
+| `go test -tags qa ./deploy/test/...` ≥ 95% pass rate (skips counted as pass) | Test output | QA-on-call | ☐ | |
+| `go test -race -count=10 ./internal/...` 0 races | `tool-output/race-x10.txt` | QA-on-call | ☐ | |
+| Coverage ≥ thresholds in `ci.yml` (service / handler / crypto / local-issuer / acme / stepca / mcp) | `tool-output/cover-summary.txt` | QA-on-call | ☐ | |
+| Helm chart `helm lint && helm template` clean | `tool-output/helm.txt` | DevOps-on-call | ☐ | |
+| All `t.Skip` sites have current rationales (see Bundle O audit; CI guard catches new orphans) | `make qa-stats` t.Skip count | QA-on-call | ☐ | |
+| Frontend: Vitest run clean; per-page coverage ≥ 70% | `web/tool-output/vitest.txt` | Frontend-on-call | ☐ | |
+| Manual Parts 23, 24, 55, 56 executed (or explicit defer with rationale) | This sheet | QA-on-call | ☐ | |
+| Demo stack `docker compose up -d --build` smoke (`/health` 200, `/ready` 200) | curl receipt | QA-on-call | ☐ | |
+| `govulncheck ./...` clean (or deferred-call advisories tracked in `gap-backlog`) | `tool-output/govulncheck.json` | Security-on-call | ☐ | |
+| QA-doc drift guards green (Part-count + cert-count) | CI run URL | QA-on-call | ☐ | |
+| FSM transition coverage tables (`coverage-audit-2026-04-27/tables/fsm-coverage.md`) — Existential FSMs ≥80% legal + 100% illegal | This sheet | QA-on-call | ☐ | |
+
+**Sign-off owner:** ______________________ &nbsp;&nbsp;**Date:** ______ &nbsp;&nbsp;**Tag:** v__.__.__
+
+## Mutation Testing Targets & Kill Rate
+
+Mutation testing exposes which assertions are actually load-bearing — tests can pass against broken code if mutations survive, which is a coverage trap. The audit's Phase 0 attempted to run `go-mutesting` on the Existential cluster but was blocked by a Go 1.25 / arm64 incompatibility in `osutil@v1.6.1` (uses `syscall.Dup2` which is undefined on linux/arm64). The operator-runnable workaround uses a fork that targets `unix.Dup3` instead.
+
+| Package | Risk class | Target kill rate | Last measured | Tool |
+|---|---|---|---|---|
+| `internal/crypto` | Existential | ≥90% | unmeasured (sandbox-blocked, operator-runnable) | go-mutesting |
+| `internal/pkcs7` | Existential | ≥90% | unmeasured | go-mutesting |
+| `internal/connector/issuer/local` | Existential | ≥90% | unmeasured | go-mutesting |
+| `internal/connector/issuer/acme` | Existential | ≥80% (catch-up; failure-mode coverage 55.6% per Bundle J) | unmeasured | go-mutesting |
+| `internal/connector/issuer/stepca` | Existential | ≥85% (post-Bundle-L.B coverage at 90.4%) | unmeasured | go-mutesting |
+| `internal/api/middleware` | High | ≥80% | unmeasured | go-mutesting |
+| `internal/validation` | Existential (CWE-78 / CWE-113 boundary) | ≥90% | unmeasured | go-mutesting |
+| `web/src/utils/safeHtml.ts` | Frontend (XSS gate) | ≥90% | unmeasured | Stryker |
+
+### Operator command (per package)
+
+```bash
+# Use the avito-tech fork that supports linux/arm64 + Go 1.25.
+go install github.com/avito-tech/go-mutesting/cmd/go-mutesting@latest
+
+mkdir -p tool-output
+$(go env GOPATH)/bin/go-mutesting --debug ./internal/crypto/... \
+  > tool-output/mutation-crypto.txt 2>&1
+grep -oE 'mutation score is [0-9.]+' tool-output/mutation-crypto.txt | tail -1
+```
+
+**Acceptance:** ≥80% (Existential) / ≥70% (High). Anything below is a Medium finding; triage entries go in `coverage-audit-2026-04-27/gap-backlog.md`. This subsection moves mutation testing from "future work" to "documented release gate."
+
+## Adding New Tests
+
+When a new feature ships:
+
+1. **Add a Part section** in `qa_test.go` following the numbering in `docs/testing-guide.md`
+2. **API tests**: use `c.get()`, `c.post()`, `c.bodyStr()`, `c.getJSON()`, `c.timedGet()`
+3. **Source checks**: use `fileExists(t, "relative/path")` and `fileContains(t, "path", "substring")`
+4. **DB checks**: use `openQADB(t)` and `db.queryInt(t, "SELECT ...")`
+5. **Cleanup**: always use `t.Cleanup()` for data created during tests
+6. **Skip if external**: use `t.Skip("Requires X — manual test")` with a clear reason
+
+## Version History
+
+- **v1.3** (April 2026, post-Bundle-P) — QA Doc Strengthening shipped. New top-of-doc Test Suite Health dashboard (regenerated via `make qa-stats`). New Coverage by Risk Class table after the Coverage Map. New Release Day Sign-Off Matrix and Mutation Testing Targets sections. CI seed-count + Part-count drift guards land in `.github/workflows/ci.yml` so future doc drift fails CI. Bundle P closes M-007 / M-010 / M-011 / M-012 (structural strengthening) + M-008 (Mutation Testing Targets).
+- **v1.2** (April 2026, post-coverage-audit) — Documented Parts 55–56 (I-004 Agent Soft-Retirement, I-005 Notification Retry & Dead-Letter) and surfaced Parts 23–24 (S/MIME & EKU; OCSP/CRL) as not-yet-automated. 56 Parts total in `testing-guide.md`; 49 live `Part_*` automation wrappers in `qa_test.go` + 4 new `Skip` stubs for Parts 23/24/55/56 = 53 wrappers (Parts 15–17 remain covered by source-checks in Parts 42–46). Reconciled seed-data section to actual `seed_demo.sql` counts (12 agents, 13 issuers; certs were already accurate at 32). Bundle I of the 2026-04-27 coverage-audit closure plan.
+- **v1.1** (April 2026) — Added Parts 53–54 (M47: Kubernetes Secrets target + AWS ACM PCA issuer). 54 Parts total, ~164 automated subtests.
+- **v1.0** (April 2026) — Initial release covering all 52 Parts of testing-guide.md v2.1. Replaces `qa-smoke-test.sh`.

docs/quickstart.md

@@ -43,6 +43,8 @@ On Linux, follow the official Docker install guide for your distribution.
 
 ## Start Everything
 
+### Docker Compose (Quick Start)
+
 ```bash
 git clone https://github.com/shankar0123/certctl.git
 cd certctl
@@ -58,6 +60,39 @@ cp deploy/.env.example deploy/.env
 docker compose -f deploy/docker-compose.yml up -d --build
 ```
 
+> **Warning:** Edit `POSTGRES_PASSWORD` *before* the very first `docker compose up`. Postgres seeds the password into its data directory only on first boot of an empty volume — after that, the password is baked into `pg_authid` and the env var is ignored. If you boot once with the default and later change `POSTGRES_PASSWORD` in `.env`, the certctl-server container picks up the new value but postgres still authenticates against the old one, and the server logs `pq: password authentication failed for user "certctl"` (SQLSTATE 28P01). Two ways out: tear down the volume with `docker compose -f deploy/docker-compose.yml down -v` (this **deletes all data**) and bring up fresh, or rotate non-destructively with `docker compose -f deploy/docker-compose.yml exec postgres psql -U certctl -c "ALTER ROLE certctl PASSWORD '<new>';"` and then restart certctl-server with the matching `POSTGRES_PASSWORD`.
+
+### Docker Compose Environments
+
+The `deploy/` directory contains four compose files for different use cases:
+
+| File | Purpose | How to run |
+|------|---------|------------|
+| `docker-compose.yml` | **Base platform.** PostgreSQL + certctl server + agent. Clean dashboard with onboarding wizard — use this for production or first-time setup. | `docker compose -f deploy/docker-compose.yml up --build` |
+| `docker-compose.demo.yml` | **Demo data override.** Layers 180 days of realistic seed data (15 certs, 5 agents, multiple issuers) onto the base. Dashboard charts and tables look populated on first boot. | `docker compose -f deploy/docker-compose.yml -f deploy/docker-compose.demo.yml up --build` |
+| `docker-compose.dev.yml` | **Development override.** Adds PgAdmin (port 5050), debug-level logging, and a Delve debugger port (40000) for the server. | `docker compose -f deploy/docker-compose.yml -f deploy/docker-compose.dev.yml up --build` |
+| `docker-compose.test.yml` | **Integration test environment.** 7 containers on a static-IP subnet: PostgreSQL, certctl server+agent, step-ca, Pebble ACME server, challenge test server, and NGINX. Runs the full issuance→deployment→verification flow against real CA backends. Standalone — does not combine with the base file. | `docker compose -f deploy/docker-compose.test.yml up --build` |
+
+Override files are layered onto the base with multiple `-f` flags. The test environment is self-contained and runs independently. To reset any environment's data, add `down -v` to remove volumes.
+
+For a deep dive into every service, environment variable, and networking decision, see the [Docker Compose Environments Guide](../deploy/ENVIRONMENTS.md).
+
+### Kubernetes with Helm
+
+For production deployments on Kubernetes, use the Helm chart:
+
+```bash
+helm install certctl deploy/helm/certctl/ \
+  --create-namespace --namespace certctl \
+  --set server.auth.apiKey="your-secure-api-key" \
+  --set postgresql.auth.password="your-db-password" \
+  --set ingress.enabled=true \
+  --set ingress.hosts[0].host="certctl.example.com" \
+  --set ingress.hosts[0].tls=true
+```
+
+The chart includes: server Deployment (with configurable replicas, health probes, security context), PostgreSQL StatefulSet with persistent volumes, agent DaemonSet (one agent per infrastructure node), optional Ingress with TLS, and ServiceAccount with RBAC. All certctl configuration options are exposed in `values.yaml` — customize issuer settings, target connectors, scheduler intervals, and notifier credentials there.
+
 Wait about 30 seconds for PostgreSQL to initialize, then verify:
 
 ```bash
@@ -72,22 +107,30 @@ certctl-server       Up (healthy)
 certctl-agent        Up
 ```
 
+The control plane is HTTPS-only as of v2.2. The `certctl-tls-init` init container in the shipped `deploy/docker-compose.yml` self-signs a cert on first boot and drops it into a named volume. Extract the CA bundle once and reuse it for every API call in this guide:
+
 ```bash
-curl http://localhost:8443/health
+export CA=/tmp/certctl-ca.crt
+docker compose -f deploy/docker-compose.yml exec -T certctl-server \
+  cat /etc/certctl/tls/ca.crt > "$CA"
+
+curl --cacert "$CA" https://localhost:8443/health
 ```
 ```json
 {"status":"healthy"}
 ```
 
+If you're bringing your own cert (internal CA, cert-manager, operator-supplied Secret), see [`docs/tls.md`](tls.md) for the full provisioning matrix. If you're cutting over an existing install, see [`docs/upgrade-to-tls.md`](upgrade-to-tls.md) for the failure modes (out-of-date `http://…` agents fail at the TLS handshake) and the one-step procedure.
+
 ## Open the Dashboard
 
-Open **http://localhost:8443** in your browser.
+Open **https://localhost:8443** in your browser. Your browser will warn about the self-signed cert — that's expected for the demo bootstrap. Trust the CA bundle you just exported, or click through the warning.
 
 > **Note:** The Docker Compose demo runs with authentication disabled (`CERTCTL_AUTH_TYPE=none`) so you can explore immediately. For production, set `CERTCTL_AUTH_TYPE=api-key` and `CERTCTL_AUTH_SECRET=<your-secret>` in your environment, then pass `Authorization: Bearer <your-secret>` on all API requests. The dashboard will prompt for your API key on first load.
 >
 > **Key rotation:** `CERTCTL_AUTH_SECRET` accepts comma-separated keys (e.g., `CERTCTL_AUTH_SECRET=new-key,old-key`). Both keys are valid simultaneously, enabling zero-downtime rotation: add the new key, roll clients over, then remove the old key.
 
-The dashboard comes pre-loaded with 15 demo certificates across multiple teams, environments, and statuses — expiring certs, expired certs, active certs, failed renewals. A realistic snapshot of what certificate management looks like in a real organization.
+The dashboard comes pre-loaded with 35 demo certificates across 5 issuers, 8 agents, and 90 days of job history — expiring certs, expired certs, active certs, failed renewals, revocations, discovery scans, and approval workflows. A realistic snapshot of what certificate management looks like in a real organization.
 
 ### What you're looking at
 
@@ -109,7 +152,7 @@ Explore the sidebar: Certificates, Agents, Policies, Jobs, Audit Trail, Notifica
 
 **"I need to approve a renewal before it proceeds"** — Click "Jobs" in the sidebar. You'll see an amber banner: "2 jobs awaiting approval." These are renewal jobs for `auth-production` and `payments-production` that require human sign-off before proceeding. Click Approve or Reject with a reason — the decision is recorded in the audit trail.
 
-**"Show me the agent fleet"** — Click "Agents." Four agents online, one offline. Click "Fleet Overview" for OS/architecture grouping, version distribution, and per-platform listing. Agents generate ECDSA P-256 keys locally — private keys never leave your infrastructure.
+**"Show me the agent fleet"** — Click "Agents." Eight agents across Linux, macOS, and Windows platforms—most online, showing OS, architecture, IP, and version metadata. A ninth entry (server-scanner) is the sentinel agent used for network certificate discovery. Click "Fleet Overview" for OS/architecture grouping, version distribution, and per-platform listing. Agents generate ECDSA P-256 keys locally — private keys never leave your infrastructure.
 
 **"What about bulk operations?"** — On the Certificates page, select multiple certificates with checkboxes. A bulk action bar appears: trigger renewal, revoke with reason codes, or reassign ownership — all with progress tracking. At 47-day lifespans with hundreds of certs, bulk operations aren't optional.
 
@@ -121,62 +164,64 @@ Everything you see in the dashboard is backed by the REST API. All endpoints liv
 
 ### Core operations
 
+Every request below uses `--cacert "$CA"` to pin the self-signed CA bundle extracted above. In production, point `$CA` at your internal CA root or the bundle you distributed to the fleet.
+
 ```bash
 # List all certificates
-curl -s http://localhost:8443/api/v1/certificates | jq .
+curl --cacert "$CA" -s https://localhost:8443/api/v1/certificates | jq .
 
 # Filter by status
-curl -s "http://localhost:8443/api/v1/certificates?status=Expiring" | jq .
+curl --cacert "$CA" -s "https://localhost:8443/api/v1/certificates?status=Expiring" | jq .
 
 # Filter by environment
-curl -s "http://localhost:8443/api/v1/certificates?environment=production" | jq .
+curl --cacert "$CA" -s "https://localhost:8443/api/v1/certificates?environment=production" | jq .
 
 # Get a specific certificate
-curl -s http://localhost:8443/api/v1/certificates/mc-api-prod | jq .
+curl --cacert "$CA" -s https://localhost:8443/api/v1/certificates/mc-api-prod | jq .
 
 # Get deployment targets for a certificate
-curl -s http://localhost:8443/api/v1/certificates/mc-api-prod/deployments | jq .
+curl --cacert "$CA" -s https://localhost:8443/api/v1/certificates/mc-api-prod/deployments | jq .
 
 # List agents
-curl -s http://localhost:8443/api/v1/agents | jq .
+curl --cacert "$CA" -s https://localhost:8443/api/v1/agents | jq .
 
 # Check agent pending work
-curl -s http://localhost:8443/api/v1/agents/ag-web-prod/work | jq .
+curl --cacert "$CA" -s https://localhost:8443/api/v1/agents/ag-web-prod/work | jq .
 
 # View audit trail
-curl -s http://localhost:8443/api/v1/audit | jq .
+curl --cacert "$CA" -s https://localhost:8443/api/v1/audit | jq .
 
 # View policies and violations
-curl -s http://localhost:8443/api/v1/policies | jq .
-curl -s http://localhost:8443/api/v1/policies/pr-require-owner/violations | jq .
+curl --cacert "$CA" -s https://localhost:8443/api/v1/policies | jq .
+curl --cacert "$CA" -s https://localhost:8443/api/v1/policies/pr-require-owner/violations | jq .
 
 # Notifications
-curl -s http://localhost:8443/api/v1/notifications | jq .
+curl --cacert "$CA" -s https://localhost:8443/api/v1/notifications | jq .
 
 # Profiles and agent groups
-curl -s http://localhost:8443/api/v1/profiles | jq .
-curl -s http://localhost:8443/api/v1/agent-groups | jq .
+curl --cacert "$CA" -s https://localhost:8443/api/v1/profiles | jq .
+curl --cacert "$CA" -s https://localhost:8443/api/v1/agent-groups | jq .
 ```
 
 ### Sorting, filtering, and pagination
 
 ```bash
 # Sort by expiration date (ascending)
-curl -s "http://localhost:8443/api/v1/certificates?sort=notAfter" | jq .
+curl --cacert "$CA" -s "https://localhost:8443/api/v1/certificates?sort=notAfter" | jq .
 
 # Sort descending (prefix with -)
-curl -s "http://localhost:8443/api/v1/certificates?sort=-createdAt" | jq .
+curl --cacert "$CA" -s "https://localhost:8443/api/v1/certificates?sort=-createdAt" | jq .
 
 # Time-range filters (RFC3339)
-curl -s "http://localhost:8443/api/v1/certificates?expires_before=2026-05-01T00:00:00Z" | jq .
-curl -s "http://localhost:8443/api/v1/certificates?created_after=2026-03-01T00:00:00Z" | jq .
+curl --cacert "$CA" -s "https://localhost:8443/api/v1/certificates?expires_before=2026-05-01T00:00:00Z" | jq .
+curl --cacert "$CA" -s "https://localhost:8443/api/v1/certificates?created_after=2026-03-01T00:00:00Z" | jq .
 
 # Sparse fields — request only what you need
-curl -s "http://localhost:8443/api/v1/certificates?fields=id,common_name,status,expires_at" | jq .
+curl --cacert "$CA" -s "https://localhost:8443/api/v1/certificates?fields=id,common_name,status,expires_at" | jq .
 
 # Cursor pagination — efficient for large inventories
-curl -s "http://localhost:8443/api/v1/certificates?page_size=5" | jq '{next_cursor: .next_cursor, count: (.data | length)}'
-curl -s "http://localhost:8443/api/v1/certificates?cursor=<next_cursor_value>&page_size=5" | jq .
+curl --cacert "$CA" -s "https://localhost:8443/api/v1/certificates?page_size=5" | jq '{next_cursor: .next_cursor, count: (.data | length)}'
+curl --cacert "$CA" -s "https://localhost:8443/api/v1/certificates?cursor=<next_cursor_value>&page_size=5" | jq .
 ```
 
 Supported sort fields: `notAfter`, `expiresAt`, `createdAt`, `updatedAt`, `commonName`, `name`, `status`, `environment`.
@@ -185,22 +230,22 @@ Supported sort fields: `notAfter`, `expiresAt`, `createdAt`, `updatedAt`, `commo
 
 ```bash
 # Dashboard summary
-curl -s http://localhost:8443/api/v1/stats/summary | jq .
+curl --cacert "$CA" -s https://localhost:8443/api/v1/stats/summary | jq .
 
 # Certificates by status
-curl -s http://localhost:8443/api/v1/stats/certificates-by-status | jq .
+curl --cacert "$CA" -s https://localhost:8443/api/v1/stats/certificates-by-status | jq .
 
 # Expiration timeline (next 90 days)
-curl -s "http://localhost:8443/api/v1/stats/expiration-timeline?days=90" | jq .
+curl --cacert "$CA" -s "https://localhost:8443/api/v1/stats/expiration-timeline?days=90" | jq .
 
 # Job trends (last 30 days)
-curl -s "http://localhost:8443/api/v1/stats/job-trends?days=30" | jq .
+curl --cacert "$CA" -s "https://localhost:8443/api/v1/stats/job-trends?days=30" | jq .
 
 # JSON metrics
-curl -s http://localhost:8443/api/v1/metrics | jq .
+curl --cacert "$CA" -s https://localhost:8443/api/v1/metrics | jq .
 
 # Prometheus format (for Prometheus, Grafana Agent, Datadog)
-curl -s http://localhost:8443/api/v1/metrics/prometheus
+curl --cacert "$CA" -s https://localhost:8443/api/v1/metrics/prometheus
 ```
 
 ## Create Your First Certificate
@@ -208,7 +253,7 @@ curl -s http://localhost:8443/api/v1/metrics/prometheus
 Create a certificate record that certctl will track, renew, and deploy automatically.
 
 ```bash
-curl -s -X POST http://localhost:8443/api/v1/certificates \
+curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/certificates \
   -H "Content-Type: application/json" \
   -d '{
     "name": "My First Certificate",
@@ -231,31 +276,34 @@ CERT_ID="<paste the id from the response>"
 
 Trigger renewal:
 ```bash
-curl -s -X POST http://localhost:8443/api/v1/certificates/$CERT_ID/renew | jq .
+curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/certificates/$CERT_ID/renew | jq .
 ```
 
 Check the result:
 ```bash
-curl -s http://localhost:8443/api/v1/certificates/$CERT_ID | jq .
+curl --cacert "$CA" -s https://localhost:8443/api/v1/certificates/$CERT_ID | jq .
 ```
 
-Refresh the dashboard at http://localhost:8443 — your new certificate appears in the inventory.
+Refresh the dashboard at https://localhost:8443 — your new certificate appears in the inventory.
 
 ### Revoke a certificate
 
 When a private key is compromised or a service is decommissioned:
 
 ```bash
-curl -s -X POST http://localhost:8443/api/v1/certificates/$CERT_ID/revoke \
+curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/certificates/$CERT_ID/revoke \
   -H "Content-Type: application/json" \
   -d '{"reason": "superseded"}' | jq .
 ```
 
 Supported RFC 5280 reason codes: `unspecified`, `keyCompromise`, `caCompromise`, `affiliationChanged`, `superseded`, `cessationOfOperation`, `certificateHold`, `privilegeWithdrawn`.
 
-Confirm via CRL:
+Confirm via the unauthenticated DER CRL (RFC 5280 §5, RFC 8615):
 ```bash
-curl -s http://localhost:8443/api/v1/crl | jq .
+# Fetch the CRL without any API key — relying parties shouldn't need one.
+# The CRL path is unauthenticated, but it's still served over TLS.
+curl --cacert "$CA" -s https://localhost:8443/.well-known/pki/crl/iss-local -o /tmp/crl.der
+openssl crl -inform der -in /tmp/crl.der -noout -text | head -40
 ```
 
 ### Interactive approval workflow
@@ -264,15 +312,15 @@ For high-value certificates where you want human oversight. The demo includes 2
 
 ```bash
 # List jobs awaiting approval (demo includes 2)
-curl -s "http://localhost:8443/api/v1/jobs?status=AwaitingApproval" | jq '.data[] | {id, certificate_id, status}'
+curl --cacert "$CA" -s "https://localhost:8443/api/v1/jobs?status=AwaitingApproval" | jq '.data[] | {id, certificate_id, status}'
 
 # Approve a pending job
-curl -s -X POST http://localhost:8443/api/v1/jobs/JOB_ID/approve \
+curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/jobs/JOB_ID/approve \
   -H "Content-Type: application/json" \
   -d '{"reason": "Approved for production deployment"}' | jq .
 
 # Reject a pending job
-curl -s -X POST http://localhost:8443/api/v1/jobs/JOB_ID/reject \
+curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/jobs/JOB_ID/reject \
   -H "Content-Type: application/json" \
   -d '{"reason": "Key type does not meet compliance requirements"}' | jq .
 ```
@@ -298,7 +346,7 @@ export CERTCTL_DISCOVERY_DIRS="/etc/nginx/certs,/etc/ssl/certs,/var/lib/certs"
 export CERTCTL_NETWORK_SCAN_ENABLED=true
 
 # Create a scan target
-curl -s -X POST http://localhost:8443/api/v1/network-scan-targets \
+curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/network-scan-targets \
   -H "Content-Type: application/json" \
   -d '{
     "name": "Internal Network",
@@ -310,20 +358,20 @@ curl -s -X POST http://localhost:8443/api/v1/network-scan-targets \
   }' | jq .
 
 # Trigger an immediate scan
-curl -s -X POST http://localhost:8443/api/v1/network-scan-targets/nst-internal-network/scan | jq .
+curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/network-scan-targets/nst-internal-network/scan | jq .
 ```
 
 ### Triage discovered certificates
 
 ```bash
 # List discovered certs
-curl -s "http://localhost:8443/api/v1/discovered-certificates?agent_id=agent-nginx-prod" | jq .
+curl --cacert "$CA" -s "https://localhost:8443/api/v1/discovered-certificates?agent_id=agent-nginx-prod" | jq .
 
 # Summary counts
-curl -s http://localhost:8443/api/v1/discovery-summary | jq .
+curl --cacert "$CA" -s https://localhost:8443/api/v1/discovery-summary | jq .
 
 # Claim a discovered cert (bring under management)
-curl -s -X POST "http://localhost:8443/api/v1/discovered-certificates/DISCOVERY_ID/claim" \
+curl --cacert "$CA" -s -X POST "https://localhost:8443/api/v1/discovered-certificates/DISCOVERY_ID/claim" \
   -H "Content-Type: application/json" \
   -d '{"managed_certificate_id": "mc-api-prod"}' | jq .
 ```
@@ -333,8 +381,9 @@ curl -s -X POST "http://localhost:8443/api/v1/discovered-certificates/DISCOVERY_
 ```bash
 cd cmd/cli && go build -o certctl-cli .
 
-export CERTCTL_SERVER_URL="http://localhost:8443"
+export CERTCTL_SERVER_URL="https://localhost:8443"
 export CERTCTL_API_KEY="test-key-123"
+export CERTCTL_SERVER_CA_BUNDLE_PATH="$CA"   # or pass --ca-bundle; --insecure for dev self-signed
 
 ./certctl-cli certs list               # List certificates
 ./certctl-cli certs get mc-api-prod    # Certificate details
@@ -346,35 +395,66 @@ export CERTCTL_API_KEY="test-key-123"
 ./certctl-cli status                   # Health + stats
 ```
 
+## Scheduled Certificate Digest Emails
+
+Enable automatic HTML digest emails with certificate stats, expiration timeline, and job health:
+
+```bash
+# Set SMTP configuration
+export CERTCTL_SMTP_HOST=smtp.gmail.com
+export CERTCTL_SMTP_PORT=587
+export CERTCTL_SMTP_USERNAME=admin@example.com
+export CERTCTL_SMTP_PASSWORD=your-app-password
+export CERTCTL_SMTP_FROM_ADDRESS=certctl@example.com
+export CERTCTL_SMTP_USE_TLS=true
+
+# Enable digest and set recipients
+export CERTCTL_DIGEST_ENABLED=true
+export CERTCTL_DIGEST_INTERVAL=24h
+export CERTCTL_DIGEST_RECIPIENTS=ops@example.com,security@example.com
+```
+
+Preview the digest HTML before enabling scheduled delivery:
+```bash
+curl --cacert "$CA" https://localhost:8443/api/v1/digest/preview | jq '.html' | grep -o '<html>' # Shows HTML is ready
+
+# Trigger a digest send immediately (outside of schedule)
+curl --cacert "$CA" -X POST https://localhost:8443/api/v1/digest/send
+```
+
+If no recipients are configured (`CERTCTL_DIGEST_RECIPIENTS` empty), the digest falls back to certificate owner emails. Digests include total certificates, expiring soon, expired, active agents, completed/failed jobs (30-day summary), and a table of expiring certs color-coded by urgency (7/14/30 days).
+
 ## MCP Server (AI Integration)
 
 ```bash
 cd cmd/mcp-server && go build -o mcp-server .
 
-export CERTCTL_SERVER_URL="http://localhost:8443"
+export CERTCTL_SERVER_URL="https://localhost:8443"
 export CERTCTL_API_KEY="test-key-123"
+export CERTCTL_SERVER_CA_BUNDLE_PATH="$CA"   # MCP is env-vars-only; no CLI flags
 
 ./mcp-server
 ```
 
-Exposes 78 MCP tools covering the REST API via stdio transport. Ask Claude: "What certificates are expiring in the next 30 days?", "Revoke the payments cert due to key compromise", "Show me the audit trail."
+Exposes the full REST API via MCP over stdio transport. Ask Claude: "What certificates are expiring in the next 30 days?", "Revoke the payments cert due to key compromise", "Show me the audit trail."
 
 ## Demo Data Reference
 
 | Resource | Count | Examples |
 |----------|-------|---------|
-| Teams | 5 | Platform, Security, Payments, Frontend, Data |
-| Owners | 5 | Alice, Bob, Carol, Dave, Eve |
-| Issuers | 4 | Local Dev CA, Let's Encrypt Staging, step-ca Internal, DigiCert (disabled) |
-| Agents | 6 | ag-web-prod, ag-web-staging, ag-lb-prod, ag-iis-prod, ag-data-prod, server-scanner (network discovery) |
-| Targets | 5 | NGINX (prod/staging/data), F5 LB, IIS |
-| Certificates | 15 | Various statuses: Active, Expiring, Expired, Failed, Wildcard |
-| Discovered Certs | 9 | 5 Unmanaged (filesystem + network), 2 Managed (linked), 1 Dismissed, network-discovered expired printer cert |
-| Discovery Scans | 3 | Agent filesystem scans + network TLS scan |
-| Network Scan Targets | 3 | DC1 Web Servers, DC2 Application Tier, DMZ Public Endpoints |
-| Jobs (Approval) | 2 | AwaitingApproval renewal jobs for auth-prod and payments-prod |
+| Teams | 6 | Platform, Security, Payments, Frontend, Data, DevOps |
+| Owners | 6 | Alice, Bob, Carol, Dave, Eve, Frank |
+| Issuers | 5 | Local Dev CA, Let's Encrypt Staging, step-ca Internal, ZeroSSL (EAB), Custom OpenSSL CA |
+| Agents | 9 | 8 real agents (linux/darwin/windows, amd64/arm64) + server-scanner (network discovery) |
+| Targets | 8 | NGINX prod, NGINX staging, NGINX data, HAProxy, Apache, IIS, Traefik, Caddy |
+| Certificates | 35 | Active, Expiring, Expired, Failed, Revoked, RenewalInProgress, Wildcard, S/MIME |
+| Jobs | 50+ | 90 days of issuance, renewal, deployment jobs + 2 AwaitingApproval |
+| Discovered Certs | 12 | Unmanaged (filesystem + network), Managed (linked), Dismissed |
+| Discovery Scans | 8 | Historical + recent agent filesystem scans + network TLS scans |
+| Network Scan Targets | 4 | DC1 Web Servers, DC2 Application Tier, DMZ Public Endpoints, Edge Locations |
+| Audit Events | 55+ | 90 days of lifecycle events (issuance, renewal, deployment, revocation, discovery) |
 | Policies | 4 | Required owner, allowed environments, max lifetime, min renewal window |
-| Profiles | 4 | Standard TLS, Internal mTLS, Short-Lived, High Security |
+| Profiles | 5 | Standard TLS, Internal mTLS, Short-Lived, High Security, S/MIME Email |
 | Agent Groups | 5 | Linux agents, ARM agents, Production subnet, etc. |
 
 ## Dashboard Demo Mode
@@ -413,7 +493,10 @@ The `-v` flag removes the PostgreSQL data volume for a clean slate.
 
 ## What's Next
 
+**Ready to deploy with your stack?** The [Deployment Examples](examples.md) page has 5 turnkey docker-compose scenarios — pick the one closest to your setup and have it running in minutes. It also covers migration paths from Certbot, acme.sh, and cert-manager.
+
+- **[Deployment Examples](examples.md)** — ACME+NGINX, wildcard DNS-01, private CA+Traefik, step-ca+HAProxy, multi-issuer
 - **[Advanced Demo](demo-advanced.md)** — Issue a real certificate via the Local CA end-to-end
 - **[Architecture](architecture.md)** — How the control plane, agents, and connectors work together
-- **[Connector Guide](connectors.md)** — Build custom connectors for your infrastructure
+- **[Connector Reference](connectors.md)** — Configuration for all 7 issuers and 10 targets
 - **[Concepts Guide](concepts.md)** — TLS certificates, CAs, and private keys explained from scratch

docs/security.md

@@ -0,0 +1,169 @@
+# certctl Security Posture & Operator Guidance
+
+This document collects the operator-facing security guidance that the source
+code's per-finding comment blocks reference. Each section names the audit
+finding it closes, the threat model, and the operator action required (if
+any).
+
+## OCSP responder availability
+
+**Audit reference:** Bundle C / M-020. CWE-770 (uncontrolled resource
+consumption); RFC 6960 (OCSP); RFC 7633 (Must-Staple).
+
+certctl ships an OCSP responder at `/.well-known/pki/ocsp/{issuer_id}/{serial}`
+that signs a fresh response per request. Pre-Bundle-C the unauth handler
+chain had no rate limit, so an attacker could DoS the responder and force
+fail-open relying parties to accept revoked certificates as valid. Bundle C
+adds the same per-key rate limiter to the unauth chain that the authenticated
+chain has used since Bundle B. Per-IP keying applies because OCSP traffic is
+unauthenticated.
+
+The rate limiter alone does not solve the underlying revocation-bypass risk.
+**The architectural fix is for issued certificates to carry the OCSP
+Must-Staple TLS Feature extension** (RFC 7633, OID 1.3.6.1.5.5.7.1.24). When
+present, conforming TLS clients refuse to negotiate a session unless the
+server staples a fresh signed OCSP response in the TLS handshake. This shifts
+revocation enforcement from the client's discretion (which most fail-open by
+default) to a hard requirement that the connection cannot complete without
+proof of non-revocation.
+
+### Operator action
+
+For certificates issued to systems where revocation correctness matters:
+
+1. **Configure the issuer profile to set `must-staple: true`.** Out-of-the-box
+   profiles in `migrations/seed.sql` do not set this; operators add it at
+   profile-creation time via the API or by editing seed data.
+2. **Confirm the relying party honors the extension.** OpenSSL ≥ 1.1.0,
+   Firefox, and Chrome 84+ all enforce Must-Staple. Older clients silently
+   ignore it.
+3. **Confirm the deployment target is configured for OCSP stapling** so the
+   server can actually deliver the stapled response in the handshake.
+   - **nginx:** `ssl_stapling on; ssl_stapling_verify on;`
+   - **Apache:** `SSLUseStapling on`
+   - **HAProxy:** `set ssl ocsp-response /path/to/response.der`
+   - **Envoy:** `ocsp_staple_policy: must_staple`
+
+### What this does NOT cover
+
+- **CRL fallback.** Must-Staple does not affect CRL behavior. Operators with
+  CRL-based relying parties should use the rate-limit + caching defense
+  alone; there is no client-side equivalent to Must-Staple for CRLs.
+- **Self-issued certs in air-gapped networks.** When the relying party
+  cannot reach the OCSP responder at all (the threat model the audit
+  cited), Must-Staple is the only mechanism that closes the bypass. CRL
+  distribution similarly requires the relying party to fetch the CRL,
+  which is also subject to the same network-availability concern.
+
+## Postgres transport encryption
+
+See [docs/database-tls.md](database-tls.md). Bundle B / M-018.
+
+## Encryption at rest
+
+Bundle B / M-001. PBKDF2-SHA256 at 600,000 rounds (OWASP 2024 Password
+Storage Cheat Sheet floor) for the operator-supplied passphrase that
+derives the AES-256-GCM key for sensitive config columns. v3 blob format
+with a per-ciphertext random salt; v1/v2 read fallback for legacy rows.
+See [internal/crypto/encryption.go](../internal/crypto/encryption.go) and
+the accompanying tests for the format spec.
+
+## Authentication surface
+
+Bundle B / M-002. Two layers decide auth-exempt status:
+
+1. **Router layer:** `internal/api/router/router.go::AuthExemptRouterRoutes`
+   — the 4 endpoints registered via direct `r.mux.Handle` without going
+   through the middleware chain (`/health`, `/ready`, `/api/v1/auth/info`,
+   `/api/v1/version`).
+2. **Dispatch layer:** `internal/api/router/router.go::AuthExemptDispatchPrefixes`
+   — URL-prefix routing in `cmd/server/main.go::buildFinalHandler` for
+   `/.well-known/pki/*`, `/.well-known/est/*`, and `/scep[/...]*`.
+
+Both lists have AST-walking regression tests (`auth_exempt_test.go`) that
+fail CI if a new bypass lands without an updating the documented constant.
+
+## Per-user rate limiting
+
+Bundle B / M-025. Authenticated callers are bucketed by API-key name;
+unauthenticated callers (probes, OCSP relying parties, EST/SCEP enrollees)
+are bucketed by source IP. `RPS` and `BurstSize` are per-key budgets.
+`PerUserRPS` / `PerUserBurstSize` give authenticated clients a separate
+budget when set non-zero.
+
+## API key rotation
+
+**Audit reference:** L-004. CWE-924 (improper enforcement of message integrity during transmission in a communication channel) — operator UX variant.
+
+certctl's API keys are configured via the `CERTCTL_API_KEYS_NAMED` env var
+(format `name1:key1,name2:key2:admin`) and parsed at startup into an
+in-memory list. There is no DB-resident key store, no GUI, no `/api/v1/keys`
+endpoint — the env var IS the key inventory.
+
+Pre-Bundle-G the env var rejected duplicate names, so rotating a key
+required: stop accepting OLDKEY → restart → roll NEWKEY out. Any client
+polling against OLDKEY during the restart window hit a 401.
+
+Bundle G adds a **double-key rotation window**: two entries can share a
+name during the rollover, and both keys validate. Operators run the
+rotation as:
+
+1. **Generate the new key.** `openssl rand -hex 32` produces a 256-bit
+   value with sufficient entropy.
+
+2. **Append the new entry to `CERTCTL_API_KEYS_NAMED`** alongside the
+   existing one:
+   ```
+   CERTCTL_API_KEYS_NAMED="alice:OLDKEY:admin,alice:NEWKEY:admin"
+   ```
+   Both entries MUST carry the same admin flag — startup fails loud if
+   they don't (a non-admin shouldn't share an identity with an admin).
+
+3. **Restart certctl.** A startup INFO log confirms the rotation window
+   is active:
+   ```
+   INFO api-key rotation window active name=alice entries=2 see=docs/security.md::api-key-rotation
+   ```
+
+4. **Roll the new key out to all clients.** Both keys validate during
+   this phase. Audit-trail actor + per-user rate-limit bucket stay
+   consistent across the rollover (both entries produce the same
+   `UserKey` context value, the shared name).
+
+5. **Remove the old entry** from `CERTCTL_API_KEYS_NAMED`:
+   ```
+   CERTCTL_API_KEYS_NAMED="alice:NEWKEY:admin"
+   ```
+
+6. **Restart certctl.** OLDKEY now fails with 401. Rotation complete.
+
+The rotation window has no operator-set timeout — it lasts for as long
+as both entries are in the env var. Best practice is a 24-72h window
+covering a full deploy cadence; if a client hasn't rolled to NEWKEY by
+the end of step 4, extend the window before step 5.
+
+### What the contract guarantees
+
+- Two entries with the same `name`: **allowed** if both have the same
+  `admin` flag.
+- Two entries with the same `name` but mismatched admin: **rejected at
+  startup** (privilege escalation guard).
+- Two entries with the same `(name, key)` pair: **rejected at startup**
+  (typo guard — rotation requires DIFFERENT keys under the same name).
+- Single-entry steady state: unchanged from pre-Bundle-G behavior.
+
+### What the contract does NOT do
+
+- **No automatic expiration of OLDKEY.** The operator removes the entry
+  in step 5; certctl doesn't track timestamps. A future enhancement
+  could add a `rotated_at` annotation if operators ask for it.
+- **No GUI / API for key management.** Keys are env-var only by design;
+  building a key-management surface is a separate feature project.
+- **No revocation list.** If a key leaks, the only path is to remove it
+  from the env var and restart. That's appropriate for a small env-var
+  inventory; it would not scale to a per-user-key-issued model.
+
+## Reporting a vulnerability
+
+Email `certctl@proton.me`. Coordinated disclosure preferred; we will
+acknowledge within 72h.

docs/testing-strategy.md

@@ -0,0 +1,198 @@
+# certctl Testing Strategy & Deep-Scan Operator Runbook
+
+This doc covers the **testing topology** (per-PR fast gates vs. daily deep-scan
+gates), and the **operator runbook** for re-running each deep-scan tool locally
+when the CI receipt is ambiguous or when an operator wants to validate a fix
+before the next scheduled scan.
+
+For the manual end-to-end QA playbook, see [`testing-guide.md`](testing-guide.md).
+For the security posture / per-finding closure log, see [`security.md`](security.md).
+
+## CI workflow split
+
+certctl runs two GitHub Actions workflows:
+
+- **`.github/workflows/ci.yml`** — runs on every push/PR. Fast feedback only.
+  Includes `gofmt`, `go vet`, `golangci-lint`, `go test -short -count=1`,
+  `govulncheck`, the per-layer coverage gates, and the regression-grep guards
+  (the M-009 mutation budget, the L-001 InsecureSkipVerify guard, the H-001
+  Dockerfile SHA-pin guard, the M-012 USER-directive guard, etc.).
+- **`.github/workflows/security-deep-scan.yml`** — runs daily 06:00 UTC and on
+  manual dispatch. Heavyweight tools that need docker, network egress to
+  scanner registries, or wall-clock budgets the per-PR check can't tolerate.
+  Includes `gosec`, `osv-scanner`, the `-race -count=10` full-suite run,
+  `trivy` image scan, `syft` SBOM, ZAP baseline DAST, `nuclei`,
+  `schemathesis` OpenAPI fuzz, `testssl.sh`, `go-mutesting` mutation testing,
+  and `semgrep p/react-security`.
+
+Receipts from each scheduled run are uploaded as a 30-day-retention artefact
+named `security-deep-scan-<run-id>`. Audit them via the GitHub Actions UI;
+download the artefact zip for any scan that surfaces a finding.
+
+## Operator runbook — local re-run procedures
+
+These are the same commands the workflow runs, intended for an operator with
+a workstation that has docker + the Go toolchain installed. The local-run
+shape is identical to CI; the difference is wall-clock and the artefact
+location (CI uploads; local writes to `$PWD`).
+
+### Mutation testing (D-003)
+
+**Tool:** [`go-mutesting`](https://github.com/zimmski/go-mutesting). Mutates
+each AST node in turn (flips comparisons, swaps return values, removes
+statements) and re-runs the package's tests. A mutant is **killed** if any
+test fails; **surviving** mutants indicate a coverage gap (no test caught
+the bug the mutant introduced).
+
+**Targets:** the three security-critical packages whose coverage gate is
+**85%** in `ci.yml`:
+
+- `internal/crypto/`
+- `internal/pkcs7/`
+- `internal/connector/issuer/local/`
+
+**Acceptance threshold:** ≥80% mutation kill ratio per package. Surviving
+mutants below that threshold get triaged in
+`cowork/comprehensive-audit-2026-04-25/d003-mutation-results.md` — either
+ship a targeted unit test that kills the mutant, or document an
+equivalent-mutation justification.
+
+**Local run:**
+
+```
+go install github.com/zimmski/go-mutesting/cmd/go-mutesting@latest
+for pkg in ./internal/crypto/... ./internal/pkcs7/... ./internal/connector/issuer/local/...; do
+  echo "=== $pkg ==="
+  $(go env GOPATH)/bin/go-mutesting "$pkg"
+done
+```
+
+The tool prints one line per mutant (`PASS` = killed, `FAIL` = surviving)
+plus a per-package summary `The mutation score is X.YZ`. CPU-bound, single
+core, takes ~10 minutes on a 2024-era laptop for the three packages combined.
+
+**Sandbox note:** `go-mutesting` writes a mutant copy of the source tree to
+`/tmp/go-mutesting/` per run; needs ≥2 GB free disk. Sandboxed CI runners
+are sized for this; constrained dev sandboxes are not.
+
+### DAST baseline (D-004)
+
+**Tool:** [OWASP ZAP `baseline`](https://www.zaproxy.org/docs/docker/baseline-scan/).
+Spiders the running server's URL surface and runs the OWASP-ZAP active+passive
+rule pack. **Baseline** mode skips the destructive active-scan rules; it's safe
+against a non-throwaway environment.
+
+**Target:** the live `deploy/docker-compose.yml` stack on `https://localhost:8443`.
+
+**Acceptance:** zero HIGH/CRITICAL alerts. WARN/INFO alerts get triaged in the
+ZAP report; some are unavoidable (e.g., HSTS preload-list nag is a deployment
+recommendation, not a server defect).
+
+**Local run:**
+
+```
+docker compose -f deploy/docker-compose.yml up -d
+sleep 20  # wait for /ready to flip OK; check `curl --cacert deploy/test/certs/ca.crt https://localhost:8443/ready`
+docker run --rm --network host \
+  -v "$PWD":/zap/wrk \
+  ghcr.io/zaproxy/zaproxy:stable \
+  zap-baseline.py -t https://localhost:8443 \
+  -r zap-report.html -J zap-report.json
+docker compose -f deploy/docker-compose.yml down
+```
+
+The HTML report opens in a browser; the JSON is machine-readable for triage.
+
+### TLS audit (D-005)
+
+**Tool:** [`testssl.sh`](https://testssl.sh/). Probes the TLS handshake and
+each enabled cipher suite; reports protocol-version weaknesses, cipher
+weaknesses, certificate-chain issues, and known CVE patterns (Heartbleed,
+ROBOT, BEAST, etc.).
+
+**Target:** the live stack on `https://localhost:8443`.
+
+**Acceptance:** zero HIGH/CRITICAL findings. certctl pins
+`tls.Config.MinVersion = tls.VersionTLS13` (`cmd/server/tls.go`), so anything
+that surfaces is either (a) a real defect, (b) a testssl false positive, or
+(c) a deployment-config issue worth documenting in the operator runbook.
+
+**Local run:**
+
+```
+docker compose -f deploy/docker-compose.yml up -d
+sleep 20
+docker run --rm --network host \
+  -v "$PWD":/data \
+  drwetter/testssl.sh:latest \
+  --jsonfile /data/testssl.json https://localhost:8443
+docker compose -f deploy/docker-compose.yml down
+
+# Filter to actionable severities
+jq '[.scanResult[] | select(.severity == "HIGH" or .severity == "CRITICAL")]' testssl.json
+```
+
+### Frontend semgrep (D-007)
+
+**Tool:** [`semgrep`](https://semgrep.dev/) with the maintained
+[`p/react-security` ruleset](https://semgrep.dev/p/react-security). Catches
+React-specific XSS / injection patterns: `dangerouslySetInnerHTML` without
+sanitization, `target="_blank"` without `rel="noopener noreferrer"`,
+`href={userInput}`, `eval`, `document.write`, etc.
+
+**Target:** the frontend source tree at `web/src/`.
+
+**Acceptance:** zero findings. Bundle 8 already verified
+`dangerouslySetInnerHTML` count at zero and the `target="_blank"`
+rel-noopener pin via simple grep guards in `ci.yml`; semgrep adds defence
+in depth — it catches escape patterns the greps don't see (e.g.,
+`href={user_input}`, runtime `eval`, `document.write`).
+
+**Local run:**
+
+```
+docker run --rm -v "$PWD":/src returntocorp/semgrep:latest \
+  semgrep --config=p/react-security --json /src/web/src \
+  > semgrep-react.json
+
+# Count findings
+jq '.results | length' semgrep-react.json
+
+# Pretty-print findings
+jq '.results[] | {rule_id: .check_id, path, line: .start.line, message: .extra.message}' semgrep-react.json
+```
+
+If the count is non-zero, every result has a `check_id` (e.g.
+`react.dangerouslySetInnerHTML`) and a `message` describing the escape
+pattern. Triage each: either fix the call site, or — for legitimate edge
+cases — add a `// nosem: <check_id> — <reason>` directive on the
+preceding line.
+
+## Cadence
+
+| Tool                 | Trigger                            | Wall-clock | Owner          |
+|----------------------|------------------------------------|------------|----------------|
+| go-mutesting         | daily deep-scan + manual dispatch  | ~10 min    | maintainers    |
+| ZAP baseline (DAST)  | daily deep-scan + manual dispatch  | ~5 min     | maintainers    |
+| testssl.sh           | daily deep-scan + manual dispatch  | ~3 min     | maintainers    |
+| semgrep react        | daily deep-scan + manual dispatch  | ~1 min     | maintainers    |
+| `make verify`        | every commit (pre-push)            | ~1 min     | every developer |
+| ci.yml fast gates    | every push/PR                      | ~3 min     | every developer |
+
+Re-run any of the deep-scan tools locally when:
+
+- A CI receipt surfaces an unexpected finding and you want to bisect against
+  a local change before pushing.
+- You're cutting a release tag and want belt-and-suspenders evidence beyond
+  the most recent scheduled scan.
+- You're adding a new feature in the relevant surface (crypto code →
+  re-run mutation testing; new HTTP handler → re-run schemathesis + ZAP;
+  new TLS-config knob → re-run testssl).
+
+## Related docs
+
+- [`docs/security.md`](security.md) — security posture, per-finding closure log.
+- [`docs/testing-guide.md`](testing-guide.md) — manual end-to-end QA playbook.
+- [`.github/workflows/ci.yml`](../.github/workflows/ci.yml) — per-PR fast gates.
+- [`.github/workflows/security-deep-scan.yml`](../.github/workflows/security-deep-scan.yml) — daily deep-scan gates.
+- [`scripts/install-security-tools.sh`](../scripts/install-security-tools.sh) — Go-host-installed tools (the docker-based tools are not in this script).

docs/tls.md

@@ -0,0 +1,214 @@
+# TLS on the Control Plane
+
+certctl's control plane is HTTPS-only as of v2.2. There is no plaintext `http://` listener, no `auto` mode, no dual-listener bridge, no TLS 1.2 escape hatch. The server refuses to start without a cert+key pair, the agent/CLI/MCP clients reject `http://` URLs at startup, and the Helm chart refuses to render without either an operator-supplied Secret or a cert-manager Certificate CR.
+
+This doc covers four cert provisioning patterns, SIGHUP-based cert rotation, and the client-side CA-trust configuration agents and the CLI need to talk to the server. If you are upgrading from a pre-HTTPS release and want the step-by-step cutover procedure, read [`upgrade-to-tls.md`](upgrade-to-tls.md) first and come back here for reference.
+
+## What you get
+
+The server binds TLS 1.3 only with an explicit curve preference of `[X25519, P-256]`. TLS 1.3 cipher suites are non-negotiable (all three mandatory suites — AES-128-GCM-SHA256, AES-256-GCM-SHA384, CHACHA20-POLY1305-SHA256 — are always offered), so there is no `CipherSuites` knob to misconfigure. No TLS 1.2 fallback is available.
+
+Two env vars are required on the server:
+
+- `CERTCTL_SERVER_TLS_CERT_PATH` — filesystem path to the PEM-encoded server certificate
+- `CERTCTL_SERVER_TLS_KEY_PATH` — filesystem path to the PEM-encoded private key that signs the cert
+
+Both paths are read during a fail-loud preflight in `cmd/server/main.go` (see `preflightServerTLS` in `cmd/server/tls.go`). If either is unset, unreadable, or the cert+key pair does not round-trip through `tls.LoadX509KeyPair`, the process refuses to start and emits a diagnostic pointing back at this doc. The rationale lives in §3 of the HTTPS-Everywhere milestone: a cert-lifecycle product should not silently bind plaintext.
+
+## Pattern 1 — Self-signed bootstrap for docker-compose demos
+
+This is the default for the `deploy/docker-compose.yml` stack. It exists so `docker compose up -d --build` just works on a laptop without the operator standing up a CA first. It is not appropriate for any non-demo environment.
+
+An init container named `certctl-tls-init` runs once before the server starts. It uses the `alpine/openssl` image and generates an ECDSA-P256 self-signed cert (SHA-256 signature):
+
+```
+openssl req -x509 -newkey ec \
+  -pkeyopt ec_paramgen_curve:P-256 \
+  -nodes \
+  -keyout /etc/certctl/tls/server.key \
+  -out   /etc/certctl/tls/server.crt \
+  -days 3650 \
+  -subj "/CN=certctl-server" \
+  -addext "subjectAltName=DNS:certctl-server,DNS:localhost,IP:127.0.0.1,IP:::1"
+```
+
+**Why ECDSA-P256 and not ed25519.** The pre-v2.0.48 demo bootstrap used ed25519 (small keys, fast signatures). Apple's TLS stack — Safari Network Framework and the macOS-bundled LibreSSL 3.3.6 `/usr/bin/curl` — does not advertise ed25519 in the ClientHello `signature_algorithms` extension for server certs, so an ed25519 server cert was rejected at handshake with `tls: peer doesn't support any of the certificate's signature algorithms` on the server side (and the generic TLS handshake error on the client side). Homebrew OpenSSL 3.x, Chrome, Firefox, and Linux curl all accepted ed25519 — Apple was the outlier. ECDSA-P256 with SHA-256 is universally supported, so the demo bootstrap uses it by default. To pick up the new algorithm on an existing demo install, tear the volume down and rebuild: `docker compose -f deploy/docker-compose.yml down -v && docker compose -f deploy/docker-compose.yml up -d --build`. **Helm and operator-supplied-Secret users (Patterns 2 and 3) are unaffected** — they bring their own cert, and `cmd/server/tls.go` is algorithm-agnostic (TLS 1.3 with curve preference `[X25519, P-256]` for key exchange — no constraint on the server cert's signature algorithm).
+
+The cert, its matching key, and a copy of the cert published as `ca.crt` land in a named volume (`certs`) mounted at `/etc/certctl/tls/` in the server container (read-only) and the agent container (read-only). The bootstrap is idempotent — if `server.crt`, `server.key`, and `ca.crt` are already present on the volume, the init container logs `TLS cert already present at …` and exits cleanly.
+
+Single-cert design. CN is `certctl-server` to match the Docker-network hostname. The SAN list is `[certctl-server, localhost, 127.0.0.1, ::1]`, which covers both container-internal agent→server traffic and operator browser/curl access to `https://localhost:8443`. There is no separate intermediate/root chain — the server cert and the CA bundle are the same PEM. This is the whole point of a demo bootstrap.
+
+To force regeneration (rotate the demo cert), tear the volume down: `docker compose down -v`. The next `up` re-runs the init container.
+
+The server's Docker healthcheck and the agent both verify against `/etc/certctl/tls/ca.crt`; no `-k` / `InsecureSkipVerify` anywhere in the default stack.
+
+## Pattern 2 — Operator-supplied `kubernetes.io/tls` Secret (Helm)
+
+This is the default path for Helm installs. The operator provisions a Secret of type `kubernetes.io/tls` holding `tls.crt` + `tls.key` (and optionally `ca.crt` for mounting a CA bundle to clients in the same cluster) from whatever source they already trust — their internal CA, a manually-issued cert, step-ca, AWS ACM PCA exported to PEM, or the output of the self-signed bootstrap pattern above copied into a cluster Secret.
+
+```
+kubectl create secret tls certctl-server-tls \
+  --cert=server.crt \
+  --key=server.key \
+  --namespace certctl
+```
+
+Then:
+
+```
+helm install certctl deploy/helm/certctl \
+  --namespace certctl \
+  --set server.tls.existingSecret=certctl-server-tls
+```
+
+The Secret is mounted read-only at `/etc/certctl/tls/` in the server pod. The `CERTCTL_SERVER_TLS_CERT_PATH` and `CERTCTL_SERVER_TLS_KEY_PATH` env vars are wired to `tls.crt` and `tls.key` keys inside that mount. If `ca.crt` is absent from the Secret, clients that need a CA bundle should use `tls.crt` as the bundle (self-signed case) or mount a separate ConfigMap with the root chain (operator-CA case).
+
+If the operator sets neither `server.tls.existingSecret` nor `server.tls.certManager.enabled=true`, `helm template` / `helm install` fails at render-time with a diagnostic pointing at this doc. The guard is implemented in `deploy/helm/certctl/templates/_helpers.tpl` under the `certctl.tls.required` helper. This is deliberate: the HTTPS-only server would crash-loop on an empty path, so we fail earlier at Helm-render time.
+
+## Pattern 3 — cert-manager `Certificate` CR (Helm, opt-in)
+
+For clusters that already run cert-manager, the chart can provision a `Certificate` CR that writes into the Secret the server pod reads from. This is opt-in — the default is `server.tls.certManager.enabled: false` — because not every cluster has cert-manager installed, and we refuse to ship a chart that silently depends on an external controller.
+
+```
+helm install certctl deploy/helm/certctl \
+  --namespace certctl \
+  --set server.tls.certManager.enabled=true \
+  --set server.tls.certManager.issuerRef.name=my-cluster-issuer \
+  --set server.tls.certManager.issuerRef.kind=ClusterIssuer
+```
+
+The rendered `Certificate` (see `deploy/helm/certctl/templates/server-certificate.yaml`) writes `tls.crt` + `tls.key` + `ca.crt` into the Secret named by `server.tls.certManager.secretName` (defaults to `<fullname>-tls`). The server pod reads from that same Secret; the agent DaemonSet mounts the same Secret as its CA bundle source.
+
+cert-manager handles rotation. certctl-server handles in-place reload — see the SIGHUP section below.
+
+The chart enforces that if `server.tls.certManager.enabled=true`, `server.tls.certManager.issuerRef.name` must also be set. An empty `issuerRef.name` makes `helm template` fail with a diagnostic naming the missing flag.
+
+## Pattern 4 — Manually-issued from an internal CA
+
+For operators running neither Helm nor docker-compose (bare-metal / custom orchestration), the server just needs two files on disk pointed at by `CERTCTL_SERVER_TLS_CERT_PATH` and `CERTCTL_SERVER_TLS_KEY_PATH`. Issue the cert from your internal CA with:
+
+- CN matching the hostname your agents and operators use to dial the server (e.g., `certctl.prod.example.com`)
+- SAN list covering every hostname and IP that appears in `CERTCTL_SERVER_URL` values across your agent fleet
+- Key usage: digital signature + key encipherment
+- Extended key usage: server auth
+
+Store the key with mode `0600` and owner matching the UID the server runs as (`1000` in our shipped Dockerfile). The server process reads both files during `preflightServerTLS` at startup and again on every SIGHUP.
+
+The full CA chain that signed the server cert should be distributed to agents, CLI operators, and MCP clients as their `CERTCTL_SERVER_CA_BUNDLE_PATH` — see the client section below.
+
+## SIGHUP cert rotation
+
+The server wraps its cert+key pair in a `*certHolder` (see `cmd/server/tls.go`) that guards the loaded `*tls.Certificate` under a `sync.Mutex`. The `*tls.Config` wires `GetCertificate` to the holder, so every new inbound TLS handshake reads whatever cert the holder currently has.
+
+Send `SIGHUP` to the server PID and the holder re-reads both files from disk. On success, the next new connection uses the new cert; in-flight requests finish on the previous cert. A log line goes out:
+
+```
+TLS cert reloaded via SIGHUP cert_path=/etc/certctl/tls/server.crt key_path=/etc/certctl/tls/server.key
+```
+
+On failure (missing file, malformed PEM, key does not sign cert), the old cert is retained and an error logs:
+
+```
+TLS cert reload failed; continuing with previous cert cert_path=… key_path=… error=…
+```
+
+This is deliberately fail-safe on reload (as opposed to fail-loud on startup). A cert-manager renewal race, a partially-copied file, a typo in a rotation script — none of those should crash a running server and drop every agent connection. The operator sees the error in logs, fixes the underlying issue, and sends another `SIGHUP`.
+
+Pair with cert-manager, certbot `--post-hook`, or any rotation tool that can fire a signal. For docker-compose, `docker compose kill -s HUP certctl-server` works. For Kubernetes, reload is typically handled by cert-manager updating the Secret and the mounted file changing on the next kubelet sync — no explicit SIGHUP needed if the volume mount is `subPath`-free.
+
+Startup is a different story. If the cert is missing or malformed at process start, the server exits non-zero rather than binding plaintext or attempting a retry loop. That's the HTTPS-only contract.
+
+## Client-side TLS: agents, CLI, MCP
+
+Everything that talks to the server enforces HTTPS on the URL.
+
+### Agent
+
+`CERTCTL_SERVER_URL` must be `https://…`. `http://`, bare hostnames, `ftp://`, `ws://`, and empty strings are rejected at startup by `validateHTTPSScheme` in `cmd/agent/main.go` with a diagnostic pointing at `upgrade-to-tls.md`. There is no warning-and-proceed path.
+
+Two additional env vars control how the agent verifies the server cert:
+
+- `CERTCTL_SERVER_CA_BUNDLE_PATH` — filesystem path to a PEM-encoded CA bundle that signed the server cert. Loaded into `*tls.Config.RootCAs` on the agent's HTTP client. If unset, the agent falls back to the OS system trust store.
+- `CERTCTL_SERVER_TLS_INSECURE_SKIP_VERIFY` — defaults to `false`. Setting it to `true` skips verification entirely. **Dev-only escape hatch.** The agent logs a prominent warning at startup (`TLS certificate verification is disabled … never enable this in production`). Use this only when dialing a demo server whose cert you haven't bothered to mount into the agent container.
+
+Equivalent CLI flags: `--ca-bundle <path>` and `--insecure-skip-verify`.
+
+If both the CA bundle and `InsecureSkipVerify=true` are set, `InsecureSkipVerify` wins — it's the whole point of the flag. Don't do this in production.
+
+### CLI (`certctl-cli`)
+
+Same contract as the agent:
+
+- `CERTCTL_SERVER_URL` defaults to `https://` scheme; `http://` rejected at startup
+- `--ca-bundle <path>` flag or `CERTCTL_SERVER_CA_BUNDLE_PATH` env var — CA bundle for server cert verification
+- `--insecure` flag or `CERTCTL_SERVER_TLS_INSECURE_SKIP_VERIFY=true` — skip verification (dev only)
+- Error diagnostic on empty URL explicitly mentions both `--server` and `CERTCTL_SERVER_URL` so operators see the right knob to turn
+
+The CLI shares the URL-scheme validation with the agent; the test pins in `cmd/cli/main_test.go:TestValidateHTTPSScheme` cover the full rejection matrix.
+
+### MCP server (`certctl-mcp-server`)
+
+Same three controls as CLI, env-var-driven only (no flags — MCP runs as a stdio subprocess and inherits env from the launching LLM client):
+
+- `CERTCTL_SERVER_URL` must start with `https://`
+- `CERTCTL_SERVER_CA_BUNDLE_PATH` optional CA bundle
+- `CERTCTL_SERVER_TLS_INSECURE_SKIP_VERIFY` optional skip
+
+Claude Desktop / other MCP client configs should set all three in the tool's env block.
+
+## Troubleshooting: fail-loud preflight errors
+
+Every preflight failure message ends with `(see docs/tls.md)` so this doc is the first hit when an operator searches. Common failures:
+
+**`CERTCTL_SERVER_TLS_CERT_PATH is empty: HTTPS-only control plane refuses to start`**
+Set the env var. For docker-compose this is already set to `/etc/certctl/tls/server.crt` in the shipped compose file — if you're seeing this, check the `certctl-tls-init` service logs to see why the init container didn't populate the volume. For Helm, check that `server.tls.existingSecret` or `server.tls.certManager.enabled=true` is set.
+
+**`TLS cert file "…" unreadable: …`**
+The cert path is set but `os.Stat` failed. Check filesystem permissions — the server runs as UID 1000 in our shipped Dockerfile; the cert needs to be readable by that UID. Typos in the path also land here.
+
+**`TLS cert/key pair invalid (cert="…" key="…"): …`**
+Both files exist but `tls.LoadX509KeyPair` refused them. Typical causes: the private key does not sign the certificate, the key is encrypted with a passphrase (not supported — remove the passphrase with `openssl pkey` before mounting), or one of the two is DER-encoded instead of PEM. Re-issue the pair from the same CA call and re-mount.
+
+**Client side: `tls: failed to verify certificate: x509: certificate signed by unknown authority`**
+The client did not trust the CA that signed the server cert. Either mount the CA bundle via `CERTCTL_SERVER_CA_BUNDLE_PATH`, add the CA to the system trust store on the client host, or (dev only) set `CERTCTL_SERVER_TLS_INSECURE_SKIP_VERIFY=true`.
+
+**Client side: `tls: first record does not look like a TLS handshake`**
+The client is speaking plaintext HTTP to an HTTPS server (or vice-versa). Check that `CERTCTL_SERVER_URL` starts with `https://`. If you are upgrading from a pre-v2.2 release and your agents are old, they will surface this error until you roll the DaemonSet — see [`upgrade-to-tls.md`](upgrade-to-tls.md).
+
+## InsecureSkipVerify justifications (Audit L-001)
+
+`crypto/tls.Config.InsecureSkipVerify` short-circuits standard certificate
+chain validation. Each production use site below has a justification —
+the shape is "this code path is fundamentally pre-trust or
+trust-from-context, and chain validation in the stdlib path is not the
+right tool". Test-only sites are not enumerated here.
+
+The CI grep guard `Forbidden bare InsecureSkipVerify regression guard
+(L-001)` in `.github/workflows/ci.yml` fails the build if any new
+`InsecureSkipVerify: true` lands in a non-test file without a
+`//nolint:gosec` comment carrying a justification — adding a new entry
+to this table is the right way to extend the surface.
+
+| Site (file:line) | Trigger | Justification |
+|---|---|---|
+| `cmd/agent/main.go:59,125,136,1259,1262` | `--insecure-skip-verify` CLI flag | Dev escape hatch; docs/tls.md and the agent install script direct operators to use a real CA bundle in production. The server emits a startup WARN when set. |
+| `cmd/agent/verify.go:70,78` | TLS deployment verification probe | The agent is verifying that its own freshly-deployed cert is being served. The chain may be self-signed or signed by an upstream the agent host doesn't trust; what matters is the leaf-cert match against what the agent just deployed. The verifier compares the served leaf bytes to the expected leaf, not the chain. |
+| `internal/tlsprobe/probe.go:33,47,54` | Network scanner / discovery probe | Discovery's job is to find every cert on the network, including expired, self-signed, and not-yet-deployed certs. Validating the chain would silently skip the broken-cert results that are precisely what operators want to know about. |
+| `internal/mcp/client.go:35` | MCP CLI `--insecure` flag | Dev escape hatch for local-only MCP testing against a self-signed control plane. |
+| `internal/cli/client.go:39` | `certctl --insecure` flag | Same shape as the agent flag — local dev only. |
+| `internal/connector/target/f5/f5.go:128` | F5 BIG-IP iControl REST | F5 default install ships with a self-signed cert; operators who haven't replaced it use `config.Insecure`. The connector logs this on every dial and the operator-facing config docs this. |
+| `internal/connector/issuer/acme/acme.go:146` | Pebble (ACME test server) | Hard-coded for tests that drive against Pebble locally. Pebble issues self-signed; verifying the chain would defeat the purpose. |
+| `internal/service/network_scan.go:460` | Network scanner probe | Same rationale as `tlsprobe/probe.go` above — discovery surfaces broken certs by design. |
+
+**What is NOT covered by this list:** `*_test.go` files use
+`InsecureSkipVerify` freely against `httptest.Server` instances; that's a
+test-fixture pattern, not a production trust decision. The grep guard
+ignores `_test.go`.
+
+## Related docs
+
+- [`upgrade-to-tls.md`](upgrade-to-tls.md) — one-step cutover from pre-HTTPS releases
+- [`quickstart.md`](quickstart.md) — docker-compose walkthrough with HTTPS examples
+- [`test-env.md`](test-env.md) — integration test environment (also HTTPS-only)
+- [`security.md`](security.md) — overall security posture, OCSP Must-Staple guidance, encryption-at-rest spec
+- Milestone spec: `prompts/https-everywhere-milestone.md` (authoritative source for locked decisions)

docs/upgrade-to-tls.md

@@ -0,0 +1,194 @@
+# Upgrading to HTTPS-Everywhere (v2.2)
+
+certctl's control plane is HTTPS-only as of v2.2. There is no `http` mode, no `auto` mode, no dual-listener bind, no N-release migration window. The cutover is a single step. Out-of-date agents that still point at `http://…` fail at the TCP/TLS handshake layer on first connect after the upgrade and stay `Offline` in the dashboard until their env block is updated and the fleet is rolled.
+
+This doc walks operators through the cutover for the two shipped deployment topologies — docker-compose and Helm — and documents the failure modes and rollback posture explicitly.
+
+For the deep-dive on cert provisioning patterns, SIGHUP cert reload, and client-side CA-trust configuration, read [`tls.md`](tls.md). This doc is the narrow "how do I upgrade" procedure.
+
+## Preconditions
+
+Before you start, confirm:
+
+- **Shell access** to the server host and every agent host. The cutover requires you to restart the server and update every agent's env block.
+- **A cert+key source** for the server. Pick one:
+  - An internal CA that can issue a server cert (CN + SAN list covering every hostname / IP agents dial).
+  - A `cert-manager` install in the target Kubernetes cluster, plus a `ClusterIssuer` or `Issuer` you're willing to reference.
+  - Willingness to use the self-signed bootstrap that the shipped `deploy/docker-compose.yml` generates automatically. This is the right choice for dev and demo; it is the wrong choice for production.
+- **A maintenance window.** Out-of-date agents break at the TLS handshake and stay offline until rolled. Schedule the upgrade so the agent fleet can be updated in the same window as the server.
+- **Backups.** This is a one-way door (see the Rollback section below). Snapshot your PostgreSQL database before `docker compose down` or `helm upgrade`.
+
+There is no schema migration tied to this release; the only at-rest state that changes is the `certs` named volume (docker-compose) or the `tls.crt`/`tls.key` Secret (Helm).
+
+## Procedure — docker-compose operators
+
+The shipped `deploy/docker-compose.yml` includes a `certctl-tls-init` init container that self-signs an ECDSA-P256 (SHA-256 signature) cert on first boot and drops `server.crt`, `server.key`, and `ca.crt` into a named volume mounted read-only at `/etc/certctl/tls/` on the server and agent containers. No manual cert provisioning is required for the default stack. (Pre-v2.0.48 this was an ed25519 cert; see [`tls.md`](tls.md) Pattern 1 for the rationale and the `down -v && up --build` migration note.)
+
+1. **Pull the HTTPS-everywhere release.** From the repo root:
+
+   ```
+   git pull
+   ```
+
+   Confirm you're on a tag or `master` that contains the `certctl-tls-init` service in `deploy/docker-compose.yml`. Grep for it: `grep certctl-tls-init deploy/docker-compose.yml` should hit.
+
+2. **Stop the old plaintext cluster.**
+
+   ```
+   docker compose -f deploy/docker-compose.yml down
+   ```
+
+   Do not pass `-v`; keeping the PostgreSQL volume preserves your cert inventory, audit trail, and job history across the upgrade.
+
+3. **Bring the cluster back up with the HTTPS build.**
+
+   ```
+   docker compose -f deploy/docker-compose.yml up -d --build
+   ```
+
+   The `certctl-tls-init` service runs once, generates the self-signed cert into the `certs` volume, and exits with code 0. The server container waits for `certctl-tls-init` via `depends_on: { condition: service_completed_successfully }` and only starts once the cert material is on disk. The server's Docker healthcheck now uses `curl --cacert /etc/certctl/tls/ca.crt -f https://localhost:8443/health`, so the container only becomes healthy once the HTTPS listener is up and serving the bundled cert correctly.
+
+4. **Verify the HTTPS endpoint from the host.**
+
+   ```
+   curl --cacert $(docker compose -f deploy/docker-compose.yml exec -T certctl-server cat /etc/certctl/tls/ca.crt) https://localhost:8443/health
+   ```
+
+   Expect `{"status":"ok"}` with HTTP 200. If you get a TLS verification error, the CA bundle wasn't read correctly — re-run the `exec -T` command and pipe the output directly into `--cacert @-` or save it to a local file first. If you get `connection refused`, the server never finished startup — check `docker compose logs certctl-server` for a fail-loud preflight diagnostic pointing at `docs/tls.md`.
+
+5. **Confirm the bundled agent reconnects.** Agents inside the compose stack pick up the new URL (`CERTCTL_SERVER_URL=https://certctl-server:8443`) and the bundled CA (`CERTCTL_SERVER_CA_BUNDLE_PATH=/etc/certctl/tls/ca.crt`) from their env block automatically — no per-agent change needed. Tail the agent log:
+
+   ```
+   docker compose -f deploy/docker-compose.yml logs -f certctl-agent
+   ```
+
+   You should see `heartbeat sent` within 30 seconds. In the dashboard (`https://localhost:8443`), the agent should show as `Online`.
+
+**External agents** running outside the compose network (e.g., the `install-agent.sh`-installed systemd service on a separate host) need their env block updated manually before the cutover — see the Agent env block section below.
+
+## Procedure — Helm operators
+
+The Helm chart does not self-sign. It refuses to render (`helm template` exits non-zero) unless you configure one of two cert sources: an operator-supplied Secret, or a cert-manager `Certificate` CR. See [`tls.md`](tls.md) for the full pattern catalog.
+
+1. **Provision cert material.** Pick one of:
+
+   - **Operator-supplied Secret.** Issue a cert from your internal CA (or any other source) and load it into a `kubernetes.io/tls` Secret in the certctl namespace:
+
+     ```
+     kubectl create secret tls certctl-server-tls \
+       --cert=server.crt --key=server.key \
+       --namespace certctl
+     ```
+
+   - **cert-manager.** Set `server.tls.certManager.enabled=true` on the upgrade and reference an existing `ClusterIssuer` or `Issuer`:
+
+     ```
+     --set server.tls.certManager.enabled=true
+     --set server.tls.certManager.issuerRef.name=my-cluster-issuer
+     --set server.tls.certManager.issuerRef.kind=ClusterIssuer
+     ```
+
+2. **Upgrade the release.**
+
+   ```
+   helm upgrade certctl deploy/helm/certctl \
+     --namespace certctl \
+     --set server.tls.existingSecret=certctl-server-tls
+   ```
+
+   (Or the `certManager` variant.) If you omit both `server.tls.existingSecret` and `server.tls.certManager.enabled`, the chart fails at render time with a diagnostic pointing at `docs/tls.md`. That guard exists precisely so you catch the missing config at `helm upgrade` time, not at pod-crash-loop time.
+
+3. **Verify the HTTPS endpoint from inside the cluster.** Port-forward and curl with the CA bundle:
+
+   ```
+   kubectl port-forward -n certctl svc/certctl-server 8443:8443 &
+   kubectl get secret -n certctl certctl-server-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/certctl-ca.crt
+   curl --cacert /tmp/certctl-ca.crt https://localhost:8443/health
+   ```
+
+   Expect `{"status":"ok"}`. If the Secret does not contain a `ca.crt` key (operator-supplied Secrets often don't), use `tls.crt` as the bundle instead — for a self-signed cert the two files are identical, and for a cert chained to an internal CA you should separately distribute the root CA bundle via ConfigMap or mounted file.
+
+4. **Update every agent manifest.** Agents outside this Helm release (or in a separately-managed DaemonSet) need their env block updated:
+
+   ```
+   - name: CERTCTL_SERVER_URL
+     value: "https://certctl-server.certctl.svc.cluster.local:8443"
+   - name: CERTCTL_SERVER_CA_BUNDLE_PATH
+     value: "/etc/certctl/tls/ca.crt"
+   ```
+
+   Mount the server's Secret (or a separate CA-bundle Secret / ConfigMap) at `/etc/certctl/tls/` as a read-only volume. If you bundle the agent via the shipped Helm chart's DaemonSet, the wiring is already done — set `agent.enabled=true` and the chart mounts the same Secret.
+
+5. **Roll the agent DaemonSet.**
+
+   ```
+   kubectl rollout restart ds/certctl-agent -n certctl
+   kubectl rollout status ds/certctl-agent -n certctl
+   ```
+
+   Every agent pod restarts with the new URL + CA bundle and reconnects on HTTPS. The dashboard shows agents flip from `Offline` to `Online` as pods finish rolling.
+
+## Agent env block — external hosts
+
+Agents installed on bare-metal or VM hosts via `install-agent.sh` (systemd on Linux, launchd on macOS) read config from `/etc/certctl/agent.env` (Linux) or `~/Library/Application Support/certctl/agent.env` (macOS). On cutover, append or update:
+
+```
+CERTCTL_SERVER_URL=https://certctl.example.com:8443
+CERTCTL_SERVER_CA_BUNDLE_PATH=/etc/certctl/tls/ca.crt
+# CERTCTL_SERVER_TLS_INSECURE_SKIP_VERIFY=false    # Dev only. Never set to true in production.
+```
+
+Distribute the CA bundle (the same `ca.crt` the server holds, or the root chain if you issued the server cert from an intermediate) to every agent host. The path under `CERTCTL_SERVER_CA_BUNDLE_PATH` must be readable by the UID the agent service runs as.
+
+Restart the service after editing:
+
+- Linux: `systemctl restart certctl-agent`
+- macOS: `launchctl kickstart -k system/com.certctl.agent`
+
+The agent refuses to start on an `http://` URL and exits with a pre-flight diagnostic that names this doc. That rejection happens before any network call — no spurious half-connected state.
+
+## Failure mode
+
+Out-of-date agents still configured with `CERTCTL_SERVER_URL=http://…` fail on first reconnect after the cutover. The failure surfaces as one of:
+
+- `dial tcp …: connect: connection refused` — the server is no longer listening on a plaintext port. The new release binds only a TLS listener; attempting a plaintext `connect()` gets refused at the kernel level because nothing holds the socket.
+- `tls: first record does not look like a TLS handshake` — depending on timing and proxy layers (e.g., a load balancer that accepts the TCP connection before forwarding), the client may negotiate TCP, send an HTTP request line, and have the server's TLS stack reject it.
+
+Agents in this state surface as `Offline` in the dashboard. They stay offline until their env block is updated and the service restarts. There is no graceful 400-with-migration-URL response because there is no HTTP listener to serve one from — the entire plaintext call path is removed by design.
+
+If you see an unexpected agent stay `Offline` past the cutover window, SSH to the host and check the agent log. On a systemd host:
+
+```
+journalctl -u certctl-agent -n 100
+```
+
+Look for `URL scheme "http" is not supported: HTTPS-only control plane refuses to start (see docs/upgrade-to-tls.md)`. That's the pre-flight rejection. Update `CERTCTL_SERVER_URL`, restart the service, and the agent reconnects.
+
+## Rollback
+
+**There is no rollback window.** The upgrade is a one-way door. The rationale lives in §3.7 of `prompts/https-everywhere-milestone.md`: a cert-lifecycle product that bridges back to plaintext after committing to HTTPS is advertising that its own security posture is negotiable.
+
+If you need to revert, you have two options:
+
+1. **Stay on the pre-HTTPS release.** Do not upgrade until you are ready to run HTTPS on the control plane. Pin your `docker-compose.yml` or `helm upgrade` command to the last pre-v2.2 tag.
+2. **Rollback the release.** `helm rollback certctl <previous-revision>` or `git checkout <previous-tag> && docker compose up -d --build`. This rolls back the server, the compose topology, and the Helm chart in lockstep. Your PostgreSQL volume — cert inventory, audit trail, jobs — survives the rollback; nothing in this milestone changes the database schema.
+
+Option 2 drops you back to the plaintext world. It should be treated as an emergency measure, not a supported migration path.
+
+## After the cutover
+
+Once every agent is `Online`, confirm a few invariants:
+
+- `curl -sS -o /dev/null -w "%{http_code}\n" http://localhost:8443/health` returns `000` with `Connection refused` (no HTTP listener). Plaintext is gone.
+- `openssl s_client -connect localhost:8443 -tls1_2 </dev/null` fails the handshake. TLS 1.2 is rejected.
+- `openssl s_client -connect localhost:8443 -tls1_3 </dev/null` succeeds and prints the server's SAN list. TLS 1.3 is live.
+- A cert rotation test: overwrite the server cert on disk, `kill -HUP` the server PID, confirm the new cert serves on the next `openssl s_client -connect … -showcerts` without a process restart. See the SIGHUP section in [`tls.md`](tls.md).
+
+Update your runbooks. Every `http://certctl.example.com` URL in internal documentation, monitoring config, and on-call playbooks should become `https://certctl.example.com` plus a CA-trust note.
+
+## Related docs
+
+- [`tls.md`](tls.md) — cert provisioning patterns, SIGHUP rotation, troubleshooting
+- [`quickstart.md`](quickstart.md) — docker-compose walkthrough (post-HTTPS)
+- [`test-env.md`](test-env.md) — integration test environment (HTTPS-only)
+- Milestone spec: `prompts/https-everywhere-milestone.md`

docs/upgrade-to-v2-jwt-removal.md

@@ -0,0 +1,155 @@
+# Upgrading past G-1 — `CERTCTL_AUTH_TYPE=jwt` removal
+
+If your certctl deployment currently sets `CERTCTL_AUTH_TYPE=jwt` (or `server.auth.type=jwt` in Helm), the next certctl upgrade will fail-fast at startup with a dedicated diagnostic. This guide explains why, what to switch to, and how to keep JWT/OIDC at your edge.
+
+For everyone else — operators running `api-key` or `none` — this upgrade is a no-op. Skip to [`upgrade-to-tls.md`](upgrade-to-tls.md) for the v2.2 HTTPS-everywhere migration if you haven't done that one yet.
+
+## Why we removed it
+
+Pre-G-1, the config validator at `internal/config/config.go` accepted three values for `CERTCTL_AUTH_TYPE`: `api-key`, `jwt`, and `none`. The startup log line at `cmd/server/main.go` faithfully echoed `"authentication enabled" "type"="jwt"` when an operator picked `jwt`. Reasonable people read that and concluded JWT auth was on.
+
+It wasn't. Grep `internal/ cmd/` for `NewJWT`, `JWTMiddleware`, or `jwt.Parse` — pre-G-1, there were zero matches in production code. The auth-middleware wiring at `cmd/server/main.go:653` unconditionally called `middleware.NewAuthWithNamedKeys(namedKeys)` regardless of `cfg.Auth.Type`. So `CERTCTL_AUTH_TYPE=jwt` just routed every request through the api-key bearer middleware, comparing the incoming `Authorization: Bearer <something>` against whatever string the operator put in `CERTCTL_AUTH_SECRET`. Real JWT clients got 401 (the api-key middleware saw the JWT string as a literal token and compared bytes). Operators who treated `CERTCTL_AUTH_SECRET` as a JWT signing secret (and therefore handled it less carefully than an api-key) handed an attacker an api-key. Silent auth downgrade — a security finding masquerading as a config option.
+
+We chose to remove the option rather than implement JWT middleware. Implementing real JWT/OIDC requires jwks vs static-secret rotation, claim mapping (which claim is the actor / the admin flag?), expiry enforcement, audience and issuer validation, key rollover semantics, and regression coverage at the same depth as the existing api-key path. That's a feature, not a fix. The audit-recommended structural fix — and the one that actually closes the hazard — is to fail loudly instead of silently downgrading.
+
+## What changes at startup
+
+Post-G-1, a binary started with `CERTCTL_AUTH_TYPE=jwt` exits non-zero before opening the listener:
+
+```
+Failed to load configuration: CERTCTL_AUTH_TYPE=jwt is no longer accepted
+(G-1 silent auth downgrade): no JWT middleware ships with certctl. To use
+JWT/OIDC, run an authenticating gateway (oauth2-proxy / Envoy ext_authz /
+Traefik ForwardAuth / Pomerium) in front of certctl and set
+CERTCTL_AUTH_TYPE=none on the upstream. See docs/architecture.md
+"Authenticating-gateway pattern" and docs/upgrade-to-v2-jwt-removal.md
+for the migration walkthrough
+```
+
+Helm operators get the same shape at `helm install` / `helm upgrade` template time: `server.auth.type=jwt` is rejected by the chart's `certctl.validateAuthType` template helper before any Kubernetes object is rendered.
+
+The CI-side regression guard at `.github/workflows/ci.yml` blocks any future PR that re-introduces `"jwt"` as an auth-type literal in production code or spec.
+
+## Recovery — pick one
+
+### Option A — switch to `api-key` (you weren't actually using JWT)
+
+If your `CERTCTL_AUTH_SECRET` was a single high-entropy token and your clients sent it as `Authorization: Bearer <token>`, you were already using api-key auth — you just had `CERTCTL_AUTH_TYPE` set to the wrong string. Flip it:
+
+```
+# .env (docker-compose)
+CERTCTL_AUTH_TYPE=api-key
+CERTCTL_AUTH_SECRET=<your-existing-token>
+```
+
+```
+# Helm
+helm upgrade <release> deploy/helm/certctl/ \
+  --reuse-values \
+  --set server.auth.type=api-key \
+  --set server.auth.apiKey=<your-existing-token>
+```
+
+No client changes needed — the same Bearer token continues to work. The startup log will now read `"authentication enabled" "type"="api-key"`, which matches what was actually happening pre-G-1.
+
+### Option B — front certctl with an authenticating gateway
+
+If you genuinely need JWT, OIDC, mTLS, or SAML, run an authenticating gateway in front of certctl and let the gateway terminate the federated identity protocol. Configure certctl for `CERTCTL_AUTH_TYPE=none`:
+
+```
+CERTCTL_AUTH_TYPE=none
+```
+
+Then put an oauth2-proxy / Envoy `ext_authz` / Traefik `ForwardAuth` / Pomerium / Authelia (etc.) in the network path between operators and certctl. The gateway validates the identity and proxies the authenticated request to certctl as a same-origin call on a private network.
+
+### Concrete walkthrough — oauth2-proxy + certctl on docker-compose
+
+This is the simplest production-grade JWT/OIDC shape. It assumes you have an OIDC provider (Okta, Auth0, Google Workspace, Keycloak, Dex) and a registered client_id / client_secret.
+
+```yaml
+# deploy/docker-compose.gateway.yml — overlay on the base compose file
+services:
+  oauth2-proxy:
+    image: quay.io/oauth2-proxy/oauth2-proxy:latest
+    command:
+      - --provider=oidc
+      - --oidc-issuer-url=https://<your-issuer>/
+      - --client-id=${OIDC_CLIENT_ID}
+      - --client-secret=${OIDC_CLIENT_SECRET}
+      - --cookie-secret=${OAUTH2_PROXY_COOKIE_SECRET}  # openssl rand -base64 32
+      - --upstream=http://certctl-server:8443  # internal-network only; certctl listens on 8443
+      - --http-address=0.0.0.0:4180
+      - --email-domain=*
+      - --pass-access-token=true
+      - --pass-authorization-header=true
+      - --set-authorization-header=true       # forwards a bearer token upstream
+      - --skip-provider-button=true
+      - --reverse-proxy=true
+    ports:
+      - "443:4180"
+    depends_on:
+      - certctl-server
+    networks:
+      - certctl-network
+
+  certctl-server:
+    environment:
+      CERTCTL_AUTH_TYPE: none   # gateway terminates auth — see docs/upgrade-to-v2-jwt-removal.md
+      # ... rest of the certctl env block unchanged
+```
+
+Operators hit `https://<your-host>/`, get redirected through the OIDC provider, land back at oauth2-proxy with a session cookie, and oauth2-proxy proxies their request to certctl on the internal Docker network. certctl itself is HTTPS-only on `:8443` (TLS 1.3, see [`tls.md`](tls.md)) but operator browsers never see that hop directly. Bind certctl-server's `:8443` to the internal Docker network only — do NOT publish it to the host. The audit trail will record the actor as the gateway-forwarded identity if you also configure a small bearer-token-mapping shim at the gateway (most production deployments do this with a per-user api-key issued by the gateway after OIDC validation).
+
+### Traefik ForwardAuth pattern (Kubernetes)
+
+Same shape, kubernetes-flavored:
+
+```yaml
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: oidc-forward-auth
+spec:
+  forwardAuth:
+    address: http://oauth2-proxy.auth.svc.cluster.local:4180
+    trustForwardHeader: true
+    authResponseHeaders:
+      - X-Auth-Request-User
+      - X-Auth-Request-Email
+      - Authorization
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: certctl
+spec:
+  routes:
+    - match: Host(`certctl.example.com`)
+      kind: Rule
+      middlewares:
+        - name: oidc-forward-auth
+      services:
+        - name: certctl-server
+          port: 8443
+```
+
+The certctl Helm release runs with `server.auth.type=none`. The Traefik IngressRoute attaches `oidc-forward-auth` as a middleware so every request is OIDC-validated by oauth2-proxy before reaching certctl.
+
+### Envoy `ext_authz` pattern
+
+For service-mesh deployments (Istio, Consul, plain Envoy), the `ext_authz` filter calls out to an external authorization service per-request. Same outcome: certctl runs `CERTCTL_AUTH_TYPE=none` and Envoy + your authz service handle JWT/OIDC/mTLS at the mesh edge. See the [Envoy ext_authz docs](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/ext_authz_filter) for the configuration surface.
+
+## Rollback
+
+Pre-G-1 binaries silently accepted `CERTCTL_AUTH_TYPE=jwt` and routed through the api-key middleware. Downgrading the binary is the only mechanical rollback path, and it puts you back into the silent-downgrade state — which is exactly what the G-1 audit finding is about. We don't recommend it. If something is forcing your hand, capture the operational issue you're hitting and open a GitHub issue against the certctl repo with the SHAs involved; the Authenticating-gateway pattern was specifically designed to cover the use cases that historically led operators to set `CERTCTL_AUTH_TYPE=jwt`.
+
+There is no on-disk state that changes with this upgrade — no migrations to roll back, no encrypted config to re-encode, no certificates to re-issue. The change is entirely in the config-validation surface and the helm-chart template guard.
+
+## Cross-references
+
+- [`architecture.md`](architecture.md) — "Authenticating-gateway pattern (JWT, OIDC, mTLS)" section.
+- [`tls.md`](tls.md) — TLS provisioning patterns. The gateway proxying to certctl-server still needs to trust certctl's TLS cert; same patterns apply.
+- [`../deploy/helm/certctl/README.md`](../deploy/helm/certctl/README.md) — Helm-chart-flavored guidance.
+- `internal/config/config.go::ValidAuthTypes` — the single source of truth for what's accepted post-G-1.
+- `internal/repository/postgres/db.go::wrapPingError` — unrelated; pattern for runtime diagnostic of operator misconfiguration.
+- `coverage-gap-audit-2026-04-24-v5/unified-audit.md` — the audit finding (`cat-g-jwt_silent_auth_downgrade`).