certctl/deploy/test/loadtest

certctl Load-Test Harness

Closes the #8 acquisition-readiness blocker from the 2026-05-01 issuer coverage audit (cowork/issuer-coverage-audit-2026-05-01/RESULTS.md). Pre-fix, certctl had zero benchmarks or load tests for any API path; an acquirer evaluating "can certctl handle our 50k-cert fleet at 47-day rotation" had nothing to point at. This harness is the substantiation.

What it measures

A k6 driver hits two scenarios in parallel for 5 minutes at a fixed 50 req/s:

1. POST /api/v1/certificates — the issuance-acceptance hot path. Exercises auth, JSON decode, validation, service.CreateCertificate, and the managed_certificates insert. This is the operator-facing request-acceptance throughput an automation client (Terraform, Crossplane, GitOps controller) would generate.
2. GET /api/v1/certificates?per_page=50 — the most-trafficked read endpoint. Exercises pagination + filtering on the cert list query.

Latency is reported as avg / min / med / p95 / p99 / max. The error floor is < 1% (any 4xx/5xx counts as failed).

What it explicitly does NOT measure

• Issuer connector latency. Connector calls (DigiCert, ACME, Vault, AWS ACM PCA, etc.) happen asynchronously via the renewal scheduler. Their latency is pinned by the certctl_issuance_duration_seconds{issuer_type=...} Prometheus histogram (audit fix #4). Driving them through k6 would load-test someone else's API, which is wrong.
• Full ACME enrollment flow. The audit prompt mentioned ACME-via- pebble; sustained 100/s through a multi-RTT order/challenge/finalize flow requires pebble tuning + crypto helpers k6 doesn't ship out of the box. Deferred to a follow-up.
• Bulk-revoke / bulk-renew. Those are admin endpoints with their own throughput characteristics and warrant a separate scenario.
• Scheduler concurrency under bulk renewal. That's audit fix #9's scope; the harness here measures the API tier, not the scheduler.

Threshold contract

Any future change that breaches one of these fails the test:

Scenario	p95	p99	Error rate
issuance_acceptance	< 2 s	< 5 s	n/a
list_certificates	< 800 ms	< 2 s	n/a
All requests	n/a	n/a	< 1%

These are the regression guards, not the SLO. The SLO is whatever the operator chooses based on the baseline below.

How to run

From the repo root:

make loadtest

This:

1. Builds the certctl image from the repo root Dockerfile.
2. Spins up postgres, the tls-init bootstrap, certctl-server (with CERTCTL_DEMO_SEED=true so the FK rows the script needs exist), and the k6 driver.
3. Runs the k6 script for ~5 minutes 5 seconds (5s stagger between scenarios + 5m duration).
4. Prints the summary text to stdout.
5. Exits non-zero if any threshold was breached.

The full machine-readable summary lands at deploy/test/loadtest/results/summary.json (gitignored). The human-readable summary lands at results/summary.txt.

To run against a server already booted on the host (skip the compose spin-up):

docker run --rm \
  -e CERTCTL_BASE=https://localhost:8443 \
  -e CERTCTL_TOKEN=load-test-token \
  -e K6_INSECURE_SKIP_TLS_VERIFY=true \
  -v "$(pwd)/deploy/test/loadtest/k6.js:/scripts/k6.js:ro" \
  -v "$(pwd)/deploy/test/loadtest/results:/results" \
  --network host \
  grafana/k6:0.54.0 run /scripts/k6.js

Current baseline

The first operator run captures real numbers and commits them into this section. Pre-baseline this section reads "TBD — operator captures on first make loadtest run." The numbers below are the agreed minimum-acceptable thresholds, not the captured baseline; once captured, the baseline goes here as a separate row so future regressions have a diff target.

Scenario	p50	p95	p99	Error rate
issuance_acceptance (threshold)	—	< 2 s	< 5 s	< 1%
issuance_acceptance (baseline)	TBD	TBD	TBD	TBD
list_certificates (threshold)	—	< 800 ms	< 2 s	< 1%
list_certificates (baseline)	TBD	TBD	TBD	TBD

Methodology pinned at baseline capture:

• Hardware: TBD (operator's workstation specs at capture time).
• Postgres: 16-alpine, default config.
• certctl: image built from this repo at the commit referenced below.
• Concurrency: 50 req/s sustained per scenario (100 req/s total).
• Duration: 5 minutes per scenario, 5s stagger.
• Auth: api-key (Bearer token, single key).
• Encryption: CERTCTL_CONFIG_ENCRYPTION_KEY set (32+ bytes).

To recapture the baseline after a tuning commit:

make loadtest
# Inspect deploy/test/loadtest/results/summary.txt for the new numbers.
# Update the table above + the methodology line, commit alongside the
# tuning commit.

Interpreting a regression

If a future PR's make loadtest run pushes p99 above the threshold, the make target exits non-zero and CI fails. The summary.txt prints which threshold breached. Triage:

1. Look at the per-scenario http_req_duration p95 + p99 in summary.json. If only one scenario regressed, the change is localized to that endpoint's hot path.
2. Look at the iteration_duration per scenario — if total iteration time grew but http_req_duration is flat, the latency is in k6 client setup (rare; suggests something changed in the script).
3. Compare against the committed baseline. If p99 was 800 ms at baseline and is now 1.5 s but still under the 5 s threshold, the change is below the regression guard but still meaningful — flag in the PR description.

The harness deliberately does NOT auto-tune. Tuning is informed by the data; tuning commits land separately, each with their own captured baseline update.

CI cadence

Defined in .github/workflows/loadtest.yml:

• workflow_dispatch — manual trigger from the Actions tab. Used before tagging a release or after a meaningful tuning commit.
• Weekly cron — Mondays at 06:00 UTC. Catches gradual regressions from cumulative changes that no single PR triggered.

The workflow does not run per-push. Load tests are minutes long and would not provide useful per-PR signal; per-push pressure goes through make verify (which is fast) and the deploy-vendor-e2e job.

Files in this directory

deploy/test/loadtest/
├── README.md         (this file)
├── docker-compose.yml
├── k6.js             (the load script)
├── certs/            (gitignored — tls-init writes here)
└── results/          (gitignored — k6 writes summary.{json,txt} here)

Audit reference

cowork/issuer-coverage-audit-2026-05-01/RESULTS.md Top-10 fix #8.