certctl/deploy/test/loadtest

certctl Load-Test Harness

Closes the #8 acquisition-readiness blocker from the 2026-05-01 issuer coverage audit (the 2026-05-01 issuer coverage audit). Pre-fix, certctl had zero benchmarks or load tests for any API path; an acquirer evaluating "can certctl handle our 50k-cert fleet at 47-day rotation" had nothing to point at. This harness is the substantiation.

What it measures

A k6 driver hits two scenarios in parallel for 5 minutes at a fixed 50 req/s:

1. POST /api/v1/certificates — the issuance-acceptance hot path. Exercises auth, JSON decode, validation, service.CreateCertificate, and the managed_certificates insert. This is the operator-facing request-acceptance throughput an automation client (Terraform, Crossplane, GitOps controller) would generate.
2. GET /api/v1/certificates?per_page=50 — the most-trafficked read endpoint. Exercises pagination + filtering on the cert list query.

Latency is reported as avg / min / med / p95 / p99 / max. The error floor is < 1% (any 4xx/5xx counts as failed).

What it explicitly does NOT measure

• Issuer connector latency. Connector calls (DigiCert, ACME, Vault, AWS ACM PCA, etc.) happen asynchronously via the renewal scheduler. Their latency is pinned by the certctl_issuance_duration_seconds{issuer_type=...} Prometheus histogram (audit fix #4). Driving them through k6 would load-test someone else's API, which is wrong.
• Full ACME enrollment flow. The audit prompt mentioned ACME-via- pebble; sustained 100/s through a multi-RTT order/challenge/finalize flow requires pebble tuning + crypto helpers k6 doesn't ship out of the box. Deferred to a follow-up.
• Bulk-revoke / bulk-renew. Those are admin endpoints with their own throughput characteristics and warrant a separate scenario.
• Scheduler concurrency under bulk renewal. That's audit fix #9's scope; the harness here measures the API tier, not the scheduler.

Threshold contract

Any future change that breaches one of these fails the test:

Scenario	p95	p99	Error rate
issuance_acceptance	< 2 s	< 5 s	n/a
list_certificates	< 800 ms	< 2 s	n/a
All requests	n/a	n/a	< 1%

These are the regression guards, not the SLO. The SLO is whatever the operator chooses based on the baseline below.

How to run

From the repo root:

make loadtest

This:

1. Builds the certctl image from the repo root Dockerfile.
2. Spins up postgres, the tls-init bootstrap, certctl-server (with CERTCTL_DEMO_SEED=true so the FK rows the script needs exist), and the k6 driver.
3. Runs the k6 script for ~5 minutes 5 seconds (5s stagger between scenarios + 5m duration).
4. Prints the summary text to stdout.
5. Exits non-zero if any threshold was breached.

The full machine-readable summary lands at deploy/test/loadtest/results/summary.json (gitignored). The human-readable summary lands at results/summary.txt.

To run against a server already booted on the host (skip the compose spin-up):

docker run --rm \
  -e CERTCTL_BASE=https://localhost:8443 \
  -e CERTCTL_TOKEN=load-test-token \
  -e K6_INSECURE_SKIP_TLS_VERIFY=true \
  -v "$(pwd)/deploy/test/loadtest/k6.js:/scripts/k6.js:ro" \
  -v "$(pwd)/deploy/test/loadtest/results:/results" \
  --network host \
  grafana/k6:0.54.0 run /scripts/k6.js

Current baseline

The first operator run captures real numbers and commits them into this section. Pre-baseline this section reads "TBD — operator captures on first make loadtest run." The numbers below are the agreed minimum-acceptable thresholds, not the captured baseline; once captured, the baseline goes here as a separate row so future regressions have a diff target.

Scenario	p50	p95	p99	Error rate
issuance_acceptance (threshold)	—	< 2 s	< 5 s	< 1%
issuance_acceptance (baseline)1	2.12 ms	6.19 ms	8.58 ms	0.00%
list_certificates (threshold)	—	< 800 ms	< 2 s	< 1%
list_certificates (baseline)1	2.12 ms	6.19 ms	8.58 ms	0.00%

Methodology of the sandbox-placeholder capture above:

• Hardware: Linux/aarch64 unprivileged sandbox (uid 1019, no root, ~1.2 GiB free disk). NOT canonical hardware.
• Postgres: 14.22 (Ubuntu, native binaries, unix-socket dir /tmp/pg-sock), unix sockets only, port 55432.
• certctl: built from HEAD via go build -o bin/certctl-server ./cmd/server.
• Concurrency: 50 req/s sustained per scenario, both scenarios in parallel (= 100 req/s combined).
• Duration: 10 seconds per scenario (NOT 5 minutes — sandbox bash-call budget is bounded; canonical-hardware run uses 5 minutes).
• TLS: ECDSA-P256 self-signed localhost cert at /tmp/certctl-tls/.
• Auth: api-key, single Bearer token (CERTCTL_AUTH_SECRET=load-test-token).
• Rate limiting: disabled (CERTCTL_RATE_LIMIT_ENABLED=false) — without this, the 100 req/s combined load trips the default token-bucket and drives error rate to ~40%, masking real latency.
• Encryption: CERTCTL_CONFIG_ENCRYPTION_KEY set (32+ bytes).
• Captured: 2026-05-02. Total: 1002 requests, 100.15 req/s sustained, 0 failures, 100% checks passed. Raw summary.json is not committed (gitignored per the existing results/ convention).

Methodology pinned at canonical baseline capture (replace placeholder):

• Hardware: GitHub-hosted ubuntu-latest runner (4 vCPU / 16 GiB / SSD). Run via gh workflow run loadtest.yml; raw summary.json is available for 90 days as a workflow artifact.
• Postgres: 16-alpine in compose, default config.
• certctl: image built from this repo at the commit referenced below.
• Concurrency: 50 req/s sustained per scenario (100 req/s total).
• Duration: 5 minutes per scenario, 5s stagger.
• Auth: api-key (Bearer token, single key).
• Encryption: CERTCTL_CONFIG_ENCRYPTION_KEY set (32+ bytes).

To recapture the baseline after a tuning commit:

make loadtest
# Inspect deploy/test/loadtest/results/summary.txt for the new numbers.
# Update the table above + the methodology line, commit alongside the
# tuning commit.

Interpreting a regression

If a future PR's make loadtest run pushes p99 above the threshold, the make target exits non-zero and CI fails. The summary.txt prints which threshold breached. Triage:

1. Look at the per-scenario http_req_duration p95 + p99 in summary.json. If only one scenario regressed, the change is localized to that endpoint's hot path.
2. Look at the iteration_duration per scenario — if total iteration time grew but http_req_duration is flat, the latency is in k6 client setup (rare; suggests something changed in the script).
3. Compare against the committed baseline. If p99 was 800 ms at baseline and is now 1.5 s but still under the 5 s threshold, the change is below the regression guard but still meaningful — flag in the PR description.

The harness deliberately does NOT auto-tune. Tuning is informed by the data; tuning commits land separately, each with their own captured baseline update.

CI cadence

Defined in .github/workflows/loadtest.yml:

• workflow_dispatch — manual trigger from the Actions tab. Used before tagging a release or after a meaningful tuning commit.
• Weekly cron — Mondays at 06:00 UTC. Catches gradual regressions from cumulative changes that no single PR triggered.

The workflow does not run per-push. Load tests are minutes long and would not provide useful per-PR signal; per-push pressure goes through make verify (which is fast) and the deploy-vendor-e2e job.

Connector-tier baseline (Bundle 10 of the 2026-05-02 deployment-target audit)

Bundle 10 extended the harness to cover per-target-type handshake throughput in addition to the API-tier issuance/list throughput documented above. The docker-compose stack now boots four target sidecars (nginx, apache, haproxy, f5-mock) each serving a starter cert from a shared target-tls-init container, and k6 runs four additional scenarios — nginx_handshake, apache_handshake, haproxy_handshake, f5_handshake — at sustained 100 conns/min for 5 minutes against each.

What the connector tier measures

End-to-end TCP connect + TLS handshake + tiny HTTP request/response latency per target type, tagged via the k6 target_type label so summary.json's connector_tier section breaks the numbers out per sidecar:

{
  "connector_tier": {
    "nginx":   { "p50": ..., "p95": ..., "p99": ..., "error_rate": ..., "iterations": ... },
    "apache":  { ... },
    "haproxy": { ... },
    "f5":      { ... }
  }
}

This validates the target sidecar daemons are operational under sustained connection load. Procurement asks "can certctl's nginx target handle 5,000 endpoints at 47-day rotation?" — the connector code's correctness is pinned by per-connector unit tests; the underlying daemon's connection-rate ceiling is what these scenarios pin.

What the connector tier explicitly does NOT measure (v1)

• The full agent-driven deploy hot path. v1 measures handshake throughput against the sidecars directly. v2 of the harness is a follow-up that POSTs cert requests bound to per-target-type targets, polls the deployments endpoint until the agent reports complete, and measures the full POST → poll → cert-served loop. v2 needs the agent registration + target-binding API surface plumbed end-to-end in the loadtest stack — meaningful work, but not a blocker for the connection- rate procurement question.
• Kubernetes connector. kind-in-docker requires privileged: true and is operationally fragile in CI. Deferred until Bundle 2 (real k8s.io/client-go) lands and a CI-friendly envtest harness is wired.
• Real F5 BIG-IP. The harness uses the in-tree f5-mock-icontrol Go server (already used by the deploy-vendor-e2e CI job). Real F5 appliance benchmarking is out of scope; operators with a real F5 vagrant box per docs/connector-f5.md can substitute it manually.

Threshold contract

Defined in k6.js's thresholds block. Any change pushing past these fails the test:

Target type	p95	p99	Error rate
nginx	< 1 s	< 3 s	< 1% (global)
apache	< 1 s	< 3 s	< 1% (global)
haproxy	< 1 s	< 3 s	< 1% (global)
f5	< 1.5 s	< 5 s	< 1% (global)

f5-mock's threshold is looser because the iControl REST handler does slightly more work per request (login+upload+install dance the F5 connector itself drives — not exercised here, but the daemon's request handler is heavier).

Connector-tier captured baseline

Target type	p50	p95	p99	Error rate	Iterations
nginx (threshold)	—	< 1 s	< 3 s	< 1%	n/a
nginx (baseline)	TBD	TBD	TBD	TBD	TBD
apache (threshold)	—	< 1 s	< 3 s	< 1%	n/a
apache (baseline)	TBD	TBD	TBD	TBD	TBD
haproxy (threshold)	—	< 1 s	< 3 s	< 1%	n/a
haproxy (baseline)	TBD	TBD	TBD	TBD	TBD
f5 (threshold)	—	< 1.5 s	< 5 s	< 1%	n/a
f5 (baseline)	TBD	TBD	TBD	TBD	TBD

The em-dash placeholders are deliberate: do not commit numeric values without running the loadtest on canonical hardware first. Numbers from a developer laptop are misleading. The first gh workflow run loadtest.yml on a clean GitHub runner captures the baseline; commit the captured numbers into the table above as a follow-up commit alongside the methodology line.

Methodology pinned at baseline capture (canonical hardware):

• Hardware: GitHub-hosted ubuntu-latest runners (currently 4 vCPU / 16 GiB / SSD-backed). Operator captures from gh workflow run loadtest.yml to keep the hardware constant across runs.
• Sidecar images: nginx:1.27-alpine, httpd:2.4-alpine, haproxy:2.9-alpine, in-tree f5-mock-icontrol (built from deploy/test/f5-mock-icontrol/).
• Concurrency: 100 conns/min sustained per target type (400 conns/min total across the four target scenarios + 100 req/s on the API tier).
• Duration: 5 minutes per scenario, 10s stagger between API tier and connector tier so warmup overlap doesn't skew the first 30 seconds.
• TLS: starter cert from target-tls-init (ECDSA P-256, multi-SAN). The loadtest scenarios connect with K6_INSECURE_SKIP_TLS_VERIFY=true.

To recapture the connector-tier baseline after a tuning commit affecting target sidecars or the connector code:

make loadtest
# Inspect deploy/test/loadtest/results/summary.json for the
# connector_tier object and update the table above.

Files in this directory

deploy/test/loadtest/
├── README.md         (this file)
├── docker-compose.yml
├── k6.js             (the load script)
├── certs/            (gitignored — tls-init writes here)
├── fixtures/         (Bundle 10: target sidecar configs + shared starter cert)
│   ├── nginx.conf
│   ├── httpd.conf
│   ├── haproxy.cfg
│   └── target-certs/ (gitignored — target-tls-init writes here)
└── results/          (gitignored — k6 writes summary.{json,txt} here)

ACME flows (Phase 5)

The deploy/test/loadtest/k6/acme_flow.js scenario hammers the unauthenticated ACME surface (directory + new-nonce + ARI synthetic lookups) at constant 100 VUs for 5 minutes. JWS-signed paths (new-account / new-order / finalize) are intentionally out of scope: k6 doesn't ship JWS, and bundling lego inside k6 would obscure the underlying-server p95 we're trying to measure. Instead, the make acme-rfc-conformance-test target drives lego against the same stack for the full happy-path conformance gate.

Run it:

cd deploy/test/loadtest
docker compose up -d certctl postgres
k6 run --env CERTCTL_ACME_DIRECTORY=https://localhost:8443/acme/profile/prof-test/directory \
       k6/acme_flow.js

Baseline (ACME flows, 100 VUs × 5m)

The baseline is operator-captured on a workstation-class machine with a single certctl-server container + a single postgres container. Re-capture after schema migrations or transport changes; commit the new numbers so regressions are visible in code review.

Metric	Threshold	Last captured	Notes
directory_duration p95	< 500 ms	operator	Unauth GET; cache-friendly.
new_nonce_duration p95	< 300 ms	operator	Single Postgres INSERT under the hood.
renewal_info_duration p95 (synthetic id)	< 800 ms	operator	Synthetic cert-id → 4xx fast path.
http_req_failed rate	< 1%	operator	Should be ~0 — failures here mean transport issues.

Capture command: make loadtest after pointing the compose stack at the ACME flow scenario. Operators with kind / cert-manager available should pair this with make acme-cert-manager-test for end-to-end verification.

Scale tier (Phase 8 SCALE-H2, 2026-05-14)

Phase 8 closure added three new k6 scenarios that exercise the scale-relevant load surfaces the API tier and connector tier left uncovered:

Scenario	k6 file	Seed	Make target
Bulk-renewal under load	k6/bulk_renewal.js	seed/01_bulk_renewal_certs.sql (10K certs)	make loadtest-scale-bulk
ACME enrollment burst	k6/acme_burst.js	(none — unauth surface)	make loadtest-scale-acme
Agent heartbeat storm	k6/agent_storm.js	seed/02_agent_fleet.sql (5K agents)	make loadtest-scale-agent

The scale-tier scenarios live behind the scale compose profile so the default make loadtest (API tier + connector tier, ~7 min) stays fast. Run all three serially with make loadtest-scale, or trigger the loadtest.yml workflow's k6-scale matrix jobs from the Actions tab for canonical-hardware capture.

Operator-facing baseline table + threshold contracts + documented limitations live in docs/operator/scale.md under the "Scale-tier scenarios (SCALE-H2, Phase 8)" section. Treat that as the canonical source — this README only links.

The seed fixtures + their idempotency contract are documented in seed/README.md.

Audit references

• API tier: 2026-05-01 issuer coverage audit fix #8.
• Connector tier: 2026-05-02 deployment-target audit Bundle 10.
• ACME flows: Phase 5 master prompt (project notes).
• Scale tier: 2026-05-14 architecture diligence Phase 8 (SCALE-H2).

---

1. Sandbox-aggregate placeholder — captured at HEAD on a Linux/aarch64 unprivileged sandbox (no Docker, no GitHub-hosted runner). Both rows show the same aggregate combined-load numbers because the sandbox run did not break out per-scenario tags in summary.json. Treat these as a sanity floor (proof the API tier handles 100 req/s combined with zero errors and sub-10ms p99), not as the per-scenario baselines the threshold contract is written against. Replace via gh workflow run loadtest.yml on the canonical ubuntu-latest runner — that produces per-scenario tagged metrics in summary.json. ↩︎