# Operator scale guide > Last reviewed: 2026-05-14 Use this when: - You're sizing a new certctl deployment for a target fleet count. - You're scaling an existing deployment up from demo (15 certs / 1 agent) to production (1K+ certs / 100+ agents). - An auditor asks "what does this scale to?" and you want a documented answer that isn't "we haven't measured." ## DB connection pool certctl's PostgreSQL connection pool is the single largest scale lever. Pool exhaustion looks like 503s + agent poll timeouts + scheduler falling behind on its loops. The default ships at 50 max open connections (`CERTCTL_DATABASE_MAX_CONNS=50`), with idle = max/5 = 10 under the existing `internal/repository/postgres/db.go::NewDBWithMaxConns` contract. Operator-tune ladder: | Fleet size | `CERTCTL_DATABASE_MAX_CONNS` | Postgres `max_connections` | Notes | |---|---|---|---| | ≤ 500 certs / 100 agents | `50` (default) | `100` (PG default) | Demo + small deployments. Pool default sized for this. | | 5K certs / 1K agents | `100` | `200` | Postgres needs an explicit bump from the 100 default; reload required. | | 50K certs / 10K agents | `200` | `400` | Plus dedicated Postgres VM (separate from server host); shared_buffers ≥ 1Gi. | Always leave headroom in Postgres's `max_connections` for backups (`pg_dump` opens its own connection), ad-hoc psql sessions, and replicas. The ratio `(server pool size × replicas) + 20` is a safe floor for Postgres's `max_connections`. **Numbers above the small-fleet row are operator-tuning starting points, not validated ceilings.** Phase 8 of the architecture diligence remediation will replace these with measured values from synthetic fleets; until then, capture your own observations in a loadtest log and tune against them. ## Scheduler tick budgets certctl has 15 scheduler loops, each with its own cadence (internal/scheduler/scheduler.go). The renewal scan is the hottest loop on large fleets: it pulls every managed certificate, applies each profile's renewal policy, and dispatches an issuance job per cert that meets the threshold. The default cadence is `1h` (`CERTCTL_SCHEDULER_RENEWAL_CHECK_INTERVAL`). Phase 6 SCALE-M5 closure (2026-05-14) added per-ticker jitter via the `internal/scheduler.JitteredTicker` wrapper. Each loop's interval is unchanged; the wrapper adds ±10% randomized delay per tick so multiple loops with the same nominal cadence don't co-fire and cause hour- boundary CPU + DB spikes. For most fleets the visible effect is a smoother CPU graph during the renewal scan. **Renewal-sweep semaphore (SCALE-L1).** The renewal loop dispatches concurrent issuance work behind a per-tick semaphore (default `CERTCTL_RENEWAL_CONCURRENCY=25`). Under tick-budget pressure (a tick that exceeds the loop interval), the semaphore can hold the entire concurrency cap until the context cancels at next-tick boundary — which is intentional. The drain happens via context cancellation; new work isn't started past the deadline. Tests in `internal/scheduler/` pin this drain behavior. Operators on large fleets should: 1. Bump `CERTCTL_RENEWAL_CONCURRENCY` to 50 or 100 if the renewal scan consistently exceeds tick budget. 2. Also bump `CERTCTL_DATABASE_MAX_CONNS` proportionally — each concurrent renewal task opens its own pool connection during issuance / deployment. 3. Watch for the "renewal scan complete" log line per tick. If it's consistently late, you're under-provisioned. ## Async CA polling budgets (SCALE-M3) DigiCert, Entrust, GlobalSign, and Sectigo are async issuers — they accept a CSR, queue it on the CA side, and return a polling token. The certctl server polls the CA's status endpoint until the cert is ready or the deadline expires. The default poll-deadline is 10 minutes wall-clock (`asyncpoll.DefaultMaxWait`); after that the issuance returns `StillPending` and the scheduler re-enqueues the job for the next tick. Priority chain when picking the actual deadline (highest → lowest): 1. Per-connector env: `CERTCTL_DIGICERT_POLL_MAX_WAIT_SECONDS`, `CERTCTL_ENTRUST_POLL_MAX_WAIT_SECONDS`, `CERTCTL_GLOBALSIGN_POLL_MAX_WAIT_SECONDS`, `CERTCTL_SECTIGO_POLL_MAX_WAIT_SECONDS`. 2. Global env: `CERTCTL_ASYNC_POLL_MAX_WAIT_SECONDS` (sets the process-wide default for all async-CA connectors that didn't set their per-connector value). 3. Package const: `asyncpoll.DefaultMaxWait = 10 * time.Minute`. Operators with slow async CAs (Entrust certificate-mode in particular can take 15-30 minutes during business hours) should raise the per-connector value rather than the global; that way fast issuers don't pay the polling cost. ## Cursor pagination caching (SCALE-L2) Phase 6 SCALE-L2 closure (2026-05-14) added an ETag middleware at `internal/api/middleware/etag.go` covering the top-5 read endpoints: `/api/v1/certificates`, `/api/v1/jobs`, `/api/v1/agents`, `/api/v1/audit`, `/api/v1/discovery/certificates`. The ETag is derived from `(max-row-updated-at, row-count)` for the requested filter; repeated requests with the same query return `304 Not Modified` when the underlying data hasn't changed. The dashboard benefits most — its polling loop on the certificates page is the single largest read-traffic source on most deployments. When the cache is effective, repeated reads bypass the `SELECT COUNT(*) FROM ` query entirely. The cache invalidates on any mutation to the table (the row-count + max-updated-at hash flips). Operators don't need to do anything to opt in — the middleware is wired around the top-5 endpoints unconditionally. If you want to verify it's working, check the `ETag:` response header on a list endpoint and repeat the request with the same value in an `If-None-Match:` header — the second request should return 304 with an empty body. ## Scale-tier scenarios (SCALE-H2, Phase 8) Phase 8 (2026-05-14) extended the k6 load-test harness with three new scenarios that exercise the scale-relevant load surfaces the original API tier left uncovered. They live behind a compose profile gate (`docker compose --profile scale`) so the default `make loadtest` stays focused on per-PR regression scope. The full set runs weekly on the same `loadtest.yml` cron as the API + connector tier. | Scenario | k6 file | Seed fixture | Sustained load | |---|---|---|---| | Bulk-renewal under load | `deploy/test/loadtest/k6/bulk_renewal.js` | 10,000 managed_certificates (`seed/01_bulk_renewal_certs.sql`) | 5 req/s POST `/api/v1/certificates/bulk-renew` × 5 min | | ACME enrollment burst | `deploy/test/loadtest/k6/acme_burst.js` | (none — unauth surface) | 200 concurrent VUs × directory/nonce/ARI × 5 min | | Agent heartbeat storm | `deploy/test/loadtest/k6/agent_storm.js` | 5,000 agents (`seed/02_agent_fleet.sql`) | 167 req/s POST `/api/v1/agents/{id}/heartbeat` × 5 min | ### Threshold contracts (regression guards, NOT measured baselines) | Scenario | Metric | Threshold | |---|---|---| | Bulk-renewal | `http_req_duration{scenario:bulk_renewal}` p99 | < 5 s | | Bulk-renewal | `http_req_duration{scenario:bulk_renewal}` p95 | < 2 s | | Bulk-renewal | `http_req_failed{scenario:bulk_renewal}` | < 1% | | ACME burst | `acme_directory_duration` p95 | < 500 ms | | ACME burst | `acme_new_nonce_duration` p95 | < 300 ms | | ACME burst | `acme_renewal_info_duration` p95 | < 800 ms | | ACME burst | `http_req_failed{server_error:true}` 5xx-only | < 0.1% | | Agent storm | `http_req_duration{scenario:agent_storm}` p99 | < 1 s | | Agent storm | `http_req_duration{scenario:agent_storm}` p95 | < 500 ms | | Agent storm | `http_req_failed{scenario:agent_storm}` | < 0.1% | 429 rate-limit responses on the ACME burst are EXPECTED — Phase 5's per-account rate limiter SHOULD fire at sustained 200-VU pressure. The custom `acme_rate_limited_count` Counter tracks how often it fires; `acme_rate_limit_shape_ok` Counter verifies every 429 returns the RFC 7807 `application/problem+json` shape with the `urn:ietf:params:acme:error:rateLimited` type. A regression that returned plain-text 429 or a different problem type would surface as `(rate_limited_count - shape_ok_count) > 0` in the summary. ### Measured baseline — TBD pending canonical-hardware capture The Phase 8 scenarios shipped 2026-05-14. Baseline capture on a canonical `ubuntu-latest` GitHub runner is the next operational step; until then, the table below holds TBD placeholders. **Do NOT publish sandbox-captured numbers here** — the same anti-pattern the original loadtest README guards against (sandbox-aggregate placeholder vs canonical hardware) applies to Phase 8. | Scenario | p50 | p95 | p99 | Error rate | Date measured | Commit | |---|---|---|---|---|---|---| | **bulk_renewal** | TBD | TBD | TBD | TBD | — | — | | **acme_burst** directory | TBD | TBD | TBD | TBD | — | — | | **acme_burst** new-nonce | TBD | TBD | TBD | TBD | — | — | | **acme_burst** renewal-info | TBD | TBD | TBD | TBD | — | — | | **agent_storm** | TBD | TBD | TBD | TBD | — | — | Capture procedure: trigger `loadtest.yml` from the Actions tab against the current `master` SHA; wait for the `k6-scale` matrix jobs to complete; download the per-scenario summary artifacts; copy p50/p95/ p99 from `summary-.json` into the table; commit the captured numbers alongside the date + SHA. Replace this paragraph with the captured-on row when the first canonical run lands. ### How to run the scale tier locally ```sh # All three scenarios serially (~18 min total): make loadtest-scale # Individual scenarios (each ~6 min): make loadtest-scale-bulk # 10K cert bulk-renew make loadtest-scale-acme # 200 VU ACME burst make loadtest-scale-agent # 5K agent heartbeat storm ``` Each scenario boots its own copy of the loadtest compose stack (postgres + tls-init + certctl-server) plus the `scale-seed` init container that runs the SQL fixtures from `deploy/test/loadtest/seed/`. The seed is idempotent (`ON CONFLICT … DO NOTHING`) so re-running a scenario against the same compose stack is cheap. ### Documented limitations of the scale tier - **JWS-signed ACME flows are not measured.** The ACME burst scenario hits the unauthenticated directory + new-nonce + ARI surface only. Measuring the JWS-signed POST hot path (new-account / new-order / finalize) requires bundling a JWS signer into the k6 driver (k6 doesn't ship JWS). End-to-end JWS conformance is gated by `make acme-rfc-conformance-test` which drives `lego` against the same stack. - **Scheduler renewal scan throughput.** The bulk-renewal scenario measures the inbound POST throughput; the scheduler's `jobProcessorLoop` drains the enqueued jobs at a fixed per-tick budget (`CERTCTL_RENEWAL_CONCURRENCY=25` default), and the throughput of that path is not amplified by adding more inbound bulk-renew calls. A future scenario could pull `/api/v1/jobs?status=pending` and measure drain time. - **Production-sized Postgres.** The compose stack runs `postgres:16-alpine` with default config on a CI runner. Production deploys with `shared_buffers >= 1 GiB` + dedicated Postgres VM will have different query plans for the 10K-cert scan. The captured numbers translate directionally but the absolute ceiling is workload-specific — see the operator-tune ladder above for production sizing. - **Pull-only deployment model.** Agent CSR submit, work-poll, and deploy-verify paths are intentionally out of scope. The heartbeat storm exercises the highest-frequency call on a typical fleet; the work-poll path runs at the same cadence but is cheap (empty set returned 99% of the time). ## Profiling production When the above ladder doesn't fit your shape, profile against your specific workload. The [performance-baselines.md](performance-baselines.md) runbook has single-endpoint, inventory-walk, and renewal-scan recipes you can adapt. ## Related reading - [`docs/operator/performance-baselines.md`](performance-baselines.md) — per-endpoint baselines + how to re-baseline after upgrades. - [`docs/operator/runbooks/postgres-backup.md`](runbooks/postgres-backup.md) — Postgres-side backup discipline (necessary precondition for any scale tuning). - [`deploy/ENVIRONMENTS.md`](../../deploy/ENVIRONMENTS.md) — the full env-var inventory the values referenced above come from.