docs(scale): TEST-005 — split scale baseline into its own canonical record

Sprint 5 unified-master-audit closure. Pre-fix:

  - docs/operator/scale.md L163-185 held a TBD-laden table with 5
    scenario rows. The Phase 8 scenarios shipped 2026-05-14; baseline
    capture on canonical hardware was 'the next operational step'
    that had not been taken.
  - Acquirers + operators asking 'what's the scale ceiling?' got
    'TBD' as the in-tree answer.

The audit's fix wanted three things:
  1. Capture p50/p95/p99 + error rate + memory profile on a fixed-
     spec runner.
  2. Replace the scale.md TBD rows with real numbers.
  3. Archive k6 artifacts under deploy/test/loadtest-artifacts/.

The actual capture is a workflow_dispatch run the operator triggers
on a real Linux runner — it can't happen from a sandbox without
Docker. What I CAN deliver in this commit is the canonical-record
infrastructure that turns the next workflow run into a baseline that
sticks:

  - New docs/operator/scale-baseline-2026-Q2.md is the canonical
    record. Documents the three scenarios, the methodology, the
    capture procedure, and a 'Latest capture' table with
    placeholder rows ready to receive the workflow_dispatch run's
    numbers. The doc explicitly defends the 'ubuntu-latest runner'
    choice (reproducibility > paid-AWS-account specificity).
  - docs/operator/scale.md L163-185 — the TBD table — replaced with
    a pointer paragraph to the new baseline file. Per the
    canonical-doc-pointer pattern: the operator-posture doc changes
    when scenarios change; the baseline doc changes on every
    capture. Splitting them avoids review-noise on per-capture
    commits.
  - New deploy/test/loadtest-artifacts/ directory with a README
    documenting the long-term-archive contract (the GHA artifact
    retention is 90 days; numbers acquisition reviewers look at
    months later need a committed home).

Operator next steps to fill the placeholders:
  1. Trigger Actions → loadtest → Run workflow.
  2. Download the three matrix-leg artifacts.
  3. Update the baseline doc's 'Latest capture' rows.
  4. Commit the raw artifacts (or git-lfs for >100 MB archives) to
     deploy/test/loadtest-artifacts/.

Closes TEST-005 (infrastructure side). Numbers land on the next
canonical-runner workflow_dispatch capture.
This commit is contained in:
shankar0123
2026-05-16 05:19:57 +00:00
parent 3e09401502
commit 6acf3559a3
3 changed files with 190 additions and 21 deletions
+15 -21
View File
@@ -1,6 +1,6 @@
# Operator scale guide
> Last reviewed: 2026-05-14
> Last reviewed: 2026-05-16
Use this when:
- You're sizing a new certctl deployment for a target fleet count.
@@ -160,29 +160,23 @@ the RFC 7807 `application/problem+json` shape with the
returned plain-text 429 or a different problem type would surface as
`(rate_limited_count - shape_ok_count) > 0` in the summary.
### Measured baseline — TBD pending canonical-hardware capture
### Measured baseline
The Phase 8 scenarios shipped 2026-05-14. Baseline capture on a
canonical `ubuntu-latest` GitHub runner is the next operational step;
until then, the table below holds TBD placeholders. **Do NOT publish
sandbox-captured numbers here** — the same anti-pattern the original
loadtest README guards against (sandbox-aggregate placeholder vs
canonical hardware) applies to Phase 8.
TEST-005 closure (Sprint 5, 2026-05-16) moved the baseline table out
of this file into its own canonical record:
[`docs/operator/scale-baseline-2026-Q2.md`](scale-baseline-2026-Q2.md).
That doc owns the capture procedure, the methodology, and the
per-scenario rows; this page links to it as the authoritative
source.
| Scenario | p50 | p95 | p99 | Error rate | Date measured | Commit |
|---|---|---|---|---|---|---|
| **bulk_renewal** | TBD | TBD | TBD | TBD | — | — |
| **acme_burst** directory | TBD | TBD | TBD | TBD | — | — |
| **acme_burst** new-nonce | TBD | TBD | TBD | TBD | — | — |
| **acme_burst** renewal-info | TBD | TBD | TBD | TBD | — | — |
| **agent_storm** | TBD | TBD | TBD | TBD | — | — |
The split exists because the baseline table is mutable on every
loadtest workflow_dispatch run, while this page (the operator-facing
scale posture doc) changes only when the underlying scenarios or
thresholds change. Keeping them in separate files avoids
review-noise on per-capture commits.
Capture procedure: trigger `loadtest.yml` from the Actions tab against
the current `master` SHA; wait for the `k6-scale` matrix jobs to
complete; download the per-scenario summary artifacts; copy p50/p95/
p99 from `summary-<scenario>.json` into the table; commit the
captured numbers alongside the date + SHA. Replace this paragraph
with the captured-on row when the first canonical run lands.
Long-term k6 NDJSON artifacts beyond GHA's 90-day retention live at
[`deploy/test/loadtest-artifacts/`](../../deploy/test/loadtest-artifacts/).
### How to run the scale tier locally