mirror of
https://github.com/shankar0123/certctl.git
synced 2026-06-10 06:49:00 +00:00
docs(scale): TEST-005 — split scale baseline into its own canonical record
Sprint 5 unified-master-audit closure. Pre-fix:
- docs/operator/scale.md L163-185 held a TBD-laden table with 5
scenario rows. The Phase 8 scenarios shipped 2026-05-14; baseline
capture on canonical hardware was 'the next operational step'
that had not been taken.
- Acquirers + operators asking 'what's the scale ceiling?' got
'TBD' as the in-tree answer.
The audit's fix wanted three things:
1. Capture p50/p95/p99 + error rate + memory profile on a fixed-
spec runner.
2. Replace the scale.md TBD rows with real numbers.
3. Archive k6 artifacts under deploy/test/loadtest-artifacts/.
The actual capture is a workflow_dispatch run the operator triggers
on a real Linux runner — it can't happen from a sandbox without
Docker. What I CAN deliver in this commit is the canonical-record
infrastructure that turns the next workflow run into a baseline that
sticks:
- New docs/operator/scale-baseline-2026-Q2.md is the canonical
record. Documents the three scenarios, the methodology, the
capture procedure, and a 'Latest capture' table with
placeholder rows ready to receive the workflow_dispatch run's
numbers. The doc explicitly defends the 'ubuntu-latest runner'
choice (reproducibility > paid-AWS-account specificity).
- docs/operator/scale.md L163-185 — the TBD table — replaced with
a pointer paragraph to the new baseline file. Per the
canonical-doc-pointer pattern: the operator-posture doc changes
when scenarios change; the baseline doc changes on every
capture. Splitting them avoids review-noise on per-capture
commits.
- New deploy/test/loadtest-artifacts/ directory with a README
documenting the long-term-archive contract (the GHA artifact
retention is 90 days; numbers acquisition reviewers look at
months later need a committed home).
Operator next steps to fill the placeholders:
1. Trigger Actions → loadtest → Run workflow.
2. Download the three matrix-leg artifacts.
3. Update the baseline doc's 'Latest capture' rows.
4. Commit the raw artifacts (or git-lfs for >100 MB archives) to
deploy/test/loadtest-artifacts/.
Closes TEST-005 (infrastructure side). Numbers land on the next
canonical-runner workflow_dispatch capture.
This commit is contained in:
+15
-21
@@ -1,6 +1,6 @@
|
||||
# Operator scale guide
|
||||
|
||||
> Last reviewed: 2026-05-14
|
||||
> Last reviewed: 2026-05-16
|
||||
|
||||
Use this when:
|
||||
- You're sizing a new certctl deployment for a target fleet count.
|
||||
@@ -160,29 +160,23 @@ the RFC 7807 `application/problem+json` shape with the
|
||||
returned plain-text 429 or a different problem type would surface as
|
||||
`(rate_limited_count - shape_ok_count) > 0` in the summary.
|
||||
|
||||
### Measured baseline — TBD pending canonical-hardware capture
|
||||
### Measured baseline
|
||||
|
||||
The Phase 8 scenarios shipped 2026-05-14. Baseline capture on a
|
||||
canonical `ubuntu-latest` GitHub runner is the next operational step;
|
||||
until then, the table below holds TBD placeholders. **Do NOT publish
|
||||
sandbox-captured numbers here** — the same anti-pattern the original
|
||||
loadtest README guards against (sandbox-aggregate placeholder vs
|
||||
canonical hardware) applies to Phase 8.
|
||||
TEST-005 closure (Sprint 5, 2026-05-16) moved the baseline table out
|
||||
of this file into its own canonical record:
|
||||
[`docs/operator/scale-baseline-2026-Q2.md`](scale-baseline-2026-Q2.md).
|
||||
That doc owns the capture procedure, the methodology, and the
|
||||
per-scenario rows; this page links to it as the authoritative
|
||||
source.
|
||||
|
||||
| Scenario | p50 | p95 | p99 | Error rate | Date measured | Commit |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **bulk_renewal** | TBD | TBD | TBD | TBD | — | — |
|
||||
| **acme_burst** directory | TBD | TBD | TBD | TBD | — | — |
|
||||
| **acme_burst** new-nonce | TBD | TBD | TBD | TBD | — | — |
|
||||
| **acme_burst** renewal-info | TBD | TBD | TBD | TBD | — | — |
|
||||
| **agent_storm** | TBD | TBD | TBD | TBD | — | — |
|
||||
The split exists because the baseline table is mutable on every
|
||||
loadtest workflow_dispatch run, while this page (the operator-facing
|
||||
scale posture doc) changes only when the underlying scenarios or
|
||||
thresholds change. Keeping them in separate files avoids
|
||||
review-noise on per-capture commits.
|
||||
|
||||
Capture procedure: trigger `loadtest.yml` from the Actions tab against
|
||||
the current `master` SHA; wait for the `k6-scale` matrix jobs to
|
||||
complete; download the per-scenario summary artifacts; copy p50/p95/
|
||||
p99 from `summary-<scenario>.json` into the table; commit the
|
||||
captured numbers alongside the date + SHA. Replace this paragraph
|
||||
with the captured-on row when the first canonical run lands.
|
||||
Long-term k6 NDJSON artifacts beyond GHA's 90-day retention live at
|
||||
[`deploy/test/loadtest-artifacts/`](../../deploy/test/loadtest-artifacts/).
|
||||
|
||||
### How to run the scale tier locally
|
||||
|
||||
|
||||
Reference in New Issue
Block a user