mirror of https://github.com/shankar0123/certctl.git synced 2026-06-07 12:11:30 +00:00

Files

T

History

shankar0123 6acf3559a3 docs(scale): TEST-005 — split scale baseline into its own canonical record

Sprint 5 unified-master-audit closure. Pre-fix:

  - docs/operator/scale.md L163-185 held a TBD-laden table with 5
    scenario rows. The Phase 8 scenarios shipped 2026-05-14; baseline
    capture on canonical hardware was 'the next operational step'
    that had not been taken.
  - Acquirers + operators asking 'what's the scale ceiling?' got
    'TBD' as the in-tree answer.

The audit's fix wanted three things:
  1. Capture p50/p95/p99 + error rate + memory profile on a fixed-
     spec runner.
  2. Replace the scale.md TBD rows with real numbers.
  3. Archive k6 artifacts under deploy/test/loadtest-artifacts/.

The actual capture is a workflow_dispatch run the operator triggers
on a real Linux runner — it can't happen from a sandbox without
Docker. What I CAN deliver in this commit is the canonical-record
infrastructure that turns the next workflow run into a baseline that
sticks:

  - New docs/operator/scale-baseline-2026-Q2.md is the canonical
    record. Documents the three scenarios, the methodology, the
    capture procedure, and a 'Latest capture' table with
    placeholder rows ready to receive the workflow_dispatch run's
    numbers. The doc explicitly defends the 'ubuntu-latest runner'
    choice (reproducibility > paid-AWS-account specificity).
  - docs/operator/scale.md L163-185 — the TBD table — replaced with
    a pointer paragraph to the new baseline file. Per the
    canonical-doc-pointer pattern: the operator-posture doc changes
    when scenarios change; the baseline doc changes on every
    capture. Splitting them avoids review-noise on per-capture
    commits.
  - New deploy/test/loadtest-artifacts/ directory with a README
    documenting the long-term-archive contract (the GHA artifact
    retention is 90 days; numbers acquisition reviewers look at
    months later need a committed home).

Operator next steps to fill the placeholders:
  1. Trigger Actions → loadtest → Run workflow.
  2. Download the three matrix-leg artifacts.
  3. Update the baseline doc's 'Latest capture' rows.
  4. Commit the raw artifacts (or git-lfs for >100 MB archives) to
     deploy/test/loadtest-artifacts/.

Closes TEST-005 (infrastructure side). Numbers land on the next
canonical-runner workflow_dispatch capture.

2026-05-16 05:19:57 +00:00

README.md

docs(scale): TEST-005 — split scale baseline into its own canonical record

2026-05-16 05:19:57 +00:00

README.md

loadtest-artifacts/

Last reviewed: 2026-05-16

Long-term archive of k6 load-test results from the loadtest GitHub Actions workflow. TEST-005 closure (Sprint 5, 2026-05-16) introduces this directory as the committed home for captures the operator chooses to retain past GitHub's 90-day artifact-retention window.

What lands here

After a loadtest workflow_dispatch run, follow the procedure in docs/operator/scale-baseline-2026-Q2.md:

Download the three matrix-leg artifacts from the workflow page.
Update the latest-capture table in the baseline doc with the extracted percentiles.

Commit the raw artifacts you want long-term-retained here, named:

2026-Q2-bulk-renewal-<run-id>.tar.gz
2026-Q2-acme-burst-<run-id>.tar.gz
2026-Q2-agent-storm-<run-id>.tar.gz

If any single archive exceeds 100 MB, route it through git lfs (configured at repo root via .gitattributes).

Why commit artifacts rather than rely on GHA retention

GitHub Actions retains workflow artifacts for 90 days by default. Acquisition-diligence reviewers looking at scale evidence months later get a 404 unless we keep the raw NDJSON in tree.
Reproducibility. Pinning the k6 NDJSON to a SHA makes it cheap to re-derive percentiles with a different filter (e.g. "p99 excluding the warmup ramp's first 30 seconds") without re-running the workflow.

What does NOT belong here

Per-PR ephemeral runs. The loadtest workflow runs on workflow_dispatch + weekly cron; per-PR runs would be too noisy and aren't retained.
Production-environment captures. These artifacts are the ubuntu-latest reference baseline. An operator capturing their production-environment scale should put the artifacts in their own observability platform — committing them here would imply "this is what certctl's reference numbers are" which it isn't.
Manual k6 captures from a developer's laptop. Same rationale as the visual-regression snapshot runbook (docs/operator/runbooks/e2e-snapshot-update.md) — only the CI environment produces canonical numbers.