gsadmin/certctl
1
0
Fork 0
mirror of https://github.com/shankar0123/certctl.git synced 2026-06-08 19:08:51 +00:00
Code Issues Packages Projects Releases Wiki Activity

Bundle P (Coverage Audit Closure): QA doc strengthening — M-007/M-009/M-010/M-011/M-012 closed; M-008 deferred

Browse Source 
Six structural strengthenings to certctl QA documentation surface, raising
acquisition-readiness QA-doc score 4.0 -> 4.7. M-008 (per-RFC test-vector
subsections under Parts 21 + 24) deferred as 'Bundle P.2-extended' (out of
session budget; not acquisition-blocking — sharpens conformance story).

P.1 — `make qa-stats` single-source-of-truth (M-012 closed)
=========================================================
New `qa-stats` PHONY target in `Makefile` emits 14 metrics that every
count claim in `docs/qa-test-guide.md` and `docs/testing-guide.md` is
derived from: backend test files / Test functions / t.Run subtests,
frontend test files, fuzz targets, t.Skip sites, qa_test.go Part_ subtests,
testing-guide.md Parts, and unique seed IDs (mc-* / ag-* / iss-* / tgt-* /
nst-*). Iterated the seed-count regex to a deterministic
'grep -oE <prefix>-[a-z0-9_-]+ | sort -u | wc -l' form. Output emits 14
lines at HEAD; integers parse cleanly; verified against drift guards.

P.2 — CI drift guards (M-011 closed)
=========================================================
Two new CI steps in `.github/workflows/ci.yml` after coverage upload:
- Part-count drift guard: '49 of N Parts' from qa-test-guide.md vs
  '^## Part N:' header count in testing-guide.md. Fails on mismatch.
- Seed-count drift guard: '### Certificates (N total' / '### Issuers
  (N total' from qa-test-guide.md vs unique mc-* / iss-* IDs in
  seed_demo.sql with <=5pp slack on issuers (issuer rows != unique
  iss-* IDs because seed uses iss-* prefix elsewhere).
Both validated locally — pass at HEAD (56==56 Parts, 32==32 certs,
18 issuer IDs within 5pp slack of 13 issuer rows). YAML lint clean.

P.3 — Test Suite Health dashboard (Strengthening #7)
=========================================================
Single-page snapshot at top of qa-test-guide.md: file/function/subtest
counts, fuzz/skip counts, frontend test count, last-coverage-audit date
+ status, last-mutation-run date + status, race-detector status,
repository-integration test status. Designed for first-look auditor /
acquirer / new-engineer scanning.

P.4 — Coverage by Risk Class table (M-007 closed)
=========================================================
After Coverage Map in qa-test-guide.md: 6-row table (Existential /
High / Medium / Low / Frontend / Compliance) x Parts x automation
status. Cross-references each row to coverage-matrix.md. Replaces
implicit 'everything is everything' framing with explicit per-class
gates.

P.5 — Release Day Sign-Off Matrix (M-010 closed)
=========================================================
12-row release-readiness checklist in qa-test-guide.md: backend
race-clean, fuzz seed-corpus regression, frontend Vitest green, CI
drift guards green, mutation-test (sample) >= kill-rate floor, etc.
Each row cites verification command + gate value. Sign-off is 'all 12
green' — produces a per-release artifact attached to the tag.

P.6 — Mutation Testing Targets (Strengthening #5)
=========================================================
New section in qa-test-guide.md cataloging 8 packages x kill-rate
target x tool, with operator runbook citing avito-tech go-mutesting
fork (upstream zimmski/go-mutesting is sandbox-blocked on arm64 due
to syscall.Dup2). Targets aligned to risk class: Existential >=85%,
High >=75%, others tracked-not-gated.

P.7 — Per-Connector Failure-Mode Matrix (M-009 closed, condensed)
=========================================================
New 'Part 9.0 Per-Connector Failure-Mode Matrix' in
docs/testing-guide.md: 12 issuers x 8 failure modes (auth-fail / 403
/ 429+Retry-After / 5xx / malformed / DNS-failure / partial-response
/ timeout) = 96 cells with check / triangle / MISSING + Bundle
citations (J/L/M/N). Notable gaps explicitly called out: 429+Retry-
After missing for cloud-managed connectors, DNS-failure missing
across the board, partial-response missing for non-ACME / non-StepCA
connectors. Each gap is a follow-on-bundle candidate.

Verification
=========================================================
- 'make qa-stats' runs to completion, emits 14 metrics, all integers
  parse cleanly
- 'python3 -c "import yaml; yaml.safe_load(...)"' clean on ci.yml
- Both CI drift guards executed locally — both PASS at HEAD
- git diff --stat: 5 files changed, +249 / -1

Audit deliverables
=========================================================
- gap-backlog.md: strikethroughs on M-007 / M-010 / M-011 / M-012;
  partial-strike on M-009 (matrix shipped; deeper per-connector
  failure-mode test files tracked as M-009-extended); deferred-marker
  on M-008 (Bundle P.2-extended); Bundle P closure-log entry
- closure-plan.md: ticks Bundle P [x] with per-item breakdown +
  M-008 deferral note
- CHANGELOG.md: full Bundle P [unreleased] entry above Bundle O
- testing-guide.md: new Part 9.0 Per-Connector Failure-Mode Matrix
- qa-test-guide.md: 4 new sections (Test Suite Health dashboard +
  Coverage by Risk Class + Release Day Sign-Off + Mutation Testing
  Targets); version history bumped to v1.3
- Makefile: new qa-stats PHONY target
- ci.yml: 2 new drift-guard steps after coverage upload

Closes: M-007, M-010, M-011, M-012
Closes (condensed): M-009 (matrix shipped; deeper test files = M-009-extended)
Deferred: M-008 (Bundle P.2-extended; not acquisition-blocking)
Bundle: P (QA Doc Strengthening)
This commit is contained in:
cowork
2026-04-27 18:22:23 +00:00
parent 8adaed6e6d
commit 43b7323d24
5 changed files with 249 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/testing-guide.md
+31
View File
@@ -1808,6 +1808,37 @@ curl -s -w "\nHTTP %{http_code}\n" -X POST -H "$AUTH" -H "$CT" \
**Why it matters:** Issuers are the CAs that sign certificates. If issuer management is broken, no new certs can be issued.
### 9.0 Per-Connector Failure-Mode Matrix (Bundle P / Strengthening #3)
For each issuer connector, the following failure modes MUST be tested at release. Each cell cites the test that exercises it OR is marked `MISSING` (linking to `coverage-audit-2026-04-27/gap-backlog.md` for follow-on closure work). 12 issuers × 8 modes = 96 cells; condensed legend below.
**Legend:** ✓ = covered by hermetic test (httptest.Server / fake SMTP / fake SSH / etc.). △ = covered indirectly (e.g. via wrapper-layer tests; not a per-mode regression). MISSING = no test exists; track as gap-backlog row.
| Connector | 401 | 403 | 429 | 5xx | malformed | partial | timeout | DNS fail |
|---|---|---|---|---|---|---|---|---|
| ACME (RFC 8555) | ✓ B-J | ✓ B-J | △ | ✓ B-J | ✓ B-J (dir + ARI + EAB) | △ | △ | MISSING |
| StepCA (native) | ✓ B-L.B | ✓ B-L.B | MISSING | ✓ B-L.B | ✓ B-L.B (JWE round-trip) | MISSING | △ | MISSING |
| Local CA | n/a (in-process) | n/a | n/a | △ (CA load fail) | ✓ Bundle 9 | n/a | n/a | n/a |
| Vault PKI | △ | △ | MISSING | △ | △ | MISSING | △ | MISSING |
| DigiCert | △ stub | △ stub | MISSING | △ | △ | MISSING | △ | MISSING |
| Sectigo | △ stub | △ stub | MISSING | △ | △ | MISSING | △ | MISSING |
| GoogleCAS | △ stub | △ stub | MISSING | △ | △ | MISSING | △ | MISSING |
| AWS ACM-PCA | △ stub | △ stub | MISSING | △ | △ | MISSING | △ | n/a (SDK retry) |
| GlobalSign | △ stub | △ stub | MISSING | △ | △ | MISSING | △ | MISSING |
| Entrust | △ stub | △ stub | MISSING | △ | △ | MISSING | △ | MISSING |
| EJBCA | △ stub | △ stub | MISSING | △ | △ | MISSING | △ | MISSING |
| OpenSSL (script-based) | n/a | n/a | n/a | △ (script-error) | △ | n/a | △ | n/a |
**Notable gaps surfaced by this matrix:**
- 429 + Retry-After is MISSING for every cloud / SaaS issuer connector. ACME has a partial test (Bundle J's `TestGetRenewalInfo_ARI5xx` covers the 5xx wrapper but not the 429 + Retry-After honor path specifically). Tracked as M-001-extended.
- DNS-failure handling is MISSING across the board. Most connectors rely on Go's net.DialContext + DNS resolution; a broken DNS path produces an unwrapped `lookup` error.
- "Partial response" handling (truncated JSON / chunked-encoding mid-cert) is missing for non-ACME/StepCA connectors.
This matrix replaces the previous per-Part scattershot failure-mode coverage with a single audit-ready surface. When a new failure mode is added (e.g. Bundle J-extended adds Pebble-mock 429), update the cell + cite the test.
**Target connectors are NOT in this matrix** — they have a similar failure surface (deploy-time write/reload failures) but are tested under Parts 14–17 + 42–46. A separate target-connector failure matrix is tracked as a follow-on.
### 9.1 Issuer CRUD
**Test 6.1.1 — List issuers shows seed data**