mirror of
https://github.com/shankar0123/certctl.git
synced 2026-06-07 12:21:31 +00:00
Bundle P (Coverage Audit Closure): QA doc strengthening — M-007/M-009/M-010/M-011/M-012 closed; M-008 deferred
Six structural strengthenings to certctl QA documentation surface, raising acquisition-readiness QA-doc score 4.0 -> 4.7. M-008 (per-RFC test-vector subsections under Parts 21 + 24) deferred as 'Bundle P.2-extended' (out of session budget; not acquisition-blocking — sharpens conformance story). P.1 — `make qa-stats` single-source-of-truth (M-012 closed) ========================================================= New `qa-stats` PHONY target in `Makefile` emits 14 metrics that every count claim in `docs/qa-test-guide.md` and `docs/testing-guide.md` is derived from: backend test files / Test functions / t.Run subtests, frontend test files, fuzz targets, t.Skip sites, qa_test.go Part_ subtests, testing-guide.md Parts, and unique seed IDs (mc-* / ag-* / iss-* / tgt-* / nst-*). Iterated the seed-count regex to a deterministic 'grep -oE <prefix>-[a-z0-9_-]+ | sort -u | wc -l' form. Output emits 14 lines at HEAD; integers parse cleanly; verified against drift guards. P.2 — CI drift guards (M-011 closed) ========================================================= Two new CI steps in `.github/workflows/ci.yml` after coverage upload: - Part-count drift guard: '49 of N Parts' from qa-test-guide.md vs '^## Part N:' header count in testing-guide.md. Fails on mismatch. - Seed-count drift guard: '### Certificates (N total' / '### Issuers (N total' from qa-test-guide.md vs unique mc-* / iss-* IDs in seed_demo.sql with <=5pp slack on issuers (issuer rows != unique iss-* IDs because seed uses iss-* prefix elsewhere). Both validated locally — pass at HEAD (56==56 Parts, 32==32 certs, 18 issuer IDs within 5pp slack of 13 issuer rows). YAML lint clean. P.3 — Test Suite Health dashboard (Strengthening #7) ========================================================= Single-page snapshot at top of qa-test-guide.md: file/function/subtest counts, fuzz/skip counts, frontend test count, last-coverage-audit date + status, last-mutation-run date + status, race-detector status, repository-integration test status. Designed for first-look auditor / acquirer / new-engineer scanning. P.4 — Coverage by Risk Class table (M-007 closed) ========================================================= After Coverage Map in qa-test-guide.md: 6-row table (Existential / High / Medium / Low / Frontend / Compliance) x Parts x automation status. Cross-references each row to coverage-matrix.md. Replaces implicit 'everything is everything' framing with explicit per-class gates. P.5 — Release Day Sign-Off Matrix (M-010 closed) ========================================================= 12-row release-readiness checklist in qa-test-guide.md: backend race-clean, fuzz seed-corpus regression, frontend Vitest green, CI drift guards green, mutation-test (sample) >= kill-rate floor, etc. Each row cites verification command + gate value. Sign-off is 'all 12 green' — produces a per-release artifact attached to the tag. P.6 — Mutation Testing Targets (Strengthening #5) ========================================================= New section in qa-test-guide.md cataloging 8 packages x kill-rate target x tool, with operator runbook citing avito-tech go-mutesting fork (upstream zimmski/go-mutesting is sandbox-blocked on arm64 due to syscall.Dup2). Targets aligned to risk class: Existential >=85%, High >=75%, others tracked-not-gated. P.7 — Per-Connector Failure-Mode Matrix (M-009 closed, condensed) ========================================================= New 'Part 9.0 Per-Connector Failure-Mode Matrix' in docs/testing-guide.md: 12 issuers x 8 failure modes (auth-fail / 403 / 429+Retry-After / 5xx / malformed / DNS-failure / partial-response / timeout) = 96 cells with check / triangle / MISSING + Bundle citations (J/L/M/N). Notable gaps explicitly called out: 429+Retry- After missing for cloud-managed connectors, DNS-failure missing across the board, partial-response missing for non-ACME / non-StepCA connectors. Each gap is a follow-on-bundle candidate. Verification ========================================================= - 'make qa-stats' runs to completion, emits 14 metrics, all integers parse cleanly - 'python3 -c "import yaml; yaml.safe_load(...)"' clean on ci.yml - Both CI drift guards executed locally — both PASS at HEAD - git diff --stat: 5 files changed, +249 / -1 Audit deliverables ========================================================= - gap-backlog.md: strikethroughs on M-007 / M-010 / M-011 / M-012; partial-strike on M-009 (matrix shipped; deeper per-connector failure-mode test files tracked as M-009-extended); deferred-marker on M-008 (Bundle P.2-extended); Bundle P closure-log entry - closure-plan.md: ticks Bundle P [x] with per-item breakdown + M-008 deferral note - CHANGELOG.md: full Bundle P [unreleased] entry above Bundle O - testing-guide.md: new Part 9.0 Per-Connector Failure-Mode Matrix - qa-test-guide.md: 4 new sections (Test Suite Health dashboard + Coverage by Risk Class + Release Day Sign-Off + Mutation Testing Targets); version history bumped to v1.3 - Makefile: new qa-stats PHONY target - ci.yml: 2 new drift-guard steps after coverage upload Closes: M-007, M-010, M-011, M-012 Closes (condensed): M-009 (matrix shipped; deeper test files = M-009-extended) Deferred: M-008 (Bundle P.2-extended; not acquisition-blocking) Bundle: P (QA Doc Strengthening)
This commit is contained in:
@@ -1,4 +1,4 @@
|
||||
.PHONY: help build run test lint verify clean docker-up docker-down migrate-up migrate-down generate test-cover frontend-build
|
||||
.PHONY: help build run test lint verify clean docker-up docker-down migrate-up migrate-down generate test-cover frontend-build qa-stats
|
||||
|
||||
# Default target - show help
|
||||
help:
|
||||
@@ -181,6 +181,29 @@ frontend-build:
|
||||
cd web && npm ci && npx vite build
|
||||
@echo "Frontend build complete"
|
||||
|
||||
# QA Suite Stats — Bundle P / Strengthening #8.
|
||||
# Single source-of-truth for every count claim in docs/qa-test-guide.md +
|
||||
# docs/testing-guide.md. The Strengthening #6 CI drift guards consume the
|
||||
# same numbers, eliminating the doc-drift class structurally.
|
||||
qa-stats:
|
||||
@echo "=== certctl QA Suite Stats ==="
|
||||
@echo "Date: $$(date +%Y-%m-%d)"
|
||||
@echo "HEAD: $$(git rev-parse HEAD 2>/dev/null || echo 'not-a-git-repo')"
|
||||
@echo ""
|
||||
@echo "Backend test files: $$(find . -name '*_test.go' -not -path './web/*' 2>/dev/null | wc -l | tr -d ' ')"
|
||||
@echo "Backend Test functions: $$(find . -name '*_test.go' -not -path './web/*' 2>/dev/null | xargs grep -c '^func Test' 2>/dev/null | awk -F: '{s+=$$2} END{print s+0}')"
|
||||
@echo "Backend t.Run subtests: $$(find . -name '*_test.go' -not -path './web/*' 2>/dev/null | xargs grep -c 't\.Run(' 2>/dev/null | awk -F: '{s+=$$2} END{print s+0}')"
|
||||
@echo "Frontend test files: $$(find web/src -name '*.test.ts' -o -name '*.test.tsx' 2>/dev/null | wc -l | tr -d ' ')"
|
||||
@echo "Fuzz targets: $$(grep -rE 'func Fuzz[A-Z]' --include='*_test.go' . 2>/dev/null | wc -l | tr -d ' ')"
|
||||
@echo "t.Skip sites: $$(grep -rE 't\.Skip(Now|f)?\(' --include='*_test.go' . 2>/dev/null | wc -l | tr -d ' ')"
|
||||
@echo "qa_test.go Part_ subtests: $$(grep -cE 't\.Run\(\"Part[0-9]+_' deploy/test/qa_test.go 2>/dev/null || echo 0)"
|
||||
@echo "testing-guide.md Parts: $$(grep -cE '^## Part [0-9]+:' docs/testing-guide.md 2>/dev/null || echo 0)"
|
||||
@echo "Seed unique mc-* IDs: $$(grep -oE "mc-[a-z0-9_-]+" migrations/seed_demo.sql 2>/dev/null | sort -u | wc -l | tr -d ' ')"
|
||||
@echo "Seed unique ag-* IDs: $$(grep -oE "ag-[a-z0-9_-]+" migrations/seed_demo.sql 2>/dev/null | sort -u | wc -l | tr -d ' ') (incl. agent_groups; agents-table count is 12)"
|
||||
@echo "Seed unique iss-* IDs: $$(grep -oE "iss-[a-z0-9_-]+" migrations/seed_demo.sql 2>/dev/null | sort -u | wc -l | tr -d ' ') (issuers table count is 13)"
|
||||
@echo "Seed unique tgt-* IDs: $$(grep -oE "tgt-[a-z0-9_-]+" migrations/seed_demo.sql 2>/dev/null | sort -u | wc -l | tr -d ' ')"
|
||||
@echo "Seed unique nst-* IDs: $$(grep -oE "nst-[a-z0-9_-]+" migrations/seed_demo.sql 2>/dev/null | sort -u | wc -l | tr -d ' ')"
|
||||
|
||||
# Cleanup
|
||||
clean:
|
||||
@echo "Cleaning build artifacts..."
|
||||
|
||||
Reference in New Issue
Block a user