From 0692de8f28439c441d9aa6cace64175c86cb016d Mon Sep 17 00:00:00 2001 From: cowork Date: Mon, 27 Apr 2026 16:08:16 +0000 Subject: [PATCH] =?UTF-8?q?Bundle=20I=20(Coverage=20Audit=20Closure):=20QA?= =?UTF-8?q?-doc=20drift=20cleanup=20=E2=80=94=20H-007=20+=20H-008=20closed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Applies Patches 1-7 from coverage-audit-2026-04-27/tables/qa-doc-patches.md (Patch 5 re-anchored against actual HEAD seed counts after Phase 0 recon discovered the original patch's anticipated counts were themselves drifted). docs/qa-test-guide.md: - Patch 1: 'all 54 Parts' -> '49 of 56 Parts' + not-yet-automated callout - Patch 2: Totals line replaced with verified-2026-04-27 breakdown + recompute commands - Patch 3: Coverage Map gains Parts 23, 24, 55, 56 (each '0 (NOT AUTOMATED)') - Patch 4: 'Not Yet Automated' subsection added under 'What This Test Does NOT Cover' - Patch 5: Seed Data Reference re-anchored to authoritative HEAD counts: 32 certs (already correct), 12 agents (was 9), 13 issuers (was 9), 8 targets (already correct), 4 nst (already correct). Replaced narrow ID enumerations with sed | grep recompute commands. Added maintenance-note pointer to Strengthening #6 (CI guard). - Patch 6: Version History entry v1.2 added - Bonus: integration_test comparison row updated (12 agents + 13 issuers) deploy/test/qa_test.go (Patch 7): 4 new t.Run('PartN_*', ...) blocks for Parts 23, 24, 55, 56 — each calls t.Skip with a docs/testing-guide.md::Part N pointer + automation candidates. Skip-with-rationale form keeps Part numbering consistent + makes the manual-test pointer machine-readable. Replacing each Skip with a real test body is gap-backlog work. Verification: grep -cE '^## Part [0-9]+:' docs/testing-guide.md == 56 PASS grep -cE 't\.Run("Part[0-9]+_' deploy/test/qa_test.go == 53 PASS go vet -tags qa ./deploy/test/... PASS go test -tags qa -run='__nope__' ./deploy/test/... PASS (compile) (Full SKIP-grep gate requires the live demo stack; t.Skip bodies trivial.) Audit deliverables: findings.yaml: H-007 (-0014), H-008 (-0015) status open -> closed gap-backlog.md: strikethrough both rows + Bundle I closure-log entry tables/qa-doc-drift.md: 'PATCHES APPLIED' header marker (not retro-edited) acquisition-readiness.md: QA-doc rigor 2.5 -> 4.0 closure-plan.md: Bundle I checklist box ticked CHANGELOG.md: [unreleased] Bundle I entry --- CHANGELOG.md | 27 ++++++++++++++ deploy/test/qa_test.go | 40 ++++++++++++++++++++ docs/qa-test-guide.md | 84 ++++++++++++++++++++++++++++++++++-------- 3 files changed, 136 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1c0a3ec..dc0da72 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,33 @@ All notable changes to certctl are documented in this file. Dates use ISO 8601. ## [unreleased] — 2026-04-27 +### Bundle I (Coverage Audit Closure — QA Doc Cleanup): H-007 + H-008 closed + +> Applied Patches 1–7 from `coverage-audit-2026-04-27/tables/qa-doc-patches.md` to bring `docs/qa-test-guide.md` and `deploy/test/qa_test.go` back in sync with the code at HEAD. Acquisition-readiness QA-doc score lifts 2.5 → 4.0. + +`docs/qa-test-guide.md` updates: + +- **Patch 1 — Headline.** "covers all 54 Parts" → "49 of 56 Parts" + 4-not-yet-automated callout (Parts 23, 24, 55, 56). +- **Patch 2 — Totals line.** Replaced the static "~164 automated subtests" prose with a verified-2026-04-27 breakdown + recompute commands so the line stops drifting on every release. +- **Patch 3 — Coverage Map.** Added rows for Parts 23 (S/MIME & EKU), 24 (OCSP/CRL), 55 (Agent Soft-Retirement), 56 (Notification Retry & Dead-Letter) — each annotated "0 (NOT AUTOMATED)" with a `docs/testing-guide.md::Part N` pointer. +- **Patch 4 — What This Test Does NOT Cover.** New "Not Yet Automated (Parts 23, 24, 55, 56)" subsection enumerating the gaps and their manual-test rationale. +- **Patch 5 — Seed Data Reference.** Re-anchored against authoritative HEAD `migrations/seed_demo.sql` counts: **32 certs (already correct), 12 agents (was 9 — 8 named ag-* + server-scanner sentinel + 3 cloud-discovery sentinels), 13 issuers (was 9), 8 targets (already correct), 4 network scan targets (already correct).** Replaced narrow ID enumerations with `sed | grep` recompute commands so future seed additions don't silently drift the doc. Added a maintenance-note pointer to the proposed CI guard (Strengthening #6). Bundle I's Phase 0 recon discovered the original patch's anticipated counts (66 certs, 18 agents) were themselves drifted — the patch's recompute commands used overbroad regex that matched mc-* IDs across non-managed-certificates tables; corrected on the fly. +- **Patch 6 — Version History.** Added v1.2 entry citing Parts 55–56 documentation and Parts 23–24 not-yet-automated surfacing. +- Bonus fix: the integration_test comparison row "32 certs, 8 agents" → "32 certs, 12 agents, 13 issuers, 8 targets, realistic history". + +`deploy/test/qa_test.go` updates (Patch 7): + +- 4 new `t.Run("PartN_*", …)` blocks for Parts 23, 24, 55, 56. Each calls `t.Skip` with a `docs/testing-guide.md::Part N` pointer + automation-candidates list. The Skip-with-rationale form keeps Part numbering consistent in test output, makes the manual-test pointer machine-readable, and surfaces the gap to maintainers. Replacing each Skip with a real test body is gap-backlog work; this commit only closes the doc-vs-test drift. + +Verification gates met: +- `grep -cE '^## Part [0-9]+:' docs/testing-guide.md` == 56 ✓ +- `grep -cE 't\.Run\("Part[0-9]+_' deploy/test/qa_test.go` == 53 ✓ (49 live + 4 new Skip stubs) +- `go vet -tags qa ./deploy/test/...` clean +- `go test -tags qa -run='__nope__' ./deploy/test/...` PASS (compile) +- The full `go test -tags qa -run='TestQA/Part(23|24|55|56)' -v` SKIP-grep gate requires the live demo stack and is operator-runnable; the test bodies trivially `t.Skip` when reached. + +Audit deliverable updates: `findings.yaml` flips H-007 (`-0014`) and H-008 (`-0015`) status `open` → `closed` with closure_note + corrected counts; `gap-backlog.md` strikethroughs both rows + adds Bundle I closure-log entry; `tables/qa-doc-drift.md` gains a "PATCHES APPLIED 2026-04-27" header marker (preserved as audit-time snapshot, not retro-edited); `acquisition-readiness.md` "QA documentation rigor" criterion: 2.5 → 4.0; `coverage-audit-closure-plan.md` checklist ticks Bundle I. + ### Bundle 0.7 (Coverage Audit Closure): cmd/agent key-handling regression coverage — C-008 closed > Phase 0 of the 2026-04-27 coverage audit's closure plan triggered a halt-condition: `cmd/agent/keymem.go`'s two security-critical functions were at 0.0% / 11.1% line coverage despite being defense-in-depth for agent private-key memory hygiene (Bundle 9 / Audit L-002 + L-003 — agent edition). Bundle 0.7 was inserted before Bundle J as mandatory; this entry closes finding **C-008** (`CRTCTL-COVAUDIT-2026-04-27-0034`). diff --git a/deploy/test/qa_test.go b/deploy/test/qa_test.go index 7055c15..634ba76 100644 --- a/deploy/test/qa_test.go +++ b/deploy/test/qa_test.go @@ -1048,6 +1048,26 @@ func TestQA(t *testing.T) { }) }) + // =================================================================== + // Part 23: S/MIME & EKU Support — manual test (no automation yet) + // =================================================================== + t.Run("Part23_SMIMEEku", func(t *testing.T) { + t.Skip("Part 23 (S/MIME & EKU) is documented in docs/testing-guide.md::Part 23 " + + "as a manual test. Automation candidates: profile creation with SMIME EKU; " + + "issuance request with mismatched EKU should 400; issued cert MUST contain " + + "SMIMECapabilities extension when profile.allow_smime=true.") + }) + + // =================================================================== + // Part 24: OCSP Responder & DER CRL — manual test (no automation yet) + // =================================================================== + t.Run("Part24_OCSPCRL", func(t *testing.T) { + t.Skip("Part 24 (OCSP/CRL) is documented in docs/testing-guide.md::Part 24 " + + "as a manual test. Automation candidates: GET /.well-known/pki/ocsp/{issuer}/{serial} " + + "returns RFC 6960 OCSPResponse; DER CRL response is valid ASN.1 and signed by issuing CA; " + + "Must-Staple cert returns OCSP for fail-open relying parties.") + }) + // =================================================================== // Part 25: Certificate Discovery // =================================================================== @@ -1886,6 +1906,26 @@ func TestQA(t *testing.T) { fileContains(t, "migrations/seed_demo.sql", `iss-awsacmpca`) }) }) + + // =================================================================== + // Part 55: Agent Soft-Retirement (I-004) — manual test (no automation yet) + // =================================================================== + t.Run("Part55_AgentSoftRetire", func(t *testing.T) { + t.Skip("Part 55 (Agent Soft-Retirement) is documented in docs/testing-guide.md::Part 55 " + + "as a manual test. Automation candidates: POST /api/v1/agents/{id}/retire with " + + "soft=true does not delete; foreign-key cascade behavior on certs owned by retired " + + "agent; reactivation flow restores agent status.") + }) + + // =================================================================== + // Part 56: Notification Retry & Dead-Letter Queue (I-005) — manual test (no automation yet) + // =================================================================== + t.Run("Part56_NotificationDeadLetter", func(t *testing.T) { + t.Skip("Part 56 (Notification Retry/Dead-Letter) is documented in docs/testing-guide.md::Part 56 " + + "as a manual test. Automation candidates: notification with N consecutive failures " + + "transitions to status=DeadLetter; POST /api/v1/notifications/{id}/requeue resets to " + + "Pending; idempotency under concurrent retry; alert on dead-letter buildup.") + }) } // Note: uses Go 1.21+ built-in min() — no custom definition needed. diff --git a/docs/qa-test-guide.md b/docs/qa-test-guide.md index 0ad269e..09fd066 100644 --- a/docs/qa-test-guide.md +++ b/docs/qa-test-guide.md @@ -10,11 +10,13 @@ `deploy/test/qa_test.go` is a single Go test file (~1700 lines) that automates as much of `docs/testing-guide.md` as possible against a running certctl Docker Compose demo stack. It replaces the legacy `qa-smoke-test.sh` bash script. -It covers **all 54 Parts** of the testing guide: +It covers **49 of 56 Parts** of the testing guide as automation; the remaining 7 are +either manual-only by design or pending QA-suite coverage: -- **~164 automated subtests** — API calls, database queries, source file checks, performance benchmarks -- **11 skipped Parts** — with documented reasons (external CAs, Windows, browser-only, etc.) -- **Remaining ~282 manual tests** — GUI flows, scheduler timing, Docker log inspection — must be done by a human following `docs/testing-guide.md` +- **49 `Part_*` automation wrappers**, **~159 leaf subtests** — API calls, database queries, source file checks, performance benchmarks +- **11 fully skipped Parts** — with documented reasons (external CAs, Windows, browser-only, etc.) — see "What This Test Does NOT Cover" below +- **4 Parts NOT YET AUTOMATED** — Parts 23 (S/MIME & EKU), 24 (OCSP/CRL), 55 (Agent Soft-Retirement), 56 (Notification Retry & Dead-Letter) — must be tested manually per `docs/testing-guide.md` until QA-suite automation lands +- **Manual-only flows** in addition: GUI flows, scheduler timing, Docker log inspection — must be done by a human following `docs/testing-guide.md` ## Architecture @@ -118,6 +120,8 @@ This table shows what each Part tests and what's left for manual verification. | 20 | Post-Deployment Verification | 1 | 404 on nonexistent job verification | TLS probing, fingerprint comparison | | 21 | EST Server | 2 | CACerts (200 + content-type), CSRAttrs (200/204) | simpleenroll with CSR, simplereenroll, PKCS#7 parsing | | 22 | Certificate Export | 3 | PEM export, PKCS#12 export, 404 on nonexistent | Download mode, file content validation | +| 23 | S/MIME & EKU Support | 0 (NOT AUTOMATED) | — | S/MIME profile creation; EKU enforcement on issuance; SMIMECapabilities extension presence in issued cert; rejection of profile-violating EKU on CSR. Test manually per `docs/testing-guide.md::Part 23` | +| 24 | OCSP Responder & DER CRL | 0 (NOT AUTOMATED) | — | OCSP request/response (RFC 6960), DER CRL generation, status (Good/Revoked/Unknown), Must-Staple coordination. Test manually per `docs/testing-guide.md::Part 24` | | 25 | Certificate Discovery | 5 | List discovered, summary, list scan targets, create target, invalid CIDR 400 | Agent filesystem scan, claim/dismiss workflow | | 26 | Enhanced Query API | 4 | Sort descending, cursor pagination, time-range filter, invalid sort field | Field projection correctness, cursor token cycling | | 27 | Request Body Size Limits | 1 | 2MB body rejected (413/400) | Exact limit boundary (1MB) | @@ -147,8 +151,13 @@ This table shows what each Part tests and what's left for manual verification. | 52 | Helm Chart | 5 | Chart.yaml, values.yaml, 4 templates exist, securityContext, health probes | `helm template` rendering, `helm install` | | 53 | Kubernetes Secrets Target Connector (M47) | 18 | Config validation (namespace DNS-1123, secret name DNS subdomain, label keys, required fields), deployment (create/update Secret, chain concatenation, error propagation), validation (serial comparison, not-found, empty cert) | GUI target wizard KubernetesSecrets fields (namespace, secret_name, labels, kubeconfig_path), Helm RBAC toggle, TargetDetailPage type label | | 54 | AWS ACM Private CA Issuer Connector (M47) | 23 | Config validation (region, CA ARN regex, signing algorithm whitelist, validity_days, defaults), issuance (full flow, empty CSR, errors), renewal (reuses issuance), revocation (reason mapping, default, errors), GetOrderStatus completed, GetCACertPEM (success/chain/error), GetRenewalInfo nil | GUI issuer wizard AWSACMPCA fields (region, ca_arn, signing_algorithm, validity_days, template_arn), seed data visibility, create issuer flow | +| 55 | Agent Soft-Retirement (I-004) | 0 (NOT AUTOMATED) | — | Soft-retire vs hard-retire; force flag; reason capture; foreign-key cascade behavior on retired-agent cert ownership; reactivation. Test manually per `docs/testing-guide.md::Part 55` | +| 56 | Notification Retry & Dead-Letter Queue (I-005) | 0 (NOT AUTOMATED) | — | Retry loop with exponential backoff, dead-letter transition after N retries, requeue endpoint (`POST /api/v1/notifications/{id}/requeue`), idempotency on retry. Test manually per `docs/testing-guide.md::Part 56` | -**Totals:** ~164 automated subtests, 11 fully skipped Parts, ~282 manual tests remaining. +**Totals (verified 2026-04-27):** 49 `Part_*` automation wrappers, ~159 leaf subtests, 11 fully +skipped Parts, 4 Parts not yet automated (23, 24, 55, 56), and an unspecified count of manual-only +flows (GUI, scheduler timing, Docker log inspection). Run `grep -cE '^## Part [0-9]+:' docs/testing-guide.md` +and `grep -cE 't\.Run\("Part[0-9]+_' deploy/test/qa_test.go` to re-verify. ## Test Categories @@ -182,6 +191,17 @@ Timed API requests with threshold assertions: These gaps must be filled by manual testing per `docs/testing-guide.md`: +### Not Yet Automated (Parts 23, 24, 55, 56) + +These Parts are documented in `docs/testing-guide.md` but have no `Part_*` automation +in `qa_test.go` yet. They are operator-runnable from the manual playbook; QA-suite +automation should land before the next acquisition-grade release. + +- **Part 23: S/MIME & EKU Support** — profile-driven EKU enforcement; SMIMECapabilities extension +- **Part 24: OCSP Responder & DER CRL** — OCSP request/response correctness, CRL generation, Must-Staple coordination +- **Part 55: Agent Soft-Retirement (I-004)** — soft vs hard retire, FK cascade, reactivation +- **Part 56: Notification Retry & Dead-Letter Queue (I-005)** — retry semantics, dead-letter transition, requeue + ### External CA Integrations (Parts 10–13) - **Sub-CA mode** — requires CA cert+key files on disk - **ACME ARI** — requires a CA that supports RFC 9773 Renewal Information @@ -221,7 +241,7 @@ Both files live in `deploy/test/` in the same Go package (`integration_test`): | **Build tag** | `//go:build qa` | `//go:build integration` | | **Target stack** | Demo (`docker-compose.yml` + `docker-compose.demo.yml`) | Test (`docker-compose.test.yml`) | | **Port** | 8443 | Different (test stack config) | -| **Seed data** | `seed_demo.sql` (32 certs, 8 agents, realistic history) | Minimal (created by tests) | +| **Seed data** | `seed_demo.sql` (32 certs, 12 agents, 13 issuers, 8 targets, realistic history) | Minimal (created by tests) | | **CA backends** | Local CA only (demo mode) | Pebble ACME, step-ca, NGINX | | **Purpose** | Release QA — broad coverage, spot checks | Functional — end-to-end issuance, renewal, revocation against real CAs | | **Run frequency** | Before each release tag | CI on every PR | @@ -232,21 +252,54 @@ They are complementary. Integration tests prove the machinery works. QA tests pr The QA tests depend on `migrations/seed_demo.sql`. Key IDs used: -### Certificates (32 total) -`mc-api-prod`, `mc-web-prod`, `mc-pay-prod`, `mc-dash-prod`, `mc-data-prod`, `mc-search-prod`, `mc-admin-prod`, `mc-blog-prod`, `mc-docs-prod`, `mc-status-prod`, `mc-grpc-prod`, `mc-vault-prod`, `mc-consul-prod`, `mc-shop-prod`, `mc-auth-prod`, `mc-cdn-prod`, `mc-mail-prod`, `mc-ci-prod`, `mc-legacy-prod`, `mc-old-api`, `mc-wiki-prod`, `mc-api-stg`, `mc-web-stg`, `mc-pay-stg`, `mc-api-dev`, `mc-grafana-prod`, `mc-vpn-prod`, `mc-wildcard-prod`, `mc-compromised`, `mc-edge-eu`, `mc-k8s-ingress`, `mc-smime-bob` +### Certificates (32 total in `managed_certificates`) -### Agents (9 total) -`ag-web-prod`, `ag-web-staging`, `ag-lb-prod`, `ag-iis-prod`, `ag-data-prod`, `ag-edge-01`, `ag-k8s-prod`, `ag-mac-dev`, `server-scanner` (sentinel) +The full canonical list is generated by: +``` +sed -n '/^INSERT INTO managed_certificates/,/^;/p' migrations/seed_demo.sql \ + | grep -oE "^\s*\('mc-[a-z0-9_-]+" | sed -E "s/^\s*\('//" | sort -u +``` -### Issuers (9 total) -`iss-local`, `iss-acme-le`, `iss-stepca`, `iss-acme-zs`, `iss-openssl`, `iss-vault`, `iss-digicert`, `iss-sectigo`, `iss-googlecas` +Hand-listing is unsustainable as the seed grows; tests reference IDs by lookup, not by enumeration. +Sample IDs: `mc-api-prod`, `mc-web-prod`, `mc-pay-prod`, `mc-compromised`, `mc-smime-bob`, `mc-edge-eu`, `mc-k8s-ingress`, `mc-wildcard-prod`. See `migrations/seed_demo.sql:147` onward. -### Targets (8 total) +### Agents (12 total in `agents` table) + +8 named workload agents + 1 server-side sentinel + 3 cloud-discovery sentinels: + +- **Workload agents:** `ag-web-prod`, `ag-web-staging`, `ag-lb-prod`, `ag-iis-prod`, `ag-data-prod`, `ag-edge-01`, `ag-k8s-prod`, `ag-mac-dev` +- **Server-side sentinel:** `server-scanner` +- **Cloud-discovery sentinels:** `cloud-aws-sm`, `cloud-azure-kv`, `cloud-gcp-sm` + +Full list via: +``` +sed -n '/^INSERT INTO agents/,/^;/p' migrations/seed_demo.sql \ + | grep -oE "^\s*\('[a-z][a-z0-9_-]+" | sed -E "s/^\s*\('//" +``` + +(The `agent_groups` table also contains entries with `ag-*` IDs — `ag-linux-prod`, `ag-windows`, `ag-datacenter-a`, `ag-arm64`, `ag-manual` — but those are *group* IDs, not agents. Don't confuse the two.) + +### Issuers (13 total) + +`iss-local`, `iss-acme-le`, `iss-stepca`, `iss-acme-zs`, `iss-openssl`, `iss-vault`, `iss-digicert`, `iss-sectigo`, `iss-googlecas`, `iss-awsacmpca`, `iss-entrust`, `iss-globalsign`, `iss-ejbca`. + +Full list via: +``` +sed -n '/^INSERT INTO issuers/,/^;/p' migrations/seed_demo.sql \ + | grep -oE "^\s*\('iss-[a-z0-9_-]+" | sed -E "s/^\s*\('//" +``` + +### Targets (8 total in `deployment_targets`) `tgt-nginx-prod`, `tgt-nginx-staging`, `tgt-haproxy-prod`, `tgt-apache-prod`, `tgt-iis-prod`, `tgt-traefik-prod`, `tgt-caddy-prod`, `tgt-nginx-data` -### Network Scan Targets (4 total) +### Network Scan Targets (4 total in `network_scan_targets`) `nst-dc1-web`, `nst-dc2-apps`, `nst-dmz`, `nst-edge` +**Maintenance note:** when adding new seed rows, also update this section, OR remove the +per-table counts and rely on the `sed | grep` commands so the doc stops drifting on every +seed-data change. A CI guard that fails when the doc count diverges from the seed file is +proposed in `coverage-audit-2026-04-27/tables/qa-doc-strengthening.md` (Strengthening #6). + ## Troubleshooting ### "Server unreachable" on startup @@ -293,5 +346,6 @@ When a new feature ships: ## Version History -- **v1.0** (April 2026) — Initial release covering all 52 Parts of testing-guide.md v2.1. Replaces `qa-smoke-test.sh`. +- **v1.2** (April 2026, post-coverage-audit) — Documented Parts 55–56 (I-004 Agent Soft-Retirement, I-005 Notification Retry & Dead-Letter) and surfaced Parts 23–24 (S/MIME & EKU; OCSP/CRL) as not-yet-automated. 56 Parts total in `testing-guide.md`; 49 live `Part_*` automation wrappers in `qa_test.go` + 4 new `Skip` stubs for Parts 23/24/55/56 = 53 wrappers (Parts 15–17 remain covered by source-checks in Parts 42–46). Reconciled seed-data section to actual `seed_demo.sql` counts (12 agents, 13 issuers; certs were already accurate at 32). Bundle I of the 2026-04-27 coverage-audit closure plan. - **v1.1** (April 2026) — Added Parts 53–54 (M47: Kubernetes Secrets target + AWS ACM PCA issuer). 54 Parts total, ~164 automated subtests. +- **v1.0** (April 2026) — Initial release covering all 52 Parts of testing-guide.md v2.1. Replaces `qa-smoke-test.sh`.