certctl/cowork/ci-pipeline-cleanup/v2.X.0-release-notes.md

certctl v2.X.0 — CI Pipeline Cleanup

Operator-facing release notes for the ci-pipeline-cleanup master bundle. Operator picks the exact v2.X.0 from the increment-from-the-last-tag rule.

TL;DR

Restructured the on-push CI pipeline. Status checks per push drop from 19 → 7. ci.yml shrinks 1488 → ~430 lines (-71%). Three lying fields closed (staticcheck soft-gate; Bundle II's fabricated digest regex-only check; Windows matrix that validated nothing). Five new gates added (digest validity, go mod tidy drift, gofmt parity, OpenAPI ↔ handler parity, Docker build smoke).

Zero product behavior changes. No migrations, no API changes, no connector behavior changes. CI-only refactor.

What's new

scripts/ci-guards/ — extracted regression guards (Phase 1)

20 named regression guards moved from inline ci.yml bash to sibling scripts:

• G-1-jwt-auth-literal.sh, L-001-insecure-skip-verify.sh, H-001-bare-from.sh, M-012-no-root-user.sh, H-009-readme-jwt.sh, G-2-api-key-hash-json.sh, U-2-plaintext-healthcheck.sh, U-3-migration-mount.sh, D-1-D-2-statusbadge-phantom.sh, L-1-bulk-action-loop.sh, B-1-orphan-crud.sh, S-2-strings-contains-err.sh, G-3-env-docs-drift.sh, test-naming-convention.sh, S-1-hardcoded-source-counts.sh, P-1-documented-orphan-fns.sh, T-1-frontend-page-coverage.sh, bundle-8-L-015-target-blank-rel-noopener.sh, bundle-8-L-019-dangerously-set-inner-html.sh, bundle-8-M-009-bare-usemutation.sh

Each script is callable locally:

bash scripts/ci-guards/G-3-env-docs-drift.sh

CI step is a single loop that auto-picks up new scripts. Adding a new guard: drop a new <id>.sh; no ci.yml change required.

The 2 QA-doc guards (Part-count + seed-count) moved to make verify-docs instead — they protect docs-the-operator-reads, not anything the product depends on.

.github/coverage-thresholds.yml (Phase 2)

Per-package coverage floors moved out of inline bash into a YAML manifest. Each entry has floor: (integer percentage) + why: (load-bearing context — Bundle reference, HEAD measurement, gap rationale). Adding a new gated package: one YAML entry instead of ~30 lines of bash. Floors unchanged from HEAD.

staticcheck hard gate (Phase 3)

The old continue-on-error: true lying field with the "M-028 will close 6 SA1019 sites" comment is gone. Verified at HEAD: all live SA1019 sites either migrated (middleware.NewAuth → NewAuthWithNamedKeys) or suppressed inline with load-bearing rationale (csr.Attributes for RFC 2985 challengePassword; elliptic.Marshal only in byte-equivalence test). Gate now hard.

make verify parity + go mod tidy drift (Phase 4)

Two new steps in go-build-and-test:

• gofmt drift — closes the parity gap with Makefile::verify (CI was running vet + lint + test but not gofmt)
• go mod tidy drift — go mod tidy && git diff --exit-code go.mod go.sum

deploy-vendor-e2e collapsed: 12 jobs → 1 job (Phase 5)

Per-vendor matrix granularity was fake signal — verified that 115/116 vendor-edge tests are t.Log placeholders. Single job brings up all 11 sidecars at once + runs the full VendorEdge_ suite + enforces skip-count (no sidecar may silently fail to come up).

NEW scripts/ci-guards/vendor-e2e-skip-check.sh + allowlist file at scripts/ci-guards/vendor-e2e-skip-allowlist.txt (15 windows-iis- requiring tests legitimately skip on Linux per Phase 6).

Revises Bundle II frozen decision 0.9. Documented in cowork/ci-pipeline-cleanup/decisions-revised.md.

deploy-vendor-e2e-windows deleted entirely (Phase 6)

The Windows matrix can't physically work on windows-latest GitHub runners (Docker not started in Windows-containers mode by default; bridge network driver missing on Windows Docker — uses nat). Even if fixed, all 16 IIS + WinCertStore tests are t.Log placeholders.

NEW docs/connector-iis.md::Operator validation playbook documents the manual-on-Windows-host procedure operators run pre-release. The windows-iis-test sidecar stays in deploy/docker-compose.test.yml under profiles: [deploy-e2e-windows] for operator local use.

docs/deployment-vendor-matrix.md IIS + WinCertStore rows status updated pending → operator-playbook.

Revises Bundle II frozen decision 0.4. Documented in cowork/ci-pipeline-cleanup/decisions-revised.md.

NEW image-and-supply-chain job (Phases 7-9)

Top-level Ubuntu job (~3 min, parallel to go-build-and-test). Three steps:

1. Digest validity — every @sha256:<digest> ref in deploy/**/*.{yml,Dockerfile*} must resolve on its registry. Closes the H-001 lying-field gap (H-001 verifies digest presence only — Bundle II shipped 11 fabricated digests that passed H-001 and failed docker pull in CI).
2. Docker build smoke — all 4 Dockerfiles in the repo must build (Dockerfile, Dockerfile.agent, deploy/test/f5-mock-icontrol/Dockerfile, deploy/test/libest/Dockerfile).
3. OpenAPI ↔ handler operationId parity — every router route has a matching operationId in api/openapi.yaml or is documented in the new api/openapi-handler-exceptions.yaml (8 documented exceptions at HEAD: SCEP + SCEP-mTLS wire-protocol endpoints).

Coverage PR-comment action (Phase 10)

Self-hosted alternative to Codecov / Coveralls. Posts per-package coverage table as a PR comment; updates in place on subsequent pushes. No paid SaaS dependency.

make verify-docs + make verify-deploy (Phase 11)

Three-tier convention now:

• make verify — required pre-commit (gofmt + vet + lint + test)
• make verify-deploy — optional pre-push (digest validity + OpenAPI parity + Docker build smoke for server + agent)
• make verify-docs — required pre-tag (QA-doc Part-count + seed-count)

NEW docs/ci-pipeline.md (Phase 12)

Operator-facing guide to the on-push pipeline. Per-job deep-dive, guard inventory, threshold management, troubleshooting matrix, branch protection list to update.

Operator action required

After merge:

1. Update GitHub branch protection rule for master branch. Required-checks list changes from 19 entries → 7:

   • Go Build & Test
   • Frontend Build
   • Helm Chart Validation
   • deploy-vendor-e2e
   • image-and-supply-chain
   • Analyze (go)
   • Analyze (javascript-typescript)

2. (Optional) RAM-headroom verification on a test branch with the collapsed deploy-vendor-e2e job. If peak RSS > 12 GB on ubuntu-latest, fall back to bucketed matrix per cowork/ci-pipeline-cleanup/decisions-revised.md.

Rollback

If RAM headroom proves insufficient or a guard misbehaves:

• Vendor matrix collapse (Phase 5): revert that one commit; fall back to the bucketed-matrix design (3 jobs × ~4 sidecars).
• staticcheck hard gate (Phase 3): revert that one commit; flip continue-on-error: true back temporarily until the new SA1019 site is closed.
• All other phases are pure-additive or pure-extraction; reverting any single Phase commit restores the prior behavior.

Verification

make verify                          # pre-commit gate (existing)
make verify-deploy                   # optional pre-push (new)
make verify-docs                     # pre-tag (new)
bash scripts/ci-guards/*.sh          # all 20 guards locally
bash scripts/check-coverage-thresholds.sh  # only after coverage.out exists

All passing on HEAD.

Tag

Operator picks the exact v2.X.0 value. Bundle ships ~13 commits on master after the prior bundle's closing commit (HEAD 1de61e91).