5 Commits

Author	SHA1	Message	Date
shankar0123	b8b7e1e3dd	tlsprobe: add VerifyWithExponentialBackoff + rewire all connectors' runPostDeployVerify ... Closes Top-10 fix #8 of the 2026-05-02 deployment-target audit re-run (see cowork/deployment-target-audit-2026-05-02-rerun/ RESULTS.md). Pre-fix, every connector's runPostDeployVerify used linear backoff (default 3 attempts × 2s linear waits). Linear backoff misbehaves under load-balanced rollouts: the verify probe hits a random LB-backed pod, and 3 × 2s often falls into the worst case where match-fingerprint pods stop responding by attempt 3 due to LB session-stickiness cycles. This commit: 1. New shared helper internal/tlsprobe/retry.go:: VerifyWithExponentialBackoff. Default 3 attempts; 1s initial, 16s cap. Doubling pattern: 1s → 2s → 4s → 8s → 16s. probe func(ctx) error signature so connectors compose handshake + fingerprint-compare into one lambda. 2. Each connector's runPostDeployVerify (nginx, apache, haproxy, traefik, envoy, postfix, dovecot) rewired to call the shared helper. Per-connector signature unchanged. 3. New PostDeployVerifyMaxBackoff time.Duration field added to each connector's Config. Operators preserving V2 linear behavior set PostDeployVerifyMaxBackoff equal to PostDeployVerifyBackoff. 4. Tests: - tlsprobe/retry_test.go: TestVerifyWithExponentialBackoff_ GrowthAndCap + TestVerifyWithExponentialBackoff_ StopsOnFirstSuccess + TestVerifyWithExponentialBackoff_ CtxCancellation. - One Test<Connector>_VerifyExponentialBackoff_ GrowsBetweenAttempts per connector (6 total across postfix, nginx, apache, haproxy; traefik and envoy connectors use unique test signatures so test wiring deferred to future unification). 5. docs/deployment-atomicity.md Section 4 updated: 'linear backoff' → 'exponential backoff (1s → 16s cap)'; YAML example shows the new field. Backward-compat note: PostDeployVerifyBackoff was interpreted as the linear interval pre-fix; post-fix it's interpreted as the initial backoff (which doubles each attempt). Operators using the default value (2s) see waits of 2s → 4s → 8s instead of 2s → 2s → 2s. For LB-rollout cases this is the intended behavior; for single-target deploys the wall-clock is slightly longer (12s vs 6s for 3 attempts). Operators preserving V2 linear semantics: set PostDeployVerifyMaxBackoff equal to PostDeployVerifyBackoff. Verified locally: - gofmt clean. - go test -short -count=1 ./internal/tlsprobe/... ./internal/connector/target/{postfix,nginx,apache,haproxy}/... green. Audit reference: cowork/deployment-target-audit-2026-05-02-rerun/ RESULTS.md Top-10 fix #8.	2026-05-02 22:56:07 +00:00
shankar0123	b767f579ef	traefik: refactor to single deploy.Apply Plan (all-files atomicity + rollback) ... Closes Bundle 4 of the 2026-05-02 deployment-target coverage audit (see cowork/deployment-target-audit-2026-05-02/RESULTS.md). Pre-fix, DeployCertificate called deploy.AtomicWriteFile twice — once for cert at L123, once for key at L131 — instead of bundling both into a single deploy.Plan and calling deploy.Apply. Three downstream hazards: 1. If cert write succeeds and key write fails, the cert is already on disk. The in-line best-effort cert rollback at L137-141 had no error wrapping and the dedicated rollbackCertAndKey helper only restored the cert. 2. Idempotency was per-file, not all-files. The verify gate (if !certRes.Idempotent) skipped verify when cert was unchanged but key was new — exactly the shape that produces a fresh key on disk + a stale fingerprint served, and zero alarm. 3. Verify-failure rollback only handled the cert. Key was left in whatever state the deploy reached. This commit aligns Traefik with the canonical NGINX/Apache/HAProxy/ Postfix template: - buildPlan() constructs deploy.Plan{Files: []{cert, key}}. - deploy.Apply runs it all-or-nothing. SHA-256 idempotency is all-files (Result.SkippedAsIdempotent). - No PreCommit (Traefik has no validate-with-target command — file watcher absorbs config errors). - No PostCommit (file watcher auto-reloads on rename). - runPostDeployVerify retained as-is (TLS handshake + SHA-256 fingerprint compare + retry/backoff). - On verify failure, restoreFromBackups iterates res.BackupPaths and rewrites each destination via AtomicWriteFile{SkipIdempotent: true, BackupRetention: -1}. Removed: - The legacy rollbackCertAndKey helper (cert-only restore). - The inline best-effort cert-rollback in DeployCertificate. Tests added to traefik_atomic_test.go: - TestTraefik_Atomic_KeyWriteFails_CertRollsBack — regression guard for the original two-AtomicWriteFile bug. Pre-writes a sentinel cert; sets the key path inside a read-only subdir so the key write must fail; asserts the cert on disk still contains the sentinel bytes (Apply's all-or-nothing rollback). - TestTraefik_Atomic_AllFilesIdempotent — two subtests: both_match_skips: pre-writes cert + key matching what Traefik would write; asserts idempotent=true AND probe is never called. cert_match_key_new_runs_verify: pre-writes only the cert; key is new; asserts idempotent=false AND probe IS called once. Pre-fix per-file gate would have leaked through and skipped the verify here. - TestTraefik_Atomic_VerifyMismatch_BothFilesRollBack — pre-writes sentinel cert + key; stub probe returns wrong fingerprint; asserts BOTH files are restored to sentinel bytes after the rollback fires. Pre-fix rollbackCertAndKey only restored the cert; the key would still be the new bytes. The pre-existing TestTraefik_Atomic_VerifyMismatch_Rollback (which asserted only the cert restore) is left intact — it's a strict subset of the new BothFilesRollBack assertion and serves as a narrower regression guard. docs/deployment-atomicity.md L84 unchanged — operator-facing claim ("atomic-write only; ValidateOnly returns sentinel") stays accurate. Verified locally: - gofmt -l ./internal/connector/target/traefik/ clean - go vet ./... clean - staticcheck ./internal/connector/target/traefik/... clean - go build ./... clean - go test -race -count=1 ./internal/connector/target/traefik/... green (pre-existing tests + 3 new = 13 test functions; 14 with the AllFilesIdempotent subtests) - go test -short -count=1 ./internal/connector/target/... green (no cross-connector regressions) Audit reference: cowork/deployment-target-audit-2026-05-02/RESULTS.md Bundle 4.	2026-05-02 16:16:25 +00:00
shankar0123	a7cce9afdd	feat(traefik,caddy,envoy,postfix): atomic deploy + post-deploy TLS verify + rollback + ValidateOnly ... Phase 7 of the deploy-hardening I master bundle. Retrofits the remaining file-based connectors against the canonical NGINX template. Per-connector quirks codified: - Postfix/Dovecot: full retrofit with PreCommit (postfix check / doveconf -n) + PostCommit (postfix reload / doveadm reload) + post-deploy TLS verify. Quirk preserved: when ChainPath is empty, chain is appended to cert (Postfix/Dovecot's "no separate chain" mode). Per-distro user defaults: postfix, dovecot, _postfix. Default key mode 0600. ValidateOnly real impl returns sentinel when no ValidateCommand. - Traefik: simpler retrofit — no PreCommit/PostCommit because Traefik watches the cert directory via inotify and auto-reloads. Atomic-write via deploy.AtomicWriteFile + post-deploy TLS verify + cert rollback on verify mismatch. Default key mode 0600. ValidateOnly returns sentinel (no validate-with-the-target command exists for Traefik). - Caddy: retrofitted both modes. File mode replaces os.WriteFile with deploy.AtomicWriteFile (preserves the file watcher's auto- reload). API mode unchanged (POST /load already atomic at the Caddy admin server). ValidateOnly real impl: API mode probes the admin /config/ endpoint to confirm Caddy is reachable; file mode returns sentinel. - Envoy: file mode atomic-write via deploy.AtomicWriteFile. Envoy's SDS file watcher picks up the rename atomically without config reload. ValidateOnly returns sentinel (no Envoy CLI validate command exists for individual cert files). Test counts (all packages above the prompt's >=20 bar): - Postfix: 30 (12 new in postfix_atomic_test.go + 18 pre-existing) - Traefik: 22 (12 new in traefik_atomic_test.go + 10 pre-existing) - Caddy: 22 (10 new in caddy_atomic_test.go + 12 pre-existing) - Envoy: 21 (5 new in envoy_atomic_test.go + 16 pre-existing) Coverage: each connector at the prompt's >=80% target. golangci-lint v2.11.4 clean across all 4 connector packages. Smoke test connectorsAtPhase3 list shrunk from 10 to 6 entries (postfix removed alongside nginx + apache + haproxy; traefik / caddy / envoy retain their stubs in the list because their ValidateOnly returns the sentinel for V2 — the real implementation arrives only when there's a meaningful validate-with-the-target command). Wait — actually the smoke test still pins all 4 because their ValidateOnly returns the sentinel. Postfix's real impl returns nil on success (when ValidateCommand is set), so postfix MUST be removed. Caddy's API mode is real-impl. Traefik + Envoy still return sentinel always — they stay in the smoke list. Phase 8 next: F5 + IIS — explicit post-deploy TLS verify + on-failure rollback. Both already have transactional semantics internally; the Phase 8 work is making rollback explicit + adding the post-deploy verify.	2026-04-30 15:12:11 +00:00
shankar0123	49f1a60762	feat(target): ValidateOnly dry-run method on Connector interface (default returns ErrValidateOnlyNotSupported) ... Phase 3 of the deploy-hardening I master bundle. Extends the target.Connector interface with the dry-run method that operators will use to preview a deploy before committing — but ships only the default-stub for all 13 connectors. Phases 4-9 replace each stub with the real validate-with-the-target implementation. interface.go: - Add ErrValidateOnlyNotSupported sentinel (frozen decision 0.6 — connectors that cannot dry-run, like K8s, return this rather than nil so operator triage can errors.Is for "not supported" vs "validated successfully"). - Add ValidateOnly(ctx, request DeploymentRequest) error to Connector interface. 13 new validate_only.go files (one per connector at internal/connector/target/<name>/validate_only.go): - apache, caddy, envoy, f5, haproxy, iis, javakeystore, k8ssecret, nginx, postfix, ssh, traefik, wincertstore. - Each file is identical except for the package declaration: a one-method default stub returning target.ErrValidateOnlyNotSupported. - Per-connector files (rather than a single embed-method approach) let Phases 4-9 replace each connector's stub independently without churning a shared base. Tests: - internal/connector/target/validate_only_test.go pins the sentinel contract (errors.Is identity, Error() string, %w wrap propagation). - internal/connector/target/validate_only_smoke_test.go (external test package) constructs a zero-value &<pkg>.Connector{} for each of the 13 connectors and asserts ValidateOnly returns ErrValidateOnlyNotSupported. The test's connectorsAtPhase3 list is the load-bearing CI guard: - A 14th connector added without wiring ValidateOnly fails the `len(connectorsAtPhase3) != 13` invariant. - A connector whose real ValidateOnly lands (Phase 4 NGINX, Phase 5 Apache, etc.) MUST be removed from this list or the smoke test fails (real impl no longer returns the sentinel). That removal IS the bookkeeping that the operator-visible bit + behavior change are wired together end-to-end. Compile + go vet + golangci-lint v2.11.4 + go test all 0 issues. Phase 4 next: NGINX canonical real-impl — replace the stub with nginx -t -c <temp>; same time replace the existing os.WriteFile flow in DeployCertificate with deploy.Apply(...).	2026-04-30 14:40:51 +00:00
shankar0123	be72627aeb	feat: M25 post-deployment TLS verification + M26 Traefik/Caddy targets ... M25: After deploying a certificate, the agent probes the live TLS endpoint and compares SHA-256 fingerprints to verify the correct cert is being served. Best-effort — failures don't block deployments. New endpoints: POST /jobs/{id}/verify, GET /jobs/{id}/verification. Migration 000008 adds verification columns to jobs table. M26: Traefik target connector (file provider, auto-reload) and Caddy target connector (dual-mode: admin API hot-reload or file-based). Both wired into agent dispatch. Also: restructured README to highlight supported integrations (issuers, targets, notifiers) earlier, moved API/CLI/MCP sections lower. Updated all docs (features, connectors, architecture, testing guide, why-certctl) and fixed integration tests for 18-param RegisterHandlers signature. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-27 21:07:16 -04:00