feat(apache): atomic deploy + post-deploy TLS verify + rollback + ValidateOnly + test-depth uplift to 34 tests

Phase 5 of the deploy-hardening I master bundle. Mirrors the Phase 4
NGINX template for Apache httpd. Test count lifts 3 → 34 (above the
prompt's >=30 target; matches and slightly exceeds the IIS bar).

Apache-specific quirks codified in apache.go:

- Validate command convention is `apachectl configtest` (NOT
  `apachectl -t` — that flag exists but configtest is the documented
  operator-facing form).
- Reload command convention is `apachectl graceful` for zero-
  downtime worker swap (NOT `apachectl restart` which drops
  in-flight TLS sessions).
- Per-distro user defaults: Debian/Ubuntu apache2, RHEL/CentOS
  apache, Alpine httpd. pickFirstExistingUser walks the list and
  picks the one that resolves on the host; falls back to no-chown
  when none exist (cross-distro portability without operator
  config; same approach as nginx).
- Default key file mode 0600 for back-compat with operators
  relying on the historical hard-coded value (matches the
  pre-Phase-5 implementation behavior).

DeployCertificate refactor:
- Replaces the duplicated os.WriteFile chain with deploy.Apply.
- PreCommit runs the operator's ValidateCommand via the test
  seam (which wraps `sh -c <cmd>` in production).
- PostCommit runs ReloadCommand the same way.
- Post-deploy TLS verify (frozen-decision-0.3 default ON when
  Endpoint is configured): probes the configured target,
  compares leaf cert SHA-256 against deployed bytes, retries with
  exponential backoff (default 3 attempts / 2s backoff for
  load-balanced targets).
- Rollback wires: reload-fail → restore backups + retry reload;
  verify-fail → restore backups + reload again. Second-failure
  surfaces ErrRollbackFailed for operator-actionable triage.

ValidateOnly real implementation replaces the Phase 3 stub.
Returns ErrValidateOnlyNotSupported when no ValidateCommand
configured; otherwise runs the validate-with-the-target command
without touching the live cert.

Test seams (SetTestRunValidate / SetTestRunReload / SetTestProbe)
allow tests to skip exec without `apachectl` on PATH; mirror the
nginx pattern.

Tests (34 total: 31 in apache_atomic_test.go + 3 pre-existing
in apache_test.go):

- Atomic invariants (happy, validate-fail-no-files-changed,
  reload-fail-rollback, rollback-also-fail-escalation)
- SHA-256 idempotency (full skip + partial-mismatch full-deploy)
- Post-deploy verify (match-success, mismatch-rollback,
  dial-timeout-rollback, retries-until-match,
  retries-exhausted-rollback, no-endpoint-skips, disabled-skips)
- Ownership / mode preservation (existing-mode, override-wins,
  default-key-0600, default-cert-0644)
- Backup retention (keeps-N, disabled-no-backups, backup-created)
- Concurrency (same-paths-serialize)
- ValidateOnly (happy, fails, no-command-sentinel, stderr-in-error)
- Edge cases (no-chain, no-key, ctx-cancelled, verify-rollback-
  reload, deployment-id-prefix, metadata-populated)

Coverage: Apache 86.6% (above the >=85% prompt bar). Race detector
clean. golangci-lint v2.11.4 clean.

Smoke test connectorsAtPhase3 list shrunk from 12 to 11
entries (apache removed; nginx + apache now have real impls).

Phase 6 next: HAProxy (combined PEM atomic write + `haproxy -c -f`
validate + uplift 3 → >=30).
This commit is contained in:
shankar0123
2026-04-30 14:56:23 +00:00
parent 7444df01e2
commit 12e5f97f59
4 changed files with 955 additions and 146 deletions
@@ -21,7 +21,7 @@ import (
"testing"
"github.com/shankar0123/certctl/internal/connector/target"
"github.com/shankar0123/certctl/internal/connector/target/apache"
// apache removed Phase 5 — real ValidateOnly implementation now in apache.go.
"github.com/shankar0123/certctl/internal/connector/target/caddy"
"github.com/shankar0123/certctl/internal/connector/target/envoy"
"github.com/shankar0123/certctl/internal/connector/target/f5"
@@ -64,7 +64,8 @@ var connectorsAtPhase3 = []struct {
// suite.
new func() target.Connector
}{
{"apache", func() target.Connector { return &apache.Connector{} }},
// apache removed Phase 5 — its ValidateOnly is now the real
// implementation; tested directly in apache/apache_atomic_test.go.
{"caddy", func() target.Connector { return &caddy.Connector{} }},
{"envoy", func() target.Connector { return &envoy.Connector{} }},
{"f5", func() target.Connector { return &f5.Connector{} }},
@@ -84,7 +85,7 @@ var connectorsAtPhase3 = []struct {
func TestEveryConnectorDefaultsToSentinel(t *testing.T) {
// Expected list size shrinks as Phases 4-9 land their real
// ValidateOnly implementations. Phase 4 removed nginx.
const expectedAtCurrentPhase = 12
const expectedAtCurrentPhase = 11
if len(connectorsAtPhase3) != expectedAtCurrentPhase {
t.Fatalf("connectors-at-phase list = %d entries, want %d (drift in the 13-connector inventory)", len(connectorsAtPhase3), expectedAtCurrentPhase)
}