GitHub #10 reopened: operator mikeakasully cloned v2.0.50 fresh and ran the
canonical quickstart (docker compose -f deploy/docker-compose.yml up -d --build);
postgres reported unhealthy indefinitely, dependent containers never started.
Root cause: deploy/docker-compose.yml mounted a hand-curated subset of
migrations/*.up.sql + seed.sql into postgres /docker-entrypoint-initdb.d/.
Postgres applied them at initdb time. Once seed.sql referenced columns added
by migrations *after* the mounted cutoff (e.g., policy_rules.severity from
migration 000013), initdb crashed mid-seed and the container loop wedged.
Two sources of truth (compose mount list vs in-tree migration ladder)
diverged the moment a seed-touching migration shipped, and the only thing
that fixed it was hand-editing the compose file every release.
Fix: remove the dual source. Postgres boots empty; the server applies
migrations + seed at startup via RunMigrations + RunSeed. Helm has used
this pattern since day one (postgres-init emptyDir); compose now matches.
Bundled with four ride-along audit findings whose fixes share the same
schema/db code surface, so operators take the schema-change pain only once:
cat-u-seed_initdb_schema_drift [P1, primary] — initdb-mount fix
cat-o-retry_interval_unit_mismatch [P1] — column rename minutes→seconds
cat-o-notification_created_at_dead_field [P2] — add column + populate
cat-o-health_check_column_orphans [P1] — drop unwired columns
cat-u-no_version_endpoint [P2] — add /api/v1/version
Single migration (000017_db_coupling_cleanup) bundles the three schema
changes under a DO \$\$ guard so re-application is safe; reduces
operator-visible 'schema-change releases' from four to one.
Backend
- internal/repository/postgres/db.go: add RunSeed (baseline) + RunDemoSeed
(gated by CERTCTL_DEMO_SEED). Both idempotent (ON CONFLICT DO NOTHING in
every shipped INSERT) so repeated boots are safe; missing-file is no-op
so custom packaging that strips seeds still boots cleanly.
- cmd/server/main.go: invoke RunSeed (always) + RunDemoSeed (when flag set)
immediately after RunMigrations.
- internal/repository/postgres/notification.go: NotificationRepository.Create
now sets created_at (with time.Now() fallback when caller leaves it zero);
scanNotification reads it back; List + ListRetryEligible SELECT extended.
- internal/repository/postgres/renewal_policy.go: column references updated
to retry_interval_seconds across SELECT/INSERT/UPDATE sites.
- internal/api/handler/version.go: new VersionHandler exposes
{version, commit, modified, build_time, go_version} from
runtime/debug.ReadBuildInfo() with ldflags-supplied Version override.
- internal/api/router/router.go: register GET /api/v1/version through the
no-auth chain (CORS + ContentType) alongside /health, /ready,
/api/v1/auth/info.
- cmd/server/main.go: add /api/v1/version to no-auth dispatch + audit
ExcludePaths so rollout polling doesn't dominate the audit trail.
- internal/config/config.go: add DatabaseConfig.DemoSeed +
CERTCTL_DEMO_SEED env var.
Migration
- migrations/000017_db_coupling_cleanup.up.sql + .down.sql:
(1) renewal_policies.retry_interval_minutes → retry_interval_seconds
(DO \$\$ guard, idempotent re-application)
(2) notification_events ADD COLUMN created_at TIMESTAMPTZ
NOT NULL DEFAULT NOW()
(3) network_scan_targets DROP orphan health_check_enabled +
health_check_interval_seconds
- migrations/seed.sql: column reference updated to retry_interval_seconds.
- migrations/seed_demo.sql: same column rename + applied at runtime now via
RunDemoSeed (no longer initdb-mounted).
Compose
- deploy/docker-compose.yml: drop ALL initdb mounts (10 migration files +
seed.sql); add start_period: 30s to postgres + certctl-server healthchecks
to absorb the runtime migration + seed application window on first boot.
- deploy/docker-compose.test.yml: same drop (+ ghost seed_test.sql mount
removed; that file never existed); same healthcheck start_period.
- deploy/docker-compose.demo.yml: replace seed_demo.sql initdb mount with
CERTCTL_DEMO_SEED=true env var on certctl-server.
Tests
- internal/api/handler/version_handler_test.go: TestVersion_ReturnsBuildInfo,
TestVersion_RejectsNonGet, TestVersion_LdflagsOverride.
- internal/repository/postgres/seed_test.go: TestRunSeed_AppliesIdempotently,
TestRunSeed_MissingFileIsNoOp, TestRunDemoSeed_AppliesIdempotently,
TestMigration000017_RetryIntervalRename,
TestMigration000017_NotificationCreatedAt,
TestMigration000017_HealthCheckOrphansDropped (testcontainers, -short skips).
- internal/repository/postgres/notification_test.go:
TestNotificationRepository_CreatedAt_IsPersisted +
TestNotificationRepository_CreatedAt_DefaultsToNow.
CI guardrail
- .github/workflows/ci.yml: new 'Forbidden migration mount in compose initdb
(U-3)' step grep-fails the build if any migrations/*.sql or seed*.sql
re-appears in /docker-entrypoint-initdb.d in any compose file. Catches
future drift before a fresh-clone operator hits it.
Spec / Docs
- api/openapi.yaml: add /api/v1/version operation under Health tag.
- docs/architecture.md: replace the 'initdb may run the same SQL' paragraph
with a post-U-3 single-source-of-truth explanation.
- CHANGELOG.md: full unreleased-section entry covering all 5 closures,
breaking changes, and the new env var.
Audit doc
- coverage-gap-audit-2026-04-24-v5/unified-audit.md: add new P1 #14
cat-u-seed_initdb_schema_drift; flip the 4 ride-along findings to
✅ RESOLVED with closure prose pointing at this commit.
Verification: build/vet/test -short -race all clean across all touched
packages locally; govulncheck reports 0 vulnerabilities affecting our
code; OpenAPI YAML parses; CI U-3 grep guardrail clears against the
post-fix tree.
Follow-up to v2.0.47 (HTTPS-Everywhere). The Phase-3 self-signed
bootstrap sidecar shipped an ed25519 server cert. Apple's TLS stack —
Safari Network Framework and the macOS-bundled LibreSSL 3.3.6
/usr/bin/curl — does not advertise ed25519 in the ClientHello
signature_algorithms extension for server certs, so the handshake fails
with the server-side log line:
tls: peer doesn't support any of the certificate's signature algorithms
Homebrew OpenSSL 3.x, Chrome, Firefox, and Linux curl all accept
ed25519 server certs fine. Apple is the outlier. Rather than gate the
demo stack behind "install Homebrew OpenSSL first," swap the bootstrap
algorithm to ECDSA-P256 with SHA-256 — universally supported, including
on the Apple stack.
Changes
- deploy/docker-compose.yml: certctl-tls-init openssl invocation swapped
to `-newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes`; header comment
+ echo line updated; multi-line rationale paragraph added.
- deploy/docker-compose.test.yml: same openssl swap + echo update for
the test harness sidecar that writes to the bind-mounted ./test/certs
directory the Go integration_test.go pins via CERTCTL_TEST_CA_BUNDLE.
- docs/tls.md: Pattern 1 description + code block updated;
"Why ECDSA-P256 and not ed25519" rationale paragraph added covering
pre-v2.0.48 history, the Apple diagnosis, accepting clients, and
the operator migration command. Patterns 2 (existing Secret) and 3
(cert-manager) explicitly called out as unaffected.
- docs/upgrade-to-tls.md: docker-compose procedure sentence updated
with cross-reference to tls.md Pattern 1.
- docs/test-env.md: "Get the CA bundle for curl" sentence updated.
Migration
Existing demo installs must tear the `certs` named volume down to pick
up the new algorithm:
docker compose -f deploy/docker-compose.yml down -v
docker compose -f deploy/docker-compose.yml up -d --build
Not touched
- cmd/server/tls.go: algorithm-agnostic. TLS 1.3 min version with
[X25519, P-256] curve preferences for key exchange is orthogonal to
the server cert's signature algorithm. No Go code change needed.
- Helm chart: Patterns 2 and 3 operators supply their own cert; this
patch does not affect them.
- Unrelated ed25519 uses (agent key algorithm detection, profile
algorithm options, SSH key path examples, tlsprobe key metadata,
cloud discovery key-algo display): all orthogonal to the server TLS
bootstrap cert.
Incidental cleanup
- .gitignore: dropped dangling `strategy.md` entry (file doesn't exist
in repo; entry was cruft).
Breaking change release. Plaintext HTTP listener removed. The certctl
control plane now terminates TLS 1.3 on :8443 via
http.Server.ListenAndServeTLS. No CERTCTL_TLS_ENABLED=false escape
hatch. No dual-listener mode. One-step cutover per docs/upgrade-to-tls.md.
Server
- cmd/server/tls.go: certHolder with SIGHUP hot-reload + atomic cert
swap, buildServerTLSConfig (TLS 1.3 min, GetCertificate callback),
preflightServerTLS validation
- cmd/server/main.go: ListenAndServeTLS in place of ListenAndServe,
watchSIGHUP wiring, cert/key path config threading
- tls_test.go: 418-line regression coverage of reload, preflight,
callback behavior, SAN validation
Config
- CERTCTL_TLS_CERT_PATH / CERTCTL_TLS_KEY_PATH (required)
- Plaintext rejection: agents/CLI/MCP pre-flight-fail on http://
URLs with a pointer to docs/upgrade-to-tls.md
Agents, CLI, MCP
- All three pre-flight-reject http:// URLs with fail-loud diagnostic
- CERTCTL_SERVER_CA_BUNDLE_PATH for private-CA trust
- CERTCTL_SERVER_TLS_INSECURE_SKIP_VERIFY for dev-only bypass
(loud warning on startup)
- install-agent.sh emits both vars as commented template lines
docker-compose
- certctl-tls-init sidecar generates SAN-valid self-signed cert into
deploy/test/certs/ on first boot
- All demo-stack curls pin against ca.crt with --cacert
Helm chart
- Three TLS provisioning modes, exactly one required:
- server.tls.existingSecret (operator-supplied)
- server.tls.certManager.enabled (cert-manager integration)
- server.tls.selfSigned.enabled (eval only — not for production)
- server-certificate.yaml template for cert-manager mode
- helm install without a TLS source fails at template render with
a pointer to docs/tls.md
CI
- .github/workflows/ci.yml Helm Chart Validation step renders the
chart in both existingSecret and cert-manager modes, plus an
inverse guard-regression test that asserts helm template MUST
refuse to render when no TLS source is configured. Previously
the single `helm template` invocation hit the certctl.tls.required
fail-loud guard and exit-1'd CI. Four invocations now: lint
(existingSecret), template (existingSecret), template
(cert-manager), template (no args — must fail).
Integration tests
- deploy/test/integration_test.go stands up the Compose stack over
HTTPS, extracts the CA bundle, and exercises every certctl API
over https://localhost:8443
- All 34 integration subtests green (per Phase 8 local CI-parity)
Documentation
- New: docs/tls.md (provisioning patterns, rotation, SIGHUP reload)
- New: docs/upgrade-to-tls.md (one-step cutover, no-downgrade
warnings, fleet-roll sequencing)
- CHANGELOG.md: v2.2.0 "HTTPS Everywhere — The Irony" entry
(file heading unchanged; release tag is v2.0.47)
- All curls in docs/, examples/, deploy/helm/ guides use
https://localhost:8443 --cacert
Verification
- grep -rn "ListenAndServe[^T]" cmd/ internal/ → 0 hits
- grep -rn "\"http://" cmd/ internal/ → 2 benign hits (Caddy admin
API default, SSRF doc comment) — zero certctl endpoints
- Tasks #197–#206 (Phases 0–8) all closed in the tracker
Files: 65 changed, 3489 insertions, 372 deletions (pre-CI-fix).
Addresses Medium finding M-4 in the audit report. The multi-stage
Dockerfiles previously had no ARG declarations for HTTP_PROXY,
HTTPS_PROXY, or NO_PROXY, so corporate-proxy environments silently
failed at 'npm ci' (frontend stage) and 'go mod download' (Go builder).
The npm retry idiom (`npm ci --include=dev || npm ci --include=dev`)
masked the failure because the upstream 'Exit handler never called!'
bug exits 0 despite the install crash.
Fix: thread HTTP_PROXY / HTTPS_PROXY / NO_PROXY ARGs through every
Docker build stage that performs network I/O, re-export them as ENV
with both upper- and lower-case aliases (apk/curl/npm read lowercase;
Go/Node read uppercase), and forward the host shell's environment via
`build.args:` in every compose file and `build-args:` in the release
workflow's docker/build-push-action steps. Defaults are empty strings
so un-proxied builds remain byte-identical to the pre-fix tree.
Scope: Dockerfile (frontend + Go builder stages), Dockerfile.agent
(Go builder stage), deploy/docker-compose.yml (server + agent),
deploy/docker-compose.dev.yml (server + agent), deploy/docker-compose.test.yml
(server + agent), .github/workflows/release.yml (both docker/build-push-action
v6 invocations). Zero Go, web, test, or runtime code changes. Zero
base-image changes. Existing npm `||` retry idiom and `ARG TARGETARCH`
preserved verbatim.
CWE-1173 (Improper Use of Validated Input) / CWE-16 (Configuration).
Verification:
- YAML parses clean across all four compose files and release.yml.
- yamllint -d relaxed: clean exit across all five YAML files.
- All six `build.args:` blocks expose HTTP_PROXY, HTTPS_PROXY, NO_PROXY
with default-empty ${VAR:-} substitution.
- Both release.yml docker/build-push-action steps expose the same
three keys sourced from ${{ secrets.HTTP_PROXY }}, etc.
- Dockerfiles contain 5 proxy ARG declarations total (Dockerfile has 2
stages × 3 ARGs = 6 lines, Dockerfile.agent has 1 stage × 3 ARGs = 3
lines); lowercase ENV aliases verified present in every stage.
- git diff --shortstat: 6 files changed, 117 insertions(+), 0 deletions.
Pure additive.
Docker-live verification (`docker build`, `docker compose config`)
deferred to CI / post-commit smoke because the sandbox has no Docker
runtime. hadolint, go, golangci-lint, govulncheck likewise unavailable
in the sandbox; per-layer CI coverage gates (service 55%, handler 60%,
domain 40%, middleware 30%) are trivially unaffected as M-4 touches
zero Go source files.
- New deploy/ENVIRONMENTS.md: comprehensive walkthrough of all 4 compose
files with service-by-service explanations, beginner-friendly Docker
concepts, and expert-level networking/config details
- Fix docker-compose.dev.yml: agent LOG_LEVEL → CERTCTL_LOG_LEVEL (was
silently ignored without the CERTCTL_ prefix)
- Add CERTCTL_CONFIG_ENCRYPTION_KEY to base and test compose (enables
M34/M35 dynamic issuer/target config encryption)
- Add CERTCTL_DISCOVERY_DIRS to base compose agent (enables filesystem
certificate discovery in default deployment)
- Cross-link ENVIRONMENTS.md from README doc table and quickstart.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
4-step wizard (Connect CA → Deploy Agent → Add Certificate → Done) shown
on fresh installs when no user-configured issuers or certificates exist.
Auto-seeded env var issuers (source="env") are excluded from first-run
detection. Wizard state latches to prevent query refetches from dismissing
it mid-flow. Split docker-compose into clean default (wizard-compatible)
and demo override (seed_demo.sql). Added missing migrations 000009/000010
to test compose.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three bugs fixed:
- Docker Compose only mounted migration 000001; migrations 000002-000007
(profiles, agent groups, revocation, discovery, network scans) never ran,
breaking half the demo features. Now mounts all 7 migrations in order.
- Network Scans page crashed with pq.Array scan error because lib/pq
doesn't support []int, only []int64. Changed Ports field accordingly.
- Dashboard pie chart displayed "RenewalInProgress" without spaces.
Added formatStatus() helper for PascalCase → spaced display.
Also adds first-run demo experience improvements:
- 9 discovered certificates (filesystem + network scan mix)
- 3 discovery scans with recent timestamps
- 2 AwaitingApproval renewal jobs for approval workflow demo
- CERTCTL_NETWORK_SCAN_ENABLED=true in Docker Compose
- Network scan targets seeded with last_scan results
- Version badge updated to v2.0.5
- Docs updated (quickstart, advanced demo) to reference seeded data
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- POSTGRES_PASSWORD and CERTCTL_API_KEY read from .env file
- Added deploy/.env.example with documentation
- Agent key volume (agent_keys) for key persistence across restarts
- Agent healthcheck via pgrep
- Resource limits: server 1CPU/512M, agent 0.5CPU/256M
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Private keys never leave agent infrastructure. Agents generate ECDSA P-256
key pairs locally, store them with 0600 permissions, and submit only the CSR
(public key) to the control plane. New AwaitingCSR job state pauses
renewal/issuance jobs until the agent submits its CSR. Server-side keygen
retained behind CERTCTL_KEYGEN_MODE=server for demo/development.
Key changes:
- Dual keygen mode via CERTCTL_KEYGEN_MODE (agent default, server for demo)
- AwaitingCSR job state with CommonName/SANs in work response
- Agent ECDSA P-256 keygen, local key storage, CSR-only submission
- CompleteAgentCSRRenewal server-side flow for agent-submitted CSRs
- DeploymentRequest.KeyPEM for agent-provided keys during deployment
- Dockerfile.agent creates /var/lib/certctl/keys with correct ownership
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Runtime fixes:
- Fix env var mismatch (CERTCTL_DB_URL → CERTCTL_DATABASE_URL)
- Fix table name mismatches (certificates → managed_certificates, notifications → notification_events)
- Add renewal_policy_id to certificate queries
- Remove non-existent created_at from notification queries
- Add env var fallback for agent CLI flags
- Graceful degradation for missing notifiers/issuers in demo mode
- Copy web/ directory in Dockerfile for dashboard serving
Service layer:
- Implement handler-service interface pattern across all services
- Wire up certificate, agent, job, policy, team, owner, audit, notification services
Documentation:
- Add concepts.md: beginner-friendly guide to TLS, CAs, private keys
- Rewrite quickstart.md with accurate API examples matching actual handlers
- Add demo-advanced.md: interactive demo with cert issuance and automated script
- Update architecture.md with correct table names and connector interfaces
- Update connectors.md to match actual Go interface signatures
- Update demo-guide.md with cross-references to new docs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
shankar0123