The Rank 5 cloud-target seed rows in `seed_demo.sql` referenced a
non-existent `ag-server` agent_id. On every fresh-clone
`docker compose -f deploy/docker-compose.yml -f deploy/docker-compose.demo.yml up`
the server crash-looped at the demo-seed step:
pq: insert or update on table "deployment_targets" violates foreign
key constraint "deployment_targets_agent_id_fkey"
Origin: commit 9a7e818 ("docs, seed: cloud-target operator runbook +
AWS ACM / Azure KV demo seed rows") added the rows but didn't insert
or rebind to a matching agents row. The `ag-server` ID never existed
in seed_demo.sql or anywhere else.
Fix: bind the two cloud targets to the existing cloud sentinel agents
that were already inserted at lines 78-79 (alongside `cloud-gcp-sm`):
- tgt-aws-acm-prod → cloud-aws-sm
- tgt-azure-kv-prod → cloud-azure-kv
These cloud sentinels were inserted in commit 9a7e818's same family
specifically to back agentless cloud targets — exact semantic match.
Why the existing test didn't catch this:
TestRunDemoSeed_AppliesIdempotently in
internal/repository/postgres/seed_test.go calls the same RunSeed +
RunDemoSeed pair the server uses at boot, so it WOULD have caught the
FK violation. But the test depends on a live PostgreSQL container via
testcontainers-go and is gated under `testing.Short()` → the default
`go test ./... -short` lane that `make verify` runs always skipped it.
The dedicated integration lane that strips `-short` either wasn't run
on commit 9a7e818 or the failure was missed. Promoting the test out
from under `-short` is a separate hardening conversation (CI runs
need docker-in-docker which isn't free); that's out of scope for this
hotfix.
Static FK audit confirms the fix:
Defined agent IDs (12): ag-{data,edge-01,iis,k8s,lb,mac-dev,
web-prod,web-staging}-prod, cloud-{aws-sm,azure-kv,gcp-sm},
server-scanner
Referenced agent_id values in deployment_targets after fix:
ag-data-prod, ag-edge-01, ag-iis-prod, ag-k8s-prod, ag-lb-prod,
ag-web-prod, ag-web-staging, cloud-aws-sm, cloud-azure-kv
Unresolved: zero.
Acceptance gate (operator-side):
- docker compose -f deploy/docker-compose.yml \
-f deploy/docker-compose.demo.yml up -d --build
against a fresh clone — server boots clean within 30s, dashboard
at https://localhost:8443 shows the seeded demo data.
Strategic naming cleanup. Earlier doc-comments + commit messages framed Rank
4 / Rank 5 / Rank 7 work as 'Rank N of the 2026-05-03 Infisical deep-research
deliverable' — the 'Infisical' qualifier was a holdover from the original
deep-research framing where Infisical (a competing secrets-management
platform) was the comparator. Keeping the comparator's name in our source
adds noise without value; an external reader sees 'Infisical' and assumes a
dependency or shared lineage rather than reading it as the competitive
context it was.
Mechanical sed across 34 files (32 source / docs + 2 follow-up Python passes
to collapse 'deep-research deep-research' duplicates that emerged where the
original phrase wrapped across lines):
s|Infisical deep-research|deep-research|g
s|infisical-deep-research-results|deep-research-results-2026-05-03|g
s|infisical-deep-research-prompt|deep-research-prompt-2026-05-03|g
s|infisical-deep-research|deep-research|g
s|Infisical|deep-research|g
s|deep-research deep-research|deep-research|g # collapse-pass
Net diff: 63 insertions / 64 deletions across cmd/, docs/, internal/,
migrations/. Pure text substitution; zero behavior change. Code path
unchanged — go vet clean, tests for TestApproval pass on both
internal/service and internal/api/handler packages.
Workspace docs (cowork/) carry the same references and will be swept
separately — they're not under certctl/ git control. The two filename
references (cowork/infisical-deep-research-results.md +
cowork/infisical-deep-research-prompt.md) get renamed alongside that sweep
to deep-research-results-2026-05-03.md /
deep-research-prompt-2026-05-03.md so cross-references in the certctl
repo doc-comments resolve cleanly.
Wraps up Rank 5 of the 2026-05-03 Infisical deep-research deliverable
(commits edf6bee AWS + 8a56a78 Azure):
- docs/runbook-cloud-targets.md — sysadmin-grade flowchart spanning
the AWS ACM + Azure Key Vault deploy paths side-by-side. Covers
minimum IAM policy / RBAC role JSON, IRSA + AKS workload-identity
recipes, manual rollback recovery procedures (aws acm
import-certificate / az keyvault certificate import), CloudTrail
+ Activity Log forensics queries for "who wrote to this ARN /
vault cert", Prometheus cardinality + cost budget, and the
V3-Pro forward path (CloudFront / Front Door direct-attach,
ALB / App Gateway auto-bind, soft-delete recovery, GCP CM).
- migrations/seed_demo.sql — two new demo target rows (tgt-aws-
acm-prod + tgt-azure-kv-prod) so QA can exercise the per-cloud
wiring end-to-end against the demo seed without standing up
real cloud accounts.
cowork/WORKSPACE-ROADMAP.md (sibling-folder, not in this commit's
diff) was updated to mark the V2 AWS ACM + Azure KV connectors as
shipped and document the V3-Pro CloudFront / Front Door direct-attach
+ App Gateway auto-bind + soft-delete recovery + GCP CM follow-on
items.
cowork/infisical-deep-research-results.md (sibling-folder) Part 5
Rank 5 marked CLOSED with both commit SHAs.
Doc-only commit. No code changes.
Verified locally:
- go test -short -count=1 ./internal/connector/target/awsacm/...
./internal/connector/target/azurekv/... green.
- markdown lint clean against the Bundle 8 + Rank 4 runbook templates.
GitHub #10 reopened: operator mikeakasully cloned v2.0.50 fresh and ran the
canonical quickstart (docker compose -f deploy/docker-compose.yml up -d --build);
postgres reported unhealthy indefinitely, dependent containers never started.
Root cause: deploy/docker-compose.yml mounted a hand-curated subset of
migrations/*.up.sql + seed.sql into postgres /docker-entrypoint-initdb.d/.
Postgres applied them at initdb time. Once seed.sql referenced columns added
by migrations *after* the mounted cutoff (e.g., policy_rules.severity from
migration 000013), initdb crashed mid-seed and the container loop wedged.
Two sources of truth (compose mount list vs in-tree migration ladder)
diverged the moment a seed-touching migration shipped, and the only thing
that fixed it was hand-editing the compose file every release.
Fix: remove the dual source. Postgres boots empty; the server applies
migrations + seed at startup via RunMigrations + RunSeed. Helm has used
this pattern since day one (postgres-init emptyDir); compose now matches.
Bundled with four ride-along audit findings whose fixes share the same
schema/db code surface, so operators take the schema-change pain only once:
cat-u-seed_initdb_schema_drift [P1, primary] — initdb-mount fix
cat-o-retry_interval_unit_mismatch [P1] — column rename minutes→seconds
cat-o-notification_created_at_dead_field [P2] — add column + populate
cat-o-health_check_column_orphans [P1] — drop unwired columns
cat-u-no_version_endpoint [P2] — add /api/v1/version
Single migration (000017_db_coupling_cleanup) bundles the three schema
changes under a DO \$\$ guard so re-application is safe; reduces
operator-visible 'schema-change releases' from four to one.
Backend
- internal/repository/postgres/db.go: add RunSeed (baseline) + RunDemoSeed
(gated by CERTCTL_DEMO_SEED). Both idempotent (ON CONFLICT DO NOTHING in
every shipped INSERT) so repeated boots are safe; missing-file is no-op
so custom packaging that strips seeds still boots cleanly.
- cmd/server/main.go: invoke RunSeed (always) + RunDemoSeed (when flag set)
immediately after RunMigrations.
- internal/repository/postgres/notification.go: NotificationRepository.Create
now sets created_at (with time.Now() fallback when caller leaves it zero);
scanNotification reads it back; List + ListRetryEligible SELECT extended.
- internal/repository/postgres/renewal_policy.go: column references updated
to retry_interval_seconds across SELECT/INSERT/UPDATE sites.
- internal/api/handler/version.go: new VersionHandler exposes
{version, commit, modified, build_time, go_version} from
runtime/debug.ReadBuildInfo() with ldflags-supplied Version override.
- internal/api/router/router.go: register GET /api/v1/version through the
no-auth chain (CORS + ContentType) alongside /health, /ready,
/api/v1/auth/info.
- cmd/server/main.go: add /api/v1/version to no-auth dispatch + audit
ExcludePaths so rollout polling doesn't dominate the audit trail.
- internal/config/config.go: add DatabaseConfig.DemoSeed +
CERTCTL_DEMO_SEED env var.
Migration
- migrations/000017_db_coupling_cleanup.up.sql + .down.sql:
(1) renewal_policies.retry_interval_minutes → retry_interval_seconds
(DO \$\$ guard, idempotent re-application)
(2) notification_events ADD COLUMN created_at TIMESTAMPTZ
NOT NULL DEFAULT NOW()
(3) network_scan_targets DROP orphan health_check_enabled +
health_check_interval_seconds
- migrations/seed.sql: column reference updated to retry_interval_seconds.
- migrations/seed_demo.sql: same column rename + applied at runtime now via
RunDemoSeed (no longer initdb-mounted).
Compose
- deploy/docker-compose.yml: drop ALL initdb mounts (10 migration files +
seed.sql); add start_period: 30s to postgres + certctl-server healthchecks
to absorb the runtime migration + seed application window on first boot.
- deploy/docker-compose.test.yml: same drop (+ ghost seed_test.sql mount
removed; that file never existed); same healthcheck start_period.
- deploy/docker-compose.demo.yml: replace seed_demo.sql initdb mount with
CERTCTL_DEMO_SEED=true env var on certctl-server.
Tests
- internal/api/handler/version_handler_test.go: TestVersion_ReturnsBuildInfo,
TestVersion_RejectsNonGet, TestVersion_LdflagsOverride.
- internal/repository/postgres/seed_test.go: TestRunSeed_AppliesIdempotently,
TestRunSeed_MissingFileIsNoOp, TestRunDemoSeed_AppliesIdempotently,
TestMigration000017_RetryIntervalRename,
TestMigration000017_NotificationCreatedAt,
TestMigration000017_HealthCheckOrphansDropped (testcontainers, -short skips).
- internal/repository/postgres/notification_test.go:
TestNotificationRepository_CreatedAt_IsPersisted +
TestNotificationRepository_CreatedAt_DefaultsToNow.
CI guardrail
- .github/workflows/ci.yml: new 'Forbidden migration mount in compose initdb
(U-3)' step grep-fails the build if any migrations/*.sql or seed*.sql
re-appears in /docker-entrypoint-initdb.d in any compose file. Catches
future drift before a fresh-clone operator hits it.
Spec / Docs
- api/openapi.yaml: add /api/v1/version operation under Health tag.
- docs/architecture.md: replace the 'initdb may run the same SQL' paragraph
with a post-U-3 single-source-of-truth explanation.
- CHANGELOG.md: full unreleased-section entry covering all 5 closures,
breaking changes, and the new env var.
Audit doc
- coverage-gap-audit-2026-04-24-v5/unified-audit.md: add new P1 #14
cat-u-seed_initdb_schema_drift; flip the 4 ride-along findings to
✅ RESOLVED with closure prose pointing at this commit.
Verification: build/vet/test -short -race all clean across all touched
packages locally; govulncheck reports 0 vulnerabilities affecting our
code; OpenAPI YAML parses; CI U-3 grep guardrail clears against the
post-fix tree.
D-008 was a three-part drift in the policy engine that made the
D-005/D-006 remediation cosmetic below the DB layer:
(a) migrations/seed.sql INSERTed rules with pre-D-005 lowercase
types ('ownership', 'environment', 'lifetime', 'renewal_window')
that the handler validator rejects on Create/Update but that
raw SQL INSERTs bypassed entirely. At runtime evaluateRule's
switch fell through to the default "unknown policy rule type"
error branch on every demo rule × every cert × every cycle,
flooding logs while emitting zero violations.
(b) migrations/seed_demo.sql persisted lowercase severity values
('critical', 'error', 'warning') on policy_violations rows.
INSERT succeeded because that column had no CHECK, but any
frontend comparing against the canonical PolicySeverity enum
mis-categorized every seeded violation.
(c) evaluateRule hardcoded Severity: PolicySeverityWarning on
every emitted violation and ignored rule.Config entirely —
so the D-006 per-rule severity column (000013) and every
per-arm Config JSON ({allowed_issuer_ids, allowed_domains,
required_keys, allowed, lead_time_days, max_days}) was dead
data below the evaluation layer.
This commit lands (a)+(b)+(c) atomically. Shipping any subset
leaves the feature half-working.
## Changes
Domain (internal/domain/policy.go):
* Add PolicyTypeCertificateLifetime as the 6th TitleCase canonical.
Pre-D-008 the seeded "max-certificate-lifetime" rule had no engine
arm — routing it through RenewalLeadTime would conflate "how
close to expiry before we renew" with "how long can the cert
possibly be", two distinct semantics. The new type accepts
config {"max_days": int} and flags certs whose
NotAfter - NotBefore exceeds the cap.
Handler validator (internal/api/handler/validation.go):
* ValidatePolicyType allowlist grown to 6 canonicals
(AllowedIssuers, AllowedDomains, RequiredMetadata,
AllowedEnvironments, RenewalLeadTime, CertificateLifetime).
OpenAPI (api/openapi.yaml):
* PolicyType enum grown to match domain.
Frontend (web/src/api/types.ts, types.test.ts):
* POLICY_TYPES tuple gains CertificateLifetime; pin test asserts
all 6 canonicals and rejects casing drift.
Migration 000014 (policy_violations severity CHECK):
* Named CHECK constraint (policy_violations_severity_check)
mirroring 000013's allowlist, defense-in-depth at the DB layer
against future drift from bypassed writes (migrations, psql
sessions, future callers). Symmetric down migration drops by
name.
Seed data:
* migrations/seed.sql rewritten to emit TitleCase canonicals with
per-arm config JSON that actually exercises the config-consuming
paths (not the missing-field backstops):
- pr-require-owner → RequiredMetadata {"required_keys":["owner"]} Warning
- pr-allowed-environments → AllowedEnvironments {"allowed":["production","staging","development"]} Error
- pr-max-certificate-lifetime → CertificateLifetime {"max_days":90} Critical
- pr-min-renewal-window → RenewalLeadTime {"lead_time_days":14} Warning
Severities are now differentiated per rule (D-006 intent).
* migrations/seed_demo.sql violation rows flipped to TitleCase
severity ('Critical', 'Error', 'Warning') so migration 000014
applies cleanly on upgrade paths.
Engine rewrite (internal/service/policy.go):
* evaluateRule rewritten. All six arms now:
1. Parse rule.Config into the per-arm typed struct.
2. Bad JSON → log at ValidateCertificate boundary and skip
this rule (no co-located poisoning of other rules in the
same batch).
3. Empty/null Config → emit the pre-D-008 missing-field
violation (backwards compat invariant — operators who
haven't reconfigured still see the same output).
4. Violations emitted carry rule.Severity (no more hardcoded
Warning); D-006 column is now load-bearing.
* CertificateLifetime arm reads NotBefore/NotAfter from the
certificate's latest version via CertRepo. Injected via
PolicyService.SetCertRepo() setter — avoids churning ~36
NewPolicyService call sites while keeping the lifetime arm
optional (degrades to a log+skip if the setter is not wired).
Server wiring (cmd/server/main.go):
* policyService.SetCertRepo(certRepo) wired after construction.
Tests (internal/service/policy_test.go):
* 25 new subtests across 5 groups:
- TestEvaluateRule_SeverityPassThrough (6): every rule type
emits violations carrying rule.Severity, not hardcoded.
- TestEvaluateRule_ConfigConsumed (12): every per-arm Config
path exercised positive + negative.
- TestEvaluateRule_EmptyConfig_BackCompat (3): empty/null
Config still emits pre-D-008 missing-field violations.
- TestEvaluateRule_BadConfig_SkipsRule: malformed JSON logs
and skips cleanly without poisoning neighbors.
- TestEvaluateRule_CertificateLifetime_RepoScenarios (3):
ok when repo wired, log+skip when not, handles missing
NotBefore/NotAfter edges.
Provenance: D-008 surfaced during D-005/D-006 remediation review
in eef1db0. That commit added persistence and CI pins for the
severity field but did not re-verify the evaluation layer
consumed it; this finding and fix close the audit-process gap.
Add three new issuer connectors completing commercial and open-source CA
coverage. Entrust uses mTLS client certificate auth with sync/async
issuance. GlobalSign Atlas uses mTLS + API key/secret dual auth with
serial-based tracking. EJBCA supports dual auth (mTLS or OAuth2) for
self-hosted Keyfactor CAs.
Each connector implements the full issuer.Connector interface (9 methods),
includes httptest-based unit tests (~14 each), and follows established
patterns (injectable HTTP clients, RFC 5280 revocation reason mapping,
CRL/OCSP delegated to CA).
Also includes: issuer factory cases, env var seeding, config structs,
domain types, seed data (3 rows, all disabled), OpenAPI enum updates,
frontend issuer catalog entries with config fields, and full docs
(connectors.md, architecture.md, features.md, README).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rewrote docs/features.md from scratch as authoritative feature inventory
(1255 lines, every claim verified against source files).
Audited README.md and architecture.md against repo — fixed 19 stale
references: K8s Secrets status, issuer counts, dashboard page counts,
CI thresholds, missing connectors in Mermaid diagrams, OpenAPI operation
count, GetCACertPEM behavior, and V2/V4 roadmap accuracy.
Also includes related fixes discovered during audit:
- Scheduler skips expired/failed/revoked certs from auto-renewal
- Seed demo expiry dates moved outside 31-day scheduler query window
- Agent pages use correct last_heartbeat_at field name
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Google Cloud Certificate Authority Service integration via REST API
with OAuth2 service account auth (JWT→access token). Synchronous
issuance model, CA pool selection, mutex-guarded token caching,
revocation with RFC 5280 reason mapping. No Google SDK dependency —
all stdlib. 19 tests with httptest mock OAuth2 + CAS API.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Audit all docs and examples against current codebase state. Fix seed_demo.sql
domain constant casing (IssuerType, TargetType, AgentStatus) that would cause
agent dispatch failures. Fix example docker-compose health endpoints (/health
not /api/v1/health) and env var names (CERTCTL_DATABASE_URL). Update connector
counts, test numbers, and planned→implemented status across docs. Convert 3
ASCII flow diagrams to Mermaid.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add certificate export in PEM (JSON or file download) and PKCS#12 formats.
Private keys are never included — they stay on agents. Add EKU-aware
issuance threading profile EKUs (serverAuth, clientAuth, codeSigning,
emailProtection, timeStamping) through the full issuance pipeline. Fix
agent CSR SAN splitting for email addresses, adaptive KeyUsage flags for
S/MIME vs TLS, and a pre-existing generateID collision bug in deployment
job creation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three bugs fixed:
- Docker Compose only mounted migration 000001; migrations 000002-000007
(profiles, agent groups, revocation, discovery, network scans) never ran,
breaking half the demo features. Now mounts all 7 migrations in order.
- Network Scans page crashed with pq.Array scan error because lib/pq
doesn't support []int, only []int64. Changed Ports field accordingly.
- Dashboard pie chart displayed "RenewalInProgress" without spaces.
Added formatStatus() helper for PascalCase → spaced display.
Also adds first-run demo experience improvements:
- 9 discovered certificates (filesystem + network scan mix)
- 3 discovery scans with recent timestamps
- 2 AwaitingApproval renewal jobs for approval workflow demo
- CERTCTL_NETWORK_SCAN_ENABLED=true in Docker Compose
- Network scan targets seeded with last_scan results
- Version badge updated to v2.0.5
- Docs updated (quickstart, advanced demo) to reference seeded data
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
V2 audit found 3 critical number mismatches propagated across 8 files:
- Table count was 19 everywhere but actual migrations create 21 tables
- MCP tool count was 76 but tools.go registers 78 (M21/M22 additions)
- README MCP breakdown claimed 83 tools with math summing to 90
- architecture.md still had stale 860+ test count
- features.md OpenAPI claim said 93 ops but spec has 78
- mcp.md tool-per-domain table had wrong counts in 10 of 16 rows
- Added 3 network_scan_targets to seed_demo.sql for demo completeness
- Added curl examples to Agent Groups section in features.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Seed data used lowercase statuses ('active', 'expiring', 'expired',
'renewal_in_progress', 'failed') but the domain model uses PascalCase
('Active', 'Expiring', 'Expired', 'RenewalInProgress', 'Failed'). This
caused the dashboard pie chart to show two separate "active" segments
because the stats service only recalculates status for certs matching
the capitalized "Active" string.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The seed_demo.sql referenced nonexistent agent IDs (agent-web-1, agent-api-1,
agent-db-1) in the agent_group_members table, causing a FK constraint violation
on fresh database initialization. Fixed to use the actual agent IDs defined
earlier in the same file (ag-web-prod, ag-web-staging, ag-iis-prod).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sub-CA mode: Local CA loads CA cert+key from disk (CERTCTL_CA_CERT_PATH +
CERTCTL_CA_KEY_PATH) to operate as subordinate CA under enterprise root
(e.g., ADCS). Supports RSA, ECDSA, PKCS#8 keys. Validates IsCA and
KeyUsageCertSign. Falls back to self-signed when paths unset.
DNS-01 challenges: Pluggable DNSSolver interface with script-based hook
implementation. User-provided scripts create/cleanup _acme-challenge TXT
records for any DNS provider. Configurable propagation wait. Enables
wildcard certs and non-HTTP-accessible hosts.
step-ca connector: Smallstep private CA via native /sign API with JWK
provisioner auth. Issuance, renewal, revocation. Registered as iss-stepca.
23 new tests across 3 files. CI test path widened to ./internal/connector/issuer/...
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Agents now report OS, architecture, IP address, hostname, and version
via heartbeat using runtime.GOOS, runtime.GOARCH, and net.Dial. New
migration adds columns to agents table. Heartbeat handler, service,
and repository updated to accept and persist metadata. GUI shows
OS/Arch in agent list and full system info in agent detail page.
Apache httpd connector: separate cert/chain/key files, apachectl
configtest validation, graceful reload. HAProxy connector: combined
PEM file (cert+chain+key), optional config validation, reload.
Both wired into agent binary's target connector switch.
14 tests for new connectors. All existing tests updated for new
Heartbeat/UpdateHeartbeat signatures. Docs updated across README,
architecture, concepts, and connectors guides.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
shankar0123