Phase 0 closure prep: cowork/ holds the operator's internal
legal/audit/strategy artifacts — counsel-signed declaration, the
filter-repo callback for the history rewrite, the pre-rewrite bundle
backup, audit scratch HTML. These are private operator artifacts and
must never accidentally land in the public repo.
The public-facing description of the Phase 0 rewrite lives at
docs/history-normalization.md (separate commit, post-rewrite). This
gitignore entry is the pre-rewrite version so the rewrite's output
state has cowork/ ignored from commit 1.
Three findings from the certctl architecture diligence audit's Phase 1
bundle (Supply-Chain Hardening) closed together in one PR since they all
touch .github/workflows/ + repo root.
RED-1 — delete tracked precompiled binary
- deploy/test/f5-mock-icontrol/f5-mock-icontrol (8.6 MB ARM64 ELF) was
tracked alongside the Go source that builds it. The fixture's
Dockerfile already uses a multi-stage build that re-runs
'go build' inside the container (line 13), so the tracked binary
was vestigial — never actually consumed by the test wiring.
- git rm'd. Path added to .gitignore so it doesn't re-land.
- No Makefile target needed; the Dockerfile is the rebuild path.
RED-2 — SHA-pin every GitHub Action
- Pre: 37 of 41 'uses:' lines were tag-pinned (@v4 etc); only
4 were SHA-pinned (sigstore/cosign-installer + anchore/sbom-action).
- Post: 0 / 41. Every 'uses:' line is now '@<40-char-sha> # vN'
(the trailing comment preserves the human-readable version for
operator audit). SHA-pinning closes the standard supply-chain
attack vector against GitHub Actions consumers.
- SHAs resolved live via the GitHub API; spot-checked one.
TEST-L2 — npm audit hard gate
- Added 'npm audit --omit=dev --audit-level=high' step to the
Frontend Build job in ci.yml. --omit=dev excludes vitest/vite/
eslint/etc which don't ship to operators.
- Local run today: 0 vulnerabilities; gate enters with no triage
backlog. Catches future regressions.
New CI guards (regression-prevention):
- scripts/ci-guards/no-tag-pinned-actions.sh — fails the build if
a future PR adds 'uses: foo/bar@v2' instead of SHA-pinning.
- scripts/ci-guards/no-precompiled-binary.sh — runs file(1) over
git ls-files output; fails on any tracked ELF/Mach-O/PE.
- Both pass locally. CI's existing loop over scripts/ci-guards/*.sh
picks them up automatically.
Closes: cowork/certctl-architecture-diligence-audit.html#fix-RED-1,
cowork/certctl-architecture-diligence-audit.html#fix-RED-2,
cowork/certctl-architecture-diligence-audit.html#fix-TEST-L2
Follow-up to v2.0.47 (HTTPS-Everywhere). The Phase-3 self-signed
bootstrap sidecar shipped an ed25519 server cert. Apple's TLS stack —
Safari Network Framework and the macOS-bundled LibreSSL 3.3.6
/usr/bin/curl — does not advertise ed25519 in the ClientHello
signature_algorithms extension for server certs, so the handshake fails
with the server-side log line:
tls: peer doesn't support any of the certificate's signature algorithms
Homebrew OpenSSL 3.x, Chrome, Firefox, and Linux curl all accept
ed25519 server certs fine. Apple is the outlier. Rather than gate the
demo stack behind "install Homebrew OpenSSL first," swap the bootstrap
algorithm to ECDSA-P256 with SHA-256 — universally supported, including
on the Apple stack.
Changes
- deploy/docker-compose.yml: certctl-tls-init openssl invocation swapped
to `-newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes`; header comment
+ echo line updated; multi-line rationale paragraph added.
- deploy/docker-compose.test.yml: same openssl swap + echo update for
the test harness sidecar that writes to the bind-mounted ./test/certs
directory the Go integration_test.go pins via CERTCTL_TEST_CA_BUNDLE.
- docs/tls.md: Pattern 1 description + code block updated;
"Why ECDSA-P256 and not ed25519" rationale paragraph added covering
pre-v2.0.48 history, the Apple diagnosis, accepting clients, and
the operator migration command. Patterns 2 (existing Secret) and 3
(cert-manager) explicitly called out as unaffected.
- docs/upgrade-to-tls.md: docker-compose procedure sentence updated
with cross-reference to tls.md Pattern 1.
- docs/test-env.md: "Get the CA bundle for curl" sentence updated.
Migration
Existing demo installs must tear the `certs` named volume down to pick
up the new algorithm:
docker compose -f deploy/docker-compose.yml down -v
docker compose -f deploy/docker-compose.yml up -d --build
Not touched
- cmd/server/tls.go: algorithm-agnostic. TLS 1.3 min version with
[X25519, P-256] curve preferences for key exchange is orthogonal to
the server cert's signature algorithm. No Go code change needed.
- Helm chart: Patterns 2 and 3 operators supply their own cert; this
patch does not affect them.
- Unrelated ed25519 uses (agent key algorithm detection, profile
algorithm options, SSH key path examples, tlsprobe key metadata,
cloud discovery key-algo display): all orthogonal to the server TLS
bootstrap cert.
Incidental cleanup
- .gitignore: dropped dangling `strategy.md` entry (file doesn't exist
in repo; entry was cruft).
Breaking change release. Plaintext HTTP listener removed. The certctl
control plane now terminates TLS 1.3 on :8443 via
http.Server.ListenAndServeTLS. No CERTCTL_TLS_ENABLED=false escape
hatch. No dual-listener mode. One-step cutover per docs/upgrade-to-tls.md.
Server
- cmd/server/tls.go: certHolder with SIGHUP hot-reload + atomic cert
swap, buildServerTLSConfig (TLS 1.3 min, GetCertificate callback),
preflightServerTLS validation
- cmd/server/main.go: ListenAndServeTLS in place of ListenAndServe,
watchSIGHUP wiring, cert/key path config threading
- tls_test.go: 418-line regression coverage of reload, preflight,
callback behavior, SAN validation
Config
- CERTCTL_TLS_CERT_PATH / CERTCTL_TLS_KEY_PATH (required)
- Plaintext rejection: agents/CLI/MCP pre-flight-fail on http://
URLs with a pointer to docs/upgrade-to-tls.md
Agents, CLI, MCP
- All three pre-flight-reject http:// URLs with fail-loud diagnostic
- CERTCTL_SERVER_CA_BUNDLE_PATH for private-CA trust
- CERTCTL_SERVER_TLS_INSECURE_SKIP_VERIFY for dev-only bypass
(loud warning on startup)
- install-agent.sh emits both vars as commented template lines
docker-compose
- certctl-tls-init sidecar generates SAN-valid self-signed cert into
deploy/test/certs/ on first boot
- All demo-stack curls pin against ca.crt with --cacert
Helm chart
- Three TLS provisioning modes, exactly one required:
- server.tls.existingSecret (operator-supplied)
- server.tls.certManager.enabled (cert-manager integration)
- server.tls.selfSigned.enabled (eval only — not for production)
- server-certificate.yaml template for cert-manager mode
- helm install without a TLS source fails at template render with
a pointer to docs/tls.md
CI
- .github/workflows/ci.yml Helm Chart Validation step renders the
chart in both existingSecret and cert-manager modes, plus an
inverse guard-regression test that asserts helm template MUST
refuse to render when no TLS source is configured. Previously
the single `helm template` invocation hit the certctl.tls.required
fail-loud guard and exit-1'd CI. Four invocations now: lint
(existingSecret), template (existingSecret), template
(cert-manager), template (no args — must fail).
Integration tests
- deploy/test/integration_test.go stands up the Compose stack over
HTTPS, extracts the CA bundle, and exercises every certctl API
over https://localhost:8443
- All 34 integration subtests green (per Phase 8 local CI-parity)
Documentation
- New: docs/tls.md (provisioning patterns, rotation, SIGHUP reload)
- New: docs/upgrade-to-tls.md (one-step cutover, no-downgrade
warnings, fleet-roll sequencing)
- CHANGELOG.md: v2.2.0 "HTTPS Everywhere — The Irony" entry
(file heading unchanged; release tag is v2.0.47)
- All curls in docs/, examples/, deploy/helm/ guides use
https://localhost:8443 --cacert
Verification
- grep -rn "ListenAndServe[^T]" cmd/ internal/ → 0 hits
- grep -rn "\"http://" cmd/ internal/ → 2 benign hits (Caddy admin
API default, SSRF doc comment) — zero certctl endpoints
- Tasks #197–#206 (Phases 0–8) all closed in the tracker
Files: 65 changed, 3489 insertions, 372 deletions (pre-CI-fix).
Critical alerts can no longer be silently dropped by a transient
notifier failure. Failed notification attempts now ride an exponential
backoff retry loop, with a 5-attempt budget before promotion to the
dead-letter queue for operator intervention.
Schema (migration 000016, idempotent):
- retry_count INTEGER NOT NULL DEFAULT 0
- next_retry_at TIMESTAMPTZ
- last_error TEXT
- idx_notification_events_retry_sweep partial index
(next_retry_at) WHERE status='failed' AND next_retry_at IS NOT NULL
Dead rows clear next_retry_at so the index stops matching them.
Service contract:
- NotificationService.RetryFailedNotifications drives 2^n-minute
exponential backoff capped at 1h (notifRetryBackoffCap) with
5-attempt budget (notifRetryMaxAttempts).
- Exhaustion (RetryCount >= notifRetryMaxAttempts-1) promotes to
status='dead' via MarkAsDead.
- Non-terminal failures record via RecordFailedAttempt.
- Success path promotes to 'sent' without touching retry_count
(audit preserves "delivered on attempt N").
- Missing-notifier branch defensively promotes to 'sent' to avoid
wedging a row on a deleted channel.
- RequeueNotification operator escape hatch atomically resets
retry_count -> 0, next_retry_at -> NULL, last_error -> NULL,
status -> pending via notifRepo.Requeue.
Scheduler:
- New always-on notificationRetryLoop wired into the base loop set at
CERTCTL_NOTIFICATION_RETRY_INTERVAL (default 2m).
- sync/atomic.Bool idempotency guard.
- sync.WaitGroup shutdown drain via WaitForCompletion.
StatsService:
- SetNotifRepo setter pattern preserves 9 pre-existing
NewStatsService call sites (main.go + stats_test.go + 8 digest
tests) without touching the constructor signature.
- DashboardSummary.NotificationsDead populated via
notifRepo.CountByStatus(ctx, "dead") — nil-safe when unwired
(reports zero on systems without a notification repository).
- CountByStatus error is non-fatal (dashboard summary is
best-effort for this field).
- Prometheus certctl_notification_dead_total counter emitted from
the same snapshot.
Handler:
- New POST /api/v1/notifications/{id}/requeue endpoint.
- dead status surfaces to MCP + CLI.
Frontend:
- NotificationsPage gains two-tab toolbar ("All" / "Dead letter")
with queryKey: ['notifications', activeTab] so switching tabs
doesn't serve stale data until the 30s refetch.
- Dead rows surface "Retry {n}/5" + truncated last_error with
full-text title tooltip.
- Requeue mutation wrapped as
mutationFn: (id: string) => requeueNotification(id)
to prevent react-query v5's positional context argument from
leaking into the API client — pinned against future refactors
by strict-match toHaveBeenCalledWith('notif-dead-001') in
NotificationsPage.test.tsx:181.
Closes I-005.
CI failure on master (commit 3287e17) — staticcheck ST1020:
internal/api/middleware/middleware.go:125:1: ST1020: comment on exported
function NewAuthWithNamedKeys should be of the form
"NewAuthWithNamedKeys ..." (staticcheck)
When NewAuth was renamed to NewAuthWithNamedKeys during the M-002 auth
unification, the leading godoc sentence was left pointing at the old name.
Rewrite the comment so its first sentence starts with the new function
name, and expand the body to describe the named-key + admin-flag contract
introduced in 3287e17.
Also gitignore /.gopath/ — session-scoped tool install cache, same
category as /.gocache/ and /.gomodcache/.
Verification:
go vet ./internal/api/middleware/... — clean
go build ./internal/api/middleware/... — clean
go test ./internal/api/middleware/... — PASS (0.245s)
staticcheck -checks=all,<project exclusions> — clean across
middleware, handler, service, domain, cmd/server, scheduler
Closes: CI failure on 3287e17.
Closes the remaining P1 gaps from coverage-gap-audit.md (M-001/M-002/M-003/M-006)
on top of the C-001/C-002 ownership + agent-FK contract fixes landed in
a53a4b8. The work lands as a single commit spanning server, docs, tests,
and the React client.
M-002 — Named API keys with per-key actor propagation
* Migration 000014 adds the 'api_keys' table (id, name, hash,
principal, role, created_at, last_used_at, disabled_at) so every
credential carries an identifiable principal instead of the
opaque 'anonymous'/'api-key' sentinel.
* Auth middleware now rotates through configured keys, performs
constant-time hash comparison, stamps 'last_used_at', and emits
an actor struct via contextWithActor(). The audit middleware,
bulk-revocation handler, approval handlers, and MCP tool layer
now read the principal off the context and persist it on every
audit_events row.
* Regression coverage:
- internal/api/middleware/audit_test.go — actor propagation,
principal redaction for disabled keys, anonymous fallback for
unauthenticated endpoints.
- internal/api/handler/bulk_revocation_handler_test.go,
job_handler_test.go — principal-on-audit assertions.
M-003 — Authorization gates (Phase B)
* Approval handler rejects self-approval / self-rejection with 403
when the actor principal equals the job's requested_by field.
* Bulk revocation is gated behind the 'admin' role; operators and
viewers receive 403.
* Regression coverage:
- internal/service/job_test.go — TestApproveJob_NotSelf,
TestRejectJob_NotSelf.
- internal/api/handler/bulk_revocation_handler_test.go —
TestBulkRevoke_RequiresAdmin, TestBulkRevoke_AdminSucceeds.
M-006 — RFC-compliant CRL/OCSP on the unauthenticated .well-known mux
* Per RFC 8615, relying parties cannot reasonably be asked to
authenticate against the issuing certctl instance to retrieve
revocation material. CRL and OCSP move off the authenticated
'/api/v1/crl*' and '/api/v1/ocsp/*' paths onto:
GET /.well-known/pki/crl/{issuer_id}
Content-Type: application/pkix-crl (RFC 5280 §5)
GET /.well-known/pki/ocsp/{issuer_id}/{serial}
Content-Type: application/ocsp-response (RFC 6960)
* Non-standard JSON CRL shape is removed; only DER is served.
* Short-lived certificate exemption (profile TTL < 1h → skip
CRL/OCSP) is preserved; the response simply omits the serial.
* Routes are registered on the unauthenticated 'finalHandler' mux
in cmd/server/main.go alongside EST ('/.well-known/est/*') and
SCEP ('/scep'). Legacy authenticated paths return 404.
* Regression coverage:
- internal/api/handler/certificate_handler_test.go — content
type, DER parseability, 404 for unknown issuer.
- internal/api/handler/adversarial_path_test.go — unauthenticated
access asserted for CRL, OCSP, EST, SCEP.
- internal/api/router/router_test.go — route-table assertion
that '.well-known/pki/*', '.well-known/est/*', and '/scep' are
mounted on the unauthenticated branch.
M-001 — Auto-closed by M-002
EST and SCEP were already registered on the unauthenticated
'finalHandler' mux; the router comment at
internal/api/router/router.go:247 now matches reality. The
adversarial-path tests above lock the behavior in.
Verification (all gates green):
* go vet ./... — clean
* go build ./... — ok
* go test -short ./... (55+ packages) — all pass
* web/ : npm test (225 Vitest tests) — all pass
* web/ : npx tsc --noEmit — clean
* grep sweep for '/api/v1/(crl|ocsp)' — 13 surviving hits,
all intentional M-006 tombstone/relocation comments.
Documentation:
* coverage-gap-audit.md — status flips M-001/M-002/M-003/M-006 →
Fixed, with per-finding resolution paragraphs citing regression
test IDs. (Audit file lives outside this repo; see cowork root.)
* CLAUDE.md Project Status line updated with the auth-unification
closure note.
* docs/features.md, docs/architecture.md, docs/quickstart.md,
docs/concepts.md, docs/connectors.md, docs/test-env.md,
docs/testing-guide.md, docs/compliance-*.md, docs/demo-advanced.md
— refreshed for the new '.well-known/pki/*' namespace and named
API keys.
* api/openapi.yaml — documents the new unauthenticated endpoints
and removes the legacy '/api/v1/crl*' + '/api/v1/ocsp/*' paths.
.gitignore: adds '/.gocache/' and '/.gomodcache/' for the session-
scoped Go caches so they never enter the tree.
- HSM/TPM agent key storage and CA key storage moved from V5+ to V3 Pro
(enterprise compliance gate, not adoption driver)
- Renamed roadmap.md to strategy.md (gitignored, never committed)
- Updated compliance-nist.md HSM references from V5 to V3 Pro
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Frontend audit (10 categories): lifecycle fields in types, new API
functions (CRL, OCSP, deployments, updateIssuer/Target, getPolicy),
issuer/owner/profile filters on CertificatesPage, last_renewal_at
column, error_message column on JobsPage, full crypto policy UI on
ProfilesPage (key algorithms, EKUs, SAN patterns), key info + CA
badge on DiscoveryPage, edit modal on TargetDetailPage, tags field
on certificate creation, darwin→macOS mapping on AgentFleetPage.
211 Vitest tests passing.
README accuracy: test counts (1300+ Go, 211 frontend), page count
(24), demo data (32 certs, 7 issuers, 180 days), endpoint count
(97), MCP tools (80), CLI subcommands (10), moved shipped items
out of "Coming in v2.1.0".
Docs: architecture.md diagrams updated (Vault PKI, DigiCert,
Traefik, Caddy added), features.md Vault/DigiCert status updated.
Version bumped to v2.0.20. cli binary removed from git tracking.
Testing guide Part 41 added (12 auto + 9 manual tests).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Use type conversion DigestStatusCount(c) instead of struct literal
- Replace with...else-if (invalid in Go templates) with if...else-if chain
- Add *.bak and cmd/agent/*.key/*.pem to .gitignore
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements 3 deferred security tickets (TICKET-003, TICKET-007, TICKET-010)
and performs comprehensive documentation audit to eliminate drift between
code and docs.
Code changes:
- TICKET-003: Repository integration tests with testcontainers-go (50+ subtests)
- TICKET-007: CertificateService decomposition into RevocationSvc + CAOperationsSvc
- TICKET-010: Request body size limits via http.MaxBytesReader middleware
- Fix missing slog import in certificate.go after service decomposition
Documentation updates:
- README: Fix endpoint count (97→93), expand env var reference (15→39 vars)
- CLAUDE.md: Fix OpenAPI operation count (85→93), update file locations
- architecture.md: Add body size limits section, middleware chain ordering
- CONTRIBUTING.md: New contributor guide with architecture conventions,
test patterns, middleware ordering, CI thresholds
- SECURITY_REMEDIATION.md: Removed from repo (moved to cowork, gitignored)
- Test files: Add doc comments to all new test files
Documentation that should exist but doesn't yet:
- Architecture diagrams (C4 model or similar)
- Threat model document
- Testing philosophy guide
- Disaster recovery runbook
- Upgrade guide (migration between versions)
- API versioning strategy document
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
M17: Script-based issuer connector delegating sign/revoke/CRL to user-provided
scripts. Compatible with any CA tooling (OpenSSL, cfssl, custom PKI). Configurable
timeout, environment variable passthrough. 14 tests including timeout enforcement.
M16b: certctl-cli wraps all 76 REST API endpoints for terminal workflows. Supports
certs/agents/jobs list/get/renew/revoke/cancel, bulk PEM import with progress
reporting, server health status, table and JSON output formats. Zero external
dependencies (stdlib only). 14 tests with mock HTTP server.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Trim V2 roadmap to free-tier features only (GUI operations, CLI, notifiers,
Prometheus metrics, OCSP, MCP server, filesystem discovery). Move enterprise
features to V3 with deliberately vague descriptions. Remove specific version
references for F5/IIS implementations and SSE/WebSocket from docs. Add
roadmap.md to gitignore for private strategy tracking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Backend hardening:
- Fix 6 nginx.go non-constant format string build errors
- Add validation.go with hostname, PEM, and enum validators
- Apply input validation to all POST/PUT handlers (certificates,
agents, CSR, policies, teams, owners, targets, issuers)
- Fix unchecked JSON decode in TriggerDeployment handler
Frontend (Vite + React + TypeScript):
- Migrate from single-file SPA to proper build pipeline
- 7 pages: Dashboard, Certificates (list+detail), Agents, Jobs,
Notifications, Policies, Audit Trail
- TanStack Query for server state with auto-refetch intervals
- Certificate detail with version history and renewal trigger
- Job cancellation, status/type filtering, expiry countdowns
- Reusable components: DataTable, StatusBadge, ErrorState, PageHeader
- Dark theme with Tailwind CSS, sidebar nav via React Router
Server integration:
- Go server serves web/dist/ (Vite output) with SPA fallback
- Falls back to web/index.html for legacy mode
- .gitignore updated for web/node_modules/ and web/dist/
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
shankar0123