mirror of
https://github.com/shankar0123/certctl.git
synced 2026-06-08 23:28:51 +00:00
v2.0.47: HTTPS Everywhere — TLS-only control plane, agents/CLI/MCP
Breaking change release. Plaintext HTTP listener removed. The certctl control plane now terminates TLS 1.3 on :8443 via http.Server.ListenAndServeTLS. No CERTCTL_TLS_ENABLED=false escape hatch. No dual-listener mode. One-step cutover per docs/upgrade-to-tls.md. Server - cmd/server/tls.go: certHolder with SIGHUP hot-reload + atomic cert swap, buildServerTLSConfig (TLS 1.3 min, GetCertificate callback), preflightServerTLS validation - cmd/server/main.go: ListenAndServeTLS in place of ListenAndServe, watchSIGHUP wiring, cert/key path config threading - tls_test.go: 418-line regression coverage of reload, preflight, callback behavior, SAN validation Config - CERTCTL_TLS_CERT_PATH / CERTCTL_TLS_KEY_PATH (required) - Plaintext rejection: agents/CLI/MCP pre-flight-fail on http:// URLs with a pointer to docs/upgrade-to-tls.md Agents, CLI, MCP - All three pre-flight-reject http:// URLs with fail-loud diagnostic - CERTCTL_SERVER_CA_BUNDLE_PATH for private-CA trust - CERTCTL_SERVER_TLS_INSECURE_SKIP_VERIFY for dev-only bypass (loud warning on startup) - install-agent.sh emits both vars as commented template lines docker-compose - certctl-tls-init sidecar generates SAN-valid self-signed cert into deploy/test/certs/ on first boot - All demo-stack curls pin against ca.crt with --cacert Helm chart - Three TLS provisioning modes, exactly one required: - server.tls.existingSecret (operator-supplied) - server.tls.certManager.enabled (cert-manager integration) - server.tls.selfSigned.enabled (eval only — not for production) - server-certificate.yaml template for cert-manager mode - helm install without a TLS source fails at template render with a pointer to docs/tls.md CI - .github/workflows/ci.yml Helm Chart Validation step renders the chart in both existingSecret and cert-manager modes, plus an inverse guard-regression test that asserts helm template MUST refuse to render when no TLS source is configured. Previously the single `helm template` invocation hit the certctl.tls.required fail-loud guard and exit-1'd CI. Four invocations now: lint (existingSecret), template (existingSecret), template (cert-manager), template (no args — must fail). Integration tests - deploy/test/integration_test.go stands up the Compose stack over HTTPS, extracts the CA bundle, and exercises every certctl API over https://localhost:8443 - All 34 integration subtests green (per Phase 8 local CI-parity) Documentation - New: docs/tls.md (provisioning patterns, rotation, SIGHUP reload) - New: docs/upgrade-to-tls.md (one-step cutover, no-downgrade warnings, fleet-roll sequencing) - CHANGELOG.md: v2.2.0 "HTTPS Everywhere — The Irony" entry (file heading unchanged; release tag is v2.0.47) - All curls in docs/, examples/, deploy/helm/ guides use https://localhost:8443 --cacert Verification - grep -rn "ListenAndServe[^T]" cmd/ internal/ → 0 hits - grep -rn "\"http://" cmd/ internal/ → 2 benign hits (Caddy admin API default, SSRF doc comment) — zero certctl endpoints - Tasks #197–#206 (Phases 0–8) all closed in the tracker Files: 65 changed, 3489 insertions, 372 deletions (pre-CI-fix).
This commit is contained in:
@@ -39,7 +39,7 @@ Deploy certctl control plane once (Docker Compose, Kubernetes Helm chart, or sel
|
||||
```bash
|
||||
cd /opt/certctl
|
||||
docker compose up -d
|
||||
# Dashboard & API: http://localhost:8443
|
||||
# Dashboard & API: https://localhost:8443 (self-signed cert — pin with --cacert ./deploy/test/certs/ca.crt)
|
||||
```
|
||||
|
||||
**Option B: Kubernetes** (recommended for prod)
|
||||
@@ -59,7 +59,8 @@ chmod +x /usr/local/bin/certctl-agent
|
||||
|
||||
# Config
|
||||
sudo tee /etc/certctl/agent.env > /dev/null <<EOF
|
||||
CERTCTL_SERVER_URL=http://certctl-control-plane:8443
|
||||
CERTCTL_SERVER_URL=https://certctl-control-plane:8443
|
||||
CERTCTL_SERVER_CA_BUNDLE_PATH=/etc/certctl/tls/ca.crt
|
||||
CERTCTL_API_KEY=your-api-key
|
||||
CERTCTL_DISCOVERY_DIRS=/etc/nginx/certs,/etc/ssl,/etc/letsencrypt/live
|
||||
CERTCTL_KEY_DIR=/var/lib/certctl/keys
|
||||
|
||||
@@ -387,12 +387,12 @@ This requirement covers key generation, storage, rotation, and destruction. Cert
|
||||
- API key transmitted in Authorization header (not URL parameter, not cookie).
|
||||
- Browser to server: TLS.
|
||||
- Agent to server: TLS.
|
||||
- No credential logging (API key hash only, never plaintext).
|
||||
- No credential logging (audit records the per-key actor `Name`, never the Bearer token; logs redact the `Authorization` header).
|
||||
|
||||
**Evidence You Can Provide**:
|
||||
- API configuration: `CERTCTL_AUTH_TYPE=api-key` in deployment manifest.
|
||||
- Database schema: `api_keys` table showing SHA-256 hash column, not plaintext.
|
||||
- API audit log: `GET /api/v1/audit?action=api_call` showing Bearer token validation (no plaintext keys logged).
|
||||
- Key inventory: `CERTCTL_API_KEYS_NAMED` env var (format `name:key:admin,...`) — seeds the in-memory `NamedAPIKey{Name, Key, Admin}` struct at `internal/api/middleware/middleware.go:29`. Keys are constant-time-compared (`subtle.ConstantTimeCompare`) against the Bearer token. No database table stores them; protect the env var contents at rest via a secrets manager (Vault / AWS Secrets Manager / Kubernetes Secrets / Docker Secrets).
|
||||
- API audit log: `GET /api/v1/audit?action=api_call` showing per-key actor names (`Name` field of matched `NamedAPIKey`) on every call, with zero plaintext or hashed key material recorded.
|
||||
- TLS certificate on control plane: `openssl s_client -connect {server}:8443` showing valid certificate, TLS 1.2+, strong cipher.
|
||||
- GUI login flow: browser network tab showing Authorization header (token value redacted in compliance report).
|
||||
|
||||
|
||||
+16
-10
@@ -50,14 +50,17 @@ docker compose -f deploy/docker-compose.yml up -d --build
|
||||
docker compose -f deploy/docker-compose.yml ps
|
||||
```
|
||||
|
||||
Open **http://localhost:8443** in your browser alongside your terminal. You'll watch changes appear in the dashboard as you make API calls.
|
||||
Open **https://localhost:8443** in your browser alongside your terminal. The default compose stack ships a self-signed cert; your browser will show a warning the first time — click through (or trust `deploy/test/certs/ca.crt` in your OS keychain). You'll watch changes appear in the dashboard as you make API calls.
|
||||
|
||||
Set up a base variable for convenience:
|
||||
Set up base variables for convenience:
|
||||
|
||||
```bash
|
||||
API="http://localhost:8443"
|
||||
API="https://localhost:8443"
|
||||
CA="$PWD/deploy/test/certs/ca.crt" # pin the self-signed CA for curl
|
||||
```
|
||||
|
||||
Every `curl` in this guide uses `--cacert "$CA"` so the TLS handshake verifies against the compose-stack CA instead of the system trust store.
|
||||
|
||||
## How the pieces fit together
|
||||
|
||||
Before we start, here's the high-level flow of what we're about to do:
|
||||
@@ -730,7 +733,7 @@ Check the CRL (Certificate Revocation List) — served unauthenticated under the
|
||||
# DER-encoded X.509 CRL for the local CA (binary — pipe to openssl for inspection).
|
||||
# Note: no -H "Authorization: Bearer ..." — the endpoint is deliberately
|
||||
# unauthenticated. Content-Type is application/pkix-crl.
|
||||
curl -s http://localhost:8443/.well-known/pki/crl/iss-local -o /tmp/crl.der
|
||||
curl --cacert "$CA" -s https://localhost:8443/.well-known/pki/crl/iss-local -o /tmp/crl.der
|
||||
openssl crl -inform DER -in /tmp/crl.der -text -noout
|
||||
```
|
||||
|
||||
@@ -740,7 +743,7 @@ Check OCSP status (RFC 6960, also unauthenticated, `application/ocsp-response`):
|
||||
# Replace SERIAL with the actual serial number from the certificate version.
|
||||
# The embedded OCSP responder returns a signed DER response — parse it with
|
||||
# `openssl ocsp -respin` or similar tooling.
|
||||
curl -s http://localhost:8443/.well-known/pki/ocsp/iss-local/SERIAL -o /tmp/ocsp.der
|
||||
curl --cacert "$CA" -s https://localhost:8443/.well-known/pki/ocsp/iss-local/SERIAL -o /tmp/ocsp.der
|
||||
openssl ocsp -respin /tmp/ocsp.der -noverify -resp_text | head -40
|
||||
```
|
||||
|
||||
@@ -946,7 +949,8 @@ certctl includes a standalone CLI tool for command-line users:
|
||||
cd cmd/cli && go build -o certctl-cli .
|
||||
|
||||
# Export credentials
|
||||
export CERTCTL_SERVER_URL="http://localhost:8443"
|
||||
export CERTCTL_SERVER_URL="https://localhost:8443"
|
||||
export CERTCTL_SERVER_CA_BUNDLE_PATH="$PWD/deploy/test/certs/ca.crt"
|
||||
export CERTCTL_API_KEY="test-key-123"
|
||||
|
||||
# List certificates (JSON or table format)
|
||||
@@ -990,7 +994,8 @@ certctl exposes the full REST API via the Model Context Protocol (MCP), enabling
|
||||
cd cmd/mcp-server && go build -o mcp-server .
|
||||
|
||||
# Export credentials
|
||||
export CERTCTL_SERVER_URL="http://localhost:8443"
|
||||
export CERTCTL_SERVER_URL="https://localhost:8443"
|
||||
export CERTCTL_SERVER_CA_BUNDLE_PATH="$PWD/deploy/test/certs/ca.crt"
|
||||
export CERTCTL_API_KEY="test-key-123"
|
||||
|
||||
# Start the MCP server (listens on stdin/stdout)
|
||||
@@ -1048,7 +1053,7 @@ docker compose -f deploy/docker-compose.yml run -e CERTCTL_DISCOVERY_DIRS=/tmp/c
|
||||
Or with the CLI flag:
|
||||
|
||||
```bash
|
||||
certctl-agent --agent-id a-demo-1 --key-dir /tmp/keys --discovery-dirs /tmp/certs --server http://localhost:8443 --api-key test-key-123
|
||||
certctl-agent --agent-id a-demo-1 --key-dir /tmp/keys --discovery-dirs /tmp/certs --server https://localhost:8443 --ca-bundle "$CA" --api-key test-key-123
|
||||
```
|
||||
|
||||
### Network Discovery (Server-Side)
|
||||
@@ -1191,7 +1196,8 @@ Here's a single script that runs the entire demo end-to-end. Save it as `demo.sh
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
API="http://localhost:8443"
|
||||
API="https://localhost:8443"
|
||||
CA="$PWD/deploy/test/certs/ca.crt" # pin the self-signed CA for curl
|
||||
BLUE='\033[0;34m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
@@ -1299,7 +1305,7 @@ echo " 5. Revoked the certificate with RFC 5280 reason codes"
|
||||
echo " 6. Checked dashboard stats and metrics"
|
||||
echo " 7. All actions recorded in the audit trail"
|
||||
echo ""
|
||||
echo -e "Open ${GREEN}http://localhost:8443${NC} to see everything in the dashboard."
|
||||
echo -e "Open ${GREEN}https://localhost:8443${NC} to see everything in the dashboard."
|
||||
echo "Look for certificate: $CERT_ID"
|
||||
```
|
||||
|
||||
|
||||
+11
-5
@@ -29,15 +29,18 @@ The binary has zero runtime dependencies beyond the certctl server it connects t
|
||||
|
||||
## Configuration
|
||||
|
||||
The MCP server reads two environment variables:
|
||||
The MCP server reads three environment variables:
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| `CERTCTL_SERVER_URL` | No | `http://localhost:8443` | URL of the certctl REST API |
|
||||
| `CERTCTL_SERVER_URL` | No | `https://localhost:8443` | URL of the certctl REST API (HTTPS-only as of v2.2) |
|
||||
| `CERTCTL_API_KEY` | No | (empty) | API key for authentication (passed as `Bearer` token) |
|
||||
| `CERTCTL_SERVER_CA_BUNDLE_PATH` | Yes (for self-signed / internal CA) | (empty) | Path to PEM CA bundle that signed the server cert. Required when the server cert isn't rooted in the system trust store (the default compose stack ships a self-signed cert at `deploy/test/certs/ca.crt`). |
|
||||
|
||||
If your certctl server has auth enabled (the default), you must provide the API key. The MCP server passes it through to every HTTP request.
|
||||
|
||||
Since v2.2 the certctl control plane is HTTPS-only. If the server cert is self-signed or chained to an internal CA, set `CERTCTL_SERVER_CA_BUNDLE_PATH` so the MCP server can verify the TLS handshake. Never set `CERTCTL_SERVER_TLS_INSECURE_SKIP_VERIFY=true` outside local development — it disables all certificate validation.
|
||||
|
||||
## Setting Up with Claude Desktop
|
||||
|
||||
Add this to your Claude Desktop MCP configuration file (`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS, `%APPDATA%\Claude\claude_desktop_config.json` on Windows):
|
||||
@@ -48,7 +51,8 @@ Add this to your Claude Desktop MCP configuration file (`~/Library/Application S
|
||||
"certctl": {
|
||||
"command": "/path/to/certctl-mcp",
|
||||
"env": {
|
||||
"CERTCTL_SERVER_URL": "http://localhost:8443",
|
||||
"CERTCTL_SERVER_URL": "https://localhost:8443",
|
||||
"CERTCTL_SERVER_CA_BUNDLE_PATH": "/path/to/certctl/deploy/test/certs/ca.crt",
|
||||
"CERTCTL_API_KEY": "your-api-key-here"
|
||||
}
|
||||
}
|
||||
@@ -67,7 +71,8 @@ In Cursor, go to Settings → MCP Servers and add:
|
||||
"certctl": {
|
||||
"command": "/path/to/certctl-mcp",
|
||||
"env": {
|
||||
"CERTCTL_SERVER_URL": "http://localhost:8443",
|
||||
"CERTCTL_SERVER_URL": "https://localhost:8443",
|
||||
"CERTCTL_SERVER_CA_BUNDLE_PATH": "/path/to/certctl/deploy/test/certs/ca.crt",
|
||||
"CERTCTL_API_KEY": "your-api-key-here"
|
||||
}
|
||||
}
|
||||
@@ -84,7 +89,8 @@ Add certctl as an MCP server in your project's `.mcp.json`:
|
||||
"certctl": {
|
||||
"command": "/path/to/certctl-mcp",
|
||||
"env": {
|
||||
"CERTCTL_SERVER_URL": "http://localhost:8443",
|
||||
"CERTCTL_SERVER_URL": "https://localhost:8443",
|
||||
"CERTCTL_SERVER_CA_BUNDLE_PATH": "/path/to/certctl/deploy/test/certs/ca.crt",
|
||||
"CERTCTL_API_KEY": "your-api-key-here"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -34,7 +34,7 @@ cd certctl/deploy
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
Access the dashboard at `http://localhost:8443` with API key from `.env` file.
|
||||
Access the dashboard at `https://localhost:8443` with the API key from `.env`. The default compose stack ships a self-signed cert; pin with `--cacert ./deploy/test/certs/ca.crt` when calling the API from the host.
|
||||
|
||||
### 2. Deploy Agents
|
||||
|
||||
|
||||
@@ -22,7 +22,7 @@ Option A: Docker Compose (quickest for evaluation)
|
||||
```bash
|
||||
cd /opt/certctl
|
||||
docker compose up -d
|
||||
# Dashboard & API: http://localhost:8443
|
||||
# Dashboard & API: https://localhost:8443 (self-signed cert — use --cacert ./deploy/test/certs/ca.crt for the default compose stack)
|
||||
# Default API key in logs (grep CERTCTL_API_KEY docker logs certctl-server)
|
||||
```
|
||||
|
||||
@@ -45,7 +45,8 @@ chmod +x /usr/local/bin/certctl-agent
|
||||
# Create config
|
||||
sudo mkdir -p /etc/certctl /var/lib/certctl/keys
|
||||
sudo tee /etc/certctl/agent.env > /dev/null <<EOF
|
||||
CERTCTL_SERVER_URL=http://certctl-control-plane.example.com:8443
|
||||
CERTCTL_SERVER_URL=https://certctl-control-plane.example.com:8443
|
||||
CERTCTL_SERVER_CA_BUNDLE_PATH=/etc/certctl/tls/ca.crt
|
||||
CERTCTL_API_KEY=your-api-key-here
|
||||
CERTCTL_DISCOVERY_DIRS=/etc/letsencrypt/live
|
||||
CERTCTL_KEY_DIR=/var/lib/certctl/keys
|
||||
|
||||
+9
-4
@@ -68,8 +68,10 @@ The spec organizes endpoints into 16 tags:
|
||||
The spec declares a `bearerAuth` security scheme applied globally. All endpoints under `/api/v1/` require a Bearer token by default:
|
||||
|
||||
```bash
|
||||
curl -H "Authorization: Bearer your-api-key" \
|
||||
http://localhost:8443/api/v1/certificates
|
||||
# The default compose stack uses a self-signed cert; pin with --cacert
|
||||
curl --cacert ./deploy/test/certs/ca.crt \
|
||||
-H "Authorization: Bearer your-api-key" \
|
||||
https://localhost:8443/api/v1/certificates
|
||||
```
|
||||
|
||||
Three endpoints are exempt from auth (declared with `security: []` in the spec): `/health`, `/ready`, and `/api/v1/auth/info`. The auth info endpoint tells clients whether authentication is enabled and what type is required — useful for GUIs that need to show/hide a login screen.
|
||||
@@ -150,8 +152,9 @@ Import the spec directly into Postman:
|
||||
|
||||
1. Open Postman → Import → File → select `api/openapi.yaml`
|
||||
2. Postman creates a collection with all 78 documented operations organized by tag
|
||||
3. Set the `baseUrl` variable to `http://localhost:8443`
|
||||
3. Set the `baseUrl` variable to `https://localhost:8443` (HTTPS-only as of v2.2)
|
||||
4. Add an `Authorization: Bearer your-api-key` header to the collection
|
||||
5. Import the demo stack CA bundle (`deploy/test/certs/ca.crt`) into Postman's Settings → Certificates → CA Certificates, or disable certificate verification for the `localhost` host (Settings → General → SSL certificate verification)
|
||||
|
||||
## Key Schemas
|
||||
|
||||
@@ -176,8 +179,10 @@ Use the spec to generate contract tests that verify the API matches the spec:
|
||||
```bash
|
||||
# Using schemathesis for fuzz testing against the spec
|
||||
pip install schemathesis
|
||||
# The default compose stack uses a self-signed cert — export a CA bundle or set REQUESTS_CA_BUNDLE
|
||||
export REQUESTS_CA_BUNDLE=$(pwd)/deploy/test/certs/ca.crt
|
||||
schemathesis run api/openapi.yaml \
|
||||
--base-url http://localhost:8443 \
|
||||
--base-url https://localhost:8443 \
|
||||
--header "Authorization: Bearer your-api-key"
|
||||
```
|
||||
|
||||
|
||||
@@ -85,10 +85,12 @@ go test -tags qa -v -timeout 10m ./...
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `CERTCTL_QA_SERVER_URL` | `http://localhost:8443` | certctl server URL |
|
||||
| `CERTCTL_QA_SERVER_URL` | `https://localhost:8443` | certctl server URL (HTTPS-only as of v2.2) |
|
||||
| `CERTCTL_QA_API_KEY` | `change-me-in-production` | API key for Bearer auth |
|
||||
| `CERTCTL_QA_DB_URL` | `postgres://certctl:certctl@localhost:5432/certctl?sslmode=disable` | PostgreSQL connection string |
|
||||
| `CERTCTL_QA_REPO_DIR` | `../..` | Path to certctl repo root (for source file checks) |
|
||||
| `CERTCTL_QA_CA_BUNDLE` | `./certs/ca.crt` | PEM CA bundle pinned for TLS verification. The demo stack's `certctl-tls-init` container writes here. |
|
||||
| `CERTCTL_QA_INSECURE` | `false` | Set to `"true"` to skip TLS verification (e.g. before the init container finishes). Never use outside the demo harness. |
|
||||
|
||||
## Part-by-Part Coverage Map
|
||||
|
||||
@@ -256,8 +258,8 @@ docker compose -f docker-compose.yml -f docker-compose.demo.yml ps
|
||||
# Check server logs
|
||||
docker compose -f docker-compose.yml -f docker-compose.demo.yml logs certctl-server
|
||||
|
||||
# Check if the port is exposed
|
||||
curl -s http://localhost:8443/health
|
||||
# Check if the port is exposed (self-signed cert — pin CA bundle)
|
||||
curl --cacert ./deploy/test/certs/ca.crt -s https://localhost:8443/health
|
||||
```
|
||||
|
||||
### "connect to QA DB" failure
|
||||
|
||||
+59
-46
@@ -105,16 +105,24 @@ certctl-server Up (healthy)
|
||||
certctl-agent Up
|
||||
```
|
||||
|
||||
The control plane is HTTPS-only as of v2.2. The `certctl-tls-init` init container in the shipped `deploy/docker-compose.yml` self-signs a cert on first boot and drops it into a named volume. Extract the CA bundle once and reuse it for every API call in this guide:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8443/health
|
||||
export CA=/tmp/certctl-ca.crt
|
||||
docker compose -f deploy/docker-compose.yml exec -T certctl-server \
|
||||
cat /etc/certctl/tls/ca.crt > "$CA"
|
||||
|
||||
curl --cacert "$CA" https://localhost:8443/health
|
||||
```
|
||||
```json
|
||||
{"status":"healthy"}
|
||||
```
|
||||
|
||||
If you're bringing your own cert (internal CA, cert-manager, operator-supplied Secret), see [`docs/tls.md`](tls.md) for the full provisioning matrix. If you're cutting over an existing install, see [`docs/upgrade-to-tls.md`](upgrade-to-tls.md) for the failure modes (out-of-date `http://…` agents fail at the TLS handshake) and the one-step procedure.
|
||||
|
||||
## Open the Dashboard
|
||||
|
||||
Open **http://localhost:8443** in your browser.
|
||||
Open **https://localhost:8443** in your browser. Your browser will warn about the self-signed cert — that's expected for the demo bootstrap. Trust the CA bundle you just exported, or click through the warning.
|
||||
|
||||
> **Note:** The Docker Compose demo runs with authentication disabled (`CERTCTL_AUTH_TYPE=none`) so you can explore immediately. For production, set `CERTCTL_AUTH_TYPE=api-key` and `CERTCTL_AUTH_SECRET=<your-secret>` in your environment, then pass `Authorization: Bearer <your-secret>` on all API requests. The dashboard will prompt for your API key on first load.
|
||||
>
|
||||
@@ -154,62 +162,64 @@ Everything you see in the dashboard is backed by the REST API. All endpoints liv
|
||||
|
||||
### Core operations
|
||||
|
||||
Every request below uses `--cacert "$CA"` to pin the self-signed CA bundle extracted above. In production, point `$CA` at your internal CA root or the bundle you distributed to the fleet.
|
||||
|
||||
```bash
|
||||
# List all certificates
|
||||
curl -s http://localhost:8443/api/v1/certificates | jq .
|
||||
curl --cacert "$CA" -s https://localhost:8443/api/v1/certificates | jq .
|
||||
|
||||
# Filter by status
|
||||
curl -s "http://localhost:8443/api/v1/certificates?status=Expiring" | jq .
|
||||
curl --cacert "$CA" -s "https://localhost:8443/api/v1/certificates?status=Expiring" | jq .
|
||||
|
||||
# Filter by environment
|
||||
curl -s "http://localhost:8443/api/v1/certificates?environment=production" | jq .
|
||||
curl --cacert "$CA" -s "https://localhost:8443/api/v1/certificates?environment=production" | jq .
|
||||
|
||||
# Get a specific certificate
|
||||
curl -s http://localhost:8443/api/v1/certificates/mc-api-prod | jq .
|
||||
curl --cacert "$CA" -s https://localhost:8443/api/v1/certificates/mc-api-prod | jq .
|
||||
|
||||
# Get deployment targets for a certificate
|
||||
curl -s http://localhost:8443/api/v1/certificates/mc-api-prod/deployments | jq .
|
||||
curl --cacert "$CA" -s https://localhost:8443/api/v1/certificates/mc-api-prod/deployments | jq .
|
||||
|
||||
# List agents
|
||||
curl -s http://localhost:8443/api/v1/agents | jq .
|
||||
curl --cacert "$CA" -s https://localhost:8443/api/v1/agents | jq .
|
||||
|
||||
# Check agent pending work
|
||||
curl -s http://localhost:8443/api/v1/agents/ag-web-prod/work | jq .
|
||||
curl --cacert "$CA" -s https://localhost:8443/api/v1/agents/ag-web-prod/work | jq .
|
||||
|
||||
# View audit trail
|
||||
curl -s http://localhost:8443/api/v1/audit | jq .
|
||||
curl --cacert "$CA" -s https://localhost:8443/api/v1/audit | jq .
|
||||
|
||||
# View policies and violations
|
||||
curl -s http://localhost:8443/api/v1/policies | jq .
|
||||
curl -s http://localhost:8443/api/v1/policies/pr-require-owner/violations | jq .
|
||||
curl --cacert "$CA" -s https://localhost:8443/api/v1/policies | jq .
|
||||
curl --cacert "$CA" -s https://localhost:8443/api/v1/policies/pr-require-owner/violations | jq .
|
||||
|
||||
# Notifications
|
||||
curl -s http://localhost:8443/api/v1/notifications | jq .
|
||||
curl --cacert "$CA" -s https://localhost:8443/api/v1/notifications | jq .
|
||||
|
||||
# Profiles and agent groups
|
||||
curl -s http://localhost:8443/api/v1/profiles | jq .
|
||||
curl -s http://localhost:8443/api/v1/agent-groups | jq .
|
||||
curl --cacert "$CA" -s https://localhost:8443/api/v1/profiles | jq .
|
||||
curl --cacert "$CA" -s https://localhost:8443/api/v1/agent-groups | jq .
|
||||
```
|
||||
|
||||
### Sorting, filtering, and pagination
|
||||
|
||||
```bash
|
||||
# Sort by expiration date (ascending)
|
||||
curl -s "http://localhost:8443/api/v1/certificates?sort=notAfter" | jq .
|
||||
curl --cacert "$CA" -s "https://localhost:8443/api/v1/certificates?sort=notAfter" | jq .
|
||||
|
||||
# Sort descending (prefix with -)
|
||||
curl -s "http://localhost:8443/api/v1/certificates?sort=-createdAt" | jq .
|
||||
curl --cacert "$CA" -s "https://localhost:8443/api/v1/certificates?sort=-createdAt" | jq .
|
||||
|
||||
# Time-range filters (RFC3339)
|
||||
curl -s "http://localhost:8443/api/v1/certificates?expires_before=2026-05-01T00:00:00Z" | jq .
|
||||
curl -s "http://localhost:8443/api/v1/certificates?created_after=2026-03-01T00:00:00Z" | jq .
|
||||
curl --cacert "$CA" -s "https://localhost:8443/api/v1/certificates?expires_before=2026-05-01T00:00:00Z" | jq .
|
||||
curl --cacert "$CA" -s "https://localhost:8443/api/v1/certificates?created_after=2026-03-01T00:00:00Z" | jq .
|
||||
|
||||
# Sparse fields — request only what you need
|
||||
curl -s "http://localhost:8443/api/v1/certificates?fields=id,common_name,status,expires_at" | jq .
|
||||
curl --cacert "$CA" -s "https://localhost:8443/api/v1/certificates?fields=id,common_name,status,expires_at" | jq .
|
||||
|
||||
# Cursor pagination — efficient for large inventories
|
||||
curl -s "http://localhost:8443/api/v1/certificates?page_size=5" | jq '{next_cursor: .next_cursor, count: (.data | length)}'
|
||||
curl -s "http://localhost:8443/api/v1/certificates?cursor=<next_cursor_value>&page_size=5" | jq .
|
||||
curl --cacert "$CA" -s "https://localhost:8443/api/v1/certificates?page_size=5" | jq '{next_cursor: .next_cursor, count: (.data | length)}'
|
||||
curl --cacert "$CA" -s "https://localhost:8443/api/v1/certificates?cursor=<next_cursor_value>&page_size=5" | jq .
|
||||
```
|
||||
|
||||
Supported sort fields: `notAfter`, `expiresAt`, `createdAt`, `updatedAt`, `commonName`, `name`, `status`, `environment`.
|
||||
@@ -218,22 +228,22 @@ Supported sort fields: `notAfter`, `expiresAt`, `createdAt`, `updatedAt`, `commo
|
||||
|
||||
```bash
|
||||
# Dashboard summary
|
||||
curl -s http://localhost:8443/api/v1/stats/summary | jq .
|
||||
curl --cacert "$CA" -s https://localhost:8443/api/v1/stats/summary | jq .
|
||||
|
||||
# Certificates by status
|
||||
curl -s http://localhost:8443/api/v1/stats/certificates-by-status | jq .
|
||||
curl --cacert "$CA" -s https://localhost:8443/api/v1/stats/certificates-by-status | jq .
|
||||
|
||||
# Expiration timeline (next 90 days)
|
||||
curl -s "http://localhost:8443/api/v1/stats/expiration-timeline?days=90" | jq .
|
||||
curl --cacert "$CA" -s "https://localhost:8443/api/v1/stats/expiration-timeline?days=90" | jq .
|
||||
|
||||
# Job trends (last 30 days)
|
||||
curl -s "http://localhost:8443/api/v1/stats/job-trends?days=30" | jq .
|
||||
curl --cacert "$CA" -s "https://localhost:8443/api/v1/stats/job-trends?days=30" | jq .
|
||||
|
||||
# JSON metrics
|
||||
curl -s http://localhost:8443/api/v1/metrics | jq .
|
||||
curl --cacert "$CA" -s https://localhost:8443/api/v1/metrics | jq .
|
||||
|
||||
# Prometheus format (for Prometheus, Grafana Agent, Datadog)
|
||||
curl -s http://localhost:8443/api/v1/metrics/prometheus
|
||||
curl --cacert "$CA" -s https://localhost:8443/api/v1/metrics/prometheus
|
||||
```
|
||||
|
||||
## Create Your First Certificate
|
||||
@@ -241,7 +251,7 @@ curl -s http://localhost:8443/api/v1/metrics/prometheus
|
||||
Create a certificate record that certctl will track, renew, and deploy automatically.
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://localhost:8443/api/v1/certificates \
|
||||
curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/certificates \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "My First Certificate",
|
||||
@@ -264,22 +274,22 @@ CERT_ID="<paste the id from the response>"
|
||||
|
||||
Trigger renewal:
|
||||
```bash
|
||||
curl -s -X POST http://localhost:8443/api/v1/certificates/$CERT_ID/renew | jq .
|
||||
curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/certificates/$CERT_ID/renew | jq .
|
||||
```
|
||||
|
||||
Check the result:
|
||||
```bash
|
||||
curl -s http://localhost:8443/api/v1/certificates/$CERT_ID | jq .
|
||||
curl --cacert "$CA" -s https://localhost:8443/api/v1/certificates/$CERT_ID | jq .
|
||||
```
|
||||
|
||||
Refresh the dashboard at http://localhost:8443 — your new certificate appears in the inventory.
|
||||
Refresh the dashboard at https://localhost:8443 — your new certificate appears in the inventory.
|
||||
|
||||
### Revoke a certificate
|
||||
|
||||
When a private key is compromised or a service is decommissioned:
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://localhost:8443/api/v1/certificates/$CERT_ID/revoke \
|
||||
curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/certificates/$CERT_ID/revoke \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"reason": "superseded"}' | jq .
|
||||
```
|
||||
@@ -289,7 +299,8 @@ Supported RFC 5280 reason codes: `unspecified`, `keyCompromise`, `caCompromise`,
|
||||
Confirm via the unauthenticated DER CRL (RFC 5280 §5, RFC 8615):
|
||||
```bash
|
||||
# Fetch the CRL without any API key — relying parties shouldn't need one.
|
||||
curl -s http://localhost:8443/.well-known/pki/crl/iss-local -o /tmp/crl.der
|
||||
# The CRL path is unauthenticated, but it's still served over TLS.
|
||||
curl --cacert "$CA" -s https://localhost:8443/.well-known/pki/crl/iss-local -o /tmp/crl.der
|
||||
openssl crl -inform der -in /tmp/crl.der -noout -text | head -40
|
||||
```
|
||||
|
||||
@@ -299,15 +310,15 @@ For high-value certificates where you want human oversight. The demo includes 2
|
||||
|
||||
```bash
|
||||
# List jobs awaiting approval (demo includes 2)
|
||||
curl -s "http://localhost:8443/api/v1/jobs?status=AwaitingApproval" | jq '.data[] | {id, certificate_id, status}'
|
||||
curl --cacert "$CA" -s "https://localhost:8443/api/v1/jobs?status=AwaitingApproval" | jq '.data[] | {id, certificate_id, status}'
|
||||
|
||||
# Approve a pending job
|
||||
curl -s -X POST http://localhost:8443/api/v1/jobs/JOB_ID/approve \
|
||||
curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/jobs/JOB_ID/approve \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"reason": "Approved for production deployment"}' | jq .
|
||||
|
||||
# Reject a pending job
|
||||
curl -s -X POST http://localhost:8443/api/v1/jobs/JOB_ID/reject \
|
||||
curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/jobs/JOB_ID/reject \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"reason": "Key type does not meet compliance requirements"}' | jq .
|
||||
```
|
||||
@@ -333,7 +344,7 @@ export CERTCTL_DISCOVERY_DIRS="/etc/nginx/certs,/etc/ssl/certs,/var/lib/certs"
|
||||
export CERTCTL_NETWORK_SCAN_ENABLED=true
|
||||
|
||||
# Create a scan target
|
||||
curl -s -X POST http://localhost:8443/api/v1/network-scan-targets \
|
||||
curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/network-scan-targets \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "Internal Network",
|
||||
@@ -345,20 +356,20 @@ curl -s -X POST http://localhost:8443/api/v1/network-scan-targets \
|
||||
}' | jq .
|
||||
|
||||
# Trigger an immediate scan
|
||||
curl -s -X POST http://localhost:8443/api/v1/network-scan-targets/nst-internal-network/scan | jq .
|
||||
curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/network-scan-targets/nst-internal-network/scan | jq .
|
||||
```
|
||||
|
||||
### Triage discovered certificates
|
||||
|
||||
```bash
|
||||
# List discovered certs
|
||||
curl -s "http://localhost:8443/api/v1/discovered-certificates?agent_id=agent-nginx-prod" | jq .
|
||||
curl --cacert "$CA" -s "https://localhost:8443/api/v1/discovered-certificates?agent_id=agent-nginx-prod" | jq .
|
||||
|
||||
# Summary counts
|
||||
curl -s http://localhost:8443/api/v1/discovery-summary | jq .
|
||||
curl --cacert "$CA" -s https://localhost:8443/api/v1/discovery-summary | jq .
|
||||
|
||||
# Claim a discovered cert (bring under management)
|
||||
curl -s -X POST "http://localhost:8443/api/v1/discovered-certificates/DISCOVERY_ID/claim" \
|
||||
curl --cacert "$CA" -s -X POST "https://localhost:8443/api/v1/discovered-certificates/DISCOVERY_ID/claim" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"managed_certificate_id": "mc-api-prod"}' | jq .
|
||||
```
|
||||
@@ -368,8 +379,9 @@ curl -s -X POST "http://localhost:8443/api/v1/discovered-certificates/DISCOVERY_
|
||||
```bash
|
||||
cd cmd/cli && go build -o certctl-cli .
|
||||
|
||||
export CERTCTL_SERVER_URL="http://localhost:8443"
|
||||
export CERTCTL_SERVER_URL="https://localhost:8443"
|
||||
export CERTCTL_API_KEY="test-key-123"
|
||||
export CERTCTL_SERVER_CA_BUNDLE_PATH="$CA" # or pass --ca-bundle; --insecure for dev self-signed
|
||||
|
||||
./certctl-cli certs list # List certificates
|
||||
./certctl-cli certs get mc-api-prod # Certificate details
|
||||
@@ -402,10 +414,10 @@ export CERTCTL_DIGEST_RECIPIENTS=ops@example.com,security@example.com
|
||||
|
||||
Preview the digest HTML before enabling scheduled delivery:
|
||||
```bash
|
||||
curl http://localhost:8443/api/v1/digest/preview | jq '.html' | grep -o '<html>' # Shows HTML is ready
|
||||
curl --cacert "$CA" https://localhost:8443/api/v1/digest/preview | jq '.html' | grep -o '<html>' # Shows HTML is ready
|
||||
|
||||
# Trigger a digest send immediately (outside of schedule)
|
||||
curl -X POST http://localhost:8443/api/v1/digest/send
|
||||
curl --cacert "$CA" -X POST https://localhost:8443/api/v1/digest/send
|
||||
```
|
||||
|
||||
If no recipients are configured (`CERTCTL_DIGEST_RECIPIENTS` empty), the digest falls back to certificate owner emails. Digests include total certificates, expiring soon, expired, active agents, completed/failed jobs (30-day summary), and a table of expiring certs color-coded by urgency (7/14/30 days).
|
||||
@@ -415,8 +427,9 @@ If no recipients are configured (`CERTCTL_DIGEST_RECIPIENTS` empty), the digest
|
||||
```bash
|
||||
cd cmd/mcp-server && go build -o mcp-server .
|
||||
|
||||
export CERTCTL_SERVER_URL="http://localhost:8443"
|
||||
export CERTCTL_SERVER_URL="https://localhost:8443"
|
||||
export CERTCTL_API_KEY="test-key-123"
|
||||
export CERTCTL_SERVER_CA_BUNDLE_PATH="$CA" # MCP is env-vars-only; no CLI flags
|
||||
|
||||
./mcp-server
|
||||
```
|
||||
|
||||
+64
-47
@@ -16,7 +16,7 @@ You'll start 7 Docker containers that talk to each other:
|
||||
| **pebble-challtestsrv** | DNS/HTTP challenge test server for Pebble | 10.30.50.3 | Not directly — Pebble talks to it |
|
||||
| **Pebble** | A fake Let's Encrypt (tests the ACME protocol without touching the real internet) | 10.30.50.4 | Not directly — the server talks to it |
|
||||
| **step-ca** | A private Certificate Authority (think: your company's internal CA) | 10.30.50.5 | Not directly — the server talks to it |
|
||||
| **certctl-server** | The brain. API + web dashboard + scheduler + ACME challenge server | 10.30.50.6 | **http://localhost:8443** |
|
||||
| **certctl-server** | The brain. API + web dashboard + scheduler + ACME challenge server | 10.30.50.6 | **https://localhost:8443** (self-signed — see CA-bundle note below) |
|
||||
| **NGINX** | A web server. The agent deploys certificates here. | 10.30.50.7 | **https://localhost:8444** |
|
||||
| **certctl-agent** | The hands. Generates keys, deploys certs to NGINX | 10.30.50.8 | Not directly — it talks to the server |
|
||||
|
||||
@@ -123,7 +123,7 @@ docker compose -f docker-compose.test.yml up --build
|
||||
|
||||
```
|
||||
certctl-test-server | {"level":"INFO","msg":"server started","address":"0.0.0.0:8443"}
|
||||
certctl-test-agent | {"level":"INFO","msg":"agent starting","server_url":"http://certctl-server:8443"}
|
||||
certctl-test-agent | {"level":"INFO","msg":"agent starting","server_url":"https://certctl-server:8443"}
|
||||
certctl-test-stepca | Serving HTTPS on :9000 ...
|
||||
certctl-test-pebble | Listening on: 0.0.0.0:14000
|
||||
```
|
||||
@@ -159,13 +159,29 @@ certctl-test-stepca Up (healthy)
|
||||
|
||||
**If certctl-test-server says "Restarting"**: It probably started before step-ca or Pebble were ready. Wait 30 seconds and check again. If it keeps restarting, see [Troubleshooting](#troubleshooting).
|
||||
|
||||
### Get the CA bundle for curl
|
||||
|
||||
The test harness runs HTTPS-only (the `certctl-tls-init` init container self-signs an ed25519 server cert into a bind-mounted directory before the server starts — see `docker-compose.test.yml` §`certctl-tls-init` for details). The CA cert that signed it is materialized on the host at `./test/certs/ca.crt` (relative to the `deploy/` directory). Every `curl` in the rest of this doc expects it in `$CA`:
|
||||
|
||||
```bash
|
||||
export CA=$PWD/test/certs/ca.crt
|
||||
ls -la "$CA" # sanity check: file should exist and be non-empty
|
||||
curl --cacert "$CA" -f https://localhost:8443/health
|
||||
```
|
||||
|
||||
Expect `{"status":"ok"}`. If `curl` errors with `SSL certificate problem: unable to get local issuer certificate`, the init container hasn't finished yet — wait a few seconds and retry. If the file doesn't exist at all, the bind mount didn't populate; `docker compose -f docker-compose.test.yml logs certctl-tls-init` should show the self-sign ran.
|
||||
|
||||
For a full explanation of the cert provisioning patterns (self-signed bootstrap, operator-supplied, cert-manager), see [`tls.md`](tls.md). For the one-step cutover from the old plaintext test harness to HTTPS, see [`upgrade-to-tls.md`](upgrade-to-tls.md).
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Open the Dashboard
|
||||
|
||||
Open your web browser and go to:
|
||||
|
||||
**http://localhost:8443**
|
||||
**https://localhost:8443**
|
||||
|
||||
Your browser will warn you that the cert is self-signed ("Your connection is not private" / "NET::ERR_CERT_AUTHORITY_INVALID"). That's expected for the test harness — the CA that signed the cert lives at `deploy/test/certs/ca.crt` and isn't in your system trust store. Click through the warning (Chrome: "Advanced" → "Proceed"; Firefox: "Accept the Risk"; Safari: "Show Details" → "visit this website").
|
||||
|
||||
You'll see a login screen asking for an API key. Enter:
|
||||
|
||||
@@ -198,12 +214,13 @@ Go back to your second terminal. Let's verify the data loaded correctly.
|
||||
### Check the agent
|
||||
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer test-key-2026" \
|
||||
http://localhost:8443/api/v1/agents | python3 -m json.tool
|
||||
curl --cacert "$CA" -s -H "Authorization: Bearer test-key-2026" \
|
||||
https://localhost:8443/api/v1/agents | python3 -m json.tool
|
||||
```
|
||||
|
||||
**What this command does**:
|
||||
- `curl` makes an HTTP request (like a browser but from the terminal)
|
||||
- `curl` makes an HTTPS request (like a browser but from the terminal)
|
||||
- `--cacert "$CA"` pins the test harness's self-signed root as the only trust anchor for this call — matches what you exported in Step 1
|
||||
- `-s` means "silent" (don't show progress bars)
|
||||
- `-H "Authorization: Bearer test-key-2026"` sends the API key (same one you used to log in)
|
||||
- `python3 -m json.tool` formats the JSON response so it's readable
|
||||
@@ -233,8 +250,8 @@ The important parts: `"id": "agent-test-01"` and `"status": "online"`. If the st
|
||||
### Check the issuers
|
||||
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer test-key-2026" \
|
||||
http://localhost:8443/api/v1/issuers | python3 -m json.tool
|
||||
curl --cacert "$CA" -s -H "Authorization: Bearer test-key-2026" \
|
||||
https://localhost:8443/api/v1/issuers | python3 -m json.tool
|
||||
```
|
||||
|
||||
You should see three issuers:
|
||||
@@ -245,8 +262,8 @@ You should see three issuers:
|
||||
### Check the target
|
||||
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer test-key-2026" \
|
||||
http://localhost:8443/api/v1/targets | python3 -m json.tool
|
||||
curl --cacert "$CA" -s -H "Authorization: Bearer test-key-2026" \
|
||||
https://localhost:8443/api/v1/targets | python3 -m json.tool
|
||||
```
|
||||
|
||||
You should see `target-test-nginx` — the NGINX deployment target, assigned to `agent-test-01`.
|
||||
@@ -255,7 +272,7 @@ The target config uses no-op commands for `reload_command` and `validate_command
|
||||
|
||||
### See it all in the dashboard
|
||||
|
||||
Open the dashboard at http://localhost:8443 and click through the sidebar:
|
||||
Open the dashboard at https://localhost:8443 and click through the sidebar:
|
||||
- **Agents** — you should see `test-agent-01`
|
||||
- **Issuers** — you should see all three CAs
|
||||
- **Targets** — you should see `Test NGINX`
|
||||
@@ -287,7 +304,7 @@ The private key **never leaves the agent**. The server only ever sees the CSR (p
|
||||
### Step 4a: Create the certificate record
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://localhost:8443/api/v1/certificates \
|
||||
curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/certificates \
|
||||
-H "Authorization: Bearer test-key-2026" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
@@ -338,7 +355,7 @@ docker exec certctl-test-postgres psql -U certctl -d certctl -c \
|
||||
### Step 4c: Trigger issuance
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://localhost:8443/api/v1/certificates/mc-local-test/renew \
|
||||
curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/certificates/mc-local-test/renew \
|
||||
-H "Authorization: Bearer test-key-2026" | python3 -m json.tool
|
||||
```
|
||||
|
||||
@@ -395,7 +412,7 @@ The `subject` should match the domain name you chose. The `issuer` should say "c
|
||||
|
||||
### Step 4f: Check the dashboard
|
||||
|
||||
Open the dashboard at http://localhost:8443 and:
|
||||
Open the dashboard at https://localhost:8443 and:
|
||||
|
||||
1. Click **Certificates** in the sidebar — you should see `mc-local-test` with status "Active"
|
||||
2. Click on it to see the detail page — you should see version history, the signed certificate details, and the deployment timeline
|
||||
@@ -414,7 +431,7 @@ This is the real deal. ACME is the protocol that Let's Encrypt uses to issue cer
|
||||
### Step 5a: Create the certificate record
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://localhost:8443/api/v1/certificates \
|
||||
curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/certificates \
|
||||
-H "Authorization: Bearer test-key-2026" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
@@ -441,7 +458,7 @@ docker exec certctl-test-postgres psql -U certctl -d certctl -c \
|
||||
"INSERT INTO certificate_target_mappings (certificate_id, target_id) VALUES ('mc-acme-test', 'target-test-nginx') ON CONFLICT DO NOTHING;"
|
||||
|
||||
# Trigger issuance
|
||||
curl -s -X POST http://localhost:8443/api/v1/certificates/mc-acme-test/renew \
|
||||
curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/certificates/mc-acme-test/renew \
|
||||
-H "Authorization: Bearer test-key-2026" | python3 -m json.tool
|
||||
```
|
||||
|
||||
@@ -502,7 +519,7 @@ Revocation means "this certificate is no longer trusted, even though it hasn't e
|
||||
### Step 7a: Revoke the Local CA cert
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://localhost:8443/api/v1/certificates/mc-local-test/revoke \
|
||||
curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/certificates/mc-local-test/revoke \
|
||||
-H "Authorization: Bearer test-key-2026" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"reason": "superseded"}' | python3 -m json.tool
|
||||
@@ -516,7 +533,7 @@ The CRL is a DER-encoded X.509 v2 CRL (RFC 5280 §5) served under the RFC 8615 w
|
||||
|
||||
```bash
|
||||
# No Authorization header — the endpoint is public by design.
|
||||
curl -s http://localhost:8443/.well-known/pki/crl/iss-local -o /tmp/crl.der
|
||||
curl --cacert "$CA" -s https://localhost:8443/.well-known/pki/crl/iss-local -o /tmp/crl.der
|
||||
openssl crl -inform der -in /tmp/crl.der -noout -text | head -40
|
||||
```
|
||||
|
||||
@@ -533,8 +550,8 @@ Go to **Certificates** in the sidebar. The `mc-local-test` cert should now show
|
||||
The agent is configured to scan `/nginx-certs` every 6 hours for existing certificates. It already ran a scan when it started up. Let's see what it found.
|
||||
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer test-key-2026" \
|
||||
http://localhost:8443/api/v1/discovered-certificates | python3 -m json.tool
|
||||
curl --cacert "$CA" -s -H "Authorization: Bearer test-key-2026" \
|
||||
https://localhost:8443/api/v1/discovered-certificates | python3 -m json.tool
|
||||
```
|
||||
|
||||
**What you should see**: Any certificates that exist in the NGINX cert directory, including the ones you deployed in Steps 4-5. The discovery system extracts metadata (CN, SANs, issuer, expiry, fingerprint) from the PEM files.
|
||||
@@ -542,8 +559,8 @@ curl -s -H "Authorization: Bearer test-key-2026" \
|
||||
Check the summary:
|
||||
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer test-key-2026" \
|
||||
http://localhost:8443/api/v1/discovery-summary | python3 -m json.tool
|
||||
curl --cacert "$CA" -s -H "Authorization: Bearer test-key-2026" \
|
||||
https://localhost:8443/api/v1/discovery-summary | python3 -m json.tool
|
||||
```
|
||||
|
||||
This shows counts: how many are Unmanaged, Managed, and Dismissed.
|
||||
@@ -557,7 +574,7 @@ In the dashboard: click **Discovery** in the sidebar to see the triage view.
|
||||
Force a renewal on the ACME certificate to see the full cycle happen again:
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://localhost:8443/api/v1/certificates/mc-acme-test/renew \
|
||||
curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/certificates/mc-acme-test/renew \
|
||||
-H "Authorization: Bearer test-key-2026" | python3 -m json.tool
|
||||
```
|
||||
|
||||
@@ -584,7 +601,7 @@ The test environment enables EST with `CERTCTL_EST_ENABLED=true` and `CERTCTL_ES
|
||||
### Step 10a: Check available CA certificates
|
||||
|
||||
```bash
|
||||
curl -sk http://localhost:8443/.well-known/est/cacerts \
|
||||
curl --cacert "$CA" -s https://localhost:8443/.well-known/est/cacerts \
|
||||
-H "Authorization: Bearer test-key-2026"
|
||||
```
|
||||
|
||||
@@ -595,7 +612,7 @@ curl -sk http://localhost:8443/.well-known/est/cacerts \
|
||||
### Step 10b: Check CSR attributes
|
||||
|
||||
```bash
|
||||
curl -sk http://localhost:8443/.well-known/est/csrattrs \
|
||||
curl --cacert "$CA" -s https://localhost:8443/.well-known/est/csrattrs \
|
||||
-H "Authorization: Bearer test-key-2026"
|
||||
```
|
||||
|
||||
@@ -615,7 +632,7 @@ openssl req -new -newkey ec -pkeyopt ec_paramgen_curve:P-256 \
|
||||
EST_CSR=$(openssl req -in /tmp/est-test.csr -outform DER | base64 -w 0)
|
||||
|
||||
# Submit to EST simpleenroll endpoint
|
||||
curl -sk -X POST http://localhost:8443/.well-known/est/simpleenroll \
|
||||
curl --cacert "$CA" -s -X POST https://localhost:8443/.well-known/est/simpleenroll \
|
||||
-H "Authorization: Bearer test-key-2026" \
|
||||
-H "Content-Type: application/pkcs10" \
|
||||
-d "$EST_CSR"
|
||||
@@ -628,8 +645,8 @@ curl -sk -X POST http://localhost:8443/.well-known/est/simpleenroll \
|
||||
Decode and inspect the response (if you saved it to a variable):
|
||||
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer test-key-2026" \
|
||||
http://localhost:8443/api/v1/audit-events | python3 -m json.tool | head -30
|
||||
curl --cacert "$CA" -s -H "Authorization: Bearer test-key-2026" \
|
||||
https://localhost:8443/api/v1/audit-events | python3 -m json.tool | head -30
|
||||
```
|
||||
|
||||
Check the audit trail — you should see an `est_enrollment` event with the CN `est-device.certctl.test`.
|
||||
@@ -639,7 +656,7 @@ Check the audit trail — you should see an `est_enrollment` event with the CN `
|
||||
EST also supports re-enrollment (certificate renewal). The same CSR format works:
|
||||
|
||||
```bash
|
||||
curl -sk -X POST http://localhost:8443/.well-known/est/simplereenroll \
|
||||
curl --cacert "$CA" -s -X POST https://localhost:8443/.well-known/est/simplereenroll \
|
||||
-H "Authorization: Bearer test-key-2026" \
|
||||
-H "Content-Type: application/pkcs10" \
|
||||
-d "$EST_CSR"
|
||||
@@ -658,7 +675,7 @@ S/MIME certificates are used for email signing and encryption — a different us
|
||||
### Step 11a: Create an S/MIME certificate record
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://localhost:8443/api/v1/certificates \
|
||||
curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/certificates \
|
||||
-H "Authorization: Bearer test-key-2026" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
@@ -686,7 +703,7 @@ Notice:
|
||||
docker exec certctl-test-postgres psql -U certctl -d certctl -c \
|
||||
"INSERT INTO certificate_target_mappings (certificate_id, target_id) VALUES ('mc-smime-test', 'target-test-nginx') ON CONFLICT DO NOTHING;"
|
||||
|
||||
curl -s -X POST http://localhost:8443/api/v1/certificates/mc-smime-test/renew \
|
||||
curl --cacert "$CA" -s -X POST https://localhost:8443/api/v1/certificates/mc-smime-test/renew \
|
||||
-H "Authorization: Bearer test-key-2026" | python3 -m json.tool
|
||||
```
|
||||
|
||||
@@ -695,15 +712,15 @@ curl -s -X POST http://localhost:8443/api/v1/certificates/mc-smime-test/renew \
|
||||
After the agent processes the job (30-60 seconds), check the certificate details:
|
||||
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer test-key-2026" \
|
||||
http://localhost:8443/api/v1/certificates/mc-smime-test | python3 -m json.tool
|
||||
curl --cacert "$CA" -s -H "Authorization: Bearer test-key-2026" \
|
||||
https://localhost:8443/api/v1/certificates/mc-smime-test | python3 -m json.tool
|
||||
```
|
||||
|
||||
The certificate should show `"status": "active"`. To verify the EKU on the actual cert, you can export it:
|
||||
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer test-key-2026" \
|
||||
http://localhost:8443/api/v1/certificates/mc-smime-test/export/pem | python3 -m json.tool
|
||||
curl --cacert "$CA" -s -H "Authorization: Bearer test-key-2026" \
|
||||
https://localhost:8443/api/v1/certificates/mc-smime-test/export/pem | python3 -m json.tool
|
||||
```
|
||||
|
||||
If you decode the certificate PEM, you should see:
|
||||
@@ -768,16 +785,16 @@ If you have Go installed, you can build and test the CLI tool:
|
||||
go build -o certctl-cli ./cmd/cli
|
||||
|
||||
# List certificates
|
||||
./certctl-cli --server http://localhost:8443 --api-key test-key-2026 list-certs
|
||||
./certctl-cli --server https://localhost:8443 --ca-bundle "$CA" --api-key test-key-2026 list-certs
|
||||
|
||||
# Get a specific certificate
|
||||
./certctl-cli --server http://localhost:8443 --api-key test-key-2026 get-cert mc-acme-test
|
||||
./certctl-cli --server https://localhost:8443 --ca-bundle "$CA" --api-key test-key-2026 get-cert mc-acme-test
|
||||
|
||||
# Check health
|
||||
./certctl-cli --server http://localhost:8443 --api-key test-key-2026 health
|
||||
./certctl-cli --server https://localhost:8443 --ca-bundle "$CA" --api-key test-key-2026 health
|
||||
|
||||
# Get metrics (JSON format)
|
||||
./certctl-cli --server http://localhost:8443 --api-key test-key-2026 --format json metrics
|
||||
./certctl-cli --server https://localhost:8443 --ca-bundle "$CA" --api-key test-key-2026 --format json metrics
|
||||
```
|
||||
|
||||
---
|
||||
@@ -924,15 +941,15 @@ Look for error messages. Common ones:
|
||||
**Step 2**: Verify the agent is registered:
|
||||
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer test-key-2026" \
|
||||
http://localhost:8443/api/v1/agents/agent-test-01 | python3 -m json.tool
|
||||
curl --cacert "$CA" -s -H "Authorization: Bearer test-key-2026" \
|
||||
https://localhost:8443/api/v1/agents/agent-test-01 | python3 -m json.tool
|
||||
```
|
||||
|
||||
**Step 3**: Check for pending jobs:
|
||||
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer test-key-2026" \
|
||||
"http://localhost:8443/api/v1/jobs?status=Pending&status=AwaitingCSR" | python3 -m json.tool
|
||||
curl --cacert "$CA" -s -H "Authorization: Bearer test-key-2026" \
|
||||
"https://localhost:8443/api/v1/jobs?status=Pending&status=AwaitingCSR" | python3 -m json.tool
|
||||
```
|
||||
|
||||
If there are pending jobs but the agent isn't picking them up, check that the job's `agent_id` matches `agent-test-01`.
|
||||
@@ -962,8 +979,8 @@ docker exec certctl-test-nginx nginx -s reload
|
||||
**Step 3**: If the files aren't there, the deployment job hasn't completed. Check the jobs:
|
||||
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer test-key-2026" \
|
||||
"http://localhost:8443/api/v1/jobs?type=Deployment" | python3 -m json.tool
|
||||
curl --cacert "$CA" -s -H "Authorization: Bearer test-key-2026" \
|
||||
"https://localhost:8443/api/v1/jobs?type=Deployment" | python3 -m json.tool
|
||||
```
|
||||
|
||||
Look at the job status. If it's "Running" and stuck, the server's job processor may have picked it up instead of the agent (this was a known bug — the fix skips deployment jobs with `agent_id` in the server's `ProcessPendingJobs`).
|
||||
@@ -1008,7 +1025,7 @@ Change it to a different port, like:
|
||||
- "9443:8443"
|
||||
```
|
||||
|
||||
Then access the dashboard at http://localhost:9443 instead.
|
||||
Then access the dashboard at https://localhost:9443 instead.
|
||||
|
||||
### Starting completely fresh
|
||||
|
||||
@@ -1054,7 +1071,7 @@ docker compose -f docker-compose.test.yml up --build
|
||||
|
||||
| What | Value |
|
||||
|---|---|
|
||||
| Dashboard URL | http://localhost:8443 |
|
||||
| Dashboard URL | https://localhost:8443 (use `--cacert ./test/certs/ca.crt`) |
|
||||
| API key | `test-key-2026` |
|
||||
| NGINX HTTP | http://localhost:8080 |
|
||||
| NGINX HTTPS | https://localhost:8444 |
|
||||
|
||||
+179
@@ -0,0 +1,179 @@
|
||||
# TLS on the Control Plane
|
||||
|
||||
certctl's control plane is HTTPS-only as of v2.2. There is no plaintext `http://` listener, no `auto` mode, no dual-listener bridge, no TLS 1.2 escape hatch. The server refuses to start without a cert+key pair, the agent/CLI/MCP clients reject `http://` URLs at startup, and the Helm chart refuses to render without either an operator-supplied Secret or a cert-manager Certificate CR.
|
||||
|
||||
This doc covers four cert provisioning patterns, SIGHUP-based cert rotation, and the client-side CA-trust configuration agents and the CLI need to talk to the server. If you are upgrading from a pre-HTTPS release and want the step-by-step cutover procedure, read [`upgrade-to-tls.md`](upgrade-to-tls.md) first and come back here for reference.
|
||||
|
||||
## What you get
|
||||
|
||||
The server binds TLS 1.3 only with an explicit curve preference of `[X25519, P-256]`. TLS 1.3 cipher suites are non-negotiable (all three mandatory suites — AES-128-GCM-SHA256, AES-256-GCM-SHA384, CHACHA20-POLY1305-SHA256 — are always offered), so there is no `CipherSuites` knob to misconfigure. No TLS 1.2 fallback is available.
|
||||
|
||||
Two env vars are required on the server:
|
||||
|
||||
- `CERTCTL_SERVER_TLS_CERT_PATH` — filesystem path to the PEM-encoded server certificate
|
||||
- `CERTCTL_SERVER_TLS_KEY_PATH` — filesystem path to the PEM-encoded private key that signs the cert
|
||||
|
||||
Both paths are read during a fail-loud preflight in `cmd/server/main.go` (see `preflightServerTLS` in `cmd/server/tls.go`). If either is unset, unreadable, or the cert+key pair does not round-trip through `tls.LoadX509KeyPair`, the process refuses to start and emits a diagnostic pointing back at this doc. The rationale lives in §3 of the HTTPS-Everywhere milestone: a cert-lifecycle product should not silently bind plaintext.
|
||||
|
||||
## Pattern 1 — Self-signed bootstrap for docker-compose demos
|
||||
|
||||
This is the default for the `deploy/docker-compose.yml` stack. It exists so `docker compose up -d --build` just works on a laptop without the operator standing up a CA first. It is not appropriate for any non-demo environment.
|
||||
|
||||
An init container named `certctl-tls-init` runs once before the server starts. It uses the `alpine/openssl` image and generates an ed25519 self-signed cert:
|
||||
|
||||
```
|
||||
openssl req -x509 -newkey ed25519 -nodes \
|
||||
-keyout /etc/certctl/tls/server.key \
|
||||
-out /etc/certctl/tls/server.crt \
|
||||
-days 3650 \
|
||||
-subj "/CN=certctl-server" \
|
||||
-addext "subjectAltName=DNS:certctl-server,DNS:localhost,IP:127.0.0.1,IP:::1"
|
||||
```
|
||||
|
||||
The cert, its matching key, and a copy of the cert published as `ca.crt` land in a named volume (`certs`) mounted at `/etc/certctl/tls/` in the server container (read-only) and the agent container (read-only). The bootstrap is idempotent — if `server.crt`, `server.key`, and `ca.crt` are already present on the volume, the init container logs `TLS cert already present at …` and exits cleanly.
|
||||
|
||||
Single-cert design. CN is `certctl-server` to match the Docker-network hostname. The SAN list is `[certctl-server, localhost, 127.0.0.1, ::1]`, which covers both container-internal agent→server traffic and operator browser/curl access to `https://localhost:8443`. There is no separate intermediate/root chain — the server cert and the CA bundle are the same PEM. This is the whole point of a demo bootstrap.
|
||||
|
||||
To force regeneration (rotate the demo cert), tear the volume down: `docker compose down -v`. The next `up` re-runs the init container.
|
||||
|
||||
The server's Docker healthcheck and the agent both verify against `/etc/certctl/tls/ca.crt`; no `-k` / `InsecureSkipVerify` anywhere in the default stack.
|
||||
|
||||
## Pattern 2 — Operator-supplied `kubernetes.io/tls` Secret (Helm)
|
||||
|
||||
This is the default path for Helm installs. The operator provisions a Secret of type `kubernetes.io/tls` holding `tls.crt` + `tls.key` (and optionally `ca.crt` for mounting a CA bundle to clients in the same cluster) from whatever source they already trust — their internal CA, a manually-issued cert, step-ca, AWS ACM PCA exported to PEM, or the output of the self-signed bootstrap pattern above copied into a cluster Secret.
|
||||
|
||||
```
|
||||
kubectl create secret tls certctl-server-tls \
|
||||
--cert=server.crt \
|
||||
--key=server.key \
|
||||
--namespace certctl
|
||||
```
|
||||
|
||||
Then:
|
||||
|
||||
```
|
||||
helm install certctl deploy/helm/certctl \
|
||||
--namespace certctl \
|
||||
--set server.tls.existingSecret=certctl-server-tls
|
||||
```
|
||||
|
||||
The Secret is mounted read-only at `/etc/certctl/tls/` in the server pod. The `CERTCTL_SERVER_TLS_CERT_PATH` and `CERTCTL_SERVER_TLS_KEY_PATH` env vars are wired to `tls.crt` and `tls.key` keys inside that mount. If `ca.crt` is absent from the Secret, clients that need a CA bundle should use `tls.crt` as the bundle (self-signed case) or mount a separate ConfigMap with the root chain (operator-CA case).
|
||||
|
||||
If the operator sets neither `server.tls.existingSecret` nor `server.tls.certManager.enabled=true`, `helm template` / `helm install` fails at render-time with a diagnostic pointing at this doc. The guard is implemented in `deploy/helm/certctl/templates/_helpers.tpl` under the `certctl.tls.required` helper. This is deliberate: the HTTPS-only server would crash-loop on an empty path, so we fail earlier at Helm-render time.
|
||||
|
||||
## Pattern 3 — cert-manager `Certificate` CR (Helm, opt-in)
|
||||
|
||||
For clusters that already run cert-manager, the chart can provision a `Certificate` CR that writes into the Secret the server pod reads from. This is opt-in — the default is `server.tls.certManager.enabled: false` — because not every cluster has cert-manager installed, and we refuse to ship a chart that silently depends on an external controller.
|
||||
|
||||
```
|
||||
helm install certctl deploy/helm/certctl \
|
||||
--namespace certctl \
|
||||
--set server.tls.certManager.enabled=true \
|
||||
--set server.tls.certManager.issuerRef.name=my-cluster-issuer \
|
||||
--set server.tls.certManager.issuerRef.kind=ClusterIssuer
|
||||
```
|
||||
|
||||
The rendered `Certificate` (see `deploy/helm/certctl/templates/server-certificate.yaml`) writes `tls.crt` + `tls.key` + `ca.crt` into the Secret named by `server.tls.certManager.secretName` (defaults to `<fullname>-tls`). The server pod reads from that same Secret; the agent DaemonSet mounts the same Secret as its CA bundle source.
|
||||
|
||||
cert-manager handles rotation. certctl-server handles in-place reload — see the SIGHUP section below.
|
||||
|
||||
The chart enforces that if `server.tls.certManager.enabled=true`, `server.tls.certManager.issuerRef.name` must also be set. An empty `issuerRef.name` makes `helm template` fail with a diagnostic naming the missing flag.
|
||||
|
||||
## Pattern 4 — Manually-issued from an internal CA
|
||||
|
||||
For operators running neither Helm nor docker-compose (bare-metal / custom orchestration), the server just needs two files on disk pointed at by `CERTCTL_SERVER_TLS_CERT_PATH` and `CERTCTL_SERVER_TLS_KEY_PATH`. Issue the cert from your internal CA with:
|
||||
|
||||
- CN matching the hostname your agents and operators use to dial the server (e.g., `certctl.prod.example.com`)
|
||||
- SAN list covering every hostname and IP that appears in `CERTCTL_SERVER_URL` values across your agent fleet
|
||||
- Key usage: digital signature + key encipherment
|
||||
- Extended key usage: server auth
|
||||
|
||||
Store the key with mode `0600` and owner matching the UID the server runs as (`1000` in our shipped Dockerfile). The server process reads both files during `preflightServerTLS` at startup and again on every SIGHUP.
|
||||
|
||||
The full CA chain that signed the server cert should be distributed to agents, CLI operators, and MCP clients as their `CERTCTL_SERVER_CA_BUNDLE_PATH` — see the client section below.
|
||||
|
||||
## SIGHUP cert rotation
|
||||
|
||||
The server wraps its cert+key pair in a `*certHolder` (see `cmd/server/tls.go`) that guards the loaded `*tls.Certificate` under a `sync.Mutex`. The `*tls.Config` wires `GetCertificate` to the holder, so every new inbound TLS handshake reads whatever cert the holder currently has.
|
||||
|
||||
Send `SIGHUP` to the server PID and the holder re-reads both files from disk. On success, the next new connection uses the new cert; in-flight requests finish on the previous cert. A log line goes out:
|
||||
|
||||
```
|
||||
TLS cert reloaded via SIGHUP cert_path=/etc/certctl/tls/server.crt key_path=/etc/certctl/tls/server.key
|
||||
```
|
||||
|
||||
On failure (missing file, malformed PEM, key does not sign cert), the old cert is retained and an error logs:
|
||||
|
||||
```
|
||||
TLS cert reload failed; continuing with previous cert cert_path=… key_path=… error=…
|
||||
```
|
||||
|
||||
This is deliberately fail-safe on reload (as opposed to fail-loud on startup). A cert-manager renewal race, a partially-copied file, a typo in a rotation script — none of those should crash a running server and drop every agent connection. The operator sees the error in logs, fixes the underlying issue, and sends another `SIGHUP`.
|
||||
|
||||
Pair with cert-manager, certbot `--post-hook`, or any rotation tool that can fire a signal. For docker-compose, `docker compose kill -s HUP certctl-server` works. For Kubernetes, reload is typically handled by cert-manager updating the Secret and the mounted file changing on the next kubelet sync — no explicit SIGHUP needed if the volume mount is `subPath`-free.
|
||||
|
||||
Startup is a different story. If the cert is missing or malformed at process start, the server exits non-zero rather than binding plaintext or attempting a retry loop. That's the HTTPS-only contract.
|
||||
|
||||
## Client-side TLS: agents, CLI, MCP
|
||||
|
||||
Everything that talks to the server enforces HTTPS on the URL.
|
||||
|
||||
### Agent
|
||||
|
||||
`CERTCTL_SERVER_URL` must be `https://…`. `http://`, bare hostnames, `ftp://`, `ws://`, and empty strings are rejected at startup by `validateHTTPSScheme` in `cmd/agent/main.go` with a diagnostic pointing at `upgrade-to-tls.md`. There is no warning-and-proceed path.
|
||||
|
||||
Two additional env vars control how the agent verifies the server cert:
|
||||
|
||||
- `CERTCTL_SERVER_CA_BUNDLE_PATH` — filesystem path to a PEM-encoded CA bundle that signed the server cert. Loaded into `*tls.Config.RootCAs` on the agent's HTTP client. If unset, the agent falls back to the OS system trust store.
|
||||
- `CERTCTL_SERVER_TLS_INSECURE_SKIP_VERIFY` — defaults to `false`. Setting it to `true` skips verification entirely. **Dev-only escape hatch.** The agent logs a prominent warning at startup (`TLS certificate verification is disabled … never enable this in production`). Use this only when dialing a demo server whose cert you haven't bothered to mount into the agent container.
|
||||
|
||||
Equivalent CLI flags: `--ca-bundle <path>` and `--insecure-skip-verify`.
|
||||
|
||||
If both the CA bundle and `InsecureSkipVerify=true` are set, `InsecureSkipVerify` wins — it's the whole point of the flag. Don't do this in production.
|
||||
|
||||
### CLI (`certctl-cli`)
|
||||
|
||||
Same contract as the agent:
|
||||
|
||||
- `CERTCTL_SERVER_URL` defaults to `https://` scheme; `http://` rejected at startup
|
||||
- `--ca-bundle <path>` flag or `CERTCTL_SERVER_CA_BUNDLE_PATH` env var — CA bundle for server cert verification
|
||||
- `--insecure` flag or `CERTCTL_SERVER_TLS_INSECURE_SKIP_VERIFY=true` — skip verification (dev only)
|
||||
- Error diagnostic on empty URL explicitly mentions both `--server` and `CERTCTL_SERVER_URL` so operators see the right knob to turn
|
||||
|
||||
The CLI shares the URL-scheme validation with the agent; the test pins in `cmd/cli/main_test.go:TestValidateHTTPSScheme` cover the full rejection matrix.
|
||||
|
||||
### MCP server (`certctl-mcp-server`)
|
||||
|
||||
Same three controls as CLI, env-var-driven only (no flags — MCP runs as a stdio subprocess and inherits env from the launching LLM client):
|
||||
|
||||
- `CERTCTL_SERVER_URL` must start with `https://`
|
||||
- `CERTCTL_SERVER_CA_BUNDLE_PATH` optional CA bundle
|
||||
- `CERTCTL_SERVER_TLS_INSECURE_SKIP_VERIFY` optional skip
|
||||
|
||||
Claude Desktop / other MCP client configs should set all three in the tool's env block.
|
||||
|
||||
## Troubleshooting: fail-loud preflight errors
|
||||
|
||||
Every preflight failure message ends with `(see docs/tls.md)` so this doc is the first hit when an operator searches. Common failures:
|
||||
|
||||
**`CERTCTL_SERVER_TLS_CERT_PATH is empty: HTTPS-only control plane refuses to start`**
|
||||
Set the env var. For docker-compose this is already set to `/etc/certctl/tls/server.crt` in the shipped compose file — if you're seeing this, check the `certctl-tls-init` service logs to see why the init container didn't populate the volume. For Helm, check that `server.tls.existingSecret` or `server.tls.certManager.enabled=true` is set.
|
||||
|
||||
**`TLS cert file "…" unreadable: …`**
|
||||
The cert path is set but `os.Stat` failed. Check filesystem permissions — the server runs as UID 1000 in our shipped Dockerfile; the cert needs to be readable by that UID. Typos in the path also land here.
|
||||
|
||||
**`TLS cert/key pair invalid (cert="…" key="…"): …`**
|
||||
Both files exist but `tls.LoadX509KeyPair` refused them. Typical causes: the private key does not sign the certificate, the key is encrypted with a passphrase (not supported — remove the passphrase with `openssl pkey` before mounting), or one of the two is DER-encoded instead of PEM. Re-issue the pair from the same CA call and re-mount.
|
||||
|
||||
**Client side: `tls: failed to verify certificate: x509: certificate signed by unknown authority`**
|
||||
The client did not trust the CA that signed the server cert. Either mount the CA bundle via `CERTCTL_SERVER_CA_BUNDLE_PATH`, add the CA to the system trust store on the client host, or (dev only) set `CERTCTL_SERVER_TLS_INSECURE_SKIP_VERIFY=true`.
|
||||
|
||||
**Client side: `tls: first record does not look like a TLS handshake`**
|
||||
The client is speaking plaintext HTTP to an HTTPS server (or vice-versa). Check that `CERTCTL_SERVER_URL` starts with `https://`. If you are upgrading from a pre-v2.2 release and your agents are old, they will surface this error until you roll the DaemonSet — see [`upgrade-to-tls.md`](upgrade-to-tls.md).
|
||||
|
||||
## Related docs
|
||||
|
||||
- [`upgrade-to-tls.md`](upgrade-to-tls.md) — one-step cutover from pre-HTTPS releases
|
||||
- [`quickstart.md`](quickstart.md) — docker-compose walkthrough with HTTPS examples
|
||||
- [`test-env.md`](test-env.md) — integration test environment (also HTTPS-only)
|
||||
- Milestone spec: `prompts/https-everywhere-milestone.md` (authoritative source for locked decisions)
|
||||
@@ -0,0 +1,194 @@
|
||||
# Upgrading to HTTPS-Everywhere (v2.2)
|
||||
|
||||
certctl's control plane is HTTPS-only as of v2.2. There is no `http` mode, no `auto` mode, no dual-listener bind, no N-release migration window. The cutover is a single step. Out-of-date agents that still point at `http://…` fail at the TCP/TLS handshake layer on first connect after the upgrade and stay `Offline` in the dashboard until their env block is updated and the fleet is rolled.
|
||||
|
||||
This doc walks operators through the cutover for the two shipped deployment topologies — docker-compose and Helm — and documents the failure modes and rollback posture explicitly.
|
||||
|
||||
For the deep-dive on cert provisioning patterns, SIGHUP cert reload, and client-side CA-trust configuration, read [`tls.md`](tls.md). This doc is the narrow "how do I upgrade" procedure.
|
||||
|
||||
## Preconditions
|
||||
|
||||
Before you start, confirm:
|
||||
|
||||
- **Shell access** to the server host and every agent host. The cutover requires you to restart the server and update every agent's env block.
|
||||
- **A cert+key source** for the server. Pick one:
|
||||
- An internal CA that can issue a server cert (CN + SAN list covering every hostname / IP agents dial).
|
||||
- A `cert-manager` install in the target Kubernetes cluster, plus a `ClusterIssuer` or `Issuer` you're willing to reference.
|
||||
- Willingness to use the self-signed bootstrap that the shipped `deploy/docker-compose.yml` generates automatically. This is the right choice for dev and demo; it is the wrong choice for production.
|
||||
- **A maintenance window.** Out-of-date agents break at the TLS handshake and stay offline until rolled. Schedule the upgrade so the agent fleet can be updated in the same window as the server.
|
||||
- **Backups.** This is a one-way door (see the Rollback section below). Snapshot your PostgreSQL database before `docker compose down` or `helm upgrade`.
|
||||
|
||||
There is no schema migration tied to this release; the only at-rest state that changes is the `certs` named volume (docker-compose) or the `tls.crt`/`tls.key` Secret (Helm).
|
||||
|
||||
## Procedure — docker-compose operators
|
||||
|
||||
The shipped `deploy/docker-compose.yml` includes a `certctl-tls-init` init container that self-signs an ed25519 cert on first boot and drops `server.crt`, `server.key`, and `ca.crt` into a named volume mounted read-only at `/etc/certctl/tls/` on the server and agent containers. No manual cert provisioning is required for the default stack.
|
||||
|
||||
1. **Pull the HTTPS-everywhere release.** From the repo root:
|
||||
|
||||
```
|
||||
git pull
|
||||
```
|
||||
|
||||
Confirm you're on a tag or `master` that contains the `certctl-tls-init` service in `deploy/docker-compose.yml`. Grep for it: `grep certctl-tls-init deploy/docker-compose.yml` should hit.
|
||||
|
||||
2. **Stop the old plaintext cluster.**
|
||||
|
||||
```
|
||||
docker compose -f deploy/docker-compose.yml down
|
||||
```
|
||||
|
||||
Do not pass `-v`; keeping the PostgreSQL volume preserves your cert inventory, audit trail, and job history across the upgrade.
|
||||
|
||||
3. **Bring the cluster back up with the HTTPS build.**
|
||||
|
||||
```
|
||||
docker compose -f deploy/docker-compose.yml up -d --build
|
||||
```
|
||||
|
||||
The `certctl-tls-init` service runs once, generates the self-signed cert into the `certs` volume, and exits with code 0. The server container waits for `certctl-tls-init` via `depends_on: { condition: service_completed_successfully }` and only starts once the cert material is on disk. The server's Docker healthcheck now uses `curl --cacert /etc/certctl/tls/ca.crt -f https://localhost:8443/health`, so the container only becomes healthy once the HTTPS listener is up and serving the bundled cert correctly.
|
||||
|
||||
4. **Verify the HTTPS endpoint from the host.**
|
||||
|
||||
```
|
||||
curl --cacert $(docker compose -f deploy/docker-compose.yml exec -T certctl-server cat /etc/certctl/tls/ca.crt) https://localhost:8443/health
|
||||
```
|
||||
|
||||
Expect `{"status":"ok"}` with HTTP 200. If you get a TLS verification error, the CA bundle wasn't read correctly — re-run the `exec -T` command and pipe the output directly into `--cacert @-` or save it to a local file first. If you get `connection refused`, the server never finished startup — check `docker compose logs certctl-server` for a fail-loud preflight diagnostic pointing at `docs/tls.md`.
|
||||
|
||||
5. **Confirm the bundled agent reconnects.** Agents inside the compose stack pick up the new URL (`CERTCTL_SERVER_URL=https://certctl-server:8443`) and the bundled CA (`CERTCTL_SERVER_CA_BUNDLE_PATH=/etc/certctl/tls/ca.crt`) from their env block automatically — no per-agent change needed. Tail the agent log:
|
||||
|
||||
```
|
||||
docker compose -f deploy/docker-compose.yml logs -f certctl-agent
|
||||
```
|
||||
|
||||
You should see `heartbeat sent` within 30 seconds. In the dashboard (`https://localhost:8443`), the agent should show as `Online`.
|
||||
|
||||
**External agents** running outside the compose network (e.g., the `install-agent.sh`-installed systemd service on a separate host) need their env block updated manually before the cutover — see the Agent env block section below.
|
||||
|
||||
## Procedure — Helm operators
|
||||
|
||||
The Helm chart does not self-sign. It refuses to render (`helm template` exits non-zero) unless you configure one of two cert sources: an operator-supplied Secret, or a cert-manager `Certificate` CR. See [`tls.md`](tls.md) for the full pattern catalog.
|
||||
|
||||
1. **Provision cert material.** Pick one of:
|
||||
|
||||
- **Operator-supplied Secret.** Issue a cert from your internal CA (or any other source) and load it into a `kubernetes.io/tls` Secret in the certctl namespace:
|
||||
|
||||
```
|
||||
kubectl create secret tls certctl-server-tls \
|
||||
--cert=server.crt --key=server.key \
|
||||
--namespace certctl
|
||||
```
|
||||
|
||||
- **cert-manager.** Set `server.tls.certManager.enabled=true` on the upgrade and reference an existing `ClusterIssuer` or `Issuer`:
|
||||
|
||||
```
|
||||
--set server.tls.certManager.enabled=true
|
||||
--set server.tls.certManager.issuerRef.name=my-cluster-issuer
|
||||
--set server.tls.certManager.issuerRef.kind=ClusterIssuer
|
||||
```
|
||||
|
||||
2. **Upgrade the release.**
|
||||
|
||||
```
|
||||
helm upgrade certctl deploy/helm/certctl \
|
||||
--namespace certctl \
|
||||
--set server.tls.existingSecret=certctl-server-tls
|
||||
```
|
||||
|
||||
(Or the `certManager` variant.) If you omit both `server.tls.existingSecret` and `server.tls.certManager.enabled`, the chart fails at render time with a diagnostic pointing at `docs/tls.md`. That guard exists precisely so you catch the missing config at `helm upgrade` time, not at pod-crash-loop time.
|
||||
|
||||
3. **Verify the HTTPS endpoint from inside the cluster.** Port-forward and curl with the CA bundle:
|
||||
|
||||
```
|
||||
kubectl port-forward -n certctl svc/certctl-server 8443:8443 &
|
||||
kubectl get secret -n certctl certctl-server-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/certctl-ca.crt
|
||||
curl --cacert /tmp/certctl-ca.crt https://localhost:8443/health
|
||||
```
|
||||
|
||||
Expect `{"status":"ok"}`. If the Secret does not contain a `ca.crt` key (operator-supplied Secrets often don't), use `tls.crt` as the bundle instead — for a self-signed cert the two files are identical, and for a cert chained to an internal CA you should separately distribute the root CA bundle via ConfigMap or mounted file.
|
||||
|
||||
4. **Update every agent manifest.** Agents outside this Helm release (or in a separately-managed DaemonSet) need their env block updated:
|
||||
|
||||
```
|
||||
- name: CERTCTL_SERVER_URL
|
||||
value: "https://certctl-server.certctl.svc.cluster.local:8443"
|
||||
- name: CERTCTL_SERVER_CA_BUNDLE_PATH
|
||||
value: "/etc/certctl/tls/ca.crt"
|
||||
```
|
||||
|
||||
Mount the server's Secret (or a separate CA-bundle Secret / ConfigMap) at `/etc/certctl/tls/` as a read-only volume. If you bundle the agent via the shipped Helm chart's DaemonSet, the wiring is already done — set `agent.enabled=true` and the chart mounts the same Secret.
|
||||
|
||||
5. **Roll the agent DaemonSet.**
|
||||
|
||||
```
|
||||
kubectl rollout restart ds/certctl-agent -n certctl
|
||||
kubectl rollout status ds/certctl-agent -n certctl
|
||||
```
|
||||
|
||||
Every agent pod restarts with the new URL + CA bundle and reconnects on HTTPS. The dashboard shows agents flip from `Offline` to `Online` as pods finish rolling.
|
||||
|
||||
## Agent env block — external hosts
|
||||
|
||||
Agents installed on bare-metal or VM hosts via `install-agent.sh` (systemd on Linux, launchd on macOS) read config from `/etc/certctl/agent.env` (Linux) or `~/Library/Application Support/certctl/agent.env` (macOS). On cutover, append or update:
|
||||
|
||||
```
|
||||
CERTCTL_SERVER_URL=https://certctl.example.com:8443
|
||||
CERTCTL_SERVER_CA_BUNDLE_PATH=/etc/certctl/tls/ca.crt
|
||||
# CERTCTL_SERVER_TLS_INSECURE_SKIP_VERIFY=false # Dev only. Never set to true in production.
|
||||
```
|
||||
|
||||
Distribute the CA bundle (the same `ca.crt` the server holds, or the root chain if you issued the server cert from an intermediate) to every agent host. The path under `CERTCTL_SERVER_CA_BUNDLE_PATH` must be readable by the UID the agent service runs as.
|
||||
|
||||
Restart the service after editing:
|
||||
|
||||
- Linux: `systemctl restart certctl-agent`
|
||||
- macOS: `launchctl kickstart -k system/com.certctl.agent`
|
||||
|
||||
The agent refuses to start on an `http://` URL and exits with a pre-flight diagnostic that names this doc. That rejection happens before any network call — no spurious half-connected state.
|
||||
|
||||
## Failure mode
|
||||
|
||||
Out-of-date agents still configured with `CERTCTL_SERVER_URL=http://…` fail on first reconnect after the cutover. The failure surfaces as one of:
|
||||
|
||||
- `dial tcp …: connect: connection refused` — the server is no longer listening on a plaintext port. The new release binds only a TLS listener; attempting a plaintext `connect()` gets refused at the kernel level because nothing holds the socket.
|
||||
- `tls: first record does not look like a TLS handshake` — depending on timing and proxy layers (e.g., a load balancer that accepts the TCP connection before forwarding), the client may negotiate TCP, send an HTTP request line, and have the server's TLS stack reject it.
|
||||
|
||||
Agents in this state surface as `Offline` in the dashboard. They stay offline until their env block is updated and the service restarts. There is no graceful 400-with-migration-URL response because there is no HTTP listener to serve one from — the entire plaintext call path is removed by design.
|
||||
|
||||
If you see an unexpected agent stay `Offline` past the cutover window, SSH to the host and check the agent log. On a systemd host:
|
||||
|
||||
```
|
||||
journalctl -u certctl-agent -n 100
|
||||
```
|
||||
|
||||
Look for `URL scheme "http" is not supported: HTTPS-only control plane refuses to start (see docs/upgrade-to-tls.md)`. That's the pre-flight rejection. Update `CERTCTL_SERVER_URL`, restart the service, and the agent reconnects.
|
||||
|
||||
## Rollback
|
||||
|
||||
**There is no rollback window.** The upgrade is a one-way door. The rationale lives in §3.7 of `prompts/https-everywhere-milestone.md`: a cert-lifecycle product that bridges back to plaintext after committing to HTTPS is advertising that its own security posture is negotiable.
|
||||
|
||||
If you need to revert, you have two options:
|
||||
|
||||
1. **Stay on the pre-HTTPS release.** Do not upgrade until you are ready to run HTTPS on the control plane. Pin your `docker-compose.yml` or `helm upgrade` command to the last pre-v2.2 tag.
|
||||
2. **Rollback the release.** `helm rollback certctl <previous-revision>` or `git checkout <previous-tag> && docker compose up -d --build`. This rolls back the server, the compose topology, and the Helm chart in lockstep. Your PostgreSQL volume — cert inventory, audit trail, jobs — survives the rollback; nothing in this milestone changes the database schema.
|
||||
|
||||
Option 2 drops you back to the plaintext world. It should be treated as an emergency measure, not a supported migration path.
|
||||
|
||||
## After the cutover
|
||||
|
||||
Once every agent is `Online`, confirm a few invariants:
|
||||
|
||||
- `curl -sS -o /dev/null -w "%{http_code}\n" http://localhost:8443/health` returns `000` with `Connection refused` (no HTTP listener). Plaintext is gone.
|
||||
- `openssl s_client -connect localhost:8443 -tls1_2 </dev/null` fails the handshake. TLS 1.2 is rejected.
|
||||
- `openssl s_client -connect localhost:8443 -tls1_3 </dev/null` succeeds and prints the server's SAN list. TLS 1.3 is live.
|
||||
- A cert rotation test: overwrite the server cert on disk, `kill -HUP` the server PID, confirm the new cert serves on the next `openssl s_client -connect … -showcerts` without a process restart. See the SIGHUP section in [`tls.md`](tls.md).
|
||||
|
||||
Update your runbooks. Every `http://certctl.example.com` URL in internal documentation, monitoring config, and on-call playbooks should become `https://certctl.example.com` plus a CA-trust note.
|
||||
|
||||
## Related docs
|
||||
|
||||
- [`tls.md`](tls.md) — cert provisioning patterns, SIGHUP rotation, troubleshooting
|
||||
- [`quickstart.md`](quickstart.md) — docker-compose walkthrough (post-HTTPS)
|
||||
- [`test-env.md`](test-env.md) — integration test environment (HTTPS-only)
|
||||
- Milestone spec: `prompts/https-everywhere-milestone.md`
|
||||
+1
-1
@@ -107,7 +107,7 @@ The demo seeds certificates across multiple issuers, agents, and deployment targ
|
||||
```bash
|
||||
git clone https://github.com/shankar0123/certctl.git
|
||||
cd certctl/deploy && docker compose up -d
|
||||
# Dashboard at http://localhost:8443
|
||||
# Dashboard at https://localhost:8443 (self-signed cert — pin deploy/test/certs/ca.crt)
|
||||
```
|
||||
|
||||
See the [Quickstart Guide](quickstart.md) for a full walkthrough, or explore the [5 turnkey examples](../examples/) for specific scenarios (ACME+NGINX, wildcard DNS-01, private CA+Traefik, step-ca+HAProxy, multi-issuer).
|
||||
|
||||
Reference in New Issue
Block a user