The pre-G-1 config validator accepted CERTCTL_AUTH_TYPE=jwt and the
startup log faithfully echoed 'authentication enabled type=jwt'.
Reasonable people read that and concluded JWT auth was on. It wasn't.
The auth-middleware wiring at cmd/server/main.go unconditionally routed
every request through the api-key bearer middleware regardless of
cfg.Auth.Type. So CERTCTL_AUTH_TYPE=jwt quietly compared the incoming
'Authorization: Bearer <token>' against whatever string the operator put
in CERTCTL_AUTH_SECRET — real JWT clients got 401, and operators who
treated CERTCTL_AUTH_SECRET as a *signing* secret (because they thought
they were configuring JWT) had effectively handed an attacker an api-key.
A security finding masquerading as a config option.
We chose the audit-recommended structural fix: remove the option, fail
fast at startup, and add the gateway-fronting pattern as the documented
forward path. Implementing JWT middleware would have meant jwks vs
static-secret rotation, claim mapping, expiry enforcement, audience and
issuer validation, key rollover semantics, and regression coverage at the
same depth as the existing api-key path — a feature, not a fix. Operators
who genuinely need JWT/OIDC front certctl with an authenticating gateway
(oauth2-proxy / Envoy ext_authz / Traefik ForwardAuth / Pomerium /
Authelia) and run the upstream certctl with CERTCTL_AUTH_TYPE=none. Same
shape works on docker-compose and Helm.
The change is comprehensive across 7 phases — every surface that
mentioned 'jwt' as a certctl-auth-type is updated, plus structural
backstops (typed enum, runtime guard, helm template validation, CI grep
guard) so the lie can't reappear.
Files changed:
Phase 1 — production code (typed enum + jwt removal):
- internal/config/config.go: AuthType typed alias + AuthTypeAPIKey /
AuthTypeNone constants + ValidAuthTypes() helper. Validate() routes
literal 'jwt' through a dedicated multi-line diagnostic naming the
authenticating-gateway pattern, then cross-checks against
ValidAuthTypes(). Secret-required branch simplified to api-key-only.
Field comment on AuthConfig.Type rewritten to drop jwt and point at
the gateway pattern.
- internal/api/middleware/middleware.go: AuthConfig.Type field comment
references the typed config.AuthType constants.
- internal/api/handler/health.go: same treatment for HealthHandler.AuthType.
- cmd/server/main.go: defense-in-depth runtime switch immediately after
config.Load() — exits 1 on any unsupported auth-type that bypassed the
validator. Auth-disabled startup log explicitly names the
authenticating-gateway pattern.
Phase 2 — tests (Red→Green, contract pinning):
- internal/config/config_test.go: TestValidate_JWTAuth_RejectedDedicated
(two table rows pinning the dedicated G-1 error fires regardless of
whether Secret is set), TestValidAuthTypesDoesNotContainJWT (property
guard against future re-introduction),
TestValidAuthTypesIsExactly_APIKey_None (allowed-set contract),
TestValidate_GenericInvalidAuthType (pins non-jwt invalid values still
hit the generic invalid-auth-type error). Removed the prior
TestValidate_JWTAuth_MissingSecret happy-path since its premise is
inverted post-G-1.
- internal/api/handler/health_test.go: removed
TestAuthInfo_ReturnsAuthType_JWT (which baked the silent-downgrade lie
into the regression suite). Pre-existing _APIKey test continues to
cover the api-key happy path.
Phase 3 — spec, docs, env templates:
- api/openapi.yaml: auth_type enum dropped to [api-key, none] with
inline comment naming the G-1 closure.
- .env.example (root): CERTCTL_AUTH_TYPE comment block rewritten to drop
jwt and point at the gateway pattern; secret-required conditional
simplified to api-key-only.
- docs/architecture.md: middleware-stack bullet rewritten to drop the
JWT mention; new H3 'Authenticating-gateway pattern (JWT, OIDC, mTLS)'
section explaining the design rationale and listing oauth2-proxy /
Envoy ext_authz / Traefik ForwardAuth / Pomerium / Authelia / Caddy
forward_auth / Apache mod_auth_openidc / nginx auth_request as the
standard fronting options.
- docs/upgrade-to-v2-jwt-removal.md (new ~125 lines): migration guide
with preconditions, what-changes, both recovery paths, complete
docker-compose oauth2-proxy walkthrough, Traefik ForwardAuth and Envoy
ext_authz patterns, rollback posture.
Phase 4 — Helm chart (template validation + docs):
- deploy/helm/certctl/templates/_helpers.tpl: new certctl.validateAuthType
helper mirroring the existing certctl.tls.required pattern. Fails
template render on any server.auth.type outside {api-key, none} with
a multi-line diagnostic.
- deploy/helm/certctl/templates/server-deployment.yaml,
server-configmap.yaml, server-secret.yaml: invoke the helper at the
top of each template that depends on .Values.server.auth.type.
- deploy/helm/certctl/values.yaml: auth: block comment expanded with the
G-1 rationale and gateway-pattern cross-reference.
- deploy/helm/CHART_SUMMARY.md: server.auth.type table row now surfaces
the allowed set and points at the upgrade doc.
- deploy/helm/certctl/README.md: new 'JWT / OIDC via authenticating
gateway' section with a Kubernetes-flavored oauth2-proxy + certctl
walkthrough.
Phase 5 — release surface:
- CHANGELOG.md: new [unreleased] top entry with Breaking / Removed /
Added / Changed sections; explicit pointer at
docs/upgrade-to-v2-jwt-removal.md from the Breaking subsection.
Phase 6 — CI guardrail:
- .github/workflows/ci.yml: new 'Forbidden auth-type literal regression
guard (G-1)' step. Scoped patterns catch the actual regression shapes
(map literal, slice literal, switch case, OpenAPI enum, env-file
default, AuthType('jwt') cast). Comments and the dedicated rejection
branch are intentionally exempt; connector-package JWT references
(Google OAuth2 / step-ca) are exempt as out-of-scope external
protocols. Verified locally: the guard passes on the actual tree and
fires on all 4 synthetic regression patterns.
Out of scope (explicitly untouched):
- internal/connector/discovery/gcpsm/gcpsm.go — Google OAuth2 service-
account JWT (external protocol).
- internal/connector/issuer/googlecas/googlecas.go — same.
- internal/connector/issuer/stepca/stepca.go — step-ca's provisioner
one-time-token JWT for /sign API.
- docs/test-env.md, docs/connectors.md, docs/features.md — describe
external CAs' use of JWT, not certctl's auth shape.
- Implementing actual JWT middleware. Feature, not a fix.
Verification (all gates pass):
- go build ./... — clean
- go vet ./... — clean
- go test -short ./... — every package green
- go test -short -race ./internal/config/... ./internal/api/... — clean
- govulncheck ./... — no vulnerabilities in our code
- helm lint deploy/helm/certctl/ — clean
- helm template with auth.type=api-key — renders OK
- helm template with auth.type=none — renders OK
- helm template with auth.type=jwt — fails with validateAuthType
diagnostic (exit 1)
- python3 yaml.safe_load on api/openapi.yaml — parses
- CI guardrail mirror — clean on real tree, fires on all 4 synthetic
regression patterns
- Smoke test: 'CERTCTL_AUTH_TYPE=jwt ./certctl-server' exits non-zero
with: 'Failed to load configuration: CERTCTL_AUTH_TYPE=jwt is no
longer accepted (G-1 silent auth downgrade): no JWT middleware ships
with certctl. To use JWT/OIDC, run an authenticating gateway
(oauth2-proxy / Envoy ext_authz / Traefik ForwardAuth / Pomerium) in
front of certctl and set CERTCTL_AUTH_TYPE=none on the upstream.
See docs/architecture.md "Authenticating-gateway pattern" and
docs/upgrade-to-v2-jwt-removal.md for the migration walkthrough'
config pkg coverage: ValidAuthTypes 100%, Validate 94.7%, total 75.5%.
Refs: coverage-gap-audit-2026-04-24-v5/unified-audit.md
§2 P1 cluster, cat-g-jwt_silent_auth_downgrade
Audit recommendation followed verbatim: 'Remove jwt from
validAuthTypes until middleware ships'.
12 KiB
Certctl Helm Chart - Complete Summary
Overview
A production-ready Helm chart for deploying certctl (self-hosted certificate lifecycle management platform) on Kubernetes. The chart provides:
- High availability support with multi-replica deployments
- Persistent PostgreSQL database with automatic schema migration
- DaemonSet or Deployment-based agent deployment
- Comprehensive security contexts and RBAC
- Multiple deployment scenarios (dev, prod, HA, external DB)
- Full documentation and examples
Chart Metadata
- Name: certctl
- Chart Version: 0.1.0
- App Version: 2.1.0
- Type: application
- License: BSL-1.1 (converts to Apache 2.0 in 2033)
File Structure
deploy/helm/
├── README.md # Main Helm chart documentation
├── DEPLOYMENT_GUIDE.md # Step-by-step deployment guide
├── CHART_SUMMARY.md # This file
│
├── certctl/
│ ├── Chart.yaml # Chart metadata
│ ├── values.yaml # Default configuration values
│ ├── .helmignore # Files to ignore when building chart
│ │
│ └── templates/
│ ├── _helpers.tpl # Helm template helper functions
│ ├── NOTES.txt # Post-deployment notes
│ │
│ ├── server-deployment.yaml # Certctl API server deployment
│ ├── server-service.yaml # Server Kubernetes service
│ ├── server-configmap.yaml # Server configuration
│ ├── server-secret.yaml # Server secrets (API key, DB password, etc)
│ │
│ ├── postgres-statefulset.yaml # PostgreSQL database statefulset
│ ├── postgres-service.yaml # PostgreSQL headless service
│ ├── postgres-secret.yaml # Database credentials secret
│ │
│ ├── agent-daemonset.yaml # Certctl agent daemonset/deployment
│ ├── agent-configmap.yaml # Agent configuration
│ │
│ ├── ingress.yaml # Optional ingress resource
│ └── serviceaccount.yaml # ServiceAccount and RBAC
│
└── examples/
├── values-dev.yaml # Development/testing configuration
├── values-prod-ha.yaml # Production HA configuration
├── values-external-db.yaml # External PostgreSQL (RDS, Cloud SQL)
└── values-acme-dns01.yaml # ACME with DNS-01 (Let's Encrypt)
Key Components
1. Server Deployment
File: templates/server-deployment.yaml
- Manages certctl API server instances
- Configurable replicas (default: 1)
- Health checks (liveness & readiness probes)
- Security context: non-root user, read-only filesystem
- Resource limits (default: 500m CPU, 512Mi memory)
- Automatic restart on failure
Values:
server:
replicas: 1
port: 8443
auth:
type: api-key
apiKey: "REQUIRED"
resources:
requests: {cpu: 100m, memory: 128Mi}
limits: {cpu: 500m, memory: 512Mi}
2. PostgreSQL StatefulSet
File: templates/postgres-statefulset.yaml
- Persistent database storage
- Automatic schema migrations on startup
- Single replica (can be extended with external HA tools)
- Health checks via pg_isready
- Configurable storage size and class
- Security context: non-root user (UID 999)
Values:
postgresql:
enabled: true
storage:
size: 10Gi
storageClass: "" # Use default
auth:
database: certctl
username: certctl
password: "REQUIRED"
3. Agent DaemonSet/Deployment
File: templates/agent-daemonset.yaml
- DaemonSet mode: one agent per Kubernetes node
- Deployment mode: custom number of agent replicas
- Local key storage with secure permissions (0600)
- Health checks and automatic restart
- Optional certificate discovery from filesystem
Values:
agent:
enabled: true
kind: DaemonSet # or Deployment
replicas: 1 # for Deployment only
keyDir: /var/lib/certctl/keys
discoveryDirs: "/etc/ssl/certs" # optional
4. Ingress (Optional)
File: templates/ingress.yaml
- Optional HTTPS ingress
- cert-manager integration for automatic TLS
- Multiple host support
- Path-based routing
Values:
ingress:
enabled: false
className: nginx
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
hosts:
- host: certctl.example.com
paths:
- path: /
pathType: Prefix
5. ConfigMaps and Secrets
Files:
server-configmap.yaml- Non-secret server configurationserver-secret.yaml- API key, database URL, SMTP passwordpostgres-secret.yaml- Database credentialsagent-configmap.yaml- Agent configuration
All secrets are base64-encoded and stored in Kubernetes Secrets.
6. ServiceAccount and RBAC
File: templates/serviceaccount.yaml
- Optional ServiceAccount creation
- Optional RBAC (ClusterRole, ClusterRoleBinding)
- Namespace-scoped by default
Deployment Scenarios
Development Setup
Use examples/values-dev.yaml:
helm install certctl certctl/ \
--values examples/values-dev.yaml \
--set server.auth.apiKey="dev-key" \
--set postgresql.auth.password="dev-password"
Features:
- Single server replica
- Demo auth (no API key required)
- Small database (5Gi)
- LoadBalancer service for easy access
- Debug logging level
Production HA Setup
Use examples/values-prod-ha.yaml:
helm install certctl certctl/ \
--values examples/values-prod-ha.yaml \
--set server.auth.apiKey="$(openssl rand -base64 32)" \
--set postgresql.auth.password="$(openssl rand -base64 32)"
Features:
- 3 server replicas with pod anti-affinity
- Large database storage (100Gi)
- Pod disruption budgets
- Prometheus monitoring enabled
- Production resource limits
External PostgreSQL
Use examples/values-external-db.yaml:
helm install certctl certctl/ \
--values examples/values-external-db.yaml \
--set postgresql.enabled=false \
--set 'server.env.CERTCTL_DATABASE_URL=postgres://...'
Use cases:
- AWS RDS
- Google Cloud SQL
- Azure Database for PostgreSQL
- External self-managed PostgreSQL
ACME with DNS-01
Use examples/values-acme-dns01.yaml:
helm install certctl certctl/ \
--values examples/values-acme-dns01.yaml
Enables:
- Automatic certificate issuance from Let's Encrypt
- DNS-01 challenge (wildcard support)
- Custom DNS provider scripts
Configuration Options
Server Configuration
| Option | Default | Description |
|---|---|---|
server.replicas |
1 | Number of server replicas |
server.port |
8443 | Server port |
server.auth.type |
api-key | Authentication type — api-key or none (G-1: jwt removed; for JWT/OIDC use a fronting authenticating gateway, see docs/architecture.md and docs/upgrade-to-v2-jwt-removal.md) |
server.auth.apiKey |
"" | API key (REQUIRED when auth.type=api-key) |
server.logging.level |
info | Log level |
server.logging.format |
json | Log format |
PostgreSQL Configuration
| Option | Default | Description |
|---|---|---|
postgresql.enabled |
true | Enable internal PostgreSQL |
postgresql.storage.size |
10Gi | Database storage size |
postgresql.storage.storageClass |
"" | Storage class name |
postgresql.auth.password |
"" | Database password (REQUIRED) |
Agent Configuration
| Option | Default | Description |
|---|---|---|
agent.enabled |
true | Deploy agents |
agent.kind |
DaemonSet | DaemonSet or Deployment |
agent.replicas |
1 | Replicas (Deployment only) |
agent.keyDir |
/var/lib/certctl/keys | Key storage directory |
Issuer Configuration
| Option | Default | Description |
|---|---|---|
server.issuer.local.enabled |
true | Enable Local CA |
server.issuer.acme.enabled |
false | Enable ACME |
server.issuer.acme.directoryURL |
"" | ACME directory URL |
server.issuer.acme.email |
"" | ACME email |
server.issuer.acme.challengeType |
http-01 | Challenge type |
See values.yaml for complete configuration options.
Helm Template Functions
Defined in templates/_helpers.tpl:
| Function | Purpose |
|---|---|
certctl.name |
Chart name |
certctl.fullname |
Full release name |
certctl.chart |
Chart name and version |
certctl.labels |
Common labels |
certctl.selectorLabels |
Selector labels |
certctl.serverSelectorLabels |
Server selector labels |
certctl.agentSelectorLabels |
Agent selector labels |
certctl.postgresSelectorLabels |
PostgreSQL selector labels |
certctl.serviceAccountName |
ServiceAccount name |
certctl.serverImage |
Server image URI |
certctl.agentImage |
Agent image URI |
certctl.postgresImage |
PostgreSQL image URI |
certctl.databaseURL |
Database connection string |
certctl.serverURL |
Server URL for agents |
Security Features
Pod Security
- Non-root users (UID 1000 for app, UID 999 for PostgreSQL)
- Read-only root filesystems
- No privilege escalation
- Dropped capabilities (ALL)
- Resource limits to prevent DoS
Secrets Management
- All sensitive data in Kubernetes Secrets
- Base64 encoded at rest
- Can be integrated with:
- sealed-secrets
- external-secrets
- Vault
- AWS Secrets Manager
RBAC
- ServiceAccount per release
- Optional ClusterRole/ClusterRoleBinding
- Extensible for custom permissions
Network Security
- Support for Kubernetes NetworkPolicies
- Service-to-service communication via internal DNS
- Optional Ingress with TLS
Monitoring and Observability
Health Checks
- Liveness probes (detect dead containers)
- Readiness probes (detect not-ready services)
- HTTP endpoints:
/health,/readyz
Logging
- Structured JSON logging
- Request ID propagation
- Configurable log levels (debug, info, warn, error)
Metrics
- Prometheus metrics endpoint:
/api/v1/metrics/prometheus - Optional ServiceMonitor for Prometheus Operator
- Built-in metrics:
- Certificate counts by status
- Agent counts and status
- Job completion/failure rates
- Server uptime
Installation Quick Reference
# Development
helm install certctl certctl/ \
--set server.auth.apiKey=dev \
--set postgresql.auth.password=dev
# Production HA
helm install certctl certctl/ \
--values examples/values-prod-ha.yaml \
--set server.auth.apiKey="$(openssl rand -base64 32)" \
--set postgresql.auth.password="$(openssl rand -base64 32)"
# External database
helm install certctl certctl/ \
--values examples/values-external-db.yaml \
--set postgresql.enabled=false \
--set 'server.env.CERTCTL_DATABASE_URL=postgres://...'
# ACME with Let's Encrypt
helm install certctl certctl/ \
--set server.issuer.acme.enabled=true \
--set server.issuer.acme.directoryURL=https://acme-v02.api.letsencrypt.org/directory
# Check status
kubectl get pods -l app.kubernetes.io/instance=certctl
kubectl logs -l app.kubernetes.io/component=server -f
# Upgrade
helm upgrade certctl certctl/ -f new-values.yaml
# Uninstall
helm uninstall certctl
Best Practices
1. Use Secrets Management
# Use sealed-secrets
kubectl create secret generic certctl-secrets \
--from-literal=api-key="$(openssl rand -base64 32)" \
--dry-run=client -o yaml | kubeseal -f - | kubectl apply -f -
2. Configure Resource Limits
Match limits to your cluster capacity:
server:
resources:
requests: {cpu: 250m, memory: 256Mi}
limits: {cpu: 1000m, memory: 512Mi}
3. Enable HA for Production
server:
replicas: 3
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution: [...]
4. Use Persistent Storage
postgresql:
storage:
size: 100Gi
storageClass: fast-ssd
5. Enable Monitoring
monitoring:
enabled: true
serviceMonitor:
enabled: true
Documentation
- README.md - Complete Helm chart documentation
- DEPLOYMENT_GUIDE.md - Step-by-step deployment instructions
- values.yaml - Commented configuration reference
Support
For issues, questions, or contributions:
- GitHub: https://github.com/shankar0123/certctl
- Documentation: https://github.com/shankar0123/certctl/tree/main/docs
License
BSL-1.1 (Business Source License) Converts to Apache 2.0 on March 14, 2033