Files
certctl/deploy/helm/CHART_SUMMARY.md
T
shankar0123 9c1d446e40 fix(security,config): remove unimplemented JWT auth-type, close silent downgrade (G-1)
The pre-G-1 config validator accepted CERTCTL_AUTH_TYPE=jwt and the
startup log faithfully echoed 'authentication enabled type=jwt'.
Reasonable people read that and concluded JWT auth was on. It wasn't.
The auth-middleware wiring at cmd/server/main.go unconditionally routed
every request through the api-key bearer middleware regardless of
cfg.Auth.Type. So CERTCTL_AUTH_TYPE=jwt quietly compared the incoming
'Authorization: Bearer <token>' against whatever string the operator put
in CERTCTL_AUTH_SECRET — real JWT clients got 401, and operators who
treated CERTCTL_AUTH_SECRET as a *signing* secret (because they thought
they were configuring JWT) had effectively handed an attacker an api-key.
A security finding masquerading as a config option.

We chose the audit-recommended structural fix: remove the option, fail
fast at startup, and add the gateway-fronting pattern as the documented
forward path. Implementing JWT middleware would have meant jwks vs
static-secret rotation, claim mapping, expiry enforcement, audience and
issuer validation, key rollover semantics, and regression coverage at the
same depth as the existing api-key path — a feature, not a fix. Operators
who genuinely need JWT/OIDC front certctl with an authenticating gateway
(oauth2-proxy / Envoy ext_authz / Traefik ForwardAuth / Pomerium /
Authelia) and run the upstream certctl with CERTCTL_AUTH_TYPE=none. Same
shape works on docker-compose and Helm.

The change is comprehensive across 7 phases — every surface that
mentioned 'jwt' as a certctl-auth-type is updated, plus structural
backstops (typed enum, runtime guard, helm template validation, CI grep
guard) so the lie can't reappear.

Files changed:

Phase 1 — production code (typed enum + jwt removal):
- internal/config/config.go: AuthType typed alias + AuthTypeAPIKey /
  AuthTypeNone constants + ValidAuthTypes() helper. Validate() routes
  literal 'jwt' through a dedicated multi-line diagnostic naming the
  authenticating-gateway pattern, then cross-checks against
  ValidAuthTypes(). Secret-required branch simplified to api-key-only.
  Field comment on AuthConfig.Type rewritten to drop jwt and point at
  the gateway pattern.
- internal/api/middleware/middleware.go: AuthConfig.Type field comment
  references the typed config.AuthType constants.
- internal/api/handler/health.go: same treatment for HealthHandler.AuthType.
- cmd/server/main.go: defense-in-depth runtime switch immediately after
  config.Load() — exits 1 on any unsupported auth-type that bypassed the
  validator. Auth-disabled startup log explicitly names the
  authenticating-gateway pattern.

Phase 2 — tests (Red→Green, contract pinning):
- internal/config/config_test.go: TestValidate_JWTAuth_RejectedDedicated
  (two table rows pinning the dedicated G-1 error fires regardless of
  whether Secret is set), TestValidAuthTypesDoesNotContainJWT (property
  guard against future re-introduction),
  TestValidAuthTypesIsExactly_APIKey_None (allowed-set contract),
  TestValidate_GenericInvalidAuthType (pins non-jwt invalid values still
  hit the generic invalid-auth-type error). Removed the prior
  TestValidate_JWTAuth_MissingSecret happy-path since its premise is
  inverted post-G-1.
- internal/api/handler/health_test.go: removed
  TestAuthInfo_ReturnsAuthType_JWT (which baked the silent-downgrade lie
  into the regression suite). Pre-existing _APIKey test continues to
  cover the api-key happy path.

Phase 3 — spec, docs, env templates:
- api/openapi.yaml: auth_type enum dropped to [api-key, none] with
  inline comment naming the G-1 closure.
- .env.example (root): CERTCTL_AUTH_TYPE comment block rewritten to drop
  jwt and point at the gateway pattern; secret-required conditional
  simplified to api-key-only.
- docs/architecture.md: middleware-stack bullet rewritten to drop the
  JWT mention; new H3 'Authenticating-gateway pattern (JWT, OIDC, mTLS)'
  section explaining the design rationale and listing oauth2-proxy /
  Envoy ext_authz / Traefik ForwardAuth / Pomerium / Authelia / Caddy
  forward_auth / Apache mod_auth_openidc / nginx auth_request as the
  standard fronting options.
- docs/upgrade-to-v2-jwt-removal.md (new ~125 lines): migration guide
  with preconditions, what-changes, both recovery paths, complete
  docker-compose oauth2-proxy walkthrough, Traefik ForwardAuth and Envoy
  ext_authz patterns, rollback posture.

Phase 4 — Helm chart (template validation + docs):
- deploy/helm/certctl/templates/_helpers.tpl: new certctl.validateAuthType
  helper mirroring the existing certctl.tls.required pattern. Fails
  template render on any server.auth.type outside {api-key, none} with
  a multi-line diagnostic.
- deploy/helm/certctl/templates/server-deployment.yaml,
  server-configmap.yaml, server-secret.yaml: invoke the helper at the
  top of each template that depends on .Values.server.auth.type.
- deploy/helm/certctl/values.yaml: auth: block comment expanded with the
  G-1 rationale and gateway-pattern cross-reference.
- deploy/helm/CHART_SUMMARY.md: server.auth.type table row now surfaces
  the allowed set and points at the upgrade doc.
- deploy/helm/certctl/README.md: new 'JWT / OIDC via authenticating
  gateway' section with a Kubernetes-flavored oauth2-proxy + certctl
  walkthrough.

Phase 5 — release surface:
- CHANGELOG.md: new [unreleased] top entry with Breaking / Removed /
  Added / Changed sections; explicit pointer at
  docs/upgrade-to-v2-jwt-removal.md from the Breaking subsection.

Phase 6 — CI guardrail:
- .github/workflows/ci.yml: new 'Forbidden auth-type literal regression
  guard (G-1)' step. Scoped patterns catch the actual regression shapes
  (map literal, slice literal, switch case, OpenAPI enum, env-file
  default, AuthType('jwt') cast). Comments and the dedicated rejection
  branch are intentionally exempt; connector-package JWT references
  (Google OAuth2 / step-ca) are exempt as out-of-scope external
  protocols. Verified locally: the guard passes on the actual tree and
  fires on all 4 synthetic regression patterns.

Out of scope (explicitly untouched):
- internal/connector/discovery/gcpsm/gcpsm.go — Google OAuth2 service-
  account JWT (external protocol).
- internal/connector/issuer/googlecas/googlecas.go — same.
- internal/connector/issuer/stepca/stepca.go — step-ca's provisioner
  one-time-token JWT for /sign API.
- docs/test-env.md, docs/connectors.md, docs/features.md — describe
  external CAs' use of JWT, not certctl's auth shape.
- Implementing actual JWT middleware. Feature, not a fix.

Verification (all gates pass):
- go build ./... — clean
- go vet ./... — clean
- go test -short ./... — every package green
- go test -short -race ./internal/config/... ./internal/api/... — clean
- govulncheck ./... — no vulnerabilities in our code
- helm lint deploy/helm/certctl/ — clean
- helm template with auth.type=api-key — renders OK
- helm template with auth.type=none — renders OK
- helm template with auth.type=jwt — fails with validateAuthType
  diagnostic (exit 1)
- python3 yaml.safe_load on api/openapi.yaml — parses
- CI guardrail mirror — clean on real tree, fires on all 4 synthetic
  regression patterns
- Smoke test: 'CERTCTL_AUTH_TYPE=jwt ./certctl-server' exits non-zero
  with: 'Failed to load configuration: CERTCTL_AUTH_TYPE=jwt is no
  longer accepted (G-1 silent auth downgrade): no JWT middleware ships
  with certctl. To use JWT/OIDC, run an authenticating gateway
  (oauth2-proxy / Envoy ext_authz / Traefik ForwardAuth / Pomerium) in
  front of certctl and set CERTCTL_AUTH_TYPE=none on the upstream.
  See docs/architecture.md "Authenticating-gateway pattern" and
  docs/upgrade-to-v2-jwt-removal.md for the migration walkthrough'

config pkg coverage: ValidAuthTypes 100%, Validate 94.7%, total 75.5%.

Refs: coverage-gap-audit-2026-04-24-v5/unified-audit.md
      §2 P1 cluster, cat-g-jwt_silent_auth_downgrade
      Audit recommendation followed verbatim: 'Remove jwt from
      validAuthTypes until middleware ships'.
2026-04-25 00:22:23 +00:00

12 KiB

Certctl Helm Chart - Complete Summary

Overview

A production-ready Helm chart for deploying certctl (self-hosted certificate lifecycle management platform) on Kubernetes. The chart provides:

  • High availability support with multi-replica deployments
  • Persistent PostgreSQL database with automatic schema migration
  • DaemonSet or Deployment-based agent deployment
  • Comprehensive security contexts and RBAC
  • Multiple deployment scenarios (dev, prod, HA, external DB)
  • Full documentation and examples

Chart Metadata

  • Name: certctl
  • Chart Version: 0.1.0
  • App Version: 2.1.0
  • Type: application
  • License: BSL-1.1 (converts to Apache 2.0 in 2033)

File Structure

deploy/helm/
├── README.md                              # Main Helm chart documentation
├── DEPLOYMENT_GUIDE.md                    # Step-by-step deployment guide
├── CHART_SUMMARY.md                       # This file
│
├── certctl/
│   ├── Chart.yaml                         # Chart metadata
│   ├── values.yaml                        # Default configuration values
│   ├── .helmignore                        # Files to ignore when building chart
│   │
│   └── templates/
│       ├── _helpers.tpl                   # Helm template helper functions
│       ├── NOTES.txt                      # Post-deployment notes
│       │
│       ├── server-deployment.yaml         # Certctl API server deployment
│       ├── server-service.yaml            # Server Kubernetes service
│       ├── server-configmap.yaml          # Server configuration
│       ├── server-secret.yaml             # Server secrets (API key, DB password, etc)
│       │
│       ├── postgres-statefulset.yaml      # PostgreSQL database statefulset
│       ├── postgres-service.yaml          # PostgreSQL headless service
│       ├── postgres-secret.yaml           # Database credentials secret
│       │
│       ├── agent-daemonset.yaml           # Certctl agent daemonset/deployment
│       ├── agent-configmap.yaml           # Agent configuration
│       │
│       ├── ingress.yaml                   # Optional ingress resource
│       └── serviceaccount.yaml            # ServiceAccount and RBAC
│
└── examples/
    ├── values-dev.yaml                    # Development/testing configuration
    ├── values-prod-ha.yaml                # Production HA configuration
    ├── values-external-db.yaml            # External PostgreSQL (RDS, Cloud SQL)
    └── values-acme-dns01.yaml             # ACME with DNS-01 (Let's Encrypt)

Key Components

1. Server Deployment

File: templates/server-deployment.yaml

  • Manages certctl API server instances
  • Configurable replicas (default: 1)
  • Health checks (liveness & readiness probes)
  • Security context: non-root user, read-only filesystem
  • Resource limits (default: 500m CPU, 512Mi memory)
  • Automatic restart on failure

Values:

server:
  replicas: 1
  port: 8443
  auth:
    type: api-key
    apiKey: "REQUIRED"
  resources:
    requests: {cpu: 100m, memory: 128Mi}
    limits: {cpu: 500m, memory: 512Mi}

2. PostgreSQL StatefulSet

File: templates/postgres-statefulset.yaml

  • Persistent database storage
  • Automatic schema migrations on startup
  • Single replica (can be extended with external HA tools)
  • Health checks via pg_isready
  • Configurable storage size and class
  • Security context: non-root user (UID 999)

Values:

postgresql:
  enabled: true
  storage:
    size: 10Gi
    storageClass: ""  # Use default
  auth:
    database: certctl
    username: certctl
    password: "REQUIRED"

3. Agent DaemonSet/Deployment

File: templates/agent-daemonset.yaml

  • DaemonSet mode: one agent per Kubernetes node
  • Deployment mode: custom number of agent replicas
  • Local key storage with secure permissions (0600)
  • Health checks and automatic restart
  • Optional certificate discovery from filesystem

Values:

agent:
  enabled: true
  kind: DaemonSet  # or Deployment
  replicas: 1      # for Deployment only
  keyDir: /var/lib/certctl/keys
  discoveryDirs: "/etc/ssl/certs"  # optional

4. Ingress (Optional)

File: templates/ingress.yaml

  • Optional HTTPS ingress
  • cert-manager integration for automatic TLS
  • Multiple host support
  • Path-based routing

Values:

ingress:
  enabled: false
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  hosts:
    - host: certctl.example.com
      paths:
        - path: /
          pathType: Prefix

5. ConfigMaps and Secrets

Files:

  • server-configmap.yaml - Non-secret server configuration
  • server-secret.yaml - API key, database URL, SMTP password
  • postgres-secret.yaml - Database credentials
  • agent-configmap.yaml - Agent configuration

All secrets are base64-encoded and stored in Kubernetes Secrets.

6. ServiceAccount and RBAC

File: templates/serviceaccount.yaml

  • Optional ServiceAccount creation
  • Optional RBAC (ClusterRole, ClusterRoleBinding)
  • Namespace-scoped by default

Deployment Scenarios

Development Setup

Use examples/values-dev.yaml:

helm install certctl certctl/ \
  --values examples/values-dev.yaml \
  --set server.auth.apiKey="dev-key" \
  --set postgresql.auth.password="dev-password"

Features:

  • Single server replica
  • Demo auth (no API key required)
  • Small database (5Gi)
  • LoadBalancer service for easy access
  • Debug logging level

Production HA Setup

Use examples/values-prod-ha.yaml:

helm install certctl certctl/ \
  --values examples/values-prod-ha.yaml \
  --set server.auth.apiKey="$(openssl rand -base64 32)" \
  --set postgresql.auth.password="$(openssl rand -base64 32)"

Features:

  • 3 server replicas with pod anti-affinity
  • Large database storage (100Gi)
  • Pod disruption budgets
  • Prometheus monitoring enabled
  • Production resource limits

External PostgreSQL

Use examples/values-external-db.yaml:

helm install certctl certctl/ \
  --values examples/values-external-db.yaml \
  --set postgresql.enabled=false \
  --set 'server.env.CERTCTL_DATABASE_URL=postgres://...'

Use cases:

  • AWS RDS
  • Google Cloud SQL
  • Azure Database for PostgreSQL
  • External self-managed PostgreSQL

ACME with DNS-01

Use examples/values-acme-dns01.yaml:

helm install certctl certctl/ \
  --values examples/values-acme-dns01.yaml

Enables:

  • Automatic certificate issuance from Let's Encrypt
  • DNS-01 challenge (wildcard support)
  • Custom DNS provider scripts

Configuration Options

Server Configuration

Option Default Description
server.replicas 1 Number of server replicas
server.port 8443 Server port
server.auth.type api-key Authentication type — api-key or none (G-1: jwt removed; for JWT/OIDC use a fronting authenticating gateway, see docs/architecture.md and docs/upgrade-to-v2-jwt-removal.md)
server.auth.apiKey "" API key (REQUIRED when auth.type=api-key)
server.logging.level info Log level
server.logging.format json Log format

PostgreSQL Configuration

Option Default Description
postgresql.enabled true Enable internal PostgreSQL
postgresql.storage.size 10Gi Database storage size
postgresql.storage.storageClass "" Storage class name
postgresql.auth.password "" Database password (REQUIRED)

Agent Configuration

Option Default Description
agent.enabled true Deploy agents
agent.kind DaemonSet DaemonSet or Deployment
agent.replicas 1 Replicas (Deployment only)
agent.keyDir /var/lib/certctl/keys Key storage directory

Issuer Configuration

Option Default Description
server.issuer.local.enabled true Enable Local CA
server.issuer.acme.enabled false Enable ACME
server.issuer.acme.directoryURL "" ACME directory URL
server.issuer.acme.email "" ACME email
server.issuer.acme.challengeType http-01 Challenge type

See values.yaml for complete configuration options.

Helm Template Functions

Defined in templates/_helpers.tpl:

Function Purpose
certctl.name Chart name
certctl.fullname Full release name
certctl.chart Chart name and version
certctl.labels Common labels
certctl.selectorLabels Selector labels
certctl.serverSelectorLabels Server selector labels
certctl.agentSelectorLabels Agent selector labels
certctl.postgresSelectorLabels PostgreSQL selector labels
certctl.serviceAccountName ServiceAccount name
certctl.serverImage Server image URI
certctl.agentImage Agent image URI
certctl.postgresImage PostgreSQL image URI
certctl.databaseURL Database connection string
certctl.serverURL Server URL for agents

Security Features

Pod Security

  • Non-root users (UID 1000 for app, UID 999 for PostgreSQL)
  • Read-only root filesystems
  • No privilege escalation
  • Dropped capabilities (ALL)
  • Resource limits to prevent DoS

Secrets Management

  • All sensitive data in Kubernetes Secrets
  • Base64 encoded at rest
  • Can be integrated with:
    • sealed-secrets
    • external-secrets
    • Vault
    • AWS Secrets Manager

RBAC

  • ServiceAccount per release
  • Optional ClusterRole/ClusterRoleBinding
  • Extensible for custom permissions

Network Security

  • Support for Kubernetes NetworkPolicies
  • Service-to-service communication via internal DNS
  • Optional Ingress with TLS

Monitoring and Observability

Health Checks

  • Liveness probes (detect dead containers)
  • Readiness probes (detect not-ready services)
  • HTTP endpoints: /health, /readyz

Logging

  • Structured JSON logging
  • Request ID propagation
  • Configurable log levels (debug, info, warn, error)

Metrics

  • Prometheus metrics endpoint: /api/v1/metrics/prometheus
  • Optional ServiceMonitor for Prometheus Operator
  • Built-in metrics:
    • Certificate counts by status
    • Agent counts and status
    • Job completion/failure rates
    • Server uptime

Installation Quick Reference

# Development
helm install certctl certctl/ \
  --set server.auth.apiKey=dev \
  --set postgresql.auth.password=dev

# Production HA
helm install certctl certctl/ \
  --values examples/values-prod-ha.yaml \
  --set server.auth.apiKey="$(openssl rand -base64 32)" \
  --set postgresql.auth.password="$(openssl rand -base64 32)"

# External database
helm install certctl certctl/ \
  --values examples/values-external-db.yaml \
  --set postgresql.enabled=false \
  --set 'server.env.CERTCTL_DATABASE_URL=postgres://...'

# ACME with Let's Encrypt
helm install certctl certctl/ \
  --set server.issuer.acme.enabled=true \
  --set server.issuer.acme.directoryURL=https://acme-v02.api.letsencrypt.org/directory

# Check status
kubectl get pods -l app.kubernetes.io/instance=certctl
kubectl logs -l app.kubernetes.io/component=server -f

# Upgrade
helm upgrade certctl certctl/ -f new-values.yaml

# Uninstall
helm uninstall certctl

Best Practices

1. Use Secrets Management

# Use sealed-secrets
kubectl create secret generic certctl-secrets \
  --from-literal=api-key="$(openssl rand -base64 32)" \
  --dry-run=client -o yaml | kubeseal -f - | kubectl apply -f -

2. Configure Resource Limits

Match limits to your cluster capacity:

server:
  resources:
    requests: {cpu: 250m, memory: 256Mi}
    limits: {cpu: 1000m, memory: 512Mi}

3. Enable HA for Production

server:
  replicas: 3
podAntiAffinity:
  requiredDuringSchedulingIgnoredDuringExecution: [...]

4. Use Persistent Storage

postgresql:
  storage:
    size: 100Gi
    storageClass: fast-ssd

5. Enable Monitoring

monitoring:
  enabled: true
  serviceMonitor:
    enabled: true

Documentation

  • README.md - Complete Helm chart documentation
  • DEPLOYMENT_GUIDE.md - Step-by-step deployment instructions
  • values.yaml - Commented configuration reference

Support

For issues, questions, or contributions:

License

BSL-1.1 (Business Source License) Converts to Apache 2.0 on March 14, 2033