Files
certctl/deploy/helm/certctl/values.yaml
T
shankar0123 86fffa305a fix(deploy,helm,docs): published-image HEALTHCHECK speaks HTTPS + Helm /ready path + docs HTTPS sweep (U-2)
Pre-U-2 the published `ghcr.io/shankar0123/certctl-server` image
shipped with `HEALTHCHECK CMD curl -f http://localhost:8443/health`.
The server has been HTTPS-only since the v2.2 HTTPS-Everywhere milestone
(`cmd/server/main.go::ListenAndServeTLS`, no plaintext fallback, TLS
1.3 pinned), so the probe failed on every interval and Docker marked
the container `unhealthy` indefinitely. Operators inside docker-
compose / Helm / the example stacks were unaffected — compose overrides
the HEALTHCHECK with `--cacert + https://`, Helm uses explicit
`httpGet` probes that ignore Docker's HEALTHCHECK, and every example
compose file overrides with `curl -sfk https://localhost:8443/health`.
But anyone running bare `docker run` / Docker Swarm / Nomad / ECS —
exactly the "I just pulled the published image" path — saw permanent
`unhealthy` status and (depending on orchestrator policy) a restart-
loop. (Audit: cat-u-healthcheck_protocol_mismatch in
coverage-gap-audit-2026-04-24-v5/unified-audit.md.)

Recon for U-2 surfaced two adjacent bugs from the same v2.2 milestone
gap, both bundled into this commit because they share the same root
cause and the same operator surface:

  1. Helm chart `server.readinessProbe.httpGet.path` pointed at
     `/readyz`, the kube-flavored convention. The certctl server
     doesn't register `/readyz` (only `/health` and `/ready` are
     wired and bypass the auth middleware — see
     internal/api/router/router.go:81 and cmd/server/main.go:920).
     K8s readiness probes therefore got 401 (api-key auth rejection)
     or 404 (when auth was disabled), pods stayed `NotReady`
     indefinitely, and Helm rollouts stalled.

  2. The agent image (`Dockerfile.agent`) had no HEALTHCHECK at all,
     so bare-`docker run` agents got zero health signal. The
     compose override at `deploy/docker-compose.yml:173` called
     `pgrep -f certctl-agent` against the agent image, but the
     agent image didn't ship `procps` — pgrep was missing too. The
     compose probe was a latent always-fail.

We fixed all three with the audit-recommended shape (option (a) — `-k`)
plus three structural backstops:

Files changed:

Phase 1 — Dockerfile fix:
- Dockerfile: HEALTHCHECK switched from `curl -f http://localhost:8443/
  health` to `curl -fsk https://localhost:8443/health`. `-k`
  (insecure) is acceptable because the probe is localhost-to-localhost:
  the same process serving the cert is being probed, no network hop.
  Pinning `--cacert` is not viable for the published image because
  the bootstrap cert is per-deploy (generated into the `certs` named
  volume on first up; operator-supplied via Helm's `existingSecret`
  or cert-manager). Long-form docblock cross-references the audit
  closure, the compose vs Helm vs examples coverage matrix, and the
  CI guardrail.
- Dockerfile.agent: added HEALTHCHECK using `pgrep -f certctl-agent`
  matching the compose pattern. Added `procps` to the runtime apk
  install — fixes both the new image-level HEALTHCHECK AND the
  pre-existing compose probe that was silently failing.

Phase 2 — Helm readiness probe path:
- deploy/helm/certctl/values.yaml: server.readinessProbe.httpGet.path
  changed from `/readyz` to `/ready`. Liveness probe path
  (`/health`) was correct and is unchanged. Probes block now carries
  an explanatory comment naming the registered no-auth probe routes
  and the U-2 closure rationale.

Phase 3 — Image-level integration tests:
- deploy/test/healthcheck_test.go (new, //go:build integration):
  TestPublishedServerImage_HealthcheckSpecUsesHTTPS builds the server
  image, inspects `Config.Healthcheck.Test` via `docker inspect`,
  and asserts the array contains `https://localhost:8443/health` and
  `-k`, and does NOT contain `http://localhost:8443/health`
  (positive + negative regression contracts).
  TestPublishedAgentImage_HealthcheckSpecExists builds the agent image
  and asserts the HEALTHCHECK uses `pgrep` against `certctl-agent`.
  Both tests `t.Skip` cleanly when docker isn't available (sandbox /
  CI without docker-in-docker) — verified locally: tests skip with the
  diagnostic and the suite returns PASS.
  TestPublishedServerImage_HealthcheckTransitionsToHealthy is a
  documented `t.Skip` placeholder until the harness wires a sidecar
  postgres for image-level smoke; the spec-level tests above cover the
  audit-flagged regression.

Phase 4 — CI guardrail:
- .github/workflows/ci.yml: new "Forbidden plaintext HEALTHCHECK
  regression guard (U-2)" step. Scoped patterns catch
  `HEALTHCHECK.*http://` and `curl -f http://localhost:8443/health`
  in any `Dockerfile*`. Comment lines exempt; docs/upgrade-to-tls.md
  out of scope (the post-cutover invariant string at line 182 is
  intentionally a documented expected-failure assertion). Verified
  locally on the real tree (passes) and against synthetic regressions
  (each fires the guard).

Phase 5 — Docs sweep:
- docs/connectors.md: 15 stale curl examples updated from
  `http://localhost:8443/...` to `https://localhost:8443/...` with
  `--cacert "$CA"` injected on every site. Added a one-time
  introductory note documenting the `$CA` extraction with
  `docker compose ... exec ... cat /etc/certctl/tls/ca.crt`,
  matching the pattern in docs/quickstart.md. Pre-U-2 these examples
  silently failed against the HTTPS listener.

Phase 6 — Release surface:
- CHANGELOG.md: appended U-2 section to the existing [unreleased]
  block (immediately below the G-1 entry). Sections: explanatory
  blockquote covering all three bugs (primary + 2 adjacent), Fixed,
  Added, Changed.

Verification (all gates pass):
- go build ./... — clean
- go vet ./... — clean
- go vet -tags integration ./deploy/test/ — clean
- go test -short ./... — every package green
- go test -tags integration -v -run TestPublishedServerImage|TestPublishedAgentImage ./deploy/test/ —
  three tests SKIP cleanly with "docker not available" diagnostic
- helm lint deploy/helm/certctl/ — clean
- helm template smoke render — succeeds; rendered Deployment carries
  `path: /ready` and zero `/readyz` matches
- python3 yaml.safe_load on api/openapi.yaml — parses
- govulncheck ./... — no vulnerabilities in our code
- CI guardrail mirror: clean on real tree, fires on synthetic
  regression patterns

Out of scope (intentionally untouched):
- cmd/server/main.go::ListenAndServeTLS — HTTPS-only is correct,
  this finding does NOT propose adding back a plaintext listener.
- deploy/docker-compose.yml:126 HEALTHCHECK — already correct.
- deploy/docker-compose.test.yml HEALTHCHECK blocks — already correct.
- All 5 examples/*/docker-compose.yml HEALTHCHECK overrides — already
  correct (they ALSO use `-fsk https://localhost:8443/health`).
- Helm server.livenessProbe.httpGet — already uses `scheme: HTTPS` +
  `path: /health`, correct.
- docs/upgrade-to-tls.md:182 `curl ... http://localhost:8443/health`
  invariant line — that's the expected-failure assertion for the
  post-cutover state ("plaintext is gone, expect Connection refused");
  intentionally left intact.
- Go production code — this is purely a deploy-image / probe / docs /
  Helm-chart fix.

Refs: coverage-gap-audit-2026-04-24-v5/unified-audit.md
      §2 P1 cluster, cat-u-healthcheck_protocol_mismatch
      Audit recommendation followed verbatim: 'change Dockerfile:80
      to CMD curl -kf https://localhost:8443/health'.
2026-04-25 12:02:18 +00:00

542 lines
16 KiB
YAML

# Default values for certctl Helm chart
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
# Namespace override (optional)
namespace: ""
# Global configuration
commonLabels: {}
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
# ==============================================================================
# Certctl Server Configuration
# ==============================================================================
server:
# Number of replicas (for HA deployments)
replicas: 1
# Image configuration
image:
repository: ghcr.io/shankar0123/certctl
tag: "" # defaults to Chart.appVersion
pullPolicy: IfNotPresent
# Server port
port: 8443
# Resource requests and limits
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
# Pod security context
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
# Liveness and readiness probes (HTTPS-only as of v2.2).
#
# The two paths exposed for probes are `/health` and `/ready` —
# registered in internal/api/router/router.go:76-85 and bypassing the
# auth middleware via the no-auth list at cmd/server/main.go:920.
# Both serve the same JSON shape today (`{"status":"healthy"}` /
# `{"status":"ready"}`) but exist as separate routes so liveness and
# readiness can diverge in the future without renaming.
livenessProbe:
httpGet:
path: /health
port: https
scheme: HTTPS
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
# U-2 (P1, cat-u-healthcheck_protocol_mismatch — adjacent fix): pre-U-2
# the readiness probe pointed at `/readyz`, the conventional kube-flavor
# name. The certctl server doesn't register `/readyz` (only `/health`
# and `/ready`) — see cmd/server/main.go:920 and
# internal/api/router/router.go:81. K8s readiness probes therefore
# received a 404 (or, with auth enabled, a 401 from the api-key middleware
# because `/readyz` was NOT in the no-auth bypass set), pods stayed
# `NotReady` indefinitely, and Helm rollouts stalled. Post-U-2 the path
# matches a registered route.
readinessProbe:
httpGet:
path: /ready
port: https
scheme: HTTPS
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 2
# TLS configuration — REQUIRED. HTTPS is the only supported mode (v2.2+).
# Operator must configure EXACTLY ONE of:
# (a) server.tls.existingSecret: <name> # pre-existing kubernetes.io/tls Secret
# (b) server.tls.certManager.enabled: true # provision a cert-manager Certificate CR
# Refusing to set either makes `helm template` fail with a diagnostic pointing at docs/tls.md.
tls:
# Name of a pre-existing Secret (type kubernetes.io/tls) holding tls.crt + tls.key (+ optional ca.crt).
# Leave empty to fall through to the cert-manager path.
existingSecret: ""
# Mount path for the TLS Secret inside the server + agent containers.
mountPath: /etc/certctl/tls
# cert-manager auto-provisioning. Opt-in (off by default per milestone §3.4).
certManager:
enabled: false
# Secret name the cert-manager Certificate CR writes into. Agents and the server
# both read from this Secret. If empty, defaults to "<fullname>-tls".
secretName: ""
# Cert-manager issuer reference.
issuerRef:
name: "" # e.g. "letsencrypt-prod" or "internal-ca"
kind: ClusterIssuer # ClusterIssuer or Issuer
group: cert-manager.io
# Subject fields on the issued cert.
commonName: "certctl-server"
dnsNames:
- certctl-server
- localhost
# Certificate lifetime + renewal window.
duration: 2160h # 90 days
renewBefore: 360h # 15 days
# Service type (ClusterIP, LoadBalancer, NodePort)
service:
type: ClusterIP
port: 8443
annotations: {}
# Authentication configuration.
# Valid types: "api-key" (production) or "none" (demo only — disables
# authentication on the API and logs a loud Warn at server startup).
# For JWT/OIDC, run an authenticating gateway in front of certctl
# (oauth2-proxy / Envoy ext_authz / Traefik ForwardAuth / Pomerium)
# and set type=none here so the gateway terminates federated identity.
# See docs/architecture.md "Authenticating-gateway pattern".
#
# G-1 (P1): pre-G-1 the chart accepted server.auth.type=jwt and the
# certctl-server container silently routed every request through the
# api-key bearer middleware — silent auth downgrade. Post-G-1 the
# chart's `certctl.validateAuthType` template helper rejects any value
# outside {api-key, none} at template time. See
# docs/upgrade-to-v2-jwt-removal.md if you previously set type=jwt.
auth:
type: api-key
apiKey: "" # REQUIRED when type=api-key (set via --set or values override).
# Logging configuration
logging:
level: info # debug, info, warn, error
format: json # json or text
# SMTP configuration for email notifications (optional)
smtp:
enabled: false
host: ""
port: 587
username: ""
password: ""
fromAddress: ""
useTLS: true
# Certificate digest digest (periodic email summary)
digest:
enabled: false
interval: "24h"
recipients: []
# Example:
# - admin@example.com
# - ops@example.com
# Enrollment over Secure Transport (EST) configuration
est:
enabled: false
issuerID: "iss-local"
profileID: ""
# Rate limiting configuration
rateLimiting:
rps: 100 # Requests per second
burst: 200 # Burst capacity
# Network scanning configuration
networkScan:
enabled: false
interval: "6h"
# Certificate key generation mode
keygen:
mode: agent # Options: agent (production), server (demo with warning)
# CORS configuration
cors:
origins: "" # Comma-separated list, empty means deny all cross-origin requests
# Issuer connectors configuration
issuer:
local:
enabled: true
# For sub-CA mode, provide these paths:
# caCertPath: /path/to/ca.crt
# caKeyPath: /path/to/ca.key
acme:
enabled: false
directoryURL: ""
email: ""
challengeType: "http-01" # Options: http-01, dns-01, dns-persist-01
# DNS configuration (for dns-01 or dns-persist-01)
# dnsPresentScript: /path/to/dns-present.sh
# dnsCleanupScript: /path/to/dns-cleanup.sh
# dnsPropagationWait: "30s"
# dnsPersistIssuerDomain: "validation.example.com"
# EAB configuration (for ZeroSSL, Google Trust Services, etc.)
# eabKid: ""
# eabHmac: ""
stepca:
enabled: false
# rootCAPath: /path/to/root_ca.crt
# intermediateCAPath: /path/to/intermediate_ca.crt
# provisionerName: ""
# provisionerPassword: ""
openssl:
enabled: false
# signScript: /path/to/sign.sh
# revokeScript: /path/to/revoke.sh
# crlScript: /path/to/crl.sh
# timeoutSeconds: 30
# Notifier connectors configuration
notifiers:
slack:
enabled: false
# webhookUrl: ""
# channel: ""
# username: ""
# iconEmoji: ""
teams:
enabled: false
# webhookUrl: ""
pagerduty:
enabled: false
# routingKey: ""
# severity: warning
opsgenie:
enabled: false
# apiKey: ""
# priority: P3
# Additional environment variables
# Will be passed as-is to the server container
env: {}
# Example:
# CERTCTL_SCHEDULER_RENEWAL_CHECK_INTERVAL: "1h"
# CERTCTL_DATABASE_MAX_CONNS: "25"
# Additional volume mounts for custom configurations
# volumeMounts: []
# - name: ca-cert
# mountPath: /etc/ssl/certs/ca.crt
# subPath: ca.crt
# Additional volumes
# volumes: []
# - name: ca-cert
# secret:
# secretName: ca-cert
# ==============================================================================
# PostgreSQL Configuration
# ==============================================================================
postgresql:
# Enable/disable PostgreSQL (set to false if using external database)
enabled: true
# Image configuration
image:
repository: postgres
tag: "16-alpine"
pullPolicy: IfNotPresent
# Authentication
auth:
database: certctl
username: certctl
# REQUIRED — set via `--set postgresql.auth.password=<value>` or values override.
#
# WARNING (U-1): rotating this value after first deploy does NOT change the
# database password. The `postgres:16-alpine` image runs `initdb` only when
# /var/lib/postgresql/data is empty, so POSTGRES_PASSWORD is written into
# pg_authid exactly once — on the first boot of the StatefulSet's PVC.
# Subsequent rollouts pick up the new env value in the postgres container
# but the certctl-server container's CERTCTL_DATABASE_URL also picks up
# the new value, while pg_authid still expects the old one — leading to
# `pq: password authentication failed for user "certctl"` (SQLSTATE 28P01).
#
# The certctl-server emits guidance via internal/repository/postgres/db.go::
# wrapPingError when it sees SQLSTATE 28P01 at startup. To resolve in a
# Helm deployment:
# - Non-destructive (preferred for environments with data):
# kubectl exec -it <release>-postgres-0 -- \
# psql -U certctl -c "ALTER ROLE certctl PASSWORD '<new>';"
# then update the secret/values to match and let the certctl-server
# pod restart against the matching credential.
# - Destructive (DESTROYS DATA — only acceptable on dev/demo PVCs):
# helm uninstall <release> && \
# kubectl delete pvc -l app.kubernetes.io/name=certctl,app.kubernetes.io/component=postgres && \
# helm install <release> ... # PVC re-creates empty, initdb seeds new password
password: ""
# Storage configuration
storage:
size: 10Gi
storageClass: "" # Uses default StorageClass if empty
# deleteOnTermination: false # Keep data on Helm uninstall
# Resource requests and limits
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
# Pod security context
securityContext:
runAsNonRoot: true
runAsUser: 999
runAsGroup: 999
fsGroup: 999
# Liveness and readiness probes
livenessProbe:
exec:
command:
- /bin/sh
- -c
- pg_isready -U certctl -d certctl
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
exec:
command:
- /bin/sh
- -c
- pg_isready -U certctl -d certctl
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 2
# Service configuration
service:
type: ClusterIP
port: 5432
# PostgreSQL-specific settings
postgresqlConfig: {}
# Example:
# max_connections: "200"
# shared_buffers: "256MB"
# ==============================================================================
# Certctl Agent Configuration
# ==============================================================================
agent:
# Enable/disable agent deployment
enabled: true
# Deployment strategy: DaemonSet (recommended) or Deployment
kind: DaemonSet # Options: DaemonSet, Deployment
# Image configuration
image:
repository: ghcr.io/shankar0123/certctl-agent
tag: "" # defaults to Chart.appVersion
pullPolicy: IfNotPresent
# Number of replicas (for Deployment kind; ignored for DaemonSet)
replicas: 1
# Resource requests and limits
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 200m
memory: 256Mi
# Pod security context
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
# Agent name (can be overridden per pod via StatefulSet ordinals)
name: "" # If empty, uses release name
# Key storage directory
keyDir: /var/lib/certctl/keys
# Certificate discovery directories (comma-separated)
discoveryDirs: ""
# Example: "/etc/ssl/certs,/etc/pki/tls"
# Node selector for agent pods (for DaemonSet)
nodeSelector: {}
# Example:
# node-role.kubernetes.io/worker: "true"
# Tolerations for agent pods
tolerations: []
# Example:
# - key: node-role
# operator: Equal
# value: worker
# effect: NoSchedule
# Affinity rules
affinity: {}
# Additional environment variables
env: {}
# ==============================================================================
# Ingress Configuration
# ==============================================================================
ingress:
enabled: false
className: ""
annotations: {}
# kubernetes.io/ingress.class: nginx
# Optional cert-manager integration for the public-facing Ingress cert.
# This is completely independent of server.tls.* — the Ingress terminates
# an *additional* TLS hop between the internet and the in-cluster Service.
# Leave disabled unless an Ingress is exposing certctl to the outside world.
certManager:
enabled: false
issuerRef:
name: "" # e.g. "letsencrypt-prod"
kind: ClusterIssuer # ClusterIssuer or Issuer
hosts:
- host: certctl.local
paths:
- path: /
pathType: Prefix
tls: []
# - secretName: certctl-tls
# hosts:
# - certctl.local
# ==============================================================================
# Service Account Configuration
# ==============================================================================
serviceAccount:
create: true
annotations: {}
name: "" # defaults to release name if empty
# ==============================================================================
# RBAC Configuration
# ==============================================================================
rbac:
create: true
# ==============================================================================
# Kubernetes Secrets Target Connector
# ==============================================================================
kubernetesSecrets:
# Enable RBAC rules for managing TLS Secrets
enabled: false
# ==============================================================================
# Pod Disruption Budget (for HA deployments)
# ==============================================================================
podDisruptionBudget:
enabled: false
minAvailable: 1
# maxUnavailable: 1
# ==============================================================================
# Monitoring Configuration
# ==============================================================================
monitoring:
enabled: false
# Prometheus ServiceMonitor
serviceMonitor:
enabled: false
interval: 30s
scrapeTimeout: 10s
# labels: {}
# selector: {}
# ==============================================================================
# Advanced Configuration
# ==============================================================================
# Node affinity for server pods
nodeAffinity: {}
# Pod affinity for server pods
podAffinity: {}
# Pod anti-affinity for server pods (for HA)
podAntiAffinity: {}
# Example:
# podAntiAffinity:
# preferredDuringSchedulingIgnoredDuringExecution:
# - weight: 100
# podAffinityTerm:
# labelSelector:
# matchExpressions:
# - key: app.kubernetes.io/name
# operator: In
# values:
# - certctl
# topologyKey: kubernetes.io/hostname
# Custom labels for all resources
customLabels: {}
# Custom annotations for all resources
customAnnotations: {}