v2.0.47: HTTPS Everywhere — TLS-only control plane, agents/CLI/MCP

Breaking change release. Plaintext HTTP listener removed. The certctl control plane now terminates TLS 1.3 on :8443 via http.Server.ListenAndServeTLS. No CERTCTL_TLS_ENABLED=false escape hatch. No dual-listener mode. One-step cutover per docs/upgrade-to-tls.md. Server - cmd/server/tls.go: certHolder with SIGHUP hot-reload + atomic cert swap, buildServerTLSConfig (TLS 1.3 min, GetCertificate callback), preflightServerTLS validation - cmd/server/main.go: ListenAndServeTLS in place of ListenAndServe, watchSIGHUP wiring, cert/key path config threading - tls_test.go: 418-line regression coverage of reload, preflight, callback behavior, SAN validation Config - CERTCTL_TLS_CERT_PATH / CERTCTL_TLS_KEY_PATH (required) - Plaintext rejection: agents/CLI/MCP pre-flight-fail on http:// URLs with a pointer to docs/upgrade-to-tls.md Agents, CLI, MCP - All three pre-flight-reject http:// URLs with fail-loud diagnostic - CERTCTL_SERVER_CA_BUNDLE_PATH for private-CA trust - CERTCTL_SERVER_TLS_INSECURE_SKIP_VERIFY for dev-only bypass (loud warning on startup) - install-agent.sh emits both vars as commented template lines docker-compose - certctl-tls-init sidecar generates SAN-valid self-signed cert into deploy/test/certs/ on first boot - All demo-stack curls pin against ca.crt with --cacert Helm chart - Three TLS provisioning modes, exactly one required: - server.tls.existingSecret (operator-supplied) - server.tls.certManager.enabled (cert-manager integration) - server.tls.selfSigned.enabled (eval only — not for production) - server-certificate.yaml template for cert-manager mode - helm install without a TLS source fails at template render with a pointer to docs/tls.md CI - .github/workflows/ci.yml Helm Chart Validation step renders the chart in both existingSecret and cert-manager modes, plus an inverse guard-regression test that asserts helm template MUST refuse to render when no TLS source is configured. Previously the single `helm template` invocation hit the certctl.tls.required fail-loud guard and exit-1'd CI. Four invocations now: lint (existingSecret), template (existingSecret), template (cert-manager), template (no args — must fail). Integration tests - deploy/test/integration_test.go stands up the Compose stack over HTTPS, extracts the CA bundle, and exercises every certctl API over https://localhost:8443 - All 34 integration subtests green (per Phase 8 local CI-parity) Documentation - New: docs/tls.md (provisioning patterns, rotation, SIGHUP reload) - New: docs/upgrade-to-tls.md (one-step cutover, no-downgrade warnings, fleet-roll sequencing) - CHANGELOG.md: v2.2.0 "HTTPS Everywhere — The Irony" entry (file heading unchanged; release tag is v2.0.47) - All curls in docs/, examples/, deploy/helm/ guides use https://localhost:8443 --cacert Verification - grep -rn "ListenAndServe[^T]" cmd/ internal/ → 0 hits - grep -rn "\"http://" cmd/ internal/ → 2 benign hits (Caddy admin API default, SSRF doc comment) — zero certctl endpoints - Tasks #197–#206 (Phases 0–8) all closed in the tracker Files: 65 changed, 3489 insertions, 372 deletions (pre-CI-fix).
2026-06-10 04:48:52 +00:00 · 2026-04-20 03:31:05 +00:00
parent 04c7eca615
commit 52248be717
66 changed files with 3518 additions and 375 deletions
@@ -236,10 +236,12 @@ kubectl get svc -l app.kubernetes.io/instance=certctl
 kubectl get ingress
 kubectl describe ingress certctl

-# Test API connectivity
+# Test API connectivity (HTTPS-only as of v2.2)
 POD=$(kubectl get pods -l app.kubernetes.io/component=server -o jsonpath='{.items[0].metadata.name}')
 kubectl port-forward $POD 8443:8443 &
-curl -H "Authorization: Bearer $API_KEY" http://localhost:8443/health
+# If the chart provisioned a self-signed cert, fetch the CA bundle from the TLS secret first:
+#   kubectl get secret certctl-server-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/certctl-ca.crt
+curl --cacert /tmp/certctl-ca.crt -H "Authorization: Bearer $API_KEY" https://localhost:8443/health
 ```

 ### Step 6: Access the Dashboard
@@ -333,9 +335,10 @@ kubectl logs $POD | tail -20
 # Port forward to API
 kubectl port-forward svc/certctl-server 8443:8443 &

-# Create a test certificate
+# Create a test certificate (HTTPS-only as of v2.2 — pin the chart-provisioned CA bundle)
+# kubectl get secret certctl-server-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/certctl-ca.crt
 API_KEY="your-api-key"
-curl -X POST http://localhost:8443/api/v1/certificates \
+curl --cacert /tmp/certctl-ca.crt -X POST https://localhost:8443/api/v1/certificates \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
@@ -33,9 +33,11 @@ kubectl get pods -l app.kubernetes.io/instance=certctl
 # View server logs
 kubectl logs -l app.kubernetes.io/component=server -f

-# Access the API
+# Access the API (HTTPS-only as of v2.2; use --cacert or -k depending on your cert provisioning)
 kubectl port-forward svc/certctl-server 8443:8443 &
-curl http://localhost:8443/health
+# If the chart provisioned a self-signed cert, fetch the CA bundle from the secret first:
+#   kubectl get secret certctl-server-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/certctl-ca.crt
+curl --cacert /tmp/certctl-ca.crt https://localhost:8443/health
 ```

 ## Next Steps
@@ -4,36 +4,46 @@
 {{- else if contains "NodePort" .Values.server.service.type }}
  export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
  export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "certctl.fullname" . }}-server)
-  echo http://$NODE_IP:$NODE_PORT
+  echo https://$NODE_IP:$NODE_PORT
 {{- else if contains "LoadBalancer" .Values.server.service.type }}
  export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "certctl.fullname" . }}-server --template "{.status.loadBalancer.ingress[0].ip}")
-  echo http://$SERVICE_IP:{{ .Values.server.service.port }}
+  echo https://$SERVICE_IP:{{ .Values.server.service.port }}
 {{- else }}
  export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "certctl.name" . }},app.kubernetes.io/instance={{ .Release.Name }},app.kubernetes.io/component=server" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace {{ .Release.Namespace }} $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
-  echo "Visit http://127.0.0.1:8080 to use your application"
-  kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8080:$CONTAINER_PORT
+  echo "Visit https://127.0.0.1:8443 to use your application"
+  kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8443:$CONTAINER_PORT
 {{- end }}

-2. Get the default API key:
+2. Talk to the HTTPS-only server from your workstation:
+  # Export the CA bundle that signed the server cert (self-signed or cert-manager-issued)
+  kubectl get secret --namespace {{ .Release.Namespace }} {{ include "certctl.tls.secretName" . }} \
+    -o jsonpath='{.data.ca\.crt}' | base64 --decode > /tmp/certctl-ca.crt
+  # (If ca.crt is empty, fall back to tls.crt — typical when the Secret
+  #  was created from a self-signed bootstrap cert without a separate CA.)
+
+  # Adapt the URL below to match the Server URL printed in step 1.
+  curl --cacert /tmp/certctl-ca.crt https://127.0.0.1:8443/health
+
+3. Get the default API key:
  kubectl get secret --namespace {{ .Release.Namespace }} {{ include "certctl.fullname" . }}-server -o jsonpath="{.data.api-key}" | base64 --decode; echo

-3. Get PostgreSQL connection details:
+4. Get PostgreSQL connection details:
  Host: {{ include "certctl.fullname" . }}-postgres.{{ .Release.Namespace }}.svc.cluster.local
  Port: 5432
  Database: {{ .Values.postgresql.auth.database }}
  Username: {{ .Values.postgresql.auth.username }}
  Password: $(kubectl get secret --namespace {{ .Release.Namespace }} {{ include "certctl.fullname" . }}-postgres -o jsonpath="{.data.password}" | base64 --decode)

-4. Check deployment status:
+5. Check deployment status:
  kubectl get pods -n {{ .Release.Namespace }} -l app.kubernetes.io/instance={{ .Release.Name }}

-5. View server logs:
+6. View server logs:
  kubectl logs -n {{ .Release.Namespace }} -l app.kubernetes.io/name={{ include "certctl.name" . }},app.kubernetes.io/component=server -f

 {{- if .Values.agent.enabled }}

-6. View agent logs:
+7. View agent logs:
  kubectl logs -n {{ .Release.Namespace }} -l app.kubernetes.io/name={{ include "certctl.name" . }},app.kubernetes.io/component=agent -f

 {{- end }}
@@ -58,11 +68,7 @@ IMPORTANT NOTES FOR PRODUCTION:
   - Use an external PostgreSQL managed service (AWS RDS, Cloud SQL, etc.)
   - Set postgresql.enabled=false and configure CERTCTL_DATABASE_URL in values

-5. Enable HTTPS/TLS using an Ingress with certificate management:
-   - Configure cert-manager for automatic TLS certificate renewal
-   - Update ingress values with your domain and certificate issuer
-
-6. Review security contexts and network policies:
+5. Review security contexts and network policies:
   - All containers run as non-root
   - Implement network policies to restrict traffic between components
   - Consider pod security policies or security standards for your cluster
@@ -118,8 +118,54 @@ postgres://{{ .Values.postgresql.auth.username }}:$(POSTGRES_PASSWORD)@{{ includ
 {{- end }}

 {{/*
-Server URL (for agents)
+Server URL (for agents). HTTPS-only as of v2.2 — see docs/tls.md.
 */}}
 {{- define "certctl.serverURL" -}}
-http://{{ include "certctl.fullname" . }}-server:{{ .Values.server.service.port }}
+https://{{ include "certctl.fullname" . }}-server:{{ .Values.server.service.port }}
+{{- end }}
+
+{{/*
+TLS Secret name resolver.
+
+Operator-facing precedence:
+  1. server.tls.existingSecret        — operator points at a pre-existing kubernetes.io/tls Secret
+  2. server.tls.certManager.secretName — explicit secret name for the cert-manager Certificate CR
+  3. "<fullname>-tls"                  — default when cert-manager is enabled but secretName is blank
+
+Never emits an empty string — that case is already excluded by certctl.tls.required below,
+which must be invoked by any template that depends on the resolved secret name.
+*/}}
+{{- define "certctl.tls.secretName" -}}
+{{- if .Values.server.tls.existingSecret -}}
+{{- .Values.server.tls.existingSecret -}}
+{{- else if .Values.server.tls.certManager.secretName -}}
+{{- .Values.server.tls.certManager.secretName -}}
+{{- else -}}
+{{- printf "%s-tls" (include "certctl.fullname" .) -}}
+{{- end -}}
+{{- end }}
+
+{{/*
+TLS configuration gate.
+
+HTTPS is the only supported listener mode (v2.2+). The server refuses to start
+without a cert/key pair mounted at server.tls.mountPath, so `helm template` /
+`helm install` must fail loudly at render-time rather than shipping a broken
+Deployment that crash-loops with "tls config required".
+
+Operators MUST configure EXACTLY ONE of:
+  (a) server.tls.existingSecret: <name-of-kubernetes.io/tls-secret>
+  (b) server.tls.certManager.enabled: true  (+ issuerRef.name populated)
+
+Any template that mounts the TLS Secret must call
+`{{ include "certctl.tls.required" . }}` at the top so this guard runs once
+per affected resource. No-op when configured correctly.
+*/}}
+{{- define "certctl.tls.required" -}}
+{{- if and (not .Values.server.tls.existingSecret) (not .Values.server.tls.certManager.enabled) -}}
+{{- fail "\n\ncertctl refuses to start without TLS.\n\nSet EXACTLY ONE of:\n  --set server.tls.existingSecret=<your-kubernetes.io/tls-secret-name>\nOR\n  --set server.tls.certManager.enabled=true \\\n  --set server.tls.certManager.issuerRef.name=<your-issuer-or-clusterissuer>\n\nSee docs/tls.md for the full setup walkthrough, including bootstrap\nguidance for air-gapped clusters without cert-manager.\n" -}}
+{{- end -}}
+{{- if and .Values.server.tls.certManager.enabled (not .Values.server.tls.certManager.issuerRef.name) -}}
+{{- fail "\n\nserver.tls.certManager.enabled=true but server.tls.certManager.issuerRef.name is empty.\n\nSet:\n  --set server.tls.certManager.issuerRef.name=<your-issuer-or-clusterissuer>\n\nSee docs/tls.md.\n" -}}
+{{- end -}}
 {{- end }}
@@ -1,4 +1,5 @@
 {{- if .Values.agent.enabled }}
+{{- include "certctl.tls.required" . }}
 {{- if eq .Values.agent.kind "DaemonSet" }}
 apiVersion: apps/v1
 kind: DaemonSet
@@ -53,6 +54,8 @@ spec:
                  fieldPath: metadata.name
            - name: CERTCTL_KEY_DIR
              value: {{ .Values.agent.keyDir }}
+            - name: CERTCTL_SERVER_CA_BUNDLE_PATH
+              value: "{{ .Values.server.tls.mountPath }}/ca.crt"
            {{- if .Values.agent.discoveryDirs }}
            - name: CERTCTL_DISCOVERY_DIRS
              valueFrom:
@@ -70,12 +73,19 @@ spec:
              mountPath: {{ .Values.agent.keyDir }}
            - name: tmp
              mountPath: /tmp
+            - name: server-tls
+              mountPath: {{ .Values.server.tls.mountPath }}
+              readOnly: true
      volumes:
        - name: agent-keys
          emptyDir:
            sizeLimit: 1Gi
        - name: tmp
          emptyDir: {}
+        - name: server-tls
+          secret:
+            secretName: {{ include "certctl.tls.secretName" . }}
+            defaultMode: 0400
 {{- else if eq .Values.agent.kind "Deployment" }}
 apiVersion: apps/v1
 kind: Deployment
@@ -135,6 +145,8 @@ spec:
              {{- end }}
            - name: CERTCTL_KEY_DIR
              value: {{ .Values.agent.keyDir }}
+            - name: CERTCTL_SERVER_CA_BUNDLE_PATH
+              value: "{{ .Values.server.tls.mountPath }}/ca.crt"
            {{- if .Values.agent.discoveryDirs }}
            - name: CERTCTL_DISCOVERY_DIRS
              valueFrom:
@@ -152,11 +164,18 @@ spec:
              mountPath: {{ .Values.agent.keyDir }}
            - name: tmp
              mountPath: /tmp
+            - name: server-tls
+              mountPath: {{ .Values.server.tls.mountPath }}
+              readOnly: true
      volumes:
        - name: agent-keys
          emptyDir:
            sizeLimit: 1Gi
        - name: tmp
          emptyDir: {}
+        - name: server-tls
+          secret:
+            secretName: {{ include "certctl.tls.secretName" . }}
+            defaultMode: 0400
 {{- end }}
 {{- end }}
@@ -1,14 +1,24 @@
 {{- if .Values.ingress.enabled }}
+{{- if and .Values.ingress.certManager.enabled (not .Values.ingress.certManager.issuerRef.name) -}}
+{{- fail "\n\ningress.certManager.enabled=true but ingress.certManager.issuerRef.name is empty.\n\nSet:\n  --set ingress.certManager.issuerRef.name=<your-issuer-or-clusterissuer>\n\nThis is separate from server.tls.certManager — it issues the external-facing\nIngress cert, not the in-cluster server TLS cert. See docs/tls.md.\n" -}}
+{{- end -}}
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
  name: {{ include "certctl.fullname" . }}
  labels:
    {{- include "certctl.labels" . | nindent 4 }}
-  {{- with .Values.ingress.annotations }}
  annotations:
+    {{- if .Values.ingress.certManager.enabled }}
+    {{- if eq .Values.ingress.certManager.issuerRef.kind "ClusterIssuer" }}
+    cert-manager.io/cluster-issuer: {{ .Values.ingress.certManager.issuerRef.name | quote }}
+    {{- else }}
+    cert-manager.io/issuer: {{ .Values.ingress.certManager.issuerRef.name | quote }}
+    {{- end }}
+    {{- end }}
+    {{- with .Values.ingress.annotations }}
    {{- toYaml . | nindent 4 }}
-  {{- end }}
+    {{- end }}
 spec:
  {{- if .Values.ingress.className }}
  ingressClassName: {{ .Values.ingress.className }}
@@ -33,7 +43,7 @@ spec:
            pathType: {{ .pathType }}
            backend:
              service:
-                name: {{ include "certctl.fullname" . }}-server
+                name: {{ include "certctl.fullname" $ }}-server
                port:
                  number: {{ $.Values.server.service.port }}
          {{- end }}
@@ -0,0 +1,31 @@
+{{- if .Values.server.tls.certManager.enabled }}
+{{- include "certctl.tls.required" . }}
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: {{ include "certctl.fullname" . }}-server-tls
+  labels:
+    {{- include "certctl.labels" . | nindent 4 }}
+    app.kubernetes.io/component: server
+spec:
+  secretName: {{ include "certctl.tls.secretName" . }}
+  commonName: {{ .Values.server.tls.certManager.commonName | quote }}
+  dnsNames:
+    {{- range .Values.server.tls.certManager.dnsNames }}
+    - {{ . | quote }}
+    {{- end }}
+  duration: {{ .Values.server.tls.certManager.duration }}
+  renewBefore: {{ .Values.server.tls.certManager.renewBefore }}
+  usages:
+    - server auth
+    - digital signature
+    - key encipherment
+  privateKey:
+    algorithm: ECDSA
+    size: 256
+    rotationPolicy: Always
+  issuerRef:
+    name: {{ .Values.server.tls.certManager.issuerRef.name | quote }}
+    kind: {{ .Values.server.tls.certManager.issuerRef.kind }}
+    group: {{ .Values.server.tls.certManager.issuerRef.group }}
+{{- end }}
@@ -1,3 +1,4 @@
+{{- include "certctl.tls.required" . }}
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -32,7 +33,7 @@ spec:
          image: {{ include "certctl.serverImage" . }}
          imagePullPolicy: {{ .Values.server.image.pullPolicy }}
          ports:
-            - name: http
+            - name: https
              containerPort: {{ .Values.server.port }}
              protocol: TCP
          env:
@@ -40,6 +41,10 @@ spec:
              value: "0.0.0.0"
            - name: CERTCTL_SERVER_PORT
              value: "{{ .Values.server.port }}"
+            - name: CERTCTL_SERVER_TLS_CERT_PATH
+              value: "{{ .Values.server.tls.mountPath }}/tls.crt"
+            - name: CERTCTL_SERVER_TLS_KEY_PATH
+              value: "{{ .Values.server.tls.mountPath }}/tls.key"
            - name: CERTCTL_DATABASE_URL
              valueFrom:
                secretKeyRef:
@@ -172,12 +177,19 @@ spec:
          volumeMounts:
            - name: tmp
              mountPath: /tmp
+            - name: tls
+              mountPath: {{ .Values.server.tls.mountPath }}
+              readOnly: true
            {{- if .Values.server.volumeMounts }}
            {{- toYaml .Values.server.volumeMounts | nindent 12 }}
            {{- end }}
      volumes:
        - name: tmp
          emptyDir: {}
+        - name: tls
+          secret:
+            secretName: {{ include "certctl.tls.secretName" . }}
+            defaultMode: 0400
        {{- if .Values.server.volumes }}
        {{- toYaml .Values.server.volumes | nindent 8 }}
        {{- end }}
@@ -13,8 +13,8 @@ spec:
  type: {{ .Values.server.service.type }}
  ports:
    - port: {{ .Values.server.service.port }}
-      targetPort: http
+      targetPort: https
      protocol: TCP
-      name: http
+      name: https
  selector:
    {{- include "certctl.serverSelectorLabels" . | nindent 4 }}
@@ -48,11 +48,12 @@ server:
      drop:
        - ALL

-  # Liveness and readiness probes
+  # Liveness and readiness probes (HTTPS-only as of v2.2)
  livenessProbe:
    httpGet:
      path: /health
-      port: http
+      port: https
+      scheme: HTTPS
    initialDelaySeconds: 10
    periodSeconds: 10
    timeoutSeconds: 5
@@ -61,12 +62,50 @@ server:
  readinessProbe:
    httpGet:
      path: /readyz
-      port: http
+      port: https
+      scheme: HTTPS
    initialDelaySeconds: 5
    periodSeconds: 5
    timeoutSeconds: 3
    failureThreshold: 2

+  # TLS configuration — REQUIRED. HTTPS is the only supported mode (v2.2+).
+  # Operator must configure EXACTLY ONE of:
+  #   (a) server.tls.existingSecret: <name>        # pre-existing kubernetes.io/tls Secret
+  #   (b) server.tls.certManager.enabled: true     # provision a cert-manager Certificate CR
+  # Refusing to set either makes `helm template` fail with a diagnostic pointing at docs/tls.md.
+  tls:
+    # Name of a pre-existing Secret (type kubernetes.io/tls) holding tls.crt + tls.key (+ optional ca.crt).
+    # Leave empty to fall through to the cert-manager path.
+    existingSecret: ""
+
+    # Mount path for the TLS Secret inside the server + agent containers.
+    mountPath: /etc/certctl/tls
+
+    # cert-manager auto-provisioning. Opt-in (off by default per milestone §3.4).
+    certManager:
+      enabled: false
+
+      # Secret name the cert-manager Certificate CR writes into. Agents and the server
+      # both read from this Secret. If empty, defaults to "<fullname>-tls".
+      secretName: ""
+
+      # Cert-manager issuer reference.
+      issuerRef:
+        name: ""                      # e.g. "letsencrypt-prod" or "internal-ca"
+        kind: ClusterIssuer           # ClusterIssuer or Issuer
+        group: cert-manager.io
+
+      # Subject fields on the issued cert.
+      commonName: "certctl-server"
+      dnsNames:
+        - certctl-server
+        - localhost
+
+      # Certificate lifetime + renewal window.
+      duration: 2160h                 # 90 days
+      renewBefore: 360h               # 15 days
+
  # Service type (ClusterIP, LoadBalancer, NodePort)
  service:
    type: ClusterIP
@@ -356,7 +395,16 @@ ingress:
  className: ""
  annotations: {}
    # kubernetes.io/ingress.class: nginx
-    # cert-manager.io/cluster-issuer: letsencrypt-prod
+
+  # Optional cert-manager integration for the public-facing Ingress cert.
+  # This is completely independent of server.tls.* — the Ingress terminates
+  # an *additional* TLS hop between the internet and the in-cluster Service.
+  # Leave disabled unless an Ingress is exposing certctl to the outside world.
+  certManager:
+    enabled: false
+    issuerRef:
+      name: ""                      # e.g. "letsencrypt-prod"
+      kind: ClusterIssuer           # ClusterIssuer or Issuer
  hosts:
    - host: certctl.local
      paths: