certctl/docs/archive/upgrades/to-v2-jwt-removal.md

Upgrading past G-1 — CERTCTL_AUTH_TYPE=jwt removal

Last reviewed: 2026-05-05

Archived 2026-05-05. This upgrade guide applies to operators upgrading past the G-1 milestone (the CERTCTL_AUTH_TYPE=jwt removal). Current operators on post-G-1 releases don't need this. For the steady-state security posture reference, see docs/operator/security.md. Preserved here for late upgraders.

If your certctl deployment currently sets CERTCTL_AUTH_TYPE=jwt (or server.auth.type=jwt in Helm), the next certctl upgrade will fail-fast at startup with a dedicated diagnostic. This guide explains why, what to switch to, and how to keep JWT/OIDC at your edge.

For everyone else — operators running api-key or none — this upgrade is a no-op. Skip to to-tls-v2.2.md for the v2.2 HTTPS-everywhere migration if you haven't done that one yet.

Why we removed it

Pre-G-1, the config validator at internal/config/config.go accepted three values for CERTCTL_AUTH_TYPE: api-key, jwt, and none. The startup log line at cmd/server/main.go faithfully echoed "authentication enabled" "type"="jwt" when an operator picked jwt. Reasonable people read that and concluded JWT auth was on.

It wasn't. Grep internal/ cmd/ for NewJWT, JWTMiddleware, or jwt.Parse — pre-G-1, there were zero matches in production code. The auth-middleware wiring at cmd/server/main.go:653 unconditionally called middleware.NewAuthWithNamedKeys(namedKeys) regardless of cfg.Auth.Type. So CERTCTL_AUTH_TYPE=jwt just routed every request through the api-key bearer middleware, comparing the incoming Authorization: Bearer <something> against whatever string the operator put in CERTCTL_AUTH_SECRET. Real JWT clients got 401 (the api-key middleware saw the JWT string as a literal token and compared bytes). Operators who treated CERTCTL_AUTH_SECRET as a JWT signing secret (and therefore handled it less carefully than an api-key) handed an attacker an api-key. Silent auth downgrade — a security finding masquerading as a config option.

We chose to remove the option rather than implement JWT middleware. Implementing real JWT/OIDC requires jwks vs static-secret rotation, claim mapping (which claim is the actor / the admin flag?), expiry enforcement, audience and issuer validation, key rollover semantics, and regression coverage at the same depth as the existing api-key path. That's a feature, not a fix. The audit-recommended structural fix — and the one that actually closes the hazard — is to fail loudly instead of silently downgrading.

What changes at startup

Post-G-1, a binary started with CERTCTL_AUTH_TYPE=jwt exits non-zero before opening the listener:

Failed to load configuration: CERTCTL_AUTH_TYPE=jwt is no longer accepted
(G-1 silent auth downgrade): no JWT middleware ships with certctl. To use
JWT/OIDC, run an authenticating gateway (oauth2-proxy / Envoy ext_authz /
Traefik ForwardAuth / Pomerium) in front of certctl and set
CERTCTL_AUTH_TYPE=none on the upstream. See docs/architecture.md
"Authenticating-gateway pattern" and docs/upgrade-to-v2-jwt-removal.md
for the migration walkthrough

Helm operators get the same shape at helm install / helm upgrade template time: server.auth.type=jwt is rejected by the chart's certctl.validateAuthType template helper before any Kubernetes object is rendered.

The CI-side regression guard at .github/workflows/ci.yml blocks any future PR that re-introduces "jwt" as an auth-type literal in production code or spec.

Recovery — pick one

Option A — switch to api-key (you weren't actually using JWT)

If your CERTCTL_AUTH_SECRET was a single high-entropy token and your clients sent it as Authorization: Bearer <token>, you were already using api-key auth — you just had CERTCTL_AUTH_TYPE set to the wrong string. Flip it:

# .env (docker-compose)
CERTCTL_AUTH_TYPE=api-key
CERTCTL_AUTH_SECRET=<your-existing-token>

# Helm
helm upgrade <release> deploy/helm/certctl/ \
  --reuse-values \
  --set server.auth.type=api-key \
  --set server.auth.apiKey=<your-existing-token>

No client changes needed — the same Bearer token continues to work. The startup log will now read "authentication enabled" "type"="api-key", which matches what was actually happening pre-G-1.

Option B — front certctl with an authenticating gateway

If you genuinely need JWT, OIDC, mTLS, or SAML, run an authenticating gateway in front of certctl and let the gateway terminate the federated identity protocol. Configure certctl for CERTCTL_AUTH_TYPE=none:

CERTCTL_AUTH_TYPE=none

Then put an oauth2-proxy / Envoy ext_authz / Traefik ForwardAuth / Pomerium / Authelia (etc.) in the network path between operators and certctl. The gateway validates the identity and proxies the authenticated request to certctl as a same-origin call on a private network.

Concrete walkthrough — oauth2-proxy + certctl on docker-compose

This is the simplest production-grade JWT/OIDC shape. It assumes you have an OIDC provider (Okta, Auth0, Google Workspace, Keycloak, Dex) and a registered client_id / client_secret.

# deploy/docker-compose.gateway.yml — overlay on the base compose file
services:
  oauth2-proxy:
    image: quay.io/oauth2-proxy/oauth2-proxy:latest
    command:
      - --provider=oidc
      - --oidc-issuer-url=https://<your-issuer>/
      - --client-id=${OIDC_CLIENT_ID}
      - --client-secret=${OIDC_CLIENT_SECRET}
      - --cookie-secret=${OAUTH2_PROXY_COOKIE_SECRET}  # openssl rand -base64 32
      - --upstream=http://certctl-server:8443  # internal-network only; certctl listens on 8443
      - --http-address=0.0.0.0:4180
      - --email-domain=*
      - --pass-access-token=true
      - --pass-authorization-header=true
      - --set-authorization-header=true       # forwards a bearer token upstream
      - --skip-provider-button=true
      - --reverse-proxy=true
    ports:
      - "443:4180"
    depends_on:
      - certctl-server
    networks:
      - certctl-network

  certctl-server:
    environment:
      CERTCTL_AUTH_TYPE: none   # gateway terminates auth — see docs/upgrade-to-v2-jwt-removal.md
      # ... rest of the certctl env block unchanged

Operators hit https://<your-host>/, get redirected through the OIDC provider, land back at oauth2-proxy with a session cookie, and oauth2-proxy proxies their request to certctl on the internal Docker network. certctl itself is HTTPS-only on :8443 (TLS 1.3, see tls.md) but operator browsers never see that hop directly. Bind certctl-server's :8443 to the internal Docker network only — do NOT publish it to the host. The audit trail will record the actor as the gateway-forwarded identity if you also configure a small bearer-token-mapping shim at the gateway (most production deployments do this with a per-user api-key issued by the gateway after OIDC validation).

Traefik ForwardAuth pattern (Kubernetes)

Same shape, kubernetes-flavored:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: oidc-forward-auth
spec:
  forwardAuth:
    address: http://oauth2-proxy.auth.svc.cluster.local:4180
    trustForwardHeader: true
    authResponseHeaders:
      - X-Auth-Request-User
      - X-Auth-Request-Email
      - Authorization
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: certctl
spec:
  routes:
    - match: Host(`certctl.example.com`)
      kind: Rule
      middlewares:
        - name: oidc-forward-auth
      services:
        - name: certctl-server
          port: 8443

The certctl Helm release runs with server.auth.type=none. The Traefik IngressRoute attaches oidc-forward-auth as a middleware so every request is OIDC-validated by oauth2-proxy before reaching certctl.

Envoy ext_authz pattern

For service-mesh deployments (Istio, Consul, plain Envoy), the ext_authz filter calls out to an external authorization service per-request. Same outcome: certctl runs CERTCTL_AUTH_TYPE=none and Envoy + your authz service handle JWT/OIDC/mTLS at the mesh edge. See the Envoy ext_authz docs for the configuration surface.

Rollback

Pre-G-1 binaries silently accepted CERTCTL_AUTH_TYPE=jwt and routed through the api-key middleware. Downgrading the binary is the only mechanical rollback path, and it puts you back into the silent-downgrade state — which is exactly what the G-1 audit finding is about. We don't recommend it. If something is forcing your hand, capture the operational issue you're hitting and open a GitHub issue against the certctl repo with the SHAs involved; the Authenticating-gateway pattern was specifically designed to cover the use cases that historically led operators to set CERTCTL_AUTH_TYPE=jwt.

There is no on-disk state that changes with this upgrade — no migrations to roll back, no encrypted config to re-encode, no certificates to re-issue. The change is entirely in the config-validation surface and the helm-chart template guard.

Cross-references

• architecture.md — "Authenticating-gateway pattern (JWT, OIDC, mTLS)" section.
• tls.md — TLS provisioning patterns. The gateway proxying to certctl-server still needs to trust certctl's TLS cert; same patterns apply.
• ../deploy/helm/certctl/README.md — Helm-chart-flavored guidance.
• internal/config/config.go::ValidAuthTypes — the single source of truth for what's accepted post-G-1.
• internal/repository/postgres/db.go::wrapPingError — unrelated; pattern for runtime diagnostic of operator misconfiguration.
• coverage-gap-audit-2026-04-24-v5/unified-audit.md — the audit finding (cat-g-jwt_silent_auth_downgrade).