certctl/docs/operator/runbooks/config-encryption-upgrade.md

# Runbook: forcing config-encryption blob upgrades (v1/v2 → v3)

> Last reviewed: 2026-05-12

Use this when:
- You've rotated `CERTCTL_CONFIG_ENCRYPTION_KEY` and want every row in
  the database to be re-sealed under the new passphrase, not just the
  next ones to be touched.
- A v1- or v2-era encrypted blob existed in your database before you
  upgraded to a post-M-8 release and you want to retire the legacy
  read path's PBKDF2 work factor (100,000 rounds) in favor of the v3
  factor (600,000 rounds, OWASP 2024).
- You're preparing for an audit and want every at-rest encrypted blob
  to be on the same wire format.

Audience: a platform sysadmin who can run SQL against certctl's
PostgreSQL instance and exercise the GUI/REST API write paths.

For background on the v3 / v2 / v1 wire formats and the FileDriver vs
HSM threat model, read
[`docs/operator/secret-custody.md`](../secret-custody.md) first.

---

## Background: how the read fallback works

`internal/crypto/encryption.go::DecryptIfKeySet` reads three on-disk
formats in this order:

```
v3 (magic 0x03, per-ciphertext 16-byte salt, PBKDF2 600k) →
v2 (magic 0x02, per-ciphertext 16-byte salt, PBKDF2 100k) →
v1 (no magic, fixed 28-byte salt, PBKDF2 100k)
```

The fallback is AEAD-driven: if v3 decryption fails authentication, the
function tries v2; if v2 fails, v1. This is what keeps pre-M-8 v1 blobs
readable without an explicit migration.

`EncryptIfKeySet` always writes v3. As a result, any row that is
**re-written** through the normal application code path is silently
upgraded to v3 the moment it's persisted.

The implication: you do not need to "migrate" v1/v2 blobs for them to
keep working — only if you want the v1/v2 wire format physically gone
from your database.

## Procedure

### Step 1 — confirm the encryption key is set

Re-encryption obviously cannot run without a passphrase. Verify:

```bash
echo "${CERTCTL_CONFIG_ENCRYPTION_KEY:-NOT SET}" | sed -E 's/./*/g'
```

If the variable prints `NOT SET`, do not proceed — set the key in your
deployment manifest and restart the control plane first.

### Step 2 — identify which tables hold encrypted blobs

Encrypted columns in the v2.1.0 schema:

| Table              | Column                | Notes                                                                |
|---|---|---|
| `issuers`          | `encrypted_config`    | Only populated for `source='database'` rows (env-seeded rows are not encrypted) |
| `targets`          | `encrypted_config`    | Same source-based gating as issuers                                  |
| `oidc_providers`   | `client_secret_enc`   | OIDC client_secret                                                   |
| `auth_session_signing_keys` | `key_material_enc` | HMAC-SHA256 session-cookie signing key                          |

If your schema differs, derive the column list from the migration
folder:

```bash
grep -hE '_enc[ ,]|encrypted_config' migrations/*.up.sql | sort -u
```

### Step 3 — identify rows still on v1/v2

The magic byte of the blob distinguishes versions; v1 blobs start with
the random AES-GCM nonce (anything but `0x02` or `0x03` is definitely
v1), and v2 vs v3 is determined by the first byte:

```sql
-- Per-table version distribution (run against your live database)
SELECT
    SUBSTRING(encrypted_config FROM 1 FOR 1)::bytea AS magic,
    COUNT(*) AS rows
  FROM issuers
  WHERE encrypted_config IS NOT NULL
  GROUP BY magic;
```

Expected steady-state output is a single row with `magic = \x03`.
Any rows with `\x02` are v2; any rows with anything else are v1.

### Step 4 — force re-sealing

`UPDATE` the rows back to themselves through the normal application
write path. The cleanest way to do this is via the REST API or GUI,
not raw SQL — re-issuing the same `PUT /api/v1/issuers/:id` reads the
row, decrypts, then re-encrypts under v3 on the write back.

For an issuer named `iss-letsencrypt-prod`:

```bash
# Fetch then re-PUT the same body (CSRF + bearer token elided).
curl -sS https://certctl.example.com/api/v1/issuers/iss-letsencrypt-prod \
  -H "Authorization: Bearer $CERTCTL_API_KEY" \
  | jq '.' \
  | curl -sS -X PUT https://certctl.example.com/api/v1/issuers/iss-letsencrypt-prod \
      -H "Authorization: Bearer $CERTCTL_API_KEY" \
      -H "Content-Type: application/json" \
      --data-binary @-
```

Repeat for each row that the Step 3 query flagged as non-v3.

### Step 5 — verify

Re-run the Step 3 query. The output should now show only `magic =
\x03` rows.

## Special case: rotating the encryption-key passphrase

If your goal is to retire a possibly-compromised passphrase rather
than retire a legacy wire format, the order is:

1. Generate a new passphrase. Document it via your secret-management
   tool (HashiCorp Vault, AWS Secrets Manager, etc.).
2. Stop the control plane briefly so no rows are written under the
   stale passphrase during the transition window.
3. Run a one-shot decrypt-with-old / re-encrypt-with-new pass.
   certctl ships no built-in tool for this — see the open
   roadmap item below. The cleanest current approach is:
    - Start certctl with the OLD passphrase.
    - Read every encrypted column out to a JSON dump via the REST API.
    - Stop certctl. Update its env to the NEW passphrase. Restart.
    - PUT every row back from the JSON dump (the writes re-seal under
      the new passphrase).
4. Document the old passphrase as retired in your secret-management
   tool. Anyone with read access to a pre-rotation backup still needs
   it to decrypt that backup; the live database no longer needs it.

For most operators, simply rotating the passphrase and letting the
re-seal happen organically as rows are touched is acceptable — the
v3 wire format with PBKDF2 600k rounds makes offline brute-force
against the old passphrase computationally expensive.

## Open roadmap items

- Ship a built-in `certctl admin reseal --all` command that does Steps
  3 and 4 in one shot, with structured progress + audit logging.
  Tracked in [WORKSPACE-ROADMAP.md](../../WORKSPACE-ROADMAP.md).
- Surface per-table v1/v2/v3 distribution as a Prometheus gauge so
  alerting can fire on "rows on legacy format" drift.

## Related reading

- [`docs/operator/secret-custody.md`](../secret-custody.md) — the
  broader where-do-private-keys-live reference; this runbook is the
  procedural arm of that document.
- [`internal/crypto/encryption.go`](../../../internal/crypto/encryption.go)
  package comment — wire format authoritative reference.