fix(kms): resolve Vault auth from the environment at startup and add Kubernetes auth

The server startup path built its Vault backend config field by field from the
command-line struct, hardcoding VaultAuthMethod::Token and requiring a token.
KmsConfig::from_env(), which already resolved AppRole and token-file auth plus
namespace, TLS and mount settings, was never called outside tests, so those
environment variables were silently dropped whenever RUSTFS_KMS_ENABLE=true and
the documented AppRole / Vault Agent deployments could not start.

Move the assembly into vault_kv2_config_from_env / vault_transit_config_from_env
in the KMS crate and route both from_env() and init.rs through them, with the
command line supplying only the values it owns. One implementation now serves
both entry points, so they cannot drift apart again.

On top of that, add VaultAuthMethod::Kubernetes: the pod's projected
ServiceAccount token is exchanged for a lease-bound Vault token and renewed like
AppRole. The token is re-read on every login because the kubelet rotates it, and
the file mode is deliberately not checked since the kubelet mounts it
world-readable. This removes the Vault Agent sidecar requirement on Kubernetes
and leaves no credential to distribute.

VaultCliOverrides deliberately does not derive Debug: it carries the raw token,
so denying the derive turns a future interpolation into a compile error.
This commit is contained in:
唐小鸭
2026-08-14 10:24:43 +08:00
parent aa4d3317ed
commit ab7e777e55
6 changed files with 742 additions and 98 deletions
+47 -5
View File
@@ -8,9 +8,12 @@ This runbook covers how the RustFS Vault KMS backends (KV2 and Transit) authenti
| --- | --- | --- | --- | --- |
| Static token | `Token` | Whatever the operator provisioned; RustFS never renews it | None | Development; short-lived experiments |
| AppRole | `AppRole` | Lease-bound token obtained by login; renewed by RustFS | Renew at half TTL, re-login on failure | Production without a Vault Agent sidecar |
| Kubernetes | `Kubernetes` | Lease-bound token obtained by login; renewed by RustFS | Renew at half TTL, re-login on failure | Production on Kubernetes, with no credential to distribute |
| Agent token file | `TokenFile` | Owned by Vault Agent; RustFS only re-reads the sink file | File re-read once per poll interval | Production with a Vault Agent (or equivalent) managing auth |
Exactly one method must be configured. Setting `RUSTFS_KMS_VAULT_TOKEN_FILE` together with `RUSTFS_KMS_VAULT_APPROLE_ROLE_ID` or an explicit `RUSTFS_KMS_VAULT_TOKEN` is rejected at startup with a configuration error, because the effective identity would be ambiguous.
Exactly one method must be configured. Setting `RUSTFS_KMS_VAULT_TOKEN_FILE` together with any other method, or `RUSTFS_KMS_VAULT_KUBERNETES_ROLE` together with `RUSTFS_KMS_VAULT_APPROLE_ROLE_ID`, is rejected at startup with a configuration error, because the effective identity would be ambiguous. A leftover `RUSTFS_KMS_VAULT_TOKEN` alongside a configured login method is tolerated and ignored, so a stale variable cannot silently downgrade the identity.
All of these are read the same way whether the service is started with `RUSTFS_KMS_ENABLE=true` or configured later through `POST /rustfs/admin/v3/kms/configure`.
The default `dev-token` fallback for `RUSTFS_KMS_VAULT_TOKEN` is rejected outside explicit development mode (`RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS=true`), as are plain-HTTP Vault addresses and disabled TLS verification.
@@ -58,6 +61,43 @@ The secret_id file is re-read on every login attempt, so rotating the SecretID i
An empty or missing secret_id file fails the login attempt immediately (no Vault round trip) and is retried on the normal refresh cadence, so repairing the file heals the backend without a restart.
## Kubernetes authentication
On Kubernetes this is the method to prefer: the pod's own ServiceAccount is the identity, so there is no credential to distribute, rotate, or leak into a Secret.
### Vault-side setup
```shell
vault auth enable kubernetes
vault write auth/kubernetes/config \
kubernetes_host="https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT"
vault write auth/kubernetes/role/rustfs \
bound_service_account_names=rustfs \
bound_service_account_namespaces=rustfs \
token_policies=rustfs-kms \
token_ttl=1h
```
As with AppRole, keep `token_ttl` comfortably above the RustFS per-attempt timeout (default 30s).
### RustFS configuration
```shell
RUSTFS_KMS_BACKEND=vault-transit # or "vault" for the KV2 backend
RUSTFS_KMS_VAULT_ADDRESS=https://vault.vault.svc.cluster.local:8200
RUSTFS_KMS_VAULT_KUBERNETES_ROLE=rustfs
# Optional, defaults to "kubernetes":
# RUSTFS_KMS_VAULT_KUBERNETES_MOUNT=kubernetes
# Optional, defaults to the kubelet's projected token path:
# RUSTFS_KMS_VAULT_KUBERNETES_JWT_PATH=/var/run/secrets/kubernetes.io/serviceaccount/token
```
RustFS logs in at startup and renews the token at half its TTL, falling back to a fresh login exactly as AppRole does. The ServiceAccount token is re-read from disk on every login rather than cached, so a projected token the kubelet rotates is picked up without a restart.
A missing or empty token file fails the login attempt immediately (no Vault round trip) and is retried on the normal refresh cadence, so a token projected late — during a slow pod start, for example — heals the backend on its own.
## Vault Agent token file
In this mode a Vault Agent (or any equivalent process) owns authentication and token renewal, and RustFS only reads the token sink file.
@@ -101,13 +141,13 @@ If the agent stops refreshing the file that is fine — RustFS re-reads the same
## Fail-closed window
For lease-bound credentials (AppRole tokens, token files), `current()` refuses to hand out a token that is within the safety window of its expiry and has not been refreshed. Requests then fail with `KMS credentials unavailable: ...` instead of being sent with a token that could lapse mid-flight and fail unpredictably on the Vault side.
For lease-bound credentials (AppRole and Kubernetes tokens, token files), `current()` refuses to hand out a token that is within the safety window of its expiry and has not been refreshed. Requests then fail with `KMS credentials unavailable: ...` instead of being sent with a token that could lapse mid-flight and fail unpredictably on the Vault side.
- Default window: one per-attempt timeout (`RUSTFS_KMS_TIMEOUT_SECS`, default 30s) — a request issued now can legitimately stay in flight that long, so the token must outlive it.
- Override: `refresh_safety_window_secs` on the `AppRole` or `TokenFile` auth configuration.
- Override: `refresh_safety_window_secs` on the `AppRole`, `Kubernetes` or `TokenFile` auth configuration.
- Static tokens never trip the window: they carry no lease and are assumed valid until Vault says otherwise.
The window is a symptom threshold, not the fault itself: by the time it trips, refresh has been failing for roughly half the token TTL (AppRole) or two poll intervals (token file).
The window is a symptom threshold, not the fault itself: by the time it trips, refresh has been failing for roughly half the token TTL (AppRole, Kubernetes) or two poll intervals (token file).
### Troubleshooting
@@ -117,6 +157,8 @@ The window is a symptom threshold, not the fault itself: by the time it trips, r
| Renewal succeeded but re-login later fails | `Vault token renewal failed; falling back to a fresh login` followed by login errors | SecretID expired/revoked or AppRole role changed; rotate the secret_id file |
| Token file mode error at startup or during polls | `has insecure permissions` in the error | Fix the sink `mode` (0600) and the file owner; the next poll heals the provider |
| Token file missing/empty errors | `Failed to read Vault token file` / `token file ... is empty` | Vault Agent down or sink misconfigured; restart the agent, the next poll heals the provider |
| Startup fails immediately with a configuration error naming two env vars | — | Two auth methods configured at once; keep exactly one of token, AppRole, token file |
| Kubernetes login fails with a permission error | `Vault Kubernetes login failed` | The pod's ServiceAccount is not in the role's `bound_service_account_names`/`_namespaces`, or `auth/kubernetes/config` names the wrong API server |
| Kubernetes ServiceAccount token errors | `Failed to read Kubernetes ServiceAccount token` / `ServiceAccount token ... is empty` | The token is not projected into the pod (check `automountServiceAccountToken` and the volume mount); the next refresh cycle heals the provider |
| Startup fails immediately with a configuration error naming two env vars | — | Two auth methods configured at once; keep exactly one of token, AppRole, Kubernetes, token file |
When diagnosing, confirm three clocks/lifetimes in order: the Vault token TTL (`vault token lookup` with the token's accessor), the RustFS refresh cadence (half TTL or the poll interval), and the fail-closed window. The renewal task logs every failed cycle, so a silent gap in warnings combined with `CredentialsUnavailable` errors points at the process clock or a paused runtime rather than Vault.