唐小鸭
ae9fe62fb1
* fix(sse): resolve bucket default encryption per request PUT and the POST-object/extract path resolved a bucket's default encryption with a hard-coded "no explicit SSE-C" flag, so the default was layered onto a request that already carried an SSE-C header triple and then tripped that request's own mutual-exclusion check. Every bucket with default encryption refused SSE-C single PUTs with 400 InvalidArgument, while CreateMultipartUpload on the same bucket succeeded because it resolves SSE elsewhere. Both call sites now derive the flag from the request headers, as COPY already did. The bucket default's KMS key id was also inherited independently of the effective algorithm, so an explicit AES256 request against an aws:kms default bucket produced a self-contradictory algorithm/key-id pair and was rejected. The key id is now inherited only when the effective algorithm is aws:kms, matching the storage-layer resolver. Refs backlog#2368 B1, B2. * fix(sse): refuse SSE-KMS without a running KMS service A write requesting aws:kms on a node with no KMS service fell back to the node-local SSE-S3 provider: the data key was wrapped with RUSTFS_SSE_S3_MASTER_KEY while the object metadata still recorded aws:kms and the requested KMS key id. The stored object claimed a KMS protection it never had, under a key that was never consulted, and no signal distinguished it from a genuine SSE-KMS object. The managed-encryption path now asks the resolved DEK provider whether it wraps with a node-local master key and refuses SSE-KMS in that case: InvalidRequest when KMS was never configured, ServiceUnavailable when a configured service is not running. The check sits after the per-key authorization gate so an unauthorized caller still receives AccessDenied whatever the KMS runtime state is, and asks the provider rather than a parallel availability signal because the provider is what actually wraps the key. A missing master key no longer answers an SSE-KMS request with an SSE-S3-worded configuration error. The SSE-S3 local fallback is unchanged. Refs backlog#2368 B4. * fix(ecstore): restore and archive tiers in stored coordinates Multipart restore addressed the remote tier in plaintext coordinates while the copy-back reads the stored representation. Each part received a misaligned slice of the remote object whose length still satisfied the range, the hash reader and the completion size check, so the restore reported success and silently replaced the object's bytes. Encrypted and compressed multipart objects were both affected. Restore now accumulates stored part sizes, passes the stored length to the hash reader alongside the plaintext length, and validates against the stored size. The copy-back digests stored bytes, so its computed MD5 is not the object's public ETag. Restore now preserves the object ETag on both the single-part and multipart paths, and gives each restored part its own recorded part ETag rather than the object-level value. Transition also handed the tier the object's SSE headers and its RustFS-wrapped data key as request headers. Any S3 target rejected an SSE-C archive outright, an SSE-KMS archive asked the target to encrypt a second time under a key id it does not own, and the wrapped DEK left the cluster. The archive request now strips every SSE header and encryption marker with the predicate the replication path already uses; the local xl.meta keeps all of it, so read-through and restore are unaffected. Objects restored by an affected release are not detected or repaired retroactively and must be re-restored from the tier. Refs backlog#2368 B3, B5; backlog#2369 P7.1. * fix(rio): lock the v1 nonce layout within a segment Decrypting a v1 segment tried three historical nonce layouts per frame, independently for every frame. The last of them exists for streams written before 1.0.0-alpha.91, which reused a segment's part nonce for every block in it; because block zero's derived nonce equals that base nonce, a frame encrypted at index zero authenticated at any position. An attacker able to rewrite the underlying shards could replay it and have the forged plaintext returned with 200 and an unchanged length. Shard integrity uses a keyed-hash-free checksum, which such an attacker can recompute, so it is not a barrier. A segment now locks onto whichever layout decoded its first non-zero-index frame and rejects any later frame needing a different one. That leaves one residual shape: a stream built purely from repeats of frame zero has no later frame to disagree. New RUSTFS_ENCRYPTION_LEGACY_NONCE_FALLBACK (default true, so pre-alpha.91 objects keep decrypting) drops the third layout entirely when set to false, which closes it. Turning it off refuses pre-alpha.91 objects, so migrate them first by rewriting in place. Refs backlog#2369 P2. * fix(kms): reload a service that failed to start POST /rustfs/admin/v3/kms/reload short-circuited whenever the persisted configuration matched the in-memory one byte for byte. A node whose KMS failed to start keeps that configuration and sits in Error, so the documented recovery call returned "reloaded successfully" while leaving the node down. Peers reached the same path through the reload broadcast, so a cluster that lost Vault during a rolling restart had no working recovery route other than the node-local start endpoint. Reload now short-circuits only for a service that is actually running, and otherwise reconfigures, which starts a service that is not running. The AWS backend also advertised key-version enumeration through kms/status, which its own documentation says it cannot do; the capability and its golden snapshot now say false. Refs backlog#2369 P1, P7.3. * docs: record the SSE and KMS changes for 1.0.0 The Unreleased changelog section carried no entry for any encryption work merged since 1.0.0-rc.5, including three items with operational impact: the config-secret variable whose absence persists secrets in cleartext with only a warning, the v2 frame write switch and its rolling-upgrade constraint, and per-key authorization making a public bucket incompatible with SSE-KMS objects. Adds those plus this batch, including the SSE-KMS refusal as a breaking change with both routes out. Also corrects four places where documentation contradicted the code: the cleanup register still called encrypted range seek opt-in after its default flipped, the Helm README claimed vault_mount_path only applies to Transit while the template also feeds the KV2 mount, the disaster-recovery drill listed bundle contents for backends whose export is refused with 501, and the Chinese README capability table predated most of the feature set. Documents the SSE-S3 local master key as a first-class operational mode with its rotation dead end, and what the v1 frame layout does and does not authenticate. Refs backlog#2369 P5. * fix(kms): classify data-path KMS failures by what the caller can do Only "key not found" and a backend outage were classified; every other KMS failure that reached the S3 data path fell through to 500 InternalError with a generic message. A disabled or pending-deletion key, a denied KMS grant, an encryption-context mismatch, an unsupported algorithm, a credential or timeout failure, and a capability the configured backend does not have all looked identical to a server fault. SDKs therefore applied exponential backoff to configuration errors no retry can fix, and monitoring counted every one of them against the server's own error rate. Unusable-key and request-side failures now answer 400, a denied grant 403, transient backend failures 503, and a missing backend capability 501. Damaged, unreadable, or unknown-format key material keeps its 500: it is a server-side integrity fault, and existing tests pin it. The classifier is deliberately separate from the admin lifecycle mapping, which answers 404 for a missing key because there a key id is the resource being addressed; on the data path it arrives inside a request header or a bucket default. Messages either name what the caller asked for or stay generic, with deployment-side detail left on the error source the way the storage-IO mapping already does. Refs backlog#2368 B6. * fix(kms): track and renew static Vault tokens Token authentication hard-coded "this token carries no lease", so the renewal task never started, the remaining-TTL gauge was never published, and nothing looked wrong. `vault token create` grants a 768-hour TTL by default, so a cluster that had been healthy for a month turned every KMS call into a 403 and could not recover without a restart or a reconfigure. Production configuration validation only rejects the literal dev-token, so an ordinary expiring token reaches a whole cluster. The source now reads `auth/token/lookup-self` at login and adopts what Vault reports. A token with no expiry behaves exactly as before. An expiring renewable one is picked up by the existing renewal loop and renewed at half TTL like every other auth method. An expiring non-renewable one warns with its remaining lifetime and publishes the gauge, so the fail-closed window is visible before it arrives. The probe never fails the login: a policy that omits lookup-self, or a Vault that is briefly unreachable, warns and falls back to exactly the previous behaviour rather than taking down a deployment that works today. The scripted Vault test double answers the lookup out of band so existing scripts keep describing only the protocol under test. Refs backlog#2369 P3. * feat(sse): report SSE-C requests that arrive without TLS An SSE-C request carries the customer's AES key in a request header, so AWS S3 and MinIO both refuse one that did not arrive over TLS. RustFS accepted them on any transport: a plaintext hop hands the key to anyone on the path, and since the object cannot be read without that same key, the exposure lasts as long as the object does. Refusing outright is the correct end state but not a safe default to adopt inside a release window, because the project's own s3-tests and e2e lanes and most staging deployments speak plain HTTP. This release reports instead: each such request increments rustfs_ssec_plaintext_requests_total and logs one warning per process, so an operator can confirm nothing would break before the default flips. RUSTFS_SSE_C_REQUIRE_TLS=true opts into the AWS 400 now. The verdict is per connection rather than per deployment: the layer is built with whether this listener terminated TLS, and additionally accepts an https protocol forwarded by a proxy the trusted-proxy configuration already vetted. It sits beside the rate limiter, after the layer that makes a forwarded protocol trustworthy and after the request context, so a rejection can echo the request id. Refs backlog#2369 P7.2. * fix(kms): say what a node-local backend means for a cluster The Local backend keeps key material on each node's own disk and generates its Argon2id salt per node, so two nodes derive different keys from the same master_key and an object encrypted on one node cannot be decrypted on another. Behind a load balancer that surfaces as intermittent 500s on reads that succeeded moments earlier, with nothing tying the symptom to the cause: the only signal was a generic "development, testing and demos only" positioning warning that says nothing about what actually breaks. Configuring or reconfiguring Local while the deployment is distributed now logs a dedicated event and appends the consequence to the configure response, so the operator who made the change sees it. The product decision to warn rather than refuse is unchanged. Refs backlog#2369 P7.4. * docs: record the remaining SSE and KMS changes for 1.0.0 Adds changelog entries for the KMS data-path status classification, the Vault static-token lease probe, the SSE-C plaintext-transport report and its switch, and the node-local backend warning. Documents two things the backend security guide never stated: that SSE-C belongs on a secure transport, with the counter and switch to plan the change around, and that the Local backend cannot be shared by a multi-node deployment because each node derives different keys from the same master key. Refs backlog#2368 B6; backlog#2369 P3, P5, P7.2, P7.4. * fix(kms): report an unreadable key store as an outage on the S3 path A backend now distinguishes a key store it could not read from a key that is genuinely absent, but the S3 boundary collapsed the first one back onto 500 InternalError through the fallthrough for integrity faults. The distinction was therefore invisible to the client: a temporary key-directory outage looked exactly like a permanently damaged key record, and neither the status nor the metric said the request was worth retrying. An unreadable key store joins the retryable class and answers 503, next to a backend error and a credential failure. Damaged, unreadable or unknown-format key material keeps its 500. Refs backlog#2368 B6; builds on rustfs/rustfs#7470.
13 KiB
Vault KMS authentication runbook
Use this when: configuring how the Vault KMS backends (KV2 and Transit) authenticate to Vault, rotating AppRole SecretIDs or agent tokens, or diagnosing KMS credentials unavailable errors.
Source of truth: crates/kms/src/backends/vault_credentials.rs (DEFAULT_TOKEN_FILE_POLL_INTERVAL_SECS, refresh_safety_window_secs, the renewal loop and its log lines), crates/kms/src/config.rs (RUSTFS_KMS_TIMEOUT_SECS default). For what each backend stores in Vault and the KV2/Transit policy scopes, see KMS backend security properties.
Choosing an authentication method
| Method | Config tag | Credential lifetime | Background renewal | Recommended for |
|---|---|---|---|---|
| Static token | Token |
Whatever the operator provisioned; read from Vault at login | Renewed at half TTL when Vault reports the token as renewable | Development; short-lived experiments |
| AppRole | AppRole |
Lease-bound token obtained by login; renewed by RustFS | Renew at half TTL, re-login on failure | Production without a Vault Agent sidecar |
| Kubernetes | Kubernetes |
Lease-bound token obtained by login; renewed by RustFS | Renew at half TTL, re-login on failure | Production on Kubernetes, with no credential to distribute |
| Agent token file | TokenFile |
Owned by Vault Agent; RustFS only re-reads the sink file | File re-read once per poll interval | Production with a Vault Agent (or equivalent) managing auth |
Static token: what RustFS now knows about it
vault token create grants a 768-hour TTL by default, so a static token normally does expire. At login RustFS calls auth/token/lookup-self and adopts whatever Vault reports:
- No expiry (a root or periodic-root token,
ttl0): unchanged — no lease is tracked, no renewal task runs, and the token is never refused locally. - Expiring and renewable: the ordinary renewal loop takes over, renewing at half the remaining TTL and publishing the remaining-TTL gauge.
- Expiring but not renewable: a
vault_static_token_not_renewablewarning is logged with the remaining TTL, the gauge is published, and requests fail closed inside the safety window rather than lapsing mid-flight against Vault. Rotate to a fresh token, or move to AppRole, Kubernetes, or an agent-managed token file.
The probe never fails the login. A token whose policy omits lookup-self (Vault's default policy grants it), or a Vault that is unreachable at that moment, logs vault_static_token_lookup_failed and falls back to the previous behaviour — no lease tracked, no renewal — so a deployment that works today keeps working. That fallback lasts for the life of the client generation, so treat the warning as something to fix rather than tolerate.
Exactly one method must be configured. Setting RUSTFS_KMS_VAULT_TOKEN_FILE together with any other method, or RUSTFS_KMS_VAULT_KUBERNETES_ROLE together with RUSTFS_KMS_VAULT_APPROLE_ROLE_ID, is rejected at startup with a configuration error because the effective identity would be ambiguous. A leftover RUSTFS_KMS_VAULT_TOKEN alongside a configured login method is tolerated and ignored, so a stale variable cannot silently downgrade the identity.
All of these are read the same way whether the service is started with RUSTFS_KMS_ENABLE=true or configured later through POST /rustfs/admin/v3/kms/configure.
The default dev-token fallback for RUSTFS_KMS_VAULT_TOKEN is rejected outside explicit development mode (RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS=true), as are plain-HTTP Vault addresses and disabled TLS verification.
AppRole authentication
Vault-side setup
Create a policy scoped to exactly what the backend needs (see the KV2 and Transit policy examples in KMS backend security properties), then an AppRole that issues tokens carrying it:
vault policy write rustfs-kms rustfs-kms-policy.hcl
vault auth enable approle
vault write auth/approle/role/rustfs-kms \
token_policies="rustfs-kms" \
token_ttl=1h \
token_max_ttl=24h \
secret_id_ttl=90d \
secret_id_num_uses=0
Keep token_ttl comfortably above the RustFS per-attempt timeout (default 30s): the fail-closed window defaults to one attempt timeout, so a token TTL close to it leaves almost no usable lifetime.
RustFS configuration
RUSTFS_KMS_BACKEND=vault-transit # or "vault" for the KV2 backend
RUSTFS_KMS_VAULT_ADDRESS=https://vault.example.com:8200
RUSTFS_KMS_VAULT_APPROLE_ROLE_ID=<role-id>
RUSTFS_KMS_VAULT_APPROLE_SECRET_ID_FILE=/etc/rustfs/approle-secret-id
# Alternatively, inline (the file takes precedence when both are set):
# RUSTFS_KMS_VAULT_APPROLE_SECRET_ID=<secret-id>
# Optional, defaults to "approle":
# RUSTFS_KMS_VAULT_APPROLE_MOUNT=approle
RustFS logs in at startup, then renews the token at half its TTL in the background. If renewal fails (network, Vault sealed, token revoked), it falls back to a full re-login; if that also fails, it keeps retrying every few seconds until Vault recovers.
SecretID delivery and rotation
Deliver the SecretID out of band (a secrets-manager-mounted file, an init-container writing RUSTFS_KMS_VAULT_APPROLE_SECRET_ID_FILE, or Vault response wrapping unwrapped by your deployment tooling). Treat it like a password: owner-readable file permissions, never in logs or shell history.
The secret_id file is re-read on every login attempt, so rotating the SecretID needs no restart:
- Generate a new SecretID:
vault write -f auth/approle/role/rustfs-kms/secret-id. - Atomically replace the file.
- Revoke the old SecretID accessor. The already-issued token keeps renewing; the new SecretID is only needed at the next full re-login.
An empty or missing secret_id file fails the login attempt immediately (no Vault round trip). At startup the error is fatal: provider construction fails and the process exits, so a file missing at boot is recovered by restarting the process, not by an in-process retry. Once RustFS is running, the same failure is retried on the normal refresh cadence, so repairing the file mid-run heals the backend without a restart.
Kubernetes authentication
On Kubernetes this is the method to prefer: the pod's own ServiceAccount is the identity, so there is no credential to distribute, rotate, or leak into a Secret.
Vault-side setup
vault auth enable kubernetes
vault write auth/kubernetes/config \
kubernetes_host="https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT"
vault write auth/kubernetes/role/rustfs \
bound_service_account_names=rustfs \
bound_service_account_namespaces=rustfs \
token_policies=rustfs-kms \
token_ttl=1h
As with AppRole, keep token_ttl comfortably above the RustFS per-attempt timeout (default 30s).
RustFS configuration
RUSTFS_KMS_BACKEND=vault-transit # or "vault" for the KV2 backend
RUSTFS_KMS_VAULT_ADDRESS=https://vault.vault.svc.cluster.local:8200
RUSTFS_KMS_VAULT_KUBERNETES_ROLE=rustfs
# Optional, defaults to "kubernetes":
# RUSTFS_KMS_VAULT_KUBERNETES_MOUNT=kubernetes
# Optional, defaults to the kubelet's projected token path:
# RUSTFS_KMS_VAULT_KUBERNETES_JWT_PATH=/var/run/secrets/kubernetes.io/serviceaccount/token
RustFS logs in at startup and renews the token at half its TTL, falling back to a fresh login exactly as AppRole does. The ServiceAccount token is re-read from disk on every login rather than cached, so a projected token the kubelet rotates is picked up without a restart.
A missing or empty token file fails the login attempt immediately (no Vault round trip). At startup the error is fatal (provider construction fails and the process exits), so a token projected late during a slow pod start is recovered by the pod restart loop, not by an in-process retry. Once RustFS is running, a token file that goes missing or turns empty is retried on the normal refresh cadence and heals the backend on its own.
Vault Agent token file
In this mode a Vault Agent (or any equivalent process) owns authentication and token renewal, and RustFS only reads the token sink file.
Vault Agent example
auto_auth {
method "approle" {
config = {
role_id_file_path = "/etc/vault-agent/role-id"
secret_id_file_path = "/etc/vault-agent/secret-id"
}
}
sink "file" {
config = {
path = "/run/vault-agent/token"
mode = 0600
}
}
}
RustFS configuration
RUSTFS_KMS_BACKEND=vault-transit
RUSTFS_KMS_VAULT_ADDRESS=https://vault.example.com:8200
RUSTFS_KMS_VAULT_TOKEN_FILE=/run/vault-agent/token
The poll interval (poll_interval_secs in the TokenFile auth configuration, DEFAULT_TOKEN_FILE_POLL_INTERVAL_SECS, 30 seconds) controls how often the file is re-read. Each successful read grants the token an observed validity of twice the poll interval and installs a fresh client generation, so an agent-rotated token is picked up within one poll interval of the atomic replace.
Requirements enforced at every read, each failing the refresh without contacting Vault:
- The file must exist and be non-empty after trimming whitespace.
- On Unix, the file must not be readable or writable by group or other (mode
0600or stricter). Wider permissions are a hard error naming the offending mode, mirroring the SFTP host-key rule. RustFS must run as the file's owner.
If the agent stops refreshing the file, RustFS re-reads the same token and keeps going as long as the token itself is valid on the Vault side. If the file disappears or turns empty, RustFS keeps serving requests on the last-read token until the fail-closed window trips, and heals automatically once the file is restored.
Fail-closed window
For lease-bound credentials (AppRole and Kubernetes tokens, token files), current() refuses to hand out a token that is within the safety window of its expiry and has not been refreshed. Requests then fail with KMS credentials unavailable: ... instead of being sent with a token that could lapse mid-flight and fail unpredictably on the Vault side.
| Aspect | Value |
|---|---|
| Default window | One per-attempt timeout (RUSTFS_KMS_TIMEOUT_SECS, default 30s): a request issued now can legitimately stay in flight that long, so the token must outlive it. |
| Override | refresh_safety_window_secs on the AppRole, Kubernetes or TokenFile auth configuration. |
| Static tokens | Never trip the window; they carry no lease and are assumed valid until Vault says otherwise. |
The window is a symptom threshold, not the fault itself: by the time it trips, refresh has been failing for roughly half the token TTL (AppRole, Kubernetes) or two poll intervals (token file).
Troubleshooting
| Symptom | Log line to look for | Likely cause and fix |
|---|---|---|
Requests fail with KMS credentials unavailable |
Vault credential refresh failed; retrying until the credentials recover (warn, repeated) |
Vault unreachable/sealed, or the credential source is broken; the provider recovers on its own once refresh succeeds. Fix the cause; no restart needed. |
| Renewal succeeded but re-login later fails | Vault token renewal failed; falling back to a fresh login followed by login errors |
SecretID expired/revoked or AppRole role changed; rotate the secret_id file. |
| Token file mode error at startup or during polls | has insecure permissions in the error |
Fix the sink mode (0600) and the file owner; the next poll heals the provider. |
| Token file missing/empty errors | Failed to read Vault token file / token file ... is empty |
Vault Agent down or sink misconfigured; restart the agent, the next poll heals the provider. |
| Kubernetes login fails with a permission error | Vault Kubernetes login failed |
The pod's ServiceAccount is not in the role's bound_service_account_names/_namespaces, or auth/kubernetes/config names the wrong API server. |
| Kubernetes ServiceAccount token errors | Failed to read Kubernetes ServiceAccount token / ServiceAccount token ... is empty |
The token is not projected into the pod (check automountServiceAccountToken and the volume mount); the next refresh cycle heals the provider. |
| Startup fails immediately with a configuration error naming two env vars | (none) | Two auth methods configured at once; keep exactly one of token, AppRole, Kubernetes, token file. |
When diagnosing, confirm three clocks/lifetimes in order: the Vault token TTL (vault token lookup with the token's accessor), the RustFS refresh cadence (half TTL or the poll interval), and the fail-closed window. The renewal task logs every failed cycle, so a silent gap in warnings combined with CredentialsUnavailable errors points at the process clock or a paused runtime rather than Vault.