* fix(kms): fail closed on Local key records this build cannot interpret
The Local backend's protection marker is the only version discriminator its
key records have, and three readers walked past it.
`ensure_missing_salt_can_be_generated` skipped every record it could not read
or parse, so a directory whose protection state is unknown still got a fresh
salt published before startup validation failed. That write is the
irreversible step: the next startup finds a salt file, never re-enters the
guard, and the evidence that the real salt was lost is gone. Every record the
guard now rejects already failed startup key validation a few lines later, so
no directory that initializes today stops initializing.
Backup export and restore folded an unknown marker into "material corrupt" /
"bundle corrupted". The record is intact and a newer build reads it fine, so
the operator response is a version change, not a disaster recovery. Both now
classify the marker before their schema parse, sharing one probe with the
backend reader.
`list_keys` dropped any record it could not decode from the page. Concurrent
removal stays a skip; anything else fails the listing rather than answering
"these are your keys" with a set that silently omits one.
* test(kms): cover every fail-closed path around the Local protection marker
Each test fails on the pre-fix code in the way the fix is about: the salt
cases because a replacement salt is published before startup validation
fails, the export and restore cases because the verdict comes back as
corruption, and the listing case because the record is edited out of the page.
The restore commit marker's unknown-version branch had no test at all,
unlike its Vault counterpart; it is now driven from the decoder, from the
restore entry point, and from backend startup.
Also states the widened salt guard in the Local backend operations doc,
including the operator recovery path for an unrecognized record.
* fix(kms): say 'not a readable JSON object' when the marker probe cannot parse
The probe now fails on any input that is not a JSON object, not only on
malformed JSON, so the message must cover both.
* test(kms): assert the salt file before the error variant
The replacement salt is written before the error the guard reports, so the
file assertion is the one that fails on a regression.
* test(kms): guard the new backup error variant's display string
* docs: state that MinIO-encrypted objects are not readable by RustFS
Operators evaluating a MinIO migration had no warning that objects MinIO
wrote with SSE-S3, SSE-KMS, or SSE-C cannot be read back. The container
formats interoperate, so the limitation is easy to discover only after
the data has moved.
Document the limitation where a migration decision is actually made:
- minio-file-format-compat.md gains Part C, covering which object classes
transfer, the three seams that block each SSE mode with file:line
evidence, the reverse direction, and the current workarounds. It also
records that the `rio-v2` MinIO sealed-key parser does not close the
gap: the feature is absent from released artifacts, and the managed-SSE
detection gate is not feature-gated and returns before the parser runs.
- kms-backend-security.md gains an operator-facing warning next to the
backend comparison table, since configuring the static backend with
MinIO's key material looks like it should work and does not.
- s3-compatibility-matrix.md scopes its SSE row to RustFS's own
round-trip.
The read path treats an undetected MinIO-encrypted object as unencrypted
rather than failing, so all three notes tell operators to verify migrated
objects by content instead of by status code.
Refs rustfs/backlog#1638.
* docs: correct the failure mode for MinIO-encrypted objects
The read path does fail closed; the earlier text claimed ciphertext was
served as plaintext. MinIO's internal headers mark the object encrypted,
so the reader refuses when no material resolves. What is actually wrong
is the diagnosis: the refusal surfaces as a 500 InternalError.
docs(kms): correct the KV2 at-rest boundary and rotation rejection claims
The Vault KV2 section claimed the backend reports its confidentiality
boundary as `at_rest_protection: vault-kv2-acl` through `backend_info`.
That accessor, the `BackendInfo` type and its assertion test were removed
in rustfs/rustfs#5501, and no replacement reports the boundary. State
instead that the boundary is documented only, that `kms/status` exposes a
capability matrix covering supported operations rather than key-material
location, and that the backup manifest's `at_rest_protection` field is a
bundle declaration rather than a backend self-report.
The rotation section named `InvalidOperation` as the error Local and
Static return. Neither advertises `rotate`, so the product path falls to
the shared `KmsBackend` default and returns `UnsupportedCapability`;
`InvalidOperation` survives only in a test-only client helper.
* feat(kms): report configuration references that block a key deletion
Adds a KeyImpactReport that states which configuration still points at a
key, how exhaustively the sources were read, and which sources were not
consulted at all. The report deliberately carries no in-use or
safe-to-delete claim: it covers the configuration layer only, so an empty
reference list means nothing was found in the scanned sources, never that
the key is unreferenced.
Immediate deletion destroys key material without ever reaching the
deletion worker, so it never passed the worker's reference gate. The
manager now consults the same checker on that path and refuses with a
typed KeyStillReferenced error. This only ever adds a refusal; the
scheduled deletion path and the worker's blocking behaviour are
unchanged.
* test(kms): cover the immediate-deletion reference refusal
* feat(kms): surface configuration references on the admin key endpoints
DeleteKey and DescribeKey now return an impact section listing the
configuration that points at the key, so an operator scheduling a
deletion sees what will refuse to destroy the material instead of
learning it from a server-side log once the window has run out.
The section is reported, never acted on: scheduling still succeeds while
references exist, and the deletion worker's gate remains the only thing
that decides whether material is destroyed. An immediate deletion that
the manager refuses for an outstanding reference now answers 409.
* test(kms): pin the impact wire shape and the unreferenced force-delete path
* fix(kms): make the DescribeKey impact section opt-in
Collecting the section lists every bucket, and DescribeKey is polled, so
carrying that fan-out on the default read path trades a hot path's cost
for a diagnostic. It is now collected only for impact=true; without the
parameter the endpoint does exactly the work it did before and returns
no impact field.
A value that is neither true nor false is refused rather than read as
off, so a typo cannot answer a request for the section with a response
that merely lacks one. DeleteKey still reports unconditionally: that is
the request whose consequences the caller cannot otherwise see, and it
is not polled.
* fix(kms): box the query-parse refusal now that responses carry impact
The delete response grew an impact section, which pushed it past the
size clippy accepts inline in a Result. It is a full response body
rather than an error code, so it is boxed at the one place that returns
it as an error; the wire shape and the public field type are unchanged.
* feat(kms): accept the AWS backend through KMS configuration
The AWS KMS backend could be constructed but not selected: the admin
configure API had no AWS variant and startup rejected the backend name.
The configure request pins the region rather than defaulting it, because
that configuration is persisted once and replayed on every node: leaving
the region to each node's ambient provider chain would let nodes address
different regions, and therefore different keys, while reporting an
identical configuration. The request accepts no credential fields, so
credentials stay with the aws-config provider chain on each node, and
`deny_unknown_fields` refuses attempts to submit them anyway.
* test(kms): cover AWS backend selection through the service manager
An end-to-end check that an admin configure request selects the AWS
backend, builds a client, and passes the startup health check. Marked
#[ignore]: it needs real AWS credentials, though it creates no key and
is therefore not billable on its own.
fix(admin): retire the query-string form of immediate KMS key deletion
Immediate deletion destroys master key material outright, and every
object encrypted under that key becomes permanently unreadable. The
delete endpoint accepted that request as a query parameter, which is the
form most easily issued by accident and the one that made the waiting
window bypassable.
The query string can now only schedule a deletion: `force_immediate`
with any value other than `false`, or a `confirm_key_id` parameter, is
refused with 400 rather than downgraded to a scheduled deletion, so a
caller cannot read the answer as "destroyed". The JSON body form is
unchanged and remains the single way to reach the service gate that
enforces the server opt-in and the echoed confirmation.
Classify the route accordingly: `RouteRiskLevel` gains `Critical` for
routes whose worst case is permanent loss of user data, and the KMS key
deletion route is the only member, pinned in both directions by a matrix
test. Endpoint-level coverage for the 7-30 day window bound is added for
every configured backend.
Refs rustfs/backlog#1585 (part of rustfs/backlog#1562)
* feat(policy): add built-in KMS role policies
KMSKeyAdministrator, KMSKeyUser and KMSAuditor ship as canned identity
policies so operators can express KMS role separation without hand-writing
the resource grammar. They grant only kms actions, so they compose with an
existing data-plane policy, and none of them confers kms:Configure,
kms:ServiceControl, kms:ClearCache, kms:Backup or kms:Restore.
* docs(kms): document per-key KMS authorization and the role templates
* test(kms): add an end-to-end negative authorization matrix
Covers the admin and SSE-KMS planes for a wrong identity, a wrong key, a
wrong action and an explicit Deny, each preceded by a positive control so a
denial cannot be an unpropagated policy. SSE-S3 and unencrypted objects are
asserted to stay exempt.
* test(replication): pin the SSE-KMS contract with per-key authorization on
The replication worker carries no request identity, so it must stay exempt
from SSE-KMS key authorization. Running the existing contract with the
switch enabled makes a regression in that exemption visible here.
* docs(kms): describe KMS configuration convergence as implemented
* docs(kms): document the landed KMS metric families and narrow the gaps list
* docs(kms): state that no request field sets the cache metrics switch
* docs(kms): correct the describe_key cache divergence bound
* docs(kms): record the cryptographic compliance position
RustFS links no FIPS-validated cryptographic module: the rustls provider is
the ordinary aws-lc-rs build, and every data-path AEAD is RustCrypto. The
crypto crate's default-on `fips` feature only selects PBKDF2+AES-GCM over
Argon2id, with the same RustCrypto implementations behind both branches, so
it cannot support a validation claim either.
Document that status, the terminology rules for external material, the real
semantics of the `fips` feature with a rename direction, the cost of the
three routes to a stronger position, and the sequencing rules for retiring an
algorithm.
Refs rustfs/backlog#1587 (part of rustfs/backlog#1562)
* docs(kms): document the mixed-version cluster constraints
Collect the cross-version constraints that landed with versioned rotation and
the check-and-set lifecycle work: which persisted formats decode both ways,
which guarantees only hold once every node is upgraded, how long nodes can
disagree on lifecycle state, and that reconfigure is persisted cluster-wide
but applied only on the node that handled it.
Adds the recommended rolling-upgrade sequence and the list of operations to
avoid while two builds are running.
Refs rustfs/backlog#1581 (part of rustfs/backlog#1562)
* feat(kms): add AppRole configuration surface for Vault auth
Extend VaultAuthMethod::AppRole with secret_id_file (re-read on every
login so external rotation is picked up), a configurable auth mount
(default "approle"), and an optional fail-closed safety window. All new
fields are serde(default) so previously persisted configurations keep
deserializing, and the strict admin-configure deserializer accepts them
as optional.
Environment selection: setting RUSTFS_KMS_VAULT_APPROLE_ROLE_ID switches
both Vault backends to AppRole; the secret_id comes from
RUSTFS_KMS_VAULT_APPROLE_SECRET_ID_FILE (path stored, file wins) or
RUSTFS_KMS_VAULT_APPROLE_SECRET_ID, following the static secret-key file
precedent. validate() rejects AppRole configs without a role_id, without
any secret_id source, or with an empty mount.
Also append the CredentialsUnavailable error variant used by the
fail-closed credential gate.
* feat(kms): implement AppRole login with background renewal and fail-closed expiry
Implement the AppRoleLogin token source (vaultrs approle login +
renew-self) and wire lease-bound credentials through the provider:
- Each successful login/renewal installs a new client generation in the
ArcSwap; in-flight requests finish on the generation they captured.
- A background renewal task refreshes at half the lease TTL: renewable
tokens are renewed in place, everything else (or a failed renewal)
falls back to a fresh login. Auth exchanges run under the typed retry
policy (OpClass::Auth) and failed cycles retry on a fixed cadence, so
the provider recovers once Vault does.
- Fail-closed: current() refuses to hand out a token inside the
configured safety window of its expiry (default: one attempt timeout),
returning CredentialsUnavailable instead of sending a request whose
token may lapse mid-flight.
- Refreshes are single-flight: concurrent triggers for the same
generation coalesce into one login.
- The renewal task's owner handle lives on the KMS service version:
stop() shuts it down explicitly and reconfigure recycles it via
cancel-on-drop when the old version is discarded.
- The secret_id file is re-read on every login attempt; missing or empty
files fail the attempt without contacting Vault. Crate-owned copies of
tokens and secret_ids are zeroized on drop, and Debug output of every
credential-carrying type stays redacted (leak regression tests).
The renewal machinery is covered by paused-clock tests driving a
scripted token source: renew-at-half-TTL timing, login fallback,
fail-closed window entry and recovery, prompt task recycling, and
coalesced concurrent refreshes.
* feat(kms): add Vault Agent token file authentication
Add the TokenFile source: the token is read from an agent-managed sink
file (RUSTFS_KMS_VAULT_TOKEN_FILE or the TokenFile auth config) and
re-read once per poll interval (default 30s) through the existing
renewal loop, so a token rotated by the agent installs a new client
generation within one poll of the atomic replace. Each successful read
extends the token's observed validity to twice the poll interval; a
file that disappears or turns empty keeps failing the refresh until the
fail-closed window trips, and heals the provider as soon as it is
restored.
Reads are strict and never contact Vault on failure: the file must be
non-empty after trimming, and on Unix group/other permission bits are a
hard error (mirroring the SFTP host-key rule). Rotation detection uses
a content digest; the token itself is never stored on the source and
the crate-owned copy is zeroized.
Configuring the token file together with AppRole or an explicit static
token is rejected as a configuration error. All new config fields are
serde(default) and the strict admin-configure deserializer accepts the
new variant.
Covered by paused-clock tests (atomic replacement installs a new
generation next cycle, deletion fails closed and recovers, prompt task
recycling) plus negatives for missing/empty/over-permissive files and a
Debug leak regression.
* docs(kms): add Vault authentication and credential lifecycle runbook
Cover choosing between static token, AppRole, and Vault Agent token
file auth; AppRole role setup with SecretID delivery and rotation;
Agent sink deployment with the permission requirements; and the
fail-closed window semantics with a troubleshooting table keyed on the
renewal task's log lines.
* feat(kms): retain historical master key versions for Vault KV2 rotation
Rotation previously had to be rejected outright because replacing the
stored material would orphan every DEK wrapped by earlier versions.
Vault KV2 now keeps each version's material in an immutable, create-only
record at {prefix}/{key_id}/versions/{N} and treats the top-level record
as the current-version pointer plus fast-path material copy:
- decrypt resolves the envelope's master_key_version to its version
record; a missing version fails closed with KeyVersionNotFound and
never falls back to the current material
- envelopes without a version (pre-versioning writers) resolve to the
baseline_version frozen at the key's first rotation, so never-rotated
keys behave exactly as before
- generate_data_key stamps the wrapping version from the same key record
snapshot that supplied the material
- rotate_key commits in check-and-set order: freeze baseline, persist
the next version's material, then switch the current pointer; any
failure leaves the current pointer untouched, and concurrent rotations
serialize on the CAS writes with monotonically unique versions
- key listings drop the versions/ directory entries and physical key
deletion purges version records before the key record
Refs rustfs/backlog#1565
* test(kms): pin rotation contracts across backends and document retention
Completes the backlog#1565 series with the cross-backend regression net
and operator documentation:
- Vault Transit: ignored integration test proving version-prefixed
historical ciphertext still decrypts after rotation, with no
RustFS-side version bookkeeping in the envelope
- Local: offline mixed-format test interleaving pre-versioning and
versioned envelopes through a rejected rotation; the existing
rotation-rejection pinning test already covers material immutability
- docs: kms-backend-security.md gains the KV2 versioned retention
model, version record retention/destruction preconditions, and the
upgrade-before-first-rotation cluster constraint
Refs rustfs/backlog#1565
Zhengchao An