rustfs/docs/operations/kms-backend-security.md

KMS backend security properties

Use this when: choosing a KMS backend, scheduling or debugging master key rotation, planning a rolling upgrade of a cluster with KMS enabled, or auditing where master key material lives and who can read it. Source of truth: crates/kms/src/config.rs (KmsBackend, ENV_KMS_* constants), crates/kms/src/backends/{local,vault,vault_transit,aws}.rs, crates/kms/src/encryption/dek.rs (DataKeyEnvelope), rustfs/src/admin/route_policy.rs (KMS route actions).

RustFS ships several KMS backends. They differ not only in deployment effort but in where master key material lives and who can read it. Pick a backend based on the confidentiality boundary you need, not on the name alone. Related: Vault KMS authentication runbook (credential sources, refresh, fail-closed window), Cryptographic compliance positioning, Per-key KMS authorization, KMS admin API contract, KMS observability runbook.

Backend comparison

Backend	Config tag	Master key material location	At-rest protection of key material	Durability	Rotation	Intended use
Local	Local	Files under key_dir, encrypted with the configured local master key	Local master key (AES-GCM) + file permissions	Crash-durable commits on local filesystems only; see Local backend durability and deployment support matrix	Rejected by design (single material)	Development, testing and demos only; not supported for production
Static	Static	Provided out-of-band via environment/file; never persisted by RustFS	Operator-managed secret distribution	No state persisted by RustFS	Rejected (read-only backend)	Development and testing with an externally supplied key; not supported for production
Vault KV2	VaultKV2 (legacy alias Vault)	Stored directly in Vault KV v2 (Base64-encoded plaintext)	Vault ACLs + KV v2 at-rest encryption + TLS only	Delegated to Vault storage	Versioned retention (immutable per-version records + current pointer)	Deployments that accept Vault KV ACLs as the sole confidentiality boundary
Vault Transit	VaultTransit	Key-encryption keys never leave Vault; only Transit ciphertext is visible outside	Vault Transit engine (cryptographic isolation)	Delegated to Vault storage	Via Vault Transit key versioning	Deployments that need key material to be unreadable through storage APIs
AWS KMS	AWS (alias AwsKms)	Key material never leaves AWS KMS; RustFS mirrors no key state	AWS KMS (cryptographic isolation) + IAM	Delegated to AWS	On-demand RotateKeyOnDemand; prior backing keys stay usable for decryption	Deployments rooted in AWS IAM — read AWS KMS: deviations from the shared backend contract first

No KMS configured: the SSE-S3 local master key

A deployment that never configures a KMS can still serve SSE-S3 by setting RUSTFS_SSE_S3_MASTER_KEY to a base64-encoded 32-byte key. Data keys are then wrapped with that key using AES-256-GCM, on the node that serves the write. Understand three consequences before relying on it:

• The key is the whole confidentiality boundary. It lives in the process environment of every node, with no ACL, no audit trail and no policy engine in front of it.
• Objects written this way can never be rotated. There is no key record to rotate and no rewrap path; changing the value makes every object written under the old one unreadable. Migrating to a KMS later means rewriting those objects (for example with CopyObject), not reconfiguring.
• It does not serve SSE-KMS. A request for x-amz-server-side-encryption: aws:kms on a node with no running KMS is refused — 400 InvalidRequest when KMS was never configured, 503 when a configured service is not running. Earlier releases silently wrapped the data key with the local master key while still stamping aws:kms and the requested key id into the object metadata; that metadata claimed a KMS protection the object never had. If a deployment depended on that, either configure a KMS or ask for AES256.

The value is unset by default, and a deployment that neither configures a KMS nor sets it simply cannot serve SSE-S3 (the write is refused, never silently downgraded to plaintext).

Migrating from MinIO: encrypted objects do not carry over

Warning: default RustFS builds fail closed on objects that MinIO encrypted. This applies to SSE-S3, SSE-KMS, and SSE-C, whichever KMS backend you configure; configuring Static with MinIO's key material does not make them readable. Such objects list and HEAD normally (their xl.meta parses), and only the payload read fails — with S3 InvalidObjectState, never plaintext. Read a sample of encrypted objects, not just their listings, before decommissioning the MinIO deployment.

The read path exists behind the rio-v2 feature as a migration-only build, the reverse direction (RustFS-written SSE objects read by MinIO) is unsupported, and the migration options are enumerated in MinIO file-format interoperability, Part C. The AWS KMS and MinIO KES wire protocols are non-targets of that document and of this one.

Vault KV2: what the backend does and does not do

The Vault KV2 backend uses Vault purely as a secure storage service:

• Master key material is generated by RustFS and written to KV v2 as a Base64-encoded value (encrypted_key_material is an encoding, not a ciphertext).
• The backend never calls the Vault Transit engine. The mount_path configuration field and the RUSTFS_KMS_VAULT_MOUNT_PATH environment variable are deprecated leftovers: accepted for compatibility and ignored.
• Data-encryption keys (DEKs) handed to the object-encryption path are still wrapped with AES-256-GCM under the master key; the statement above concerns the master key's storage in Vault, not the DEK envelope.
• No runtime interface reports this boundary. GET /rustfs/admin/v3/kms/status names the active backend (backend_type: vault-kv2) and returns a capabilities matrix, but that matrix enumerates only supported operations. Determining the confidentiality boundary in force means reading backend_type and applying the comparison table above.
• The at_rest_protection: storage-only field carried by a KMS backup manifest declares the protection state of key material inside a backup bundle, not a property the running backend reports about itself.
• Key rotation retains every historical master key version as an immutable record under {prefix}/{key_id}/versions/{N} and only then moves the current-version pointer; see Master key rotation.

Warning: KV read access is equivalent to holding the master keys. Any Vault identity (token, AppRole, or policy) that can read the RustFS key path in KV v2 can recover the plaintext master key material and decrypt every object protected by those keys. If this is not acceptable, use the Vault Transit backend instead.

Minimal Vault policy for the KV2 backend

Scope the RustFS token/AppRole to exactly the KV v2 mount and key prefix it is configured with (defaults shown: mount secret, prefix rustfs/kms/keys), and grant no other identity read access to that subtree:

# RustFS KMS (Vault KV2 backend) — key storage only, no Transit access needed.
path "secret/data/rustfs/kms/keys/*" {
  capabilities = ["create", "read", "update"]
}

path "secret/metadata/rustfs/kms/keys/*" {
  capabilities = ["list", "read", "delete"]
}

• The trailing wildcards also cover the per-version material records under .../keys/{key_id}/versions/{N}; no extra policy paths are needed.
• delete on the metadata path is required only for permanent key deletion (force_immediate); drop it if you never hard-delete keys. RustFS refuses force_immediate unless the server sets RUSTFS_KMS_ALLOW_IMMEDIATE_DELETION=true, so leaving that gate off keeps the capability unreachable whatever the Vault policy allows.
• Do not attach sudo, wildcard mounts, or Transit paths; the KV2 backend does not use them.
• Audit KV reads on the key prefix: every read event is a potential master-key disclosure.

Master key rotation: retention, destruction, and upgrade ordering

Rotation is reachable through POST /rustfs/admin/v3/kms/keys/rotate (kms:RotateKey, high risk); it is not exposed on the S3 surface. Local and Static advertise no rotate capability (capabilities.rotate is false in the kms/status response) and reject rotation with UnsupportedCapability. Vault Transit delegates rotation to the Transit engine's own versioning (ciphertext is version-prefixed, e.g. vault:v1:...). Vault KV2 rotates by retaining every historical version, as described below.

Rotation drivers and scheduling, per backend

The rotate endpoint is one API over three different mechanisms, and which component performs the rotation decides how periodic rotation must be scheduled.

Backend	Can rotate	Who performs the rotation	How to schedule periodic rotation	Wrap ceiling
Local	No	Nobody — the rotate endpoint is refused with UnsupportedCapability	Cannot be scheduled; migrating to a rotating backend is the only path	Unmitigable
Static	No	Nobody — same refusal; the material is supplied out-of-band and read-only	Cannot be scheduled; migrate	Unmitigable
Vault KV2	Yes	RustFS owns the protocol: freeze the outgoing material as an immutable version record, persist the new material, move the current pointer with a check-and-set write	Exactly one external scheduler (cron, Kubernetes CronJob) calling the rotate endpoint with credentials scoped to kms:RotateKey	Reset by each rotation
Vault Transit	Yes	Vault's Transit engine; RustFS forwards the call and records the version bump	Vault's native auto_rotate_period on the Transit key. Do not also drive the RustFS endpoint: two owners of the version cadence means neither configured period holds. The reported key version advances only through RustFS, so on an auto-rotating key treat it as a floor	Not applicable (wraps inside Vault)
AWS KMS	Yes	AWS — the endpoint maps to RotateKeyOnDemand	AWS's native automatic rotation. Do not drive periodic rotation through the RustFS endpoint: AWS caps lifetime on-demand rotations, so a scheduler exhausts the quota and then fails forever. Keep the endpoint for incidents. RustFS neither enables nor observes AWS automatic rotation and records no rotation timestamp, so its rotation-age signals measure key age on this backend	Not applicable (wraps inside AWS)

Wrap ceiling. Where RustFS wraps DEKs locally (Local, Static, Vault KV2) every DEK is wrapped with AES-256-GCM under the master key using a random 96-bit nonce, and NIST SP 800-38D caps AES-GCM at 2^32 invocations per key under random nonces. Each encrypted-object write is one wrap, so the count tracks lifetime encrypted writes. Rotation installs fresh material and restarts the count; on Local and Static there is no rotation, so the ceiling can only be escaped by migrating.

Why there is no built-in rotation timer (KV2). A timer inside the server cannot verify the upgrade-before-first-rotation constraint, and rotation is not idempotent: without leader election, N nodes on the same schedule would advance the key version N times per period. Verify that your scheduler keeps up with the rotation readiness fields and the KmsKeyRotationOverdue alert in the KMS observability runbook.

Pre-rotation checklist (before the first rotation of any key, and before enabling any schedule):

1. Every node runs a build that understands the master_key_version envelope field — the hard constraint below. A timer cannot check this; you must.
2. No rolling upgrade is in progress — see Do not do these during a mixed-version window.
3. The retention and destruction preconditions are understood: every version record a stored DEK envelope references must remain readable, and no retention tooling prunes the version subtree.
4. For KV2, exactly one scheduler exists.
5. RUSTFS_KMS_ROTATION_MAX_AGE_SECS is set to the rotation period your policy requires, so the per-key rotation_due verdict and the rotation-age alert verify the schedule instead of assuming it.

Rotation readiness: reported, never acted on

RustFS reports which keys have outlived a period you configure; nothing consults the verdict before encrypting or decrypting, and it has no effect on readiness or liveness.

Setting	Meaning	Unset or unparsable	Floor
RUSTFS_KMS_ROTATION_MAX_AGE_SECS	Rotation period in whole seconds; keys older than this report rotation_due with reason age or never_rotated	No age verdict is reported (a warning is logged for an unparsable value) — how often keys must rotate is a compliance decision, not a built-in default	1 hour
RUSTFS_KMS_ROTATION_MAX_WRAPS	Data keys one key's material may wrap before rotation_due with reason wraps	No wrap verdict is reported	1,000,000 (wraps are accounted in reserved blocks of that size)

GET /rustfs/admin/v3/kms/keys carries rotation_due and rotation_due_reason per key: age (rotated, but longer ago than the period), never_rotated (in use longer than the period, never rotated), wraps (wrap budget exceeded; wins over age when both hold, because the AES-GCM ceiling is not negotiable), or unsupported (the backend cannot rotate). Only backends where RustFS wraps locally and can rotate count wraps (Vault KV2); Transit and AWS report no count. GET /rustfs/admin/v3/kms/keys/{key_id} does not carry these fields — its response records a creation date but no rotation timestamp, so read the verdict from the listing.

Vault KV2 versioned retention model

Each rotation writes the new version's material to {prefix}/{key_id}/versions/{N} as an immutable, create-only record, and only after that material is durably persisted does a check-and-set write move the top-level record (the current-version pointer, which also mirrors the current material as a fast path). The first rotation additionally freezes the pre-rotation material as a version record and pins it as the key's baseline_version; DEK envelopes written before versioning existed (no master_key_version field) always resolve to that baseline, never to whatever version is current.

Decryption loads exactly the version recorded in the envelope and fails closed with a typed KeyVersionNotFound error when that version's record is missing. There is deliberately no fallback to the current material: falling back would silently feed the wrong key to AEAD and mask tampered envelopes.

Retention and destruction preconditions

• Every version record that any stored DEK envelope references must remain readable. The bulk rekey sweep (POST /rustfs/admin/v3/kms/keys/rekey, kms:Rekey) rewraps stored envelopes onto the current version; until a sweep has completed with zero failures after the last rotation, assume every version of a rotated key is referenced. A completed sweep is evidence, not authority — the deletion gate stays the decision point. Replication strips encryption metadata in transit, so each replica site runs its own sweep.
• Version records are ordinary KV v2 secrets under the key subtree. Never run kv metadata delete or kv destroy against {prefix}/{key_id}/versions/*, and do not apply delete-version-after or retention tooling to that subtree. Each version record has a single KV revision, so KV max-versions settings neither protect nor endanger history — but metadata deletion removes a record entirely.
• Permanent key deletion through RustFS (force_immediate after PendingDeletion) purges the key's version records together with the key record; that is the only supported way to remove them. It requires RUSTFS_KMS_ALLOW_IMMEDIATE_DELETION=true on the server and a DELETE with a JSON body that sets force_immediate and echoes the key id as confirm_key_id; the query-parameter form is refused outright. Leave the gate off except while actively destroying keys — the pending-deletion window plus CancelKeyDeletion is the only recovery path for objects encrypted under the key.
• force_immediate is refused with 409 Conflict while any bucket's default encryption configuration names the key or the key is the KMS service default key. A scheduled deletion is not refused for that reason: it destroys nothing and stays cancellable, and the background sweep re-checks the same references before destroying material.
• For Vault Transit, retention is governed by the Transit key's min_decryption_version: never raise it above the oldest version that may still protect live ciphertext.

Reading the impact section

DeleteKey responses always carry an impact section listing the configuration that points at the key (buckets whose default encryption names it, and whether it is the service default key). DescribeKey (GET /rustfs/admin/v3/kms/keys/{key_id}) returns the same section only when asked with impact=true, because collecting it lists every bucket; a value other than true/false is rejected with 400. An absent section means "not collected", never "nothing references this key".

coverage.scanned names the sources that were read and coverage.not_scanned the ones that were not — which currently includes every object encrypted under the key. completeness is exact only over the scanned sources and unavailable when a source could not be read; both an unreadable source and an outstanding reference stop the sweep from destroying material. An empty references list does not mean the key is unused: no object metadata is consulted, so a key with no configuration references can still protect live data.

Upgrade before first rotation (hard constraint)

Do not rotate any key until every RustFS node runs a build that understands the master_key_version envelope field. Older binaries ignore the field and always decrypt with the current material: harmless while nothing has been rotated, but after a rotation they fail to decrypt every object wrapped by an earlier key version. Complete the rolling upgrade of the entire cluster first, then rotate. The rest of this constraint class is collected in Mixed-version clusters during a rolling upgrade.

SSE-C requires a secure transport

An SSE-C request carries the customer's AES key in a request header, so AWS S3 and MinIO both refuse one that did not arrive over TLS. A plaintext hop hands that key to anyone on the path, and because the object cannot be read without the same key, the exposure lasts as long as the object does.

This release reports rather than refuses, because flipping straight to a rejection would break every plaintext staging and test deployment inside a release window:

• Every SSE-C request on a plaintext transport increments rustfs_ssec_plaintext_requests_total and logs one ssec_request_without_tls warning per process.
• RUSTFS_SSE_C_REQUIRE_TLS=true (default false) refuses those requests now, with the same 400 InvalidRequest wording AWS uses. Confirm the counter reads zero before enabling it.
• The default is expected to flip in a later release.

The verdict is per connection: a listener that terminates TLS satisfies it, and so does an https protocol forwarded by a proxy the trusted-proxy configuration accepts. A direct plaintext client asserts nothing, and a forwarded protocol from an untrusted peer is not consulted.

Object ciphertext format: what the v1 frame layout does and does not authenticate

Every object RustFS writes today uses the v1 frame layout (the v2 layout exists and is read automatically, but its write switch RUSTFS_ENCRYPTION_FRAME_V2 is off by default). Each frame is authenticated with AES-256-GCM under a nonce derived from the object's base nonce and the frame's index. Three properties do not follow from that, and an operator's threat model has to account for them:

• No frame-index binding. A frame's index is not part of its associated data. Authentication proves a frame was produced under this object's key; it does not by itself prove the frame belongs at the position it occupies.
• No final-frame authentication. Nothing in a v1 stream marks the last frame, so a stream that has been cut short is not distinguishable from a shorter object by cryptographic means.
• Truncation is not detected server-side. A full GET is cut off by the length gate mid-stream and surfaces as IncompleteBody — after the response headers have already gone out. A ranged GET that ends early looks like an ordinary EOF and is not reported at all. A client that needs a truncation signal must compare the delivered length against Content-Length itself.

These matter only to an attacker who can already rewrite the underlying shards. Shard integrity uses a keyed-hash-free checksum (HighwayHash), which such an attacker can recompute, so it is not a barrier.

Two historical shapes additionally reuse a GCM nonce and cannot be repaired by any read-side change:

Shape	Written by	Consequence	Migration
Multipart objects written before 1.0.0-alpha.91	The pre-alpha.91 writer reused a segment's part nonce for every block in it	The whole segment shares one nonce; a frame from index zero authenticates anywhere in that segment	Rewrite in place with CopyObject; then set RUSTFS_ENCRYPTION_LEGACY_NONCE_FALLBACK=false
SSE-C objects written before 1.0.0-beta.9 that carry no stored IV	The nonce was derived deterministically from bucket and key	Historical versions of the same key share a nonce, which affects confidentiality as well as forgeability	Rewrite in place with CopyObject

The decrypt reader locks each segment to whichever nonce layout decoded its first non-zero-index frame, so a replayed frame is rejected as soon as any later frame disagrees. A stream that is nothing but repeats of frame zero has no later frame to disagree, so a deployment that holds no pre-alpha.91 objects should set RUSTFS_ENCRYPTION_LEGACY_NONCE_FALLBACK=false (default true) to drop that layout entirely. Turning it off refuses to decrypt pre-alpha.91 objects, so migrate first.

Mixed-version clusters during a rolling upgrade

During a rolling upgrade KMS state is shared three ways: Vault holds key records and Transit metadata, cluster storage holds the persisted KMS configuration, and each node's process memory holds caches and the live backend instance. Nodes on different builds agree on the first, may disagree on the third, and can disagree on configuration for as long as the operator leaves them running, because the reload broadcast that converges configuration is one of the things an older build rejects. This section is written for the KV2 and Transit backends; the Local backend is unsupported for multi-node deployments regardless of version (see the deployment support matrix).

Persisted formats are backward compatible in both directions

No coordinated format cutover is required; the compatibility is deliberate and covered by decode tests.

Record	Compatibility mechanism	Caveat
DEK envelopes	DataKeyEnvelope::master_key_version is optional and omitted when absent, so envelopes from non-rotating backends stay byte-identical to the historical seven-field JSON. An upgraded node resolves a pre-versioning envelope to the key's baseline_version, or to the current version for a never-rotated key. Unknown fields are skipped; a bounded field-name sample is logged at a rate-limited warn and counted by rustfs_kms_persisted_unknown_fields_total{record_kind="data-key-envelope"}	An older binary reading a new envelope ignores the version field and decrypts with the current material — the rotation constraint above
Local key records	Each <key_id>.key carries format_version: 1; records without it default to 1. A reader accepts a version at most the one it understands and rejects newer with UnsupportedFormatVersion before decrypting. Unknown fields are accepted and counted by rustfs_kms_persisted_unknown_fields_total{record_kind="local-key-record"}	Once a future version greater than 1 has written a record, do not roll back to a build predating the marker
KV2 key records	baseline_version is read with a serde default; None means "never rotated"	An old build drops the field on write-back (see below)
Transit metadata records	Decode on either build	—

DEK envelope context binding (RUSTFS_KMS_ENVELOPE_AAD)

Historically the KV2 and Local backends sealed only the DEK plaintext; the encryption_context rode in the envelope unauthenticated and was checked by field comparison alone, so a party able to rewrite the stored envelope could rewrite the context. With RUSTFS_KMS_ENVELOPE_AAD=true, newly wrapped envelopes bind the canonical context bytes as AES-GCM additional data and carry context_binding: 1; rewriting the stored context, or stripping the flag, then fails authentication. Static, Vault Transit and AWS already bound the context through their own mechanisms and are unaffected.

Rollout constraint: reading bound envelopes needs no switch, but a node that predates the field cannot open them — its unwrap runs without the additional data and fails authentication. The switch defaults off (ENV_KMS_ENVELOPE_AAD in crates/kms/src/config.rs); enable it only after every node runs a release that understands context_binding, mirroring the RUSTFS_ENCRYPTION_FRAME_V2 rollout. With the switch on, a rewrap sweep upgrades unbound envelopes to the bound format (converging to zero writes on re-run); a bound envelope never regresses to the unbound shape, and an envelope carrying an unrecognized context_binding value is refused rather than decrypted without its binding.

Guarantees that hold only once every node is upgraded

These are properties of builds from 1.0.0-rc.1 onward; a single older node removes them for the whole cluster.

Guarantee	Upgraded behaviour	What an old node does
Check-and-set lifecycle writes	Every KV2 lifecycle mutation (create, enable, disable, tag, schedule/cancel deletion) and every Transit metadata write is a versioned read followed by a check-and-set write, retried on conflict	Writes blind, so it can overwrite a check-and-set commit without any conflict being reported — the lost update the change eliminated
baseline_version survives write-back	Preserved	Reads the record without error and drops baseline_version on any write-back; the key then resolves pre-versioning envelopes to the current version, which after a rotation is the wrong material
wrap_budget_reserved only overestimates	The KV2 record's wrap counter (behind rustfs_kms_max_key_wrap_operations) is reserved in blocks and never understates	Drops the field on write-back, regressing the count toward zero. Nothing breaks — the counter is advisory and the next reservation re-establishes a floor — but do not trust a low reading taken during or shortly after a mixed-version window
Version-record awareness	Version records under {prefix}/{key_id}/versions/{N} are create-only (check-and-set of 0), so two nodes racing a version number produce exactly one creator	Never reads or writes the sub-path, and its key listing reports the KV2 directory entry (my-key/) as though it were a key

Windows in which nodes can legitimately disagree

Even with every node on the same build, some state is process-local. These windows are bounded by design, except the last one.

What can diverge	Bound	Mechanism
Transit key lifecycle state used by the encrypt and generate_data_key gates	≤ METADATA_CACHE_TTL (300 s, not tunable — this cache gates cryptographic operations)	Per-node in-process Transit metadata cache, TTL- and capacity-bounded, with targeted invalidation when a data-path call reports the key gone server-side. Builds older than 1.0.0-rc.1 held this cache with no TTL and no capacity bound: on such a node the window is "until the process restarts"
describe_key output	One metadata cache TTL: cache_ttl_seconds from the KMS configure request, 300 s when omitted, clamped down to 24 h at use (clamped rather than rejected; zero is refused while caching is enabled)	Manager-level key metadata cache; a reporting cache the KV2 state gates never read. kms service-status and the configuration endpoint report the effective post-clamp value. Builds older than 1.0.0-rc.1 ignored cache_ttl_seconds and ran 300 s while persisting 3600 s as the default, so a cluster configured before then widens to the stored 3600 s on upgrade — read the reported value back
KV2 key lifecycle state	None	The KV2 backend re-reads the key record from Vault for every lifecycle and data-key operation
Active KMS configuration	One best-effort reload broadcast; unbounded for any peer that did not apply it	See below

Configuration changes converge through a best-effort peer reload

POST /rustfs/admin/v3/kms/configure and /kms/reconfigure persist the new configuration to cluster storage at config/kms_config.json, switch the KMS service on the node that handled the request, and broadcast a reload signal once to every peer. A peer that accepts the signal re-reads the persisted configuration and reconfigures itself; a peer already running that configuration treats it as a no-op. Convergence is best effort and the request never fails on account of a peer:

• There is no background retry. A peer that is unreachable, whose build predates the KMS subsystem, or whose reload fails keeps its previous configuration until a later reconfigure reaches it or it restarts. For those peers the split is unbounded.
• The admin response reports success either way, but its message names every peer that did not converge, and the server logs one kms_peer_config_reload_failed warning per peer.
• While a split lasts, both configurations are live: if the change switched backends, or changed the Vault mount or key prefix, nodes write new key material to different places and a key created through one node is invisible to the others.

GET /rustfs/admin/v3/kms/service-status returns a cluster_config object holding one redacted configuration fingerprint per node plus a consistent flag, true only when every node answered with the same fingerprint (an unreachable peer, a build reporting no fingerprint, and an unconfigured node each read as divergent). Secrets are substituted out before fingerprinting, so the field detects a configuration split, not a credential split. Treat a configure or reconfigure whose response names unconverged peers as unfinished: re-issue it once those peers are reachable, or restart them.

Recommended rolling upgrade order

Follow the node-at-a-time procedure in the multi-node restart runbook; this adds the KMS-specific sequencing.

1. Freeze KMS administrative traffic for the duration: no key creation, enable, disable, tagging, schedule-deletion, cancel-deletion, rotation, or reconfiguration. Object read and write traffic continues normally.
2. Upgrade one node at a time, waiting for each to report ready before starting the next.
3. Verify no node is left behind before unfreezing. A single old node reintroduces blind writes and strips baseline_version on its next lifecycle write.
4. Resume administrative traffic.
5. Only then perform the first rotation of any key.
6. If the KMS configuration was changed at any point, confirm cluster_config.consistent is true in the service-status response, and re-issue the change — or restart the node — for every peer still reporting a different fingerprint.

Do not do these during a mixed-version window

• Rotate any key. Unrecoverable for objects an old node must read.
• Issue any KV2 lifecycle write to an old node. Its blind write can clobber a concurrent check-and-set commit and drops baseline_version.
• Create the same key ID from two nodes. The create path is create-only on upgraded builds, but an old node's blind write does not honor that: the later writer's material wins and every DEK wrapped with the earlier material becomes permanently unwrappable.
• Assume a disable or schedule-deletion took effect cluster-wide. Old Transit nodes cache lifecycle state without expiry; confirm per node, or restart the old nodes.
• Reconfigure the KMS backend and consider it done. The reload broadcast is exactly what an old build rejects; check the response message and cluster_config.consistent.
• Delete or prune version records under {prefix}/{key_id}/versions/* for any reason. This is never safe, mixed-version or not; see Retention and destruction preconditions.

Choosing between Vault KV2 and Vault Transit

Use Vault Transit (VaultTransit) when key material must be cryptographically isolated from anyone holding storage-level read access: Transit keeps key-encryption keys inside Vault, only ever returns ciphertext, and supports server-side key versioning and rotation. Use Vault KV2 only when you accept that the Vault ACL on the key path is the confidentiality boundary and want the operational simplicity of a single KV mount.

AWS KMS: deviations from the shared backend contract

Select it with RUSTFS_KMS_BACKEND=aws. Credentials and region resolution are delegated entirely to the standard aws-config provider chain (environment, shared profile, container/IMDS role), so RustFS never stores, persists, or redacts AWS credential material. Only two non-credential settings are read: RUSTFS_KMS_AWS_REGION and RUSTFS_KMS_AWS_ENDPOINT_URL. A plaintext (http://) endpoint override would expose every KMS request including plaintext data keys, so it is refused unless the development opt-in is set.

AWS owns key state, backing-key rotation, and the deletion window, and this backend mirrors none of it locally. Four behaviours therefore differ from every RustFS-managed backend:

Behaviour	RustFS-managed backends	AWS KMS backend
Decryption with a Disabled or PendingDeletion key	Kept working, so disabling a key never breaks reads of objects already encrypted under it	Refused by AWS. Objects encrypted under a key that is later disabled become unreadable until it is re-enabled
Key deletion	Physical deletion available	No physical delete. ScheduleKeyDeletion is the only removal path; AWS destroys the material when the 7-30 day window elapses. force_immediate is refused
Cancelling a scheduled deletion	Key returns to Enabled	Key is left Disabled; enable it explicitly
Creating a key under a caller-chosen name	The requested name becomes the key id	Refused. AWS assigns identifiers and this backend does not manage aliases

Consequences of the last row: SSE-S3 key auto-creation and the synthetic KMS probe are unavailable on this backend, because both address a key by a name they choose. Pre-create keys in AWS and reference them by AWS key id or ARN.

The AWS backend is exempt from backends::contract_tests::assert_state_machine_contract, whose assumptions (disabled keys still decrypt, cancel returns to Enabled, caller-assigned names) AWS violates. The exemption is pinned by the offline aws_backend_shared_contract_exemption_is_pinned test in crates/kms/src/backends/aws.rs; if AWS changes any of these semantics, integrate the backend into the shared driver and remove the exemption rather than weakening the shared assertions.

Key versions are opaque: RustFS reports key_version as 1 and cannot enumerate versions. Rotation uses RotateKeyOnDemand, which retains prior backing keys for decryption; AWS's automatic yearly rotation is neither enabled nor reported on by RustFS.

The KMS admin API accepts the backend as "backend_type": "AWS" (aliases aws, aws-kms, aws_kms, AwsKms) on /v3/kms/configure and /v3/kms/reconfigure. The body carries region (required), and optionally endpoint_url, default_key_id, and the shared timeout/retry/cache settings; credential fields are rejected as unknown, because every node resolves credentials through its own provider chain. region is mandatory even though RUSTFS_KMS_AWS_REGION is optional at startup: the admin configuration is replayed on every node, and a request that left the region to each node's ambient chain would let nodes address different regions — and therefore different keys — under an identical configuration. default_key_id must be an AWS key id or ARN that already exists.

Vault TLS: custom CA and mutual TLS

Both Vault backends (KV2 and Transit) support a private certificate authority and client-certificate (mTLS) authentication at the connection layer:

Setting	Environment variable	Admin configure field	Meaning
CA bundle	RUSTFS_KMS_VAULT_CA_CERT	ca_cert_path	Path to a PEM CA bundle; when set, only this bundle is trusted
Client certificate	RUSTFS_KMS_VAULT_CLIENT_CERT	client_cert_path	Path to a PEM client certificate presented to Vault; requires the client key
Client key	RUSTFS_KMS_VAULT_CLIENT_KEY	client_key_path	Path to the PEM private key matching the client certificate
Skip verification	RUSTFS_KMS_VAULT_SKIP_TLS_VERIFY	skip_tls_verify	Disables server certificate verification; gated on the insecure development defaults opt-in

Paths are read on the node applying the configuration, so the files must exist at the same path on every node. Certificate and key must be configured together; the files are read and parsed when the backend starts, so a bad path or malformed PEM fails the configuration rather than a later request. The kms/status backend summary reports has_custom_ca and has_client_identity booleans, never file contents. RustFS always sets the trust roots and identity explicitly — to the configured values or to empty — so the VAULT_CACERT, VAULT_CAPATH, VAULT_CLIENT_CERT and VAULT_CLIENT_KEY process environment variables cannot splice TLS material into the connection behind the KMS configuration.

Local backend durability and deployment support matrix

The Local backend stores one JSON record per key (<key_id>.key) plus an Argon2id salt file (.master-key.salt) inside the configured key_dir. For where the key material lives and who can read it, see the backend comparison.

Positioning

• Local is the default backend (kms_backend defaults to local) and is a development, testing and demo backend; it is not supported for production. Activating a backend whose capabilities report production_supported: false logs a kms_backend_positioning warning on every start, restart and reconfigure, and the kms/status capability matrix carries the same flag. The positioning is a warning, not a gate.
• Configuration validation enforces stricter rules outside explicit development mode: a master key is required and key_dir must not live under the process temp directory.
• The RustFS Kubernetes operator places the key directory on a PersistentVolumeClaim, so keys survive pod rescheduling.
• A multi-node deployment cannot share it. Key material lives on each node's own disk and the Argon2id salt is generated per node, so two nodes derive different keys from the same master_key. An object encrypted on node A cannot be decrypted on node B; behind a load balancer that appears as intermittent 500s on reads that succeeded a moment earlier. Configuring Local while the deployment is distributed logs kms_node_local_backend_in_distributed_deployment and appends the same warning to the kms/configure response. This stays a warning, not a gate.
• Production multi-node deployments should use the Vault Transit backend.

Deployment support matrix

Deployment	Supported	Notes
Local filesystem (ext4, XFS, APFS, ...)	Yes	The commit protocol relies on POSIX rename/hard_link atomicity and fsync durability
Kubernetes PVC	Yes	Only when the PersistentVolume is backed by a local or block filesystem; this is how the RustFS operator provisions the key directory
NFS or other shared/network filesystems	No	Network filesystems do not reliably provide the atomicity and fsync semantics the protocol depends on; an NFS-backed PersistentVolume is this case
Multiple RustFS processes sharing one key_dir	No	Concurrent key creation is linearized (hard_link refuses to clobber), but every other write is a read-modify-write with no cross-process lock, so concurrent writers can silently lose updates

Within a single process, per-key write locks serialize read-modify-write updates.

Crash recovery behavior

Every mutation of the key directory uses a durable commit protocol:

1. A temp file (<name>.tmp-<uuid>) is created exclusively in key_dir.
2. The content is written and fsynced (sync_all).
3. The file is published atomically: rename to replace an existing file, hard_link to create a new one without clobbering.
4. The parent directory is fsynced so the new directory entry is durable.

Deletion mirrors the tail of the protocol (remove_file followed by a parent directory fsync). A crash at any step leaves either the complete old state or the complete new state, plus at most an unpublished temp file. On startup the backend:

• Removes orphaned commit temp files. The matcher is strict (<prefix>.tmp-<uuid>, never anything ending in .key), so published key files are never touched.
• Validates every published .key file. A record that fails to decode fails startup rather than being silently skipped.
• Guards the salt file. If .master-key.salt is missing but the directory contains keys marked encrypted-master-key, initialization fails closed naming the salt path. A regenerated salt derives a different master key and can never decrypt those keys, so the recovery is to restore the salt file (or the whole directory) from backup, never to let a fresh salt be generated. The guard is equally strict about a record it cannot read or interpret (for example one written by a newer RustFS naming an at-rest protection this build does not implement): no replacement salt is generated for such a directory either. An empty directory, or a legacy directory predating the salt file, initializes normally.

Filesystem permissions and the boundaries the protocol assumes

• The key directory is held at 0o700 and every file published into it — key records, the salt, restore staging — is written owner-only. The requested mode is applied and re-read on the open file before the content becomes durable, so the process umask cannot widen it.
• A directory wider than 0o700 is narrowed on every start (and re-read to confirm), not refused: kubelet creates emptyDir at 0o777, several PVC provisioners mkdir -m 0777, and a --tmpfs mount lands at 1777. Only a directory this process cannot secure is fatal. Narrowing is logged with the previous mode whenever it was reachable beyond the owner.
• Publishing never writes through a symlink: hard_link refuses any existing destination (including a dangling symlink) and rename replaces the link itself. Startup removes anything wearing a commit-temp name that is not a directory, symlinks included; the protocol only ever creates temps with create_new, so such an entry is either its own leftover or something planted.

Two boundaries are not verified, and deployments should not assume them:

• Cross-device operations. The temp file is always created in the destination's own directory, so rename and hard_link never cross a filesystem; that invariant is tested, a real cross-device attempt is not. The restore staging directory is always .restore-staging inside key_dir — do not bind-mount it onto another filesystem.
• The key directory being replaced mid-commit. Paths are re-resolved from the directory name rather than held as a directory descriptor. If key_dir is swapped between the rename and the parent fsync, the fsync lands on the replacement and the call still reports success. Reaching this requires write access to the key directory's parent, which nothing checks — keep the parent owner-writable too. Closing it properly means moving to renameat/linkat against a held descriptor; it is a recorded gap.

Backing up the key directory

Back up key_dir as a whole, including the hidden .master-key.salt file. A key file on its own is not restorable: decrypting it requires the master key derived from the configured master_key and the persisted salt. Restoring a partial directory leaves the backend unable to decrypt, and the salt guard will (correctly) refuse to start. Losing the salt file with no backup means every key encrypted under it is unrecoverable. The rehearsal procedure is in the KMS disaster-recovery drill.