rustfs/docs/operations/kms-backend-security.md

KMS backend security properties

RustFS ships several KMS backends. They differ not only in deployment effort but in where master key material lives and who can read it. Pick a backend based on the confidentiality boundary you need, not on the name alone.

For how the Vault backends authenticate (static token, AppRole, Vault Agent token file) and how credential refresh and the fail-closed window behave, see the Vault KMS authentication runbook. For what may be claimed about the cryptographic implementations themselves, see Cryptographic compliance positioning.

Backend comparison

Backend	Config tag	Master key material location	At-rest protection of key material	Durability	Rotation	Intended use
Local	Local	Files under key_dir, encrypted with the configured local master key	Local master key (AES-GCM) + file permissions	Crash-durable commits on local filesystems only; see Local backend durability and deployment support matrix	Rejected by design (single material, development backend)	Development; single-node setups that accept host-level trust
Static	Static	Provided out-of-band via environment/file; never persisted by RustFS	Operator-managed secret distribution	No state persisted by RustFS	Rejected (read-only backend)	Simple deployments with an external secret manager
Vault KV2	VaultKV2 (legacy alias Vault)	Stored directly in Vault KV v2 (Base64-encoded plaintext)	Vault ACLs + KV v2 at-rest encryption + TLS only	Delegated to Vault storage	Versioned retention (immutable per-version records + current pointer)	Deployments that accept Vault KV ACLs as the sole confidentiality boundary
Vault Transit	VaultTransit	Key-encryption keys never leave Vault; only Transit ciphertext is visible outside	Vault Transit engine (cryptographic isolation)	Delegated to Vault storage	Via Vault Transit key versioning	Deployments that need key material to be unreadable through storage APIs

Vault KV2: what the backend does and does not do

The Vault KV2 backend uses Vault purely as a secure storage service:

• Master key material is generated by RustFS and written to KV v2 as a Base64-encoded value (encrypted_key_material is an encoding, not a ciphertext).
• The backend never calls the Vault Transit engine. The mount_path configuration field and the RUSTFS_KMS_VAULT_MOUNT_PATH environment variable are deprecated leftovers: they are accepted for compatibility and ignored.
• Data-encryption keys (DEKs) handed to the object-encryption path are still wrapped with AES-256-GCM under the master key; the statement above concerns the master key's storage in Vault, not the DEK envelope.
• The backend reports this boundary in its backend_info metadata as at_rest_protection: vault-kv2-acl.
• Key rotation retains every historical master key version as an immutable record under {prefix}/{key_id}/versions/{N} and only then moves the current-version pointer; see Master key rotation for the retention preconditions and the cluster-upgrade ordering constraint.

Warning: KV read access is equivalent to holding the master keys. Any Vault identity (token, AppRole, or policy) that can read the RustFS key path in KV v2 can recover the plaintext master key material and decrypt every object protected by those keys. Treat KV read grants on that path with the same care as handing out the keys themselves. If this is not acceptable, use the Vault Transit backend instead.

Minimal Vault policy for the KV2 backend

Scope the RustFS token/AppRole to exactly the KV v2 mount and key prefix it is configured with (defaults shown: mount secret, prefix rustfs/kms/keys), and grant no other identity read access to that subtree:

# RustFS KMS (Vault KV2 backend) — key storage only, no Transit access needed.
path "secret/data/rustfs/kms/keys/*" {
  capabilities = ["create", "read", "update"]
}

path "secret/metadata/rustfs/kms/keys/*" {
  capabilities = ["list", "read", "delete"]
}

Notes:

• The trailing wildcards also cover the per-version material records that rotation creates under .../keys/{key_id}/versions/{N}; no extra policy paths are needed.
• delete on the metadata path is required for permanent key deletion (force_immediate); drop it if you never hard-delete keys.
• Do not attach sudo, wildcard mounts, or Transit paths to this policy; the KV2 backend does not use them.
• Auditing KV reads on the key prefix is strongly recommended: every read event is a potential master-key disclosure.

Master key rotation: retention, destruction, and upgrade ordering

Rotation support differs per backend. Local and Static reject rotation outright (InvalidOperation); their single key material is never overwritten. Vault Transit delegates rotation to the Transit engine's own key versioning (ciphertext is version-prefixed, e.g. vault:v1:...). Vault KV2 rotates by retaining every historical version, as described below. Rotation is currently only reachable through the backend-level KmsClient::rotate_key API; it is not exposed through the admin or S3 surface.

Vault KV2 versioned retention model

Each rotation writes the new version's material to {prefix}/{key_id}/versions/{N} as an immutable, create-only record, and only after that material is durably persisted does a check-and-set write move the top-level record (the current-version pointer, which also mirrors the current material as a fast path). The first rotation additionally freezes the pre-rotation material as a version record and pins it as the key's baseline_version; DEK envelopes written before versioning existed (no master_key_version field) always resolve to that baseline, never to whatever version is current.

Decryption loads exactly the version recorded in the envelope and fails closed with a typed KeyVersionNotFound error when that version's record is missing. There is deliberately no fallback to the current material: falling back would silently feed the wrong key to AEAD and mask tampered envelopes.

Retention and destruction preconditions

• Every version record that any stored DEK envelope references must remain readable. Until an object rewrap/migration capability exists, assume every version of a rotated key is referenced: destroying a version record permanently orphans all objects whose DEKs it wrapped.
• Version records are ordinary KV v2 secrets under the key subtree. Never run kv metadata delete or kv destroy against {prefix}/{key_id}/versions/*, and do not apply delete-version-after or retention tooling to that subtree. RustFS-managed retention does not rely on KV2's own secret versioning (each version record has a single KV revision), so KV max-versions settings do not protect or endanger history — but metadata deletion always removes a record entirely.
• Permanent key deletion through RustFS (force_immediate after PendingDeletion) purges the key's version records together with the key record; that is the only supported way to remove them.
• For Vault Transit, retention is governed by the Transit key's min_decryption_version: never raise it above the oldest version that may still protect live ciphertext.

Upgrade before first rotation (hard constraint)

Do not rotate any key until every RustFS node in the cluster runs a build that understands the master_key_version envelope field. Older binaries ignore the field and always decrypt with the current material: harmless while nothing has been rotated, but after a rotation they will fail to decrypt every object wrapped by an earlier key version. Complete the rolling upgrade of the entire cluster first, then rotate.

This is the sharpest instance of a broader class of constraints; the rest are collected in Mixed-version clusters during a rolling upgrade.

Mixed-version clusters during a rolling upgrade

During a rolling upgrade the cluster runs two RustFS builds at once. That window matters more for KMS than for most subsystems, because KMS state is shared three ways: Vault holds the key records and Transit metadata, cluster storage holds the persisted KMS configuration, and each node's process memory holds caches and the live backend instance. Nodes on different builds agree on the first, may disagree on the third, and — for configuration — can disagree for as long as the operator leaves them running.

This section states only what is true of the current implementation. It is written for the KV2 and Transit backends; the Local backend is unsupported for multi-node deployments regardless of version (see the deployment support matrix).

Persisted formats are backward compatible in both directions

Nothing in this list requires a coordinated format cutover. The compatibility is deliberate and is covered by decode tests.

• DEK envelopes. DataKeyEnvelope::master_key_version is optional and omitted when absent, so envelopes written by non-rotating backends stay byte-identical to the historical seven-field JSON shape. An upgraded node reading a pre-versioning envelope resolves None to the key's recorded baseline version, or — for a key that was never rotated, and so has no baseline — to the current version, which is exactly the pre-versioning behavior.
• KV2 key records. baseline_version is read with a serde default, so records written by older builds deserialize unchanged, and None correctly means "never rotated".
• Transit metadata records. Metadata persisted in KV v2 by either build decodes on the other.

The one-way hazard is the rotation constraint above: an older binary reading a new envelope silently ignores the version field and decrypts with the current material.

Guarantees that hold only once every node is upgraded

These are properties of the upgraded code, so a single node left behind removes them for the whole cluster.

• Check-and-set lifecycle writes. Upgraded builds write every KV2 lifecycle mutation — create, enable, disable, tag metadata, schedule deletion, cancel deletion — as a versioned read followed by a check-and-set write, retrying on conflict by re-reading and re-validating the state gate (rustfs/rustfs#5518). Transit metadata writes got the same treatment (rustfs/rustfs#5520). Builds older than those write blind. A blind write from an old node can overwrite a check-and-set commit from an upgraded node without any conflict being reported, which is precisely the lost update the change was made to eliminate.
• baseline_version survives a write-back. The KV2 key record does not deny unknown fields, so an old build reads a new record without error — and drops baseline_version when it writes that record back for any reason. A key that loses its baseline resolves pre-versioning envelopes to the current version again, which after a rotation means the wrong master key material. Any lifecycle operation issued to an old node is enough to trigger this.
• Version-record awareness. Rotation stores each historical version under {prefix}/{key_id}/versions/{N} as a create-only record (check-and-set of 0), so two nodes racing the same version number produce exactly one creator; the loser adopts the persisted, never-current material or fails without touching the current pointer. Old builds have no concept of that sub-path: they never read or write it, and their key listing reports the KV2 directory entry (my-key/) as though it were a key, because the directory filter only exists in upgraded builds.

Windows in which nodes can legitimately disagree

Even with every node on the same build, some state is process-local. These windows are bounded by design, except the last one.

What can diverge	Bound	Mechanism
Transit key lifecycle state used by the encrypt and generate_data_key gates	≤ 300 s (METADATA_CACHE_TTL)	Each node caches Transit metadata in process, TTL- and capacity-bounded, with targeted invalidation when a data-path call reports the key is gone server-side. A disable or schedule-deletion performed on one node is enforced on the others within one TTL at the latest, sooner if they hit that signal.
describe_key output	≤ 300 s	The manager-level key metadata cache. This is a reporting cache; the KV2 state gates do not read it.
KV2 key lifecycle state	None	The KV2 backend re-reads the key record from Vault for every lifecycle and data-key operation, so a committed disable is effective on every upgraded node immediately.
Active KMS configuration	Until the remaining nodes are restarted	See below.

Builds older than rustfs/rustfs#5520 held the Transit metadata cache with no TTL and no capacity bound. On such a node the divergence window is not 300 seconds but "until the process restarts": it can keep encrypting under a key that another node disabled, indefinitely.

Configuration is persisted cluster-wide but applied per node

POST /rustfs/admin/v3/kms/reconfigure currently does two things: it switches the KMS service on the node that handled the request, and it persists the new configuration to cluster storage at config/kms_config.json. It does not notify peers. Every other node keeps running the configuration it started with until it is restarted, at which point it loads the persisted configuration during startup.

The practical consequences today:

• The configuration-split window has no upper bound other than the operator restarting the remaining nodes. Convergence is a restart, not a timeout.
• During the window both configurations are live. If the reconfiguration changed backends, or changed the Vault mount or key prefix, different nodes write new key material to different places, and a key created through one node is invisible to the others.
• kms status reflects the node that answered the request, so a single successful status response is not evidence that the cluster is consistent. Query every node.

Treat reconfigure as the first step of a cluster-wide operation, not as the operation itself.

Recommended rolling upgrade order

Follow the node-at-a-time procedure in the multi-node restart runbook; this adds the KMS-specific sequencing around it.

1. Freeze KMS administrative traffic for the duration: no key creation, enable, disable, tagging, schedule-deletion, cancel-deletion, rotation, or reconfiguration. Object read and write traffic continues normally.
2. Upgrade one node at a time, waiting for each to report ready before starting the next.
3. Verify no node is left behind before unfreezing. A single old node is enough to reintroduce blind writes and to strip baseline_version on its next lifecycle write.
4. Resume administrative traffic.
5. Only then perform the first rotation of any key. Once the whole cluster understands master_key_version, rotation is safe; before that it is not.
6. If the KMS configuration was changed at any point, restart the nodes that did not handle the request so they reload the persisted configuration, and confirm each one reports the intended backend.

Do not do these during a mixed-version window

• Rotate any key. This is the hard constraint stated above; a rotation is unrecoverable for objects an old node must read.
• Issue any KV2 lifecycle write to an old node. Its blind write can clobber a concurrent check-and-set commit and will drop baseline_version from the record.
• Create the same key ID from two nodes. The create path is create-only on upgraded builds, but an old node's blind write does not honor that: the later writer's material wins and every DEK already wrapped with the earlier material becomes permanently unwrappable.
• Assume a disable or schedule-deletion took effect cluster-wide. Old Transit nodes cache lifecycle state without expiry; confirm per node, or restart the old nodes, before treating a key as no longer in use.
• Reconfigure the KMS backend and consider it done. The change applies to one node and is persisted; the rest need a restart.
• Delete or prune version records under {prefix}/{key_id}/versions/* for any reason. This is never safe, mixed-version or not; see Retention and destruction preconditions.

Choosing between Vault KV2 and Vault Transit

Use Vault Transit (VaultTransit) when key material must be cryptographically isolated from anyone holding storage-level read access: Transit keeps key-encryption keys inside Vault and only ever returns ciphertext, and supports server-side key versioning/rotation.

Use Vault KV2 only when you accept that the Vault ACL on the key path is the confidentiality boundary and you want the operational simplicity of a single KV mount.

Local backend durability and deployment support matrix

The Local backend stores one JSON record per key (<key_id>.key) plus an Argon2id salt file (.master-key.salt) inside the configured key_dir. This section documents which deployments that layout supports and how the backend recovers from a crash or power loss. For where the key material lives and who can read it, see the backend comparison above.

Positioning

The facts today:

• Local is the current default backend (kms_backend defaults to local).
• The RustFS Kubernetes operator places the key directory on a PersistentVolumeClaim, so the keys survive pod rescheduling.
• The in-code documentation labels the backend "for development and testing only", and configuration validation enforces stricter rules outside explicit development mode: a master key is required and key_dir must not live under the process temp directory.
• Production multi-node deployments should use the Vault Transit backend.

The backend's final support level is positioning under review (internal tracking); this section describes what the implementation guarantees, not a commitment to a support tier.

Deployment support matrix

Deployment	Supported	Notes
Local filesystem (ext4, XFS, APFS, ...)	Yes	The commit protocol relies on POSIX rename/hard_link atomicity and fsync durability, which local filesystems provide
Kubernetes PVC	Yes	Only when the PersistentVolume is backed by a local or block filesystem; this is how the RustFS operator provisions the key directory
NFS or other shared/network filesystems	No	Network filesystems do not reliably provide the atomicity and fsync semantics the commit protocol depends on; an NFS-backed PersistentVolume is this case, not the PVC case above
Multiple RustFS processes sharing one key_dir	No	Concurrent key creation is linearized (hard_link refuses to clobber an existing key), but every other write — status updates, deletion, cancellation — is a read-modify-write with no cross-process lock, so concurrent writers can silently lose updates

Within a single process, per-key write locks serialize read-modify-write updates, so concurrent API calls against one RustFS instance are safe.

Crash recovery behavior

Every mutation of the key directory uses a durable commit protocol:

1. A temp file (<name>.tmp-<uuid>) is created exclusively in key_dir.
2. The content is written and fsynced (sync_all).
3. The file is published atomically: rename to replace an existing file, hard_link to create a new one without clobbering.
4. The parent directory is fsynced so the new directory entry is durable.

Deletion mirrors the tail of the protocol (remove_file followed by a parent directory fsync), so a deleted key cannot resurface after power loss. A crash at any step leaves either the complete old state or the complete new state, plus at most an unpublished temp file.

On startup the backend then:

• Removes orphaned commit temp files. The matcher is strict (<prefix>.tmp-<uuid>, never anything ending in .key), so published key files — including a key the user named to look like a temp file — are never touched. Publishing is atomic, so a matching leftover can only be an unpublished remnant of an interrupted commit.
• Validates every published .key file. A record that fails to decode fails startup rather than being silently skipped.
• Guards the salt file. If .master-key.salt is missing but the directory contains keys marked encrypted-master-key, initialization fails closed with a configuration error naming the salt path. A regenerated salt derives a different master key and can never decrypt those keys, so the correct recovery is to restore the salt file (or the whole directory) from backup, never to let a fresh salt be generated. An empty directory, or a legacy directory predating the salt file, still initializes normally.

Backing up the key directory

Back up key_dir as a whole, including the hidden .master-key.salt file. A key file on its own is not restorable: decrypting it requires the master key derived from the configured master_key and the persisted salt. Restoring a partial directory — key files without the salt, or the salt without the key files — leaves the backend unable to decrypt, and the salt guard above will (correctly) refuse to start with encrypted keys and no salt. Losing the salt file with no backup means every key encrypted under it is unrecoverable.