rustfs/docs/operations/kms-backend-security.md

KMS backend security properties

RustFS ships several KMS backends. They differ not only in deployment effort but in where master key material lives and who can read it. Pick a backend based on the confidentiality boundary you need, not on the name alone.

Backend comparison

Backend	Config tag	Master key material location	At-rest protection of key material	Durability	Rotation	Intended use
Local	Local	Files under key_dir, encrypted with the configured local master key	Local master key (AES-GCM) + file permissions	Crash-durable commits on local filesystems only; see Local backend durability and deployment support matrix	Rejected by design (single material, development backend)	Development; single-node setups that accept host-level trust
Static	Static	Provided out-of-band via environment/file; never persisted by RustFS	Operator-managed secret distribution	No state persisted by RustFS	Rejected (read-only backend)	Simple deployments with an external secret manager
Vault KV2	VaultKV2 (legacy alias Vault)	Stored directly in Vault KV v2 (Base64-encoded plaintext)	Vault ACLs + KV v2 at-rest encryption + TLS only	Delegated to Vault storage	Versioned retention (immutable per-version records + current pointer)	Deployments that accept Vault KV ACLs as the sole confidentiality boundary
Vault Transit	VaultTransit	Key-encryption keys never leave Vault; only Transit ciphertext is visible outside	Vault Transit engine (cryptographic isolation)	Delegated to Vault storage	Via Vault Transit key versioning	Deployments that need key material to be unreadable through storage APIs

Vault KV2: what the backend does and does not do

The Vault KV2 backend uses Vault purely as a secure storage service:

• Master key material is generated by RustFS and written to KV v2 as a Base64-encoded value (encrypted_key_material is an encoding, not a ciphertext).
• The backend never calls the Vault Transit engine. The mount_path configuration field and the RUSTFS_KMS_VAULT_MOUNT_PATH environment variable are deprecated leftovers: they are accepted for compatibility and ignored.
• Data-encryption keys (DEKs) handed to the object-encryption path are still wrapped with AES-256-GCM under the master key; the statement above concerns the master key's storage in Vault, not the DEK envelope.
• The backend reports this boundary in its backend_info metadata as at_rest_protection: vault-kv2-acl.
• Key rotation retains every historical master key version as an immutable record under {prefix}/{key_id}/versions/{N} and only then moves the current-version pointer; see Master key rotation for the retention preconditions and the cluster-upgrade ordering constraint.

Warning: KV read access is equivalent to holding the master keys. Any Vault identity (token, AppRole, or policy) that can read the RustFS key path in KV v2 can recover the plaintext master key material and decrypt every object protected by those keys. Treat KV read grants on that path with the same care as handing out the keys themselves. If this is not acceptable, use the Vault Transit backend instead.

Minimal Vault policy for the KV2 backend

Scope the RustFS token/AppRole to exactly the KV v2 mount and key prefix it is configured with (defaults shown: mount secret, prefix rustfs/kms/keys), and grant no other identity read access to that subtree:

# RustFS KMS (Vault KV2 backend) — key storage only, no Transit access needed.
path "secret/data/rustfs/kms/keys/*" {
  capabilities = ["create", "read", "update"]
}

path "secret/metadata/rustfs/kms/keys/*" {
  capabilities = ["list", "read", "delete"]
}

Notes:

• The trailing wildcards also cover the per-version material records that rotation creates under .../keys/{key_id}/versions/{N}; no extra policy paths are needed.
• delete on the metadata path is required for permanent key deletion (force_immediate); drop it if you never hard-delete keys.
• Do not attach sudo, wildcard mounts, or Transit paths to this policy; the KV2 backend does not use them.
• Auditing KV reads on the key prefix is strongly recommended: every read event is a potential master-key disclosure.

Master key rotation: retention, destruction, and upgrade ordering

Rotation support differs per backend. Local and Static reject rotation outright (InvalidOperation); their single key material is never overwritten. Vault Transit delegates rotation to the Transit engine's own key versioning (ciphertext is version-prefixed, e.g. vault:v1:...). Vault KV2 rotates by retaining every historical version, as described below. Rotation is currently only reachable through the backend-level KmsClient::rotate_key API; it is not exposed through the admin or S3 surface.

Vault KV2 versioned retention model

Each rotation writes the new version's material to {prefix}/{key_id}/versions/{N} as an immutable, create-only record, and only after that material is durably persisted does a check-and-set write move the top-level record (the current-version pointer, which also mirrors the current material as a fast path). The first rotation additionally freezes the pre-rotation material as a version record and pins it as the key's baseline_version; DEK envelopes written before versioning existed (no master_key_version field) always resolve to that baseline, never to whatever version is current.

Decryption loads exactly the version recorded in the envelope and fails closed with a typed KeyVersionNotFound error when that version's record is missing. There is deliberately no fallback to the current material: falling back would silently feed the wrong key to AEAD and mask tampered envelopes.

Retention and destruction preconditions

• Every version record that any stored DEK envelope references must remain readable. Until an object rewrap/migration capability exists, assume every version of a rotated key is referenced: destroying a version record permanently orphans all objects whose DEKs it wrapped.
• Version records are ordinary KV v2 secrets under the key subtree. Never run kv metadata delete or kv destroy against {prefix}/{key_id}/versions/*, and do not apply delete-version-after or retention tooling to that subtree. RustFS-managed retention does not rely on KV2's own secret versioning (each version record has a single KV revision), so KV max-versions settings do not protect or endanger history — but metadata deletion always removes a record entirely.
• Permanent key deletion through RustFS (force_immediate after PendingDeletion) purges the key's version records together with the key record; that is the only supported way to remove them.
• For Vault Transit, retention is governed by the Transit key's min_decryption_version: never raise it above the oldest version that may still protect live ciphertext.

Upgrade before first rotation (hard constraint)

Do not rotate any key until every RustFS node in the cluster runs a build that understands the master_key_version envelope field. Older binaries ignore the field and always decrypt with the current material: harmless while nothing has been rotated, but after a rotation they will fail to decrypt every object wrapped by an earlier key version. Complete the rolling upgrade of the entire cluster first, then rotate.

Choosing between Vault KV2 and Vault Transit

Use Vault Transit (VaultTransit) when key material must be cryptographically isolated from anyone holding storage-level read access: Transit keeps key-encryption keys inside Vault and only ever returns ciphertext, and supports server-side key versioning/rotation.

Use Vault KV2 only when you accept that the Vault ACL on the key path is the confidentiality boundary and you want the operational simplicity of a single KV mount.

Local backend durability and deployment support matrix

The Local backend stores one JSON record per key (<key_id>.key) plus an Argon2id salt file (.master-key.salt) inside the configured key_dir. This section documents which deployments that layout supports and how the backend recovers from a crash or power loss. For where the key material lives and who can read it, see the backend comparison above.

Positioning

The facts today:

• Local is the current default backend (kms_backend defaults to local).
• The RustFS Kubernetes operator places the key directory on a PersistentVolumeClaim, so the keys survive pod rescheduling.
• The in-code documentation labels the backend "for development and testing only", and configuration validation enforces stricter rules outside explicit development mode: a master key is required and key_dir must not live under the process temp directory.
• Production multi-node deployments should use the Vault Transit backend.

The backend's final support level is positioning under review (internal tracking); this section describes what the implementation guarantees, not a commitment to a support tier.

Deployment support matrix

Deployment	Supported	Notes
Local filesystem (ext4, XFS, APFS, ...)	Yes	The commit protocol relies on POSIX rename/hard_link atomicity and fsync durability, which local filesystems provide
Kubernetes PVC	Yes	Only when the PersistentVolume is backed by a local or block filesystem; this is how the RustFS operator provisions the key directory
NFS or other shared/network filesystems	No	Network filesystems do not reliably provide the atomicity and fsync semantics the commit protocol depends on; an NFS-backed PersistentVolume is this case, not the PVC case above
Multiple RustFS processes sharing one key_dir	No	Concurrent key creation is linearized (hard_link refuses to clobber an existing key), but every other write — status updates, deletion, cancellation — is a read-modify-write with no cross-process lock, so concurrent writers can silently lose updates

Within a single process, per-key write locks serialize read-modify-write updates, so concurrent API calls against one RustFS instance are safe.

Crash recovery behavior

Every mutation of the key directory uses a durable commit protocol:

1. A temp file (<name>.tmp-<uuid>) is created exclusively in key_dir.
2. The content is written and fsynced (sync_all).
3. The file is published atomically: rename to replace an existing file, hard_link to create a new one without clobbering.
4. The parent directory is fsynced so the new directory entry is durable.

Deletion mirrors the tail of the protocol (remove_file followed by a parent directory fsync), so a deleted key cannot resurface after power loss. A crash at any step leaves either the complete old state or the complete new state, plus at most an unpublished temp file.

On startup the backend then:

• Removes orphaned commit temp files. The matcher is strict (<prefix>.tmp-<uuid>, never anything ending in .key), so published key files — including a key the user named to look like a temp file — are never touched. Publishing is atomic, so a matching leftover can only be an unpublished remnant of an interrupted commit.
• Validates every published .key file. A record that fails to decode fails startup rather than being silently skipped.
• Guards the salt file. If .master-key.salt is missing but the directory contains keys marked encrypted-master-key, initialization fails closed with a configuration error naming the salt path. A regenerated salt derives a different master key and can never decrypt those keys, so the correct recovery is to restore the salt file (or the whole directory) from backup, never to let a fresh salt be generated. An empty directory, or a legacy directory predating the salt file, still initializes normally.

Backing up the key directory

Back up key_dir as a whole, including the hidden .master-key.salt file. A key file on its own is not restorable: decrypting it requires the master key derived from the configured master_key and the persisted salt. Restoring a partial directory — key files without the salt, or the salt without the key files — leaves the backend unable to decrypt, and the salt guard above will (correctly) refuse to start with encrypted keys and no salt. Losing the salt file with no backup means every key encrypted under it is unrecoverable.