* feat(kms): add data key rewrap and wrapping inspection primitives
Rewrap re-protects an existing data key envelope with the master key's
current version without touching the data key itself, which is the
precondition for ever retiring an older version: until every envelope a
version wrapped has been moved off it, destroying that version orphans
every object whose data key it wrapped.
Adds KmsBackend::rewrap_data_key and its read-only counterpart
describe_data_key_wrapping, both gated by a new BackendCapabilities::rewrap
flag and defaulting to UnsupportedCapability. Vault KV2 unwraps with the
frozen version record that wrapped the envelope and re-wraps with the
current material; Vault Transit uses the native transit/rewrap endpoint so
the data key never enters this process.
No read or write path changes: nothing calls these yet.
* test(kms): cover the rewrap primitive against a scripted Vault
* fix(kms): resolve both key materials before the data key is unwrapped
Keeps every fallible step out of the window in which the plaintext data
key exists, so no error path can drop it without zeroizing it first.
* fix(kms): fail closed on Local key records this build cannot interpret
The Local backend's protection marker is the only version discriminator its
key records have, and three readers walked past it.
`ensure_missing_salt_can_be_generated` skipped every record it could not read
or parse, so a directory whose protection state is unknown still got a fresh
salt published before startup validation failed. That write is the
irreversible step: the next startup finds a salt file, never re-enters the
guard, and the evidence that the real salt was lost is gone. Every record the
guard now rejects already failed startup key validation a few lines later, so
no directory that initializes today stops initializing.
Backup export and restore folded an unknown marker into "material corrupt" /
"bundle corrupted". The record is intact and a newer build reads it fine, so
the operator response is a version change, not a disaster recovery. Both now
classify the marker before their schema parse, sharing one probe with the
backend reader.
`list_keys` dropped any record it could not decode from the page. Concurrent
removal stays a skip; anything else fails the listing rather than answering
"these are your keys" with a set that silently omits one.
* test(kms): cover every fail-closed path around the Local protection marker
Each test fails on the pre-fix code in the way the fix is about: the salt
cases because a replacement salt is published before startup validation
fails, the export and restore cases because the verdict comes back as
corruption, and the listing case because the record is edited out of the page.
The restore commit marker's unknown-version branch had no test at all,
unlike its Vault counterpart; it is now driven from the decoder, from the
restore entry point, and from backend startup.
Also states the widened salt guard in the Local backend operations doc,
including the operator recovery path for an unrecognized record.
* fix(kms): say 'not a readable JSON object' when the marker probe cannot parse
The probe now fails on any input that is not a JSON object, not only on
malformed JSON, so the message must cover both.
* test(kms): assert the salt file before the error variant
The replacement salt is written before the error the guard reports, so the
file assertion is the one that fails on a regression.
* test(kms): guard the new backup error variant's display string
api_types carried a second DeleteKeyRequest that nothing referenced: it is
absent from the lib.rs re-export list, so rustfs_kms::DeleteKeyRequest has
always resolved to types::DeleteKeyRequest through `pub use types::*`, and
the admin handler builds the real type via `use rustfs_kms::types::*`.
The copy had also drifted apart from the type it shadowed. It still called
force_immediate "for development/testing only" and described the 7-30 day
window as advisory, and it never gained the confirm_key_id field that the
immediate-deletion gate now requires. A caller that reached into api_types
and deserialized into it would silently drop confirm_key_id.
api_types::DeleteKeyResponse stays: it is live, pinned by the
kms_management_responses_have_stable_json_shapes snapshot alongside the
list/describe/cancel response shapes, and it mirrors the admin wire
response rather than duplicating types::DeleteKeyResponse, whose fields
differ. Its doc comment now records why no request twin sits beside it.
docs(kms): correct the KV2 at-rest boundary and rotation rejection claims
The Vault KV2 section claimed the backend reports its confidentiality
boundary as `at_rest_protection: vault-kv2-acl` through `backend_info`.
That accessor, the `BackendInfo` type and its assertion test were removed
in rustfs/rustfs#5501, and no replacement reports the boundary. State
instead that the boundary is documented only, that `kms/status` exposes a
capability matrix covering supported operations rather than key-material
location, and that the backup manifest's `at_rest_protection` field is a
bundle declaration rather than a backend self-report.
The rotation section named `InvalidOperation` as the error Local and
Static return. Neither advertises `rotate`, so the product path falls to
the shared `KmsBackend` default and returns `UnsupportedCapability`;
`InvalidOperation` survives only in a test-only client helper.
* feat(kms): report configuration references that block a key deletion
Adds a KeyImpactReport that states which configuration still points at a
key, how exhaustively the sources were read, and which sources were not
consulted at all. The report deliberately carries no in-use or
safe-to-delete claim: it covers the configuration layer only, so an empty
reference list means nothing was found in the scanned sources, never that
the key is unreferenced.
Immediate deletion destroys key material without ever reaching the
deletion worker, so it never passed the worker's reference gate. The
manager now consults the same checker on that path and refuses with a
typed KeyStillReferenced error. This only ever adds a refusal; the
scheduled deletion path and the worker's blocking behaviour are
unchanged.
* test(kms): cover the immediate-deletion reference refusal
* feat(kms): surface configuration references on the admin key endpoints
DeleteKey and DescribeKey now return an impact section listing the
configuration that points at the key, so an operator scheduling a
deletion sees what will refuse to destroy the material instead of
learning it from a server-side log once the window has run out.
The section is reported, never acted on: scheduling still succeeds while
references exist, and the deletion worker's gate remains the only thing
that decides whether material is destroyed. An immediate deletion that
the manager refuses for an outstanding reference now answers 409.
* test(kms): pin the impact wire shape and the unreferenced force-delete path
* fix(kms): make the DescribeKey impact section opt-in
Collecting the section lists every bucket, and DescribeKey is polled, so
carrying that fan-out on the default read path trades a hot path's cost
for a diagnostic. It is now collected only for impact=true; without the
parameter the endpoint does exactly the work it did before and returns
no impact field.
A value that is neither true nor false is refused rather than read as
off, so a typo cannot answer a request for the section with a response
that merely lacks one. DeleteKey still reports unconditionally: that is
the request whose consequences the caller cannot otherwise see, and it
is not polled.
* fix(kms): box the query-parse refusal now that responses carry impact
The delete response grew an impact section, which pushed it past the
size clippy accepts inline in a Result. It is a full response body
rather than an error code, so it is boxed at the one place that returns
it as an error; the wire shape and the public field type are unchanged.
* fix(kms): persist and report the Vault KV2 key rotation timestamp
The rotation-age gauge reads KeyInfo::rotated_at and falls back to
created_at when it is absent. The KV2 backend never persisted a rotation
time and hardcoded None in describe_key, so a key rotated many times and
a key that was never rotated reported the same age.
Record the rotation time on the same check-and-set write that switches
the current version, and report the stored value from describe_key and
from the recovered-create path. Records written before the field existed
keep deserializing and stay unstamped: no timestamp is invented for a
rotation this node cannot vouch for.
* test(kms): cover Vault KV2 rotation timestamp persistence and legacy records
* docs(kms): correct the static backend's MinIO compatibility claim
`StaticConfig` claimed the backend derives DEKs via "HMAC-SHA256 +
AES-256-GCM, matching the MinIO builtin/static KMS wire format". Neither
half holds: the configured key is used directly as the AES-256-GCM key
with no derivation step, and wrapped DEKs are serialized as RustFS's own
`DataKeyEnvelope` JSON. MinIO's KMS ciphertext uses a different shape,
which `is_data_key_envelope` explicitly classifies as foreign (see the
`minio_legacy` case in encryption/dek.rs).
The claim as written tells a migrating operator that MinIO-written
ciphertext will open here, which it will not. Restate what the backend
actually does and point at rustfs/backlog#1638 for the real interop work.
Also refresh the neighbouring ciphertext-format note in static_kms.rs,
which still described a raw `ciphertext || nonce` layout that the JSON
envelope replaced.
* ci(minio-interop): fix the dead test selector and guard against empty runs
The job selected its tests with `-p rustfs-ecstore -E
'binary(minio_generated_read_test)'`. #5435 moved those reader tests from
crates/ecstore/tests/minio_generated_read_test.rs into the `rustfs` crate
as a `#[cfg(test)] mod`, which removed that test binary; the selector has
selected zero interop tests since. Point it at the tests where they now
live, verified locally:
cargo nextest list --run-ignored all -p rustfs --features rio-v2 \
-E 'test(minio_generated_read_test::)' # 4 tests, was 0
Add a guard step in front of the run. The old `binary(...)` form happened
to fail loudly once its binary disappeared, but the name-based form that
replaces it is a valid filterset even when it matches nothing, so a later
rename would silently reduce this job to a pass that asserts nothing. The
guard counts the selection and fails with an explicit reason; the count
comes from `filter-match.status`, since the JSON's top-level `test-count`
is the package total and ignores `-E`. `--no-tests=fail` on the run step
covers the same case if the guard is ever dropped.
Also record in the header what this job does and does not prove: MinIO
wrapped-DEK envelopes are still rejected by both envelope parsers, and
that work is tracked in rustfs/backlog#1638.
* fix(kms): page the Local key listing so the deletion sweep sees every key
The Local backend answered every ListKeys with the first `limit` entries of
`read_dir` and a hardcoded `truncated: false`, so the deletion sweep ended
after one page: on a deployment with more keys than a page, expired key
material past the first page was never destroyed, and the lifecycle gauges
published that partial page as if it were the whole key set.
Listing now orders the key set by identifier and pages through it, with the
marker as an exclusive lower bound on the identifier rather than an index, so
a key added or removed between pages — including the marker key, which the
sweep itself destroys — cannot make the listing skip keys or restart. Only the
page is read from disk, so a list costs the requested limit rather than the
size of the key set.
The pagination arithmetic lives in a shared helper so the other self-paging
backends can adopt the same semantics, and the sweep now stops instead of
re-listing when a backend hands back the cursor it was just given.
* fix(kms): give ListKeys a defined zero-limit and cursor contract
`GET /rustfs/admin/v3/kms/keys?limit=0` reached the Vault KV2 and Vault
Transit backends as a page size of zero, where the page arithmetic indexed the
element before an empty page and aborted the request. Both backends also
resolved the marker by searching for it in the key list, so a marker naming a
key that had since been removed silently restarted the listing from the
beginning instead of resuming after it.
All four self-paging backends now share one contract: a zero limit is answered
as an empty, non-truncated page without reaching the backend at all, and the
marker is an exclusive lower bound on the key identifier rather than a position
in the list. The Vault backends read metadata only for the page they return,
so a list costs the requested limit instead of the whole key set, and KV2 now
applies the usage and status filters it previously accepted and ignored.
* feat(kms): accept the AWS backend through KMS configuration
The AWS KMS backend could be constructed but not selected: the admin
configure API had no AWS variant and startup rejected the backend name.
The configure request pins the region rather than defaulting it, because
that configuration is persisted once and replayed on every node: leaving
the region to each node's ambient provider chain would let nodes address
different regions, and therefore different keys, while reporting an
identical configuration. The request accepts no credential fields, so
credentials stay with the aws-config provider chain on each node, and
`deny_unknown_fields` refuses attempts to submit them anyway.
* test(kms): cover AWS backend selection through the service manager
An end-to-end check that an admin configure request selects the AWS
backend, builds a client, and passes the startup health check. Marked
#[ignore]: it needs real AWS credentials, though it creates no key and
is therefore not billable on its own.
* test(kms): cover KV2 decrypt of pre-rotation envelopes offline
The forward half of the rotation contract - an envelope written before a
rotation still decrypts after it - was only exercised by the #[ignore]
live-Vault tests, so CI never verified it. The offline layer only had the
negative cases (a regressed version pointer must fail closed).
Drive encrypt -> rotate -> decrypt over the scripted Vault responder,
folding what each rotation writes back into the served state so the
material the decrypt resolves is the material the rotation persisted.
Also cover two consecutive rotations and pin that new envelopes carry the
rotated version.
* test(kms): cover Transit decrypt of pre-rotation data keys offline
Vault owns the transit crypto, so the offline responder cannot prove the
round trip - that stays in the #[ignore] live test. What it can pin is the
client-side wiring: after a rotation records the version bump, decrypting
a data key generated before it must forward the historical vault:v1:
ciphertext to Vault byte for byte and return the material Vault hands
back.
* feat(kms): add a disaster-recovery drill harness for the Local backend
Rehearse the full backup/restore loop offline and return machine-readable
evidence: seed a sandbox deployment, seal sample objects through the
production encryption path, export a bundle, destroy the persistence layer,
preflight, restore, and decrypt every pre-disaster object again.
The evidence records the measured recovery point (one key is written past the
snapshot fence and must stay unrecoverable), the recovery time by phase, the
manifest digest before and after, and whether the restore treated its bundle
as read-only.
* test(kms): drill the Local disaster matrix and the interrupted cutover
Runs the harness against total key-directory loss, salt loss, and a torn key
record, asserting every pre-disaster object decrypts again while work past the
snapshot fence stays lost. Two further legs crash a restore exactly at its
commit point and prove the published marker names the bundle and the files it
still owes, then that re-running rolls forward and aborting rolls back.
The Vault leg needs a real server and is ignored by default: a Vault bundle
never carries the non-exportable Transit root, so what it drills is the
refusal to proceed before the operator has restored it natively.
* feat(kms): add an operator entry point for the disaster-recovery drill
Runs one rehearsal from environment configuration and writes the evidence
bundle, exiting non-zero on a failed verdict so a scheduled drill fails its
job instead of filing a bad report. It reads the same backup-KEK variables as
the admin backup API: drilling with the KEK real bundles are sealed under is
what proves that KEK is still retrievable.
* docs(kms): add the disaster-recovery drill runbook
Documents the procedure the harness automates: what a drill measures and why
the object probe rather than the manifest digest is the acceptance criterion,
the per-backend responsibility split, the disaster matrix, how to read the
evidence bundle, the two interrupted-cutover outcomes, and the Vault variant
whose cryptographic root comes back through Vault's own flow.
* chore(typos): accept RTO as a disaster-recovery term
* feat(kms): add backup and restore admin API
Wires the merged KMS backup contract, Local export and Local restore into
the admin API: export a sealed bundle, run a zero-write restore preflight,
execute a confirmed restore, roll an interrupted restore back, and report
subsystem readiness.
- Dedicated kms:Backup / kms:Restore actions, recorded in the admin route
matrix. Neither is reachable through any other KMS action.
- Restore requires two independent confirmations: an echo of the bundle
manifest's backup id, and an explicitly named conflict policy (the
default never writes).
- The backup KEK comes from the environment and is refused when it reuses
a secret of the configured backend, compared both as the literal value
and as raw key bytes.
- No endpoint accepts a path: bundles are addressed by a validated name
under a configured root, and the restore target is always the server's
own configured key directory.
- Bundles now carry a sanitized configuration artifact built as an
allowlist projection, so a future backend credential field cannot leak
into a bundle by default. Restore verifies it and never applies it.
- Audit entries go through the existing KMS admin wiring and carry
identifiers only.
* test(kms): pin the backup admin API gates
Fixes the test KEK to a real 32-byte value and drives the export refusal
from the configured backend rather than from the handle that happens to
be available, so a Local handle cannot export on behalf of a backend
whose material RustFS does not own.
`VaultKeyData::baseline_version` pins the master key version that every
pre-versioning DEK envelope (one with no `master_key_version`) resolves to.
Builds released before versioned rotation do not know the field, and every KV2
lifecycle write — enable, disable, schedule/cancel deletion, tag, untag —
rewrites the whole record, so a single lifecycle call from an older node during
a rolling upgrade silently drops the baseline. Serde attributes cannot prevent
this: the code doing the dropping already shipped.
The loss is only latent until the next rotation. With the baseline gone,
rotation takes the first-rotation path again and freezes a *new* baseline at the
current version, so every legacy envelope permanently resolves to material that
never wrapped it. That is the point of no return, and it is the one this commit
blocks: rotation now lists the key's immutable version records first and refuses
when records exist while the record carries no baseline. Those two are created
by the same commit, so the combination can only mean the baseline was erased
afterwards. The refusal is decided from reads alone, before any write, and names
the version to restore — the oldest recorded version *is* the lost baseline,
since version records start at the baseline the first rotation froze.
Reads are diagnosed rather than blocked. A version-less envelope on a key with
no baseline resolves to the current version, which AES-256-GCM refuses to
unwrap when it is the wrong one, so no wrong plaintext can be returned. Only
after that failure does decrypt list the version records and re-report the
failure as the lost baseline. Refusing up front on the same evidence would break
reads that work today: an older node writes version-less envelopes wrapped with
whatever material is current, and those still decrypt. This also keeps the extra
listing off every read of pre-versioning data.
Self-healing (writing back `baseline_version = min(recorded)`) is deliberately
not done: during the mixed-version window that caused the loss, an older node
can erase it again on the next lifecycle call, so healing would mask an
unfinished upgrade instead of surfacing it. Moving the baseline to a KV path
older builds cannot rewrite is the real fix and is scheduled for GA.
Refs rustfs/backlog#1581 (part of rustfs/backlog#1562)
* fix(kms): honour the configured metadata cache TTL and metrics switch
KmsManager::new built the KmsCache from cache_config.max_keys alone, so
cache_config.ttl was dead configuration: every deployment ran the
hardcoded 300s window whatever the admin configure API was given, while
CacheSummary and the KMS config endpoint reported the configured value
back. cache_config.enable_metrics was never read anywhere.
Build the cache from the whole CacheConfig. The documented default is
reconciled down to the 300s the cache has always used rather than up to
the advertised 3600s, and now lives in one place (DEFAULT_CACHE_TTL)
instead of being duplicated across the four configure-request
converters, so the default path behaves exactly as before.
Behaviour change: a deployment configured through the admin API already
has ttl 3600 persisted, because the old converters wrote that default
into the stored config, so its describe_key staleness window widens from
an effective 300s to the 3600s it asked for. No cryptographic or
authorization path widens - encrypt, decrypt and generate_data_key go
straight to the backend and never read this cache. The Vault Transit
backend's own metadata cache, which does gate crypto through
ensure_key_state_allows, stays fixed at 300s and is now documented as
deliberately not operator-tunable.
The configured duration now reaches moka's builder, which panics above
1000 years, so CacheConfig::effective_ttl clamps to a 24h maximum the
way effective_timeout already clamps its own, and validate rejects a
zero TTL beside the existing max_keys check. Both config summaries and
the KMS config endpoint report the effective value, so the admin API
cannot advertise a lifetime the cache does not honour.
enable_metrics gates publication of the rustfs_kms_metadata_cache_*
families only; the counters behind the admin status API keep running
either way. No configure-request field sets it yet.
Refs rustfs/backlog#1584
* docs(kms): state why the Transit metadata TTL is not bound to the default
The comment claimed the constant matches config::DEFAULT_CACHE_TTL, which
reads as an invariant the code does not enforce. Say plainly that the
equality is a coincidence rather than a contract, and why binding the two
would be wrong: this cache gates crypto through ensure_key_state_allows,
so a later change to the operator-facing describe-cache default must not
be able to widen its staleness window.
* feat(kms): observe key lifecycle from the deletion sweep
The sweep already pages through the whole key set, so the lifecycle
gauges come out of the pages it has in hand: no extra backend call is
made for them. It publishes the number of keys awaiting their deletion
deadline, the number of tombstones an interrupted removal left behind,
and how long ago the least recently rotated usable key was rotated
(counting from creation for keys that were never rotated), plus a
per-outcome counter of what the sweep acted on.
Every gauge is a label-less aggregate: a per-key label would carry key
identifiers into the metric stream and grow the series count with the
key set, so "this key is overdue for rotation" stays a threshold for an
alerting rule to apply to the aggregate. Keys the sweep destroys drop
out of the census, and a sweep that could not finish listing leaves the
gauges at their last complete values rather than understating them.
* feat(kms): expose Vault token TTL and fail-closed state as gauges
The renewal loop already tracks token expiry, so it now publishes the
seconds left on the active Vault token and whether the provider is
refusing to serve it. The fail-closed gauge re-evaluates the very gate
`VaultCredentialProvider::current` applies, so what operators see and
what the request path does cannot drift apart.
Both waits in the loop republish on a bounded cadence, so a scrape
landing between refresh cycles never reads a TTL frozen at the last
refresh or a fail-closed state that flipped after it. That costs a timer
and no Vault traffic, and the request path stays free of metric work.
Renewal successes and failures already land in the auth operation
counters, so nothing is double-counted here. Neither gauge carries a
label: the address, mount, auth path and token are all off limits as
label values, and there is one generation to describe.
* feat(kms): record real cache hit, miss and eviction metrics
The metadata cache reported (entry_count, 0) because moka exposes no hit
or miss counts, so the miss half of every cache report was a constant.
Track lookups and removals in the cache itself: hit/miss counters on the
lookup path, a moka eviction listener classifying removals by cause, and
an entry gauge refreshed whenever the entry set changes. The counters are
exported through the metrics facade under the rustfs_kms_ prefix with
static label values only, matching the operation-policy metrics, and are
also returned as a KmsCacheStats snapshot in place of the old tuple.
Cache semantics are unchanged: capacity, TTL and invalidation points are
the same, and remove now flushes pending maintenance so the gauge and the
removal notification describe the cache the caller sees.
Refs rustfs/backlog#1584
* fix(kms): report real cache counters through the admin status API
KmsStatusResponse.cache_stats mapped the old (entry_count, 0) tuple onto
hit_count and miss_count, so operators polling KMS status read the entry
count as a hit count and a miss count that was always zero.
Map the fields to the counters they claim to be, and add entry_count and
eviction_count as additive, defaulted fields so the entry number that
hit_count used to carry is still available.
Refs rustfs/backlog#1584
* fix(kms): refresh the cache entry gauge on lookup misses
The entry gauge was published only from the write paths, so an entry
dropped by TTL expiry left `rustfs_kms_metadata_cache_entries` reporting
a population that no longer existed until the next put, remove or clear.
A cache that goes quiet — entries ageing out with no further writes —
kept over-reporting indefinitely.
Republish the gauge from the lookup path when the lookup misses. A miss
is where expiry surfaces, and moka reaps expired entries in the
maintenance it runs during that same lookup, so the count read
afterwards reflects the reaping. Hits stay free of the extra work.
* docs(kms): correct the entry gauge convergence claim on the miss path
The comment on the miss-path gauge refresh said moka reaps expired
entries in the maintenance it runs on that same lookup. It does not:
`should_apply_reads` is gated on a full read log or an elapsed
housekeeping interval, so the removal that decrements `entry_count` and
reaches the eviction listener may land on a later lookup.
The behaviour and the test are unchanged — the gauge still converges,
and the test drives `run_pending_tasks` explicitly rather than riding on
that interval. Only the stated guarantee was wrong, so say interval
instead of same-lookup and record why forcing maintenance on the read
path was not the trade taken.
* feat(kms): add link_durably primitive and restore-marker startup guard
The local backend gains the two pieces the bundle restore path builds on:
a no-clobber hard-link publish primitive whose AlreadyExists case is
idempotent only for byte-identical content, and a fail-closed startup
guard that refuses to open a key directory holding a restore cutover
marker. The durable commit protocol and the key-id containment check
become pub(crate) so the restore module reuses them instead of copies.
* feat(kms): record a master-key verifier and pre-seal decrypt probe in export
Fill the manifest's master_key_verifier slot with an opaque one-way
value (scheme-prefixed, bound to the backup id and the KDF salt) so a
restore can detect a wrong operator-supplied master key before touching
any target state, and probe-decrypt every artifact as stored under the
backup KEK before the manifest may seal — digest equality alone only
proves the ciphertext landed intact. The payload decryption tail is
factored out and shared with the restore side so producer and consumer
cannot drift on the framing.
Also drops the stale KmsClient test import orphaned by the backend
refactor (#5501); the test suite did not compile without this.
* feat(kms): restore local backend key material from sealed backup bundles
The consumer side of the Local bundle export, as a four-phase protocol:
- Dry-run: full in-memory bundle decode (digest and AEAD verification of
every artifact), KDF-drift detection against the compiled-in
derivation, deployment and injected-generation checks (strictly lower
is rejected, equal stays allowed for repeated drills), master-key
verifier check, and target conflict enumeration - with zero writes.
- Staging: artifacts are committed durably into the .restore-staging/
subdirectory (invisible to the backend's key scan and orphan-temp
matcher) and every record is decryption-probed with the derived
master key both in memory before staging and again from the staged
bytes.
- Commit marker + cutover: the durably published .restore-commit.json
marker is the single commit point; cutover publishes staged files via
link_durably (salt first, keys after), then durably removes the
marker and drops staging.
- Crash re-entry: before the marker the target top level is untouched
and a re-run starts over; with the marker published, backend startup
fails closed and a re-run with the same bundle rolls forward while
abort_local_restore rolls back. Every interruption converges to the
complete old or complete new state.
Restore never goes through LocalKmsClient::new (which would mint a
fresh salt); the only write mode is the explicit
restore-into-empty-target policy, where an orphan salt or a foreign
marker already counts as non-empty. The bundle source stays strictly
read-only.
Refs rustfs/backlog#1572
* fix(kms): drop the KmsClient trait import removed by #5501
The local backup export tests (#5499) merged after #5501 folded the
KmsClient trait into KmsBackend, leaving a dead trait import that breaks
'cargo test -p rustfs-kms' compilation on main. create_key is an inherent
method on LocalKmsClient since #5501, so the import is unnecessary.
* fix(kms): CAS Transit metadata writes and bound the metadata cache
Transit KV metadata writes were whole-record overwrites with no
precondition, so two nodes mutating the same key could silently clobber
each other's lifecycle state, and the process-local metadata cache had
neither a TTL nor a capacity bound, so a key disabled or scheduled for
deletion on one node stayed usable on every other node until restart.
- Replace write_metadata_to_kv with a versioned read
(read_metadata_from_kv_versioned) plus a check-and-set write
(cas_write_metadata_to_kv); mutate_key_metadata re-reads the
authoritative record and re-runs the state gate on every attempt, and a
lost CAS race retries with a fresh snapshot (bounded budget) instead of
replaying the stale one.
- Migrate every read-modify-write caller: enable, disable, schedule and
cancel deletion, rotate version bump, the expired-key tombstone, and
both create paths (create-only CAS that read-confirms the winner on a
lost race).
- Bound the metadata cache with moka (300s TTL, 1024 entries) and drop a
key's entry when a transit data call reports it gone server-side.
- Fail closed when the synthesized-metadata fallback cannot be read or
persisted: the fabricated Enabled record is only served after a durable
create-only CAS write, closing the gate weakening documented as a KNOWN
RISK; the persistence fallback for pre-metadata keys is kept.
Refs rustfs/backlog#1581 (part of rustfs/backlog#1562)
* fix(kms): drop stale KmsClient trait import in local_export tests
The KmsClient trait was folded into KmsBackend (#5501), but the backup
export tests merged afterwards (#5499) still imported it, breaking the
crate's test build; create_key is an inherent LocalKmsClient method, so
the import is simply unused.
* fix(kms): make Vault KV2 lifecycle writes check-and-set
Every KV2 lifecycle write used to be a blind whole-record overwrite, so
two nodes racing on the same key could lose updates: a disable racing a
rotation wrote the pre-rotation record back (rolling back the version
and material of a committed rotation), concurrent same-name creates let
the later material win (orphaning DEKs wrapped under the earlier one),
and a cancellation racing the deletion sweep could be overwritten by
the tombstone (or resurrect an already tombstoned key).
All lifecycle mutations now go through a bounded check-and-set
read-modify-write loop: each attempt re-reads the record pinned to its
KV2 secret version, re-runs the state gate against the fresh snapshot,
and writes back check-and-set against exactly that version; after
LIFECYCLE_CAS_ATTEMPTS lost races the typed conflict error surfaces.
The loop composes with the operation policy's single-attempt rule for
non-idempotent writes: each write is still sent at most once, only the
whole read-gate-write cycle repeats. create_key becomes a create-only
write (cas=0) so exactly one of two concurrent creates commits and the
loser reports KeyAlreadyExists. The blind store_key_data primitive is
now test-only.
Reads and rotation additionally fail closed when the version history is
inconsistent: resolving material through a version record above the
current pointer is refused (that state only arises when a lost update
rolled back a committed rotation), and rotation refuses to extend a
history whose records reach more than one step past the current pointer
(one step ahead is the footprint of an interrupted rotation and still
recovers through the adopt path).
Refs rustfs/backlog#1581
* feat(kms): export local backend key material as sealed backup bundles
Adds the producer side of the Local backup series on top of the #5483
contract: a directory-wide export fence gives the snapshot a single
consistent generation, every artifact is AEAD-wrapped under a
caller-supplied backup KEK that is separate from the business trust
hierarchy, and the sealed manifest with completeness marker is written
last so an interrupted export can never be mistaken for a restorable
bundle. Restore and the admin API land in follow-up changes.
* fix(kms): verify manifest digest against raw bytes, not re-serialized fields
The decode path recomputed the digest by re-serializing the parsed
manifest, which silently assumes every field's stored spelling survives
a parse-and-reprint round trip. Timestamps do not guarantee that: the
time zone annotation jiff emits depends on the host (IANA name, POSIX
TZ string, Etc/Unknown), and the legacy-compat parser rewrites
bracket-less spellings to +00:00[UTC]. On CI this made freshly written
bundles fail digest verification while passing locally.
Digest verification now operates on the raw stored bytes, normalized
only through the JSON value layer with the digest slot emptied in
place; parsed typed fields are never re-serialized on the decode path.
Sealing uses the same value-layer canonical form, and the export
additionally pins created_at to UTC so bundles are host-independent. A
regression test seals a manifest whose created_at spelling cannot
round-trip and proves decoding still verifies. One behavior sharpens:
inserting an explicit null reserved slot after sealing is now rejected
as a digest mismatch instead of being tolerated.
* fix(kms): make manifest digest canonicalization independent of map ordering
The canonical digest form serialized serde_json values directly, which
inherits the key order of serde_json's map type: sorted by default, but
insertion-ordered when any crate in the unified build graph enables the
preserve_order feature. The workspace-wide CI build unified that
feature while a per-crate local build did not, so the frozen fixture
digest matched in one environment and not the other — and a bundle
sealed by one build flavor would fail verification in the other.
Canonicalization now rebuilds every JSON object with bytewise-sorted
keys (array order preserved) before hashing, so the digest bytes are
identical regardless of feature unification. Reproduced by enabling
preserve_order in dev-dependencies (fixture test red), then verified
green with the fix under both map flavors.
Add opt-in hotpath feature surfaces to every workspace crate and wire the root rustfs feature passthrough for function, allocation, and CPU profiling.
Add a focused set of function-level measurements for scanner, heal, lock, target replay, IAM, KMS, Keystone, trusted proxy, and capacity paths without adding request-scoped primitive wrappers.
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(kms): record operation metrics in the retry policy engine
Instrument policy::execute — the single choke point every outbound Vault
call and credential exchange already flows through — so no call site
needs its own instrumentation:
- rustfs_kms_backend_operations_total (counter): operation, op_class,
outcome (success / fatal / budget_exhausted / deadline_exceeded /
cancelled)
- rustfs_kms_backend_attempt_failures_total (counter): operation,
error_class (retryable_conn / retryable_status / fatal /
attempt_timeout)
- rustfs_kms_backend_operation_duration_seconds (histogram): wall-clock
duration including retries and backoff
- rustfs_kms_backend_operation_attempts (histogram): attempts used
Metric labels carry only static enum values (operation names, classes,
outcomes) — never key identifiers, key material, ciphertext, or tokens.
Emission goes through the process-global metrics facade recorder, the
same pattern the rest of the workspace uses, so no new wiring is needed
in rustfs/src.
Tests drive a paused-clock runtime under a thread-local debugging
recorder, so counts, attempts, and even the recorded (virtual-clock)
durations are asserted deterministically with zero real sleeps.
Refs rustfs/backlog#1569 (part of rustfs/backlog#1562)
* test(kms): add Vault fault-injection matrix
Offline cases inject transport faults locally and are fully
deterministic: a refused connection is retried up to the configured
budget, and a stalled connection is cut off by the per-attempt timeout
instead of hanging. Ignored cases run against a real dev Vault
(RUSTFS_KMS_VAULT_ADDR) and pin the fail-closed auth behavior: an
invalid token and a missing key each resolve in exactly one attempt.
Every case asserts through the policy metrics recorded by a
thread-local debugging recorder, which doubles as the request-count
assertion even against a real server. Throttling and recoverable 5xx
responses cannot be forced on a stock dev Vault; those paths stay
pinned by the scripted-Vault wiring tests and the engine tests.
Refs rustfs/backlog#1569 (part of rustfs/backlog#1562)
* feat(kms): add key lifecycle operations to the backend contract
Add enable_key/disable_key/rotate_key to KmsBackend with conservative
defaults returning the typed UnsupportedCapability error, mirroring
remove_expired_key. KmsManager gains matching pass-through methods and
drops cached key metadata after every successful state mutation so the
next describe observes backend truth. The local backend overrides
enable/disable, delegating to its state-machine-gated client methods;
rotation stays rejected, matching its advertised capabilities. New
dedicated policy actions kms:EnableKey and kms:DisableKey complete the
KMS action taxonomy alongside the existing kms:RotateKey.
* feat(admin): add KMS key enable/disable/rotate endpoints
POST /v3/kms/keys/enable, /v3/kms/keys/disable and /v3/kms/keys/rotate,
following the existing /v3/kms/keys handler conventions: key_id body
with keyId query fallback, {success, message, key_id, key_metadata}
responses, and 503 JSON while the KMS service is absent. Error mapping
keeps InvalidOperation/ValidationError at 400 like the sibling handlers
and surfaces UnsupportedCapability as 501 so a backend capability gap is
never mistaken for a missing key. Existing /v3/kms/keys handlers are
untouched apart from a visibility change on a private query helper.
* feat(kms): gate scheduled key deletion on bucket encryption references
Implement the DeletionReferenceChecker seam left by the deletion worker:
before any material is destroyed, every bucket's SSE configuration is
checked for a default KMS key reference and a hit blocks the removal.
The gate fails closed - an unpublished object store, a failed bucket
listing or an unreadable per-bucket encryption config all report a
blocking reference - because destroying key material is irreversible
while a blocked removal is simply retried on the next sweep. Registered
during init_kms_system before the service can start, so every worker
spawn observes it. Storage access goes through a new kms section of the
root storage facade.
* test(kms): add a scripted loopback Vault for policy wiring tests
A minimal HTTP/1.1 responder that serves canned Vault responses in order
and records the method/path sequence, so wiring tests can assert exactly
how many requests a code path performed (retries, read-confirm) without
a live Vault server.
* feat(kms): route Vault operations through the retry policy engine
Wire every outbound vaultrs call in the KV2 and Transit backends through
policy::execute, completing the wiring half of the operation policy work
(the engine landed separately):
- Reads (KV2 read/read_metadata/read_version/list, transit read/list/
encrypt/decrypt, health checks) run as ReadIdempotent: bounded retries
with exponential backoff and jitter on 429, recoverable 5xx, and
connection-level failures; 400/401/403/404 stay fatal.
- Writes (KV2 set/CAS set/delete_metadata, transit create/update/rotate/
delete, metadata writes) run as MutatingNonIdempotent: exactly one
attempt under the per-attempt timeout, never replayed. CAS conflicts
in the rotation protocol pass through unchanged as the concurrency
signal they are.
- Each attempt takes a fresh credential snapshot, so a retry after a
credential rotation uses the new token.
- Read-confirm recovery for lost create responses: when a create finds
an existing key that is exactly what it would have produced (same
algorithm, enabled, usable material, and for request-level creates the
same usage/description/tags), it reports the stored key as the create
result instead of KeyAlreadyExists. Any divergence keeps failing.
- Deletes treat already-deleted records as completed deletes (KV2
version records; transit metadata already did), so re-running an
interrupted deletion converges.
- A failed existence pre-check inside create now fails the create
instead of falling through to a blind overwrite (fail closed).
- The policy module sheds its allow(dead_code) now that it is wired.
Wiring tests run against a scripted loopback Vault and assert request
counts and endpoints for the retry, single-attempt, CAS-conflict, and
read-confirm paths.
Refs rustfs/backlog#1569 (part of rustfs/backlog#1562)
* feat(kms): enforce shared key state machine across backends
Unify the key state x operation matrix behind a single gate in
backends/mod.rs and wire it into the Local, Vault KV2 and Vault Transit
backends: Disabled keys reject encryption, data key generation and
rotation while still allowing decryption and lifecycle recovery;
PendingDeletion keys reject everything except decryption and
cancellation (including repeated deletion scheduling); cancellation now
requires an actual pending deletion everywhere. This closes the missing
gates on KV2 encrypt/generate and Local generate_data_key, and stops
enable_key from silently reverting a pending deletion.
Decryption is deliberately left ungated in Disabled/PendingDeletion — an
explicit, documented and tested deviation from AWS KMS, since gating it
would break reads of existing objects the moment a key is disabled.
Add shared contract tests driving the full matrix offline for Local (and
via ignored tests against a live Vault for KV2/Transit), a stateless
contract for Static, an SSE-shaped regression proving existing envelopes
stay decryptable after disable, and a pin on the known-risk Enabled
default of Transit's synthesized metadata fallback.
Refs rustfs/backlog#1571 (part of rustfs/backlog#1562)
* feat(kms): persist deletion deadlines and run a restartable deletion worker (#5491)
* feat(kms): add AppRole configuration surface for Vault auth
Extend VaultAuthMethod::AppRole with secret_id_file (re-read on every
login so external rotation is picked up), a configurable auth mount
(default "approle"), and an optional fail-closed safety window. All new
fields are serde(default) so previously persisted configurations keep
deserializing, and the strict admin-configure deserializer accepts them
as optional.
Environment selection: setting RUSTFS_KMS_VAULT_APPROLE_ROLE_ID switches
both Vault backends to AppRole; the secret_id comes from
RUSTFS_KMS_VAULT_APPROLE_SECRET_ID_FILE (path stored, file wins) or
RUSTFS_KMS_VAULT_APPROLE_SECRET_ID, following the static secret-key file
precedent. validate() rejects AppRole configs without a role_id, without
any secret_id source, or with an empty mount.
Also append the CredentialsUnavailable error variant used by the
fail-closed credential gate.
* feat(kms): implement AppRole login with background renewal and fail-closed expiry
Implement the AppRoleLogin token source (vaultrs approle login +
renew-self) and wire lease-bound credentials through the provider:
- Each successful login/renewal installs a new client generation in the
ArcSwap; in-flight requests finish on the generation they captured.
- A background renewal task refreshes at half the lease TTL: renewable
tokens are renewed in place, everything else (or a failed renewal)
falls back to a fresh login. Auth exchanges run under the typed retry
policy (OpClass::Auth) and failed cycles retry on a fixed cadence, so
the provider recovers once Vault does.
- Fail-closed: current() refuses to hand out a token inside the
configured safety window of its expiry (default: one attempt timeout),
returning CredentialsUnavailable instead of sending a request whose
token may lapse mid-flight.
- Refreshes are single-flight: concurrent triggers for the same
generation coalesce into one login.
- The renewal task's owner handle lives on the KMS service version:
stop() shuts it down explicitly and reconfigure recycles it via
cancel-on-drop when the old version is discarded.
- The secret_id file is re-read on every login attempt; missing or empty
files fail the attempt without contacting Vault. Crate-owned copies of
tokens and secret_ids are zeroized on drop, and Debug output of every
credential-carrying type stays redacted (leak regression tests).
The renewal machinery is covered by paused-clock tests driving a
scripted token source: renew-at-half-TTL timing, login fallback,
fail-closed window entry and recovery, prompt task recycling, and
coalesced concurrent refreshes.
* feat(kms): add Vault Agent token file authentication
Add the TokenFile source: the token is read from an agent-managed sink
file (RUSTFS_KMS_VAULT_TOKEN_FILE or the TokenFile auth config) and
re-read once per poll interval (default 30s) through the existing
renewal loop, so a token rotated by the agent installs a new client
generation within one poll of the atomic replace. Each successful read
extends the token's observed validity to twice the poll interval; a
file that disappears or turns empty keeps failing the refresh until the
fail-closed window trips, and heals the provider as soon as it is
restored.
Reads are strict and never contact Vault on failure: the file must be
non-empty after trimming, and on Unix group/other permission bits are a
hard error (mirroring the SFTP host-key rule). Rotation detection uses
a content digest; the token itself is never stored on the source and
the crate-owned copy is zeroized.
Configuring the token file together with AppRole or an explicit static
token is rejected as a configuration error. All new config fields are
serde(default) and the strict admin-configure deserializer accepts the
new variant.
Covered by paused-clock tests (atomic replacement installs a new
generation next cycle, deletion fails closed and recovers, prompt task
recycling) plus negatives for missing/empty/over-permissive files and a
Debug leak regression.
* docs(kms): add Vault authentication and credential lifecycle runbook
Cover choosing between static token, AppRole, and Vault Agent token
file auth; AppRole role setup with SecretID delivery and rotation;
Agent sink deployment with the permission requirements; and the
fail-closed window semantics with a troubleshooting table keyed on the
renewal task's log lines.
* feat(kms): retain historical master key versions for Vault KV2 rotation
Rotation previously had to be rejected outright because replacing the
stored material would orphan every DEK wrapped by earlier versions.
Vault KV2 now keeps each version's material in an immutable, create-only
record at {prefix}/{key_id}/versions/{N} and treats the top-level record
as the current-version pointer plus fast-path material copy:
- decrypt resolves the envelope's master_key_version to its version
record; a missing version fails closed with KeyVersionNotFound and
never falls back to the current material
- envelopes without a version (pre-versioning writers) resolve to the
baseline_version frozen at the key's first rotation, so never-rotated
keys behave exactly as before
- generate_data_key stamps the wrapping version from the same key record
snapshot that supplied the material
- rotate_key commits in check-and-set order: freeze baseline, persist
the next version's material, then switch the current pointer; any
failure leaves the current pointer untouched, and concurrent rotations
serialize on the CAS writes with monotonically unique versions
- key listings drop the versions/ directory entries and physical key
deletion purges version records before the key record
Refs rustfs/backlog#1565
* test(kms): pin rotation contracts across backends and document retention
Completes the backlog#1565 series with the cross-backend regression net
and operator documentation:
- Vault Transit: ignored integration test proving version-prefixed
historical ciphertext still decrypts after rotation, with no
RustFS-side version bookkeeping in the envelope
- Local: offline mixed-format test interleaving pre-versioning and
versioned envelopes through a rejected rotation; the existing
rotation-rejection pinning test already covers material immutability
- docs: kms-backend-security.md gains the KV2 versioned retention
model, version record retention/destruction preconditions, and the
upgrade-before-first-rotation cluster constraint
Refs rustfs/backlog#1565
Contract-only module for KMS backup/restore (no handler or backend
wiring): versioned manifest schema with completeness marker and sealed
digest, the (backend, at-rest protection) responsibility matrix, typed
fail-closed errors, and the zero-write restore dry-run report. Fields
whose shape depends on in-flight contracts are reserved and reject data
in format version 1.
Rotation previously had to be rejected outright because replacing the
stored material would orphan every DEK wrapped by earlier versions.
Vault KV2 now keeps each version's material in an immutable, create-only
record at {prefix}/{key_id}/versions/{N} and treats the top-level record
as the current-version pointer plus fast-path material copy:
- decrypt resolves the envelope's master_key_version to its version
record; a missing version fails closed with KeyVersionNotFound and
never falls back to the current material
- envelopes without a version (pre-versioning writers) resolve to the
baseline_version frozen at the key's first rotation, so never-rotated
keys behave exactly as before
- generate_data_key stamps the wrapping version from the same key record
snapshot that supplied the material
- rotate_key commits in check-and-set order: freeze baseline, persist
the next version's material, then switch the current pointer; any
failure leaves the current pointer untouched, and concurrent rotations
serialize on the CAS writes with monotonically unique versions
- key listings drop the versions/ directory entries and physical key
deletion purges version records before the key record
Refs rustfs/backlog#1565
* fix(kms): repair vault test call sites missed by the timeout refactor
Three offline tests still constructed VaultKmsClient with the pre-#5472
single-argument signature, leaving cargo test -p rustfs-kms unable to
compile. Pass the same 30s attempt timeout the neighbouring tests use.
* refactor(kms): route Vault clients through a rotatable credential provider
Both Vault backends previously built a VaultClient in their constructor
and held it for the lifetime of the backend, which leaves no seam for
re-authentication: rotating credentials would require tearing down the
whole backend.
Introduce backends/vault_credentials with a TokenSource trait (only
StaticToken for now; AppRole login and agent token files land in
follow-ups) and a VaultCredentialProvider that owns the authenticated
client behind an ArcSwap. Request paths take a per-call snapshot via
current(), so a future rotation swaps in a new client generation without
interrupting calls already in flight. Tokens held by this crate are
zeroized on drop, and Debug output of every credential-carrying type is
redacted (covered by a leak regression test).
Behavior is unchanged: static token, namespace, and per-attempt timeout
feed the same VaultClientSettings as before, and AppRole configurations
are still rejected at construction with the same message.
* fix(kms): restore vault backend test compilation after timeout parameter
PR #5472 added an attempt_timeout parameter to VaultKmsClient::new while
PR #5474 landed tests still using the one-argument form, leaving
'cargo test -p rustfs-kms' unable to compile on main. Pass the same
30-second timeout the surrounding integration tests already use.
* feat(kms): add master key version to data key envelope contract
DataKeyEnvelope gains an optional master_key_version field recording
which KEK version wrapped the DEK, so rotation-aware backends can load
the matching historical material on decrypt. The field is skipped when
None, keeping envelopes from non-rotating backends byte-identical to
the historical seven-field JSON shape, and legacy envelopes without the
field deserialize to None. The envelope discriminator marker is
untouched, so mixed-format routing is unchanged in both directions.
Adds the KeyVersionNotFound typed error for version-addressed material
lookups that must fail closed instead of falling back to the current
version.
Refs rustfs/backlog#1565
* refactor(sse): decouple encryption from ecstore
* feat(kms): enhance KMS service manager with runtime state and persistence support
* feat(kms): add local key export functionality for SSE-S3 migration tests
* fix(kms): keep local key export narrowly scoped
* fix(sse): validate copy source customer algorithm
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Zhengchao An