rustfs/backlog#1636 rejected a built-in rotation scheduler: rotation is a policy decision with a per-backend cost and a hard upgrade-ordering constraint, and a server that rotated on its own would make that decision on an operator's behalf at a moment they did not choose. This is what that issue resolved to deliver instead — the signal, without the actuator.
RUSTFS_KMS_ROTATION_MAX_AGE_SECS names the period. Unset leaves the verdict unreported rather than assuming a policy, because how often keys must be rotated is a compliance decision and a built-in default would report keys as overdue against a rule nobody wrote; an unparsable value is refused the same way, loudly. Values below an hour are raised to it, since a threshold of seconds reports every key as overdue moments after it was rotated and teaches operators to ignore the signal.
KeyInfo gains rotation_due and rotation_due_reason, both additive on the wire and both filled in by the manager rather than by each backend, so no two backends can disagree about what overdue means. A backend that does not advertise rotation reports unsupported and is never reported as due — it must not be told to do something it cannot. A key with no recorded rotation is measured from creation, which is how long its material has actually been in use, and is distinguished from a stale rotation so an operator can tell "overdue again" from "never once". Ages are computed saturating, so a timestamp from a node running ahead cannot manufacture an overdue key.
The verdict is advisory in the strongest sense: nothing consults it before encrypting or decrypting, a key reported as due keeps serving traffic, and readiness is unaffected.
The single-key describe response deliberately does not carry the verdict. Its type records a creation date but no rotation timestamp, so a verdict computed there could not tell a key rotated last week from one never rotated, and reporting never_rotated for a key that was in fact rotated is worse than reporting nothing.
The wraps-based branch the issue also specifies is not implemented: it depends on the per-key wrap accounting that does not exist yet.
A key that cannot be described was handled two incompatible ways. Vault KV2 swallowed every describe failure and dropped the key from the page, so a damaged or newer-format record silently disappeared from the operator's inventory and from the deletion sweep's census. Local failed the whole listing instead, so one bad record stopped every scheduled deletion on the node for as long as the damage lasted. Both force a per-key problem into a whole-page answer.
ListKeysResponse now carries unreadable_key_ids, and the backends that read local key records classify per-key failures in one place: KeyNotFound is a concurrent deletion and is skipped, a material-level error names the key on the page, and anything else fails the listing, because it says nothing about a particular key and reporting it as key damage would turn a backend outage into a false data-loss alarm. A listing that covered the entire key set and found nothing readable still fails, since an empty page there is indistinguishable from a deployment with no keys; the guard is scoped to a page with no successor so a damaged key can never strand the keys behind it. The deletion sweep destroys the expired keys it can read, counts the unreadable ones, and withholds its lifecycle gauges rather than publishing a census over a key set it did not fully see.
Vault Transit needs the same treatment and is easy to miss: its per-key metadata records live in KV2 too, so folding every non-404 failure into a backend error left its per-key classification unreachable and one metadata record written by a newer build still failed every listing on the node.
Vault KV2 record reads gain the typed errors this needs: an unparseable body is MaterialCorrupt and an absent data envelope is MaterialMissing, where both were previously indistinguishable from Vault being unreachable. Only the parse failure's category and position are reported, because serde's own message embeds the offending scalar and that message reaches a log line and an admin HTTP body.
The admin list handlers refuse a malformed limit with 400 instead of silently substituting the default page size, and every page is capped at 1000 where it is cut, so a single request can no longer fan out one metadata lookup per key without bound. The four operation-level KMS metrics gain a backend label, since operation names are shared across backends and a Transit latency regression was previously indistinguishable from an AWS one. The Static backend captures its reported creation date once instead of reading the clock on every describe and list. POST /kms/clear-cache gains a named response type with an unchanged wire shape.
* feat(kms): add data key rewrap and wrapping inspection primitives
Rewrap re-protects an existing data key envelope with the master key's
current version without touching the data key itself, which is the
precondition for ever retiring an older version: until every envelope a
version wrapped has been moved off it, destroying that version orphans
every object whose data key it wrapped.
Adds KmsBackend::rewrap_data_key and its read-only counterpart
describe_data_key_wrapping, both gated by a new BackendCapabilities::rewrap
flag and defaulting to UnsupportedCapability. Vault KV2 unwraps with the
frozen version record that wrapped the envelope and re-wraps with the
current material; Vault Transit uses the native transit/rewrap endpoint so
the data key never enters this process.
No read or write path changes: nothing calls these yet.
* test(kms): cover the rewrap primitive against a scripted Vault
* fix(kms): resolve both key materials before the data key is unwrapped
Keeps every fallible step out of the window in which the plaintext data
key exists, so no error path can drop it without zeroizing it first.
* feat(kms): report configuration references that block a key deletion
Adds a KeyImpactReport that states which configuration still points at a
key, how exhaustively the sources were read, and which sources were not
consulted at all. The report deliberately carries no in-use or
safe-to-delete claim: it covers the configuration layer only, so an empty
reference list means nothing was found in the scanned sources, never that
the key is unreferenced.
Immediate deletion destroys key material without ever reaching the
deletion worker, so it never passed the worker's reference gate. The
manager now consults the same checker on that path and refuses with a
typed KeyStillReferenced error. This only ever adds a refusal; the
scheduled deletion path and the worker's blocking behaviour are
unchanged.
* test(kms): cover the immediate-deletion reference refusal
* feat(kms): surface configuration references on the admin key endpoints
DeleteKey and DescribeKey now return an impact section listing the
configuration that points at the key, so an operator scheduling a
deletion sees what will refuse to destroy the material instead of
learning it from a server-side log once the window has run out.
The section is reported, never acted on: scheduling still succeeds while
references exist, and the deletion worker's gate remains the only thing
that decides whether material is destroyed. An immediate deletion that
the manager refuses for an outstanding reference now answers 409.
* test(kms): pin the impact wire shape and the unreferenced force-delete path
* fix(kms): make the DescribeKey impact section opt-in
Collecting the section lists every bucket, and DescribeKey is polled, so
carrying that fan-out on the default read path trades a hot path's cost
for a diagnostic. It is now collected only for impact=true; without the
parameter the endpoint does exactly the work it did before and returns
no impact field.
A value that is neither true nor false is refused rather than read as
off, so a typo cannot answer a request for the section with a response
that merely lacks one. DeleteKey still reports unconditionally: that is
the request whose consequences the caller cannot otherwise see, and it
is not polled.
* fix(kms): box the query-parse refusal now that responses carry impact
The delete response grew an impact section, which pushed it past the
size clippy accepts inline in a Result. It is a full response body
rather than an error code, so it is boxed at the one place that returns
it as an error; the wire shape and the public field type are unchanged.
* feat(kms): add backup and restore admin API
Wires the merged KMS backup contract, Local export and Local restore into
the admin API: export a sealed bundle, run a zero-write restore preflight,
execute a confirmed restore, roll an interrupted restore back, and report
subsystem readiness.
- Dedicated kms:Backup / kms:Restore actions, recorded in the admin route
matrix. Neither is reachable through any other KMS action.
- Restore requires two independent confirmations: an echo of the bundle
manifest's backup id, and an explicitly named conflict policy (the
default never writes).
- The backup KEK comes from the environment and is refused when it reuses
a secret of the configured backend, compared both as the literal value
and as raw key bytes.
- No endpoint accepts a path: bundles are addressed by a validated name
under a configured root, and the restore target is always the server's
own configured key directory.
- Bundles now carry a sanitized configuration artifact built as an
allowlist projection, so a future backend credential field cannot leak
into a bundle by default. Restore verifies it and never applies it.
- Audit entries go through the existing KMS admin wiring and carry
identifiers only.
* test(kms): pin the backup admin API gates
Fixes the test KEK to a real 32-byte value and drives the export refusal
from the configured backend rather than from the handle that happens to
be available, so a Local handle cannot export on behalf of a backend
whose material RustFS does not own.
* fix(kms): honour the configured metadata cache TTL and metrics switch
KmsManager::new built the KmsCache from cache_config.max_keys alone, so
cache_config.ttl was dead configuration: every deployment ran the
hardcoded 300s window whatever the admin configure API was given, while
CacheSummary and the KMS config endpoint reported the configured value
back. cache_config.enable_metrics was never read anywhere.
Build the cache from the whole CacheConfig. The documented default is
reconciled down to the 300s the cache has always used rather than up to
the advertised 3600s, and now lives in one place (DEFAULT_CACHE_TTL)
instead of being duplicated across the four configure-request
converters, so the default path behaves exactly as before.
Behaviour change: a deployment configured through the admin API already
has ttl 3600 persisted, because the old converters wrote that default
into the stored config, so its describe_key staleness window widens from
an effective 300s to the 3600s it asked for. No cryptographic or
authorization path widens - encrypt, decrypt and generate_data_key go
straight to the backend and never read this cache. The Vault Transit
backend's own metadata cache, which does gate crypto through
ensure_key_state_allows, stays fixed at 300s and is now documented as
deliberately not operator-tunable.
The configured duration now reaches moka's builder, which panics above
1000 years, so CacheConfig::effective_ttl clamps to a 24h maximum the
way effective_timeout already clamps its own, and validate rejects a
zero TTL beside the existing max_keys check. Both config summaries and
the KMS config endpoint report the effective value, so the admin API
cannot advertise a lifetime the cache does not honour.
enable_metrics gates publication of the rustfs_kms_metadata_cache_*
families only; the counters behind the admin status API keep running
either way. No configure-request field sets it yet.
Refs rustfs/backlog#1584
* docs(kms): state why the Transit metadata TTL is not bound to the default
The comment claimed the constant matches config::DEFAULT_CACHE_TTL, which
reads as an invariant the code does not enforce. Say plainly that the
equality is a coincidence rather than a contract, and why binding the two
would be wrong: this cache gates crypto through ensure_key_state_allows,
so a later change to the operator-facing describe-cache default must not
be able to widen its staleness window.
* feat(kms): record real cache hit, miss and eviction metrics
The metadata cache reported (entry_count, 0) because moka exposes no hit
or miss counts, so the miss half of every cache report was a constant.
Track lookups and removals in the cache itself: hit/miss counters on the
lookup path, a moka eviction listener classifying removals by cause, and
an entry gauge refreshed whenever the entry set changes. The counters are
exported through the metrics facade under the rustfs_kms_ prefix with
static label values only, matching the operation-policy metrics, and are
also returned as a KmsCacheStats snapshot in place of the old tuple.
Cache semantics are unchanged: capacity, TTL and invalidation points are
the same, and remove now flushes pending maintenance so the gauge and the
removal notification describe the cache the caller sees.
Refs rustfs/backlog#1584
* fix(kms): report real cache counters through the admin status API
KmsStatusResponse.cache_stats mapped the old (entry_count, 0) tuple onto
hit_count and miss_count, so operators polling KMS status read the entry
count as a hit count and a miss count that was always zero.
Map the fields to the counters they claim to be, and add entry_count and
eviction_count as additive, defaulted fields so the entry number that
hit_count used to carry is still available.
Refs rustfs/backlog#1584
* fix(kms): refresh the cache entry gauge on lookup misses
The entry gauge was published only from the write paths, so an entry
dropped by TTL expiry left `rustfs_kms_metadata_cache_entries` reporting
a population that no longer existed until the next put, remove or clear.
A cache that goes quiet — entries ageing out with no further writes —
kept over-reporting indefinitely.
Republish the gauge from the lookup path when the lookup misses. A miss
is where expiry surfaces, and moka reaps expired entries in the
maintenance it runs during that same lookup, so the count read
afterwards reflects the reaping. Hits stay free of the extra work.
* docs(kms): correct the entry gauge convergence claim on the miss path
The comment on the miss-path gauge refresh said moka reaps expired
entries in the maintenance it runs on that same lookup. It does not:
`should_apply_reads` is gated on a full read log or an elapsed
housekeeping interval, so the removal that decrements `entry_count` and
reaches the eviction listener may land on a later lookup.
The behaviour and the test are unchanged — the gauge still converges,
and the test drives `run_pending_tasks` explicitly rather than riding on
that interval. Only the stated guarantee was wrong, so say interval
instead of same-lookup and record why forcing maintenance on the read
path was not the trade taken.
Add opt-in hotpath feature surfaces to every workspace crate and wire the root rustfs feature passthrough for function, allocation, and CPU profiling.
Add a focused set of function-level measurements for scanner, heal, lock, target replay, IAM, KMS, Keystone, trusted proxy, and capacity paths without adding request-scoped primitive wrappers.
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(kms): add key lifecycle operations to the backend contract
Add enable_key/disable_key/rotate_key to KmsBackend with conservative
defaults returning the typed UnsupportedCapability error, mirroring
remove_expired_key. KmsManager gains matching pass-through methods and
drops cached key metadata after every successful state mutation so the
next describe observes backend truth. The local backend overrides
enable/disable, delegating to its state-machine-gated client methods;
rotation stays rejected, matching its advertised capabilities. New
dedicated policy actions kms:EnableKey and kms:DisableKey complete the
KMS action taxonomy alongside the existing kms:RotateKey.
* feat(admin): add KMS key enable/disable/rotate endpoints
POST /v3/kms/keys/enable, /v3/kms/keys/disable and /v3/kms/keys/rotate,
following the existing /v3/kms/keys handler conventions: key_id body
with keyId query fallback, {success, message, key_id, key_metadata}
responses, and 503 JSON while the KMS service is absent. Error mapping
keeps InvalidOperation/ValidationError at 400 like the sibling handlers
and surfaces UnsupportedCapability as 501 so a backend capability gap is
never mistaken for a missing key. Existing /v3/kms/keys handlers are
untouched apart from a visibility change on a private query helper.
* feat(kms): gate scheduled key deletion on bucket encryption references
Implement the DeletionReferenceChecker seam left by the deletion worker:
before any material is destroyed, every bucket's SSE configuration is
checked for a default KMS key reference and a hit blocks the removal.
The gate fails closed - an unpublished object store, a failed bucket
listing or an unreadable per-bucket encryption config all report a
blocking reference - because destroying key material is irreversible
while a blocked removal is simply retried on the next sweep. Registered
during init_kms_system before the service can start, so every worker
spawn observes it. Storage access goes through a new kms section of the
root storage facade.
* feat(kms): enforce shared key state machine across backends
Unify the key state x operation matrix behind a single gate in
backends/mod.rs and wire it into the Local, Vault KV2 and Vault Transit
backends: Disabled keys reject encryption, data key generation and
rotation while still allowing decryption and lifecycle recovery;
PendingDeletion keys reject everything except decryption and
cancellation (including repeated deletion scheduling); cancellation now
requires an actual pending deletion everywhere. This closes the missing
gates on KV2 encrypt/generate and Local generate_data_key, and stops
enable_key from silently reverting a pending deletion.
Decryption is deliberately left ungated in Disabled/PendingDeletion — an
explicit, documented and tested deviation from AWS KMS, since gating it
would break reads of existing objects the moment a key is disabled.
Add shared contract tests driving the full matrix offline for Local (and
via ignored tests against a live Vault for KV2/Transit), a stateless
contract for Static, an SSE-shaped regression proving existing envelopes
stay decryptable after disable, and a pin on the known-risk Enabled
default of Transit's synthesized metadata fallback.
Refs rustfs/backlog#1571 (part of rustfs/backlog#1562)
* feat(kms): persist deletion deadlines and run a restartable deletion worker (#5491)
* feat(kms): implement secure handling of static KMS secret keys and enhance encryption context validation
* feat: enhance local SSE DEK handling with JSON envelope format and versioning
* feat(kms): implement key management service with local and vault backends
Signed-off-by: junxiang Mu <1948535941@qq.com>
* feat(kms): enhance security with zeroize for sensitive data and improve key management
Signed-off-by: junxiang Mu <1948535941@qq.com>
* remove Hashi word
Signed-off-by: junxiang Mu <1948535941@qq.com>
* refactor: remove unused request structs from kms handlers
Signed-off-by: junxiang Mu <1948535941@qq.com>
---------
Signed-off-by: junxiang Mu <1948535941@qq.com>
Zhengchao An