KMSKeyAdministrator, KMSKeyUser and KMSAuditor ship as canned identity
policies so operators can express KMS role separation without hand-writing
the resource grammar. They grant only kms actions, so they compose with an
existing data-plane policy, and none of them confers kms:Configure,
kms:ServiceControl, kms:ClearCache, kms:Backup or kms:Restore.
* feat(kms): add admin endpoints for key description and tag updates
Wire the KMS key metadata updates landed at the service layer to the admin
API: POST /v3/kms/keys/{update-description,tag,untag}. Each endpoint gates on
a dedicated KMS action scoped to the key its body names, records a handler-owned
audit entry for both the authorization denial and the outcome, and maps a
backend that cannot update key metadata to 501 rather than 404.
Refs rustfs/backlog#1586 (part of rustfs/backlog#1562)
* fix(admin): audit metadata attempts refused by an unavailable KMS
* test(admin): register the new KMS metadata routes in the matrix
EventName::mask() turns a leaf variant's discriminant `v` into
`1 << (v - 1)`, so the enum can hold at most 64 variants in total —
compound "All" variants included, since they consume discriminants
even though they own no bit and push every later leaf further up the
range. The last variant is KmsServiceStopped at 61, leaving 3 slots.
Past the budget, `1u64 << (v - 1)` shifts by 64 or more: debug builds
panic with a shift overflow, release builds silently mask the shift
amount down and hand back a bit that already belongs to another event.
Either failure surfaces far from the line that added the variant.
The existing test_mask_bit_budget_is_not_exhausted only bounds the
events listed in the test-local ALL_EVENT_NAMES, so a variant nobody
remembered to list went unchecked. Add three layers instead:
- A const assertion on LAST_EVENT_NAME_VALUE, anchored on the last
variant, that fails `cargo build` once the discriminant passes 64.
- An exhaustive-match tripwire next to ALL_EVENT_NAMES, so adding a
variant breaks compilation with a non-exhaustive-patterns error that
points at the budget notes.
- Tests pinning that discriminants stay dense and fully listed, that
every leaf mask is non-zero and owns a unique bit (catching the
release-mode wrapped-shift collision), and that the last variant
still lands inside the u64.
Document the budget and the three guards on mask() so the next person
adding an event sees the remaining headroom.
Refs: rustfs/backlog#1572
* feat(kms): add backup and restore admin API
Wires the merged KMS backup contract, Local export and Local restore into
the admin API: export a sealed bundle, run a zero-write restore preflight,
execute a confirmed restore, roll an interrupted restore back, and report
subsystem readiness.
- Dedicated kms:Backup / kms:Restore actions, recorded in the admin route
matrix. Neither is reachable through any other KMS action.
- Restore requires two independent confirmations: an echo of the bundle
manifest's backup id, and an explicitly named conflict policy (the
default never writes).
- The backup KEK comes from the environment and is refused when it reuses
a secret of the configured backend, compared both as the literal value
and as raw key bytes.
- No endpoint accepts a path: bundles are addressed by a validated name
under a configured root, and the restore target is always the server's
own configured key directory.
- Bundles now carry a sanitized configuration artifact built as an
allowlist projection, so a future backend credential field cannot leak
into a bundle by default. Restore verifies it and never applies it.
- Audit entries go through the existing KMS admin wiring and carry
identifiers only.
* test(kms): pin the backup admin API gates
Fixes the test KEK to a real 32-byte value and drives the export refusal
from the configured backend rather than from the handle that happens to
be available, so a Local handle cannot export on behalf of a backend
whose material RustFS does not own.
`VaultKeyData::baseline_version` pins the master key version that every
pre-versioning DEK envelope (one with no `master_key_version`) resolves to.
Builds released before versioned rotation do not know the field, and every KV2
lifecycle write — enable, disable, schedule/cancel deletion, tag, untag —
rewrites the whole record, so a single lifecycle call from an older node during
a rolling upgrade silently drops the baseline. Serde attributes cannot prevent
this: the code doing the dropping already shipped.
The loss is only latent until the next rotation. With the baseline gone,
rotation takes the first-rotation path again and freezes a *new* baseline at the
current version, so every legacy envelope permanently resolves to material that
never wrapped it. That is the point of no return, and it is the one this commit
blocks: rotation now lists the key's immutable version records first and refuses
when records exist while the record carries no baseline. Those two are created
by the same commit, so the combination can only mean the baseline was erased
afterwards. The refusal is decided from reads alone, before any write, and names
the version to restore — the oldest recorded version *is* the lost baseline,
since version records start at the baseline the first rotation froze.
Reads are diagnosed rather than blocked. A version-less envelope on a key with
no baseline resolves to the current version, which AES-256-GCM refuses to
unwrap when it is the wrong one, so no wrong plaintext can be returned. Only
after that failure does decrypt list the version records and re-report the
failure as the lost baseline. Refusing up front on the same evidence would break
reads that work today: an older node writes version-less envelopes wrapped with
whatever material is current, and those still decrypt. This also keeps the extra
listing off every read of pre-versioning data.
Self-healing (writing back `baseline_version = min(recorded)`) is deliberately
not done: during the mixed-version window that caused the loss, an older node
can erase it again on the next lifecycle call, so healing would mask an
unfinished upgrade instead of surfacing it. Moving the baseline to a KV path
older builds cannot rewrite is the real fix and is scheduled for GA.
Refs rustfs/backlog#1581 (part of rustfs/backlog#1562)
* fix(kms): honour the configured metadata cache TTL and metrics switch
KmsManager::new built the KmsCache from cache_config.max_keys alone, so
cache_config.ttl was dead configuration: every deployment ran the
hardcoded 300s window whatever the admin configure API was given, while
CacheSummary and the KMS config endpoint reported the configured value
back. cache_config.enable_metrics was never read anywhere.
Build the cache from the whole CacheConfig. The documented default is
reconciled down to the 300s the cache has always used rather than up to
the advertised 3600s, and now lives in one place (DEFAULT_CACHE_TTL)
instead of being duplicated across the four configure-request
converters, so the default path behaves exactly as before.
Behaviour change: a deployment configured through the admin API already
has ttl 3600 persisted, because the old converters wrote that default
into the stored config, so its describe_key staleness window widens from
an effective 300s to the 3600s it asked for. No cryptographic or
authorization path widens - encrypt, decrypt and generate_data_key go
straight to the backend and never read this cache. The Vault Transit
backend's own metadata cache, which does gate crypto through
ensure_key_state_allows, stays fixed at 300s and is now documented as
deliberately not operator-tunable.
The configured duration now reaches moka's builder, which panics above
1000 years, so CacheConfig::effective_ttl clamps to a 24h maximum the
way effective_timeout already clamps its own, and validate rejects a
zero TTL beside the existing max_keys check. Both config summaries and
the KMS config endpoint report the effective value, so the admin API
cannot advertise a lifetime the cache does not honour.
enable_metrics gates publication of the rustfs_kms_metadata_cache_*
families only; the counters behind the admin status API keep running
either way. No configure-request field sets it yet.
Refs rustfs/backlog#1584
* docs(kms): state why the Transit metadata TTL is not bound to the default
The comment claimed the constant matches config::DEFAULT_CACHE_TTL, which
reads as an invariant the code does not enforce. Say plainly that the
equality is a coincidence rather than a contract, and why binding the two
would be wrong: this cache gates crypto through ensure_key_state_allows,
so a later change to the operator-facing describe-cache default must not
be able to widen its staleness window.
Emit zero tombstones for removed bucket-level replication backlog series and retire the corresponding metric descriptors after the configured tombstone window, matching the existing replication bandwidth lifecycle.
Co-authored-by: heihutu <heihutu@gmail.com>
test(replication): distinguish backlog from failures
Extend the replication backlog e2e to assert that historical failed counters can remain non-zero after recovery while current backlog and MRF pending gauges settle to zero.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(s3-types): add KMS service-control audit events
Configuration changes and service start/stop are management-plane actions
with no event name of their own, so they could not reach the audit
pipeline at all. Append three variants for them, following the existing
rule that KMS events are audit-only and live outside the `s3:` namespace,
so no bucket notification selector can expand to them.
* feat(admin): audit KMS management operations
Every KMS admin endpoint now builds an OperationContext from the
authenticated caller and hands it to the KMS layer, so the record the
manager already produces carries the principal, source address and
canonical request id instead of the internal placeholder.
A new adapter maps those records onto the server's existing AuditEntry
format and installs itself as the KMS audit sink at service assembly, so
KMS activity reaches the targets a deployment already operates. The
handlers emit directly for what the KMS layer cannot see: a request the
authorization gate rejects, and the endpoints with no context-aware KMS
entry point (data-key derivation and service control).
Only the failure class is recorded, never the error message, and the new
module joins the logging guardrail's checked files alongside the handlers
it serves.
Add RAII guards for replication runtime backlog tickets so active worker and queue counters unwind on every terminal path.
Expose node-local MRF pending, dropped, missed, and flush-failure metrics through the bucket replication Prometheus collector while keeping the existing current backlog and durable MRF gauges additive.
Update durable MRF summary maintenance to aggregate incrementally during the persister loop, avoiding repeated full-entry scans on each successful flush.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(kms): observe key lifecycle from the deletion sweep
The sweep already pages through the whole key set, so the lifecycle
gauges come out of the pages it has in hand: no extra backend call is
made for them. It publishes the number of keys awaiting their deletion
deadline, the number of tombstones an interrupted removal left behind,
and how long ago the least recently rotated usable key was rotated
(counting from creation for keys that were never rotated), plus a
per-outcome counter of what the sweep acted on.
Every gauge is a label-less aggregate: a per-key label would carry key
identifiers into the metric stream and grow the series count with the
key set, so "this key is overdue for rotation" stays a threshold for an
alerting rule to apply to the aggregate. Keys the sweep destroys drop
out of the census, and a sweep that could not finish listing leaves the
gauges at their last complete values rather than understating them.
* feat(kms): expose Vault token TTL and fail-closed state as gauges
The renewal loop already tracks token expiry, so it now publishes the
seconds left on the active Vault token and whether the provider is
refusing to serve it. The fail-closed gauge re-evaluates the very gate
`VaultCredentialProvider::current` applies, so what operators see and
what the request path does cannot drift apart.
Both waits in the loop republish on a bounded cadence, so a scrape
landing between refresh cycles never reads a TTL frozen at the last
refresh or a fail-closed state that flipped after it. That costs a timer
and no Vault traffic, and the request path stays free of metric work.
Renewal successes and failures already land in the auth operation
counters, so nothing is double-counted here. Neither gauge carries a
label: the address, mount, auth path and token are all off limits as
label values, and there is one generation to describe.
* feat(kms): record real cache hit, miss and eviction metrics
The metadata cache reported (entry_count, 0) because moka exposes no hit
or miss counts, so the miss half of every cache report was a constant.
Track lookups and removals in the cache itself: hit/miss counters on the
lookup path, a moka eviction listener classifying removals by cause, and
an entry gauge refreshed whenever the entry set changes. The counters are
exported through the metrics facade under the rustfs_kms_ prefix with
static label values only, matching the operation-policy metrics, and are
also returned as a KmsCacheStats snapshot in place of the old tuple.
Cache semantics are unchanged: capacity, TTL and invalidation points are
the same, and remove now flushes pending maintenance so the gauge and the
removal notification describe the cache the caller sees.
Refs rustfs/backlog#1584
* fix(kms): report real cache counters through the admin status API
KmsStatusResponse.cache_stats mapped the old (entry_count, 0) tuple onto
hit_count and miss_count, so operators polling KMS status read the entry
count as a hit count and a miss count that was always zero.
Map the fields to the counters they claim to be, and add entry_count and
eviction_count as additive, defaulted fields so the entry number that
hit_count used to carry is still available.
Refs rustfs/backlog#1584
* fix(kms): refresh the cache entry gauge on lookup misses
The entry gauge was published only from the write paths, so an entry
dropped by TTL expiry left `rustfs_kms_metadata_cache_entries` reporting
a population that no longer existed until the next put, remove or clear.
A cache that goes quiet — entries ageing out with no further writes —
kept over-reporting indefinitely.
Republish the gauge from the lookup path when the lookup misses. A miss
is where expiry surfaces, and moka reaps expired entries in the
maintenance it runs during that same lookup, so the count read
afterwards reflects the reaping. Hits stay free of the extra work.
* docs(kms): correct the entry gauge convergence claim on the miss path
The comment on the miss-path gauge refresh said moka reaps expired
entries in the maintenance it runs on that same lookup. It does not:
`should_apply_reads` is gated on a full read log or an elapsed
housekeeping interval, so the removal that decrements `entry_count` and
reaches the eviction listener may land on a later lookup.
The behaviour and the test are unchanged — the gauge still converges,
and the test drives `run_pending_tasks` explicitly rather than riding on
that interval. Only the stated guarantee was wrong, so say interval
instead of same-lookup and record why forcing maintenance on the read
path was not the trade taken.
A streaming GET holds snapshot leases on the object's data directories,
and DeleteObjects defers their physical cleanup until the leases are
released. A DeleteBucket issued inside that window passes the xl.meta
emptiness check but fails closed in the non-force delete_volume tree
removal on the leftover part files, returning BucketNotEmpty for a
logically empty bucket.
This is what intermittently failed the s3tests
test_encryption_sse_c_multipart_bad_download teardown in CI: the test
never reads its 30MiB GET body, so the server-side stream (and its
leases) stays alive until the connection drops, racing the teardown's
DeleteObjects + DeleteBucket sequence. With the body held open the
failure reproduces 5/5 locally; after this change it passes 20/20, and
the real s3-tests case passes 20 consecutive runs.
delete_volume now executes the registry-tracked pending deferred
deletions for the volume before removing the directory tree. Only data
dirs whose logical delete already committed are touched; unknown files
still fail closed with VolumeNotEmpty.
* feat(kms): add link_durably primitive and restore-marker startup guard
The local backend gains the two pieces the bundle restore path builds on:
a no-clobber hard-link publish primitive whose AlreadyExists case is
idempotent only for byte-identical content, and a fail-closed startup
guard that refuses to open a key directory holding a restore cutover
marker. The durable commit protocol and the key-id containment check
become pub(crate) so the restore module reuses them instead of copies.
* feat(kms): record a master-key verifier and pre-seal decrypt probe in export
Fill the manifest's master_key_verifier slot with an opaque one-way
value (scheme-prefixed, bound to the backup id and the KDF salt) so a
restore can detect a wrong operator-supplied master key before touching
any target state, and probe-decrypt every artifact as stored under the
backup KEK before the manifest may seal — digest equality alone only
proves the ciphertext landed intact. The payload decryption tail is
factored out and shared with the restore side so producer and consumer
cannot drift on the framing.
Also drops the stale KmsClient test import orphaned by the backend
refactor (#5501); the test suite did not compile without this.
* feat(kms): restore local backend key material from sealed backup bundles
The consumer side of the Local bundle export, as a four-phase protocol:
- Dry-run: full in-memory bundle decode (digest and AEAD verification of
every artifact), KDF-drift detection against the compiled-in
derivation, deployment and injected-generation checks (strictly lower
is rejected, equal stays allowed for repeated drills), master-key
verifier check, and target conflict enumeration - with zero writes.
- Staging: artifacts are committed durably into the .restore-staging/
subdirectory (invisible to the backend's key scan and orphan-temp
matcher) and every record is decryption-probed with the derived
master key both in memory before staging and again from the staged
bytes.
- Commit marker + cutover: the durably published .restore-commit.json
marker is the single commit point; cutover publishes staged files via
link_durably (salt first, keys after), then durably removes the
marker and drops staging.
- Crash re-entry: before the marker the target top level is untouched
and a re-run starts over; with the marker published, backend startup
fails closed and a re-run with the same bundle rolls forward while
abort_local_restore rolls back. Every interruption converges to the
complete old or complete new state.
Restore never goes through LocalKmsClient::new (which would mint a
fresh salt); the only write mode is the explicit
restore-into-empty-target policy, where an orphan salt or a foreign
marker already counts as non-empty. The bundle source stays strictly
read-only.
Refs rustfs/backlog#1572
Bring the dormant Resource::Kms variant to life: arn:aws:kms:::key/<key_id>
patterns (empty-account form, wildcards allowed in the id, alias/<name>
reserved as parse-only) now parse, validate, serialize back, and are matched
by KMS statements against the requested key id carried in Args::object.
Statements without KMS resources keep the legacy match-every-key behaviour,
as do call sites that pass no key resource, so nothing changes until the
admin/SSE authorization paths start passing key ids.
Statement validation rejects KMS resources on non-KMS statements, and bucket
policy validation rejects KMS actions and resources outright while stored
policies keep deserializing; evaluation skips pure-KMS bucket policy
statements with a warning. A kms:Decrypt action is added for the upcoming
SSE-KMS read path.
Refs rustfs/backlog#1582 (part of rustfs/backlog#1562)
* fix(kms): drop the KmsClient trait import removed by #5501
The local backup export tests (#5499) merged after #5501 folded the
KmsClient trait into KmsBackend, leaving a dead trait import that breaks
'cargo test -p rustfs-kms' compilation on main. create_key is an inherent
method on LocalKmsClient since #5501, so the import is unnecessary.
* fix(kms): CAS Transit metadata writes and bound the metadata cache
Transit KV metadata writes were whole-record overwrites with no
precondition, so two nodes mutating the same key could silently clobber
each other's lifecycle state, and the process-local metadata cache had
neither a TTL nor a capacity bound, so a key disabled or scheduled for
deletion on one node stayed usable on every other node until restart.
- Replace write_metadata_to_kv with a versioned read
(read_metadata_from_kv_versioned) plus a check-and-set write
(cas_write_metadata_to_kv); mutate_key_metadata re-reads the
authoritative record and re-runs the state gate on every attempt, and a
lost CAS race retries with a fresh snapshot (bounded budget) instead of
replaying the stale one.
- Migrate every read-modify-write caller: enable, disable, schedule and
cancel deletion, rotate version bump, the expired-key tombstone, and
both create paths (create-only CAS that read-confirms the winner on a
lost race).
- Bound the metadata cache with moka (300s TTL, 1024 entries) and drop a
key's entry when a transit data call reports it gone server-side.
- Fail closed when the synthesized-metadata fallback cannot be read or
persisted: the fabricated Enabled record is only served after a durable
create-only CAS write, closing the gate weakening documented as a KNOWN
RISK; the persistence fallback for pre-metadata keys is kept.
Refs rustfs/backlog#1581 (part of rustfs/backlog#1562)
* fix(kms): drop stale KmsClient trait import in local_export tests
The KmsClient trait was folded into KmsBackend (#5501), but the backup
export tests merged afterwards (#5499) still imported it, breaking the
crate's test build; create_key is an inherent LocalKmsClient method, so
the import is simply unused.
* fix(kms): make Vault KV2 lifecycle writes check-and-set
Every KV2 lifecycle write used to be a blind whole-record overwrite, so
two nodes racing on the same key could lose updates: a disable racing a
rotation wrote the pre-rotation record back (rolling back the version
and material of a committed rotation), concurrent same-name creates let
the later material win (orphaning DEKs wrapped under the earlier one),
and a cancellation racing the deletion sweep could be overwritten by
the tombstone (or resurrect an already tombstoned key).
All lifecycle mutations now go through a bounded check-and-set
read-modify-write loop: each attempt re-reads the record pinned to its
KV2 secret version, re-runs the state gate against the fresh snapshot,
and writes back check-and-set against exactly that version; after
LIFECYCLE_CAS_ATTEMPTS lost races the typed conflict error surfaces.
The loop composes with the operation policy's single-attempt rule for
non-idempotent writes: each write is still sent at most once, only the
whole read-gate-write cycle repeats. create_key becomes a create-only
write (cas=0) so exactly one of two concurrent creates commits and the
loser reports KeyAlreadyExists. The blind store_key_data primitive is
now test-only.
Reads and rotation additionally fail closed when the version history is
inconsistent: resolving material through a version record above the
current pointer is refused (that state only arises when a lost update
rolled back a committed rotation), and rotation refuses to extend a
history whose records reach more than one step past the current pointer
(one step ahead is the footprint of an interrupted rotation and still
recovers through the adopt path).
Refs rustfs/backlog#1581
* feat(kms): export local backend key material as sealed backup bundles
Adds the producer side of the Local backup series on top of the #5483
contract: a directory-wide export fence gives the snapshot a single
consistent generation, every artifact is AEAD-wrapped under a
caller-supplied backup KEK that is separate from the business trust
hierarchy, and the sealed manifest with completeness marker is written
last so an interrupted export can never be mistaken for a restorable
bundle. Restore and the admin API land in follow-up changes.
* fix(kms): verify manifest digest against raw bytes, not re-serialized fields
The decode path recomputed the digest by re-serializing the parsed
manifest, which silently assumes every field's stored spelling survives
a parse-and-reprint round trip. Timestamps do not guarantee that: the
time zone annotation jiff emits depends on the host (IANA name, POSIX
TZ string, Etc/Unknown), and the legacy-compat parser rewrites
bracket-less spellings to +00:00[UTC]. On CI this made freshly written
bundles fail digest verification while passing locally.
Digest verification now operates on the raw stored bytes, normalized
only through the JSON value layer with the digest slot emptied in
place; parsed typed fields are never re-serialized on the decode path.
Sealing uses the same value-layer canonical form, and the export
additionally pins created_at to UTC so bundles are host-independent. A
regression test seals a manifest whose created_at spelling cannot
round-trip and proves decoding still verifies. One behavior sharpens:
inserting an explicit null reserved slot after sealing is now rejected
as a digest mismatch instead of being tolerated.
* fix(kms): make manifest digest canonicalization independent of map ordering
The canonical digest form serialized serde_json values directly, which
inherits the key order of serde_json's map type: sorted by default, but
insertion-ordered when any crate in the unified build graph enables the
preserve_order feature. The workspace-wide CI build unified that
feature while a per-crate local build did not, so the frozen fixture
digest matched in one environment and not the other — and a bundle
sealed by one build flavor would fail verification in the other.
Canonicalization now rebuilds every JSON object with bytewise-sorted
keys (array order preserved) before hashing, so the digest bytes are
identical regardless of feature unification. Reproduced by enabling
preserve_order in dev-dependencies (fixture test red), then verified
green with the fix under both map flavors.
Add opt-in hotpath feature surfaces to every workspace crate and wire the root rustfs feature passthrough for function, allocation, and CPU profiling.
Add a focused set of function-level measurements for scanner, heal, lock, target replay, IAM, KMS, Keystone, trusted proxy, and capacity paths without adding request-scoped primitive wrappers.
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(kms): record operation metrics in the retry policy engine
Instrument policy::execute — the single choke point every outbound Vault
call and credential exchange already flows through — so no call site
needs its own instrumentation:
- rustfs_kms_backend_operations_total (counter): operation, op_class,
outcome (success / fatal / budget_exhausted / deadline_exceeded /
cancelled)
- rustfs_kms_backend_attempt_failures_total (counter): operation,
error_class (retryable_conn / retryable_status / fatal /
attempt_timeout)
- rustfs_kms_backend_operation_duration_seconds (histogram): wall-clock
duration including retries and backoff
- rustfs_kms_backend_operation_attempts (histogram): attempts used
Metric labels carry only static enum values (operation names, classes,
outcomes) — never key identifiers, key material, ciphertext, or tokens.
Emission goes through the process-global metrics facade recorder, the
same pattern the rest of the workspace uses, so no new wiring is needed
in rustfs/src.
Tests drive a paused-clock runtime under a thread-local debugging
recorder, so counts, attempts, and even the recorded (virtual-clock)
durations are asserted deterministically with zero real sleeps.
Refs rustfs/backlog#1569 (part of rustfs/backlog#1562)
* test(kms): add Vault fault-injection matrix
Offline cases inject transport faults locally and are fully
deterministic: a refused connection is retried up to the configured
budget, and a stalled connection is cut off by the per-attempt timeout
instead of hanging. Ignored cases run against a real dev Vault
(RUSTFS_KMS_VAULT_ADDR) and pin the fail-closed auth behavior: an
invalid token and a missing key each resolve in exactly one attempt.
Every case asserts through the policy metrics recorded by a
thread-local debugging recorder, which doubles as the request-count
assertion even against a real server. Throttling and recoverable 5xx
responses cannot be forced on a stock dev Vault; those paths stay
pinned by the scripted-Vault wiring tests and the engine tests.
Refs rustfs/backlog#1569 (part of rustfs/backlog#1562)
overtrue