* feat(kms): add a disaster-recovery drill harness for the Local backend
Rehearse the full backup/restore loop offline and return machine-readable
evidence: seed a sandbox deployment, seal sample objects through the
production encryption path, export a bundle, destroy the persistence layer,
preflight, restore, and decrypt every pre-disaster object again.
The evidence records the measured recovery point (one key is written past the
snapshot fence and must stay unrecoverable), the recovery time by phase, the
manifest digest before and after, and whether the restore treated its bundle
as read-only.
* test(kms): drill the Local disaster matrix and the interrupted cutover
Runs the harness against total key-directory loss, salt loss, and a torn key
record, asserting every pre-disaster object decrypts again while work past the
snapshot fence stays lost. Two further legs crash a restore exactly at its
commit point and prove the published marker names the bundle and the files it
still owes, then that re-running rolls forward and aborting rolls back.
The Vault leg needs a real server and is ignored by default: a Vault bundle
never carries the non-exportable Transit root, so what it drills is the
refusal to proceed before the operator has restored it natively.
* feat(kms): add an operator entry point for the disaster-recovery drill
Runs one rehearsal from environment configuration and writes the evidence
bundle, exiting non-zero on a failed verdict so a scheduled drill fails its
job instead of filing a bad report. It reads the same backup-KEK variables as
the admin backup API: drilling with the KEK real bundles are sealed under is
what proves that KEK is still retrievable.
* docs(kms): add the disaster-recovery drill runbook
Documents the procedure the harness automates: what a drill measures and why
the object probe rather than the manifest digest is the acceptance criterion,
the per-backend responsibility split, the disaster matrix, how to read the
evidence bundle, the two interrupted-cutover outcomes, and the Vault variant
whose cryptographic root comes back through Vault's own flow.
* chore(typos): accept RTO as a disaster-recovery term
* feat(kms): record real cache hit, miss and eviction metrics
The metadata cache reported (entry_count, 0) because moka exposes no hit
or miss counts, so the miss half of every cache report was a constant.
Track lookups and removals in the cache itself: hit/miss counters on the
lookup path, a moka eviction listener classifying removals by cause, and
an entry gauge refreshed whenever the entry set changes. The counters are
exported through the metrics facade under the rustfs_kms_ prefix with
static label values only, matching the operation-policy metrics, and are
also returned as a KmsCacheStats snapshot in place of the old tuple.
Cache semantics are unchanged: capacity, TTL and invalidation points are
the same, and remove now flushes pending maintenance so the gauge and the
removal notification describe the cache the caller sees.
Refs rustfs/backlog#1584
* fix(kms): report real cache counters through the admin status API
KmsStatusResponse.cache_stats mapped the old (entry_count, 0) tuple onto
hit_count and miss_count, so operators polling KMS status read the entry
count as a hit count and a miss count that was always zero.
Map the fields to the counters they claim to be, and add entry_count and
eviction_count as additive, defaulted fields so the entry number that
hit_count used to carry is still available.
Refs rustfs/backlog#1584
* fix(kms): refresh the cache entry gauge on lookup misses
The entry gauge was published only from the write paths, so an entry
dropped by TTL expiry left `rustfs_kms_metadata_cache_entries` reporting
a population that no longer existed until the next put, remove or clear.
A cache that goes quiet — entries ageing out with no further writes —
kept over-reporting indefinitely.
Republish the gauge from the lookup path when the lookup misses. A miss
is where expiry surfaces, and moka reaps expired entries in the
maintenance it runs during that same lookup, so the count read
afterwards reflects the reaping. Hits stay free of the extra work.
* docs(kms): correct the entry gauge convergence claim on the miss path
The comment on the miss-path gauge refresh said moka reaps expired
entries in the maintenance it runs on that same lookup. It does not:
`should_apply_reads` is gated on a full read log or an elapsed
housekeeping interval, so the removal that decrements `entry_count` and
reaches the eviction listener may land on a later lookup.
The behaviour and the test are unchanged — the gauge still converges,
and the test drives `run_pending_tasks` explicitly rather than riding on
that interval. Only the stated guarantee was wrong, so say interval
instead of same-lookup and record why forcing maintenance on the read
path was not the trade taken.
* refactor(sse): decouple encryption from ecstore
* feat(kms): enhance KMS service manager with runtime state and persistence support
* feat(kms): add local key export functionality for SSE-S3 migration tests
* fix(kms): keep local key export narrowly scoped
* fix(sse): validate copy source customer algorithm
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Zhengchao An