* fix(kms): drop stale KmsClient trait import in local_export tests
The KmsClient trait was folded into KmsBackend (#5501), but the backup
export tests merged afterwards (#5499) still imported it, breaking the
crate's test build; create_key is an inherent LocalKmsClient method, so
the import is simply unused.
* fix(kms): make Vault KV2 lifecycle writes check-and-set
Every KV2 lifecycle write used to be a blind whole-record overwrite, so
two nodes racing on the same key could lose updates: a disable racing a
rotation wrote the pre-rotation record back (rolling back the version
and material of a committed rotation), concurrent same-name creates let
the later material win (orphaning DEKs wrapped under the earlier one),
and a cancellation racing the deletion sweep could be overwritten by
the tombstone (or resurrect an already tombstoned key).
All lifecycle mutations now go through a bounded check-and-set
read-modify-write loop: each attempt re-reads the record pinned to its
KV2 secret version, re-runs the state gate against the fresh snapshot,
and writes back check-and-set against exactly that version; after
LIFECYCLE_CAS_ATTEMPTS lost races the typed conflict error surfaces.
The loop composes with the operation policy's single-attempt rule for
non-idempotent writes: each write is still sent at most once, only the
whole read-gate-write cycle repeats. create_key becomes a create-only
write (cas=0) so exactly one of two concurrent creates commits and the
loser reports KeyAlreadyExists. The blind store_key_data primitive is
now test-only.
Reads and rotation additionally fail closed when the version history is
inconsistent: resolving material through a version record above the
current pointer is refused (that state only arises when a lost update
rolled back a committed rotation), and rotation refuses to extend a
history whose records reach more than one step past the current pointer
(one step ahead is the footprint of an interrupted rotation and still
recovers through the adopt path).
Refs rustfs/backlog#1581
* test(kms): add a scripted loopback Vault for policy wiring tests
A minimal HTTP/1.1 responder that serves canned Vault responses in order
and records the method/path sequence, so wiring tests can assert exactly
how many requests a code path performed (retries, read-confirm) without
a live Vault server.
* feat(kms): route Vault operations through the retry policy engine
Wire every outbound vaultrs call in the KV2 and Transit backends through
policy::execute, completing the wiring half of the operation policy work
(the engine landed separately):
- Reads (KV2 read/read_metadata/read_version/list, transit read/list/
encrypt/decrypt, health checks) run as ReadIdempotent: bounded retries
with exponential backoff and jitter on 429, recoverable 5xx, and
connection-level failures; 400/401/403/404 stay fatal.
- Writes (KV2 set/CAS set/delete_metadata, transit create/update/rotate/
delete, metadata writes) run as MutatingNonIdempotent: exactly one
attempt under the per-attempt timeout, never replayed. CAS conflicts
in the rotation protocol pass through unchanged as the concurrency
signal they are.
- Each attempt takes a fresh credential snapshot, so a retry after a
credential rotation uses the new token.
- Read-confirm recovery for lost create responses: when a create finds
an existing key that is exactly what it would have produced (same
algorithm, enabled, usable material, and for request-level creates the
same usage/description/tags), it reports the stored key as the create
result instead of KeyAlreadyExists. Any divergence keeps failing.
- Deletes treat already-deleted records as completed deletes (KV2
version records; transit metadata already did), so re-running an
interrupted deletion converges.
- A failed existence pre-check inside create now fails the create
instead of falling through to a blind overwrite (fail closed).
- The policy module sheds its allow(dead_code) now that it is wired.
Wiring tests run against a scripted loopback Vault and assert request
counts and endpoints for the retry, single-attempt, CAS-conflict, and
read-confirm paths.
Refs rustfs/backlog#1569 (part of rustfs/backlog#1562)
houseme