feat(kms): route Vault operations through the retry policy engine (#5495)

* test(kms): add a scripted loopback Vault for policy wiring tests

A minimal HTTP/1.1 responder that serves canned Vault responses in order
and records the method/path sequence, so wiring tests can assert exactly
how many requests a code path performed (retries, read-confirm) without
a live Vault server.

* feat(kms): route Vault operations through the retry policy engine

Wire every outbound vaultrs call in the KV2 and Transit backends through
policy::execute, completing the wiring half of the operation policy work
(the engine landed separately):

- Reads (KV2 read/read_metadata/read_version/list, transit read/list/
  encrypt/decrypt, health checks) run as ReadIdempotent: bounded retries
  with exponential backoff and jitter on 429, recoverable 5xx, and
  connection-level failures; 400/401/403/404 stay fatal.
- Writes (KV2 set/CAS set/delete_metadata, transit create/update/rotate/
  delete, metadata writes) run as MutatingNonIdempotent: exactly one
  attempt under the per-attempt timeout, never replayed. CAS conflicts
  in the rotation protocol pass through unchanged as the concurrency
  signal they are.
- Each attempt takes a fresh credential snapshot, so a retry after a
  credential rotation uses the new token.
- Read-confirm recovery for lost create responses: when a create finds
  an existing key that is exactly what it would have produced (same
  algorithm, enabled, usable material, and for request-level creates the
  same usage/description/tags), it reports the stored key as the create
  result instead of KeyAlreadyExists. Any divergence keeps failing.
- Deletes treat already-deleted records as completed deletes (KV2
  version records; transit metadata already did), so re-running an
  interrupted deletion converges.
- A failed existence pre-check inside create now fails the create
  instead of falling through to a blind overwrite (fail closed).
- The policy module sheds its allow(dead_code) now that it is wired.

Wiring tests run against a scripted loopback Vault and assert request
counts and endpoints for the retry, single-attempt, CAS-conflict, and
read-confirm paths.

Refs rustfs/backlog#1569 (part of rustfs/backlog#1562)
This commit is contained in:
Zhengchao An
2026-07-31 08:51:02 +08:00
committed by GitHub
parent b4901abd17
commit cad8246ffb
7 changed files with 1029 additions and 175 deletions
+2 -1
View File
@@ -75,7 +75,8 @@ anyhow = { workspace = true }
insta = { workspace = true, features = ["yaml", "json"] }
tempfile = { workspace = true }
temp-env = { workspace = true }
tokio = { workspace = true, features = ["test-util"] }
# "net" backs the scripted loopback Vault used by the policy wiring tests.
tokio = { workspace = true, features = ["net", "test-util"] }
[features]
default = []