A key that cannot be described was handled two incompatible ways. Vault KV2 swallowed every describe failure and dropped the key from the page, so a damaged or newer-format record silently disappeared from the operator's inventory and from the deletion sweep's census. Local failed the whole listing instead, so one bad record stopped every scheduled deletion on the node for as long as the damage lasted. Both force a per-key problem into a whole-page answer.
ListKeysResponse now carries unreadable_key_ids, and the backends that read local key records classify per-key failures in one place: KeyNotFound is a concurrent deletion and is skipped, a material-level error names the key on the page, and anything else fails the listing, because it says nothing about a particular key and reporting it as key damage would turn a backend outage into a false data-loss alarm. A listing that covered the entire key set and found nothing readable still fails, since an empty page there is indistinguishable from a deployment with no keys; the guard is scoped to a page with no successor so a damaged key can never strand the keys behind it. The deletion sweep destroys the expired keys it can read, counts the unreadable ones, and withholds its lifecycle gauges rather than publishing a census over a key set it did not fully see.
Vault Transit needs the same treatment and is easy to miss: its per-key metadata records live in KV2 too, so folding every non-404 failure into a backend error left its per-key classification unreachable and one metadata record written by a newer build still failed every listing on the node.
Vault KV2 record reads gain the typed errors this needs: an unparseable body is MaterialCorrupt and an absent data envelope is MaterialMissing, where both were previously indistinguishable from Vault being unreachable. Only the parse failure's category and position are reported, because serde's own message embeds the offending scalar and that message reaches a log line and an admin HTTP body.
The admin list handlers refuse a malformed limit with 400 instead of silently substituting the default page size, and every page is capped at 1000 where it is cut, so a single request can no longer fan out one metadata lookup per key without bound. The four operation-level KMS metrics gain a backend label, since operation names are shared across backends and a Transit latency regression was previously indistinguishable from an AWS one. The Static backend captures its reported creation date once instead of reading the clock on every describe and list. POST /kms/clear-cache gains a named response type with an unchanged wire shape.
* docs(kms): add the object-side bulk rekey job contract
Define the design contract for the bulk DEK re-wrap job before any of it is
built: work unit granularity, idempotency model, failure semantics, the
exclusion list, the metadata write constraints, and the ownership model.
The job re-wraps envelopes only and never rewrites object bodies, and it
never destroys a superseded key version: a half-finished run is a fully
serviceable state precisely because the old version still decrypts, so
folding destruction into the job would turn a resumable action into
irreversible loss on partial failure.
Records two positions that diverge from the originating request. Pause is
not provided; cancel plus cursor restart plus rate control covers what pause
is actually asked for, and none of the seven existing long-job frameworks
has a pause state. And the KEK version a given envelope was sealed under is
not observable today, from object metadata or from the decrypt response, so
recognizing the target state requires the re-wrap primitive to record a
version witness - stated here as the one interface both sides must agree on.
Refs rustfs/backlog#1642 (part of rustfs/backlog#1562)
* docs(kms): correct how the wrapping KEK version is recovered
The first draft claimed the wrapping KEK version was not observable at all,
and built the idempotent-skip and completion-evidence conclusions on that.
It is observable for every backend that rotates, so that framing was wrong.
The sealed blob under x-rustfs-encryption-key is the base64 of the backend
ciphertext, which is DataKeyEnvelope JSON; the read path in sse.rs already
discriminates on it via is_data_key_envelope. Vault KV2 records the version
in the envelope's master_key_version, and Transit leaves that field None on
purpose because its ciphertext self-describes as vault:vN:. Local and Static
hardcode None because neither rotates. Only AWS is genuinely opaque.
So the skip is achievable, and the honest statement is its cost: a base64
decode plus a JSON parse per scanned work unit, which belongs in the rate
budget. AWS becomes a scope-admission refusal instead of a silent
every-object rewrite on re-run.
Two traps replace the old over-broad claim. A bare None means three
different things across backends, so version extraction must be dispatched
by backend. And Local omits the version precisely because rotation is
rejected there, which couples it to backlog#1565: whichever change gives
Local a rotation history must start recording the version in the same
change, or Local joins AWS in the unreadable column.
The requirement left on the re-wrap primitive shrinks accordingly, from
"record a version" to exposing one backend-dispatched accessor and
reporting "already at target state" as a distinct outcome.
Refs rustfs/backlog#1642 (part of rustfs/backlog#1562)
Zhengchao An