Documents the procedure the harness automates: what a drill measures and why
the object probe rather than the manifest digest is the acceptance criterion,
the per-backend responsibility split, the disaster matrix, how to read the
evidence bundle, the two interrupted-cutover outcomes, and the Vault variant
whose cryptographic root comes back through Vault's own flow.
Runs one rehearsal from environment configuration and writes the evidence
bundle, exiting non-zero on a failed verdict so a scheduled drill fails its
job instead of filing a bad report. It reads the same backup-KEK variables as
the admin backup API: drilling with the KEK real bundles are sealed under is
what proves that KEK is still retrievable.
Runs the harness against total key-directory loss, salt loss, and a torn key
record, asserting every pre-disaster object decrypts again while work past the
snapshot fence stays lost. Two further legs crash a restore exactly at its
commit point and prove the published marker names the bundle and the files it
still owes, then that re-running rolls forward and aborting rolls back.
The Vault leg needs a real server and is ignored by default: a Vault bundle
never carries the non-exportable Transit root, so what it drills is the
refusal to proceed before the operator has restored it natively.
Rehearse the full backup/restore loop offline and return machine-readable
evidence: seed a sandbox deployment, seal sample objects through the
production encryption path, export a bundle, destroy the persistence layer,
preflight, restore, and decrypt every pre-disaster object again.
The evidence records the measured recovery point (one key is written past the
snapshot fence and must stay unrecoverable), the recovery time by phase, the
manifest digest before and after, and whether the restore treated its bundle
as read-only.
* feat(kms): add backup and restore admin API
Wires the merged KMS backup contract, Local export and Local restore into
the admin API: export a sealed bundle, run a zero-write restore preflight,
execute a confirmed restore, roll an interrupted restore back, and report
subsystem readiness.
- Dedicated kms:Backup / kms:Restore actions, recorded in the admin route
matrix. Neither is reachable through any other KMS action.
- Restore requires two independent confirmations: an echo of the bundle
manifest's backup id, and an explicitly named conflict policy (the
default never writes).
- The backup KEK comes from the environment and is refused when it reuses
a secret of the configured backend, compared both as the literal value
and as raw key bytes.
- No endpoint accepts a path: bundles are addressed by a validated name
under a configured root, and the restore target is always the server's
own configured key directory.
- Bundles now carry a sanitized configuration artifact built as an
allowlist projection, so a future backend credential field cannot leak
into a bundle by default. Restore verifies it and never applies it.
- Audit entries go through the existing KMS admin wiring and carry
identifiers only.
* test(kms): pin the backup admin API gates
Fixes the test KEK to a real 32-byte value and drives the export refusal
from the configured backend rather than from the handle that happens to
be available, so a Local handle cannot export on behalf of a backend
whose material RustFS does not own.
* feat(kms): add link_durably primitive and restore-marker startup guard
The local backend gains the two pieces the bundle restore path builds on:
a no-clobber hard-link publish primitive whose AlreadyExists case is
idempotent only for byte-identical content, and a fail-closed startup
guard that refuses to open a key directory holding a restore cutover
marker. The durable commit protocol and the key-id containment check
become pub(crate) so the restore module reuses them instead of copies.
* feat(kms): record a master-key verifier and pre-seal decrypt probe in export
Fill the manifest's master_key_verifier slot with an opaque one-way
value (scheme-prefixed, bound to the backup id and the KDF salt) so a
restore can detect a wrong operator-supplied master key before touching
any target state, and probe-decrypt every artifact as stored under the
backup KEK before the manifest may seal — digest equality alone only
proves the ciphertext landed intact. The payload decryption tail is
factored out and shared with the restore side so producer and consumer
cannot drift on the framing.
Also drops the stale KmsClient test import orphaned by the backend
refactor (#5501); the test suite did not compile without this.
* feat(kms): restore local backend key material from sealed backup bundles
The consumer side of the Local bundle export, as a four-phase protocol:
- Dry-run: full in-memory bundle decode (digest and AEAD verification of
every artifact), KDF-drift detection against the compiled-in
derivation, deployment and injected-generation checks (strictly lower
is rejected, equal stays allowed for repeated drills), master-key
verifier check, and target conflict enumeration - with zero writes.
- Staging: artifacts are committed durably into the .restore-staging/
subdirectory (invisible to the backend's key scan and orphan-temp
matcher) and every record is decryption-probed with the derived
master key both in memory before staging and again from the staged
bytes.
- Commit marker + cutover: the durably published .restore-commit.json
marker is the single commit point; cutover publishes staged files via
link_durably (salt first, keys after), then durably removes the
marker and drops staging.
- Crash re-entry: before the marker the target top level is untouched
and a re-run starts over; with the marker published, backend startup
fails closed and a re-run with the same bundle rolls forward while
abort_local_restore rolls back. Every interruption converges to the
complete old or complete new state.
Restore never goes through LocalKmsClient::new (which would mint a
fresh salt); the only write mode is the explicit
restore-into-empty-target policy, where an orphan salt or a foreign
marker already counts as non-empty. The bundle source stays strictly
read-only.
Refs rustfs/backlog#1572
* feat(kms): export local backend key material as sealed backup bundles
Adds the producer side of the Local backup series on top of the #5483
contract: a directory-wide export fence gives the snapshot a single
consistent generation, every artifact is AEAD-wrapped under a
caller-supplied backup KEK that is separate from the business trust
hierarchy, and the sealed manifest with completeness marker is written
last so an interrupted export can never be mistaken for a restorable
bundle. Restore and the admin API land in follow-up changes.
* fix(kms): verify manifest digest against raw bytes, not re-serialized fields
The decode path recomputed the digest by re-serializing the parsed
manifest, which silently assumes every field's stored spelling survives
a parse-and-reprint round trip. Timestamps do not guarantee that: the
time zone annotation jiff emits depends on the host (IANA name, POSIX
TZ string, Etc/Unknown), and the legacy-compat parser rewrites
bracket-less spellings to +00:00[UTC]. On CI this made freshly written
bundles fail digest verification while passing locally.
Digest verification now operates on the raw stored bytes, normalized
only through the JSON value layer with the digest slot emptied in
place; parsed typed fields are never re-serialized on the decode path.
Sealing uses the same value-layer canonical form, and the export
additionally pins created_at to UTC so bundles are host-independent. A
regression test seals a manifest whose created_at spelling cannot
round-trip and proves decoding still verifies. One behavior sharpens:
inserting an explicit null reserved slot after sealing is now rejected
as a digest mismatch instead of being tolerated.
* fix(kms): make manifest digest canonicalization independent of map ordering
The canonical digest form serialized serde_json values directly, which
inherits the key order of serde_json's map type: sorted by default, but
insertion-ordered when any crate in the unified build graph enables the
preserve_order feature. The workspace-wide CI build unified that
feature while a per-crate local build did not, so the frozen fixture
digest matched in one environment and not the other — and a bundle
sealed by one build flavor would fail verification in the other.
Canonicalization now rebuilds every JSON object with bytewise-sorted
keys (array order preserved) before hashing, so the digest bytes are
identical regardless of feature unification. Reproduced by enabling
preserve_order in dev-dependencies (fixture test red), then verified
green with the fix under both map flavors.
Contract-only module for KMS backup/restore (no handler or backend
wiring): versioned manifest schema with completeness marker and sealed
digest, the (backend, at-rest protection) responsibility matrix, typed
fail-closed errors, and the zero-write restore dry-run report. Fields
whose shape depends on in-flight contracts are reserved and reject data
in format version 1.
overtrue