feat(kms): restore local backend key material from sealed backup bundles (#5522)

* feat(kms): add link_durably primitive and restore-marker startup guard

The local backend gains the two pieces the bundle restore path builds on:
a no-clobber hard-link publish primitive whose AlreadyExists case is
idempotent only for byte-identical content, and a fail-closed startup
guard that refuses to open a key directory holding a restore cutover
marker. The durable commit protocol and the key-id containment check
become pub(crate) so the restore module reuses them instead of copies.

* feat(kms): record a master-key verifier and pre-seal decrypt probe in export

Fill the manifest's master_key_verifier slot with an opaque one-way
value (scheme-prefixed, bound to the backup id and the KDF salt) so a
restore can detect a wrong operator-supplied master key before touching
any target state, and probe-decrypt every artifact as stored under the
backup KEK before the manifest may seal — digest equality alone only
proves the ciphertext landed intact. The payload decryption tail is
factored out and shared with the restore side so producer and consumer
cannot drift on the framing.

Also drops the stale KmsClient test import orphaned by the backend
refactor (#5501); the test suite did not compile without this.

* feat(kms): restore local backend key material from sealed backup bundles

The consumer side of the Local bundle export, as a four-phase protocol:

- Dry-run: full in-memory bundle decode (digest and AEAD verification of
  every artifact), KDF-drift detection against the compiled-in
  derivation, deployment and injected-generation checks (strictly lower
  is rejected, equal stays allowed for repeated drills), master-key
  verifier check, and target conflict enumeration - with zero writes.
- Staging: artifacts are committed durably into the .restore-staging/
  subdirectory (invisible to the backend's key scan and orphan-temp
  matcher) and every record is decryption-probed with the derived
  master key both in memory before staging and again from the staged
  bytes.
- Commit marker + cutover: the durably published .restore-commit.json
  marker is the single commit point; cutover publishes staged files via
  link_durably (salt first, keys after), then durably removes the
  marker and drops staging.
- Crash re-entry: before the marker the target top level is untouched
  and a re-run starts over; with the marker published, backend startup
  fails closed and a re-run with the same bundle rolls forward while
  abort_local_restore rolls back. Every interruption converges to the
  complete old or complete new state.

Restore never goes through LocalKmsClient::new (which would mint a
fresh salt); the only write mode is the explicit
restore-into-empty-target policy, where an orphan salt or a foreign
marker already counts as non-empty. The bundle source stays strictly
read-only.

Refs rustfs/backlog#1572
This commit is contained in:
Zhengchao An
2026-08-01 10:05:47 +08:00
committed by GitHub
parent b6d4689c75
commit 739efaaea1
4 changed files with 2217 additions and 30 deletions
+9 -3
View File
@@ -16,9 +16,10 @@
//!
//! The contract side defines the versioned backup manifest, the per-backend
//! responsibility matrix, typed failure modes, and the restore dry-run
//! report. [`local_export`] implements the producer side for the Local
//! backend as a crate-internal API; restore orchestration and the admin API
//! build on these pieces in follow-up changes.
//! report. [`local_export`] implements the producer side and
//! [`local_restore`] the consumer side for the Local backend as
//! crate-internal APIs; the admin API builds on these pieces in follow-up
//! changes.
//!
//! # Bundle model
//!
@@ -51,6 +52,7 @@ mod capability;
mod dry_run;
mod error;
pub mod local_export;
pub mod local_restore;
mod manifest;
pub use capability::{AtRestProtection, BackupBackendKind, BackupResponsibility};
@@ -62,6 +64,10 @@ pub use local_export::{
BackupKek, LOCAL_BUNDLE_MANIFEST_FILE, LocalBackupExportRequest, decrypt_bundle_artifact, export_local_backup,
read_local_bundle_manifest,
};
pub use local_restore::{
LocalRestoreReport, LocalRestoreRequest, RestoreConflictPolicy, abort_local_restore, dry_run_local_restore,
restore_local_backup,
};
pub use manifest::{
AeadAlgorithm, ArtifactDescriptor, ArtifactKind, BackupKekDescriptor, BackupManifest, CompletenessState, ContentDigest,
DigestAlgorithm, LocalKdfDescriptor, LocalKeyDerivation, ReservedSlot,