mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-05 21:07:43 +00:00
Zhengchao An
3cfe867dff
* feat(kms): add a disaster-recovery drill harness for the Local backend Rehearse the full backup/restore loop offline and return machine-readable evidence: seed a sandbox deployment, seal sample objects through the production encryption path, export a bundle, destroy the persistence layer, preflight, restore, and decrypt every pre-disaster object again. The evidence records the measured recovery point (one key is written past the snapshot fence and must stay unrecoverable), the recovery time by phase, the manifest digest before and after, and whether the restore treated its bundle as read-only. * test(kms): drill the Local disaster matrix and the interrupted cutover Runs the harness against total key-directory loss, salt loss, and a torn key record, asserting every pre-disaster object decrypts again while work past the snapshot fence stays lost. Two further legs crash a restore exactly at its commit point and prove the published marker names the bundle and the files it still owes, then that re-running rolls forward and aborting rolls back. The Vault leg needs a real server and is ignored by default: a Vault bundle never carries the non-exportable Transit root, so what it drills is the refusal to proceed before the operator has restored it natively. * feat(kms): add an operator entry point for the disaster-recovery drill Runs one rehearsal from environment configuration and writes the evidence bundle, exiting non-zero on a failed verdict so a scheduled drill fails its job instead of filing a bad report. It reads the same backup-KEK variables as the admin backup API: drilling with the KEK real bundles are sealed under is what proves that KEK is still retrievable. * docs(kms): add the disaster-recovery drill runbook Documents the procedure the harness automates: what a drill measures and why the object probe rather than the manifest digest is the acceptance criterion, the per-backend responsibility split, the disaster matrix, how to read the evidence bundle, the two interrupted-cutover outcomes, and the Vault variant whose cryptographic root comes back through Vault's own flow. * chore(typos): accept RTO as a disaster-recovery term
90 lines
4.1 KiB
Rust
90 lines
4.1 KiB
Rust
// Copyright 2024 RustFS Team
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
//! Backup/restore contracts and backup production for KMS state.
|
|
//!
|
|
//! The contract side defines the versioned backup manifest, the per-backend
|
|
//! responsibility matrix, typed failure modes, and the restore dry-run
|
|
//! report. [`local_export`] implements the producer side and
|
|
//! [`local_restore`] the consumer side for the Local backend;
|
|
//! [`vault_restore`] orchestrates the consumer side for the Vault backends,
|
|
//! whose cryptographic root is restored by Vault's own disaster-recovery
|
|
//! flow. All are crate-internal APIs; the admin API builds on these pieces in
|
|
//! follow-up changes. [`drill`] rehearses the whole loop end to end and turns
|
|
//! it into machine-readable evidence.
|
|
//!
|
|
//! # Bundle model
|
|
//!
|
|
//! A backup bundle is a set of AEAD-encrypted artifacts described by a single
|
|
//! [`BackupManifest`]. All state in a bundle belongs to one snapshot
|
|
//! generation — there is no partially consistent bundle. The bundle is
|
|
//! protected by a backup KEK that is deliberately outside the business KMS
|
|
//! trust hierarchy, and the manifest is sealed with a completeness marker and
|
|
//! a final digest; a bundle that never reached its marker is permanently
|
|
//! non-restorable.
|
|
//!
|
|
//! # Restore ordering
|
|
//!
|
|
//! Restore implementations must follow this order: re-establish the external
|
|
//! trust root first (Vault/HSM native restore where one exists), then
|
|
//! material and version records into staging, then metadata and
|
|
//! configuration, then verification, and only then an explicit atomic
|
|
//! cutover. A dry-run ([`RestoreDryRunReport`]) performs zero writes.
|
|
//!
|
|
//! # Deliberately unfrozen
|
|
//!
|
|
//! Fields whose shape depends on contracts still in flight are reserved
|
|
//! rather than guessed (see [`ReservedSlot`]): the per-key version inventory
|
|
//! (backlog#1565) and capability discovery (backlog#1571). Alias and policy
|
|
//! artifacts are reserved names for features that do not exist yet. Reserved
|
|
//! slots reject data in format version 1 and become real types in a later
|
|
//! format version.
|
|
|
|
mod capability;
|
|
pub mod drill;
|
|
mod dry_run;
|
|
mod error;
|
|
pub mod local_export;
|
|
pub mod local_restore;
|
|
mod manifest;
|
|
pub mod vault_restore;
|
|
|
|
pub use capability::{AtRestProtection, BackupBackendKind, BackupResponsibility};
|
|
pub use drill::{
|
|
DRILL_EVIDENCE_FORMAT_VERSION, DrillBundleEvidence, DrillDataset, DrillDisaster, DrillEvidence, DrillPhase, DrillPhaseTiming,
|
|
DrillRecoveryEvidence, DrillRequest, DrillRpoEvidence, DrillVerdict, EnvelopeProbe, run_local_drill,
|
|
};
|
|
pub use dry_run::{
|
|
ExternalDependencyMismatch, RestoreBlocker, RestoreBlockerCode, RestoreConflict, RestoreConflictKind, RestoreDryRunReport,
|
|
};
|
|
pub use error::BackupError;
|
|
pub use local_export::{
|
|
BackupKek, LOCAL_BUNDLE_MANIFEST_FILE, LocalBackupExportRequest, decrypt_bundle_artifact, export_local_backup,
|
|
read_bundle_manifest, read_local_bundle_manifest,
|
|
};
|
|
pub use local_restore::{
|
|
LocalRestoreReport, LocalRestoreRequest, RestoreConflictPolicy, abort_local_restore, dry_run_local_restore,
|
|
restore_local_backup,
|
|
};
|
|
pub use manifest::{
|
|
AeadAlgorithm, ArtifactDescriptor, ArtifactKind, BackupKekDescriptor, BackupManifest, CompletenessState, ContentDigest,
|
|
DigestAlgorithm, LocalKdfDescriptor, LocalKeyDerivation, ReservedSlot, VaultExternalReferences, VaultKvRecordReference,
|
|
VaultTransitReference,
|
|
};
|
|
pub use vault_restore::{
|
|
VaultRestoreAbortReport, VaultRestoreAbortSkip, VaultRestoreAbortSkipReason, VaultRestoreClient, VaultRestoreMismatch,
|
|
VaultRestoreReport, VaultRestoreRequest, VaultRestoreSequence, VaultRestoreStage, VaultRestoreTarget, abort_vault_restore,
|
|
dry_run_vault_restore, restore_vault_backup,
|
|
};
|