feat(kms): add a disaster-recovery drill harness for KMS backups (#5587)

* feat(kms): add a disaster-recovery drill harness for the Local backend

Rehearse the full backup/restore loop offline and return machine-readable
evidence: seed a sandbox deployment, seal sample objects through the
production encryption path, export a bundle, destroy the persistence layer,
preflight, restore, and decrypt every pre-disaster object again.

The evidence records the measured recovery point (one key is written past the
snapshot fence and must stay unrecoverable), the recovery time by phase, the
manifest digest before and after, and whether the restore treated its bundle
as read-only.

* test(kms): drill the Local disaster matrix and the interrupted cutover

Runs the harness against total key-directory loss, salt loss, and a torn key
record, asserting every pre-disaster object decrypts again while work past the
snapshot fence stays lost. Two further legs crash a restore exactly at its
commit point and prove the published marker names the bundle and the files it
still owes, then that re-running rolls forward and aborting rolls back.

The Vault leg needs a real server and is ignored by default: a Vault bundle
never carries the non-exportable Transit root, so what it drills is the
refusal to proceed before the operator has restored it natively.

* feat(kms): add an operator entry point for the disaster-recovery drill

Runs one rehearsal from environment configuration and writes the evidence
bundle, exiting non-zero on a failed verdict so a scheduled drill fails its
job instead of filing a bad report. It reads the same backup-KEK variables as
the admin backup API: drilling with the KEK real bundles are sealed under is
what proves that KEK is still retrievable.

* docs(kms): add the disaster-recovery drill runbook

Documents the procedure the harness automates: what a drill measures and why
the object probe rather than the manifest digest is the acceptance criterion,
the per-backend responsibility split, the disaster matrix, how to read the
evidence bundle, the two interrupted-cutover outcomes, and the Vault variant
whose cryptographic root comes back through Vault's own flow.

* chore(typos): accept RTO as a disaster-recovery term

_typos.toml

@@ -47,6 +47,9 @@ consts = "consts"
 Hashi = "Hashi"  # HashiCorp
 # Accept alternate spelling used in parser/XML comments.
 unparseable = "unparseable"
+# Disaster-recovery objectives: recovery time and recovery point.
+RTO = "RTO"
+rto = "rto"
 
 [files]
 extend-exclude = []

crates/kms/examples/kms_dr_drill.rs

@@ -0,0 +1,187 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Operator entry point for the KMS disaster-recovery drill.
+//!
+//! Runs one rehearsal inside a scratch workspace and writes the evidence
+//! bundle. Everything it touches lives under that workspace: it never reads,
+//! writes, or restores into a running deployment's key directory. See
+//! `docs/operations/kms-disaster-recovery-drill.md` for the procedure and for
+//! how to size the dataset so the measured recovery time transfers.
+//!
+//! Exit status is the verdict: 0 when every check held, 1 otherwise, so a
+//! scheduled drill fails its job instead of quietly filing a bad report.
+
+use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64};
+use rustfs_kms::backup::{BackupKek, DrillDataset, DrillDisaster, DrillRequest, DrillVerdict, run_local_drill};
+use std::path::PathBuf;
+use std::process::ExitCode;
+use zeroize::Zeroizing;
+
+const ENV_WORKSPACE: &str = "RUSTFS_KMS_DRILL_WORKSPACE";
+const ENV_DRILL_ID: &str = "RUSTFS_KMS_DRILL_ID";
+const ENV_DISASTER: &str = "RUSTFS_KMS_DRILL_DISASTER";
+const ENV_DEPLOYMENT: &str = "RUSTFS_KMS_DRILL_DEPLOYMENT";
+const ENV_MASTER_KEY: &str = "RUSTFS_KMS_DRILL_MASTER_KEY";
+const ENV_KEYS: &str = "RUSTFS_KMS_DRILL_KEYS";
+const ENV_OBJECTS_PER_KEY: &str = "RUSTFS_KMS_DRILL_OBJECTS_PER_KEY";
+const ENV_OBJECT_BYTES: &str = "RUSTFS_KMS_DRILL_OBJECT_BYTES";
+const ENV_EVIDENCE: &str = "RUSTFS_KMS_DRILL_EVIDENCE";
+const ENV_FILE_PERMISSIONS: &str = "RUSTFS_KMS_DRILL_FILE_PERMISSIONS";
+
+// Deliberately the same names the admin backup API reads: drilling with the
+// KEK the real backups are sealed under is what proves that KEK is still
+// retrievable, which is the part of a recovery no bundle can attest to.
+const ENV_KEK: &str = "RUSTFS_KMS_BACKUP_KEK";
+const ENV_KEK_ID: &str = "RUSTFS_KMS_BACKUP_KEK_ID";
+const ENV_KEK_VERSION: &str = "RUSTFS_KMS_BACKUP_KEK_VERSION";
+
+fn usage() -> String {
+    format!(
+        "KMS disaster-recovery drill.\n\
+         \n\
+         Required:\n  \
+           {ENV_WORKSPACE}      absolute path to a scratch directory the drill owns\n  \
+           {ENV_MASTER_KEY}     at-rest master key for the throwaway rehearsal deployment\n  \
+           {ENV_KEK}            base64 of the 32-byte backup KEK\n  \
+           {ENV_KEK_ID}         identifier recorded in the bundle manifest\n\
+         \n\
+         Optional:\n  \
+           {ENV_KEK_VERSION}    backup KEK version (default 1)\n  \
+           {ENV_DRILL_ID}       drill identifier and key-id prefix (default kms-dr-drill)\n  \
+           {ENV_DEPLOYMENT}     deployment identity recorded in the bundle (default kms-dr-drill)\n  \
+           {ENV_DISASTER}       key-directory-lost | master-key-salt-lost | key-record-corrupted\n  \
+           {ENV_KEYS}           master keys to rehearse with (default 4)\n  \
+           {ENV_OBJECTS_PER_KEY} objects sealed per key (default 2)\n  \
+           {ENV_OBJECT_BYTES}   plaintext bytes per object (default 4096)\n  \
+           {ENV_FILE_PERMISSIONS} octal key-file mode (default 600)\n  \
+           {ENV_EVIDENCE}       evidence output path (default <workspace>/evidence.json)"
+    )
+}
+
+fn required(name: &str) -> Result<String, String> {
+    std::env::var(name)
+        .ok()
+        .filter(|value| !value.trim().is_empty())
+        .ok_or_else(|| format!("{name} is required\n\n{}", usage()))
+}
+
+fn optional_usize(name: &str, default: usize) -> Result<usize, String> {
+    match std::env::var(name) {
+        Ok(value) => value
+            .trim()
+            .parse()
+            .map_err(|_| format!("{name} must be an unsigned integer")),
+        Err(_) => Ok(default),
+    }
+}
+
+fn disaster_from_env() -> Result<DrillDisaster, String> {
+    match std::env::var(ENV_DISASTER).ok().as_deref().map(str::trim) {
+        None | Some("") | Some("key-directory-lost") => Ok(DrillDisaster::KeyDirectoryLost),
+        Some("master-key-salt-lost") => Ok(DrillDisaster::MasterKeySaltLost),
+        Some("key-record-corrupted") => Ok(DrillDisaster::KeyRecordCorrupted),
+        Some(other) => Err(format!(
+            "{ENV_DISASTER}={other:?} is not a known disaster; use key-directory-lost, master-key-salt-lost, or key-record-corrupted"
+        )),
+    }
+}
+
+fn kek_from_env() -> Result<BackupKek, String> {
+    let raw = Zeroizing::new(required(ENV_KEK)?);
+    let decoded = Zeroizing::new(BASE64.decode(raw.trim()).map_err(|_| format!("{ENV_KEK} must be base64"))?);
+    if decoded.len() != 32 {
+        return Err(format!("{ENV_KEK} must decode to exactly 32 bytes"));
+    }
+    let mut material = [0u8; 32];
+    material.copy_from_slice(&decoded);
+    let version = optional_usize(ENV_KEK_VERSION, 1)?;
+    let version = u32::try_from(version).map_err(|_| format!("{ENV_KEK_VERSION} is out of range"))?;
+    BackupKek::new(required(ENV_KEK_ID)?, version, material).map_err(|error| error.to_string())
+}
+
+async fn run() -> Result<DrillVerdict, String> {
+    let workspace = PathBuf::from(required(ENV_WORKSPACE)?);
+    if !workspace.is_absolute() {
+        return Err(format!("{ENV_WORKSPACE} must be an absolute path"));
+    }
+    let drill_id = std::env::var(ENV_DRILL_ID)
+        .ok()
+        .filter(|value| !value.trim().is_empty())
+        .unwrap_or_else(|| "kms-dr-drill".to_string());
+    let file_permissions = match std::env::var(ENV_FILE_PERMISSIONS) {
+        Ok(value) => Some(u32::from_str_radix(value.trim(), 8).map_err(|_| format!("{ENV_FILE_PERMISSIONS} must be octal"))?),
+        Err(_) => Some(0o600),
+    };
+
+    let request = DrillRequest {
+        deployment_identity: std::env::var(ENV_DEPLOYMENT)
+            .ok()
+            .filter(|value| !value.trim().is_empty())
+            .unwrap_or_else(|| drill_id.clone()),
+        drill_id,
+        workspace: workspace.clone(),
+        rustfs_version: env!("CARGO_PKG_VERSION").to_string(),
+        // A drill always restores into a target that has observed nothing, so
+        // any generation is monotonic; the number is recorded as evidence of
+        // which snapshot the rehearsal covered.
+        snapshot_generation: 1,
+        dataset: DrillDataset {
+            keys: optional_usize(ENV_KEYS, 4)?,
+            objects_per_key: optional_usize(ENV_OBJECTS_PER_KEY, 2)?,
+            object_bytes: optional_usize(ENV_OBJECT_BYTES, 4096)?,
+        },
+        disaster: disaster_from_env()?,
+        master_key: required(ENV_MASTER_KEY)?,
+        file_permissions,
+    };
+
+    let kek = kek_from_env()?;
+    let evidence = run_local_drill(&kek, &request).await.map_err(|error| error.to_string())?;
+
+    let evidence_path = std::env::var(ENV_EVIDENCE)
+        .ok()
+        .filter(|value| !value.trim().is_empty())
+        .map_or_else(|| workspace.join("evidence.json"), PathBuf::from);
+    let encoded = evidence.encode().map_err(|error| error.to_string())?;
+    std::fs::write(&evidence_path, &encoded).map_err(|error| format!("cannot write {}: {error}", evidence_path.display()))?;
+
+    eprintln!("verdict:  {:?}", evidence.verdict);
+    eprintln!("disaster: {:?}", evidence.disaster);
+    eprintln!(
+        "objects:  {}/{} historical objects decrypted after the restore",
+        evidence.envelope_probes.iter().filter(|probe| probe.verified).count(),
+        evidence.envelope_probes.len()
+    );
+    eprintln!("rpo:      {} ms of work past the snapshot fence was lost", evidence.rpo.rpo_window_millis);
+    eprintln!("rto:      {} ms of recovery work", evidence.rto_millis);
+    for finding in &evidence.findings {
+        eprintln!("finding:  {finding}");
+    }
+    eprintln!("evidence: {}", evidence_path.display());
+
+    Ok(evidence.verdict)
+}
+
+#[tokio::main]
+async fn main() -> ExitCode {
+    match run().await {
+        Ok(DrillVerdict::Passed) => ExitCode::SUCCESS,
+        Ok(DrillVerdict::Failed) => ExitCode::FAILURE,
+        Err(message) => {
+            eprintln!("{message}");
+            ExitCode::FAILURE
+        }
+    }
+}

crates/kms/src/backup/mod.rs

@@ -21,7 +21,8 @@
 //! [`vault_restore`] orchestrates the consumer side for the Vault backends,
 //! whose cryptographic root is restored by Vault's own disaster-recovery
 //! flow. All are crate-internal APIs; the admin API builds on these pieces in
-//! follow-up changes.
+//! follow-up changes. [`drill`] rehearses the whole loop end to end and turns
+//! it into machine-readable evidence.
 //!
 //! # Bundle model
 //!
@@ -51,6 +52,7 @@
 //! format version.
 
 mod capability;
+pub mod drill;
 mod dry_run;
 mod error;
 pub mod local_export;
@@ -59,6 +61,10 @@ mod manifest;
 pub mod vault_restore;
 
 pub use capability::{AtRestProtection, BackupBackendKind, BackupResponsibility};
+pub use drill::{
+    DRILL_EVIDENCE_FORMAT_VERSION, DrillBundleEvidence, DrillDataset, DrillDisaster, DrillEvidence, DrillPhase, DrillPhaseTiming,
+    DrillRecoveryEvidence, DrillRequest, DrillRpoEvidence, DrillVerdict, EnvelopeProbe, run_local_drill,
+};
 pub use dry_run::{
     ExternalDependencyMismatch, RestoreBlocker, RestoreBlockerCode, RestoreConflict, RestoreConflictKind, RestoreDryRunReport,
 };

crates/kms/src/backup/vault_restore.rs

@@ -114,10 +114,13 @@ use vaultrs::api::transit::responses::ReadKeyData;
 use vaultrs::error::ClientError;
 use vaultrs::{kv2, transit::key};
 
+// The bundle layout names are pub(crate) so the drill harness
+// (`crate::backup::drill`) addresses the exact paths a Vault bundle uses
+// instead of copies that could drift.
 /// Bundle-relative directory holding one KV record artifact per key.
-const VAULT_RECORD_ARTIFACT_DIR: &str = "vault/records";
+pub(crate) const VAULT_RECORD_ARTIFACT_DIR: &str = "vault/records";
 /// Bundle-relative suffix of a KV record artifact.
-const VAULT_RECORD_ARTIFACT_SUFFIX: &str = ".json.enc";
+pub(crate) const VAULT_RECORD_ARTIFACT_SUFFIX: &str = ".json.enc";
 /// KV path segment (under the bundle's path prefix) of the restore commit
 /// marker. The leading dot keeps it out of the key id space the backends
 /// address; a bundle naming a key id equal to it is rejected.

docs/operations/kms-disaster-recovery-drill.md

@@ -0,0 +1,113 @@
+# KMS disaster-recovery drill
+
+A KMS backup that has never been restored is a hypothesis. This runbook turns it into evidence: it rehearses the complete loop — back up, lose the persistence layer, preflight, restore, and read historical objects again — and files a machine-readable evidence bundle for each run. For what each backend's backup actually covers, see [KMS backend security properties](kms-backend-security.md); for the metrics and alerts around KMS operations, see the [KMS observability runbook](kms-observability-runbook.md).
+
+The acceptance criterion of a drill is not that files came back. It is that objects encrypted before the disaster decrypt after the restore. The harness keeps the ciphertext and encryption metadata of every object it sealed before the disaster and, once the restore is complete, decrypts each one through a freshly opened backend and compares against the pre-disaster digest. Anything less proves only that a bundle is well formed.
+
+## Scope
+
+The drill covers the **Local** backend, which is the only backend RustFS produces a full-material bundle for. The responsibility split is deliberate and is described in `crates/kms/src/backup/capability.rs`:
+
+| Backend | What a RustFS bundle carries | What restores it |
+| --- | --- | --- |
+| Local | Key records, all stored versions, the KDF salt, sanitized configuration | The RustFS restore in this runbook |
+| Static | Non-sensitive references only | The operator re-supplies the secret out of band |
+| Vault KV2 + Transit | KV metadata and Transit ciphertext references | Vault's native snapshot restore, then the RustFS orchestration |
+| Vault Transit | Metadata, configuration references, verification data | Vault's native snapshot restore, then the RustFS orchestration |
+
+For the Vault backends there is no RustFS-side export, so there is no loop for a drill to close end to end: the cryptographic root is non-exportable and comes back through Vault's own disaster-recovery flow. What RustFS owns there is the refusal to proceed before that has happened, plus the ordering of everything after it. Rehearse it with the Vault section below.
+
+## What the drill measures
+
+**Recovery point (RPO).** The harness creates one key *after* the export fence closed and seals objects under it. After the restore that key is absent and its objects do not decrypt — as they must not. The recovery point of a KMS backup is therefore the snapshot generation of the last bundle, not the moment of the disaster, and `rpo.rpo_window_millis` in the evidence is the width of that window in the rehearsal. In production the same window is the age of your newest bundle, which is what your backup schedule sets.
+
+**Recovery time (RTO).** `rto_millis` is the sum of the phases an operator waits on: quarantine, preflight, restore, and verification. Seeding, sealing and the disaster itself are drill scaffolding and are excluded. Human decision time is excluded too and belongs to your incident process, not to this number. Per-phase costs are in `timings`.
+
+An RTO figure only transfers to production if the rehearsed deployment is comparable, so size `RUSTFS_KMS_DRILL_KEYS` to the number of master keys the real deployment holds. Both numbers are recorded in the evidence (`dataset`), so a stale measurement is visibly stale.
+
+## Running a drill
+
+```bash
+export RUSTFS_KMS_DRILL_WORKSPACE=/var/lib/rustfs/dr-drill/$(date -u +%Y%m%dT%H%M%SZ)
+export RUSTFS_KMS_DRILL_MASTER_KEY='<throwaway at-rest key for the rehearsal deployment>'
+export RUSTFS_KMS_BACKUP_KEK='<base64 of the 32-byte backup KEK>'
+export RUSTFS_KMS_BACKUP_KEK_ID='<kek identifier>'
+export RUSTFS_KMS_DRILL_KEYS=64
+
+cargo run -q -p rustfs-kms --example kms_dr_drill
+```
+
+The workspace must be an absolute path that does not already hold a rehearsal; give each run its own directory so the evidence and the quarantined state stay side by side. The runner exits 0 only when every check held, so a scheduled drill fails its job on a bad result instead of quietly filing a bad report.
+
+`RUSTFS_KMS_BACKUP_KEK` and `RUSTFS_KMS_BACKUP_KEK_ID` are the same variables the admin backup API reads. Use the KEK real bundles are sealed under: retrieving it is the one part of a recovery no bundle can attest to, and a drill that invents its own KEK does not test it. The rehearsal deployment's at-rest master key is separate and throwaway — the drill creates and destroys that deployment itself and never touches the running KMS.
+
+Optional variables: `RUSTFS_KMS_DRILL_DISASTER` (see below), `RUSTFS_KMS_DRILL_ID`, `RUSTFS_KMS_DRILL_DEPLOYMENT`, `RUSTFS_KMS_DRILL_OBJECTS_PER_KEY`, `RUSTFS_KMS_DRILL_OBJECT_BYTES`, `RUSTFS_KMS_DRILL_FILE_PERMISSIONS`, `RUSTFS_KMS_BACKUP_KEK_VERSION`, and `RUSTFS_KMS_DRILL_EVIDENCE` (evidence path, default `<workspace>/evidence.json`).
+
+## Disaster matrix
+
+Run all three; they exercise different failure surfaces and converge on the same procedure, which is the point — an operator does not have to diagnose the failure mode before acting.
+
+| `RUSTFS_KMS_DRILL_DISASTER` | Simulates |
+| --- | --- |
+| `key-directory-lost` (default) | The whole key directory is gone: lost volume, wiped host |
+| `master-key-salt-lost` | Records survive but the KDF salt is gone, so no material unwraps |
+| `key-record-corrupted` | One key record is truncated in place: torn write, partial media failure |
+
+Bundle-side faults — tampered, truncated, wrong-KEK, incomplete, unknown-version — are not drill scenarios. They are covered exhaustively and deterministically by the unit tests of `crates/kms/src/backup/local_export.rs` and `crates/kms/src/backup/local_restore.rs`, and re-running them as a drill would add fixtures without adding evidence.
+
+## Recovery procedure
+
+The drill executes exactly this sequence; running it by hand against a real deployment is the same procedure with your own paths.
+
+1. **Stop the KMS.** A restore must never race a running backend.
+2. **Quarantine, do not delete.** Move the damaged key directory aside rather than clearing it. The drill records the quarantine path and its contents in the evidence, and nothing damaged is ever destroyed: a restore that turns out to be the wrong call has to stay reversible, and forensics need the original bytes. Restore refuses a non-empty target anyway, including one holding only an orphan salt.
+3. **Preflight.** A dry-run decodes the whole bundle, checks the KDF descriptor against this build, verifies the operator-supplied master key against the bundle's one-way verifier, and enumerates conflicts. It writes nothing; the drill proves that by digesting the target before and after. Read every blocker before going further.
+4. **Restore.** Artifacts are staged inside the target, decryption-probed, then published through a commit marker and an atomic cutover.
+5. **Verify.** Open the restored directory and decrypt historical objects. This is the acceptance criterion, not step 4.
+6. **Observation period.** Keep the quarantined directory and the bundle until you have observed the recovered deployment serving reads. Nothing in the restore path deletes old material for you.
+
+## Reading the evidence
+
+`evidence.json` is `DrillEvidence` (`format_version` 1). It carries identifiers, digests and durations only: no key material, no master key, no plaintext, no ciphertext. Archive it next to the incident or audit record.
+
+| Field | Why it matters |
+| --- | --- |
+| `verdict`, `findings` | The result and every check that did not hold |
+| `envelope_probes[].verified` | Per-object proof that a historical data key still unwraps |
+| `rpo.post_snapshot_objects_recovered` | Must be `0`; anything else means the bundle was not a point-in-time snapshot |
+| `bundle.manifest_digest` vs `manifest_digest_after_recovery`, `bundle.source_unchanged` | The restore treated its bundle as strictly read-only |
+| `recovery.dry_run_zero_write` | The preflight wrote nothing to the target |
+| `recovery.commit_marker_cleared` | The cutover completed rather than leaving the target mid-restore |
+| `recovery.repeat_restore_refused` and `..._left_target_unchanged` | Re-running the procedure cannot damage a healthy deployment |
+| `dataset`, `timings`, `rto_millis`, `rpo.rpo_window_millis` | The measurement, and the deployment size it is valid for |
+
+## Interruptions and re-entry
+
+A restore has exactly one commit point: the durably published `.restore-commit.json` marker. Before it, the target's top level is untouched and re-running starts over. With it published, the backend refuses to start — a key directory mid-cutover must not serve requests — and you have two ways out:
+
+- **Roll forward**: re-run the restore with the same bundle. The marker is bound to the bundle by backup id and manifest digest, so a different bundle is refused rather than merged.
+- **Roll back**: abort the restore, which takes back exactly the files the marker names, then the marker, then the staged state. The target returns to its pre-restore state and a fresh restore can start.
+
+Both paths are exercised by the drill's interrupted-cutover tests in `crates/kms/src/backup/drill.rs`, which crash a restore precisely at its commit point rather than simulating the resulting state. Every interruption converges on the complete old state or the complete new state; there is no half-activated outcome.
+
+## Vault backends
+
+There is no RustFS-side Vault export, so a Vault drill is not the same closed loop. Rehearse it as:
+
+1. Restore the Vault cluster from its own native snapshot, following your Vault runbook. Transit keys are non-exportable and RustFS never attempts to import them.
+2. Run the RustFS preflight against the recovered cluster. It verifies cluster, namespace, mounts, the Transit key's identity and its version window, and the KV generation, and refuses on the first mismatch. A Transit key that was not restored, a `min_decryption_version` raised above what the bundle still needs, or a version history restored below the bundle's snapshot point are each reported as a distinct mismatch.
+3. Only then let the restore publish KV records, in the order material and versions, then metadata, then configuration and cutover.
+
+`crates/kms/src/backup/drill.rs` carries an `#[ignore]`d leg for step 2 that drills the refusal against a real server. It needs a Vault with the transit engine enabled and reads `RUSTFS_KMS_VAULT_ADDR` and `RUSTFS_KMS_VAULT_TOKEN`:
+
+```bash
+cargo test -p rustfs-kms --lib backup::drill -- --ignored --nocapture
+```
+
+## Running the drill matrix as tests
+
+```bash
+cargo test -p rustfs-kms --lib backup::drill
+```
+
+This runs the offline matrix — all three disasters, the evidence contract, and both interrupted-cutover outcomes — with no external dependencies. Use it in CI to keep the harness itself honest; use the operator entry point above to produce evidence for a real deployment size.