feat(kms): add a disaster-recovery drill harness for KMS backups (#5587)

* feat(kms): add a disaster-recovery drill harness for the Local backend

Rehearse the full backup/restore loop offline and return machine-readable
evidence: seed a sandbox deployment, seal sample objects through the
production encryption path, export a bundle, destroy the persistence layer,
preflight, restore, and decrypt every pre-disaster object again.

The evidence records the measured recovery point (one key is written past the
snapshot fence and must stay unrecoverable), the recovery time by phase, the
manifest digest before and after, and whether the restore treated its bundle
as read-only.

* test(kms): drill the Local disaster matrix and the interrupted cutover

Runs the harness against total key-directory loss, salt loss, and a torn key
record, asserting every pre-disaster object decrypts again while work past the
snapshot fence stays lost. Two further legs crash a restore exactly at its
commit point and prove the published marker names the bundle and the files it
still owes, then that re-running rolls forward and aborting rolls back.

The Vault leg needs a real server and is ignored by default: a Vault bundle
never carries the non-exportable Transit root, so what it drills is the
refusal to proceed before the operator has restored it natively.

* feat(kms): add an operator entry point for the disaster-recovery drill

Runs one rehearsal from environment configuration and writes the evidence
bundle, exiting non-zero on a failed verdict so a scheduled drill fails its
job instead of filing a bad report. It reads the same backup-KEK variables as
the admin backup API: drilling with the KEK real bundles are sealed under is
what proves that KEK is still retrievable.

* docs(kms): add the disaster-recovery drill runbook

Documents the procedure the harness automates: what a drill measures and why
the object probe rather than the manifest digest is the acceptance criterion,
the per-backend responsibility split, the disaster matrix, how to read the
evidence bundle, the two interrupted-cutover outcomes, and the Vault variant
whose cryptographic root comes back through Vault's own flow.

* chore(typos): accept RTO as a disaster-recovery term

crates/kms/src/backup/mod.rs

@@ -21,7 +21,8 @@
 //! [`vault_restore`] orchestrates the consumer side for the Vault backends,
 //! whose cryptographic root is restored by Vault's own disaster-recovery
 //! flow. All are crate-internal APIs; the admin API builds on these pieces in
-//! follow-up changes.
+//! follow-up changes. [`drill`] rehearses the whole loop end to end and turns
+//! it into machine-readable evidence.
 //!
 //! # Bundle model
 //!
@@ -51,6 +52,7 @@
 //! format version.
 
 mod capability;
+pub mod drill;
 mod dry_run;
 mod error;
 pub mod local_export;
@@ -59,6 +61,10 @@ mod manifest;
 pub mod vault_restore;
 
 pub use capability::{AtRestProtection, BackupBackendKind, BackupResponsibility};
+pub use drill::{
+    DRILL_EVIDENCE_FORMAT_VERSION, DrillBundleEvidence, DrillDataset, DrillDisaster, DrillEvidence, DrillPhase, DrillPhaseTiming,
+    DrillRecoveryEvidence, DrillRequest, DrillRpoEvidence, DrillVerdict, EnvelopeProbe, run_local_drill,
+};
 pub use dry_run::{
     ExternalDependencyMismatch, RestoreBlocker, RestoreBlockerCode, RestoreConflict, RestoreConflictKind, RestoreDryRunReport,
 };

crates/kms/src/backup/vault_restore.rs

@@ -114,10 +114,13 @@ use vaultrs::api::transit::responses::ReadKeyData;
 use vaultrs::error::ClientError;
 use vaultrs::{kv2, transit::key};
 
+// The bundle layout names are pub(crate) so the drill harness
+// (`crate::backup::drill`) addresses the exact paths a Vault bundle uses
+// instead of copies that could drift.
 /// Bundle-relative directory holding one KV record artifact per key.
-const VAULT_RECORD_ARTIFACT_DIR: &str = "vault/records";
+pub(crate) const VAULT_RECORD_ARTIFACT_DIR: &str = "vault/records";
 /// Bundle-relative suffix of a KV record artifact.
-const VAULT_RECORD_ARTIFACT_SUFFIX: &str = ".json.enc";
+pub(crate) const VAULT_RECORD_ARTIFACT_SUFFIX: &str = ".json.enc";
 /// KV path segment (under the bundle's path prefix) of the restore commit
 /// marker. The leading dot keeps it out of the key id space the backends
 /// address; a bundle naming a key id equal to it is rejected.