test(e2e): add disk fault injection harness and reliability tests (#4223)

Add crates/e2e_test/src/chaos.rs with in-process fault-injection
primitives for a single-node multi-disk RustFS server: take a disk
offline (rename to <dir>.offline), bring it back online, replace a
disk with a fresh empty directory, corrupt an object's erasure shard
(part.* byte flips, xl.meta untouched), and SIGKILL/restart the server
with the same volumes and port.

Add reliability tests on a 4-disk (EC 2+2) topology, verified via
sha256 manifests recorded at write time:

- degraded read/write with one disk offline (incl. multipart object)
- bitrot read-through with two corrupted shards per object
- fresh-disk replacement healed via admin deep heal after a SIGKILL
  restart

Also reuse the shared signed_admin_post helper from chaos.rs in the
heal regression suite instead of a duplicated local copy.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Zhengchao An
2026-07-03 11:37:32 +08:00
committed by GitHub
parent cf056b39e3
commit 7329816ed7
4 changed files with 576 additions and 44 deletions
+8
View File
@@ -19,6 +19,14 @@ mod storage_api;
#[cfg(test)]
pub mod common;
// In-process fault-injection primitives (disk offline/replacement, shard corruption)
#[cfg(test)]
pub mod chaos;
// Reliability tests built on the fault-injection harness
#[cfg(test)]
mod reliability_disk_fault_test;
#[cfg(test)]
mod version_id_regression_test;