test(ecstore): deterministic rename_data crash-consistency harness (#4478)

The rename_data commit sequence is the highest-risk durability path in the
store, and the HP-1/HP-4/HP-5 work (fsync coalescing, group commit, relaxed
durability tiers) all need a standing gate that proves a power loss mid-commit
can never leave a mixed or corrupt object. There was one graceful-rollback
failpoint but no crash-consistency coverage.

Add a deterministic harness that models a hard power loss — the commit
sequence stops dead at the armed step with no in-process cleanup — and then
reopens the disk to assert the raw on-disk state is coherent:

- Two pre-commit injection points, RenameDataCrashPoint::{AfterDataRename,
  AfterBackupBeforeMetaCommit}, constructed at the real commit-path call sites
  but gated so the production build compiles them to a const-false no-op
  (mirrors the existing should_fail_before_old_metadata_backup pattern).
- A parameterized scenario over {strict, relaxed} durability x {both crash
  points} x {overwrite, fresh object}: seed, stage a replacement, inject,
  reopen, and assert the object reads back as exactly the old version (or does
  not exist when there was no old version) with its data dir intact, never the
  half-committed new one. The un-injected run asserts the commit makes the new
  version visible. Relaxed is held to the same old-or-new invariant as strict
  (only the durability window widens), which is exactly the property the
  durability-relaxation work must not break (rustfs/backlog#878 hard rule).
- Wire an explicit `ecstore-crash-consistency` gate into the destructive
  profile of run_ecstore_validation_suite.sh so it runs in the standing suite
  (coordinates with the #878 destructive profile rather than building a
  parallel one).

Production behavior is unchanged: the injection guards are no-ops outside
tests, and the existing rename_data rollback tests still pass.

Refs: rustfs/backlog#935 (HP-14 crash-consistency harness), rustfs/backlog#896
(test plan), rustfs/backlog#878 (ECStore validation suite), rustfs/backlog#936

Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
2026-07-08 20:53:49 +08:00
committed by GitHub
parent 4fe0789fb1
commit 54872d52dc
2 changed files with 264 additions and 0 deletions
+10
View File
@@ -505,6 +505,16 @@ run_full_profile() {
run_destructive_profile() {
run_full_profile
# Deterministic rename_data crash-consistency harness (rustfs/backlog#935
# HP-14 / test plan rustfs/backlog#896): injects a hard power loss at each
# pre-commit step across strict/relaxed durability and asserts the object
# reopens as old-or-new, never a mixed/corrupt version (rustfs/backlog#878
# hard rule). Unit-level, so it runs even when --skip-e2e drops the cluster
# scenarios below.
run_step "ecstore-crash-consistency" \
cargo test -p rustfs-ecstore --lib disk::local::test::crash_consistency -- --test-threads=1
if [[ "$SKIP_E2E" == "true" ]]; then
record_skip "e2e-destructive" "--skip-e2e was set"
return