rustfs/docs/testing/ecstore-validation-suite-design.md

ECStore Validation Suite Design

This document defines the validation suite RustFS should use before claiming ECStore erasure-coding correctness, durability, and fault-resilience coverage. It is a suite design, not a claim that all tests already exist.

Goal

Provide one command that runs the full ECStore confidence suite and produces a single pass/fail result plus artifacts. The command should exercise both:

• white-box invariants in rustfs-ecstore, rustfs-filemeta, and related helpers;
• black-box S3 and admin behavior against real single-node and distributed erasure deployments.

The suite can reduce release risk; it cannot prove the absence of all defects. It must therefore combine deterministic matrices, negative tests, fuzz/corpus tests, chaos tests, and explicit artifact review.

Proposed Entry Point

Use the top-level runner for repeatable local, CI, and release checks:

scripts/run_ecstore_validation_suite.sh --profile full

Profiles:

Profile	Purpose	Expected cost
quick	PR smoke for EC logic and existing single-node reliability tests.	minutes
full	Release gate: white-box, e2e, chaos, S3 compatibility subset, coverage.	hours
destructive	Manual/nightly gate: distributed 4-node/16-disk, crash/restart, rebalance/decommission fault injection.	hours+
fuzz	Malformed metadata/RPC/corpus fuzzing with fixed seed output.	bounded by budget

The current runner wires the existing high-signal checks. The matrix below still tracks required follow-up coverage before full can be treated as a complete release gate.

The runner writes artifacts under target/ecstore-validation/<timestamp>/:

• command transcript and environment;
• randomized seeds;
• object manifests with SHA256;
• per-disk layout snapshots before and after mutation;
• server logs;
• junit/nextest output;
• blackbox-matrix.tsv with selected black-box and fixture gates;
• coverage report;
• failure reproducer instructions.

Acceptance Rules

The suite passes only when all selected profile commands pass and every scenario asserts both API-visible behavior and on-disk state where applicable.

Required fail-closed rules:

• never return corrupted object bytes;
• never silently accept forged or split-brain metadata;
• never downgrade write quorum to read quorum;
• never leave a mixed old/new object after partial commit;
• never panic on malformed EC metadata;
• return typed errors or quorum failures for invalid states.

Unit-test coverage is a hard release-gate input: rustfs-ecstore unit line coverage for the EC-critical scope must be at least 95%, with 100% as the target for EC read, write, decode, heal, metadata quorum, and rollback paths. A lower threshold is only acceptable for a temporary, explicitly documented exception tied to missing testability or unreachable code. Full-crate coverage is still reported as an observation metric, but it must not hide EC regressions behind unrelated modules.

Tests must not pass by only checking constants, helper calls, deleted branches, or implementation details. Every test needs a reader-facing or storage-state assertion.

Fixture-backed checks are optional for local smoke runs, but explicit in the artifact stream. --require-fixtures turns missing legacy or MinIO generated fixtures into an early ecstore-fixture-gate failure before expensive black-box steps run. The MinIO generated fixture gate requires both RUSTFS_MINIO_FIXTURE_ROOT and RUSTFS_MINIO_STATIC_KMS_KEY_B64.

White-Box Matrix

Erasure Algorithm

Target files:

• crates/ecstore/src/erasure/coding/erasure.rs
• crates/ecstore/src/erasure/coding/encode.rs
• crates/ecstore/src/erasure/coding/decode.rs
• crates/ecstore/src/erasure/coding/decode_reader.rs
• crates/ecstore/src/erasure/codec/bridge.rs

Coverage:

Area	Scenarios	Assertions
Shard geometry	legacy/current shard-size formulas; lengths 0, 1, block-1, block, block+1, multi-block tail	no divide-by-zero; shard/file/range offsets match expected
Encode/decode	(data, parity) sets 2+2, 4+2, 8+8; random payloads; missing shards up to parity	reconstructed data equals original
Negative reconstruction	missing shards above parity; inconsistent shard lengths; corrupt surplus parity	typed error, no partial success
Source verification	missing data shard plus extra parity source	rebuilt parity must match source parity
Legacy compatibility	old shard formula and legacy checksum data	legacy files decode and heal correctly
Streaming decode	legacy engine vs RustFS codec engine on same stripe stream	bytes and errors are equivalent
Range output	head/middle/tail/suffix; cross-block and final-short-stripe ranges	exact byte range, no over-read/under-read

Add property tests with fixed replay seeds for payload, range, and missing-shard selection.

Bitrot and Reader Alignment

Target files:

• crates/ecstore/src/erasure/coding/bitrot.rs
• crates/ecstore/src/erasure/coding/decode.rs
• crates/ecstore/src/set_disk/core/io_primitives.rs
• crates/ecstore/src/set_disk/shard_source.rs

Coverage:

Area	Scenarios	Assertions
Hash framing	valid hash+data; wrong hash; truncated hash; truncated data	invalid data never succeeds
Short shard	short read under normal hash, skip_verify, and hash-none	UnexpectedEof or equivalent typed error
Lockstep reads	mid-stream data shard failure; pending/timeout reader; final short stripe	each live reader advances exactly one stripe; failed reader retires
Adaptive reads	hedged parity fallback and timeout retirement	no shard desync; reconstructed bytes match original
Shard source order	out-of-order read completion and missing slots	slots resolve by shard index
Deferred readers	data-blocks-first setup opens deferred parity at correct offset	parity fallback uses aligned data

Use instrumented readers that record shard index, stripe index, read count, offset, and retirement reason.

Metadata, Quorum, and Commit Atomicity

Target files:

• crates/filemeta/src/filemeta/version.rs
• crates/ecstore/src/set_disk/read.rs
• crates/ecstore/src/set_disk/metadata.rs
• crates/ecstore/src/set_disk/ops/object.rs
• crates/ecstore/src/set_disk/core/io_primitives.rs
• crates/ecstore/src/disk/local.rs

Coverage:

Area	Scenarios	Assertions
Metadata tamper	same version_id/mod_time, divergent data dir, parts, ETag, size, checksum, inline flag, erasure distribution	previous committed version or read quorum error; no arbitrary latest
Early stop	valid quorum, stale quorum, corrupt trailing disks, slow trailing disks	early-stop only on safe identity
Quorum downgrade	read quorum vs write quorum; delete marker quorum; version-not-found quorum	no mutation below write quorum
Rename atomicity	failure before data rename, after data rename, after metadata rename, cleanup failure	object is old or new; never mixed
Rollback	failed commit quorum and stale temp data	rollback preserves old metadata/data
Malformed metadata	oversized lengths, bad CRC, invalid versions, invalid UUID/timestamp/enum, huge parts	bounded memory; typed error; no panic

Fault injection should be explicit and deterministic, preferably through local disk mocks for unit tests and process-level disk manipulation for e2e tests.

Black-Box Matrix

Use crates/e2e_test for real S3/admin behavior and extend crates/e2e_test/src/chaos.rs rather than duplicating ad hoc helpers.

The current runner emits blackbox-matrix.tsv for every invocation. It is a machine-readable manifest of selected black-box scenarios, their commands, required fixture environment, and whether each row is enabled, disabled, or missing an optional/required fixture.

Current runner rows:

Profile	Scenario	Gate	Fixture env
quick	single-node disk fault read/write	e2e black box	none
quick	degraded erasure disk rebuild	e2e black box	none
quick	namespace lock quorum under EC ops	e2e black box	none
full	legacy bitrot read fixture restore	fixture black box	RUSTFS_LEGACY_TEST_ROOT, RUSTFS_LEGACY_TEST_DISK
full	MinIO generated encrypted read and negative restore fixture	fixture black box	RUSTFS_MINIO_FIXTURE_ROOT, RUSTFS_MINIO_STATIC_KMS_KEY_B64
full	S3 multipart/range/versioning/delete subset	S3 black box	none
destructive	distributed cluster concurrency	e2e black box	none
destructive	stale multipart cleanup cluster	e2e black box	none
destructive	delete marker migration semantics	e2e black box	none

Single-Node 4-Disk EC

Scenario	Required assertions
baseline PUT/GET/HEAD/List for tiny, inline, block-boundary, multi-block, multipart objects	SHA256 manifest matches; metadata is consistent on all disks
one disk offline during read	existing objects readable; no corrupted bytes
one disk offline during write	write succeeds only when write quorum holds; restored disk is healed
above-parity disk loss	GET/PUT fails with quorum error; no partial bytes accepted
corrupt data shard and parity shard	GET returns original bytes or fails closed; read-repair/heal restores
corrupt inline xl.meta	fail closed or heal; no forged inline data
range read with offline/corrupt shard	exact range bytes; invalid ranges produce expected S3 errors
multipart part resend and concurrent same-part writes	final complete object matches chosen committed parts
crash during multipart complete/put/delete	after restart only old or new full version is visible

Existing anchors:

• crates/e2e_test/src/reliability_disk_fault_test.rs
• crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs
• crates/e2e_test/src/chaos.rs

Distributed 4-Node / 16-Disk EC

Scenario	Required assertions
node/disk outage while reading large objects	no EOF/truncation; SHA256 manifest matches
write while one remote node is down	write follows quorum; later heal reconstructs remote disk
remote shard bitrot	degraded read uses clean shards; no bad bytes
concurrent GET/PUT/DELETE/List on same key	no 500 for expected conflicts; no dirty reads
range GET matrix for large objects	sequential and parallel ranges match full-object hash
internode timeout/slow disk	typed error or fallback; no desync

This layer is mandatory because single-process unit tests cannot prove RPC, HTTP/2, timeout, and distributed quorum behavior.

Versioning, Delete Markers, and Migration

Scenario	Required assertions
latest delete marker	GET/HEAD/ListObjectVersions match S3 semantics
explicit versionId for old versions	exact old bytes and metadata
suspended/null version	no version ordering regression
delete marker during heal/rebalance/decommission	marker visibility and history are preserved
orphan directory cleanup	real objects are not purged; tombstones behave correctly

Heal, Rebalance, and Decommission

Scenario	Required assertions
auto heal and admin deep heal	data hash unchanged; xl.meta and format data rebuilt
heal interruption/restart	idempotent recovery; no dangling temp objects
two-pool rebalance with versioned/multipart objects	source and target pools have consistent versions
decommission cancel/restart/finalize	target readable; source cleanup safe
rebalance/decommission with node outage	progress resumes; no duplicate or missing versions

Existing scripts under scripts/test/decommission_*.sh should be wrapped into the destructive profile only after they emit machine-readable pass/fail artifacts.

S3 Compatibility and Large Object Gates

The full profile should include a targeted S3 compatibility subset, not the entire compatibility suite by default:

TESTEXPR="multipart or range or versioning or delete" \
DEPLOY_MODE=build \
MAXFAIL=0 \
./scripts/s3-tests/run.sh

Large object gates:

• scripts/run_get_codec_streaming_smoke.sh for legacy/codec GET parity;
• scripts/run_gt1g_get_http_matrix.sh for sequential and parallel range GET;
• scripts/run_gt1g_multipart_put_matrix.sh for multipart PUT paths.

These should be artifact-producing optional stages in full or destructive profiles, not hidden local-only commands.

Fuzz and Corpus Gates

Add bounded fuzz/corpus tests for:

• xl.meta MessagePack and legacy filemeta versions;
• protobuf/RPC payload decoding;
• checksum and bitrot headers;
• range offset/length overflow;
• object names with path traversal, encoded separators, and symlink components;
• huge inline metadata and huge part counts.

Each corpus failure must save the input bytes and the minimized reproducer under the suite artifact directory.

Coverage Snapshot Target

The runner enforces unit line coverage for rustfs-ecstore in full and destructive profiles. The default threshold is 95%, and the target remains 100% for EC-critical paths:

scripts/run_ecstore_validation_suite.sh --profile full --unit-coverage-min 95
scripts/run_ecstore_validation_suite.sh --profile full --unit-coverage-min 100
scripts/run_ecstore_validation_suite.sh --profile full --unit-coverage-scope crate

The default hard gate scope is ec-critical, covering:

• crates/ecstore/src/erasure/**
• crates/ecstore/src/set_disk/read.rs
• crates/ecstore/src/set_disk/shard_source.rs
• crates/ecstore/src/set_disk/metadata.rs
• crates/ecstore/src/set_disk/ops/object.rs
• crates/ecstore/src/set_disk/core/io_primitives.rs
• crates/ecstore/src/disk/local.rs

Minimum release-gate target:

• cargo-llvm-cov must be installed unless --skip-coverage is explicitly set;
• rustfs-ecstore EC-critical unit line coverage is at least 95%;
• EC read/write/decode/heal/quorum/rollback code should trend toward 100%;
• full-crate unit line coverage is recorded for visibility but is not the EC hard gate;
• all HIGH rows in this document have positive and negative tests;
• changed EC/read/write/heal lines are covered;
• branch coverage is reviewed for reconstruction, quorum, and error paths;
• uncovered branches are either intentionally unreachable or tracked.

The quick profile also runs cargo test -p rustfs-ecstore --lib -- --test-threads=1 so the white-box smoke includes all current rustfs-ecstore library unit tests without parallel test context cross-talk, not only focused EC filters.

Coverage artifacts:

• target/ecstore-validation/<timestamp>/coverage/ecstore/lcov.info
• target/ecstore-validation/<timestamp>/coverage/ecstore/summary.tsv
• target/ecstore-validation/<timestamp>/coverage/ecstore/files.tsv

Local validation on 2026-07-07 found that the current suite is not yet at the target: full-crate unit line coverage was 69.32%; the EC-critical scope was 84.32% after the first negative read/write/recovery additions. The lowest EC-critical files were set_disk/ops/object.rs, io_primitives.rs, disk/local.rs, bitrot.rs, and encode.rs. These are gaps to close before the default 95% gate can pass.

Initial Command Set

Use the runner first:

scripts/run_ecstore_validation_suite.sh --profile quick
scripts/run_ecstore_validation_suite.sh --profile full
scripts/run_ecstore_validation_suite.sh --profile destructive
scripts/run_ecstore_validation_suite.sh --profile fuzz

The split-run equivalent for the quick profile is:

cargo test -p rustfs-filemeta --lib
cargo test -p rustfs-ecstore --lib erasure
cargo test -p rustfs-ecstore --lib set_disk::read
cargo test -p rustfs-ecstore --lib set_disk::core::io_primitives
cargo test -p rustfs-ecstore --lib set_disk::tests::test_rename_data_quorum_failure_rolls_back_destination_object
cargo test -p rustfs-ecstore --lib disk::local
cargo test -p rustfs-ecstore --lib -- --test-threads=1
cargo test --package e2e_test reliability_disk_fault_test -- --nocapture
cargo test --package e2e_test heal_erasure_disk_rebuild_test -- --nocapture
cargo test --package e2e_test namespace_lock_quorum_test -- --nocapture

Fixture-backed tests should run when the fixture path is present:

cargo test -p rustfs-ecstore --test legacy_bitrot_read_test -- --nocapture
cargo test -p rustfs-ecstore --features rio-v2 --test minio_generated_read_test -- --ignored --nocapture

Multi-Expert Adversarial Review Summary

Three independent read-only reviews were run for this design:

Reviewer	Main challenge	Resulting requirement
Algorithm correctness	A single run only samples one timing/layout/hash combination.	matrix/property tests plus instrumented reader alignment checks
Black-box reliability	Existing tests miss distributed, shard-desync, range+fault, and migration-fault combinations.	add 4-node/16-disk and destructive profiles
Security and fault review	Metadata tamper, quorum downgrade, bitrot bypass, and rename atomicity must be end-to-end.	HIGH matrix rows block completion claims

Current verdict: design accepted as a target. The runner exists and captures artifacts; implementation remains incomplete until the missing matrix rows have artifact-backed evidence.