mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-06 03:59:14 +00:00
docs(knowledge-base): prune stale content and add agent-facing index (#7035)
This commit is contained in:
Zhengchao An
@@ -1,294 +1,62 @@
|
||||
# ECStore Validation Suite Design
|
||||
# ECStore Validation Suite
|
||||
|
||||
This document defines the validation suite RustFS should use before claiming
|
||||
ECStore erasure-coding correctness, durability, and fault-resilience coverage.
|
||||
It is a suite design, not a claim that all tests already exist.
|
||||
**Use this when:** you run or extend `scripts/run_ecstore_validation_suite.sh`, or add an erasure-coding test and need the scenario row it must satisfy.
|
||||
**Source of truth:** `scripts/run_ecstore_validation_suite.sh` (profiles, flags, step commands, coverage scope, default thresholds); `docs/architecture/erasure-coding.md` (the invariants the rows enforce).
|
||||
|
||||
## Goal
|
||||
|
||||
Provide one command that runs the full ECStore confidence suite and produces a
|
||||
single pass/fail result plus artifacts. The command should exercise both:
|
||||
|
||||
- white-box invariants in `rustfs-ecstore`, `rustfs-filemeta`, and related
|
||||
helpers;
|
||||
- black-box S3 and admin behavior against real single-node and distributed
|
||||
erasure deployments.
|
||||
|
||||
The suite can reduce release risk; it cannot prove the absence of all defects.
|
||||
It must therefore combine deterministic matrices, negative tests, fuzz/corpus
|
||||
tests, chaos tests, and explicit artifact review.
|
||||
|
||||
## Proposed Entry Point
|
||||
|
||||
Use the top-level runner for repeatable local, CI, and release checks:
|
||||
## Runner contract
|
||||
|
||||
```bash
|
||||
scripts/run_ecstore_validation_suite.sh --profile full
|
||||
scripts/run_ecstore_validation_suite.sh --profile <quick|full|destructive|fuzz> \
|
||||
[--out-dir <path>] [--skip-e2e] [--skip-s3-tests] [--skip-coverage] [--require-fixtures] \
|
||||
[--unit-coverage-min <percent>] [--unit-coverage-scope <ec-critical|crate>] [--dry-run]
|
||||
```
|
||||
|
||||
Profiles:
|
||||
The runner is a local and release-validation tool; no CI workflow invokes it.
|
||||
|
||||
| Profile | Purpose | Expected cost |
|
||||
| --- | --- | --- |
|
||||
| `quick` | PR smoke for EC logic and existing single-node reliability tests. | minutes |
|
||||
| `full` | Release gate: white-box, e2e, chaos, S3 compatibility subset, coverage. | hours |
|
||||
| `destructive` | Manual/nightly gate: distributed 4-node/16-disk, crash/restart, rebalance/decommission fault injection. | hours+ |
|
||||
| `fuzz` | Malformed metadata/RPC/corpus fuzzing with fixed seed output. | bounded by budget |
|
||||
| Profile | Steps (each profile includes the one above it) | Cost |
|
||||
|---|---|---|
|
||||
| `quick` | `rustfs-filemeta` lib tests; focused `rustfs-ecstore` lib filters (`erasure`, `set_disk::read`, `set_disk::core::io_primitives`, the rename-rollback test, `disk::local`); the two isolated global-state tests named at the top of the runner; the whole `rustfs-ecstore --lib` with `--test-threads=1`; e2e `reliability_disk_fault_test`, `heal_erasure_disk_rebuild_test`, `namespace_lock_quorum_test` | minutes |
|
||||
| `full` | fixture gate; legacy-bitrot and MinIO generated-read fixture tests; s3-tests subset `TESTEXPR="multipart or range or versioning or delete"` with `DEPLOY_MODE=build MAXFAIL=0`; unit coverage gate | hours |
|
||||
| `destructive` | `disk::local::test::crash_consistency` (power loss at each pre-commit step; object reopens as old or new, never mixed); e2e `cluster_concurrency_test`, `stale_multipart_cleanup_cluster_test`, `delete_marker_migration_semantics_test` | hours+ |
|
||||
| `fuzz` | `scripts/fuzz/run.sh` bounded by `MAX_TOTAL_TIME` (default in the runner) | bounded |
|
||||
|
||||
The current runner wires the existing high-signal checks. The matrix below
|
||||
still tracks required follow-up coverage before `full` can be treated as a
|
||||
complete release gate.
|
||||
Artifacts under `<out-dir>` (default `target/ecstore-validation/<timestamp>/`):
|
||||
|
||||
The runner writes artifacts under
|
||||
`target/ecstore-validation/<timestamp>/`:
|
||||
| File | Content |
|
||||
|---|---|
|
||||
| `run-metadata.env` | profile, flags, thresholds, environment |
|
||||
| `summary.tsv` | one row per step: pass / fail / skip with reason |
|
||||
| `blackbox-matrix.tsv` | every black-box and fixture row with command, fixture env, and status `enabled` / `disabled` / `missing-optional` / `missing-required`; written on every invocation regardless of profile |
|
||||
| `logs/*.log` | per-step transcripts |
|
||||
| `coverage/ecstore/lcov.info`, `summary.tsv`, `files.tsv` | coverage export, gate summary, per-file table (`full` and `destructive`) |
|
||||
|
||||
- command transcript and environment;
|
||||
- randomized seeds;
|
||||
- object manifests with SHA256;
|
||||
- per-disk layout snapshots before and after mutation;
|
||||
- server logs;
|
||||
- junit/nextest output;
|
||||
- `blackbox-matrix.tsv` with selected black-box and fixture gates;
|
||||
- coverage report;
|
||||
- failure reproducer instructions.
|
||||
Fixture rows:
|
||||
|
||||
## Acceptance Rules
|
||||
| Row | Profile | Fixture env |
|
||||
|---|---|---|
|
||||
| legacy bitrot read (`crates/ecstore/tests/legacy_bitrot_read_test.rs`) | `full` | `RUSTFS_LEGACY_TEST_ROOT`, `RUSTFS_LEGACY_TEST_DISK` |
|
||||
| MinIO generated encrypted read and negative restore (`storage::minio_generated_read_test` in `rustfs/src/storage/mod.rs`, `--features rio-v2`, `--ignored`) | `full` | `RUSTFS_MINIO_FIXTURE_ROOT`, `RUSTFS_MINIO_STATIC_KMS_KEY_B64` |
|
||||
|
||||
The suite passes only when all selected profile commands pass and every
|
||||
scenario asserts both API-visible behavior and on-disk state where applicable.
|
||||
A missing fixture is recorded as a `missing-optional` skip. `--require-fixtures` turns it into an early `ecstore-fixture-gate` failure before the expensive black-box steps run.
|
||||
|
||||
Required fail-closed rules:
|
||||
## Acceptance rules
|
||||
|
||||
A run passes only when every selected step passes. Every scenario asserts API-visible behaviour and, where applicable, on-disk state; a test that only checks constants, helper calls, deleted branches, or implementation details does not satisfy a row.
|
||||
|
||||
Fail-closed invariants every row enforces:
|
||||
|
||||
- never return corrupted object bytes;
|
||||
- never silently accept forged or split-brain metadata;
|
||||
- never downgrade write quorum to read quorum;
|
||||
- never leave a mixed old/new object after partial commit;
|
||||
- never leave a mixed old/new object after a partial commit;
|
||||
- never panic on malformed EC metadata;
|
||||
- return typed errors or quorum failures for invalid states.
|
||||
|
||||
Unit-test coverage is a hard release-gate input: `rustfs-ecstore` unit line
|
||||
coverage for the EC-critical scope must be at least 95%, with 100% as the
|
||||
target for EC read, write, decode, heal, metadata quorum, and rollback paths.
|
||||
A lower threshold is only acceptable for a temporary, explicitly documented
|
||||
exception tied to missing testability or unreachable code. Full-crate coverage
|
||||
is still reported as an observation metric, but it must not hide EC regressions
|
||||
behind unrelated modules.
|
||||
Fault injection is explicit and deterministic: local disk mocks for unit tests, process-level disk manipulation (`crates/e2e_test/src/chaos.rs`) for e2e tests. Property tests replay a fixed seed for payload, range, and missing-shard selection.
|
||||
|
||||
Tests must not pass by only checking constants, helper calls, deleted branches,
|
||||
or implementation details. Every test needs a reader-facing or storage-state
|
||||
assertion.
|
||||
### Coverage gate
|
||||
|
||||
Fixture-backed checks are optional for local smoke runs, but explicit in the
|
||||
artifact stream. `--require-fixtures` turns missing legacy or MinIO generated
|
||||
fixtures into an early `ecstore-fixture-gate` failure before expensive black-box
|
||||
steps run. The MinIO generated fixture gate requires both
|
||||
`RUSTFS_MINIO_FIXTURE_ROOT` and `RUSTFS_MINIO_STATIC_KMS_KEY_B64`.
|
||||
|
||||
## White-Box Matrix
|
||||
|
||||
### Erasure Algorithm
|
||||
|
||||
Target files:
|
||||
|
||||
- `crates/ecstore/src/erasure/coding/erasure.rs`
|
||||
- `crates/ecstore/src/erasure/coding/encode.rs`
|
||||
- `crates/ecstore/src/erasure/coding/decode.rs`
|
||||
- `crates/ecstore/src/erasure/coding/decode_reader.rs`
|
||||
- `crates/ecstore/src/erasure/codec/bridge.rs`
|
||||
|
||||
Coverage:
|
||||
|
||||
| Area | Scenarios | Assertions |
|
||||
| --- | --- | --- |
|
||||
| Shard geometry | legacy/current shard-size formulas; lengths `0`, `1`, `block-1`, `block`, `block+1`, multi-block tail | no divide-by-zero; shard/file/range offsets match expected |
|
||||
| Encode/decode | `(data, parity)` sets `2+2`, `4+2`, `8+8`; random payloads; missing shards up to parity | reconstructed data equals original |
|
||||
| Negative reconstruction | missing shards above parity; inconsistent shard lengths; corrupt surplus parity | typed error, no partial success |
|
||||
| Source verification | missing data shard plus extra parity source | rebuilt parity must match source parity |
|
||||
| Legacy compatibility | old shard formula and legacy checksum data | legacy files decode and heal correctly |
|
||||
| Streaming decode | legacy engine vs RustFS codec engine on same stripe stream | bytes and errors are equivalent |
|
||||
| Range output | head/middle/tail/suffix; cross-block and final-short-stripe ranges | exact byte range, no over-read/under-read |
|
||||
|
||||
Add property tests with fixed replay seeds for payload, range, and missing-shard
|
||||
selection.
|
||||
|
||||
### Bitrot and Reader Alignment
|
||||
|
||||
Target files:
|
||||
|
||||
- `crates/ecstore/src/erasure/coding/bitrot.rs`
|
||||
- `crates/ecstore/src/erasure/coding/decode.rs`
|
||||
- `crates/ecstore/src/set_disk/core/io_primitives.rs`
|
||||
- `crates/ecstore/src/set_disk/shard_source.rs`
|
||||
|
||||
Coverage:
|
||||
|
||||
| Area | Scenarios | Assertions |
|
||||
| --- | --- | --- |
|
||||
| Hash framing | valid hash+data; wrong hash; truncated hash; truncated data | invalid data never succeeds |
|
||||
| Short shard | short read under normal hash, `skip_verify`, and hash-none | `UnexpectedEof` or equivalent typed error |
|
||||
| Lockstep reads | mid-stream data shard failure; pending/timeout reader; final short stripe | each live reader advances exactly one stripe; failed reader retires |
|
||||
| Adaptive reads | hedged parity fallback and timeout retirement | no shard desync; reconstructed bytes match original |
|
||||
| Shard source order | out-of-order read completion and missing slots | slots resolve by shard index |
|
||||
| Deferred readers | data-blocks-first setup opens deferred parity at correct offset | parity fallback uses aligned data |
|
||||
|
||||
Use instrumented readers that record shard index, stripe index, read count,
|
||||
offset, and retirement reason.
|
||||
|
||||
### Metadata, Quorum, and Commit Atomicity
|
||||
|
||||
Target files:
|
||||
|
||||
- `crates/filemeta/src/filemeta/version.rs`
|
||||
- `crates/ecstore/src/set_disk/read.rs`
|
||||
- `crates/ecstore/src/set_disk/metadata.rs`
|
||||
- `crates/ecstore/src/set_disk/ops/object.rs`
|
||||
- `crates/ecstore/src/set_disk/core/io_primitives.rs`
|
||||
- `crates/ecstore/src/disk/local.rs`
|
||||
|
||||
Coverage:
|
||||
|
||||
| Area | Scenarios | Assertions |
|
||||
| --- | --- | --- |
|
||||
| Metadata tamper | same `version_id`/`mod_time`, divergent data dir, parts, ETag, size, checksum, inline flag, erasure distribution | previous committed version or read quorum error; no arbitrary latest |
|
||||
| Early stop | valid quorum, stale quorum, corrupt trailing disks, slow trailing disks | early-stop only on safe identity |
|
||||
| Quorum downgrade | read quorum vs write quorum; delete marker quorum; version-not-found quorum | no mutation below write quorum |
|
||||
| Rename atomicity | failure before data rename, after data rename, after metadata rename, cleanup failure | object is old or new; never mixed |
|
||||
| Rollback | failed commit quorum and stale temp data | rollback preserves old metadata/data |
|
||||
| Malformed metadata | oversized lengths, bad CRC, invalid versions, invalid UUID/timestamp/enum, huge parts | bounded memory; typed error; no panic |
|
||||
|
||||
Fault injection should be explicit and deterministic, preferably through local
|
||||
disk mocks for unit tests and process-level disk manipulation for e2e tests.
|
||||
|
||||
## Black-Box Matrix
|
||||
|
||||
Use `crates/e2e_test` for real S3/admin behavior and extend
|
||||
`crates/e2e_test/src/chaos.rs` rather than duplicating ad hoc helpers.
|
||||
|
||||
The current runner emits `blackbox-matrix.tsv` for every invocation. It is a
|
||||
machine-readable manifest of selected black-box scenarios, their commands,
|
||||
required fixture environment, and whether each row is enabled, disabled, or
|
||||
missing an optional/required fixture.
|
||||
|
||||
Current runner rows:
|
||||
|
||||
| Profile | Scenario | Gate | Fixture env |
|
||||
| --- | --- | --- | --- |
|
||||
| `quick` | single-node disk fault read/write | e2e black box | none |
|
||||
| `quick` | degraded erasure disk rebuild | e2e black box | none |
|
||||
| `quick` | namespace lock quorum under EC ops | e2e black box | none |
|
||||
| `full` | legacy bitrot read fixture restore | fixture black box | `RUSTFS_LEGACY_TEST_ROOT`, `RUSTFS_LEGACY_TEST_DISK` |
|
||||
| `full` | MinIO generated encrypted read and negative restore fixture | fixture black box | `RUSTFS_MINIO_FIXTURE_ROOT`, `RUSTFS_MINIO_STATIC_KMS_KEY_B64` |
|
||||
| `full` | S3 multipart/range/versioning/delete subset | S3 black box | none |
|
||||
| `destructive` | distributed cluster concurrency | e2e black box | none |
|
||||
| `destructive` | stale multipart cleanup cluster | e2e black box | none |
|
||||
| `destructive` | delete marker migration semantics | e2e black box | none |
|
||||
|
||||
### Single-Node 4-Disk EC
|
||||
|
||||
| Scenario | Required assertions |
|
||||
| --- | --- |
|
||||
| baseline PUT/GET/HEAD/List for tiny, inline, block-boundary, multi-block, multipart objects | SHA256 manifest matches; metadata is consistent on all disks |
|
||||
| one disk offline during read | existing objects readable; no corrupted bytes |
|
||||
| one disk offline during write | write succeeds only when write quorum holds; restored disk is healed |
|
||||
| above-parity disk loss | GET/PUT fails with quorum error; no partial bytes accepted |
|
||||
| corrupt data shard and parity shard | GET returns original bytes or fails closed; read-repair/heal restores |
|
||||
| corrupt inline `xl.meta` | fail closed or heal; no forged inline data |
|
||||
| range read with offline/corrupt shard | exact range bytes; invalid ranges produce expected S3 errors |
|
||||
| multipart part resend and concurrent same-part writes | final complete object matches chosen committed parts |
|
||||
| crash during multipart complete/put/delete | after restart only old or new full version is visible |
|
||||
|
||||
Existing anchors:
|
||||
|
||||
- `crates/e2e_test/src/reliability_disk_fault_test.rs`
|
||||
- `crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs`
|
||||
- `crates/e2e_test/src/chaos.rs`
|
||||
|
||||
### Distributed 4-Node / 16-Disk EC
|
||||
|
||||
| Scenario | Required assertions |
|
||||
| --- | --- |
|
||||
| node/disk outage while reading large objects | no EOF/truncation; SHA256 manifest matches |
|
||||
| write while one remote node is down | write follows quorum; later heal reconstructs remote disk |
|
||||
| remote shard bitrot | degraded read uses clean shards; no bad bytes |
|
||||
| concurrent GET/PUT/DELETE/List on same key | no 500 for expected conflicts; no dirty reads |
|
||||
| range GET matrix for large objects | sequential and parallel ranges match full-object hash |
|
||||
| internode timeout/slow disk | typed error or fallback; no desync |
|
||||
|
||||
This layer is mandatory because single-process unit tests cannot prove RPC,
|
||||
HTTP/2, timeout, and distributed quorum behavior.
|
||||
|
||||
### Versioning, Delete Markers, and Migration
|
||||
|
||||
| Scenario | Required assertions |
|
||||
| --- | --- |
|
||||
| latest delete marker | GET/HEAD/ListObjectVersions match S3 semantics |
|
||||
| explicit `versionId` for old versions | exact old bytes and metadata |
|
||||
| suspended/null version | no version ordering regression |
|
||||
| delete marker during heal/rebalance/decommission | marker visibility and history are preserved |
|
||||
| orphan directory cleanup | real objects are not purged; tombstones behave correctly |
|
||||
|
||||
### Heal, Rebalance, and Decommission
|
||||
|
||||
| Scenario | Required assertions |
|
||||
| --- | --- |
|
||||
| auto heal and admin deep heal | data hash unchanged; `xl.meta` and format data rebuilt |
|
||||
| heal interruption/restart | idempotent recovery; no dangling temp objects |
|
||||
| two-pool rebalance with versioned/multipart objects | source and target pools have consistent versions |
|
||||
| decommission cancel/restart/finalize | target readable; source cleanup safe |
|
||||
| rebalance/decommission with node outage | progress resumes; no duplicate or missing versions |
|
||||
|
||||
Existing scripts under `scripts/test/decommission_*.sh` should be wrapped into
|
||||
the `destructive` profile only after they emit machine-readable pass/fail
|
||||
artifacts.
|
||||
|
||||
## S3 Compatibility and Large Object Gates
|
||||
|
||||
The full profile should include a targeted S3 compatibility subset, not the
|
||||
entire compatibility suite by default:
|
||||
|
||||
```bash
|
||||
TESTEXPR="multipart or range or versioning or delete" \
|
||||
DEPLOY_MODE=build \
|
||||
MAXFAIL=0 \
|
||||
./scripts/s3-tests/run.sh
|
||||
```
|
||||
|
||||
Large object gates:
|
||||
|
||||
- `scripts/run_get_codec_streaming_smoke.sh` for legacy/codec GET parity;
|
||||
- `scripts/run_gt1g_get_http_matrix.sh` for sequential and parallel range GET;
|
||||
- `scripts/run_gt1g_multipart_put_matrix.sh` for multipart PUT paths.
|
||||
|
||||
These should be artifact-producing optional stages in `full` or `destructive`
|
||||
profiles, not hidden local-only commands.
|
||||
|
||||
## Fuzz and Corpus Gates
|
||||
|
||||
Add bounded fuzz/corpus tests for:
|
||||
|
||||
- `xl.meta` MessagePack and legacy filemeta versions;
|
||||
- protobuf/RPC payload decoding;
|
||||
- checksum and bitrot headers;
|
||||
- range offset/length overflow;
|
||||
- object names with path traversal, encoded separators, and symlink components;
|
||||
- huge inline metadata and huge part counts.
|
||||
|
||||
Each corpus failure must save the input bytes and the minimized reproducer under
|
||||
the suite artifact directory.
|
||||
|
||||
## Coverage Snapshot Target
|
||||
|
||||
The runner enforces unit line coverage for `rustfs-ecstore` in `full` and
|
||||
`destructive` profiles. The default threshold is 95%, and the target remains
|
||||
100% for EC-critical paths:
|
||||
|
||||
```bash
|
||||
scripts/run_ecstore_validation_suite.sh --profile full --unit-coverage-min 95
|
||||
scripts/run_ecstore_validation_suite.sh --profile full --unit-coverage-min 100
|
||||
scripts/run_ecstore_validation_suite.sh --profile full --unit-coverage-scope crate
|
||||
```
|
||||
|
||||
The default hard gate scope is `ec-critical`, covering:
|
||||
`full` and `destructive` run `cargo llvm-cov -p rustfs-ecstore --lib` and fail when line coverage of the gate scope is below `--unit-coverage-min`. The default minimum and the 100% target for EC read, write, decode, heal, metadata-quorum, and rollback paths are the `UNIT_COVERAGE_*` constants at the top of the runner. `cargo-llvm-cov` must be installed unless `--skip-coverage` is passed explicitly. The default scope `ec-critical` is:
|
||||
|
||||
- `crates/ecstore/src/erasure/**`
|
||||
- `crates/ecstore/src/set_disk/read.rs`
|
||||
@@ -298,79 +66,103 @@ The default hard gate scope is `ec-critical`, covering:
|
||||
- `crates/ecstore/src/set_disk/core/io_primitives.rs`
|
||||
- `crates/ecstore/src/disk/local.rs`
|
||||
|
||||
Minimum release-gate target:
|
||||
`--unit-coverage-scope crate` measures the whole crate instead; that number is an observation metric and must not hide EC regressions behind unrelated modules. Lowering the minimum requires a documented exception tied to missing testability or unreachable code. Uncovered branches in reconstruction, quorum, and error paths are either intentionally unreachable or tracked.
|
||||
|
||||
- `cargo-llvm-cov` must be installed unless `--skip-coverage` is explicitly set;
|
||||
- `rustfs-ecstore` EC-critical unit line coverage is at least 95%;
|
||||
- EC read/write/decode/heal/quorum/rollback code should trend toward 100%;
|
||||
- full-crate unit line coverage is recorded for visibility but is not the EC
|
||||
hard gate;
|
||||
- all HIGH rows in this document have positive and negative tests;
|
||||
- changed EC/read/write/heal lines are covered;
|
||||
- branch coverage is reviewed for reconstruction, quorum, and error paths;
|
||||
- uncovered branches are either intentionally unreachable or tracked.
|
||||
## White-box scenario matrix
|
||||
|
||||
The `quick` profile also runs
|
||||
`cargo test -p rustfs-ecstore --lib -- --test-threads=1` so the white-box smoke
|
||||
includes all current `rustfs-ecstore` library unit tests without parallel test
|
||||
context cross-talk, not only focused EC filters.
|
||||
Each row is a scenario the unit layer covers; a new EC unit test names the row it satisfies.
|
||||
|
||||
Coverage artifacts:
|
||||
### Erasure algorithm (`crates/ecstore/src/erasure/`)
|
||||
|
||||
- `target/ecstore-validation/<timestamp>/coverage/ecstore/lcov.info`
|
||||
- `target/ecstore-validation/<timestamp>/coverage/ecstore/summary.tsv`
|
||||
- `target/ecstore-validation/<timestamp>/coverage/ecstore/files.tsv`
|
||||
|
||||
Local validation on 2026-07-07 found that the current suite is not yet at the
|
||||
target: full-crate unit line coverage was 69.32%; the EC-critical scope was
|
||||
84.32% after the first negative read/write/recovery additions. The lowest
|
||||
EC-critical files were `set_disk/ops/object.rs`, `io_primitives.rs`,
|
||||
`disk/local.rs`, `bitrot.rs`, and `encode.rs`. These are gaps to close before
|
||||
the default 95% gate can pass.
|
||||
|
||||
## Initial Command Set
|
||||
|
||||
Use the runner first:
|
||||
|
||||
```bash
|
||||
scripts/run_ecstore_validation_suite.sh --profile quick
|
||||
scripts/run_ecstore_validation_suite.sh --profile full
|
||||
scripts/run_ecstore_validation_suite.sh --profile destructive
|
||||
scripts/run_ecstore_validation_suite.sh --profile fuzz
|
||||
```
|
||||
|
||||
The split-run equivalent for the quick profile is:
|
||||
|
||||
```bash
|
||||
cargo test -p rustfs-filemeta --lib
|
||||
cargo test -p rustfs-ecstore --lib erasure
|
||||
cargo test -p rustfs-ecstore --lib set_disk::read
|
||||
cargo test -p rustfs-ecstore --lib set_disk::core::io_primitives
|
||||
cargo test -p rustfs-ecstore --lib set_disk::tests::test_rename_data_quorum_failure_rolls_back_destination_object
|
||||
cargo test -p rustfs-ecstore --lib disk::local
|
||||
cargo test -p rustfs-ecstore --lib -- --test-threads=1
|
||||
cargo test --package e2e_test reliability_disk_fault_test -- --nocapture
|
||||
cargo test --package e2e_test heal_erasure_disk_rebuild_test -- --nocapture
|
||||
cargo test --package e2e_test namespace_lock_quorum_test -- --nocapture
|
||||
```
|
||||
|
||||
Fixture-backed tests should run when the fixture path is present:
|
||||
|
||||
```bash
|
||||
cargo test -p rustfs-ecstore --test legacy_bitrot_read_test -- --nocapture
|
||||
cargo test -p rustfs --features rio-v2 storage::minio_generated_read_test --lib -- --ignored --nocapture
|
||||
```
|
||||
|
||||
## Multi-Expert Adversarial Review Summary
|
||||
|
||||
Three independent read-only reviews were run for this design:
|
||||
|
||||
| Reviewer | Main challenge | Resulting requirement |
|
||||
| Area | Scenarios | Assertions |
|
||||
| --- | --- | --- |
|
||||
| Algorithm correctness | A single run only samples one timing/layout/hash combination. | matrix/property tests plus instrumented reader alignment checks |
|
||||
| Black-box reliability | Existing tests miss distributed, shard-desync, range+fault, and migration-fault combinations. | add 4-node/16-disk and destructive profiles |
|
||||
| Security and fault review | Metadata tamper, quorum downgrade, bitrot bypass, and rename atomicity must be end-to-end. | HIGH matrix rows block completion claims |
|
||||
| Shard geometry | legacy/current shard-size formulas; lengths `0`, `1`, `block-1`, `block`, `block+1`, multi-block tail | no divide-by-zero; shard/file/range offsets match expected |
|
||||
| Encode/decode | `(data, parity)` sets `2+2`, `4+2`, `8+8`; random payloads; missing shards up to parity | reconstructed data equals original |
|
||||
| Negative reconstruction | missing shards above parity; inconsistent shard lengths; corrupt surplus parity | typed error, no partial success |
|
||||
| Source verification | missing data shard plus extra parity source | rebuilt parity matches source parity |
|
||||
| Legacy compatibility | old shard formula and legacy checksum data | legacy files decode and heal correctly |
|
||||
| Streaming decode | legacy engine vs RustFS codec engine on the same stripe stream | bytes and errors are equivalent |
|
||||
| Range output | head/middle/tail/suffix; cross-block and final-short-stripe ranges | exact byte range, no over-read or under-read |
|
||||
|
||||
Current verdict: design accepted as a target. The runner exists and captures
|
||||
artifacts; implementation remains incomplete until the missing matrix rows have
|
||||
artifact-backed evidence.
|
||||
### Bitrot and reader alignment (`erasure/coding/bitrot.rs`, `decode.rs`, `set_disk/core/io_primitives.rs`, `set_disk/shard_source.rs`)
|
||||
|
||||
| Area | Scenarios | Assertions |
|
||||
| --- | --- | --- |
|
||||
| Hash framing | valid hash+data; wrong hash; truncated hash; truncated data | invalid data never succeeds |
|
||||
| Short shard | short read under normal hash, `skip_verify`, and hash-none | `UnexpectedEof` or equivalent typed error |
|
||||
| Lockstep reads | mid-stream data shard failure; pending/timeout reader; final short stripe | each live reader advances exactly one stripe; failed reader retires |
|
||||
| Adaptive reads | hedged parity fallback and timeout retirement | no shard desync; reconstructed bytes match original |
|
||||
| Shard source order | out-of-order read completion and missing slots | slots resolve by shard index |
|
||||
| Deferred readers | data-blocks-first setup opens deferred parity at the correct offset | parity fallback uses aligned data |
|
||||
|
||||
Instrumented readers record shard index, stripe index, read count, offset, and retirement reason.
|
||||
|
||||
### Metadata, quorum, and commit atomicity (`crates/filemeta/src/filemeta/version.rs`, `set_disk/read.rs`, `set_disk/metadata.rs`, `set_disk/ops/object.rs`, `disk/local.rs`)
|
||||
|
||||
| Area | Scenarios | Assertions |
|
||||
| --- | --- | --- |
|
||||
| Metadata tamper | same `version_id`/`mod_time`, divergent data dir, parts, ETag, size, checksum, inline flag, erasure distribution | previous committed version or read-quorum error; no arbitrary latest |
|
||||
| Early stop | valid quorum, stale quorum, corrupt trailing disks, slow trailing disks | early-stop only on safe identity |
|
||||
| Quorum downgrade | read quorum vs write quorum; delete-marker quorum; version-not-found quorum | no mutation below write quorum |
|
||||
| Rename atomicity | failure before data rename, after data rename, after metadata rename, cleanup failure | object is old or new; never mixed |
|
||||
| Rollback | failed commit quorum and stale temp data | rollback preserves old metadata and data |
|
||||
| Malformed metadata | oversized lengths, bad CRC, invalid versions, invalid UUID/timestamp/enum, huge parts | bounded memory; typed error; no panic |
|
||||
|
||||
## Black-box scenario matrix
|
||||
|
||||
Real S3 and admin behaviour runs through `crates/e2e_test`. Extend `crates/e2e_test/src/chaos.rs` rather than adding ad hoc fault helpers.
|
||||
|
||||
### Single-node 4-disk EC (`reliability_disk_fault_test.rs`, `heal_erasure_disk_rebuild_test.rs`, `chaos.rs`)
|
||||
|
||||
| Scenario | Required assertions |
|
||||
| --- | --- |
|
||||
| baseline PUT/GET/HEAD/List for tiny, inline, block-boundary, multi-block, multipart objects | SHA256 manifest matches; metadata is consistent on all disks |
|
||||
| one disk offline during read | existing objects readable; no corrupted bytes |
|
||||
| one disk offline during write | write succeeds only when write quorum holds; restored disk is healed |
|
||||
| above-parity disk loss | GET/PUT fails with a quorum error; no partial bytes accepted |
|
||||
| corrupt data shard and parity shard | GET returns original bytes or fails closed; read-repair/heal restores |
|
||||
| corrupt inline `xl.meta` | fail closed or heal; no forged inline data |
|
||||
| range read with offline/corrupt shard | exact range bytes; invalid ranges produce the expected S3 errors |
|
||||
| multipart part resend and concurrent same-part writes | final object matches the chosen committed parts |
|
||||
| crash during multipart complete/put/delete | after restart only the old or the new full version is visible |
|
||||
|
||||
### Distributed 4-node / 16-disk EC (`cluster_concurrency_test.rs`, `namespace_lock_quorum_test.rs`, `stale_multipart_cleanup_cluster_test.rs`)
|
||||
|
||||
Single-process unit tests cannot prove RPC, HTTP/2, timeout, or distributed-quorum behaviour, so this layer is mandatory.
|
||||
|
||||
| Scenario | Required assertions |
|
||||
| --- | --- |
|
||||
| node/disk outage while reading large objects | no EOF/truncation; SHA256 manifest matches |
|
||||
| write while one remote node is down | write follows quorum; later heal reconstructs the remote disk |
|
||||
| remote shard bitrot | degraded read uses clean shards; no bad bytes |
|
||||
| concurrent GET/PUT/DELETE/List on the same key | no 500 for expected conflicts; no dirty reads |
|
||||
| range GET matrix for large objects | sequential and parallel ranges match the full-object hash |
|
||||
| internode timeout / slow disk | typed error or fallback; no desync |
|
||||
|
||||
### Versioning, delete markers, and migration (`delete_marker_migration_semantics_test.rs`)
|
||||
|
||||
| Scenario | Required assertions |
|
||||
| --- | --- |
|
||||
| latest delete marker | GET/HEAD/ListObjectVersions match S3 semantics |
|
||||
| explicit `versionId` for old versions | exact old bytes and metadata |
|
||||
| suspended/null version | no version-ordering regression |
|
||||
| delete marker during heal/rebalance/decommission | marker visibility and history are preserved |
|
||||
| orphan directory cleanup | real objects are not purged; tombstones behave correctly |
|
||||
|
||||
### Heal, rebalance, and decommission
|
||||
|
||||
| Scenario | Required assertions |
|
||||
| --- | --- |
|
||||
| auto heal and admin deep heal | data hash unchanged; `xl.meta` and format data rebuilt |
|
||||
| heal interruption/restart | idempotent recovery; no dangling temp objects |
|
||||
| two-pool rebalance with versioned/multipart objects | source and target pools have consistent versions |
|
||||
| decommission cancel/restart/finalize | target readable; source cleanup safe |
|
||||
| rebalance/decommission with node outage | progress resumes; no duplicate or missing versions |
|
||||
|
||||
`scripts/test/decommission_*.sh` cover parts of this table but are not runner steps because they emit no machine-readable pass/fail artifact.
|
||||
|
||||
## Large-object and fuzz gates
|
||||
|
||||
Stand-alone harnesses indexed in `scripts/README.md`, not runner steps: `scripts/run_get_codec_streaming_smoke.sh` (legacy vs codec GET parity), `scripts/run_gt1g_get_http_matrix.sh` (sequential and parallel range GET above 1 GiB), `scripts/run_gt1g_multipart_put_matrix.sh` (multipart PUT above 1 GiB).
|
||||
|
||||
The `fuzz` profile delegates to `scripts/fuzz/run.sh`; targets, corpus rules, and crash-reproducer locations are in `fuzz/README.md`. Malformed-storage-input surfaces the corpus must cover: `xl.meta` MessagePack and legacy filemeta versions, RPC payload decoding, checksum and bitrot headers, range offset/length overflow, object names with path traversal or encoded separators, huge inline metadata and part counts.
|
||||
|
||||
Reference in New Issue
Block a user