# RustFS Testing

**Use this when:** you need to pick a test layer for a change, name a test so a gate keeps selecting it, understand why `#[serial]` does nothing under nextest, or handle a flaky test. **Source of truth:** `.config/nextest.toml` (profiles, test-groups, quarantine), `.config/make/tests.mak` (`make test`), `.github/workflows/*.yml` (what runs when; matrix in [ci-gates.md](ci-gates.md)).

## Test taxonomy

Pick the lowest layer that can prove the change; add a higher-layer test only when the behaviour is not observable below it.

| Layer | What it covers | Entry command | When it runs (details: [ci-gates.md](ci-gates.md)) |
|---|---|---|---|
| Unit & crate integration | Per-crate logic and in-process integration tests | `cargo nextest run --all --exclude e2e_test` (or `-p <crate>`); `make test` wraps it | Non-documentation PRs, required through `Test and Lint` (`ci` profile); documentation-only selection skips the Rust jobs |
| ecstore black-box | Erasure-coded read/write/recovery validation; profiles `quick` / `full` / `destructive` / `fuzz` | `scripts/run_ecstore_validation_suite.sh --profile quick` | Local and release validation only; not wired into any workflow. Contract: [ecstore-validation-suite-design.md](ecstore-validation-suite-design.md) |
| e2e (`e2e_test` crate) | A real `rustfs` binary per test, driven over the S3, admin, and protocol APIs | `python3 scripts/e2e_binary.py build --features e2e-test-hooks`, then `python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke -p e2e_test` | Non-documentation PRs: `e2e-smoke`, required through `Test and Lint`; merge queue / main push: `e2e-full`; nightly: `e2e-repl-nightly`, `e2e-nightly`, `e2e-protocols`, `e2e-distributed`. Guide: [`crates/e2e_test/README.md`](../../crates/e2e_test/README.md); 4-node 4-disk map: [distributed-e2e.md](distributed-e2e.md) |
| Outbound target matrix | Replication of every object shape (empty, plain, retention, legal hold, multipart, checksummed, SSE-S3) against every remote-target failure mode the fake target models; an explicit expectation table pins known-red cells to an open issue | `python3 scripts/e2e_binary.py build`, then `python3 scripts/e2e_binary.py run -- cargo nextest run -p e2e_test -E 'test(/^replication_target_matrix_test::/)'` | With `e2e-repl-nightly`; required locally for any change to outbound client defaults (SOP: [`docs/postmortems/2026-09-03-replication-checksum-default-regression.md`](../postmortems/2026-09-03-replication-checksum-default-regression.md)) |
| s3s-e2e conformance | External S3 conformance tool against a live server | `./scripts/e2e-run.sh ./target/debug/rustfs <data-dir>` | Non-documentation PRs, required through `Test and Lint` (second half of the `End-to-End Tests` job) |
| S3 compatibility | `ceph/s3-tests` (boto3; allow-list `scripts/s3-tests/implemented_tests.txt`) and MinIO `mint` | `scripts/s3-tests/run.sh`; mint via `.github/workflows/mint.yml` | Non-documentation PRs: implemented and lifecycle S3 lanes required through `Test and Lint`; the separate s3-tests workflow provides additional PR reporting and a weekly full sweep; mint: weekly, report-only |
| Chaos / fault-injection | Single-node disk fault injection (`crates/e2e_test/src/chaos.rs`, `crates/e2e_test/src/fault_proxy.rs`) plus the 4-node kill/fresh-drive/blackhole cases in `crates/e2e_test/src/distributed/chaos_test.rs` | Part of the e2e crate (`e2e-reliability` and `e2e-distributed`) | Reliability cases with `e2e-full`; 4-node chaos on storage-sensitive PRs and nightly via `e2e-distributed` |
| Fuzz | `cargo-fuzz` targets over untrusted parsing surfaces; isolated sub-workspace under `fuzz/` | `./scripts/fuzz/run.sh` (see [`fuzz/README.md`](../../fuzz/README.md)) | PR smoke on the paths listed in `.github/workflows/fuzz.yml`, plus nightly corpus |
| Benchmarks | Criterion benches under each crate's `benches/` | `cargo bench -p <crate>` | On demand; never a gate |
| Conditional reads | Paired ordinary/hit/miss GET latency, throughput and read-lock measurements | `scripts/run_conditional_read_abba.py` | Local conditional-read acceptance; [guide and scope](conditional-read-benchmark.md) |

Every script named above is indexed with status and wiring in [`scripts/README.md`](../../scripts/README.md). Fixed GHSA advisories map to named regression tests in [security-regressions.md](security-regressions.md).

For assistant setup, instruction-loading checks, and scenario-based validation of delivery-policy changes, use [Agent Setup and Instruction Validation](agent-instructions.md).

The [ECStore core invariant gates](ecstore-core-invariants.md) map required native commit, rollback, lock, read, multipart and metadata regressions to the existing CI run and distinguish them from process and network fault evidence.

The [scanner checkpoint fixture](scanner-checkpoint-fixture.md) diagnoses retained subtree coverage across budget interruption, persistence, reload, and plan invalidation.

The [scanner cache cost profile](scanner-cache-cost.md) separates clone, subtree copy, encoding, and counted save costs without changing production cache behavior.

The [scanner memory acceptance runner](scanner-memory-acceptance.md) verifies owned S3 datasets, scanner restart progress, and Linux process/cgroup memory evidence under a fixed container limit.

The [Actions resource accounting guide](ci-resource-accounting.md) covers all attempts, cross-workflow attribution and unknown costs. The [runner capability and storage pilot](ci-runner-pilot.md) provides manual preflight and fixed-source build comparisons without changing required job placement.

The [Pool layout compatibility reference](pool-layout-compatibility.md) defines the topology and EC regression matrix for single-drive, single-node multi-drive, and multi-node expansion pools.

The [Scanner/Heal topology protection gates](scanner-heal-topology-gates.md) map G-EXP, G-REB and G-DEC and their fault windows to the `scanner_heal_topology_` cases, their recorded inventory, and the basis of their timeouts.

The [immutable validation candidate reference](validation-candidates.md) describes isolated package builds and exact candidate resolution for the current auto-testing fleet.

The [distributed multipart RPC delay gates](multipart-rpc-delay.md) define authenticated method-entry barriers, latency and peer-loss oracles, and the boundary between hook evidence and production incident attribution.

## Naming conventions

### Reserved test-name substrings (migration gate)

`scripts/check_migration_gate_count.sh` (runs in `Test and Lint`) selects migration-critical tests by name substring and fails when the count drops below `.config/migration-gate-floor.txt`. A rename that drops a substring silently thins the gate, so these substrings are reserved:

| Substring | Guards |
|---|---|
| `data_movement` | Cross-pool / cross-set object data-movement proofs |
| `rebalance` | Pool rebalance correctness |
| `decommission` | Pool decommission correctness |
| `source_cleanup` | Post-migration source cleanup |
| `delete_marker` | Delete-marker handling across migration |

A renamed or replaced test must retain the protected invariant and remain selected. Lower the floor only for an explicitly reviewed change to the coverage policy, with evidence that removed cases are redundant or obsolete and required behavior remains protected; never lower it to hide a failure. The list above mirrors the script; change both together.

### General naming

- Name a regression test after what it pins (issue or advisory number, or the invariant) so `rg` finds the guard for a past bug.
- e2e lane membership is selected by test-name patterns in `.config/nextest.toml` (for example `_real_dual_node` / `_real_single_node` route replication tests to the nightly lane). Follow the existing marker of the suite you extend.
- Symbol naming follows the Rust API Guidelines (see `AGENTS.md`).

## nextest and `#[serial]`

`cargo-nextest` is the runner: `make test` requires it and CI installs it. nextest runs every test in its own process, so `serial_test`'s in-process `#[serial]` mutex does **not** serialize tests against each other; it only affects the plain `cargo test` fallback. Cross-test serialization under nextest comes from a `[test-groups]` entry with `max-threads = 1` in `.config/nextest.toml` (for example `ecstore-serial-flaky`, `e2e-reliability`) or from a `-j 1` lane. Prefer making tests self-isolating (per-test instance context, random port, own temp dir) over adding serialization. `RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 make test` runs plain `cargo test`; its results are not authoritative because `[test-groups]` do not apply.

Package-scoped ECStore runs do not inherit workspace feature unification. The native decommission lock, capacity-lease and restart fixtures in `core::pools_test::decommission_lock_order_tests` require `test-util`. Select that module separately and fail an empty run, so unrelated tests matching another branch of a combined filter cannot hide its absence:

```bash
cargo nextest run --profile ci -p rustfs-ecstore --lib --features test-util \
  --no-tests fail -E 'test(/^core::pools_test::decommission_lock_order_tests::/)'
```

Time-driven tests use paused tokio time (`start_paused` plus `tokio::time::advance`) or explicit synchronization instead of fixed `sleep` windows.

### Profiles

All profiles are defined in `.config/nextest.toml`; its block comments hold the filters and rationale.

| Profile | Role |
|---|---|
| `default` | Local runs; never retries |
| `ci` | PR gate for everything except `e2e_test`; global `retries = 0` plus the quarantine list |
| `e2e-smoke` | PR subset of `e2e_test` |
| `e2e-full` | Merge-queue / main-push single-node e2e lane |
| `e2e-repl-nightly` | Nightly slow / cross-process replication lane |
| `e2e-nightly` | Nightly serial multi-process cluster fault lane |
| `e2e-distributed` | Storage-sensitive PR and nightly 4-node 4-disk S3 / lock / versioning / replication / quota / expand / decommission / rebalance / site-replication / chaos / upgrade (history + IAM AK/SK) lane |
| `e2e-protocols` | Nightly fixed-port FTPS/SFTP/WebDAV lane, run with `-j 1` |

Membership of each e2e profile is pinned by a digest in `.config/e2e-<profile>-selection.txt` and checked by `scripts/check_test_wiring.py --check-profile <profile>` before the lane runs. To list what a profile selects on your platform (the result is platform-dependent because some modules are linux-only):

```bash
cargo nextest list -p e2e_test --profile e2e-smoke --message-format json \
  | jq -r '.["rust-suites"][].testcases | to_entries[] | select(.value["filter-match"].status == "matches") | .key | split("::")[0]' \
  | sort | uniq -c
```

## Flake policy

A flaky test fails non-deterministically without a corresponding code change. Retry semantics live in `.config/nextest.toml`: `default` never retries, `ci` has global `retries = 0`, and only quarantined tests get `retries = 2` under `ci`. A quarantined test that passes on retry is marked `flaky` in `target/nextest/ci/junit.xml` (uploaded as a CI artifact); that marker, not a green check, is how a live flake stays visible.

1. **Discover** — a non-deterministic failure (CI or local) or a `flaky` JUnit marker.
2. **Track within 24h** — use an existing matching open issue or, when posting is authorized, create one describing the symptom, evidence, suspected cause, and affected suite. Otherwise prepare that report and state the missing authority. Keep failed-attempt evidence; an explained diagnostic rerun does not erase the failure or prove a fix.
3. **Quarantine only when authorized** — prefer a deterministic fix. When a reviewed quarantine is needed, add a `[[profile.ci.overrides]]` entry with `retries = 2` and a comment linking exactly one OPEN issue. Record ownership and a deadline; the current quarantine list is the `[[profile.ci.overrides]]` block in `.config/nextest.toml`.
4. **Resolve within 30 days** — make the test robust and remove the entry. Delete a test only when review establishes it is obsolete or redundant and required behavior remains protected; a flake alone does not justify deleting coverage. An entry without a live OPEN issue link is a policy violation.

## Coverage

- `.github/workflows/coverage.yml` measures workspace line coverage on its schedule and on manual dispatch: `cargo llvm-cov nextest --workspace --exclude e2e_test` under the `ci` profile, the same scope as the PR gate. The per-crate table lands in the job summary; lcov and JSON exports are uploaded as an artifact (retention set in the workflow).
- PRs touching the paths listed in `coverage.yml` also run a report-only comparison against `.config/coverage-baselines.toml` via `scripts/check_security_coverage.py`: a regression is recorded in the summary without failing the job; missing or malformed coverage evidence fails closed.
- `make coverage` (`.config/make/coverage.mak`) is the local equivalent; it writes `target/llvm-cov/lcov.info` and `coverage.json` and prints the same table via `scripts/coverage_per_crate.py`.
- Not measured: doctests (`ci.yml` runs them uninstrumented) and the `e2e_test` crate.
- A baseline change needs a linked coverage run and a reviewed explanation in the PR.
- For requested coverage assessments, reuse a report only when its revision, scope, and format match. If measurement is necessary, check disk/tool availability and use package/test-scoped `cargo llvm-cov` with the repository nextest configuration; whole-workspace measurement requires that requested scope.
- Missing branch/changed-line metrics are unknown, not zero. If measurement cannot run, mark coverage unverified and propose tests from inspected code without invented percentages.
- Rank evidenced gaps by changed behavior, data loss/security/compatibility risk, and uncovered branches. Pair each gap with a focused test, failure criterion, and scoped verification; do not introduce a threshold unless required.