gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-10-27 14:27:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
gateway/integration
rustfs/docs/testing/README.md
T
Chris b06923beaf test(e2e): add scanner/heal topology protection matrix (#8705) 
* test(e2e): add scanner/heal topology protection matrix

Add the S00 protection baseline for the Scanner/Heal program
(rustfs/backlog#2873, program rustfs/backlog#2872): seven
scanner_heal_topology_ cases in the distributed e2e lane that pin pool
expansion, rebalance and decommission behavior before the runtime changes.

The fixture seeds a deterministic inventory (empty, inline, external,
multipart, special and nested keys; a versioned bucket with an explicit
null version, three history versions and a delete marker per key; an
Object Lock bucket with legal hold and COMPLIANCE retention), records
every version with its requested and listed version id, length and
SHA-256, verifies the manifest through every node, reads physical
placement from xl.meta on the pool drives, and reads the strict
durability mode back from the admin API and the node logs.

Fault cases hold the mover at proven windows through file barriers that
compile only with the e2e-test-hooks feature: before the multipart target
commit and before source cleanup (shared by decommission and rebalance),
plus the existing pool-metadata before_publish cutpoint. An unreached
window fails the case. DistCluster gains capture_logs_under so a case can
read node startup events without a global environment variable.

docs/testing/scanner-heal-topology-gates.md maps the gates to the tests
and records the timeout basis.

* test(e2e): keep the barrier failure message static

The error(format!) ratchet rejects new formatted io::Error::other call
sites in ecstore. The reached observation already records the stage and
object, so the injected failure needs no per-call detail.

* test(e2e): require e2e-test-hooks for topology fault cases

The four scanner_heal_topology_ fault cases inject at server barriers that
exist only with the e2e-test-hooks feature. Against a default server they
waited for the barrier timeout and reported it as not reached. Check the
verified binary's features first so the missing prerequisite fails
immediately and is named.

* ci(e2e): build the distributed lane with e2e-test-hooks

Every other e2e lane builds and runs the fault-test hooks server. The
distributed lane alone used the default server, so the topology fault
cases could never reach their barriers there. Build and run with the same
feature so the receipt matches, and update the documented local commands.
The upgrade cases spawn the pinned previous release directly and do not
compare feature sets.

* test(e2e): keep topology artifacts under RUSTFS_E2E_LOG_DIR

The topology fixture captured node logs, barrier observations and evidence
in a temp dir removed on drop, so the distributed workflow uploaded nothing
for these cases. Fall back to RUSTFS_E2E_LOG_DIR, which the workflow
uploads, when no explicit evidence directory is set.

* test(e2e): remove retired pools before restart; race the pre-ACK start

The queue case restarted with decommissioned pools 0 and 1 still on the
command line, which the server rightly refuses. Add
remove_single_node_pools and restart without them, then expect the
remaining pools renumbered to 0 and 1.

The pre-ACK case now races the start request against the barrier. A start
that returns first, or stays in flight past the documented 60 s budget, is
reported as such instead of as an unreached barrier, and the evidence
records the reach latency.

capture_logs_under now overrides a log path assigned from
RUSTFS_E2E_LOG_DIR, so the seed node logs into the case directory too.

* test(e2e): bound fault-window waits by the movement budget

On Linux CI a busy target capacity gate delayed the first fault object by
about two minutes, past the 120 s wait borrowed from the server's release
poll. Wait for a movement window within the 240 s movement budget instead,
and fail at once when the decommission or rebalance ends without entering
the window, so an uninjected fault is never counted.

* test(e2e): bound status probes while waiting for a fault window

The admin decommission status request can wait on a mover held at a test
barrier until the barrier's own 120 s release timeout. The window wait
issued that request each loop, so a reached barrier went unnoticed until
the server timed it out and failed the decommission, which surfaced as a
false resume failure. Bound each status probe to 5 s and treat a slow probe
as still running.

* test(e2e): retry a busy decommission start

A decommission start that meets a busy movement gate can answer a
retryable 503 instead of waiting. The harness treated any non-proof
failure as fatal. Share one start helper that retries 503 within the
case's movement budget and the fleet-proof 500 for 45 s, and keeps every
other status fatal. The pre-ACK case retries 503 within its window budget.
2026-10-21 22:08:17 +09:00

105 lines
11 KiB
Markdown
Raw Permalink Blame History

# RustFS Testing
**Use this when:** you need to pick a test layer for a change, name a test so a gate keeps selecting it, understand why `#[serial]` does nothing under nextest, or handle a flaky test.
**Source of truth:** `.config/nextest.toml` (profiles, test-groups, quarantine), `.config/make/tests.mak` (`make test`), `.github/workflows/*.yml` (what runs when; matrix in [ci-gates.md](ci-gates.md)).
## Test taxonomy
Pick the lowest layer that can prove the change; add a higher-layer test only when the behaviour is not observable below it.
| Layer | What it covers | Entry command | When it runs (details: [ci-gates.md](ci-gates.md)) |
|---|---|---|---|
| Unit & crate integration | Per-crate logic and in-process integration tests | `cargo nextest run --all --exclude e2e_test` (or `-p <crate>`); `make test` wraps it | Every PR, required (`Test and Lint`, `ci` profile) |
| ecstore black-box | Erasure-coded read/write/recovery validation; profiles `quick` / `full` / `destructive` / `fuzz` | `scripts/run_ecstore_validation_suite.sh --profile quick` | Local and release validation only; not wired into any workflow. Contract: [ecstore-validation-suite-design.md](ecstore-validation-suite-design.md) |
| e2e (`e2e_test` crate) | A real `rustfs` binary per test, driven over the S3, admin, and protocol APIs | `python3 scripts/e2e_binary.py build --features e2e-test-hooks`, then `python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke -p e2e_test` | PR: `e2e-smoke` (report-only); merge queue / main push: `e2e-full`; nightly: `e2e-repl-nightly`, `e2e-nightly`, `e2e-protocols`, `e2e-distributed`. Guide: [`crates/e2e_test/README.md`](../../crates/e2e_test/README.md); 4-node 4-disk map: [distributed-e2e.md](distributed-e2e.md) |
| Outbound target matrix | Replication of every object shape (empty, plain, retention, legal hold, multipart, checksummed, SSE-S3) against every remote-target failure mode the fake target models; an explicit expectation table pins known-red cells to an open issue | `python3 scripts/e2e_binary.py build`, then `python3 scripts/e2e_binary.py run -- cargo nextest run -p e2e_test -E 'test(/^replication_target_matrix_test::/)'` | With `e2e-repl-nightly`; required locally for any change to outbound client defaults (SOP: [`docs/postmortems/2026-09-03-replication-checksum-default-regression.md`](../postmortems/2026-09-03-replication-checksum-default-regression.md)) |
| s3s-e2e conformance | External S3 conformance tool against a live server | `./scripts/e2e-run.sh ./target/debug/rustfs <data-dir>` | PR, report-only (second half of the `End-to-End Tests` job) |
| S3 compatibility | `ceph/s3-tests` (boto3; allow-list `scripts/s3-tests/implemented_tests.txt`) and MinIO `mint` | `scripts/s3-tests/run.sh`; mint via `.github/workflows/mint.yml` | s3-tests: PR report-only plus a weekly full sweep; mint: weekly, report-only |
| Chaos / fault-injection | Single-node disk fault injection (`crates/e2e_test/src/chaos.rs`, `crates/e2e_test/src/fault_proxy.rs`) plus the 4-node kill/fresh-drive/blackhole cases in `crates/e2e_test/src/distributed/chaos_test.rs` | Part of the e2e crate (`e2e-reliability` and `e2e-distributed`) | Reliability cases with `e2e-full`; 4-node chaos on storage-sensitive PRs and nightly via `e2e-distributed` |
| Fuzz | `cargo-fuzz` targets over untrusted parsing surfaces; isolated sub-workspace under `fuzz/` | `./scripts/fuzz/run.sh` (see [`fuzz/README.md`](../../fuzz/README.md)) | PR smoke on the paths listed in `.github/workflows/fuzz.yml`, plus nightly corpus |
| Benchmarks | Criterion benches under each crate's `benches/` | `cargo bench -p <crate>` | On demand; never a gate |
| Conditional reads | Paired ordinary/hit/miss GET latency, throughput and read-lock measurements | `scripts/run_conditional_read_abba.py` | Local conditional-read acceptance; [guide and scope](conditional-read-benchmark.md) |
Every script named above is indexed with status and wiring in [`scripts/README.md`](../../scripts/README.md). Fixed GHSA advisories map to named regression tests in [security-regressions.md](security-regressions.md).
The [ECStore core invariant gates](ecstore-core-invariants.md) map required native commit, rollback, lock, read, multipart and metadata regressions to the existing CI run and distinguish them from process and network fault evidence.
The [scanner checkpoint fixture](scanner-checkpoint-fixture.md) diagnoses retained subtree coverage across budget interruption, persistence, reload, and plan invalidation.
The [scanner cache cost profile](scanner-cache-cost.md) separates clone, subtree copy, encoding, and counted save costs without changing production cache behavior.
The [Pool layout compatibility reference](pool-layout-compatibility.md) defines the topology and EC regression matrix for single-drive, single-node multi-drive, and multi-node expansion pools.
The [Scanner/Heal topology protection gates](scanner-heal-topology-gates.md) map G-EXP, G-REB and G-DEC and their fault windows to the `scanner_heal_topology_` cases, their recorded inventory, and the basis of their timeouts.
## Naming conventions
### Reserved test-name substrings (migration gate)
`scripts/check_migration_gate_count.sh` (runs in `Test and Lint`) selects migration-critical tests by name substring and fails when the count drops below `.config/migration-gate-floor.txt`. A rename that drops a substring silently thins the gate, so these substrings are reserved:
| Substring | Guards |
|---|---|
| `data_movement` | Cross-pool / cross-set object data-movement proofs |
| `rebalance` | Pool rebalance correctness |
| `decommission` | Pool decommission correctness |
| `source_cleanup` | Post-migration source cleanup |
| `delete_marker` | Delete-marker handling across migration |
A deliberate reduction lowers the floor in the same PR. The list above mirrors the script; change both together.
### General naming
- Name a regression test after what it pins (issue or advisory number, or the invariant) so `rg` finds the guard for a past bug.
- e2e lane membership is selected by test-name patterns in `.config/nextest.toml` (for example `_real_dual_node` / `_real_single_node` route replication tests to the nightly lane). Follow the existing marker of the suite you extend.
- Symbol naming follows the Rust API Guidelines (see `AGENTS.md`).
## nextest and `#[serial]`
`cargo-nextest` is the runner: `make test` requires it and CI installs it. nextest runs every test in its own process, so `serial_test`'s in-process `#[serial]` mutex does **not** serialize tests against each other; it only affects the plain `cargo test` fallback. Cross-test serialization under nextest comes from a `[test-groups]` entry with `max-threads = 1` in `.config/nextest.toml` (for example `ecstore-serial-flaky`, `e2e-reliability`) or from a `-j 1` lane. Prefer making tests self-isolating (per-test instance context, random port, own temp dir) over adding serialization. `RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 make test` runs plain `cargo test`; its results are not authoritative because `[test-groups]` do not apply.
Time-driven tests use paused tokio time (`start_paused` plus `tokio::time::advance`) or explicit synchronization instead of fixed `sleep` windows.
### Profiles
All profiles are defined in `.config/nextest.toml`; its block comments hold the filters and rationale.
| Profile | Role |
|---|---|
| `default` | Local runs; never retries |
| `ci` | PR gate for everything except `e2e_test`; global `retries = 0` plus the quarantine list |
| `e2e-smoke` | PR subset of `e2e_test` |
| `e2e-full` | Merge-queue / main-push single-node e2e lane |
| `e2e-repl-nightly` | Nightly slow / cross-process replication lane |
| `e2e-nightly` | Nightly serial multi-process cluster fault lane |
| `e2e-distributed` | Storage-sensitive PR and nightly 4-node 4-disk S3 / lock / versioning / replication / quota / expand / decommission / rebalance / site-replication / chaos / upgrade (history + IAM AK/SK) lane |
| `e2e-protocols` | Nightly fixed-port FTPS/SFTP/WebDAV lane, run with `-j 1` |
Membership of each e2e profile is pinned by a digest in `.config/e2e-<profile>-selection.txt` and checked by `scripts/check_test_wiring.py --check-profile <profile>` before the lane runs. To list what a profile selects on your platform (the result is platform-dependent because some modules are linux-only):
```bash
cargo nextest list -p e2e_test --profile e2e-smoke --message-format json \
| jq -r '.["rust-suites"][].testcases | to_entries[] | select(.value["filter-match"].status == "matches") | .key | split("::")[0]' \
| sort | uniq -c
```
## Flake policy
A flaky test fails non-deterministically without a corresponding code change. Retry semantics live in `.config/nextest.toml`: `default` never retries, `ci` has global `retries = 0`, and only quarantined tests get `retries = 2` under `ci`. A quarantined test that passes on retry is marked `flaky` in `target/nextest/ci/junit.xml` (uploaded as a CI artifact); that marker, not a green check, is how a live flake stays visible.
1. **Discover** — a non-deterministic failure (CI or local) or a `flaky` JUnit marker.
2. **Open an issue within 24h** describing symptom, suspected cause, and affected suite. No silent re-runs.
3. **Quarantine** — add a `[[profile.ci.overrides]]` entry with `retries = 2` and a comment linking exactly one OPEN issue. The current quarantine list is the `[[profile.ci.overrides]]` block in `.config/nextest.toml`.
4. **Fix or delete within 30 days** — make the test robust and remove the entry, or delete the test. An entry without a live OPEN issue link is a policy violation.
## Coverage
- `.github/workflows/coverage.yml` measures workspace line coverage on its schedule and on manual dispatch: `cargo llvm-cov nextest --workspace --exclude e2e_test` under the `ci` profile, the same scope as the PR gate. The per-crate table lands in the job summary; lcov and JSON exports are uploaded as an artifact (retention set in the workflow).
- PRs touching the paths listed in `coverage.yml` also run a report-only comparison against `.config/coverage-baselines.toml` via `scripts/check_security_coverage.py`: a regression is recorded in the summary without failing the job; missing or malformed coverage evidence fails closed.
- `make coverage` (`.config/make/coverage.mak`) is the local equivalent; it writes `target/llvm-cov/lcov.info` and `coverage.json` and prints the same table via `scripts/coverage_per_crate.py`.
- Not measured: doctests (`ci.yml` runs them uninstrumented) and the `e2e_test` crate.
- A baseline change needs a linked coverage run and a reviewed explanation in the PR.
- For requested coverage assessments, reuse a report only when its revision, scope, and format match. If measurement is necessary, check disk/tool availability and use package/test-scoped `cargo llvm-cov` with the repository nextest configuration; whole-workspace measurement requires that requested scope.
- Missing branch/changed-line metrics are unknown, not zero. If measurement cannot run, mark coverage unverified and propose tests from inspected code without invented percentages.
- Rank evidenced gaps by changed behavior, data loss/security/compatibility risk, and uncovered branches. Pair each gap with a focused test, failure criterion, and scoped verification; do not introduce a threshold unless required.
Reference in New Issue View Git Blame Copy Permalink