diff --git a/CLAUDE.md b/CLAUDE.md index 2bfb9f6d0..5e75fe3cb 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -31,6 +31,8 @@ make build-docker BUILD_OS=ubuntu22.04 - Architecture, layering, crate map: [ARCHITECTURE.md](ARCHITECTURE.md) - Migration guardrails & readiness contracts: [docs/architecture/](docs/architecture/README.md) - CI gates: `.github/workflows/ci.yml` (source of truth; never copy its steps into docs) +- Test-layer taxonomy, per-layer entry commands, serial/nextest rules, flake + policy: [docs/testing/README.md](docs/testing/README.md) - Tier/ILM transition debugging (xl.meta inspection, versionId tracing): [docs/operations/tier-ilm-debugging.md](docs/operations/tier-ilm-debugging.md) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c4f5b079b..6414fa9d7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -68,6 +68,8 @@ make pre-pr > `make test` requires [cargo-nextest](https://nexte.st) (CI runs it and only nextest honours `.config/nextest.toml` test-groups). Install it with `cargo install cargo-nextest --locked` or a prebuilt binary (see https://nexte.st/docs/installation/). To run the plain `cargo test` fallback anyway (results not authoritative — serialization semantics differ from CI), set `RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1`. +> For the full test-layer taxonomy (unit / ecstore black-box / e2e / s3s-e2e / S3 compatibility / chaos / fuzz / bench), each layer's entry command, the naming conventions the migration gate depends on, and the serial/nextest rules, see [docs/testing/README.md](docs/testing/README.md). + ### 🔒 Automated Pre-commit Hooks #### What `make pre-commit` and `make pre-pr` actually run diff --git a/docs/testing/README.md b/docs/testing/README.md index 05a335477..22882b81d 100644 --- a/docs/testing/README.md +++ b/docs/testing/README.md @@ -1,15 +1,67 @@ # RustFS Testing > **Owner: backlog#1153 (infra-11).** This file is the authoritative home for -> test taxonomy, naming conventions, and serial/quarantine rules. It is -> currently a **skeleton** — infra-11 fills in the remaining sections. The -> event × budget × required matrix lives in `docs/testing/ci-gates.md` -> (ci-15); this file links to it rather than duplicating counts. +> the test-layer taxonomy, naming conventions, and serial/nextest rules. The +> event × budget × required-status matrix is owned separately by +> `docs/testing/ci-gates.md` (backlog#1149 ci-15); this file links to it rather +> than duplicating counts, timeouts, or required-check names. ## Test taxonomy -_TODO (infra-11): unit / e2e-smoke / e2e-full / e2e-nightly / protocols / -s3-tests / fuzz / perf, with naming conventions._ +RustFS layers its tests from cheap-and-narrow to expensive-and-broad. Higher +layers catch what lower layers cannot but cost more wall-clock and setup, so +each layer has a clear entry point and a clear "when". Pick the lowest layer +that can prove your change; add a higher-layer test only when the behaviour is +not observable below it. + +| Layer | What it covers | Entry command | When it runs | +|---|---|---|---| +| Unit & crate integration | Per-crate logic and in-process integration tests, run under nextest | `cargo nextest run --all --exclude e2e_test` (or `-p `) | Every PR (required) | +| ecstore black-box | Erasure-coded read/write/recovery validation of the ecstore stack | `scripts/run_ecstore_validation_suite.sh --profile quick` | Local / release validation (not in CI workflows) | +| e2e (`e2e_test` crate) | Full server spun up per test, driven over the S3 API | `cargo nextest run --profile e2e-smoke -p e2e_test` | PR smoke lane + scheduled full/nightly lanes | +| s3s-e2e conformance | External S3 conformance tool run against a live rustfs server | `./scripts/e2e-run.sh ./target/debug/rustfs /tmp/rustfs-e2e-data` | Per-PR e2e gate (`e2e-tests` job) | +| S3 compatibility | Third-party suites: `ceph/s3-tests` (boto3) and MinIO `mint` (many SDKs) | `scripts/s3-tests/run.sh` (mint: `.github/workflows/mint.yml`) | s3-tests: per-PR gate; mint: scheduled, report-only | +| Chaos / fault-injection | Multi-node, power-loss, and disk-fault harness | — (harness planned) | Planned — tracked in backlog#1100 | +| Fuzz | `cargo-fuzz` targets over untrusted parsing/validation surfaces | `./scripts/fuzz/run.sh` (or `cd fuzz && cargo +nightly fuzz run `) | PR smoke + nightly corpus (`.github/workflows/fuzz.yml`) | +| Benchmarks | Criterion micro/throughput benchmarks | `cargo bench -p ` | On-demand / local | + +> The **When it runs** column is a qualitative pointer only. The authoritative +> event × timeout × required-status matrix lives in `docs/testing/ci-gates.md` +> (ci-15) — do not duplicate its numbers here. + +Layer notes: + +- **Unit & crate integration** — the primary gate. `make test` wraps + `cargo nextest run --all --exclude e2e_test`; CI runs the same set under the + strict `ci` profile (`.config/nextest.toml`). Requires `cargo-nextest` (see + [Serial execution & nextest profiles](#serial-execution--nextest-profiles)). +- **ecstore black-box** — `scripts/run_ecstore_validation_suite.sh` has four + profiles (`quick` / `full` / `destructive` / `fuzz`); `quick` is the + PR-smoke-sized core read/write/recovery pass. It is a local and + release-validation tool, not wired into a CI workflow. +- **e2e** — each test spawns its own single-node rustfs server on a random port + with an isolated temp dir, so the suite is parallel-safe. The `e2e-smoke` + profile is the single PR wiring mechanism; slow/cross-process suites run in + scheduled lanes (`e2e-repl-nightly`, and the reliability group). Full + contributor guide: [`crates/e2e_test/README.md`](../../crates/e2e_test/README.md); + per-module counts: [`e2e-suite-inventory.md`](e2e-suite-inventory.md). +- **s3s-e2e** — an external black-box conformance tool installed in CI; locally + run it against a freshly built binary with `scripts/e2e-run.sh + `. +- **S3 compatibility** — `ceph/s3-tests` exercises S3 semantics through boto3 + and is gated on a committed allow-list + (`scripts/s3-tests/implemented_tests.txt`); MinIO `mint` runs many real + client SDKs and is report-only until suites pass reliably (see the header of + `.github/workflows/mint.yml`). +- **Chaos / fault-injection** — multi-node, power-loss, and disk-fault + scenarios are out of scope for this repo's in-tree suites; the harness is + tracked in backlog#1100. (Single-node disk-fault e2e tests already live in the + e2e crate's `e2e-reliability` group.) +- **Fuzz** — the `cargo-fuzz` harness is an isolated sub-workspace under + `fuzz/` (kept out of the root workspace on purpose); see + [`fuzz/README.md`](../../fuzz/README.md) for targets and corpus rules. +- **Benchmarks** — Criterion benches live under each crate's `benches/`. They + are not a gate; run them locally to compare before/after on a specific crate. ### Security advisory regression tests @@ -18,10 +70,87 @@ advisory -> test map lives in [`docs/testing/security-regressions.md`](security-regressions.md). sec-14 (backlog#1151) formalizes the written admission policy in `AGENTS.md`. -## Serial groups & CI profiles +## Naming conventions -_TODO (infra-11): document the `[test-groups]` mechanism and the `default` vs -`ci` nextest profiles. See `.config/nextest.toml`._ +### Reserved test-name substrings (migration gate) + +The migration-critical CI gate selects tests **by name substring** rather than +by module path, so a rename that drops the substring silently thins the gate. +These substrings are therefore reserved: keep them in the test name when a test +proves migration-critical behaviour. + +| Substring | Guards | +|---|---| +| `data_movement` | Cross-pool / cross-set object data-movement proofs | +| `rebalance` | Pool rebalance correctness | +| `decommission` | Pool decommission correctness | +| `source_cleanup` | Post-migration source cleanup | +| `delete_marker` | Delete-marker handling across migration | + +The gate is enforced by +[`scripts/check_migration_gate_count.sh`](../../scripts/check_migration_gate_count.sh), +which counts the selected tests and fails if the count drops below the committed +floor in +[`.config/migration-gate-floor.txt`](../../.config/migration-gate-floor.txt). +A deliberate reduction must lower the floor in the same PR, so the change is +reviewable in the diff (backlog#1153 infra-12). The substring list above is the +same one the gate uses; keep the two in sync when adding a reserved word. + +### General naming + +- Name a regression test after what it pins: the issue or advisory number + (`..._regression_test`, `..._issue_NNNN_...`) or the invariant it protects, so + a reviewer can find the guard for a past bug by grepping. +- e2e lane membership is driven by test-name patterns in `.config/nextest.toml` + (for example the `_real_dual_node` / `_real_single_node` markers route + replication tests into the nightly lane). Follow the existing marker when + adding a test to an established suite; see + [`crates/e2e_test/README.md`](../../crates/e2e_test/README.md). +- Follow the Rust API Guidelines for symbol naming (see `AGENTS.md`). + +## Serial execution & nextest profiles + +**`cargo-nextest` is the runner.** `make test` requires it and CI installs it; +plain `cargo test` is not a faithful substitute because the two runners execute +tests differently. + +- **Install:** `cargo install cargo-nextest --locked`, or a prebuilt binary + (faster) from . +- **Escape hatch:** `RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 make test` runs the + plain `cargo test` fallback, but the results are **not authoritative** — + serialization semantics differ from CI and `[test-groups]` do not apply. + +**Why `#[serial]` is mostly a no-op under nextest.** nextest runs every test in +its own process, so `serial_test`'s in-process `#[serial]` mutex does **not** +serialize tests against each other. The mechanism that actually serializes +across nextest's process boundary is a nextest `[test-groups]` entry with +`max-threads = 1` (for example `ecstore-serial-flaky` and `e2e-reliability` in +`.config/nextest.toml`). Consequences: + +- Do not add `#[serial]` expecting cross-test isolation under nextest. If two + tests genuinely share process/global state or a fixed external resource, + serialize them with a `[test-groups]` entry, or make each test self-isolating + (per-test instance context — see backlog#1153 infra-7 / infra-8). +- A large share of the repo's existing `#[serial]` markers only affect the + `cargo test` fallback. The serial-debt census and removal plan are tracked in + backlog#1153 infra-7. + +**Profiles** (all defined in `.config/nextest.toml`): + +- `default` — local runs. **Never retries**: a red test locally is a real + failure to investigate, not noise to retry away. +- `ci` — the strict CI gate: global `retries = 0` plus a narrowly-scoped + quarantine list (`retries = 2`) for tests with a tracked OPEN flake issue. +- `e2e-smoke` — the PR smoke subset of the `e2e_test` crate (the single wiring + mechanism for e2e in PR CI). +- `e2e-repl-nightly` — the scheduled slow/cross-process replication lane. + +### Time control (paused vs real clock) + +Time-driven tests should prefer paused time (`tokio::time` with `start_paused` +and `advance`) or explicit event synchronization over fixed `sleep` race +windows. The written convention and the `docs/testing/time-control.md` guide are +added by backlog#1153 infra-4. ## Flake policy