mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-05 19:55:37 +00:00
docs(knowledge-base): prune stale content and add agent-facing index (#7035)
This commit is contained in:
Zhengchao An
+49
-189
@@ -1,87 +1,30 @@
|
||||
# RustFS Testing
|
||||
|
||||
> **Owner: backlog#1153 (infra-11).** This file is the authoritative home for
|
||||
> the test-layer taxonomy, naming conventions, and serial/nextest rules. The
|
||||
> event × budget × required-status matrix is owned separately by
|
||||
> `docs/testing/ci-gates.md` (backlog#1149 ci-15); this file links to it rather
|
||||
> than duplicating counts, timeouts, or required-check names.
|
||||
**Use this when:** you need to pick a test layer for a change, name a test so a gate keeps selecting it, understand why `#[serial]` does nothing under nextest, or handle a flaky test.
|
||||
**Source of truth:** `.config/nextest.toml` (profiles, test-groups, quarantine), `.config/make/tests.mak` (`make test`), `.github/workflows/*.yml` (what runs when; matrix in [ci-gates.md](ci-gates.md)).
|
||||
|
||||
## Test taxonomy
|
||||
|
||||
RustFS layers its tests from cheap-and-narrow to expensive-and-broad. Higher
|
||||
layers catch what lower layers cannot but cost more wall-clock and setup, so
|
||||
each layer has a clear entry point and a clear "when". Pick the lowest layer
|
||||
that can prove your change; add a higher-layer test only when the behaviour is
|
||||
not observable below it.
|
||||
Pick the lowest layer that can prove the change; add a higher-layer test only when the behaviour is not observable below it.
|
||||
|
||||
| Layer | What it covers | Entry command | When it runs |
|
||||
| Layer | What it covers | Entry command | When it runs (details: [ci-gates.md](ci-gates.md)) |
|
||||
|---|---|---|---|
|
||||
| Unit & crate integration | Per-crate logic and in-process integration tests, run under nextest | `cargo nextest run --all --exclude e2e_test` (or `-p <crate>`) | Every PR (required) |
|
||||
| ecstore black-box | Erasure-coded read/write/recovery validation of the ecstore stack | `scripts/run_ecstore_validation_suite.sh --profile quick` | Local / release validation (not in CI workflows) |
|
||||
| e2e (`e2e_test` crate) | Full server spun up per test, driven over the S3 API | `cargo nextest run --profile e2e-smoke -p e2e_test` | PR smoke lane + scheduled full/nightly lanes |
|
||||
| s3s-e2e conformance | External S3 conformance tool run against a live rustfs server | `./scripts/e2e-run.sh ./target/debug/rustfs /tmp/rustfs-e2e-data` | Per-PR e2e gate (`e2e-tests` job) |
|
||||
| S3 compatibility | Third-party suites: `ceph/s3-tests` (boto3) and MinIO `mint` (many SDKs) | `scripts/s3-tests/run.sh` (mint: `.github/workflows/mint.yml`) | s3-tests: per-PR gate; mint: scheduled, report-only |
|
||||
| Chaos / fault-injection | Multi-node, power-loss, and disk-fault harness | — (harness planned) | Planned — tracked in backlog#1100 |
|
||||
| Fuzz | `cargo-fuzz` targets over untrusted parsing/validation surfaces | `./scripts/fuzz/run.sh` (or `cd fuzz && cargo +nightly fuzz run <target>`) | PR smoke + nightly corpus (`.github/workflows/fuzz.yml`) |
|
||||
| Benchmarks | Criterion micro/throughput benchmarks | `cargo bench -p <crate>` | On-demand / local |
|
||||
| Unit & crate integration | Per-crate logic and in-process integration tests | `cargo nextest run --all --exclude e2e_test` (or `-p <crate>`); `make test` wraps it | Every PR, required (`Test and Lint`, `ci` profile) |
|
||||
| ecstore black-box | Erasure-coded read/write/recovery validation; profiles `quick` / `full` / `destructive` / `fuzz` | `scripts/run_ecstore_validation_suite.sh --profile quick` | Local and release validation only; not wired into any workflow. Contract: [ecstore-validation-suite-design.md](ecstore-validation-suite-design.md) |
|
||||
| e2e (`e2e_test` crate) | A real `rustfs` binary per test, driven over the S3, admin, and protocol APIs | `cargo nextest run --profile e2e-smoke -p e2e_test` | PR: `e2e-smoke` (report-only); merge queue / main push: `e2e-full`; nightly: `e2e-repl-nightly`, `e2e-nightly`, `e2e-protocols`. Guide: [`crates/e2e_test/README.md`](../../crates/e2e_test/README.md) |
|
||||
| s3s-e2e conformance | External S3 conformance tool against a live server | `./scripts/e2e-run.sh ./target/debug/rustfs <data-dir>` | PR, report-only (second half of the `End-to-End Tests` job) |
|
||||
| S3 compatibility | `ceph/s3-tests` (boto3; allow-list `scripts/s3-tests/implemented_tests.txt`) and MinIO `mint` | `scripts/s3-tests/run.sh`; mint via `.github/workflows/mint.yml` | s3-tests: PR report-only plus a weekly full sweep; mint: weekly, report-only |
|
||||
| Chaos / fault-injection | Single-node disk fault injection (`crates/e2e_test/src/chaos.rs`, `crates/e2e_test/src/fault_proxy.rs`) used by the reliability and heal e2e modules | Part of the e2e crate (`e2e-reliability` test-group) | With the `e2e-full` and nightly e2e lanes. A multi-node power-loss harness is not in tree |
|
||||
| Fuzz | `cargo-fuzz` targets over untrusted parsing surfaces; isolated sub-workspace under `fuzz/` | `./scripts/fuzz/run.sh` (see [`fuzz/README.md`](../../fuzz/README.md)) | PR smoke on the paths listed in `.github/workflows/fuzz.yml`, plus nightly corpus |
|
||||
| Benchmarks | Criterion benches under each crate's `benches/` | `cargo bench -p <crate>` | On demand; never a gate |
|
||||
|
||||
> The **When it runs** column is a qualitative pointer only. The authoritative
|
||||
> event × timeout × required-status matrix lives in `docs/testing/ci-gates.md`
|
||||
> (ci-15) — do not duplicate its numbers here.
|
||||
|
||||
Layer notes:
|
||||
|
||||
- **Unit & crate integration** — the primary gate. `make test` wraps
|
||||
`cargo nextest run --all --exclude e2e_test`; CI runs the same set under the
|
||||
strict `ci` profile (`.config/nextest.toml`). Requires `cargo-nextest` (see
|
||||
[Serial execution & nextest profiles](#serial-execution--nextest-profiles)).
|
||||
- **ecstore black-box** — `scripts/run_ecstore_validation_suite.sh` has four
|
||||
profiles (`quick` / `full` / `destructive` / `fuzz`); `quick` is the
|
||||
PR-smoke-sized core read/write/recovery pass. It is a local and
|
||||
release-validation tool, not wired into a CI workflow.
|
||||
- **e2e** — each test spawns its own single-node rustfs server on a random port
|
||||
with an isolated temp dir, so the suite is parallel-safe. The `e2e-smoke`
|
||||
profile is the single PR wiring mechanism; slow/cross-process suites run in
|
||||
scheduled lanes (`e2e-repl-nightly`, and the reliability group). Full
|
||||
contributor guide: [`crates/e2e_test/README.md`](../../crates/e2e_test/README.md);
|
||||
per-module counts: [`e2e-suite-inventory.md`](e2e-suite-inventory.md).
|
||||
- **s3s-e2e** — an external black-box conformance tool installed in CI; locally
|
||||
run it against a freshly built binary with `scripts/e2e-run.sh <binary>
|
||||
<data-dir>`.
|
||||
- **S3 compatibility** — `ceph/s3-tests` exercises S3 semantics through boto3
|
||||
and is gated on a committed allow-list
|
||||
(`scripts/s3-tests/implemented_tests.txt`); MinIO `mint` runs many real
|
||||
client SDKs and is report-only until suites pass reliably (see the header of
|
||||
`.github/workflows/mint.yml`).
|
||||
- **Chaos / fault-injection** — multi-node, power-loss, and disk-fault
|
||||
scenarios are out of scope for this repo's in-tree suites; the harness is
|
||||
tracked in backlog#1100. (Single-node disk-fault e2e tests already live in the
|
||||
e2e crate's `e2e-reliability` group.)
|
||||
- **Fuzz** — the `cargo-fuzz` harness is an isolated sub-workspace under
|
||||
`fuzz/` (kept out of the root workspace on purpose); see
|
||||
[`fuzz/README.md`](../../fuzz/README.md) for targets and corpus rules.
|
||||
- **Benchmarks** — Criterion benches live under each crate's `benches/`. They
|
||||
are not a gate; run them locally to compare before/after on a specific crate.
|
||||
|
||||
Script inventory: every entry under `scripts/` — including the runners above —
|
||||
is indexed in [`scripts/README.md`](../../scripts/README.md) with its status
|
||||
(ci-gate / dev-tool / archived) and wiring.
|
||||
|
||||
### Security advisory regression tests
|
||||
|
||||
Fixed GHSA advisories map to named, discoverable regression tests. The
|
||||
advisory -> test map lives in
|
||||
[`docs/testing/security-regressions.md`](security-regressions.md). sec-14
|
||||
(backlog#1151) formalizes the written admission policy in `AGENTS.md`.
|
||||
Every script named above is indexed with status and wiring in [`scripts/README.md`](../../scripts/README.md). Fixed GHSA advisories map to named regression tests in [security-regressions.md](security-regressions.md).
|
||||
|
||||
## Naming conventions
|
||||
|
||||
### Reserved test-name substrings (migration gate)
|
||||
|
||||
The migration-critical CI gate selects tests **by name substring** rather than
|
||||
by module path, so a rename that drops the substring silently thins the gate.
|
||||
These substrings are therefore reserved: keep them in the test name when a test
|
||||
proves migration-critical behaviour.
|
||||
`scripts/check_migration_gate_count.sh` (runs in `Test and Lint`) selects migration-critical tests by name substring and fails when the count drops below `.config/migration-gate-floor.txt`. A rename that drops a substring silently thins the gate, so these substrings are reserved:
|
||||
|
||||
| Substring | Guards |
|
||||
|---|---|
|
||||
@@ -91,138 +34,55 @@ proves migration-critical behaviour.
|
||||
| `source_cleanup` | Post-migration source cleanup |
|
||||
| `delete_marker` | Delete-marker handling across migration |
|
||||
|
||||
The gate is enforced by
|
||||
[`scripts/check_migration_gate_count.sh`](../../scripts/check_migration_gate_count.sh),
|
||||
which counts the selected tests and fails if the count drops below the committed
|
||||
floor in
|
||||
[`.config/migration-gate-floor.txt`](../../.config/migration-gate-floor.txt).
|
||||
A deliberate reduction must lower the floor in the same PR, so the change is
|
||||
reviewable in the diff (backlog#1153 infra-12). The substring list above is the
|
||||
same one the gate uses; keep the two in sync when adding a reserved word.
|
||||
A deliberate reduction lowers the floor in the same PR. The list above mirrors the script; change both together.
|
||||
|
||||
### General naming
|
||||
|
||||
- Name a regression test after what it pins: the issue or advisory number
|
||||
(`..._regression_test`, `..._issue_NNNN_...`) or the invariant it protects, so
|
||||
a reviewer can find the guard for a past bug by grepping.
|
||||
- e2e lane membership is driven by test-name patterns in `.config/nextest.toml`
|
||||
(for example the `_real_dual_node` / `_real_single_node` markers route
|
||||
replication tests into the nightly lane). Follow the existing marker when
|
||||
adding a test to an established suite; see
|
||||
[`crates/e2e_test/README.md`](../../crates/e2e_test/README.md).
|
||||
- Follow the Rust API Guidelines for symbol naming (see `AGENTS.md`).
|
||||
- Name a regression test after what it pins (issue or advisory number, or the invariant) so `rg` finds the guard for a past bug.
|
||||
- e2e lane membership is selected by test-name patterns in `.config/nextest.toml` (for example `_real_dual_node` / `_real_single_node` route replication tests to the nightly lane). Follow the existing marker of the suite you extend.
|
||||
- Symbol naming follows the Rust API Guidelines (see `AGENTS.md`).
|
||||
|
||||
## Serial execution & nextest profiles
|
||||
## nextest and `#[serial]`
|
||||
|
||||
**`cargo-nextest` is the runner.** `make test` requires it and CI installs it;
|
||||
plain `cargo test` is not a faithful substitute because the two runners execute
|
||||
tests differently.
|
||||
`cargo-nextest` is the runner: `make test` requires it and CI installs it. nextest runs every test in its own process, so `serial_test`'s in-process `#[serial]` mutex does **not** serialize tests against each other; it only affects the plain `cargo test` fallback. Cross-test serialization under nextest comes from a `[test-groups]` entry with `max-threads = 1` in `.config/nextest.toml` (for example `ecstore-serial-flaky`, `e2e-reliability`) or from a `-j 1` lane. Prefer making tests self-isolating (per-test instance context, random port, own temp dir) over adding serialization. `RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 make test` runs plain `cargo test`; its results are not authoritative because `[test-groups]` do not apply.
|
||||
|
||||
- **Install:** `cargo install cargo-nextest --locked`, or a prebuilt binary
|
||||
(faster) from <https://nexte.st/docs/installation/>.
|
||||
- **Escape hatch:** `RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 make test` runs the
|
||||
plain `cargo test` fallback, but the results are **not authoritative** —
|
||||
serialization semantics differ from CI and `[test-groups]` do not apply.
|
||||
Time-driven tests use paused tokio time (`start_paused` plus `tokio::time::advance`) or explicit synchronization instead of fixed `sleep` windows.
|
||||
|
||||
**Why `#[serial]` is mostly a no-op under nextest.** nextest runs every test in
|
||||
its own process, so `serial_test`'s in-process `#[serial]` mutex does **not**
|
||||
serialize tests against each other. The mechanism that actually serializes
|
||||
across nextest's process boundary is a nextest `[test-groups]` entry with
|
||||
`max-threads = 1` (for example `ecstore-serial-flaky` and `e2e-reliability` in
|
||||
`.config/nextest.toml`). Consequences:
|
||||
### Profiles
|
||||
|
||||
- Do not add `#[serial]` expecting cross-test isolation under nextest. If two
|
||||
tests genuinely share process/global state or a fixed external resource,
|
||||
serialize them with a `[test-groups]` entry, or make each test self-isolating
|
||||
(per-test instance context — see backlog#1153 infra-7 / infra-8).
|
||||
- A large share of the repo's existing `#[serial]` markers only affect the
|
||||
`cargo test` fallback. The serial-debt census and removal plan are tracked in
|
||||
backlog#1153 infra-7.
|
||||
All profiles are defined in `.config/nextest.toml`; its block comments hold the filters and rationale.
|
||||
|
||||
**Profiles** (all defined in `.config/nextest.toml`):
|
||||
| Profile | Role |
|
||||
|---|---|
|
||||
| `default` | Local runs; never retries |
|
||||
| `ci` | PR gate for everything except `e2e_test`; global `retries = 0` plus the quarantine list |
|
||||
| `e2e-smoke` | PR subset of `e2e_test` |
|
||||
| `e2e-full` | Merge-queue / main-push single-node e2e lane |
|
||||
| `e2e-repl-nightly` | Nightly slow / cross-process replication lane |
|
||||
| `e2e-nightly` | Nightly serial multi-process cluster fault lane |
|
||||
| `e2e-protocols` | Nightly fixed-port FTPS/SFTP/WebDAV lane, run with `-j 1` |
|
||||
|
||||
- `default` — local runs. **Never retries**: a red test locally is a real
|
||||
failure to investigate, not noise to retry away.
|
||||
- `ci` — the strict CI gate: global `retries = 0` plus a narrowly-scoped
|
||||
quarantine list (`retries = 2`) for tests with a tracked OPEN flake issue.
|
||||
- `e2e-smoke` — the PR smoke subset of the `e2e_test` crate (the single wiring
|
||||
mechanism for e2e in PR CI).
|
||||
- `e2e-repl-nightly` — the scheduled slow/cross-process replication lane.
|
||||
Membership of each e2e profile is pinned by a digest in `.config/e2e-<profile>-selection.txt` and checked by `scripts/check_test_wiring.py --check-profile <profile>` before the lane runs. To list what a profile selects on your platform (the result is platform-dependent because some modules are linux-only):
|
||||
|
||||
### Time control (paused vs real clock)
|
||||
|
||||
Time-driven tests should prefer paused time (`tokio::time` with `start_paused`
|
||||
and `advance`) or explicit event synchronization over fixed `sleep` race
|
||||
windows. The written convention and the `docs/testing/time-control.md` guide are
|
||||
added by backlog#1153 infra-4.
|
||||
|
||||
## Coverage
|
||||
|
||||
Workspace line coverage is measured weekly. Pull requests that touch iam, kms,
|
||||
policy, or crypto also run a non-required, report-only comparison against
|
||||
`.config/coverage-baselines.toml`. During calibration, a regression is recorded
|
||||
in the job summary without failing the job; missing or malformed coverage
|
||||
evidence still fails closed (backlog#1153 infra-6).
|
||||
|
||||
- **CI**: `.github/workflows/coverage.yml` runs every Sunday and on manual
|
||||
dispatch: `cargo llvm-cov nextest --workspace --exclude e2e_test` under the
|
||||
`ci` nextest profile — the same scope and profile as the PR test gate. The
|
||||
per-crate line-coverage table lands in the run's job summary; the lcov +
|
||||
JSON exports are uploaded as a `coverage-lcov-<run>` artifact kept for
|
||||
90 days. Scheduled failures open/append the `[scheduled-failure] coverage`
|
||||
issue via the shared alert action (ci-8).
|
||||
- **Local**: `make coverage` is the equivalent (slow — instrumented rebuild
|
||||
plus the full suite). It prints the same per-crate table via
|
||||
`scripts/coverage_per_crate.py` and writes `target/llvm-cov/lcov.info` and
|
||||
`coverage.json`.
|
||||
- **Security-critical ratchet**: relevant pull requests compare iam / kms /
|
||||
policy / crypto line coverage with the versioned baseline. Drops greater than
|
||||
the configured one-percentage-point calibration threshold are marked
|
||||
`REGRESSION (report-only)`. The weekly summary runs the same comparison so
|
||||
calibration continues even when no relevant pull request is open. Baseline
|
||||
changes require a linked coverage run and a reviewed explanation.
|
||||
- **Trend comparison**: each run's job summary is the weekly per-crate
|
||||
snapshot — open two runs from the Actions history (workflow "coverage") and
|
||||
compare their tables. For line-level diffs, download the two runs'
|
||||
`coverage-lcov-*` artifacts and compare the `lcov.info` files with your lcov
|
||||
tooling of choice.
|
||||
- **Not measured**: doctests (ci.yml runs them uninstrumented; covering them
|
||||
would require a nightly toolchain) and the `e2e_test` crate (excluded from
|
||||
the unit gate; its lanes are described in the taxonomy above).
|
||||
```bash
|
||||
cargo nextest list -p e2e_test --profile e2e-smoke --message-format json \
|
||||
| jq -r '.["rust-suites"][].testcases | to_entries[] | select(.value["filter-match"].status == "matches") | .key | split("::")[0]' \
|
||||
| sort | uniq -c
|
||||
```
|
||||
|
||||
## Flake policy
|
||||
|
||||
A flaky test is one that fails non-deterministically without a corresponding
|
||||
code change. Flakes erode trust in the gate and block tightening required
|
||||
checks, so they are handled on a strict, time-boxed loop.
|
||||
A flaky test fails non-deterministically without a corresponding code change. Retry semantics live in `.config/nextest.toml`: `default` never retries, `ci` has global `retries = 0`, and only quarantined tests get `retries = 2` under `ci`. A quarantined test that passes on retry is marked `flaky` in `target/nextest/ci/junit.xml` (uploaded as a CI artifact); that marker, not a green check, is how a live flake stays visible.
|
||||
|
||||
**Retry semantics (source of truth: `.config/nextest.toml`):**
|
||||
1. **Discover** — a non-deterministic failure (CI or local) or a `flaky` JUnit marker.
|
||||
2. **Open an issue within 24h** describing symptom, suspected cause, and affected suite. No silent re-runs.
|
||||
3. **Quarantine** — add a `[[profile.ci.overrides]]` entry with `retries = 2` and a comment linking exactly one OPEN issue. The current quarantine list is the `[[profile.ci.overrides]]` block in `.config/nextest.toml`.
|
||||
4. **Fix or delete within 30 days** — make the test robust and remove the entry, or delete the test. An entry without a live OPEN issue link is a policy violation.
|
||||
|
||||
- The **local `default` profile never retries.** A red test on your machine is
|
||||
a real failure to investigate, not noise to paper over.
|
||||
- The **CI `ci` profile runs with global `retries = 0`.** A new race must fail
|
||||
on its first occurrence so the first crime scene is never masked.
|
||||
- Only tests on the **quarantine list** get `retries = 2`, and only under the
|
||||
`ci` profile. Each quarantine entry MUST link exactly one OPEN issue.
|
||||
- **JUnit flaky markers are the observable.** A quarantined test that passes
|
||||
only after a retry is marked `flaky` in `target/nextest/ci/junit.xml`
|
||||
(uploaded as a CI artifact). That marker — not a green check — is how we see
|
||||
a flake is still live.
|
||||
## Coverage
|
||||
|
||||
**Lifecycle of a flake:**
|
||||
|
||||
1. **Discover** — a test fails non-deterministically (CI or local), or shows a
|
||||
`flaky` marker in the JUnit report.
|
||||
2. **Open an issue within 24h** — file/track an issue describing the flake
|
||||
(symptom, suspected cause, affected suite). No silent re-runs.
|
||||
3. **Quarantine** — add the test to the quarantine override block in
|
||||
`.config/nextest.toml` with a comment linking that OPEN issue. This grants
|
||||
`retries = 2` under CI so the flake stops reddening unrelated PRs, while the
|
||||
`flaky` marker keeps it visible.
|
||||
4. **Fix or delete within 30 days** — make the test robust (then remove the
|
||||
quarantine entry) or delete the test. A quarantine entry may not outlive its
|
||||
fix window; an entry without a live OPEN issue link is a policy violation.
|
||||
|
||||
First quarantine members: the two backlog#937 ecstore groups
|
||||
(`concurrent_resend_same_part_commits_one_generation` and
|
||||
`store::bucket::tests::bucket_delete_*`).
|
||||
- `.github/workflows/coverage.yml` measures workspace line coverage on its schedule and on manual dispatch: `cargo llvm-cov nextest --workspace --exclude e2e_test` under the `ci` profile, the same scope as the PR gate. The per-crate table lands in the job summary; lcov and JSON exports are uploaded as an artifact (retention set in the workflow).
|
||||
- PRs touching the paths listed in `coverage.yml` also run a report-only comparison against `.config/coverage-baselines.toml` via `scripts/check_security_coverage.py`: a regression is recorded in the summary without failing the job; missing or malformed coverage evidence fails closed.
|
||||
- `make coverage` (`.config/make/coverage.mak`) is the local equivalent; it writes `target/llvm-cov/lcov.info` and `coverage.json` and prints the same table via `scripts/coverage_per_crate.py`.
|
||||
- Not measured: doctests (`ci.yml` runs them uninstrumented) and the `e2e_test` crate.
|
||||
- A baseline change needs a linked coverage run and a reviewed explanation in the PR.
|
||||
|
||||
+68
-120
@@ -1,154 +1,102 @@
|
||||
# CI gate matrix
|
||||
|
||||
This file is the source of truth for which validation runs on each event, its
|
||||
configured wall-clock budget, and whether it can block a merge. Test taxonomy,
|
||||
naming, and nextest serialization rules remain in [README.md](README.md); e2e
|
||||
membership and counts remain in
|
||||
[e2e-suite-inventory.md](e2e-suite-inventory.md).
|
||||
**Use this when:** a check is red and you need to know whether it blocks the merge, which workflow and job produced it, and how to reproduce it locally.
|
||||
**Source of truth:** the live `main` ruleset (command below) for required status; `.github/workflows/<file>.yml` for triggers, `paths`, `timeout-minutes`, and cron; `.config/nextest.toml` for e2e profile filters; `.github/scheduled-validations.json` for the freshness-watchdog list.
|
||||
|
||||
The distinction between **required** and **report-only** is load-bearing:
|
||||
a failing job blocks a merge only when its exact check name is present in the
|
||||
live `main` ruleset. A workflow name, a `merge_group` trigger, or a red PR check
|
||||
does not make a job required by itself.
|
||||
A job blocks a merge only when its exact check name is in the live `main` ruleset. A workflow name, a `merge_group` trigger, or a red PR check does not make a job required by itself.
|
||||
|
||||
## Required merge checks
|
||||
|
||||
The live `main` ruleset (`6436880`) currently requires exactly these contexts:
|
||||
The `main` ruleset (`6436880`) requires exactly these contexts, with `strict_required_status_checks_policy=false`:
|
||||
|
||||
| Required context | Producer | Validation |
|
||||
|---|---|---|
|
||||
| `CLA Check` | `.github/workflows/cla.yml` | Contributor agreement |
|
||||
| `Quick Checks` | `.github/workflows/ci.yml` | Formatting and repository guard scripts |
|
||||
| `Test and Lint` | `.github/workflows/ci.yml` | Clippy, workspace nextest excluding `e2e_test`, doctests, and migration proofs |
|
||||
| `CLA Check` | `cla.yml` | Contributor agreement |
|
||||
| `Quick Checks` | `ci.yml` job `quick-checks` | Formatting and repository guard scripts |
|
||||
| `Test and Lint` | `ci.yml` job `test-and-lint` | Clippy, workspace nextest (`ci` profile, excluding `e2e_test`), doctests, migration-gate count (`scripts/check_migration_gate_count.sh`) |
|
||||
|
||||
For pull requests limited to the paths excluded by the main CI workflow,
|
||||
`.github/workflows/ci-docs-only.yml` reports `Quick Checks` and
|
||||
`Test and Lint` under the same names. It runs the real quick checks and the
|
||||
planning-document guard; it does not claim that Rust compilation or runtime
|
||||
tests ran. Despite the workflow name, these paths also include selected deploy,
|
||||
workflow, and lock files.
|
||||
For PRs limited to the `paths-ignore` list in `ci.yml`, `ci-docs-only.yml` reports `Quick Checks` and `Test and Lint` under the same names; it runs the quick checks and `scripts/check_no_planning_docs.sh`, not a Rust build or tests. `scripts/check_ci_paths_sync.sh` keeps the two path lists aligned.
|
||||
|
||||
Verify the live rule rather than trusting this snapshot before changing merge
|
||||
policy:
|
||||
Verify the live rule before changing merge policy:
|
||||
|
||||
```bash
|
||||
gh api repos/rustfs/rustfs/rulesets/6436880 \
|
||||
--jq '.rules[] | select(.type == "required_status_checks") | .parameters'
|
||||
```
|
||||
|
||||
The ruleset currently has `strict_required_status_checks_policy=false`.
|
||||
`Continuous Integration` accepts `merge_group` events and runs `e2e-full` for
|
||||
them, but `End-to-End Tests (full merge gate)` is not currently a required
|
||||
context. Therefore the repository is prepared to test a merge-queue SHA, but
|
||||
the workflow alone does not prove that every merge passed that lane.
|
||||
Promotion rule: never promote a report-only lane to required from one green run. Require at least 14 days and 30 representative PRs with at least 99% complete execution, then update the ruleset and this file together.
|
||||
|
||||
## Pull request and merge matrix
|
||||
|
||||
Budgets below are job `timeout-minutes`, not typical runtimes. “Report-only”
|
||||
means the result is visible and actionable but is not in the live required
|
||||
context list.
|
||||
"Report-only" means visible and actionable but not in the required list. Budgets are each job's `timeout-minutes` in the named workflow and are not copied here.
|
||||
|
||||
| Event | Validation | Budget | Merge status | Reproduction |
|
||||
|---|---|---:|---|---|
|
||||
| PR, non-doc change | `Quick Checks` | 10 min | Required | `make pre-commit` (broader local umbrella) |
|
||||
| PR, non-doc change | `Test and Lint` | 90 min | Required | `cargo nextest run --profile ci --all --exclude e2e_test` |
|
||||
| PR, non-doc change | `Typos` | 10 min | Report-only | `typos` |
|
||||
| PR, non-doc change | `ILM Integration (serial)` | 90 min | Report-only | Use the exact command in `.github/workflows/ci.yml` |
|
||||
| PR, non-doc change | rio-v2 / swift / sftp test-and-lint variants | 90 min each | Report-only | `cargo nextest run` with the workflow's feature set |
|
||||
| PR, non-doc change | `Build RustFS Debug Binary` | 30 min | Report-only; prerequisite for black-box lanes | `cargo build -p rustfs --bins` |
|
||||
| PR, non-doc change | `io_uring Integration (real)` | 30 min | Report-only | `cargo test -p rustfs-ecstore --lib uring_ -- --test-threads=1 --nocapture` |
|
||||
| PR, non-doc change | `End-to-End Tests` (`e2e-smoke` plus `s3s-e2e`) | 30 min | Report-only | `cargo nextest run --profile e2e-smoke -p e2e_test`; then `./scripts/e2e-run.sh ./target/debug/rustfs <data-dir>` |
|
||||
| PR, non-doc change | `S3 Implemented Tests` | 60 min | Report-only | Build `rustfs`, then run `scripts/s3-tests/run.sh` with `DEPLOY_MODE=binary`, `TEST_MODE=single`, and `MAXFAIL=0` |
|
||||
| PR, non-doc change | `S3 Lifecycle Behavior Tests` | 30 min | Report-only | Use the accelerated scanner environment in `.github/workflows/ci.yml` with `scripts/s3-tests/run.sh` |
|
||||
| PR touching dependency or workflow inputs | Cargo Deny / Workflow Pin Report / Dependency Review | 20 / 5 / 30 min | Report-only | `cargo deny check`; `scripts/security/check_workflow_pins.sh` |
|
||||
| PR touching architecture rules or architecture docs | `Architecture Migration Rules` | 10 min | Report-only | `scripts/check_architecture_migration_rules.sh` |
|
||||
| PR touching Nix or workspace manifests | `Nix Build & Check` | 60 min | Report-only | `nix flake check` |
|
||||
| PR limited to main-CI-excluded paths | companion `Quick Checks` and `Test and Lint` | 10 min each | Required | `git diff --check`; `make doc-paths-check` when documentation paths changed |
|
||||
| `merge_group` | Standard CI plus `e2e-full` | 55 min for `e2e-full` | Standard required contexts only; `e2e-full` report-only | `cargo nextest run --profile e2e-full -p e2e_test` |
|
||||
| Push to `main` | Standard CI plus `e2e-full` | 55 min for `e2e-full` | Post-merge detection | Same as `merge_group` |
|
||||
| PR touching fuzz inputs or harness paths | Build plus five 60-second fuzz smoke targets | 60 min build; 30 min per target | Report-only | `MAX_TOTAL_TIME=60 ./scripts/fuzz/run.sh` |
|
||||
| PR touching selected ecstore disk/format paths | `Rename Safety` on Windows | 60 min | Report-only | Run the four `cargo test -p rustfs-ecstore --lib <filter>` commands in `windows-filesystem.yml` on Windows |
|
||||
| Event | Check name | Workflow / job | Merge status | Reproduce |
|
||||
|---|---|---|---|---|
|
||||
| PR, non-doc change | `Quick Checks` | `ci.yml` `quick-checks` | Required | `make pre-commit` |
|
||||
| PR, non-doc change | `Test and Lint` | `ci.yml` `test-and-lint` | Required | `cargo clippy --all-targets -- -D warnings`; `cargo nextest run --profile ci --all --exclude e2e_test`; `cargo test --all --doc`; `scripts/check_migration_gate_count.sh` |
|
||||
| PR, non-doc change | `Typos` | `ci.yml` `typos` | Report-only | `typos` |
|
||||
| PR, non-doc change | `ILM Integration (serial)` | `ci.yml` `test-ilm-integration-serial` | Report-only | exact command in the job |
|
||||
| PR, non-doc change | `Test and Lint (rio-v2)`, `Test and Lint (swift)`, `Test and Lint (sftp)` | `ci.yml` `test-and-lint-rio-v2`, `test-and-lint-protocols` | Report-only | `cargo nextest run` with the job's `--features` |
|
||||
| PR, non-doc change | `Connect Short Credential Boundary` | `ci.yml` `connect-short-credential-boundary` | Report-only | `cargo test -p rustfs --test connect_registration --features connect-e2e-short-credentials`; `cargo check -p rustfs --release --features connect-e2e-short-credentials` must fail |
|
||||
| PR, non-doc change | `Build RustFS Debug Binary` | `ci.yml` `build-rustfs-debug-binary` | Report-only; prerequisite for the black-box jobs | `cargo build -p rustfs --bins` |
|
||||
| PR, non-doc change | `io_uring Integration (real)` | `ci.yml` `uring-integration` | Report-only | `cargo test -p rustfs-ecstore --lib uring_ -- --test-threads=1 --nocapture` |
|
||||
| PR, non-doc change | `End-to-End Tests` | `ci.yml` `e2e-tests` | Report-only | `cargo nextest run --profile e2e-smoke -p e2e_test`, then `./scripts/e2e-run.sh ./target/debug/rustfs <data-dir>`; membership guards `scripts/check_test_wiring.py --check-profile e2e-smoke <listing.json>` and `scripts/check_security_smoke_count.sh check <listing.json>` |
|
||||
| PR, non-doc change | `S3 Implemented Tests` | `ci.yml` `s3-implemented-tests` | Report-only | build `rustfs`, then `scripts/s3-tests/run.sh` with the job's `DEPLOY_MODE` / `TEST_MODE` / `MAXFAIL` env |
|
||||
| PR, non-doc change | `S3 Lifecycle Behavior Tests` | `ci.yml` `s3-lifecycle-behavior-tests` | Report-only | `scripts/s3-tests/run.sh` with the job's accelerated-scanner env |
|
||||
| PR touching `paths` in `audit.yml` | `Cargo Deny`, `Workflow Pin Report`, `Dependency Review` | `audit.yml` `cargo-deny`, `workflow-pin-report`, `dependency-review` | Report-only | `cargo deny check`; `scripts/security/check_workflow_pins.sh` |
|
||||
| PR touching `paths` in `architecture-migration-rules.yml` | `Architecture Migration Rules` | `architecture-migration-rules.yml` `architecture-migration-rules` | Report-only | `scripts/check_architecture_migration_rules.sh` |
|
||||
| PR touching `paths` in `nix.yml` | `Nix Build & Check` | `nix.yml` `nix-validation` | Report-only | `nix flake check` |
|
||||
| PR touching `paths` in `fuzz.yml` | `Build Fuzz Harness`, `Smoke / <target>` | `fuzz.yml` `fuzz-build`, `pr-fuzz-smoke` | Report-only | `MAX_TOTAL_TIME=60 ./scripts/fuzz/run.sh` |
|
||||
| PR touching `paths` in `windows-filesystem.yml` | `Rename Safety` | `windows-filesystem.yml` `rename-safety` | Report-only | the `cargo test -p rustfs-ecstore --lib <filter>` commands in the job, on Windows |
|
||||
| PR touching `paths` in `coverage.yml` | `Workspace line coverage` | `coverage.yml` `coverage` | Report-only | `make coverage`; `python3 scripts/check_security_coverage.py target/llvm-cov/coverage.json` |
|
||||
| PR touching `paths` in `e2e-upgrade.yml` | `Direct upgrade from rc.2` | `e2e-upgrade.yml` `direct-upgrade` | Report-only | the `cargo test --locked -p e2e_test` command in the job with `RUSTFS_UPGRADE_SOURCE_BINARY` pointing at the pinned previous release |
|
||||
| PR touching `paths` in `oidc-keycloak.yml` | `OIDC Keycloak live gate` | `oidc-keycloak.yml` `oidc-keycloak-live` | Report-only | `cargo build --locked -p rustfs --bin rustfs`, then `bash scripts/test/oidc_keycloak_live.sh ./target/debug/rustfs` |
|
||||
| PR touching `paths` in `targets-integration.yml` | `PostgreSQL, MySQL, AMQP, and NATS` | `targets-integration.yml` `targets-live` | Report-only | start the containers as in the job, export the `RUSTFS_TEST_*` DSNs, then the job's `cargo test --locked -p rustfs-targets --test <name> -- --ignored --test-threads=1` commands |
|
||||
| PR limited to main-CI-excluded paths | `Quick Checks`, `Test and Lint` | `ci-docs-only.yml` `quick-checks`, `test-and-lint` | Required | `git diff --check`; `make doc-paths-check`; `scripts/check_no_planning_docs.sh` |
|
||||
| `merge_group`; push to `main` | `End-to-End Tests (full merge gate)` | `ci.yml` `e2e-full` | Report-only | `cargo nextest run --profile e2e-full -p e2e_test` |
|
||||
|
||||
The authoritative e2e filters live in `.config/nextest.toml`; extend a profile
|
||||
instead of adding a second ad-hoc selector. Before a profile runs,
|
||||
`scripts/check_test_wiring.py` compares its exact membership to the committed
|
||||
digest so a silent test drop fails closed.
|
||||
e2e filters live in `.config/nextest.toml`; extend a profile instead of adding a second selector. Before a profile runs, `scripts/check_test_wiring.py` compares its listing to the committed digest in `.config/e2e-<profile>-selection.txt`, so a silent test drop fails closed.
|
||||
|
||||
## Scheduled and manual validation
|
||||
## Scheduled validation
|
||||
|
||||
Scheduled lanes are independent fault domains. They do not block a pull
|
||||
request, but their workflow-local gate can fail the run and scheduled failures
|
||||
are routed to the shared failure-issue action. The scheduled-validation
|
||||
watchdog and freshness workflow separately detect incomplete runs and missing
|
||||
schedules.
|
||||
Scheduled lanes never block a PR. Their workflow-local gate fails the run, scheduled failures route to the shared failure-issue action, and `scheduled-validation-freshness.yml` fails when a workflow listed in `.github/scheduled-validations.json` has not run within its `max_age_hours` (a `never_ran_grace_until` entry covers the window before a newly enabled cron's first slot). Cadence is qualitative here; the cron lives in each workflow's `on.schedule`.
|
||||
|
||||
| Cadence (UTC unless noted) | Workflow / validation | Budget | Verdict and artifacts | Reproduction |
|
||||
|---|---|---:|---|---|
|
||||
| Daily 02:17 | Fuzz: five nightly corpus targets | 60 min build; 60 min per target | Gate; corpus/crash artifacts, scheduled failure alert | `MAX_TOTAL_TIME=<seconds> ./scripts/fuzz/run.sh` |
|
||||
| Dormant (cron 03:17 once re-enabled) | MinIO interop (EC + SSE read parity) | 40 min | Manually disabled in the Actions settings (backlog#1603) and therefore outside the freshness list; re-add it to `.github/scheduled-validations.json` when re-enabling | Follow the pinned Docker fixture steps in `minio-interop.yml` |
|
||||
| Daily 04:29 | Replication / cluster-fault / protocol e2e | 45 / 90 / 90 min | Three independent gates; JUnit, membership, and server logs | `cargo nextest run --profile e2e-repl-nightly -p e2e_test`; `--profile e2e-nightly`; `-j 1 --profile e2e-protocols` |
|
||||
| Daily 06:31 | Warp performance A/B | 180 min | Regression budget gate; A/B summaries and server logs | `bash scripts/run_hotpath_warp_abba.sh --help` |
|
||||
| Daily 00:07 Asia/Shanghai (16:07 UTC previous day) | Nightly GNU build and Vault lanes | 150 / 90 / 60 min | Build, live Vault, and HA failover gates | Use the commands and pinned Vault images in `nightly-gnu.yml` |
|
||||
| Daily 03:23 | Security Audit | 20 / 5 min, plus 30 min on PR dependency review | Cargo Deny and workflow-pin gates; scheduled failure alert | `cargo deny check`; `scripts/security/check_workflow_pins.sh` |
|
||||
| Daily 23:47 | Scheduled Validation Freshness | 10 min | Fails when a critical schedule was never created or is stale; an entry may carry `never_ran_grace_until` to cover the window before a newly enabled cron's first slot | Dispatch `scheduled-validation-freshness.yml` |
|
||||
| Sunday 00:11 | Full `Continuous Integration` matrix | Per-job budgets above | Weekly variant coverage, including dormant rio-v2 binary/e2e lanes | Dispatch `ci.yml` |
|
||||
| Sunday 01:13 | Seven-platform build matrix | 150 min per platform | Build/package integrity; scheduled failure alert | Dispatch `build.yml` with an exact platform set |
|
||||
| Sunday 02:19 | Ceph s3-tests full sweep: single and real four-node, four shards each | 180 min per shard | Compatibility gate; report, JUnit, exact node IDs, and server logs | `scripts/s3-tests/run.sh` against an existing single or distributed target |
|
||||
| Sunday 06:41 | Mint | 120 min | **Report-only by design**; per-suite PASS/FAIL/NA and raw `log.json` | Reproduce the pinned Docker sequence in `mint.yml` or dispatch it |
|
||||
| Sunday 07:43 | Workspace line coverage | 120 min | Report-only trend; lcov and JSON retained 90 days | `make coverage` |
|
||||
| Monthly, day 1 06:37 | Runner Hygiene | 15 min | Validates runner ephemerality; scheduled failure alert | Dispatch `runner-hygiene.yml` |
|
||||
| Workflow (cadence) | Jobs | Verdict and artifacts | In freshness list | Reproduce |
|
||||
|---|---|---|---|---|
|
||||
| `ci.yml` (weekly) | full matrix, including the schedule/dispatch-only rio-v2 jobs `build-rustfs-debug-binary-rio-v2` and `e2e-tests-rio-v2` | per-job | yes | dispatch `ci.yml` |
|
||||
| `build.yml` (weekly) | `build-rustfs` over the six-target platform matrix in `prepare-platform-matrix` (four Linux, macOS aarch64, Windows x86_64) | build/package integrity | yes | dispatch `build.yml` with an exact platform set |
|
||||
| `e2e-replication-nightly.yml` (nightly) | `repl-nightly`, `cluster-nightly`, `protocols-nightly` | three independent gates; JUnit, membership listing, server logs | yes | `cargo nextest run --profile e2e-repl-nightly -p e2e_test`; `--profile e2e-nightly`; `-j 1 --profile e2e-protocols` |
|
||||
| `e2e-s3tests.yml` (weekly) | `s3tests` (single and distributed, four shards each), `upstream-head-canary` | compatibility gate; report, JUnit, node IDs, server logs | yes | `scripts/s3-tests/run.sh` against an existing single or distributed target |
|
||||
| `fuzz.yml` (nightly) | `nightly-fuzz-corpus` per target | gate; corpus and crash artifacts | yes | `MAX_TOTAL_TIME=<seconds> ./scripts/fuzz/run.sh` |
|
||||
| `minio-interop.yml` (nightly) | `minio-interop` | EC + SSE read-parity gate | yes, with `never_ran_grace_until` | pinned Docker fixture steps in the workflow |
|
||||
| `performance-ab.yml` (nightly) | `warp-ab` | regression-budget gate; A/B summaries and server logs | yes | `bash scripts/run_hotpath_warp_abba.sh --help` |
|
||||
| `nightly-gnu.yml` (nightly) | `build`, `kms-vault-lane`, `kms-vault-ha-failover` | build, live Vault, and HA failover gates | yes | commands and pinned Vault images in the workflow |
|
||||
| `audit.yml` (nightly) | `cargo-deny`, `workflow-pin-report` | dependency and workflow-pin gates | yes | `cargo deny check`; `scripts/security/check_workflow_pins.sh` |
|
||||
| `mint.yml` (weekly) | `mint` | report-only by design; per-suite PASS/FAIL/NA and raw `log.json` | yes | pinned Docker sequence in the workflow |
|
||||
| `coverage.yml` (weekly) | `coverage` | report-only trend; lcov and JSON artifact | yes | `make coverage` |
|
||||
| `runner-hygiene.yml` (monthly) | `check-ephemerality` | runner ephemerality | yes | dispatch |
|
||||
| `e2e-upgrade.yml` (weekly) | `direct-upgrade` | upgrade gate; server logs | no | see the PR row |
|
||||
| `oidc-keycloak.yml` (weekly) | `oidc-keycloak-live` | live OIDC gate | no | see the PR row |
|
||||
| `targets-integration.yml` (nightly) | `targets-live` | live target gate; container logs | no | see the PR row |
|
||||
| `scheduled-validation-freshness.yml` (nightly) | `check-freshness` | fails on a never-created or stale schedule | n/a | dispatch |
|
||||
|
||||
Manual `workflow_dispatch` exists for the scheduled workflows above. Manual
|
||||
runs are debugging evidence and intentionally do not open scheduled-failure
|
||||
issues. A manual performance run may explicitly allow a known regression; that
|
||||
override must not be treated as an ordinary passing baseline.
|
||||
Manual `workflow_dispatch` runs are debugging evidence and do not open scheduled-failure issues. A manual performance run may explicitly allow a known regression; that override is not a passing baseline.
|
||||
|
||||
## Release validation
|
||||
|
||||
Release validation is post-merge and tag-driven; it does not substitute for a
|
||||
pull-request gate.
|
||||
Post-merge and tag-driven; not a substitute for a PR gate.
|
||||
|
||||
| Event | Validation | Budget | Result |
|
||||
|---|---|---:|---|
|
||||
| Push to `main` or weekly schedule | `Build and Release` platform matrix | 150 min per platform | Build artifacts for all selected targets; no release publication on a main push |
|
||||
| Valid release or preview tag | `Build and Release` plus asset checks | 150 min per platform | Draft release, checksummed assets, and publish step |
|
||||
| Successful non-preview release-tag build | Docker image build and image scan | 60 min build; 30 min scan | Multi-architecture images plus vulnerability report |
|
||||
| Successful release-tag build | DEB/RPM packaging | 30 min per architecture | Packages and checksum files uploaded to the release |
|
||||
| Successful non-preview release-tag build | Helm template test and package | 30 min build; 30 min publish | Versioned chart and repository index |
|
||||
| Trigger | Workflow / job | Result |
|
||||
|---|---|---|
|
||||
| Push to `main`, weekly schedule, dispatch | `build.yml` `build-rustfs` (a development build on a main push restricts the matrix to the Linux targets) | build artifacts; no release publication |
|
||||
| Valid release or preview tag | `build.yml` `build-rustfs`, `create-release`, `upload-release-assets`, `publish-release` | draft release, checksummed assets, publish |
|
||||
| Successful non-preview release-tag build (`workflow_run`) | `docker.yml` `build-docker`, `scan-docker-image` | multi-architecture images and vulnerability report |
|
||||
| Successful release-tag build (`workflow_run`) | `package.yml` `package` | DEB/RPM packages and checksums uploaded to the release |
|
||||
| Successful non-preview release-tag build (`workflow_run`) | `helm-package.yml` `build-helm-package`, `publish-helm-package` | versioned chart and repository index |
|
||||
| Final tag's release published | `build.yml` `cleanup-preview-releases` | deletes every `<target>-preview.<N>` Release for that target; the tags are kept |
|
||||
|
||||
Use an exact preview tag for end-to-end release rehearsal. Manual dispatches
|
||||
are backfill/debug paths and do not prove the automatic `workflow_run` chain.
|
||||
|
||||
A preview Release is internal validation state, not a deliverable: after the
|
||||
final tag's release is published, `cleanup-preview-releases` deletes every
|
||||
`<target>-preview.<N>` Release for that target. The tags themselves are kept, so
|
||||
the validated commit stays traceable.
|
||||
|
||||
## Evidence requirements
|
||||
|
||||
A green check is useful only when it proves the intended behavior ran:
|
||||
|
||||
- Record the exact commit SHA and run URL.
|
||||
- Separate product failure from runner prerequisites, service readiness, and
|
||||
cancellation. Repair the precondition, then rerun the exact workload.
|
||||
- Preserve membership manifests, JUnit, raw compatibility logs, seeds, and
|
||||
server logs where the workflow provides them.
|
||||
- For a bug fix or a new fault checker, provide sensitivity evidence: the old
|
||||
behavior or an intentional mutation must fail the new oracle, and the fixed
|
||||
behavior must pass it.
|
||||
- Never promote a report-only lane to required from one green run. Require at
|
||||
least 14 days and 30 representative pull requests with at least 99% complete
|
||||
execution, then update the ruleset and this table together.
|
||||
Use an exact preview tag for an end-to-end release rehearsal. Manual dispatches are backfill/debug paths and do not prove the automatic `workflow_run` chain.
|
||||
|
||||
## Change checklist
|
||||
|
||||
Update this file in the same pull request when any of these change:
|
||||
|
||||
- workflow triggers, job names, timeouts, or nextest profile ownership;
|
||||
- required status contexts or strict/merge-queue policy;
|
||||
- scheduled cadence, alert routing, artifact contract, or local reproduction;
|
||||
- report-only versus gating semantics.
|
||||
|
||||
Do not copy per-module test counts here. Update
|
||||
[e2e-suite-inventory.md](e2e-suite-inventory.md) and its enforced membership
|
||||
digest instead.
|
||||
Update this file in the same PR when a job or check name changes, a workflow gains or loses a `pull_request` or `schedule` trigger, required contexts or strict/merge-queue policy change, report-only vs gating semantics change, or `.github/scheduled-validations.json` membership changes. Do not copy timeouts, crons, or test counts here.
|
||||
|
||||
@@ -1,107 +0,0 @@
|
||||
# e2e_test suite inventory
|
||||
|
||||
> Authoritative per-module test counts for the `e2e_test` crate (backlog#1149
|
||||
> ci-4), generated from `cargo nextest list -p e2e_test`. Regenerate with:
|
||||
> ```bash
|
||||
> cargo nextest list -p e2e_test --message-format json | jq -r '.["rust-suites"][]?.testcases | to_entries[] | select(.value.ignored == false) | .key | split("::")[0]' | sort | uniq -c
|
||||
> ```
|
||||
> Modules marked ✅ are in the PR smoke profile `e2e-smoke`; 🌙 marks the
|
||||
> cluster, protocol, and replication subsets in the consolidated nightly
|
||||
> workflow. The `e2e-full` merge/main profile covers the remaining default
|
||||
> single-node tests. Committed test-ID digests are enforced before each run.
|
||||
> Note: counts exclude `#[ignore]`d tests (nextest lists them separately).
|
||||
> Managed-SSE (SSE-S3/SSE-KMS) replication contracts assert successful
|
||||
> re-encryption on the target (backlog#1783); SSE-C replication still pins a
|
||||
> fail-closed FAILED contract until ciphertext passthrough lands.
|
||||
|
||||
| module | tests | PR smoke |
|
||||
|---|---|---|
|
||||
| admin_auth_test | 4 | ✅ |
|
||||
| admin_iam_crud_test | 3 | ✅ |
|
||||
| admin_mfa_test | 4 | |
|
||||
| admin_pools_test | 1 | ✅ |
|
||||
| admin_timeout_regression_test | 1 | 🌙 |
|
||||
| anonymous_access_test | 4 | ✅ |
|
||||
| api_rate_limit_test | 3 | |
|
||||
| archive_download_integrity_test | 13 | |
|
||||
| bucket_logging_test | 3 | |
|
||||
| bucket_policy_check_test | 1 | ✅ |
|
||||
| bucket_stats_regression_test | 3 | |
|
||||
| chaos | 2 | |
|
||||
| checksum_upload_test | 7 | |
|
||||
| cluster_concurrency_test | 3 | 🌙 |
|
||||
| cluster_multidrive_pool_test | 4 | 🌙 |
|
||||
| common | 17 | |
|
||||
| compression_test | 6 | ✅ |
|
||||
| connection_cap_test | 2 | |
|
||||
| console_smoke_test | 1 | ✅ |
|
||||
| content_encoding_test | 3 | ✅ |
|
||||
| copy_object_checksum_test | 7 | |
|
||||
| copy_object_metadata_test | 4 | ✅ |
|
||||
| copy_object_tagging_test | 2 | ✅ |
|
||||
| copy_object_version_restore_test | 2 | |
|
||||
| copy_source_invalid_date_test | 1 | ✅ |
|
||||
| create_bucket_region_test | 2 | ✅ |
|
||||
| data_usage_test | 2 | |
|
||||
| degraded_listing_availability_test | 1 | 🌙 |
|
||||
| degraded_read_eof_regression_test | 3 | |
|
||||
| delete_marker_migration_semantics_test | 2 | ✅ |
|
||||
| delete_object_no_content_length_test | 1 | |
|
||||
| delete_objects_versioning_test | 2 | ✅ |
|
||||
| delete_regression_test | 5 | |
|
||||
| distributed_startup_regression_test | 3 | |
|
||||
| existing_object_tag_policy_test | 4 | |
|
||||
| fake_s3_target | 6 | ✅ |
|
||||
| fault_proxy | 7 | |
|
||||
| get_codec_streaming_compat_test | 1 | |
|
||||
| get_stream_failure_observability_test | 1 | |
|
||||
| group_delete_test | 4 | |
|
||||
| head_object_consistency_test | 1 | ✅ |
|
||||
| head_object_range_test | 1 | ✅ |
|
||||
| heal_erasure_disk_rebuild_test | 6 | 🌙 |
|
||||
| inline_fast_path_cluster_test | 16 | |
|
||||
| internode_rpc_signature_e2e_test | 5 | |
|
||||
| kms | 50 | |
|
||||
| leading_slash_key_test | 2 | ✅ |
|
||||
| lifecycle_regression_test | 4 | |
|
||||
| list_buckets_auth_test | 1 | ✅ |
|
||||
| list_buckets_double_slash_test | 3 | ✅ |
|
||||
| list_buckets_iam_filter_test | 1 | ✅ |
|
||||
| list_object_versions_metadata_extension_test | 1 | |
|
||||
| list_object_versions_regression_test | 2 | ✅ |
|
||||
| list_objects_duplicates_test | 3 | ✅ |
|
||||
| list_objects_v2_metadata_extension_test | 1 | |
|
||||
| list_objects_v2_pagination_test | 12 | ✅ |
|
||||
| listing_regression_test | 4 | |
|
||||
| mc_mirror_small_bucket_test | 1 | |
|
||||
| multipart_auth_test | 75 | |
|
||||
| multipart_storage_class_test | 3 | ✅ |
|
||||
| namespace_lock_quorum_test | 2 | 🌙 |
|
||||
| negative_sigv4_test | 7 | ✅ |
|
||||
| notification_startup_regression_test | 2 | |
|
||||
| notification_webhook_test | 3 | ✅ |
|
||||
| object_lambda_test | 16 | 🌙 |
|
||||
| object_lock | 34 | |
|
||||
| overwrite_cleanup_regression_test | 1 | |
|
||||
| policy | 6 | |
|
||||
| presigned_negative_test | 7 | ✅ |
|
||||
| protocols | 16 | 🌙 |
|
||||
| quota_test | 14 | |
|
||||
| reliability_disk_fault_test | 4 | |
|
||||
| reliant | 43 | 20 ✅ |
|
||||
| replication_extension_test | 76 | 20 ✅ +56 🌙 |
|
||||
| replication_lww_receiver_test | 1 | |
|
||||
| security_boundary_test | 4 | |
|
||||
| server_startup_failfast_test | 1 | |
|
||||
| snowball_auto_extract_test | 6 | |
|
||||
| special_chars_test | 14 | ✅ |
|
||||
| ssec_copy_test | 2 | ✅ |
|
||||
| stale_multipart_cleanup_cluster_test | 1 | 🌙 |
|
||||
| storage_class_capability_test | 4 | ✅ |
|
||||
| sts_query_compat_test | 6 | ✅ |
|
||||
| tier_transition_regression_test | 3 | |
|
||||
| tls_gen | 3 | |
|
||||
| tls_hot_reload_test | 1 | ✅ |
|
||||
| version_id_regression_test | 10 | ✅ |
|
||||
|
||||
**Total listed: 622 tests across 86 modules · PR smoke: 165 tests / 36 modules · merge/main full: 495 tests / 77 modules · nightly replication: 56 tests · nightly cluster faults: 32 tests / 7 modules · nightly protocols: 16 tests** · updated 2026-08-31.
|
||||
@@ -1,294 +1,62 @@
|
||||
# ECStore Validation Suite Design
|
||||
# ECStore Validation Suite
|
||||
|
||||
This document defines the validation suite RustFS should use before claiming
|
||||
ECStore erasure-coding correctness, durability, and fault-resilience coverage.
|
||||
It is a suite design, not a claim that all tests already exist.
|
||||
**Use this when:** you run or extend `scripts/run_ecstore_validation_suite.sh`, or add an erasure-coding test and need the scenario row it must satisfy.
|
||||
**Source of truth:** `scripts/run_ecstore_validation_suite.sh` (profiles, flags, step commands, coverage scope, default thresholds); `docs/architecture/erasure-coding.md` (the invariants the rows enforce).
|
||||
|
||||
## Goal
|
||||
|
||||
Provide one command that runs the full ECStore confidence suite and produces a
|
||||
single pass/fail result plus artifacts. The command should exercise both:
|
||||
|
||||
- white-box invariants in `rustfs-ecstore`, `rustfs-filemeta`, and related
|
||||
helpers;
|
||||
- black-box S3 and admin behavior against real single-node and distributed
|
||||
erasure deployments.
|
||||
|
||||
The suite can reduce release risk; it cannot prove the absence of all defects.
|
||||
It must therefore combine deterministic matrices, negative tests, fuzz/corpus
|
||||
tests, chaos tests, and explicit artifact review.
|
||||
|
||||
## Proposed Entry Point
|
||||
|
||||
Use the top-level runner for repeatable local, CI, and release checks:
|
||||
## Runner contract
|
||||
|
||||
```bash
|
||||
scripts/run_ecstore_validation_suite.sh --profile full
|
||||
scripts/run_ecstore_validation_suite.sh --profile <quick|full|destructive|fuzz> \
|
||||
[--out-dir <path>] [--skip-e2e] [--skip-s3-tests] [--skip-coverage] [--require-fixtures] \
|
||||
[--unit-coverage-min <percent>] [--unit-coverage-scope <ec-critical|crate>] [--dry-run]
|
||||
```
|
||||
|
||||
Profiles:
|
||||
The runner is a local and release-validation tool; no CI workflow invokes it.
|
||||
|
||||
| Profile | Purpose | Expected cost |
|
||||
| --- | --- | --- |
|
||||
| `quick` | PR smoke for EC logic and existing single-node reliability tests. | minutes |
|
||||
| `full` | Release gate: white-box, e2e, chaos, S3 compatibility subset, coverage. | hours |
|
||||
| `destructive` | Manual/nightly gate: distributed 4-node/16-disk, crash/restart, rebalance/decommission fault injection. | hours+ |
|
||||
| `fuzz` | Malformed metadata/RPC/corpus fuzzing with fixed seed output. | bounded by budget |
|
||||
| Profile | Steps (each profile includes the one above it) | Cost |
|
||||
|---|---|---|
|
||||
| `quick` | `rustfs-filemeta` lib tests; focused `rustfs-ecstore` lib filters (`erasure`, `set_disk::read`, `set_disk::core::io_primitives`, the rename-rollback test, `disk::local`); the two isolated global-state tests named at the top of the runner; the whole `rustfs-ecstore --lib` with `--test-threads=1`; e2e `reliability_disk_fault_test`, `heal_erasure_disk_rebuild_test`, `namespace_lock_quorum_test` | minutes |
|
||||
| `full` | fixture gate; legacy-bitrot and MinIO generated-read fixture tests; s3-tests subset `TESTEXPR="multipart or range or versioning or delete"` with `DEPLOY_MODE=build MAXFAIL=0`; unit coverage gate | hours |
|
||||
| `destructive` | `disk::local::test::crash_consistency` (power loss at each pre-commit step; object reopens as old or new, never mixed); e2e `cluster_concurrency_test`, `stale_multipart_cleanup_cluster_test`, `delete_marker_migration_semantics_test` | hours+ |
|
||||
| `fuzz` | `scripts/fuzz/run.sh` bounded by `MAX_TOTAL_TIME` (default in the runner) | bounded |
|
||||
|
||||
The current runner wires the existing high-signal checks. The matrix below
|
||||
still tracks required follow-up coverage before `full` can be treated as a
|
||||
complete release gate.
|
||||
Artifacts under `<out-dir>` (default `target/ecstore-validation/<timestamp>/`):
|
||||
|
||||
The runner writes artifacts under
|
||||
`target/ecstore-validation/<timestamp>/`:
|
||||
| File | Content |
|
||||
|---|---|
|
||||
| `run-metadata.env` | profile, flags, thresholds, environment |
|
||||
| `summary.tsv` | one row per step: pass / fail / skip with reason |
|
||||
| `blackbox-matrix.tsv` | every black-box and fixture row with command, fixture env, and status `enabled` / `disabled` / `missing-optional` / `missing-required`; written on every invocation regardless of profile |
|
||||
| `logs/*.log` | per-step transcripts |
|
||||
| `coverage/ecstore/lcov.info`, `summary.tsv`, `files.tsv` | coverage export, gate summary, per-file table (`full` and `destructive`) |
|
||||
|
||||
- command transcript and environment;
|
||||
- randomized seeds;
|
||||
- object manifests with SHA256;
|
||||
- per-disk layout snapshots before and after mutation;
|
||||
- server logs;
|
||||
- junit/nextest output;
|
||||
- `blackbox-matrix.tsv` with selected black-box and fixture gates;
|
||||
- coverage report;
|
||||
- failure reproducer instructions.
|
||||
Fixture rows:
|
||||
|
||||
## Acceptance Rules
|
||||
| Row | Profile | Fixture env |
|
||||
|---|---|---|
|
||||
| legacy bitrot read (`crates/ecstore/tests/legacy_bitrot_read_test.rs`) | `full` | `RUSTFS_LEGACY_TEST_ROOT`, `RUSTFS_LEGACY_TEST_DISK` |
|
||||
| MinIO generated encrypted read and negative restore (`storage::minio_generated_read_test` in `rustfs/src/storage/mod.rs`, `--features rio-v2`, `--ignored`) | `full` | `RUSTFS_MINIO_FIXTURE_ROOT`, `RUSTFS_MINIO_STATIC_KMS_KEY_B64` |
|
||||
|
||||
The suite passes only when all selected profile commands pass and every
|
||||
scenario asserts both API-visible behavior and on-disk state where applicable.
|
||||
A missing fixture is recorded as a `missing-optional` skip. `--require-fixtures` turns it into an early `ecstore-fixture-gate` failure before the expensive black-box steps run.
|
||||
|
||||
Required fail-closed rules:
|
||||
## Acceptance rules
|
||||
|
||||
A run passes only when every selected step passes. Every scenario asserts API-visible behaviour and, where applicable, on-disk state; a test that only checks constants, helper calls, deleted branches, or implementation details does not satisfy a row.
|
||||
|
||||
Fail-closed invariants every row enforces:
|
||||
|
||||
- never return corrupted object bytes;
|
||||
- never silently accept forged or split-brain metadata;
|
||||
- never downgrade write quorum to read quorum;
|
||||
- never leave a mixed old/new object after partial commit;
|
||||
- never leave a mixed old/new object after a partial commit;
|
||||
- never panic on malformed EC metadata;
|
||||
- return typed errors or quorum failures for invalid states.
|
||||
|
||||
Unit-test coverage is a hard release-gate input: `rustfs-ecstore` unit line
|
||||
coverage for the EC-critical scope must be at least 95%, with 100% as the
|
||||
target for EC read, write, decode, heal, metadata quorum, and rollback paths.
|
||||
A lower threshold is only acceptable for a temporary, explicitly documented
|
||||
exception tied to missing testability or unreachable code. Full-crate coverage
|
||||
is still reported as an observation metric, but it must not hide EC regressions
|
||||
behind unrelated modules.
|
||||
Fault injection is explicit and deterministic: local disk mocks for unit tests, process-level disk manipulation (`crates/e2e_test/src/chaos.rs`) for e2e tests. Property tests replay a fixed seed for payload, range, and missing-shard selection.
|
||||
|
||||
Tests must not pass by only checking constants, helper calls, deleted branches,
|
||||
or implementation details. Every test needs a reader-facing or storage-state
|
||||
assertion.
|
||||
### Coverage gate
|
||||
|
||||
Fixture-backed checks are optional for local smoke runs, but explicit in the
|
||||
artifact stream. `--require-fixtures` turns missing legacy or MinIO generated
|
||||
fixtures into an early `ecstore-fixture-gate` failure before expensive black-box
|
||||
steps run. The MinIO generated fixture gate requires both
|
||||
`RUSTFS_MINIO_FIXTURE_ROOT` and `RUSTFS_MINIO_STATIC_KMS_KEY_B64`.
|
||||
|
||||
## White-Box Matrix
|
||||
|
||||
### Erasure Algorithm
|
||||
|
||||
Target files:
|
||||
|
||||
- `crates/ecstore/src/erasure/coding/erasure.rs`
|
||||
- `crates/ecstore/src/erasure/coding/encode.rs`
|
||||
- `crates/ecstore/src/erasure/coding/decode.rs`
|
||||
- `crates/ecstore/src/erasure/coding/decode_reader.rs`
|
||||
- `crates/ecstore/src/erasure/codec/bridge.rs`
|
||||
|
||||
Coverage:
|
||||
|
||||
| Area | Scenarios | Assertions |
|
||||
| --- | --- | --- |
|
||||
| Shard geometry | legacy/current shard-size formulas; lengths `0`, `1`, `block-1`, `block`, `block+1`, multi-block tail | no divide-by-zero; shard/file/range offsets match expected |
|
||||
| Encode/decode | `(data, parity)` sets `2+2`, `4+2`, `8+8`; random payloads; missing shards up to parity | reconstructed data equals original |
|
||||
| Negative reconstruction | missing shards above parity; inconsistent shard lengths; corrupt surplus parity | typed error, no partial success |
|
||||
| Source verification | missing data shard plus extra parity source | rebuilt parity must match source parity |
|
||||
| Legacy compatibility | old shard formula and legacy checksum data | legacy files decode and heal correctly |
|
||||
| Streaming decode | legacy engine vs RustFS codec engine on same stripe stream | bytes and errors are equivalent |
|
||||
| Range output | head/middle/tail/suffix; cross-block and final-short-stripe ranges | exact byte range, no over-read/under-read |
|
||||
|
||||
Add property tests with fixed replay seeds for payload, range, and missing-shard
|
||||
selection.
|
||||
|
||||
### Bitrot and Reader Alignment
|
||||
|
||||
Target files:
|
||||
|
||||
- `crates/ecstore/src/erasure/coding/bitrot.rs`
|
||||
- `crates/ecstore/src/erasure/coding/decode.rs`
|
||||
- `crates/ecstore/src/set_disk/core/io_primitives.rs`
|
||||
- `crates/ecstore/src/set_disk/shard_source.rs`
|
||||
|
||||
Coverage:
|
||||
|
||||
| Area | Scenarios | Assertions |
|
||||
| --- | --- | --- |
|
||||
| Hash framing | valid hash+data; wrong hash; truncated hash; truncated data | invalid data never succeeds |
|
||||
| Short shard | short read under normal hash, `skip_verify`, and hash-none | `UnexpectedEof` or equivalent typed error |
|
||||
| Lockstep reads | mid-stream data shard failure; pending/timeout reader; final short stripe | each live reader advances exactly one stripe; failed reader retires |
|
||||
| Adaptive reads | hedged parity fallback and timeout retirement | no shard desync; reconstructed bytes match original |
|
||||
| Shard source order | out-of-order read completion and missing slots | slots resolve by shard index |
|
||||
| Deferred readers | data-blocks-first setup opens deferred parity at correct offset | parity fallback uses aligned data |
|
||||
|
||||
Use instrumented readers that record shard index, stripe index, read count,
|
||||
offset, and retirement reason.
|
||||
|
||||
### Metadata, Quorum, and Commit Atomicity
|
||||
|
||||
Target files:
|
||||
|
||||
- `crates/filemeta/src/filemeta/version.rs`
|
||||
- `crates/ecstore/src/set_disk/read.rs`
|
||||
- `crates/ecstore/src/set_disk/metadata.rs`
|
||||
- `crates/ecstore/src/set_disk/ops/object.rs`
|
||||
- `crates/ecstore/src/set_disk/core/io_primitives.rs`
|
||||
- `crates/ecstore/src/disk/local.rs`
|
||||
|
||||
Coverage:
|
||||
|
||||
| Area | Scenarios | Assertions |
|
||||
| --- | --- | --- |
|
||||
| Metadata tamper | same `version_id`/`mod_time`, divergent data dir, parts, ETag, size, checksum, inline flag, erasure distribution | previous committed version or read quorum error; no arbitrary latest |
|
||||
| Early stop | valid quorum, stale quorum, corrupt trailing disks, slow trailing disks | early-stop only on safe identity |
|
||||
| Quorum downgrade | read quorum vs write quorum; delete marker quorum; version-not-found quorum | no mutation below write quorum |
|
||||
| Rename atomicity | failure before data rename, after data rename, after metadata rename, cleanup failure | object is old or new; never mixed |
|
||||
| Rollback | failed commit quorum and stale temp data | rollback preserves old metadata/data |
|
||||
| Malformed metadata | oversized lengths, bad CRC, invalid versions, invalid UUID/timestamp/enum, huge parts | bounded memory; typed error; no panic |
|
||||
|
||||
Fault injection should be explicit and deterministic, preferably through local
|
||||
disk mocks for unit tests and process-level disk manipulation for e2e tests.
|
||||
|
||||
## Black-Box Matrix
|
||||
|
||||
Use `crates/e2e_test` for real S3/admin behavior and extend
|
||||
`crates/e2e_test/src/chaos.rs` rather than duplicating ad hoc helpers.
|
||||
|
||||
The current runner emits `blackbox-matrix.tsv` for every invocation. It is a
|
||||
machine-readable manifest of selected black-box scenarios, their commands,
|
||||
required fixture environment, and whether each row is enabled, disabled, or
|
||||
missing an optional/required fixture.
|
||||
|
||||
Current runner rows:
|
||||
|
||||
| Profile | Scenario | Gate | Fixture env |
|
||||
| --- | --- | --- | --- |
|
||||
| `quick` | single-node disk fault read/write | e2e black box | none |
|
||||
| `quick` | degraded erasure disk rebuild | e2e black box | none |
|
||||
| `quick` | namespace lock quorum under EC ops | e2e black box | none |
|
||||
| `full` | legacy bitrot read fixture restore | fixture black box | `RUSTFS_LEGACY_TEST_ROOT`, `RUSTFS_LEGACY_TEST_DISK` |
|
||||
| `full` | MinIO generated encrypted read and negative restore fixture | fixture black box | `RUSTFS_MINIO_FIXTURE_ROOT`, `RUSTFS_MINIO_STATIC_KMS_KEY_B64` |
|
||||
| `full` | S3 multipart/range/versioning/delete subset | S3 black box | none |
|
||||
| `destructive` | distributed cluster concurrency | e2e black box | none |
|
||||
| `destructive` | stale multipart cleanup cluster | e2e black box | none |
|
||||
| `destructive` | delete marker migration semantics | e2e black box | none |
|
||||
|
||||
### Single-Node 4-Disk EC
|
||||
|
||||
| Scenario | Required assertions |
|
||||
| --- | --- |
|
||||
| baseline PUT/GET/HEAD/List for tiny, inline, block-boundary, multi-block, multipart objects | SHA256 manifest matches; metadata is consistent on all disks |
|
||||
| one disk offline during read | existing objects readable; no corrupted bytes |
|
||||
| one disk offline during write | write succeeds only when write quorum holds; restored disk is healed |
|
||||
| above-parity disk loss | GET/PUT fails with quorum error; no partial bytes accepted |
|
||||
| corrupt data shard and parity shard | GET returns original bytes or fails closed; read-repair/heal restores |
|
||||
| corrupt inline `xl.meta` | fail closed or heal; no forged inline data |
|
||||
| range read with offline/corrupt shard | exact range bytes; invalid ranges produce expected S3 errors |
|
||||
| multipart part resend and concurrent same-part writes | final complete object matches chosen committed parts |
|
||||
| crash during multipart complete/put/delete | after restart only old or new full version is visible |
|
||||
|
||||
Existing anchors:
|
||||
|
||||
- `crates/e2e_test/src/reliability_disk_fault_test.rs`
|
||||
- `crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs`
|
||||
- `crates/e2e_test/src/chaos.rs`
|
||||
|
||||
### Distributed 4-Node / 16-Disk EC
|
||||
|
||||
| Scenario | Required assertions |
|
||||
| --- | --- |
|
||||
| node/disk outage while reading large objects | no EOF/truncation; SHA256 manifest matches |
|
||||
| write while one remote node is down | write follows quorum; later heal reconstructs remote disk |
|
||||
| remote shard bitrot | degraded read uses clean shards; no bad bytes |
|
||||
| concurrent GET/PUT/DELETE/List on same key | no 500 for expected conflicts; no dirty reads |
|
||||
| range GET matrix for large objects | sequential and parallel ranges match full-object hash |
|
||||
| internode timeout/slow disk | typed error or fallback; no desync |
|
||||
|
||||
This layer is mandatory because single-process unit tests cannot prove RPC,
|
||||
HTTP/2, timeout, and distributed quorum behavior.
|
||||
|
||||
### Versioning, Delete Markers, and Migration
|
||||
|
||||
| Scenario | Required assertions |
|
||||
| --- | --- |
|
||||
| latest delete marker | GET/HEAD/ListObjectVersions match S3 semantics |
|
||||
| explicit `versionId` for old versions | exact old bytes and metadata |
|
||||
| suspended/null version | no version ordering regression |
|
||||
| delete marker during heal/rebalance/decommission | marker visibility and history are preserved |
|
||||
| orphan directory cleanup | real objects are not purged; tombstones behave correctly |
|
||||
|
||||
### Heal, Rebalance, and Decommission
|
||||
|
||||
| Scenario | Required assertions |
|
||||
| --- | --- |
|
||||
| auto heal and admin deep heal | data hash unchanged; `xl.meta` and format data rebuilt |
|
||||
| heal interruption/restart | idempotent recovery; no dangling temp objects |
|
||||
| two-pool rebalance with versioned/multipart objects | source and target pools have consistent versions |
|
||||
| decommission cancel/restart/finalize | target readable; source cleanup safe |
|
||||
| rebalance/decommission with node outage | progress resumes; no duplicate or missing versions |
|
||||
|
||||
Existing scripts under `scripts/test/decommission_*.sh` should be wrapped into
|
||||
the `destructive` profile only after they emit machine-readable pass/fail
|
||||
artifacts.
|
||||
|
||||
## S3 Compatibility and Large Object Gates
|
||||
|
||||
The full profile should include a targeted S3 compatibility subset, not the
|
||||
entire compatibility suite by default:
|
||||
|
||||
```bash
|
||||
TESTEXPR="multipart or range or versioning or delete" \
|
||||
DEPLOY_MODE=build \
|
||||
MAXFAIL=0 \
|
||||
./scripts/s3-tests/run.sh
|
||||
```
|
||||
|
||||
Large object gates:
|
||||
|
||||
- `scripts/run_get_codec_streaming_smoke.sh` for legacy/codec GET parity;
|
||||
- `scripts/run_gt1g_get_http_matrix.sh` for sequential and parallel range GET;
|
||||
- `scripts/run_gt1g_multipart_put_matrix.sh` for multipart PUT paths.
|
||||
|
||||
These should be artifact-producing optional stages in `full` or `destructive`
|
||||
profiles, not hidden local-only commands.
|
||||
|
||||
## Fuzz and Corpus Gates
|
||||
|
||||
Add bounded fuzz/corpus tests for:
|
||||
|
||||
- `xl.meta` MessagePack and legacy filemeta versions;
|
||||
- protobuf/RPC payload decoding;
|
||||
- checksum and bitrot headers;
|
||||
- range offset/length overflow;
|
||||
- object names with path traversal, encoded separators, and symlink components;
|
||||
- huge inline metadata and huge part counts.
|
||||
|
||||
Each corpus failure must save the input bytes and the minimized reproducer under
|
||||
the suite artifact directory.
|
||||
|
||||
## Coverage Snapshot Target
|
||||
|
||||
The runner enforces unit line coverage for `rustfs-ecstore` in `full` and
|
||||
`destructive` profiles. The default threshold is 95%, and the target remains
|
||||
100% for EC-critical paths:
|
||||
|
||||
```bash
|
||||
scripts/run_ecstore_validation_suite.sh --profile full --unit-coverage-min 95
|
||||
scripts/run_ecstore_validation_suite.sh --profile full --unit-coverage-min 100
|
||||
scripts/run_ecstore_validation_suite.sh --profile full --unit-coverage-scope crate
|
||||
```
|
||||
|
||||
The default hard gate scope is `ec-critical`, covering:
|
||||
`full` and `destructive` run `cargo llvm-cov -p rustfs-ecstore --lib` and fail when line coverage of the gate scope is below `--unit-coverage-min`. The default minimum and the 100% target for EC read, write, decode, heal, metadata-quorum, and rollback paths are the `UNIT_COVERAGE_*` constants at the top of the runner. `cargo-llvm-cov` must be installed unless `--skip-coverage` is passed explicitly. The default scope `ec-critical` is:
|
||||
|
||||
- `crates/ecstore/src/erasure/**`
|
||||
- `crates/ecstore/src/set_disk/read.rs`
|
||||
@@ -298,79 +66,103 @@ The default hard gate scope is `ec-critical`, covering:
|
||||
- `crates/ecstore/src/set_disk/core/io_primitives.rs`
|
||||
- `crates/ecstore/src/disk/local.rs`
|
||||
|
||||
Minimum release-gate target:
|
||||
`--unit-coverage-scope crate` measures the whole crate instead; that number is an observation metric and must not hide EC regressions behind unrelated modules. Lowering the minimum requires a documented exception tied to missing testability or unreachable code. Uncovered branches in reconstruction, quorum, and error paths are either intentionally unreachable or tracked.
|
||||
|
||||
- `cargo-llvm-cov` must be installed unless `--skip-coverage` is explicitly set;
|
||||
- `rustfs-ecstore` EC-critical unit line coverage is at least 95%;
|
||||
- EC read/write/decode/heal/quorum/rollback code should trend toward 100%;
|
||||
- full-crate unit line coverage is recorded for visibility but is not the EC
|
||||
hard gate;
|
||||
- all HIGH rows in this document have positive and negative tests;
|
||||
- changed EC/read/write/heal lines are covered;
|
||||
- branch coverage is reviewed for reconstruction, quorum, and error paths;
|
||||
- uncovered branches are either intentionally unreachable or tracked.
|
||||
## White-box scenario matrix
|
||||
|
||||
The `quick` profile also runs
|
||||
`cargo test -p rustfs-ecstore --lib -- --test-threads=1` so the white-box smoke
|
||||
includes all current `rustfs-ecstore` library unit tests without parallel test
|
||||
context cross-talk, not only focused EC filters.
|
||||
Each row is a scenario the unit layer covers; a new EC unit test names the row it satisfies.
|
||||
|
||||
Coverage artifacts:
|
||||
### Erasure algorithm (`crates/ecstore/src/erasure/`)
|
||||
|
||||
- `target/ecstore-validation/<timestamp>/coverage/ecstore/lcov.info`
|
||||
- `target/ecstore-validation/<timestamp>/coverage/ecstore/summary.tsv`
|
||||
- `target/ecstore-validation/<timestamp>/coverage/ecstore/files.tsv`
|
||||
|
||||
Local validation on 2026-07-07 found that the current suite is not yet at the
|
||||
target: full-crate unit line coverage was 69.32%; the EC-critical scope was
|
||||
84.32% after the first negative read/write/recovery additions. The lowest
|
||||
EC-critical files were `set_disk/ops/object.rs`, `io_primitives.rs`,
|
||||
`disk/local.rs`, `bitrot.rs`, and `encode.rs`. These are gaps to close before
|
||||
the default 95% gate can pass.
|
||||
|
||||
## Initial Command Set
|
||||
|
||||
Use the runner first:
|
||||
|
||||
```bash
|
||||
scripts/run_ecstore_validation_suite.sh --profile quick
|
||||
scripts/run_ecstore_validation_suite.sh --profile full
|
||||
scripts/run_ecstore_validation_suite.sh --profile destructive
|
||||
scripts/run_ecstore_validation_suite.sh --profile fuzz
|
||||
```
|
||||
|
||||
The split-run equivalent for the quick profile is:
|
||||
|
||||
```bash
|
||||
cargo test -p rustfs-filemeta --lib
|
||||
cargo test -p rustfs-ecstore --lib erasure
|
||||
cargo test -p rustfs-ecstore --lib set_disk::read
|
||||
cargo test -p rustfs-ecstore --lib set_disk::core::io_primitives
|
||||
cargo test -p rustfs-ecstore --lib set_disk::tests::test_rename_data_quorum_failure_rolls_back_destination_object
|
||||
cargo test -p rustfs-ecstore --lib disk::local
|
||||
cargo test -p rustfs-ecstore --lib -- --test-threads=1
|
||||
cargo test --package e2e_test reliability_disk_fault_test -- --nocapture
|
||||
cargo test --package e2e_test heal_erasure_disk_rebuild_test -- --nocapture
|
||||
cargo test --package e2e_test namespace_lock_quorum_test -- --nocapture
|
||||
```
|
||||
|
||||
Fixture-backed tests should run when the fixture path is present:
|
||||
|
||||
```bash
|
||||
cargo test -p rustfs-ecstore --test legacy_bitrot_read_test -- --nocapture
|
||||
cargo test -p rustfs --features rio-v2 storage::minio_generated_read_test --lib -- --ignored --nocapture
|
||||
```
|
||||
|
||||
## Multi-Expert Adversarial Review Summary
|
||||
|
||||
Three independent read-only reviews were run for this design:
|
||||
|
||||
| Reviewer | Main challenge | Resulting requirement |
|
||||
| Area | Scenarios | Assertions |
|
||||
| --- | --- | --- |
|
||||
| Algorithm correctness | A single run only samples one timing/layout/hash combination. | matrix/property tests plus instrumented reader alignment checks |
|
||||
| Black-box reliability | Existing tests miss distributed, shard-desync, range+fault, and migration-fault combinations. | add 4-node/16-disk and destructive profiles |
|
||||
| Security and fault review | Metadata tamper, quorum downgrade, bitrot bypass, and rename atomicity must be end-to-end. | HIGH matrix rows block completion claims |
|
||||
| Shard geometry | legacy/current shard-size formulas; lengths `0`, `1`, `block-1`, `block`, `block+1`, multi-block tail | no divide-by-zero; shard/file/range offsets match expected |
|
||||
| Encode/decode | `(data, parity)` sets `2+2`, `4+2`, `8+8`; random payloads; missing shards up to parity | reconstructed data equals original |
|
||||
| Negative reconstruction | missing shards above parity; inconsistent shard lengths; corrupt surplus parity | typed error, no partial success |
|
||||
| Source verification | missing data shard plus extra parity source | rebuilt parity matches source parity |
|
||||
| Legacy compatibility | old shard formula and legacy checksum data | legacy files decode and heal correctly |
|
||||
| Streaming decode | legacy engine vs RustFS codec engine on the same stripe stream | bytes and errors are equivalent |
|
||||
| Range output | head/middle/tail/suffix; cross-block and final-short-stripe ranges | exact byte range, no over-read or under-read |
|
||||
|
||||
Current verdict: design accepted as a target. The runner exists and captures
|
||||
artifacts; implementation remains incomplete until the missing matrix rows have
|
||||
artifact-backed evidence.
|
||||
### Bitrot and reader alignment (`erasure/coding/bitrot.rs`, `decode.rs`, `set_disk/core/io_primitives.rs`, `set_disk/shard_source.rs`)
|
||||
|
||||
| Area | Scenarios | Assertions |
|
||||
| --- | --- | --- |
|
||||
| Hash framing | valid hash+data; wrong hash; truncated hash; truncated data | invalid data never succeeds |
|
||||
| Short shard | short read under normal hash, `skip_verify`, and hash-none | `UnexpectedEof` or equivalent typed error |
|
||||
| Lockstep reads | mid-stream data shard failure; pending/timeout reader; final short stripe | each live reader advances exactly one stripe; failed reader retires |
|
||||
| Adaptive reads | hedged parity fallback and timeout retirement | no shard desync; reconstructed bytes match original |
|
||||
| Shard source order | out-of-order read completion and missing slots | slots resolve by shard index |
|
||||
| Deferred readers | data-blocks-first setup opens deferred parity at the correct offset | parity fallback uses aligned data |
|
||||
|
||||
Instrumented readers record shard index, stripe index, read count, offset, and retirement reason.
|
||||
|
||||
### Metadata, quorum, and commit atomicity (`crates/filemeta/src/filemeta/version.rs`, `set_disk/read.rs`, `set_disk/metadata.rs`, `set_disk/ops/object.rs`, `disk/local.rs`)
|
||||
|
||||
| Area | Scenarios | Assertions |
|
||||
| --- | --- | --- |
|
||||
| Metadata tamper | same `version_id`/`mod_time`, divergent data dir, parts, ETag, size, checksum, inline flag, erasure distribution | previous committed version or read-quorum error; no arbitrary latest |
|
||||
| Early stop | valid quorum, stale quorum, corrupt trailing disks, slow trailing disks | early-stop only on safe identity |
|
||||
| Quorum downgrade | read quorum vs write quorum; delete-marker quorum; version-not-found quorum | no mutation below write quorum |
|
||||
| Rename atomicity | failure before data rename, after data rename, after metadata rename, cleanup failure | object is old or new; never mixed |
|
||||
| Rollback | failed commit quorum and stale temp data | rollback preserves old metadata and data |
|
||||
| Malformed metadata | oversized lengths, bad CRC, invalid versions, invalid UUID/timestamp/enum, huge parts | bounded memory; typed error; no panic |
|
||||
|
||||
## Black-box scenario matrix
|
||||
|
||||
Real S3 and admin behaviour runs through `crates/e2e_test`. Extend `crates/e2e_test/src/chaos.rs` rather than adding ad hoc fault helpers.
|
||||
|
||||
### Single-node 4-disk EC (`reliability_disk_fault_test.rs`, `heal_erasure_disk_rebuild_test.rs`, `chaos.rs`)
|
||||
|
||||
| Scenario | Required assertions |
|
||||
| --- | --- |
|
||||
| baseline PUT/GET/HEAD/List for tiny, inline, block-boundary, multi-block, multipart objects | SHA256 manifest matches; metadata is consistent on all disks |
|
||||
| one disk offline during read | existing objects readable; no corrupted bytes |
|
||||
| one disk offline during write | write succeeds only when write quorum holds; restored disk is healed |
|
||||
| above-parity disk loss | GET/PUT fails with a quorum error; no partial bytes accepted |
|
||||
| corrupt data shard and parity shard | GET returns original bytes or fails closed; read-repair/heal restores |
|
||||
| corrupt inline `xl.meta` | fail closed or heal; no forged inline data |
|
||||
| range read with offline/corrupt shard | exact range bytes; invalid ranges produce the expected S3 errors |
|
||||
| multipart part resend and concurrent same-part writes | final object matches the chosen committed parts |
|
||||
| crash during multipart complete/put/delete | after restart only the old or the new full version is visible |
|
||||
|
||||
### Distributed 4-node / 16-disk EC (`cluster_concurrency_test.rs`, `namespace_lock_quorum_test.rs`, `stale_multipart_cleanup_cluster_test.rs`)
|
||||
|
||||
Single-process unit tests cannot prove RPC, HTTP/2, timeout, or distributed-quorum behaviour, so this layer is mandatory.
|
||||
|
||||
| Scenario | Required assertions |
|
||||
| --- | --- |
|
||||
| node/disk outage while reading large objects | no EOF/truncation; SHA256 manifest matches |
|
||||
| write while one remote node is down | write follows quorum; later heal reconstructs the remote disk |
|
||||
| remote shard bitrot | degraded read uses clean shards; no bad bytes |
|
||||
| concurrent GET/PUT/DELETE/List on the same key | no 500 for expected conflicts; no dirty reads |
|
||||
| range GET matrix for large objects | sequential and parallel ranges match the full-object hash |
|
||||
| internode timeout / slow disk | typed error or fallback; no desync |
|
||||
|
||||
### Versioning, delete markers, and migration (`delete_marker_migration_semantics_test.rs`)
|
||||
|
||||
| Scenario | Required assertions |
|
||||
| --- | --- |
|
||||
| latest delete marker | GET/HEAD/ListObjectVersions match S3 semantics |
|
||||
| explicit `versionId` for old versions | exact old bytes and metadata |
|
||||
| suspended/null version | no version-ordering regression |
|
||||
| delete marker during heal/rebalance/decommission | marker visibility and history are preserved |
|
||||
| orphan directory cleanup | real objects are not purged; tombstones behave correctly |
|
||||
|
||||
### Heal, rebalance, and decommission
|
||||
|
||||
| Scenario | Required assertions |
|
||||
| --- | --- |
|
||||
| auto heal and admin deep heal | data hash unchanged; `xl.meta` and format data rebuilt |
|
||||
| heal interruption/restart | idempotent recovery; no dangling temp objects |
|
||||
| two-pool rebalance with versioned/multipart objects | source and target pools have consistent versions |
|
||||
| decommission cancel/restart/finalize | target readable; source cleanup safe |
|
||||
| rebalance/decommission with node outage | progress resumes; no duplicate or missing versions |
|
||||
|
||||
`scripts/test/decommission_*.sh` cover parts of this table but are not runner steps because they emit no machine-readable pass/fail artifact.
|
||||
|
||||
## Large-object and fuzz gates
|
||||
|
||||
Stand-alone harnesses indexed in `scripts/README.md`, not runner steps: `scripts/run_get_codec_streaming_smoke.sh` (legacy vs codec GET parity), `scripts/run_gt1g_get_http_matrix.sh` (sequential and parallel range GET above 1 GiB), `scripts/run_gt1g_multipart_put_matrix.sh` (multipart PUT above 1 GiB).
|
||||
|
||||
The `fuzz` profile delegates to `scripts/fuzz/run.sh`; targets, corpus rules, and crash-reproducer locations are in `fuzz/README.md`. Malformed-storage-input surfaces the corpus must cover: `xl.meta` MessagePack and legacy filemeta versions, RPC payload decoding, checksum and bitrot headers, range offset/length overflow, object names with path traversal or encoded separators, huge inline metadata and part counts.
|
||||
|
||||
@@ -1,58 +0,0 @@
|
||||
# Backlog #2007 Coalescer Delay Validation
|
||||
|
||||
`scripts/issue_2007_coalescer_prometheus_report.py` is a read-only Prometheus
|
||||
report helper for validating whether the GET metadata `ReadVersion` coalescer
|
||||
default can move from `200us` to `50us`.
|
||||
|
||||
The benchmark itself is intentionally external to this helper: use the same
|
||||
main build, bucket/object set, workload, and
|
||||
`RUSTFS_BATCH_READ_VERSION_SERVER_PARALLELISM=4` for both cells. Only switch:
|
||||
|
||||
```bash
|
||||
RUSTFS_GET_METADATA_READ_VERSION_COALESCE=auto
|
||||
RUSTFS_GET_METADATA_READ_VERSION_COALESCE_DELAY_MICROS=200
|
||||
RUSTFS_GET_METADATA_READ_VERSION_COALESCE_DELAY_MICROS=50
|
||||
```
|
||||
|
||||
After each measured workload window, collect a report from Prometheus:
|
||||
|
||||
```bash
|
||||
scripts/issue_2007_coalescer_prometheus_report.py \
|
||||
--query-url http://prometheus.example:9090 \
|
||||
--profile delay-200us \
|
||||
--window 180s \
|
||||
--rustfs-selector 'server=~"node[5-8]"' \
|
||||
--node-selector 'instance=~"node[5-8].*"'
|
||||
|
||||
scripts/issue_2007_coalescer_prometheus_report.py \
|
||||
--query-url http://prometheus.example:9090 \
|
||||
--profile delay-50us \
|
||||
--window 180s \
|
||||
--rustfs-selector 'server=~"node[5-8]"' \
|
||||
--node-selector 'instance=~"node[5-8].*"'
|
||||
```
|
||||
|
||||
The output is Markdown and is suitable for attaching to the issue alongside the
|
||||
warp throughput, average latency, p95, p99, and TTFB p99 from the fixed
|
||||
workload run.
|
||||
|
||||
Required RustFS signals:
|
||||
|
||||
- `grpc_read_version` and `grpc_batch_read_version` outgoing request increases.
|
||||
- Coalescer batch distribution from
|
||||
`rustfs_get_metadata_read_version_coalescer_total{event="attempted_batch"}`.
|
||||
- `batch_read_version_coalescer_wait`, `batch_read_version_rpc_roundtrip`,
|
||||
`batch_read_version_disk_read`, and `batch_read_version_response_map` p99.
|
||||
|
||||
Required host-cost signals:
|
||||
|
||||
- CPU busy from `node_cpu_seconds_total`.
|
||||
- Network RX/TX from `node_network_receive_bytes_total` and
|
||||
`node_network_transmit_bytes_total`.
|
||||
- Disk read await, average queue depth, and utilization from node-exporter disk
|
||||
counters.
|
||||
|
||||
If a section reports `UNAVAILABLE`, treat that evidence as missing rather than
|
||||
zero. Do not use a default-change PR until the `50us` cell has stable
|
||||
throughput/latency benefit and CPU, network, and disk cost are available and
|
||||
acceptable.
|
||||
@@ -1,44 +0,0 @@
|
||||
# Backlog #1649 Prometheus smoke
|
||||
|
||||
`scripts/prometheus_metrics_1649_smoke.py` is a read-only environment check for
|
||||
the metric dimensions delivered by backlog #1649 and issues #1650-#1653. It
|
||||
uses Prometheus' instant-query API and does not start, stop, reconfigure, or
|
||||
load RustFS nodes.
|
||||
|
||||
Run the parser and selector self-test without a live environment:
|
||||
|
||||
```bash
|
||||
python3 scripts/prometheus_metrics_1649_smoke.py --self-test
|
||||
```
|
||||
|
||||
For a live cluster, pass a Prometheus base URL (or its `/api/v1/query`
|
||||
endpoint), one or more expected server label values, and the built-in profile:
|
||||
|
||||
```bash
|
||||
python3 scripts/prometheus_metrics_1649_smoke.py \
|
||||
--query-url http://prometheus.example:9090 \
|
||||
--profile backlog-1649 \
|
||||
--server rustfs-node1 \
|
||||
--server rustfs-node2
|
||||
```
|
||||
|
||||
The profile checks the disk, scanner, ILM, audit, and notification series and
|
||||
their required labels. It also requires the legacy aggregate audit and
|
||||
notification series, so an additive label change cannot silently break
|
||||
existing dashboards.
|
||||
|
||||
Dynamic series retirement is checked with an exact label set after the
|
||||
scheduler retirement window has elapsed:
|
||||
|
||||
```bash
|
||||
python3 scripts/prometheus_metrics_1649_smoke.py \
|
||||
--query-url http://prometheus.example:9090 \
|
||||
--retired 'rustfs_scanner_bucket_drive_result_total|server=node1,bucket=removed,drive=d1,result=success' \
|
||||
--retired 'rustfs_audit_total_messages_by_server|server=node1,target_id=removed'
|
||||
```
|
||||
|
||||
`--require METRIC|key=value,...` requires a matching series;
|
||||
`--require-labels METRIC|key1,key2` requires every returned series to carry
|
||||
the named labels. Use `--bearer` for a bearer token or `--basic` for a
|
||||
`username:password` credential when Prometheus is protected. Do not put
|
||||
credentials in committed commands, logs, or issue comments.
|
||||
@@ -1,93 +1,41 @@
|
||||
# Security Advisory Regression Tests
|
||||
|
||||
Every fixed RustFS GitHub Security Advisory (GHSA) should map to at least one
|
||||
named, discoverable regression test. The convention is: name the test (or a
|
||||
helper / doc comment on the exact assertion) after the advisory so that
|
||||
**Use this when:** fixing or reviewing a GHSA, or checking that an advisory's guard actually executes in CI.
|
||||
**Source of truth:** the test files named below (`rg -i ghsa` finds them); `.config/nextest.toml` for lane membership; `.github/workflows/ci.yml` and `.github/workflows/e2e-replication-nightly.yml` for execution.
|
||||
|
||||
```bash
|
||||
rg -i "ghsa|3p3x|r5qv"
|
||||
```
|
||||
Every fixed RustFS GitHub Security Advisory maps to at least one named regression test. Name the test (or the helper / doc comment on the exact assertion) after the advisory so `rg -i "ghsa|<id>"` finds the guard, and a future fix of a still-open advisory is forced to update its pinned test (red -> green).
|
||||
|
||||
finds the guard for any advisory, and a future fix of a still-open advisory is
|
||||
forced to update its pinned test (red -> green).
|
||||
## Advisory -> test map
|
||||
|
||||
> This is the lightweight inventory. sec-14 (backlog#1151) formalizes the
|
||||
> written admission policy in `AGENTS.md`; keep this file as the map.
|
||||
|
||||
## Advisory -> test mapping
|
||||
|
||||
| Advisory | Class | Fix PR | Named regression tests | Layer |
|
||||
| Advisory | Class | Fix | Named regression tests | Layer |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| [GHSA-3p3x-734c-h5vx](https://github.com/rustfs/rustfs/security/advisories/GHSA-3p3x-734c-h5vx) | Constant-time secret comparison on WebDAV/FTPS password login | rustfs/rustfs#4403 | `assert_ftps_ghsa_3p3x_wrong_credentials_rejected` (`crates/e2e_test/src/protocols/ftps_core.rs`); `GHSA-3p3x` auth-failure block in `test_webdav_core_operations` (`crates/e2e_test/src/protocols/webdav_core.rs`) | e2e (protocols suite) |
|
||||
| [GHSA-3p3x-734c-h5vx](https://github.com/rustfs/rustfs/security/advisories/GHSA-3p3x-734c-h5vx) | Constant-time secret comparison on WebDAV/FTPS password login | rustfs/rustfs#4403 | `assert_ftps_ghsa_3p3x_wrong_credentials_rejected` (`crates/e2e_test/src/protocols/ftps_core.rs`); `GHSA-3p3x` auth-failure block in `test_webdav_core_operations` (`crates/e2e_test/src/protocols/webdav_core.rs`) | e2e (`e2e-protocols`) |
|
||||
| [GHSA-r5qv-rc46-hv8q](https://github.com/rustfs/rustfs/security/advisories/GHSA-r5qv-rc46-hv8q) | Internode RPC authentication must fail closed | rustfs/rustfs#4402 | `ghsa_r5qv_resolve_shared_secret_rejects_default_fallback`, `ghsa_r5qv_verify_rpc_signature_fails_closed_on_missing_or_invalid_auth` (`crates/ecstore/src/cluster/rpc/http_auth.rs`) | unit |
|
||||
| [GHSA-m77q-r63m-pj89](https://github.com/rustfs/rustfs/security/advisories/GHSA-m77q-r63m-pj89) | STS JWTs signed with shared root secret (intentionally unfixed) | n/a | `test_ghsa_m77q_sts_session_token_signed_with_root_secret` (flow-level pin: signing key == root secret, root-only decode, authorizes) and `test_created_sts_credentials_authorize_with_session_token_claims` (`crates/iam/src/sys.rs`); `token_signing_key` doc (`crates/iam/src/root_credentials.rs`) — pin current by-design behavior; fixing m77q must update red -> green | unit |
|
||||
| [GHSA-5354-r3w2-34m8](https://github.com/rustfs/rustfs/security/advisories/GHSA-5354-r3w2-34m8) | Service-account parent must stay within caller scope — a non-owner holding `CreateServiceAccountAdminAction` could parent a service account to the root credential and authenticate as owner | rustfs/rustfs#5141 | `ghsa_5354_non_owner_service_account_parent_confined_to_scope` and the `add_service_account_parent_within_scope` invariant it pins (`rustfs/src/admin/handlers/service_account.rs`) | unit |
|
||||
| [GHSA-3ppv-fx5m-m749](https://github.com/rustfs/rustfs/security/advisories/GHSA-3ppv-fx5m-m749) | Versioned object reads must be authorized against `s3:GetObjectVersion`, not `s3:GetObject` (`get_object`, CopyObject source, UploadPartCopy source) | rustfs/rustfs#5142 | `ghsa_3ppv_versioned_read_selects_get_object_version_action` and the `versioned_read_action` helper it pins (`rustfs/src/storage/access.rs`) | unit |
|
||||
| [GHSA-v9cp-qfw9-9pfp](https://github.com/rustfs/rustfs/security/advisories/GHSA-v9cp-qfw9-9pfp) | `ForAllValues:`/`ForAnyValue:` negated string operators were transposed — negation was applied to the aggregate quantified result instead of the per-value predicate | pending | `ghsa_v9cp_for_all_values_not_equals_partial_overlap`, `ghsa_v9cp_for_any_value_not_equals_partial_overlap` and the absent-key/positive-quantifier cases beside them (`crates/policy/tests/quantified_negation.rs`) — the value set must partially overlap the policy set, contained or disjoint sets cannot tell the two quantifiers apart | unit (crate test) |
|
||||
| [GHSA-6r96-hmgc-726c](https://github.com/rustfs/rustfs/security/advisories/GHSA-6r96-hmgc-726c) | Request headers must not populate server-derived IAM condition keys (`userid`, `groups`, `jwt:`/`ldap:` claim names) | pending | `ghsa_6r96_identity_condition_keys_ignore_spoofed_headers`, `ghsa_6r96_claim_condition_keys_ignore_spoofed_headers`, and `test_request_headers_still_reach_conditions` which guards the reserved set from growing too broad (`rustfs/src/auth.rs`) | unit |
|
||||
| [GHSA-x298-9x87-fvjq](https://github.com/rustfs/rustfs/security/advisories/GHSA-x298-9x87-fvjq) | The anonymous ListObjectVersions -> `s3:ListBucket` fallback must reach the same public-access gates as a direct grant | pending | `ghsa_x298_anonymous_list_object_versions_denied_when_restrict_public_buckets_enabled` (`crates/e2e_test/src/anonymous_access_test.rs`) — asserts 200 before the public-access block is applied so the test proves the gate rather than a broken fallback | e2e |
|
||||
| [GHSA-g3vq-vv42-f647](https://github.com/rustfs/rustfs/security/advisories/GHSA-g3vq-vv42-f647) | FTPS `MKD` must clear the `s3:CreateBucket` authorization boundary before reaching the backend | pending | `ghsa_g3vq_mkd_denied_before_reaching_backend` (`crates/protocols/src/ftps/driver.rs`) — primes `create_bucket` to succeed so the assertion distinguishes "denied at authorization" from "backend refused" | unit |
|
||||
| [GHSA-m77q-r63m-pj89](https://github.com/rustfs/rustfs/security/advisories/GHSA-m77q-r63m-pj89) | STS JWTs signed with the shared root secret (intentionally unfixed) | n/a; tests pin the by-design behaviour and must flip red -> green when m77q is fixed | `test_ghsa_m77q_sts_session_token_signed_with_root_secret`, `test_created_sts_credentials_authorize_with_session_token_claims` (`crates/iam/src/sys.rs`); `token_signing_key` doc (`crates/iam/src/root_credentials.rs`) | unit |
|
||||
| [GHSA-5354-r3w2-34m8](https://github.com/rustfs/rustfs/security/advisories/GHSA-5354-r3w2-34m8) | Service-account parent must stay within caller scope; a non-owner with `CreateServiceAccountAdminAction` could parent a service account to root | rustfs/rustfs#5141 | `ghsa_5354_non_owner_service_account_parent_confined_to_scope`, `ghsa_5354_scope_guard_matches_owner_or_self_scope_for_derived_credentials`, and the `add_service_account_parent_within_scope` invariant they pin (`rustfs/src/admin/handlers/service_account.rs`) | unit |
|
||||
| [GHSA-3ppv-fx5m-m749](https://github.com/rustfs/rustfs/security/advisories/GHSA-3ppv-fx5m-m749) | Versioned reads (`get_object`, CopyObject source, UploadPartCopy source) authorize against `s3:GetObjectVersion`, not `s3:GetObject` | rustfs/rustfs#5142 | `ghsa_3ppv_versioned_read_selects_get_object_version_action` and the `versioned_read_action` helper it pins (`rustfs/src/storage/access.rs`) | unit |
|
||||
| [GHSA-v9cp-qfw9-9pfp](https://github.com/rustfs/rustfs/security/advisories/GHSA-v9cp-qfw9-9pfp) | `ForAllValues:`/`ForAnyValue:` negated string operators applied negation to the aggregate instead of the per-value predicate | fixed, GHSA private-fork merge | `ghsa_v9cp_for_all_values_not_equals_partial_overlap`, `ghsa_v9cp_for_any_value_not_equals_partial_overlap` and the absent-key/positive-quantifier cases beside them (`crates/policy/tests/quantified_negation.rs`); the value set must partially overlap the policy set, since contained or disjoint sets cannot tell the quantifiers apart | crate test |
|
||||
| [GHSA-6r96-hmgc-726c](https://github.com/rustfs/rustfs/security/advisories/GHSA-6r96-hmgc-726c) | Request headers must not populate server-derived IAM condition keys (`userid`, `groups`, `jwt:`/`ldap:` claims) | fixed, GHSA private-fork merge | `ghsa_6r96_identity_condition_keys_ignore_spoofed_headers`, `ghsa_6r96_claim_condition_keys_ignore_spoofed_headers`, and `test_request_headers_still_reach_conditions`, which keeps the reserved set from growing too broad (`rustfs/src/auth.rs`) | unit |
|
||||
| [GHSA-x298-9x87-fvjq](https://github.com/rustfs/rustfs/security/advisories/GHSA-x298-9x87-fvjq) | Anonymous ListObjectVersions -> `s3:ListBucket` fallback must reach the same public-access gates as a direct grant | fixed, GHSA private-fork merge | `ghsa_x298_anonymous_list_object_versions_denied_when_restrict_public_buckets_enabled` (`crates/e2e_test/src/anonymous_access_test.rs`); asserts 200 before the public-access block is applied so it proves the gate, not a broken fallback | e2e (`e2e-smoke`) |
|
||||
| [GHSA-g3vq-vv42-f647](https://github.com/rustfs/rustfs/security/advisories/GHSA-g3vq-vv42-f647) | FTPS `MKD` must clear the `s3:CreateBucket` authorization boundary before reaching the backend | fixed, GHSA private-fork merge | `ghsa_g3vq_mkd_denied_before_reaching_backend` (`crates/protocols/src/ftps/driver.rs`); primes `create_bucket` to succeed so the assertion distinguishes "denied at authorization" from "backend refused" | unit (`ftps` feature) |
|
||||
|
||||
## Where these run (CI-execution map)
|
||||
## Where these run
|
||||
|
||||
Every security regression must land where CI actually runs it — a named test in
|
||||
an unexecuted suite is theater. The suites split across three execution paths by
|
||||
topology:
|
||||
| Layer | Command | Lane | Guard |
|
||||
| --- | --- | --- | --- |
|
||||
| Unit and crate tests (`ghsa_r5qv_*`, the m77q pins, `ghsa_5354_*`, `ghsa_3ppv_*`, `ghsa_6r96_*`, `ghsa_v9cp_*`, `ghsa_g3vq_*`) | `cargo nextest run --profile ci --all --exclude e2e_test` | every PR, `Test and Lint` (required) | none needed; the workspace pass runs every unit and crate test |
|
||||
| S3-API negative-auth e2e (`negative_sigv4_test`, `presigned_negative_test`, `admin_auth_test`) | `cargo nextest run --profile e2e-smoke -p e2e_test` | every PR, `End-to-End Tests` (report-only) | `scripts/check_security_smoke_count.sh` with the floor in `.config/security-smoke-floor.txt`, run in the `e2e-tests` job; fails when a rename drops one of these modules out of the smoke filter |
|
||||
| Other S3 e2e guards (`anonymous_access_test`) | `cargo nextest run --profile e2e-smoke -p e2e_test` | every PR, `End-to-End Tests` (report-only) | `scripts/check_test_wiring.py --check-profile e2e-smoke` digest |
|
||||
| Protocol e2e (`protocols::test_protocol_core_suite`, GHSA-3p3x) | `RUSTFS_BUILD_FEATURES=ftps,webdav,sftp cargo nextest run -j 1 --profile e2e-protocols -p e2e_test` | nightly, `e2e-replication-nightly.yml` job `protocols-nightly`; not PR-gated | `scripts/check_test_wiring.py --check-profile e2e-protocols` digest |
|
||||
|
||||
- **Unit tests** — `ghsa_r5qv_*` (`crates/ecstore`), the GHSA-m77q STS
|
||||
pinning (`crates/iam`), `ghsa_5354_*` / `ghsa_3ppv_*` / `ghsa_6r96_*`
|
||||
(`rustfs` lib), `ghsa_v9cp_*` (`crates/policy` crate test), and
|
||||
`ghsa_g3vq_*` (`crates/protocols`) run automatically in the default CI pass
|
||||
(`cargo nextest run --profile ci --all --exclude e2e_test`) — no special
|
||||
wiring. This is the CI-executed regression for the RPC fail-closed (r5qv),
|
||||
STS-signing (m77q), service-account parent-scope (5354), versioned-read
|
||||
authorization (3ppv), condition-key injection (6r96), quantified-negation
|
||||
(v9cp), and FTPS MKD authorization (g3vq) advisories.
|
||||
Notes:
|
||||
|
||||
`ghsa_g3vq_*` sits behind `rustfs-protocols`' `ftps` feature, which is off by
|
||||
default for that crate alone. It still runs in the workspace pass because the
|
||||
`rustfs` crate defaults to `["ftps", "webdav"]` and cargo unifies features
|
||||
across the build. Running `cargo test -p rustfs-protocols` on its own silently
|
||||
skips it — use `--features ftps` in that case. The `protocol-features` CI
|
||||
matrix covers `swift` and `sftp` only, for the same reason: those are not in
|
||||
any default feature set, so they need explicit jobs.
|
||||
- **S3-API negative-auth e2e (e2e-smoke, PR-gated)** — the attacker-facing S3
|
||||
auth-rejection suites run on every PR via the `e2e-smoke` nextest profile
|
||||
(`.config/nextest.toml`), which each spawns its own server on a random port
|
||||
and is parallel-safe:
|
||||
- `negative_sigv4_test` — tampered/wrong-key/skewed header SigV4 (sec-1)
|
||||
- `presigned_negative_test` — expired/tampered/wrong-key presigned URLs (sec-2)
|
||||
- `admin_auth_test` — non-admin denial + root-credential lifecycle (sec-4)
|
||||
|
||||
A count-floor guard (`scripts/check_security_smoke_count.sh`, floor in
|
||||
`.config/security-smoke-floor.txt`) runs in the `e2e-tests` CI job and fails if
|
||||
a rename drops any of these out of the smoke filter (infra-12 mechanism). This
|
||||
is the sec-5 wiring: those merged suites were dead weight until listed here.
|
||||
- **Protocol e2e (WebDAV/FTPS constant-time login, GHSA-3p3x)** — lives in the
|
||||
`e2e_test` protocols suite (`test_protocol_core_suite`), which binds **fixed
|
||||
ports**, needs the `ftps,webdav` build features, and is `#[serial]`. Those
|
||||
three properties make it **structurally incompatible** with the random-port,
|
||||
default-feature, parallel `e2e-smoke` profile: it cannot be a filterset change,
|
||||
so sec-5 (scoped to a filterset change, global ruling G5) does not wire it.
|
||||
It runs manually today:
|
||||
|
||||
```bash
|
||||
RUSTFS_BUILD_FEATURES=ftps,webdav cargo test --package e2e_test \
|
||||
test_protocol_core_suite -- --test-threads=1 --nocapture
|
||||
```
|
||||
|
||||
**Open gap:** GHSA-3p3x's *e2e* layer has no PR-gated CI execution. A protocols
|
||||
e2e CI lane is a ci-domain concern (a new profile/job, not a filterset change);
|
||||
it is deliberately out of sec-5's boundary and left as a follow-up.
|
||||
- `test_protocol_core_suite` is a single `#[tokio::test]` (not `#[serial]`) that binds fixed ports and needs the `ftps,webdav` build features; the nightly job serializes it with `-j 1`. It cannot join the random-port, default-feature `e2e-smoke` profile. Targeted local run: `crates/e2e_test/src/protocols/README.md`.
|
||||
- `ghsa_g3vq_*` sits behind the `ftps` feature of `rustfs-protocols`, which is off by default for that crate alone. It still runs in the workspace pass because the `rustfs` crate defaults to `["ftps", "webdav"]` and cargo unifies features across the build; `cargo test -p rustfs-protocols` on its own skips it, so pass `--features ftps`. The `protocol-features` matrix in `ci.yml` covers only `swift` and `sftp` for the same reason.
|
||||
|
||||
## Adding a new advisory guard
|
||||
|
||||
1. Reproduce the advisory's bypass form as a focused negative test.
|
||||
2. Name the test (or the helper/assertion) `ghsa_<id>_*`, or attach a
|
||||
`GHSA-<id>` doc comment with the advisory URL and fix PR.
|
||||
2. Name the test (or the helper/assertion) `ghsa_<id>_*`, or attach a `GHSA-<id>` doc comment with the advisory URL.
|
||||
3. Add a row to the table above.
|
||||
4. Land it where CI runs it (see the map above): a unit guard in the default CI
|
||||
pass; an S3-API negative-auth e2e in the `e2e-smoke` filter (add the module to
|
||||
`.config/nextest.toml` and bump `.config/security-smoke-floor.txt`); a
|
||||
fixed-port protocol e2e in the protocols suite (still manual — see the open
|
||||
gap above).
|
||||
4. Land it where it runs (table above): a unit guard needs nothing extra; an S3 e2e guard joins the `e2e-smoke` filter in `.config/nextest.toml` with its digest updated, and a negative-auth module also bumps `.config/security-smoke-floor.txt`; a fixed-port protocol guard goes into the protocols suite (nightly lane).
|
||||
|
||||
Reference in New Issue
Block a user