Zhengchao An
5c7c757a30
Activate the 36 dormant replication e2e tests in crates/e2e_test/src/replication_extension_test.rs (zero ran anywhere before). Split via the ci-4 nextest profile mechanism, no hand-rolled cargo-test lane: - PR smoke (profile.e2e-smoke, existing e2e-tests job): the 20 fast bucket-replication tests (target-registration / replication-check / list / remove / delete admin paths) that validate config synchronously and never wait for async convergence. Each spawns its own single-node rustfs server(s) on random ports with isolated temp dirs, so parallel-safe by construction (serial_test's #[serial] is a no-op under nextest's process-per-test model; no test-group needed). - Nightly (profile.e2e-repl-nightly + .github/workflows/e2e-replication-nightly.yml): the remaining 16 = 6 slow data-plane tests + 9 _real_dual_node + 1 _real_single_node. Defined as 'replication module MINUS the PR allowlist' so new replication tests default to nightly and are never silently unrun. The nightly workflow builds the binary once, installs awscurl so the STS dual-node test runs (skips gracefully with a visible log line otherwise), and routes scheduled failures through .github/actions/schedule-failure-issue (ci-8). Explicit division of labor with ci-5 e2e-full: these run only here. Counts (cargo nextest list): e2e-smoke 83 (63 + 20), e2e-repl-nightly 16. Docs updated: e2e-suite-inventory.md, e2e_test/README.md. Refs backlog#1147 repl-1, backlog#1155.
RustFS Testing
Owner: backlog#1153 (infra-11). This file is the authoritative home for test taxonomy, naming conventions, and serial/quarantine rules. It is currently a skeleton — infra-11 fills in the remaining sections. The event × budget × required matrix lives in
docs/testing/ci-gates.md(ci-15); this file links to it rather than duplicating counts.
Test taxonomy
TODO (infra-11): unit / e2e-smoke / e2e-full / e2e-nightly / protocols / s3-tests / fuzz / perf, with naming conventions.
Serial groups & CI profiles
TODO (infra-11): document the [test-groups] mechanism and the default vs
ci nextest profiles. See .config/nextest.toml.
Flake policy
A flaky test is one that fails non-deterministically without a corresponding code change. Flakes erode trust in the gate and block tightening required checks, so they are handled on a strict, time-boxed loop.
Retry semantics (source of truth: .config/nextest.toml):
- The local
defaultprofile never retries. A red test on your machine is a real failure to investigate, not noise to paper over. - The CI
ciprofile runs with globalretries = 0. A new race must fail on its first occurrence so the first crime scene is never masked. - Only tests on the quarantine list get
retries = 2, and only under theciprofile. Each quarantine entry MUST link exactly one OPEN issue. - JUnit flaky markers are the observable. A quarantined test that passes
only after a retry is marked
flakyintarget/nextest/ci/junit.xml(uploaded as a CI artifact). That marker — not a green check — is how we see a flake is still live.
Lifecycle of a flake:
- Discover — a test fails non-deterministically (CI or local), or shows a
flakymarker in the JUnit report. - Open an issue within 24h — file/track an issue describing the flake (symptom, suspected cause, affected suite). No silent re-runs.
- Quarantine — add the test to the quarantine override block in
.config/nextest.tomlwith a comment linking that OPEN issue. This grantsretries = 2under CI so the flake stops reddening unrelated PRs, while theflakymarker keeps it visible. - Fix or delete within 30 days — make the test robust (then remove the quarantine entry) or delete the test. A quarantine entry may not outlive its fix window; an entry without a live OPEN issue link is a policy violation.
First quarantine members: the two backlog#937 ecstore groups
(concurrent_resend_same_part_commits_one_generation and
store::bucket::tests::bucket_delete_*).