mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-27 08:38:58 +00:00
Zhengchao An
89ea931ee1
test(ci): add strict nextest ci profile with quarantine + flake policy Formalize the existing ecstore-serial-flaky mechanism into a strict CI gate (ci-10, absorbs infra-15; backlog#1149). - .config/nextest.toml: add [profile.ci] with global retries=0 (never mask a new race's first occurrence), fail-fast=false, and JUnit output at target/nextest/ci/junit.xml. Add a quarantine section where flaky tests get retries=2 under the ci profile only; each entry links one OPEN issue. First members are the two backlog#937 ecstore groups (concurrent_resend_same_part_commits_one_generation and store::bucket::tests::bucket_delete_*), which keep their existing ecstore-serial-flaky test-group serialization. Local default profile still never retries. - .github/workflows/ci.yml: run the main test step with --profile ci and upload the JUnit report (if: always(), 3-day retention, run-number in name). The migration-proof step stays on the default profile to avoid clobbering the ci JUnit artifact (its tests are not quarantined). - docs/testing/README.md: new skeleton (owned by backlog#1153 infra-11) holding the flake policy: discover -> open issue within 24h -> quarantine with issue link -> fix or delete within 30 days. AGENTS.md points to it. Refs: rustfs/backlog#1149, rustfs/backlog#937, rustfs/backlog#1155
55 lines
2.6 KiB
Markdown
55 lines
2.6 KiB
Markdown
# RustFS Testing
|
||
|
||
> **Owner: backlog#1153 (infra-11).** This file is the authoritative home for
|
||
> test taxonomy, naming conventions, and serial/quarantine rules. It is
|
||
> currently a **skeleton** — infra-11 fills in the remaining sections. The
|
||
> event × budget × required matrix lives in `docs/testing/ci-gates.md`
|
||
> (ci-15); this file links to it rather than duplicating counts.
|
||
|
||
## Test taxonomy
|
||
|
||
_TODO (infra-11): unit / e2e-smoke / e2e-full / e2e-nightly / protocols /
|
||
s3-tests / fuzz / perf, with naming conventions._
|
||
|
||
## Serial groups & CI profiles
|
||
|
||
_TODO (infra-11): document the `[test-groups]` mechanism and the `default` vs
|
||
`ci` nextest profiles. See `.config/nextest.toml`._
|
||
|
||
## Flake policy
|
||
|
||
A flaky test is one that fails non-deterministically without a corresponding
|
||
code change. Flakes erode trust in the gate and block tightening required
|
||
checks, so they are handled on a strict, time-boxed loop.
|
||
|
||
**Retry semantics (source of truth: `.config/nextest.toml`):**
|
||
|
||
- The **local `default` profile never retries.** A red test on your machine is
|
||
a real failure to investigate, not noise to paper over.
|
||
- The **CI `ci` profile runs with global `retries = 0`.** A new race must fail
|
||
on its first occurrence so the first crime scene is never masked.
|
||
- Only tests on the **quarantine list** get `retries = 2`, and only under the
|
||
`ci` profile. Each quarantine entry MUST link exactly one OPEN issue.
|
||
- **JUnit flaky markers are the observable.** A quarantined test that passes
|
||
only after a retry is marked `flaky` in `target/nextest/ci/junit.xml`
|
||
(uploaded as a CI artifact). That marker — not a green check — is how we see
|
||
a flake is still live.
|
||
|
||
**Lifecycle of a flake:**
|
||
|
||
1. **Discover** — a test fails non-deterministically (CI or local), or shows a
|
||
`flaky` marker in the JUnit report.
|
||
2. **Open an issue within 24h** — file/track an issue describing the flake
|
||
(symptom, suspected cause, affected suite). No silent re-runs.
|
||
3. **Quarantine** — add the test to the quarantine override block in
|
||
`.config/nextest.toml` with a comment linking that OPEN issue. This grants
|
||
`retries = 2` under CI so the flake stops reddening unrelated PRs, while the
|
||
`flaky` marker keeps it visible.
|
||
4. **Fix or delete within 30 days** — make the test robust (then remove the
|
||
quarantine entry) or delete the test. A quarantine entry may not outlive its
|
||
fix window; an entry without a live OPEN issue link is a policy violation.
|
||
|
||
First quarantine members: the two backlog#937 ecstore groups
|
||
(`concurrent_resend_same_part_commits_one_generation` and
|
||
`store::bucket::tests::bucket_delete_*`).
|