test(ci): strict nextest ci profile with quarantine + flake policy (#4666)

test(ci): add strict nextest ci profile with quarantine + flake policy

Formalize the existing ecstore-serial-flaky mechanism into a strict CI gate
(ci-10, absorbs infra-15; backlog#1149).

- .config/nextest.toml: add [profile.ci] with global retries=0 (never mask a
  new race's first occurrence), fail-fast=false, and JUnit output at
  target/nextest/ci/junit.xml. Add a quarantine section where flaky tests get
  retries=2 under the ci profile only; each entry links one OPEN issue. First
  members are the two backlog#937 ecstore groups
  (concurrent_resend_same_part_commits_one_generation and
  store::bucket::tests::bucket_delete_*), which keep their existing
  ecstore-serial-flaky test-group serialization. Local default profile still
  never retries.
- .github/workflows/ci.yml: run the main test step with --profile ci and upload
  the JUnit report (if: always(), 3-day retention, run-number in name). The
  migration-proof step stays on the default profile to avoid clobbering the ci
  JUnit artifact (its tests are not quarantined).
- docs/testing/README.md: new skeleton (owned by backlog#1153 infra-11) holding
  the flake policy: discover -> open issue within 24h -> quarantine with issue
  link -> fix or delete within 30 days. AGENTS.md points to it.

Refs: rustfs/backlog#1149, rustfs/backlog#937, rustfs/backlog#1155
This commit is contained in:
Zhengchao An
2026-07-10 20:08:31 +08:00
committed by GitHub
parent f0b1202b65
commit 89ea931ee1
5 changed files with 128 additions and 1 deletions
+54
View File
@@ -0,0 +1,54 @@
# RustFS Testing
> **Owner: backlog#1153 (infra-11).** This file is the authoritative home for
> test taxonomy, naming conventions, and serial/quarantine rules. It is
> currently a **skeleton** — infra-11 fills in the remaining sections. The
> event × budget × required matrix lives in `docs/testing/ci-gates.md`
> (ci-15); this file links to it rather than duplicating counts.
## Test taxonomy
_TODO (infra-11): unit / e2e-smoke / e2e-full / e2e-nightly / protocols /
s3-tests / fuzz / perf, with naming conventions._
## Serial groups & CI profiles
_TODO (infra-11): document the `[test-groups]` mechanism and the `default` vs
`ci` nextest profiles. See `.config/nextest.toml`._
## Flake policy
A flaky test is one that fails non-deterministically without a corresponding
code change. Flakes erode trust in the gate and block tightening required
checks, so they are handled on a strict, time-boxed loop.
**Retry semantics (source of truth: `.config/nextest.toml`):**
- The **local `default` profile never retries.** A red test on your machine is
a real failure to investigate, not noise to paper over.
- The **CI `ci` profile runs with global `retries = 0`.** A new race must fail
on its first occurrence so the first crime scene is never masked.
- Only tests on the **quarantine list** get `retries = 2`, and only under the
`ci` profile. Each quarantine entry MUST link exactly one OPEN issue.
- **JUnit flaky markers are the observable.** A quarantined test that passes
only after a retry is marked `flaky` in `target/nextest/ci/junit.xml`
(uploaded as a CI artifact). That marker — not a green check — is how we see
a flake is still live.
**Lifecycle of a flake:**
1. **Discover** — a test fails non-deterministically (CI or local), or shows a
`flaky` marker in the JUnit report.
2. **Open an issue within 24h** — file/track an issue describing the flake
(symptom, suspected cause, affected suite). No silent re-runs.
3. **Quarantine** — add the test to the quarantine override block in
`.config/nextest.toml` with a comment linking that OPEN issue. This grants
`retries = 2` under CI so the flake stops reddening unrelated PRs, while the
`flaky` marker keeps it visible.
4. **Fix or delete within 30 days** — make the test robust (then remove the
quarantine entry) or delete the test. A quarantine entry may not outlive its
fix window; an entry without a live OPEN issue link is a policy violation.
First quarantine members: the two backlog#937 ecstore groups
(`concurrent_resend_same_part_commits_one_generation` and
`store::bucket::tests::bucket_delete_*`).