mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-08 06:13:14 +00:00
test(ci): strict nextest ci profile with quarantine + flake policy (#4666)
test(ci): add strict nextest ci profile with quarantine + flake policy Formalize the existing ecstore-serial-flaky mechanism into a strict CI gate (ci-10, absorbs infra-15; backlog#1149). - .config/nextest.toml: add [profile.ci] with global retries=0 (never mask a new race's first occurrence), fail-fast=false, and JUnit output at target/nextest/ci/junit.xml. Add a quarantine section where flaky tests get retries=2 under the ci profile only; each entry links one OPEN issue. First members are the two backlog#937 ecstore groups (concurrent_resend_same_part_commits_one_generation and store::bucket::tests::bucket_delete_*), which keep their existing ecstore-serial-flaky test-group serialization. Local default profile still never retries. - .github/workflows/ci.yml: run the main test step with --profile ci and upload the JUnit report (if: always(), 3-day retention, run-number in name). The migration-proof step stays on the default profile to avoid clobbering the ci JUnit artifact (its tests are not quarantined). - docs/testing/README.md: new skeleton (owned by backlog#1153 infra-11) holding the flake policy: discover -> open issue within 24h -> quarantine with issue link -> fix or delete within 30 days. AGENTS.md points to it. Refs: rustfs/backlog#1149, rustfs/backlog#937, rustfs/backlog#1155
This commit is contained in:
@@ -0,0 +1,54 @@
|
||||
# RustFS Testing
|
||||
|
||||
> **Owner: backlog#1153 (infra-11).** This file is the authoritative home for
|
||||
> test taxonomy, naming conventions, and serial/quarantine rules. It is
|
||||
> currently a **skeleton** — infra-11 fills in the remaining sections. The
|
||||
> event × budget × required matrix lives in `docs/testing/ci-gates.md`
|
||||
> (ci-15); this file links to it rather than duplicating counts.
|
||||
|
||||
## Test taxonomy
|
||||
|
||||
_TODO (infra-11): unit / e2e-smoke / e2e-full / e2e-nightly / protocols /
|
||||
s3-tests / fuzz / perf, with naming conventions._
|
||||
|
||||
## Serial groups & CI profiles
|
||||
|
||||
_TODO (infra-11): document the `[test-groups]` mechanism and the `default` vs
|
||||
`ci` nextest profiles. See `.config/nextest.toml`._
|
||||
|
||||
## Flake policy
|
||||
|
||||
A flaky test is one that fails non-deterministically without a corresponding
|
||||
code change. Flakes erode trust in the gate and block tightening required
|
||||
checks, so they are handled on a strict, time-boxed loop.
|
||||
|
||||
**Retry semantics (source of truth: `.config/nextest.toml`):**
|
||||
|
||||
- The **local `default` profile never retries.** A red test on your machine is
|
||||
a real failure to investigate, not noise to paper over.
|
||||
- The **CI `ci` profile runs with global `retries = 0`.** A new race must fail
|
||||
on its first occurrence so the first crime scene is never masked.
|
||||
- Only tests on the **quarantine list** get `retries = 2`, and only under the
|
||||
`ci` profile. Each quarantine entry MUST link exactly one OPEN issue.
|
||||
- **JUnit flaky markers are the observable.** A quarantined test that passes
|
||||
only after a retry is marked `flaky` in `target/nextest/ci/junit.xml`
|
||||
(uploaded as a CI artifact). That marker — not a green check — is how we see
|
||||
a flake is still live.
|
||||
|
||||
**Lifecycle of a flake:**
|
||||
|
||||
1. **Discover** — a test fails non-deterministically (CI or local), or shows a
|
||||
`flaky` marker in the JUnit report.
|
||||
2. **Open an issue within 24h** — file/track an issue describing the flake
|
||||
(symptom, suspected cause, affected suite). No silent re-runs.
|
||||
3. **Quarantine** — add the test to the quarantine override block in
|
||||
`.config/nextest.toml` with a comment linking that OPEN issue. This grants
|
||||
`retries = 2` under CI so the flake stops reddening unrelated PRs, while the
|
||||
`flaky` marker keeps it visible.
|
||||
4. **Fix or delete within 30 days** — make the test robust (then remove the
|
||||
quarantine entry) or delete the test. A quarantine entry may not outlive its
|
||||
fix window; an entry without a live OPEN issue link is a policy violation.
|
||||
|
||||
First quarantine members: the two backlog#937 ecstore groups
|
||||
(`concurrent_resend_same_part_commits_one_generation` and
|
||||
`store::bucket::tests::bucket_delete_*`).
|
||||
Reference in New Issue
Block a user