ci(s3-tests): pin upstream suite, add weekly full sweep and compat report (#4204)

Reworks the S3 compatibility test harness for reproducibility and faster
feedback:

- Pin ceph/s3-tests to a fixed commit (S3TESTS_REV, fetch-by-SHA) so the
  449-test PR gate is reproducible; previously every run cloned upstream
  master, letting test renames or assertion changes break CI silently.
- PR gate (ci.yml s3-implemented-tests): MAXFAIL=0 + XDIST=4 so a single
  CI round reports every failure in parallel instead of stopping at the
  first one serially.
- Add TEST_SCOPE=all to run.sh to run the entire upstream suite, and
  report_compat.py to diff junit results against the classification
  lists (regressions, promotion candidates, unclassified tests).
- Rewrite e2e-s3tests.yml: delegate execution to run.sh (single source
  of truth; also fixes the broken config generation that left S3_PORT
  empty), add a weekly scheduled full sweep that fails only on whitelist
  regressions, and fix the multi-node topology to a real distributed
  cluster (endpoint-style RUSTFS_VOLUMES) instead of four independent
  single-node stores behind a load balancer.
- Docs: rewrite stale .github/s3tests/README.md (marker-era strategy),
  update scripts/s3-tests/README.md, fix dead build_testexpr.sh
  reference in S3_COMPAT_WORKFLOW.md, drop legacy non_standard_tests.txt.

All 747 classified test names verified present at the pinned revision;
13 upstream tests are currently unclassified and will surface in the
first full-sweep report.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Zhengchao An
2026-07-02 23:43:19 +08:00
committed by GitHub
parent 31c026dd4f
commit 3eeb459ece
8 changed files with 472 additions and 639 deletions
+3 -8
View File
@@ -21,19 +21,14 @@ git merge upstream/main
### Step 2: Select candidate tests
Pick ~20 tests from `unimplemented_tests.txt` and write them to `selected_tests.txt`:
```bash
TESTEXPR=$(scripts/s3-tests/build_testexpr.sh selected_tests.txt) \
DEPLOY_MODE=build MAXFAIL=0 ./scripts/s3-tests/run.sh
```
Or run them directly:
Pick candidate tests from `unimplemented_tests.txt` and run them via a pytest `-k` expression:
```bash
TESTEXPR="test_foo or test_bar" DEPLOY_MODE=build MAXFAIL=0 ./scripts/s3-tests/run.sh
```
Alternatively, check the latest weekly full-sweep report (the `e2e-s3tests` workflow's job summary, or `artifacts/s3tests-single/compat-report.md`) — its "promotion candidates" section lists tests that already pass and only need reclassification.
Review `artifacts/s3tests-single/pytest.log` for results.
### Step 3: Triage failures