mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-20 11:32:19 +00:00
ci(s3-tests): pin upstream suite, add weekly full sweep and compat report (#4204)
Reworks the S3 compatibility test harness for reproducibility and faster feedback: - Pin ceph/s3-tests to a fixed commit (S3TESTS_REV, fetch-by-SHA) so the 449-test PR gate is reproducible; previously every run cloned upstream master, letting test renames or assertion changes break CI silently. - PR gate (ci.yml s3-implemented-tests): MAXFAIL=0 + XDIST=4 so a single CI round reports every failure in parallel instead of stopping at the first one serially. - Add TEST_SCOPE=all to run.sh to run the entire upstream suite, and report_compat.py to diff junit results against the classification lists (regressions, promotion candidates, unclassified tests). - Rewrite e2e-s3tests.yml: delegate execution to run.sh (single source of truth; also fixes the broken config generation that left S3_PORT empty), add a weekly scheduled full sweep that fails only on whitelist regressions, and fix the multi-node topology to a real distributed cluster (endpoint-style RUSTFS_VOLUMES) instead of four independent single-node stores behind a load balancer. - Docs: rewrite stale .github/s3tests/README.md (marker-era strategy), update scripts/s3-tests/README.md, fix dead build_testexpr.sh reference in S3_COMPAT_WORKFLOW.md, drop legacy non_standard_tests.txt. All 747 classified test names verified present at the pinned revision; 13 upstream tests are currently unclassified and will surface in the first full-sweep report. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -21,19 +21,14 @@ git merge upstream/main
|
||||
|
||||
### Step 2: Select candidate tests
|
||||
|
||||
Pick ~20 tests from `unimplemented_tests.txt` and write them to `selected_tests.txt`:
|
||||
|
||||
```bash
|
||||
TESTEXPR=$(scripts/s3-tests/build_testexpr.sh selected_tests.txt) \
|
||||
DEPLOY_MODE=build MAXFAIL=0 ./scripts/s3-tests/run.sh
|
||||
```
|
||||
|
||||
Or run them directly:
|
||||
Pick candidate tests from `unimplemented_tests.txt` and run them via a pytest `-k` expression:
|
||||
|
||||
```bash
|
||||
TESTEXPR="test_foo or test_bar" DEPLOY_MODE=build MAXFAIL=0 ./scripts/s3-tests/run.sh
|
||||
```
|
||||
|
||||
Alternatively, check the latest weekly full-sweep report (the `e2e-s3tests` workflow's job summary, or `artifacts/s3tests-single/compat-report.md`) — its "promotion candidates" section lists tests that already pass and only need reclassification.
|
||||
|
||||
Review `artifacts/s3tests-single/pytest.log` for results.
|
||||
|
||||
### Step 3: Triage failures
|
||||
|
||||
Reference in New Issue
Block a user