ci(s3-tests): pin upstream suite, add weekly full sweep and compat report (#4204)

Reworks the S3 compatibility test harness for reproducibility and faster
feedback:

- Pin ceph/s3-tests to a fixed commit (S3TESTS_REV, fetch-by-SHA) so the
  449-test PR gate is reproducible; previously every run cloned upstream
  master, letting test renames or assertion changes break CI silently.
- PR gate (ci.yml s3-implemented-tests): MAXFAIL=0 + XDIST=4 so a single
  CI round reports every failure in parallel instead of stopping at the
  first one serially.
- Add TEST_SCOPE=all to run.sh to run the entire upstream suite, and
  report_compat.py to diff junit results against the classification
  lists (regressions, promotion candidates, unclassified tests).
- Rewrite e2e-s3tests.yml: delegate execution to run.sh (single source
  of truth; also fixes the broken config generation that left S3_PORT
  empty), add a weekly scheduled full sweep that fails only on whitelist
  regressions, and fix the multi-node topology to a real distributed
  cluster (endpoint-style RUSTFS_VOLUMES) instead of four independent
  single-node stores behind a load balancer.
- Docs: rewrite stale .github/s3tests/README.md (marker-era strategy),
  update scripts/s3-tests/README.md, fix dead build_testexpr.sh
  reference in S3_COMPAT_WORKFLOW.md, drop legacy non_standard_tests.txt.

All 747 classified test names verified present at the pinned revision;
13 upstream tests are currently unclassified and will surface in the
first full-sweep report.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

scripts/s3-tests/README.md

@@ -22,7 +22,7 @@ The script will automatically:
 4. **Prepare environment**:
    - Generate s3tests configuration from template
    - Provision alt user via admin API
-   - Clone s3-tests repository if missing
+   - Fetch s3-tests at the pinned revision (`S3TESTS_REV`)
    - Install missing dependencies (awscurl, tox, gettext)
 5. **Run tests**: Execute ceph s3-tests via tox with configured filters
 6. **Collect results**: Save logs and test results in `artifacts/s3tests-${TEST_MODE}/`
@@ -48,7 +48,7 @@ DEPLOY_MODE=build ./scripts/s3-tests/run.sh
 - Automatically compiles RustFS binary if it doesn't exist
 - If binary exists and was compiled less than **30 minutes** ago, compilation is skipped (unless `--no-cache` is specified)
 - Automatically starts the service after compilation
-- Automatically clones s3-tests repository if missing
+- Automatically fetches s3-tests at the pinned revision if missing
 - Automatically installs missing dependencies (awscurl, tox, gettext)
 - **Automatic Cleanup**: Process is automatically stopped when script exits
 
@@ -67,7 +67,7 @@ DEPLOY_MODE=binary RUSTFS_BINARY=./target/release/rustfs ./scripts/s3-tests/run.
 **Behavior**:
 - Uses existing binary file (must exist, script will not compile)
 - Automatically starts the service using the specified binary
-- Automatically clones s3-tests repository if missing
+- Automatically fetches s3-tests at the pinned revision if missing
 - Automatically installs missing dependencies (awscurl, tox, gettext)
 - **Automatic Cleanup**: Process is automatically stopped when script exits
 
@@ -82,7 +82,7 @@ DEPLOY_MODE=docker ./scripts/s3-tests/run.sh
 **Behavior**:
 - Automatically builds Docker image using `Dockerfile.source`
 - Creates Docker network (`rustfs-net`) if it doesn't exist
-- Automatically clones s3-tests repository if missing
+- Automatically fetches s3-tests at the pinned revision if missing
 - Automatically installs missing dependencies (awscurl, tox, gettext)
 - **Automatic Cleanup**: Container and network are automatically removed when script exits
 
@@ -100,7 +100,7 @@ DEPLOY_MODE=existing S3_HOST=192.168.1.100 S3_PORT=9000 ./scripts/s3-tests/run.s
 **Behavior**:
 - Skips service startup and port availability checks
 - Connects directly to the specified service endpoint
-- Automatically clones s3-tests repository if missing
+- Automatically fetches s3-tests at the pinned revision if missing
 - Automatically installs missing dependencies (awscurl, tox, gettext)
 - **Note**: The service must already have the alt user (`rustfsalt`) provisioned, or the script will provision it automatically
 
@@ -141,8 +141,15 @@ DEPLOY_MODE=existing S3_HOST=192.168.1.100 S3_PORT=9000 ./scripts/s3-tests/run.s
 ### Test Parameters
 
 - `TEST_MODE`: Test mode (default: `single`)
-- `MAXFAIL`: Stop after N failures (default: `1`)
+- `TEST_SCOPE`: Test scope (default: `implemented`)
+  - `implemented`: run only the `implemented_tests.txt` whitelist (the PR gate)
+  - `all`: run the entire upstream suite (used by the weekly full sweep)
+- `MAXFAIL`: Stop after N failures, `0` = never stop (default: `1`)
 - `XDIST`: Enable parallel execution with N workers (default: `0`, disabled)
+- `S3TESTS_REPO`: s3-tests repository URL (default: `https://github.com/ceph/s3-tests.git`)
+- `S3TESTS_REV`: Pinned s3-tests commit for reproducible runs
+  - Bump deliberately: upstream changes can rename tests or change assertions,
+    so a bump usually requires reclassifying the test list files
 - `MARKEXPR`: pytest marker expression for filtering tests
   - Default: no marker filtering; file-based test lists control the selected tests
   - Can be customized to test specific marker groups
@@ -246,6 +253,10 @@ Test results are saved in the `artifacts/s3tests-${TEST_MODE}/` directory (defau
 
 - `junit.xml`: Test results in JUnit format (compatible with CI/CD systems)
 - `pytest.log`: Detailed pytest logs with full test output
+- `compat-report.md`: Classification report generated by `report_compat.py` —
+  regressions against `implemented_tests.txt`, promotion candidates (tests
+  that pass but are still listed as unimplemented/excluded), and tests missing
+  from every list
 - `rustfs-${TEST_MODE}/rustfs.log`: RustFS service logs
 - `rustfs-${TEST_MODE}/inspect.json`: Service metadata (PID, binary path, mode, etc.)
 
@@ -316,8 +327,9 @@ The script will automatically install the following dependencies if missing (no
   - Linux: Automatically installs via `sudo apt-get install gettext-base`
   - **Note**: macOS installation may require manual intervention if brew fails
 
-- **s3-tests repository**: Automatically cloned if not present
-  - Source: `https://github.com/ceph/s3-tests.git`
+- **s3-tests repository**: Automatically fetched if not present
+  - Source: `https://github.com/ceph/s3-tests.git`, pinned to the commit in
+    `S3TESTS_REV` (see run.sh) for reproducible runs
   - Location: `${PROJECT_ROOT}/s3-tests`
 
 **Note**: The script adds `$HOME/.local/bin` to `PATH` automatically, so auto-installed Python tools are accessible.
@@ -427,29 +439,23 @@ awscurl --service s3 --region us-east-1 \
 
 ## Workflow Integration
 
-This script mirrors the GitHub Actions workflow defined in `.github/workflows/e2e-s3tests.yml`.
+This script is the single source of truth for running ceph s3-tests against
+RustFS. Two GitHub Actions workflows delegate to it:
 
-The script follows the same steps:
+- **PR gate** (`.github/workflows/ci.yml`, job `s3-implemented-tests`): runs
+  the `implemented_tests.txt` whitelist against a single-node debug binary on
+  every pull request (`DEPLOY_MODE=binary`, `MAXFAIL=0`, `XDIST=4`). Any
+  failure blocks the PR.
+- **Full sweep** (`.github/workflows/e2e-s3tests.yml`): weekly scheduled (and
+  manually dispatchable) run of the ENTIRE upstream suite (`TEST_SCOPE=all`)
+  against a Docker deployment — single node or a real 4-node distributed
+  cluster behind HAProxy. The sweep fails only on regressions in the
+  implemented whitelist; everything else is reported by `report_compat.py`
+  as promotion candidates or unclassified tests.
 
-1. Check port availability (skip for existing mode)
-2. Build/start RustFS service (varies by deployment mode)
-3. Wait for service to be fully ready:
-   - Check process/container status
-   - Check port is listening
-   - Wait for "server started successfully" log message
-   - Verify S3 API is responding
-4. Generate s3tests configuration from template
-5. Provision alt user for s3-tests via admin API
-6. Run ceph s3-tests with tox
-7. Collect logs and results
-
-### Key Improvements Over Workflow
-
-- **Smart compilation**: Skips rebuild if binary is recent (< 30 minutes)
-- **Better health checks**: Log-based readiness detection instead of blind waiting
-- **Port conflict detection**: Prevents conflicts before starting service
-- **Proxy handling**: Automatically disables proxy for localhost
-- **Configurable paths**: All paths (data, configs, artifacts) can be customized
+Keeping both workflows on this script means local runs, the PR gate, and the
+scheduled sweep always execute tests the same way (same pinned s3-tests
+revision, same config template, same user provisioning).
 
 ## See Also