ci(s3-tests): pin upstream suite, add weekly full sweep and compat report (#4204)

Reworks the S3 compatibility test harness for reproducibility and faster feedback: - Pin ceph/s3-tests to a fixed commit (S3TESTS_REV, fetch-by-SHA) so the 449-test PR gate is reproducible; previously every run cloned upstream master, letting test renames or assertion changes break CI silently. - PR gate (ci.yml s3-implemented-tests): MAXFAIL=0 + XDIST=4 so a single CI round reports every failure in parallel instead of stopping at the first one serially. - Add TEST_SCOPE=all to run.sh to run the entire upstream suite, and report_compat.py to diff junit results against the classification lists (regressions, promotion candidates, unclassified tests). - Rewrite e2e-s3tests.yml: delegate execution to run.sh (single source of truth; also fixes the broken config generation that left S3_PORT empty), add a weekly scheduled full sweep that fails only on whitelist regressions, and fix the multi-node topology to a real distributed cluster (endpoint-style RUSTFS_VOLUMES) instead of four independent single-node stores behind a load balancer. - Docs: rewrite stale .github/s3tests/README.md (marker-era strategy), update scripts/s3-tests/README.md, fix dead build_testexpr.sh reference in S3_COMPAT_WORKFLOW.md, drop legacy non_standard_tests.txt. All 747 classified test names verified present at the pinned revision; 13 upstream tests are currently unclassified and will surface in the first full-sweep report. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 16:28:15 +00:00 · 2026-07-02 23:43:19 +08:00
parent 31c026dd4f
commit 3eeb459ece
8 changed files with 472 additions and 639 deletions
@@ -22,7 +22,7 @@ The script will automatically:
 4. **Prepare environment**:
   - Generate s3tests configuration from template
   - Provision alt user via admin API
-   - Clone s3-tests repository if missing
+   - Fetch s3-tests at the pinned revision (`S3TESTS_REV`)
   - Install missing dependencies (awscurl, tox, gettext)
 5. **Run tests**: Execute ceph s3-tests via tox with configured filters
 6. **Collect results**: Save logs and test results in `artifacts/s3tests-${TEST_MODE}/`
@@ -48,7 +48,7 @@ DEPLOY_MODE=build ./scripts/s3-tests/run.sh
 - Automatically compiles RustFS binary if it doesn't exist
 - If binary exists and was compiled less than **30 minutes** ago, compilation is skipped (unless `--no-cache` is specified)
 - Automatically starts the service after compilation
- Automatically clones s3-tests repository if missing
+- Automatically fetches s3-tests at the pinned revision if missing
 - Automatically installs missing dependencies (awscurl, tox, gettext)
 - **Automatic Cleanup**: Process is automatically stopped when script exits

@@ -67,7 +67,7 @@ DEPLOY_MODE=binary RUSTFS_BINARY=./target/release/rustfs ./scripts/s3-tests/run.
 **Behavior**:
 - Uses existing binary file (must exist, script will not compile)
 - Automatically starts the service using the specified binary
- Automatically clones s3-tests repository if missing
+- Automatically fetches s3-tests at the pinned revision if missing
 - Automatically installs missing dependencies (awscurl, tox, gettext)
 - **Automatic Cleanup**: Process is automatically stopped when script exits

@@ -82,7 +82,7 @@ DEPLOY_MODE=docker ./scripts/s3-tests/run.sh
 **Behavior**:
 - Automatically builds Docker image using `Dockerfile.source`
 - Creates Docker network (`rustfs-net`) if it doesn't exist
- Automatically clones s3-tests repository if missing
+- Automatically fetches s3-tests at the pinned revision if missing
 - Automatically installs missing dependencies (awscurl, tox, gettext)
 - **Automatic Cleanup**: Container and network are automatically removed when script exits

@@ -100,7 +100,7 @@ DEPLOY_MODE=existing S3_HOST=192.168.1.100 S3_PORT=9000 ./scripts/s3-tests/run.s
 **Behavior**:
 - Skips service startup and port availability checks
 - Connects directly to the specified service endpoint
- Automatically clones s3-tests repository if missing
+- Automatically fetches s3-tests at the pinned revision if missing
 - Automatically installs missing dependencies (awscurl, tox, gettext)
 - **Note**: The service must already have the alt user (`rustfsalt`) provisioned, or the script will provision it automatically

@@ -141,8 +141,15 @@ DEPLOY_MODE=existing S3_HOST=192.168.1.100 S3_PORT=9000 ./scripts/s3-tests/run.s
 ### Test Parameters

 - `TEST_MODE`: Test mode (default: `single`)
- `MAXFAIL`: Stop after N failures (default: `1`)
+- `TEST_SCOPE`: Test scope (default: `implemented`)
+  - `implemented`: run only the `implemented_tests.txt` whitelist (the PR gate)
+  - `all`: run the entire upstream suite (used by the weekly full sweep)
+- `MAXFAIL`: Stop after N failures, `0` = never stop (default: `1`)
 - `XDIST`: Enable parallel execution with N workers (default: `0`, disabled)
+- `S3TESTS_REPO`: s3-tests repository URL (default: `https://github.com/ceph/s3-tests.git`)
+- `S3TESTS_REV`: Pinned s3-tests commit for reproducible runs
+  - Bump deliberately: upstream changes can rename tests or change assertions,
+    so a bump usually requires reclassifying the test list files
 - `MARKEXPR`: pytest marker expression for filtering tests
  - Default: no marker filtering; file-based test lists control the selected tests
  - Can be customized to test specific marker groups
@@ -246,6 +253,10 @@ Test results are saved in the `artifacts/s3tests-${TEST_MODE}/` directory (defau

 - `junit.xml`: Test results in JUnit format (compatible with CI/CD systems)
 - `pytest.log`: Detailed pytest logs with full test output
+- `compat-report.md`: Classification report generated by `report_compat.py` —
+  regressions against `implemented_tests.txt`, promotion candidates (tests
+  that pass but are still listed as unimplemented/excluded), and tests missing
+  from every list
 - `rustfs-${TEST_MODE}/rustfs.log`: RustFS service logs
 - `rustfs-${TEST_MODE}/inspect.json`: Service metadata (PID, binary path, mode, etc.)

@@ -316,8 +327,9 @@ The script will automatically install the following dependencies if missing (no
  - Linux: Automatically installs via `sudo apt-get install gettext-base`
  - **Note**: macOS installation may require manual intervention if brew fails

- **s3-tests repository**: Automatically cloned if not present
-  - Source: `https://github.com/ceph/s3-tests.git`
+- **s3-tests repository**: Automatically fetched if not present
+  - Source: `https://github.com/ceph/s3-tests.git`, pinned to the commit in
+    `S3TESTS_REV` (see run.sh) for reproducible runs
  - Location: `${PROJECT_ROOT}/s3-tests`

 **Note**: The script adds `$HOME/.local/bin` to `PATH` automatically, so auto-installed Python tools are accessible.
@@ -427,29 +439,23 @@ awscurl --service s3 --region us-east-1 \

 ## Workflow Integration

-This script mirrors the GitHub Actions workflow defined in `.github/workflows/e2e-s3tests.yml`.
+This script is the single source of truth for running ceph s3-tests against
+RustFS. Two GitHub Actions workflows delegate to it:

-The script follows the same steps:
+- **PR gate** (`.github/workflows/ci.yml`, job `s3-implemented-tests`): runs
+  the `implemented_tests.txt` whitelist against a single-node debug binary on
+  every pull request (`DEPLOY_MODE=binary`, `MAXFAIL=0`, `XDIST=4`). Any
+  failure blocks the PR.
+- **Full sweep** (`.github/workflows/e2e-s3tests.yml`): weekly scheduled (and
+  manually dispatchable) run of the ENTIRE upstream suite (`TEST_SCOPE=all`)
+  against a Docker deployment — single node or a real 4-node distributed
+  cluster behind HAProxy. The sweep fails only on regressions in the
+  implemented whitelist; everything else is reported by `report_compat.py`
+  as promotion candidates or unclassified tests.

-1. Check port availability (skip for existing mode)
-2. Build/start RustFS service (varies by deployment mode)
-3. Wait for service to be fully ready:
-   - Check process/container status
-   - Check port is listening
-   - Wait for "server started successfully" log message
-   - Verify S3 API is responding
-4. Generate s3tests configuration from template
-5. Provision alt user for s3-tests via admin API
-6. Run ceph s3-tests with tox
-7. Collect logs and results
-
-### Key Improvements Over Workflow
-
- **Smart compilation**: Skips rebuild if binary is recent (< 30 minutes)
- **Better health checks**: Log-based readiness detection instead of blind waiting
- **Port conflict detection**: Prevents conflicts before starting service
- **Proxy handling**: Automatically disables proxy for localhost
- **Configurable paths**: All paths (data, configs, artifacts) can be customized
+Keeping both workflows on this script means local runs, the PR gate, and the
+scheduled sweep always execute tests the same way (same pinned s3-tests
+revision, same config template, same user provisioning).

 ## See Also

@@ -21,19 +21,14 @@ git merge upstream/main

 ### Step 2: Select candidate tests

-Pick ~20 tests from `unimplemented_tests.txt` and write them to `selected_tests.txt`:
-
-```bash
-TESTEXPR=$(scripts/s3-tests/build_testexpr.sh selected_tests.txt) \
-  DEPLOY_MODE=build MAXFAIL=0 ./scripts/s3-tests/run.sh
-```
-
-Or run them directly:
+Pick candidate tests from `unimplemented_tests.txt` and run them via a pytest `-k` expression:

 ```bash
 TESTEXPR="test_foo or test_bar" DEPLOY_MODE=build MAXFAIL=0 ./scripts/s3-tests/run.sh
 ```

+Alternatively, check the latest weekly full-sweep report (the `e2e-s3tests` workflow's job summary, or `artifacts/s3tests-single/compat-report.md`) — its "promotion candidates" section lists tests that already pass and only need reclassification.
+
 Review `artifacts/s3tests-single/pytest.log` for results.

 ### Step 3: Triage failures
@@ -1,236 +0,0 @@
-# Legacy non-standard S3 test list (compatibility fallback)
-# =========================================================
-#
-# This file is kept for backward compatibility.
-# Active exclusion classification now uses excluded_tests.txt.
-# These tests use vendor-specific extensions not part of AWS S3 API.
-#
-# Exclusion reasons:
-# - fails_on_aws marker: Ceph-specific features
-# - X-RGW-* headers: Ceph proprietary headers
-# - allowUnordered: Ceph-specific query parameter
-# - Bucket Logging: Ceph-specific logging extensions
-# - Lifecycle: Ceph-specific lifecycle expiration behavior
-# - SSE-KMS: Ceph-specific KMS extensions
-# - Error format differences: Minor response format variations
-
-test_account_usage
-test_atomic_conditional_write_1mb
-test_bucket_get_location
-test_bucket_head_extended
-test_bucket_header_acl_grants
-test_bucket_list_return_data
-test_bucket_list_return_data_versioning
-test_bucket_logging_bucket_acl_required
-test_bucket_logging_bucket_auth_type
-test_bucket_logging_cleanup_bucket_concurrent_deletion_j
-test_bucket_logging_cleanup_bucket_concurrent_deletion_j_single
-test_bucket_logging_cleanup_bucket_concurrent_deletion_s
-test_bucket_logging_cleanup_bucket_concurrent_deletion_s_single
-test_bucket_logging_cleanup_bucket_deletion_j
-test_bucket_logging_cleanup_bucket_deletion_j_single
-test_bucket_logging_cleanup_bucket_deletion_s
-test_bucket_logging_cleanup_bucket_deletion_s_single
-test_bucket_logging_cleanup_concurrent_disabling_j
-test_bucket_logging_cleanup_concurrent_disabling_j_single
-test_bucket_logging_cleanup_concurrent_disabling_s
-test_bucket_logging_cleanup_concurrent_disabling_s_single
-test_bucket_logging_cleanup_concurrent_updating_j
-test_bucket_logging_cleanup_concurrent_updating_j_single
-test_bucket_logging_cleanup_concurrent_updating_s
-test_bucket_logging_cleanup_concurrent_updating_s_single
-test_bucket_logging_cleanup_disabling_j
-test_bucket_logging_cleanup_disabling_j_single
-test_bucket_logging_cleanup_disabling_s
-test_bucket_logging_cleanup_disabling_s_single
-test_bucket_logging_cleanup_updating_j
-test_bucket_logging_cleanup_updating_j_single
-test_bucket_logging_cleanup_updating_s
-test_bucket_logging_cleanup_updating_s_single
-test_bucket_logging_concurrent_flush_j
-test_bucket_logging_concurrent_flush_j_single
-test_bucket_logging_concurrent_flush_s
-test_bucket_logging_concurrent_flush_s_single
-test_bucket_logging_conf_concurrent_updating_pfx_j
-test_bucket_logging_conf_concurrent_updating_pfx_s
-test_bucket_logging_conf_concurrent_updating_roll_j
-test_bucket_logging_conf_concurrent_updating_roll_s
-test_bucket_logging_conf_updating_pfx_j
-test_bucket_logging_conf_updating_pfx_s
-test_bucket_logging_conf_updating_roll_j
-test_bucket_logging_conf_updating_roll_s
-test_bucket_logging_copy_objects
-test_bucket_logging_copy_objects_bucket
-test_bucket_logging_copy_objects_bucket_versioned
-test_bucket_logging_copy_objects_versioned
-test_bucket_logging_delete_objects
-test_bucket_logging_delete_objects_versioned
-test_bucket_logging_event_type_j
-test_bucket_logging_event_type_s
-test_bucket_logging_flush_empty
-test_bucket_logging_flush_j
-test_bucket_logging_flush_j_single
-test_bucket_logging_flush_s
-test_bucket_logging_flush_s_single
-test_bucket_logging_get_objects
-test_bucket_logging_get_objects_versioned
-test_bucket_logging_head_objects
-test_bucket_logging_head_objects_versioned
-test_bucket_logging_key_filter_j
-test_bucket_logging_key_filter_s
-test_bucket_logging_mpu_copy
-test_bucket_logging_mpu_copy_versioned
-test_bucket_logging_mpu_j
-test_bucket_logging_mpu_s
-test_bucket_logging_mpu_versioned_j
-test_bucket_logging_mpu_versioned_s
-test_bucket_logging_mtime
-test_bucket_logging_multi_delete
-test_bucket_logging_multi_delete_versioned
-test_bucket_logging_multiple_prefixes
-test_bucket_logging_notupdating_j
-test_bucket_logging_notupdating_j_single
-test_bucket_logging_notupdating_s
-test_bucket_logging_notupdating_s_single
-test_bucket_logging_object_acl_required
-test_bucket_logging_object_meta
-test_bucket_logging_part_cleanup_concurrent_deletion_j
-test_bucket_logging_part_cleanup_concurrent_deletion_s
-test_bucket_logging_part_cleanup_concurrent_disabling_j
-test_bucket_logging_part_cleanup_concurrent_disabling_s
-test_bucket_logging_part_cleanup_concurrent_updating_j
-test_bucket_logging_part_cleanup_concurrent_updating_s
-test_bucket_logging_part_cleanup_deletion_j
-test_bucket_logging_part_cleanup_deletion_s
-test_bucket_logging_part_cleanup_disabling_j
-test_bucket_logging_part_cleanup_disabling_s
-test_bucket_logging_part_cleanup_updating_j
-test_bucket_logging_part_cleanup_updating_s
-test_bucket_logging_partitioned_key
-test_bucket_logging_permission_change_j
-test_bucket_logging_permission_change_s
-test_bucket_logging_put_and_flush
-test_bucket_logging_put_concurrency
-test_bucket_logging_put_objects
-test_bucket_logging_put_objects_versioned
-test_bucket_logging_roll_time
-test_bucket_logging_simple_key
-test_bucket_logging_single_prefix
-test_bucket_logging_target_cleanup_j
-test_bucket_logging_target_cleanup_j_single
-test_bucket_logging_target_cleanup_s
-test_bucket_logging_target_cleanup_s_single
-test_bucket_policy_get_obj_acl_existing_tag
-test_bucket_policy_get_obj_existing_tag
-test_bucket_policy_get_obj_tagging_existing_tag
-test_bucket_policy_put_obj_copy_source
-test_bucket_policy_put_obj_copy_source_meta
-test_bucket_policy_put_obj_request_obj_tag
-test_bucket_policy_put_obj_tagging_existing_tag
-test_bucket_policy_set_condition_operator_end_with_IfExists
-test_bucket_policy_upload_part_copy
-test_bucket_recreate_new_acl
-test_bucket_recreate_overwrite_acl
-test_cors_presigned_get_object_tenant_v2
-test_cors_presigned_get_object_v2
-test_cors_presigned_put_object_tenant_v2
-test_cors_presigned_put_object_v2
-test_create_bucket_bucket_owner_enforced
-test_create_bucket_bucket_owner_preferred
-test_create_bucket_object_writer
-test_delete_marker_expiration
-test_delete_marker_nonversioned
-test_delete_marker_suspended
-test_delete_marker_versioned
-test_delete_object_current_if_match
-test_delete_object_current_if_match_last_modified_time
-test_delete_object_current_if_match_size
-test_delete_object_if_match
-test_delete_object_if_match_last_modified_time
-test_delete_object_if_match_size
-test_delete_object_version_if_match
-test_delete_object_version_if_match_last_modified_time
-test_delete_object_version_if_match_size
-test_delete_objects_current_if_match
-test_delete_objects_current_if_match_last_modified_time
-test_delete_objects_current_if_match_size
-test_delete_objects_if_match
-test_delete_objects_if_match_last_modified_time
-test_delete_objects_if_match_size
-test_delete_objects_version_if_match
-test_delete_objects_version_if_match_last_modified_time
-test_delete_objects_version_if_match_size
-test_encryption_sse_c_multipart_invalid_chunks_1
-test_encryption_sse_c_multipart_invalid_chunks_2
-test_get_multipart_checksum_object_attributes
-test_head_bucket_usage
-test_lifecycle_cloud_multiple_transition
-test_lifecycle_cloud_transition
-test_lifecycle_cloud_transition_large_obj
-test_lifecycle_deletemarker_expiration
-test_lifecycle_deletemarker_expiration_with_days_tag
-test_lifecycle_expiration
-test_lifecycle_expiration_days0
-test_lifecycle_expiration_newer_noncurrent
-test_lifecycle_expiration_noncur_tags1
-test_lifecycle_expiration_size_gt
-test_lifecycle_expiration_size_lt
-test_lifecycle_expiration_tags2
-test_lifecycle_expiration_versioned_tags2
-test_lifecycle_multipart_expiration
-test_lifecycle_noncur_cloud_transition
-test_lifecycle_noncur_expiration
-test_lifecycle_noncur_transition
-test_lifecycle_transition
-test_lifecycle_transition_single_rule_multi_trans
-test_lifecyclev2_expiration
-test_list_buckets_anonymous
-test_list_buckets_paginated
-test_list_multipart_upload
-test_list_multipart_upload_owner
-test_multipart_get_part
-test_multipart_single_get_part
-test_multipart_sse_c_get_part
-test_multipart_upload
-test_multipart_upload_small
-test_multipart_use_cksum_helper_crc32
-test_multipart_use_cksum_helper_crc32c
-test_multipart_use_cksum_helper_crc64nvme
-test_multipart_use_cksum_helper_sha1
-test_multipart_use_cksum_helper_sha256
-test_non_multipart_get_part
-test_non_multipart_sse_c_get_part
-test_object_header_acl_grants
-test_object_raw_get_x_amz_expires_out_max_range
-test_object_raw_get_x_amz_expires_out_positive_range
-test_object_raw_put_authenticated_expired
-test_object_read_unreadable
-test_object_requestid_matches_header_on_error
-test_object_set_get_unicode_metadata
-test_object_write_with_chunked_transfer_encoding
-test_post_object_invalid_date_format
-test_post_object_invalid_request_field_value
-test_post_object_missing_policy_condition
-test_post_object_request_missing_policy_specified_field
-test_post_object_set_key_from_filename
-test_post_object_success_redirect_action
-test_post_object_tags_anonymous_request
-test_post_object_wrong_bucket
-test_put_bucket_logging_account_j
-test_put_bucket_logging_account_s
-test_put_bucket_logging_extensions
-test_put_bucket_logging_policy_wildcard_objects
-test_put_bucket_logging_tenant_j
-test_put_bucket_logging_tenant_s
-test_put_bucket_ownership_bucket_owner_enforced
-test_put_bucket_ownership_bucket_owner_preferred
-test_put_bucket_ownership_object_writer
-test_put_object_current_if_match
-test_read_through
-test_restore_noncur_obj
-test_restore_object_permanent
-test_restore_object_temporary
-test_sse_kms_post_object_authenticated_request
-test_versioned_object_acl_no_version_specified
-test_versioning_multi_object_delete_with_marker_create
-test_versioning_stack_delete_merkers
@@ -0,0 +1,196 @@
+#!/usr/bin/env python3
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Diff a ceph/s3-tests junit.xml result against the RustFS test classification lists.
+
+Classifies every executed test into:
+  - regressions:            listed in implemented_tests.txt but failed/errored
+  - promotion candidates:   passed but listed in unimplemented_tests.txt / excluded_tests.txt
+  - unclassified passes:    passed but not present in any list (new upstream tests)
+  - unclassified failures:  failed and not present in any list (new upstream tests)
+
+Writes a markdown report and prints a summary to stdout. Exit code is 0 unless
+--fail-on-regression is given and at least one regression was found.
+"""
+
+from __future__ import annotations
+
+import argparse
+import pathlib
+import sys
+import xml.etree.ElementTree as ET
+
+LIST_FILES = {
+    "implemented": "implemented_tests.txt",
+    "unimplemented": "unimplemented_tests.txt",
+    "excluded": "excluded_tests.txt",
+}
+
+
+def load_list(path: pathlib.Path) -> set[str]:
+    names: set[str] = set()
+    if not path.is_file():
+        return names
+    for line in path.read_text(encoding="utf-8").splitlines():
+        line = line.strip()
+        if line and not line.startswith("#"):
+            names.add(line)
+    return names
+
+
+def base_name(testcase_name: str) -> str:
+    """Strip pytest parametrization (test_foo[param]) to match list entries."""
+    return testcase_name.split("[", 1)[0]
+
+
+def parse_junit(path: pathlib.Path) -> dict[str, str]:
+    """Return {test name: status} with status in passed/failed/error/skipped.
+
+    Parametrized cases collapse onto their base name; any failing variant marks
+    the whole test failed.
+    """
+    results: dict[str, str] = {}
+    severity = {"skipped": 0, "passed": 1, "failed": 2, "error": 2}
+    root = ET.parse(path).getroot()
+    for case in root.iter("testcase"):
+        name = base_name(case.get("name", ""))
+        if not name:
+            continue
+        if case.find("failure") is not None:
+            status = "failed"
+        elif case.find("error") is not None:
+            status = "error"
+        elif case.find("skipped") is not None:
+            status = "skipped"
+        else:
+            status = "passed"
+        prev = results.get(name)
+        if prev is None or severity[status] > severity[prev]:
+            results[name] = status
+    return results
+
+
+def render_section(title: str, rows: list[str], hint: str = "") -> list[str]:
+    lines = [f"## {title} ({len(rows)})", ""]
+    if hint:
+        lines += [hint, ""]
+    if rows:
+        lines += [f"- `{name}`" for name in sorted(rows)]
+    else:
+        lines.append("_none_")
+    lines.append("")
+    return lines
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--junit", required=True, type=pathlib.Path, help="junit.xml produced by pytest")
+    parser.add_argument(
+        "--lists-dir",
+        type=pathlib.Path,
+        default=pathlib.Path(__file__).resolve().parent,
+        help="directory containing the *_tests.txt classification files",
+    )
+    parser.add_argument("--output", type=pathlib.Path, help="write the markdown report to this path")
+    parser.add_argument(
+        "--fail-on-regression",
+        action="store_true",
+        help="exit non-zero when a test from implemented_tests.txt failed",
+    )
+    args = parser.parse_args()
+
+    if not args.junit.is_file():
+        print(f"[ERROR] junit file not found: {args.junit}", file=sys.stderr)
+        return 2
+
+    lists = {key: load_list(args.lists_dir / fname) for key, fname in LIST_FILES.items()}
+    results = parse_junit(args.junit)
+
+    regressions: list[str] = []
+    promotions: dict[str, list[str]] = {"unimplemented": [], "excluded": []}
+    unclassified_passed: list[str] = []
+    unclassified_failed: list[str] = []
+    counts = {"passed": 0, "failed": 0, "error": 0, "skipped": 0}
+
+    for name, status in results.items():
+        counts[status] += 1
+        if status in ("failed", "error"):
+            if name in lists["implemented"]:
+                regressions.append(name)
+            elif name not in lists["unimplemented"] and name not in lists["excluded"]:
+                unclassified_failed.append(name)
+        elif status == "passed":
+            if name in lists["unimplemented"]:
+                promotions["unimplemented"].append(name)
+            elif name in lists["excluded"]:
+                promotions["excluded"].append(name)
+            elif name not in lists["implemented"]:
+                unclassified_passed.append(name)
+
+    lines = [
+        "# S3 compatibility report",
+        "",
+        f"Executed: {len(results)} tests — "
+        f"{counts['passed']} passed, {counts['failed']} failed, "
+        f"{counts['error']} errored, {counts['skipped']} skipped.",
+        "",
+    ]
+    lines += render_section(
+        "Regressions",
+        regressions,
+        "Listed in `implemented_tests.txt` but failing — these gate PRs and must be fixed.",
+    )
+    lines += render_section(
+        "Promotion candidates (from unimplemented_tests.txt)",
+        promotions["unimplemented"],
+        "Now passing — move to `implemented_tests.txt` to lock in the coverage.",
+    )
+    lines += render_section(
+        "Promotion candidates (from excluded_tests.txt)",
+        promotions["excluded"],
+        "Passing despite being excluded — re-evaluate the exclusion.",
+    )
+    lines += render_section(
+        "Unclassified passes",
+        unclassified_passed,
+        "Passing but absent from every list — add to `implemented_tests.txt`.",
+    )
+    lines += render_section(
+        "Unclassified failures",
+        unclassified_failed,
+        "Failing and absent from every list — triage into `unimplemented_tests.txt` or `excluded_tests.txt`.",
+    )
+
+    report = "\n".join(lines)
+    if args.output:
+        args.output.parent.mkdir(parents=True, exist_ok=True)
+        args.output.write_text(report, encoding="utf-8")
+        print(f"[INFO] Report written to {args.output}")
+
+    print(
+        f"[INFO] {len(regressions)} regression(s), "
+        f"{len(promotions['unimplemented']) + len(promotions['excluded']) + len(unclassified_passed)} promotion candidate(s), "
+        f"{len(unclassified_failed)} unclassified failure(s)"
+    )
+    for name in sorted(regressions):
+        print(f"[REGRESSION] {name}")
+
+    if args.fail_on_regression and regressions:
+        return 1
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
@@ -43,6 +43,20 @@ S3_PORT="${S3_PORT:-9000}"
 TEST_MODE="${TEST_MODE:-single}"
 MAXFAIL="${MAXFAIL:-1}"
 XDIST="${XDIST:-0}"
+# Test scope:
+#   "implemented" (default) - run only the implemented_tests.txt whitelist (PR gate)
+#   "all"                   - run the entire upstream suite (scheduled full sweep)
+TEST_SCOPE="${TEST_SCOPE:-implemented}"
+if [[ "${TEST_SCOPE}" != "implemented" && "${TEST_SCOPE}" != "all" ]]; then
+    echo "[ERROR] Invalid TEST_SCOPE: ${TEST_SCOPE} (must be \"implemented\" or \"all\")" >&2
+    exit 1
+fi
+
+# Upstream ceph/s3-tests suite, pinned for reproducible runs.
+# Bump S3TESTS_REV deliberately: upstream changes can rename tests or change
+# assertions, so a bump usually requires reclassifying the test list files.
+S3TESTS_REPO="${S3TESTS_REPO:-https://github.com/ceph/s3-tests.git}"
+S3TESTS_REV="${S3TESTS_REV:-5522d1c351f75bc00ae0f64f742f3f095f5939d9}"

 # Compatibility default for the s3-tests harness:
 # this script provisions multiple local export directories on the same physical disk.
@@ -146,7 +160,12 @@ fi
 #   2. Easy maintenance - edit txt files to add/remove tests
 #   3. Separation of concerns - test classification vs test execution
 # =============================================================================
-if [[ -z "${TESTEXPR:-}" ]]; then
+TESTEXPR="${TESTEXPR:-}"
+if [[ -n "${TESTEXPR}" ]]; then
+    log_info "Using custom TESTEXPR selection: ${TESTEXPR}"
+elif [[ "${TEST_SCOPE}" == "all" ]]; then
+    log_info "TEST_SCOPE=all: running the entire upstream test suite (no whitelist)"
+else
    if [[ -f "${IMPLEMENTED_TESTS_FILE}" ]]; then
        log_info "Loading test list from: ${IMPLEMENTED_TESTS_FILE}"
        load_testnodes_from_file "${IMPLEMENTED_TESTS_FILE}"
@@ -240,8 +259,11 @@ Environment Variables:
  S3_ALT_ACCESS_KEY      - Alt user access key (default: rustfsalt)
  S3_ALT_SECRET_KEY      - Alt user secret key (default: rustfsalt)
  RUSTFS_SSE_S3_MASTER_KEY - Optional base64 32-byte key for local managed SSE fallback
-  MAXFAIL                - Stop after N failures (default: 1)
+  MAXFAIL                - Stop after N failures, 0 = never stop (default: 1)
  XDIST                  - Enable parallel execution with N workers (default: 0)
+  TEST_SCOPE             - "implemented" (whitelist, default) or "all" (entire upstream suite)
+  S3TESTS_REPO           - s3-tests repository URL (default: https://github.com/ceph/s3-tests.git)
+  S3TESTS_REV            - Pinned s3-tests commit; bump deliberately and reclassify test lists
  MARKEXPR               - pytest marker expression (default: no marker filtering)
  TESTEXPR               - pytest -k expression (overrides implemented_tests.txt node list)
  S3TESTS_CONF_TEMPLATE  - Path to s3tests config template (default: .github/s3tests/s3tests.conf)
@@ -819,14 +841,23 @@ awscurl \

 log_info "Alt user provisioned successfully"

-# Step 8: Prepare s3-tests
-log_info "Preparing s3-tests..."
-if [ ! -d "${PROJECT_ROOT}/s3-tests" ]; then
-    git clone --depth 1 https://github.com/ceph/s3-tests.git "${PROJECT_ROOT}/s3-tests" || {
-        log_error "Failed to clone s3-tests"
+# Step 8: Prepare s3-tests, pinned to S3TESTS_REV for reproducible runs
+log_info "Preparing s3-tests at ${S3TESTS_REV}..."
+if [ ! -d "${PROJECT_ROOT}/s3-tests/.git" ]; then
+    git init -q "${PROJECT_ROOT}/s3-tests"
+    git -C "${PROJECT_ROOT}/s3-tests" remote add origin "${S3TESTS_REPO}"
+fi
+if ! git -C "${PROJECT_ROOT}/s3-tests" cat-file -e "${S3TESTS_REV}^{commit}" 2>/dev/null; then
+    git -C "${PROJECT_ROOT}/s3-tests" fetch --depth 1 origin "${S3TESTS_REV}" || {
+        log_error "Failed to fetch s3-tests revision ${S3TESTS_REV} from ${S3TESTS_REPO}"
        exit 1
    }
 fi
+# -f discards local edits left by previous runs (e.g. pytest-xdist appended to requirements.txt)
+git -C "${PROJECT_ROOT}/s3-tests" checkout -qf --detach "${S3TESTS_REV}" || {
+    log_error "Failed to checkout s3-tests revision ${S3TESTS_REV}"
+    exit 1
+}

 cd "${PROJECT_ROOT}/s3-tests"

@@ -862,8 +893,11 @@ fi
 PYTEST_SELECTION_ARGS=()
 if [[ "${USE_FILE_TEST_NODES}" == "true" ]]; then
    PYTEST_SELECTION_ARGS=("${TEST_NODE_ARGS[@]}")
-else
+elif [[ -n "${TESTEXPR}" ]]; then
    PYTEST_SELECTION_ARGS=("${S3_TEST_FILE}" -k "${TESTEXPR}")
+else
+    # TEST_SCOPE=all: the whole upstream suite
+    PYTEST_SELECTION_ARGS=("${S3_TEST_FILE}")
 fi

 # Run tests from s3tests/functional
@@ -894,6 +928,16 @@ elif [ "${DEPLOY_MODE}" = "existing" ]; then
    echo "{\"host\": \"${S3_HOST}\", \"port\": ${S3_PORT}, \"mode\": \"existing\"}" > "${ARTIFACTS_DIR}/rustfs-${TEST_MODE}/inspect.json" || true
 fi

+# Step 11: Classification report (informational, never fails the run)
+REPORT_SCRIPT="${SCRIPT_DIR}/report_compat.py"
+if [ -f "${REPORT_SCRIPT}" ] && [ -f "${ARTIFACTS_DIR}/junit.xml" ]; then
+    python3 "${REPORT_SCRIPT}" \
+        --junit "${ARTIFACTS_DIR}/junit.xml" \
+        --lists-dir "${SCRIPT_DIR}" \
+        --output "${ARTIFACTS_DIR}/compat-report.md" \
+        || log_warn "Compatibility report generation failed"
+fi
+
 # Summary
 if [ ${TEST_EXIT_CODE} -eq 0 ]; then
    log_info "Tests completed successfully!"