From 3eeb459ecebb63cf43424334488e8cbc7257259f Mon Sep 17 00:00:00 2001 From: Zhengchao An Date: Thu, 2 Jul 2026 23:43:19 +0800 Subject: [PATCH] ci(s3-tests): pin upstream suite, add weekly full sweep and compat report (#4204) Reworks the S3 compatibility test harness for reproducibility and faster feedback: - Pin ceph/s3-tests to a fixed commit (S3TESTS_REV, fetch-by-SHA) so the 449-test PR gate is reproducible; previously every run cloned upstream master, letting test renames or assertion changes break CI silently. - PR gate (ci.yml s3-implemented-tests): MAXFAIL=0 + XDIST=4 so a single CI round reports every failure in parallel instead of stopping at the first one serially. - Add TEST_SCOPE=all to run.sh to run the entire upstream suite, and report_compat.py to diff junit results against the classification lists (regressions, promotion candidates, unclassified tests). - Rewrite e2e-s3tests.yml: delegate execution to run.sh (single source of truth; also fixes the broken config generation that left S3_PORT empty), add a weekly scheduled full sweep that fails only on whitelist regressions, and fix the multi-node topology to a real distributed cluster (endpoint-style RUSTFS_VOLUMES) instead of four independent single-node stores behind a load balancer. - Docs: rewrite stale .github/s3tests/README.md (marker-era strategy), update scripts/s3-tests/README.md, fix dead build_testexpr.sh reference in S3_COMPAT_WORKFLOW.md, drop legacy non_standard_tests.txt. All 747 classified test names verified present at the pinned revision; 13 upstream tests are currently unclassified and will surface in the first full-sweep report. Co-authored-by: Claude Fable 5 --- .github/s3tests/README.md | 124 +++---- .github/workflows/ci.yml | 3 +- .github/workflows/e2e-s3tests.yml | 417 ++++++++---------------- scripts/s3-tests/README.md | 64 ++-- scripts/s3-tests/S3_COMPAT_WORKFLOW.md | 11 +- scripts/s3-tests/non_standard_tests.txt | 236 -------------- scripts/s3-tests/report_compat.py | 196 +++++++++++ scripts/s3-tests/run.sh | 60 +++- 8 files changed, 472 insertions(+), 639 deletions(-) delete mode 100644 scripts/s3-tests/non_standard_tests.txt create mode 100644 scripts/s3-tests/report_compat.py diff --git a/.github/s3tests/README.md b/.github/s3tests/README.md index af61ed25a..8f61534ec 100644 --- a/.github/s3tests/README.md +++ b/.github/s3tests/README.md @@ -1,103 +1,81 @@ # S3 Compatibility Tests Configuration -This directory contains the configuration for running [Ceph S3 compatibility tests](https://github.com/ceph/s3-tests) against RustFS. +This directory contains the configuration template for running +[Ceph S3 compatibility tests](https://github.com/ceph/s3-tests) against RustFS. + +The test runner itself lives in [`scripts/s3-tests/`](../../scripts/s3-tests/) +— see its README for how tests are selected, executed, and reported. This +directory only holds the shared config template. ## Configuration File -The `s3tests.conf` file is based on the official `s3tests.conf.SAMPLE` from the ceph/s3-tests repository. It uses environment variable substitution via `envsubst` to configure the endpoint and credentials. +`s3tests.conf` is based on the official `s3tests.conf.SAMPLE` from the +ceph/s3-tests repository. It is a template: `scripts/s3-tests/run.sh` renders +it with `envsubst` before every run. -### Key Configuration Points +Required environment variables (all have defaults in `run.sh`): -- **Host**: Set via `${S3_HOST}` environment variable (e.g., `rustfs-single` for single-node, `lb` for multi-node) -- **Port**: 9000 (standard RustFS port) -- **Credentials**: Uses `${S3_ACCESS_KEY}` and `${S3_SECRET_KEY}` from workflow environment -- **TLS**: Disabled (`is_secure = False`) +| Variable | Meaning | +|----------|---------| +| `S3_HOST` | RustFS endpoint host | +| `S3_PORT` | RustFS endpoint port | +| `S3_ACCESS_KEY` / `S3_SECRET_KEY` | main user credentials | +| `S3_ALT_ACCESS_KEY` / `S3_ALT_SECRET_KEY` | alt user credentials (must differ from main) | -## Test Execution Strategy +TLS is disabled (`is_secure = False`). -### Network Connectivity Fix +## Test Selection Strategy -Tests run inside a Docker container on the `rustfs-net` network, which allows them to resolve and connect to the RustFS container hostnames. This fixes the "Temporary failure in name resolution" error that occurred when tests ran on the GitHub runner host. +Tests are selected by exact pytest node ids from classification list files in +`scripts/s3-tests/` — **not** by pytest markers: -### Performance Optimizations +- `implemented_tests.txt` — expected to pass; run as the per-PR gate + (`ci.yml`, job `s3-implemented-tests`) +- `unimplemented_tests.txt` — standard S3 features not yet implemented +- `excluded_tests.txt` — intentionally excluded (vendor-specific behavior or + product decisions) -1. **Parallel Execution**: Uses `pytest-xdist` with `-n 4` to run tests in parallel across 4 workers -2. **Load Distribution**: Uses `--dist=loadgroup` to distribute test groups across workers -3. **Fail-Fast**: Uses `--maxfail=50` to stop after 50 failures, saving time on catastrophic failures +The upstream s3-tests repository is pinned to a fixed commit (`S3TESTS_REV` in +`run.sh`) so results are reproducible; bump it deliberately and reclassify the +lists when upstream changes. -### Feature Filtering +## Where Tests Run -Tests are filtered using pytest markers (`-m`) to skip features not yet supported by RustFS: - -- `lifecycle` - Bucket lifecycle policies -- `versioning` - Object versioning -- `s3website` - Static website hosting -- `bucket_logging` - Bucket logging -- `encryption` / `sse_s3` - Server-side encryption -- `cloud_transition` / `cloud_restore` - Cloud storage transitions -- `lifecycle_expiration` / `lifecycle_transition` - Lifecycle operations - -This filtering: -1. Reduces test execution time significantly (from 1+ hour to ~10-15 minutes) -2. Focuses on features RustFS currently supports -3. Avoids hundreds of expected failures +- **Per PR**: `.github/workflows/ci.yml` job `s3-implemented-tests` runs the + implemented whitelist against a single-node debug binary. Any failure blocks + the PR. +- **Weekly + manual**: `.github/workflows/e2e-s3tests.yml` runs the full + upstream suite (`TEST_SCOPE=all`) against a Docker deployment (single node + or a 4-node distributed cluster behind HAProxy). It fails only on + regressions in the implemented whitelist and publishes a classification + report (`compat-report.md`, also shown in the job summary) listing promotion + candidates and unclassified tests. ## Running Tests Locally -### Single-Node Test - ```bash -# Set credentials -export S3_ACCESS_KEY=rustfsadmin -export S3_SECRET_KEY=rustfsadmin +# Whitelist run against a locally built binary (the same thing the PR gate does) +./scripts/s3-tests/run.sh -# Start RustFS container -docker run -d --name rustfs-single \ - --network rustfs-net \ - -e RUSTFS_ADDRESS=0.0.0.0:9000 \ - -e RUSTFS_ACCESS_KEY=$S3_ACCESS_KEY \ - -e RUSTFS_SECRET_KEY=$S3_SECRET_KEY \ - -e RUSTFS_VOLUMES="/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3" \ - rustfs-ci +# Full sweep, never stop on failures, 4 pytest workers +TEST_SCOPE=all MAXFAIL=0 XDIST=4 ./scripts/s3-tests/run.sh -# Generate config -export S3_HOST=rustfs-single -envsubst < .github/s3tests/s3tests.conf > /tmp/s3tests.conf - -# Run tests -docker run --rm \ - --network rustfs-net \ - -v /tmp/s3tests.conf:/etc/s3tests.conf:ro \ - python:3.12-slim \ - bash -c ' - apt-get update -qq && apt-get install -y -qq git - git clone --depth 1 https://github.com/ceph/s3-tests.git /s3-tests - cd /s3-tests - pip install -q -r requirements.txt pytest-xdist - S3TEST_CONF=/etc/s3tests.conf pytest -v -n 4 \ - s3tests/functional/test_s3.py \ - -m "not lifecycle and not versioning and not s3website and not bucket_logging and not encryption and not sse_s3" - ' +# A specific test against an already-running server +DEPLOY_MODE=existing TESTEXPR="test_bucket_list_empty" ./scripts/s3-tests/run.sh ``` -## Test Results Interpretation - -- **PASSED**: Test succeeded, feature works correctly -- **FAILED**: Test failed, indicates a potential bug or incompatibility -- **ERROR**: Test setup failed (e.g., network issues, missing dependencies) -- **SKIPPED**: Test skipped due to marker filtering +Results land in `artifacts/s3tests-single/` (`junit.xml`, `pytest.log`, +`compat-report.md`). ## Adding New Feature Support -When adding support for a new S3 feature to RustFS: - -1. Remove the corresponding marker from the filter in `.github/workflows/e2e-s3tests.yml` -2. Run the tests to verify compatibility -3. Fix any failing tests -4. Update this README to reflect the newly supported feature +Follow [`scripts/s3-tests/S3_COMPAT_WORKFLOW.md`](../../scripts/s3-tests/S3_COMPAT_WORKFLOW.md). +In short: fix the behavior, verify with `TESTEXPR=...`, then move the test +from `unimplemented_tests.txt` to `implemented_tests.txt` so the PR gate locks +it in. ## References - [Ceph S3 Tests Repository](https://github.com/ceph/s3-tests) - [S3 API Compatibility](https://docs.aws.amazon.com/AmazonS3/latest/API/) -- [pytest-xdist Documentation](https://pytest-xdist.readthedocs.io/) +- [RustFS test runner](../../scripts/s3-tests/README.md) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 555728228..7a432d500 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -429,7 +429,8 @@ jobs: DEPLOY_MODE=binary \ RUSTFS_BINARY=./target/debug/rustfs \ TEST_MODE=single \ - MAXFAIL=1 \ + MAXFAIL=0 \ + XDIST=4 \ S3_PORT="${S3_PORT}" \ DATA_ROOT="${RUN_ROOT}" \ S3TESTS_CONF=artifacts/s3tests-single/s3tests.conf \ diff --git a/.github/workflows/e2e-s3tests.yml b/.github/workflows/e2e-s3tests.yml index e79ffa521..101a52665 100644 --- a/.github/workflows/e2e-s3tests.yml +++ b/.github/workflows/e2e-s3tests.yml @@ -12,31 +12,59 @@ # See the License for the specific language governing permissions and # limitations under the License. +# Full ceph/s3-tests sweep against RustFS. +# +# The per-PR compatibility gate lives in ci.yml (job s3-implemented-tests) and +# runs only the implemented_tests.txt whitelist. This workflow complements it: +# +# - Scheduled weekly full sweep (TEST_SCOPE=all): runs the ENTIRE upstream +# suite and reports promotion candidates (tests that newly pass) and +# unclassified tests. The job fails only on regressions in the implemented +# whitelist or on infrastructure errors — expected failures from +# not-yet-implemented features do not turn the run red. +# - Manual runs (workflow_dispatch): same, with configurable mode/scope. +# +# All test execution is delegated to scripts/s3-tests/run.sh (single source of +# truth, also used locally and by the PR gate). This workflow only provisions +# the server topology: a single node, or a real 4-node distributed cluster +# (erasure-coded across nodes) behind an HAProxy round-robin load balancer. + name: e2e-s3tests on: workflow_dispatch: inputs: test-mode: - description: "Test mode to run" + description: "Server topology" required: true type: choice default: "single" options: - single - multi + scope: + description: "Test scope: full upstream suite or implemented whitelist" + required: true + type: choice + default: "all" + options: + - all + - implemented xdist: - description: "Enable pytest-xdist (parallel). '0' to disable." + description: "pytest-xdist workers. '0' to disable." + required: false + default: "4" + maxfail: + description: "Stop after N failures. '0' to run everything." required: false default: "0" - maxfail: - description: "Stop after N failures (debug friendly)" - required: false - default: "1" markexpr: - description: "pytest -m expression (feature filters)" + description: "Optional pytest -m expression" required: false - default: "not lifecycle and not versioning and not s3website and not bucket_logging and not encryption" + default: "" + schedule: + # Weekly full sweep (Sunday 02:00 UTC): full suite, single-node topology. + - cron: "0 2 * * 0" env: # main user @@ -47,11 +75,20 @@ env: S3_ALT_SECRET_KEY: rustfsalt S3_REGION: us-east-1 + S3_HOST: 127.0.0.1 + S3_PORT: "9000" RUST_LOG: info PLATFORM: linux/amd64 BUILDX_CACHE_SCOPE: rustfs-e2e-s3tests-source + # Scheduled runs have no inputs; default to a single-node full sweep. + TEST_MODE: ${{ github.event.inputs.test-mode || 'single' }} + TEST_SCOPE: ${{ github.event.inputs.scope || 'all' }} + XDIST: ${{ github.event.inputs.xdist || '4' }} + MAXFAIL: ${{ github.event.inputs.maxfail || '0' }} + MARKEXPR: ${{ github.event.inputs.markexpr || '' }} + concurrency: group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event.inputs['test-mode'] || 'single' }} cancel-in-progress: true @@ -61,10 +98,9 @@ defaults: shell: bash jobs: - s3tests-single: - if: github.event.inputs['test-mode'] == 'single' - runs-on: ubicloud-standard-2 - timeout-minutes: 120 + s3tests: + runs-on: ubicloud-standard-4 + timeout-minutes: 180 steps: - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 @@ -93,222 +129,61 @@ jobs: -t rustfs-ci \ -f Dockerfile.source . - - name: Create network - run: docker network inspect rustfs-net >/dev/null 2>&1 || docker network create rustfs-net - - - name: Remove existing rustfs-single (if any) - run: docker rm -f rustfs-single >/dev/null 2>&1 || true - - name: Start single RustFS + if: env.TEST_MODE == 'single' run: | + docker network inspect rustfs-net >/dev/null 2>&1 || docker network create rustfs-net + docker rm -f rustfs-single >/dev/null 2>&1 || true docker run -d --name rustfs-single \ --network rustfs-net \ - -p 9000:9000 \ + -p ${S3_PORT}:9000 \ -e RUSTFS_ADDRESS=0.0.0.0:9000 \ - -e RUSTFS_ACCESS_KEY=$S3_ACCESS_KEY \ - -e RUSTFS_SECRET_KEY=$S3_SECRET_KEY \ - -e RUSTFS_VOLUMES="/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3" \ + -e RUSTFS_ACCESS_KEY=${S3_ACCESS_KEY} \ + -e RUSTFS_SECRET_KEY=${S3_SECRET_KEY} \ + -e RUSTFS_VOLUMES="/data/rustfs{0...3}" \ -v /tmp/rustfs-single:/data \ rustfs-ci - - name: Wait for RustFS ready + - name: Start 4-node distributed cluster + if: env.TEST_MODE == 'multi' run: | - for i in {1..60}; do - if curl -sf http://127.0.0.1:9000/health >/dev/null 2>&1; then - echo "RustFS is ready" - exit 0 - fi + # A real distributed deployment: every node lists all endpoints in + # RUSTFS_VOLUMES so data is erasure-coded ACROSS nodes. Do not use + # node-local volume paths here — that would create four independent + # single-node stores behind the load balancer. + cat > compose.yml </dev/null)" != "true" ]; then - echo "RustFS container not running" >&2 - docker logs rustfs-single || true - exit 1 - fi - - sleep 2 - done - - echo "Health check timed out" >&2 - docker logs rustfs-single || true - exit 1 - - - name: Generate s3tests config - run: | - export S3_HOST=127.0.0.1 - envsubst < .github/s3tests/s3tests.conf > s3tests.conf - - - name: Provision s3-tests alt user (required by suite) - run: | - # Admin API requires AWS SigV4 signing. awscurl is used by RustFS codebase as well. - awscurl \ - --service s3 \ - --region "${S3_REGION}" \ - --access_key "${S3_ACCESS_KEY}" \ - --secret_key "${S3_SECRET_KEY}" \ - -X PUT \ - -H 'Content-Type: application/json' \ - -d '{"secretKey":"'"${S3_ALT_SECRET_KEY}"'","status":"enabled","policy":"readwrite"}' \ - "http://127.0.0.1:9000/rustfs/admin/v3/add-user?accessKey=${S3_ALT_ACCESS_KEY}" - - # Explicitly attach built-in policy via policy mapping. - # s3-tests relies on alt client being able to ListBuckets during setup cleanup. - awscurl \ - --service s3 \ - --region "${S3_REGION}" \ - --access_key "${S3_ACCESS_KEY}" \ - --secret_key "${S3_SECRET_KEY}" \ - -X PUT \ - "http://127.0.0.1:9000/rustfs/admin/v3/set-user-or-group-policy?policyName=readwrite&userOrGroup=${S3_ALT_ACCESS_KEY}&isGroup=false" - - # Sanity check: alt user can list buckets (should not be AccessDenied). - awscurl \ - --service s3 \ - --region "${S3_REGION}" \ - --access_key "${S3_ALT_ACCESS_KEY}" \ - --secret_key "${S3_ALT_SECRET_KEY}" \ - -X GET \ - "http://127.0.0.1:9000/" >/dev/null - - - name: Prepare s3-tests - run: | - git clone --depth 1 https://github.com/ceph/s3-tests.git s3-tests - - - name: Run ceph s3-tests (debug friendly) - run: | - export PATH="$HOME/.local/bin:$PATH" - mkdir -p artifacts/s3tests-single - - cd s3-tests - - set -o pipefail - - MAXFAIL="${{ github.event.inputs.maxfail }}" - if [ -z "$MAXFAIL" ]; then MAXFAIL="1"; fi - - MARKEXPR="${{ github.event.inputs.markexpr }}" - if [ -z "$MARKEXPR" ]; then MARKEXPR="not lifecycle and not versioning and not s3website and not bucket_logging and not encryption"; fi - - XDIST="${{ github.event.inputs.xdist }}" - if [ -z "$XDIST" ]; then XDIST="0"; fi - XDIST_ARGS="" - if [ "$XDIST" != "0" ]; then - # Add pytest-xdist to requirements.txt so tox installs it inside - # its virtualenv. Installing outside tox does NOT work. - echo "pytest-xdist" >> requirements.txt - XDIST_ARGS="-n $XDIST --dist=loadgroup" - fi - - # Run tests from s3tests/functional (boto2+boto3 combined directory). - S3TEST_CONF=${GITHUB_WORKSPACE}/s3tests.conf \ - tox -- \ - -vv -ra --showlocals --tb=long \ - --maxfail="$MAXFAIL" \ - --junitxml=${GITHUB_WORKSPACE}/artifacts/s3tests-single/junit.xml \ - $XDIST_ARGS \ - s3tests/functional/test_s3.py \ - -m "$MARKEXPR" \ - 2>&1 | tee ${GITHUB_WORKSPACE}/artifacts/s3tests-single/pytest.log - - - name: Collect RustFS logs - if: always() - run: | - mkdir -p artifacts/rustfs-single - docker logs rustfs-single > artifacts/rustfs-single/rustfs.log 2>&1 || true - docker inspect rustfs-single > artifacts/rustfs-single/inspect.json || true - - - name: Upload artifacts - if: always() && env.ACT != 'true' - uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 - with: - name: s3tests-single - path: artifacts/** - - s3tests-multi: - if: github.event_name == 'workflow_dispatch' && github.event.inputs['test-mode'] == 'multi' - runs-on: ubicloud-standard-2 - timeout-minutes: 150 - steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 - - - name: Cache pip downloads - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6 - with: - path: ~/.cache/pip - key: ${{ runner.os }}-pip-e2e-s3tests-${{ hashFiles('.github/workflows/e2e-s3tests.yml') }} - restore-keys: | - ${{ runner.os }}-pip-e2e-s3tests- - - - name: Install Python tools - run: | - python3 -m pip install --user --upgrade pip awscurl tox - echo "$HOME/.local/bin" >> "$GITHUB_PATH" - - - name: Enable buildx - uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3 - - - name: Build RustFS image (source, cached) - run: | - DOCKER_BUILDKIT=1 docker buildx build --load \ - --platform ${PLATFORM} \ - --cache-from type=gha,scope=${BUILDX_CACHE_SCOPE} \ - --cache-to type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE} \ - -t rustfs-ci \ - -f Dockerfile.source . - - - name: Prepare cluster compose - run: | - cat > compose.yml <<'EOF' services: rustfs1: - image: rustfs-ci + <<: *rustfs-node hostname: rustfs1 - networks: [rustfs-net] - environment: - RUSTFS_ADDRESS: "0.0.0.0:9000" - RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY} - RUSTFS_SECRET_KEY: ${S3_SECRET_KEY} - RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3" - volumes: - - rustfs1-data:/data + volumes: [rustfs1-data:/data] rustfs2: - image: rustfs-ci + <<: *rustfs-node hostname: rustfs2 - networks: [rustfs-net] - environment: - RUSTFS_ADDRESS: "0.0.0.0:9000" - RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY} - RUSTFS_SECRET_KEY: ${S3_SECRET_KEY} - RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3" - volumes: - - rustfs2-data:/data + volumes: [rustfs2-data:/data] rustfs3: - image: rustfs-ci + <<: *rustfs-node hostname: rustfs3 - networks: [rustfs-net] - environment: - RUSTFS_ADDRESS: "0.0.0.0:9000" - RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY} - RUSTFS_SECRET_KEY: ${S3_SECRET_KEY} - RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3" - volumes: - - rustfs3-data:/data + volumes: [rustfs3-data:/data] rustfs4: - image: rustfs-ci + <<: *rustfs-node hostname: rustfs4 - networks: [rustfs-net] - environment: - RUSTFS_ADDRESS: "0.0.0.0:9000" - RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY} - RUSTFS_SECRET_KEY: ${S3_SECRET_KEY} - RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3" - volumes: - - rustfs4-data:/data + volumes: [rustfs4-data:/data] lb: image: haproxy:2.9 hostname: lb networks: [rustfs-net] ports: - - "9000:9000" + - "${S3_PORT}:9000" volumes: - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro networks: @@ -340,104 +215,78 @@ jobs: server s4 rustfs4:9000 check EOF - - name: Launch cluster - run: docker compose -f compose.yml up -d + docker compose -f compose.yml up -d - - name: Wait for LB ready + - name: Wait for RustFS ready run: | - for i in {1..90}; do - if curl -sf http://127.0.0.1:9000/health >/dev/null 2>&1; then - echo "Load balancer is ready" + for i in {1..120}; do + if curl -sf "http://${S3_HOST}:${S3_PORT}/health" >/dev/null 2>&1; then + echo "RustFS is ready" exit 0 fi sleep 2 done - echo "LB or backend not ready" >&2 - docker compose -f compose.yml logs --tail=200 || true + echo "Health check timed out" >&2 + if [ "${TEST_MODE}" = "single" ]; then + docker logs rustfs-single || true + else + docker compose -f compose.yml logs --tail=200 || true + fi exit 1 - - name: Generate s3tests config + - name: Run ceph s3-tests run: | - export S3_HOST=127.0.0.1 - envsubst < .github/s3tests/s3tests.conf > s3tests.conf + set +e + DEPLOY_MODE=existing \ + TEST_MODE="${TEST_MODE}" \ + TEST_SCOPE="${TEST_SCOPE}" \ + XDIST="${XDIST}" \ + MAXFAIL="${MAXFAIL}" \ + MARKEXPR="${MARKEXPR}" \ + ./scripts/s3-tests/run.sh + RC=$? + set -e - - name: Provision s3-tests alt user (required by suite) - run: | - awscurl \ - --service s3 \ - --region "${S3_REGION}" \ - --access_key "${S3_ACCESS_KEY}" \ - --secret_key "${S3_SECRET_KEY}" \ - -X PUT \ - -H 'Content-Type: application/json' \ - -d '{"secretKey":"'"${S3_ALT_SECRET_KEY}"'","status":"enabled","policy":"readwrite"}' \ - "http://127.0.0.1:9000/rustfs/admin/v3/add-user?accessKey=${S3_ALT_ACCESS_KEY}" - - awscurl \ - --service s3 \ - --region "${S3_REGION}" \ - --access_key "${S3_ACCESS_KEY}" \ - --secret_key "${S3_SECRET_KEY}" \ - -X PUT \ - "http://127.0.0.1:9000/rustfs/admin/v3/set-user-or-group-policy?policyName=readwrite&userOrGroup=${S3_ALT_ACCESS_KEY}&isGroup=false" - - awscurl \ - --service s3 \ - --region "${S3_REGION}" \ - --access_key "${S3_ALT_ACCESS_KEY}" \ - --secret_key "${S3_ALT_SECRET_KEY}" \ - -X GET \ - "http://127.0.0.1:9000/" >/dev/null - - - name: Prepare s3-tests - run: | - git clone --depth 1 https://github.com/ceph/s3-tests.git s3-tests - - - name: Run ceph s3-tests (multi, debug friendly) - run: | - export PATH="$HOME/.local/bin:$PATH" - mkdir -p artifacts/s3tests-multi - - cd s3-tests - - set -o pipefail - - MAXFAIL="${{ github.event.inputs.maxfail }}" - if [ -z "$MAXFAIL" ]; then MAXFAIL="1"; fi - - MARKEXPR="${{ github.event.inputs.markexpr }}" - if [ -z "$MARKEXPR" ]; then MARKEXPR="not lifecycle and not versioning and not s3website and not bucket_logging and not encryption"; fi - - XDIST="${{ github.event.inputs.xdist }}" - if [ -z "$XDIST" ]; then XDIST="0"; fi - XDIST_ARGS="" - if [ "$XDIST" != "0" ]; then - # Add pytest-xdist to requirements.txt so tox installs it inside - # its virtualenv. Installing outside tox does NOT work. - echo "pytest-xdist" >> requirements.txt - XDIST_ARGS="-n $XDIST --dist=loadgroup" + if [ "${TEST_SCOPE}" = "implemented" ]; then + # Whitelist run: every failure is a regression. + exit "${RC}" fi - # Run tests from s3tests/functional (boto2+boto3 combined directory). - S3TEST_CONF=${GITHUB_WORKSPACE}/s3tests.conf \ - tox -- \ - -vv -ra --showlocals --tb=long \ - --maxfail="$MAXFAIL" \ - --junitxml=${GITHUB_WORKSPACE}/artifacts/s3tests-multi/junit.xml \ - $XDIST_ARGS \ - s3tests/functional/test_s3.py \ - -m "$MARKEXPR" \ - 2>&1 | tee ${GITHUB_WORKSPACE}/artifacts/s3tests-multi/pytest.log + # Full sweep: failures outside the implemented whitelist are + # inventory (promotion candidates / unimplemented features), not a + # gate. Fail only on whitelist regressions or infrastructure errors. + JUNIT="artifacts/s3tests-${TEST_MODE}/junit.xml" + if [ ! -f "${JUNIT}" ]; then + echo "No junit.xml produced — infrastructure failure (exit ${RC})" >&2 + exit "${RC}" + fi + python3 scripts/s3-tests/report_compat.py \ + --junit "${JUNIT}" \ + --lists-dir scripts/s3-tests \ + --fail-on-regression - - name: Collect logs + - name: Publish compatibility report if: always() run: | - mkdir -p artifacts/cluster - docker compose -f compose.yml logs --no-color > artifacts/cluster/cluster.log 2>&1 || true + REPORT="artifacts/s3tests-${TEST_MODE}/compat-report.md" + if [ -f "${REPORT}" ]; then + cat "${REPORT}" >> "$GITHUB_STEP_SUMMARY" + fi + + - name: Collect RustFS logs + if: always() + run: | + mkdir -p "artifacts/rustfs-${TEST_MODE}" + if [ "${TEST_MODE}" = "single" ]; then + docker logs rustfs-single > "artifacts/rustfs-${TEST_MODE}/rustfs.log" 2>&1 || true + docker inspect rustfs-single > "artifacts/rustfs-${TEST_MODE}/inspect.json" || true + else + docker compose -f compose.yml logs --no-color > "artifacts/rustfs-${TEST_MODE}/cluster.log" 2>&1 || true + fi - name: Upload artifacts if: always() && env.ACT != 'true' uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 with: - name: s3tests-multi + name: s3tests-${{ env.TEST_MODE }} path: artifacts/** diff --git a/scripts/s3-tests/README.md b/scripts/s3-tests/README.md index 6643f5b83..5638da283 100644 --- a/scripts/s3-tests/README.md +++ b/scripts/s3-tests/README.md @@ -22,7 +22,7 @@ The script will automatically: 4. **Prepare environment**: - Generate s3tests configuration from template - Provision alt user via admin API - - Clone s3-tests repository if missing + - Fetch s3-tests at the pinned revision (`S3TESTS_REV`) - Install missing dependencies (awscurl, tox, gettext) 5. **Run tests**: Execute ceph s3-tests via tox with configured filters 6. **Collect results**: Save logs and test results in `artifacts/s3tests-${TEST_MODE}/` @@ -48,7 +48,7 @@ DEPLOY_MODE=build ./scripts/s3-tests/run.sh - Automatically compiles RustFS binary if it doesn't exist - If binary exists and was compiled less than **30 minutes** ago, compilation is skipped (unless `--no-cache` is specified) - Automatically starts the service after compilation -- Automatically clones s3-tests repository if missing +- Automatically fetches s3-tests at the pinned revision if missing - Automatically installs missing dependencies (awscurl, tox, gettext) - **Automatic Cleanup**: Process is automatically stopped when script exits @@ -67,7 +67,7 @@ DEPLOY_MODE=binary RUSTFS_BINARY=./target/release/rustfs ./scripts/s3-tests/run. **Behavior**: - Uses existing binary file (must exist, script will not compile) - Automatically starts the service using the specified binary -- Automatically clones s3-tests repository if missing +- Automatically fetches s3-tests at the pinned revision if missing - Automatically installs missing dependencies (awscurl, tox, gettext) - **Automatic Cleanup**: Process is automatically stopped when script exits @@ -82,7 +82,7 @@ DEPLOY_MODE=docker ./scripts/s3-tests/run.sh **Behavior**: - Automatically builds Docker image using `Dockerfile.source` - Creates Docker network (`rustfs-net`) if it doesn't exist -- Automatically clones s3-tests repository if missing +- Automatically fetches s3-tests at the pinned revision if missing - Automatically installs missing dependencies (awscurl, tox, gettext) - **Automatic Cleanup**: Container and network are automatically removed when script exits @@ -100,7 +100,7 @@ DEPLOY_MODE=existing S3_HOST=192.168.1.100 S3_PORT=9000 ./scripts/s3-tests/run.s **Behavior**: - Skips service startup and port availability checks - Connects directly to the specified service endpoint -- Automatically clones s3-tests repository if missing +- Automatically fetches s3-tests at the pinned revision if missing - Automatically installs missing dependencies (awscurl, tox, gettext) - **Note**: The service must already have the alt user (`rustfsalt`) provisioned, or the script will provision it automatically @@ -141,8 +141,15 @@ DEPLOY_MODE=existing S3_HOST=192.168.1.100 S3_PORT=9000 ./scripts/s3-tests/run.s ### Test Parameters - `TEST_MODE`: Test mode (default: `single`) -- `MAXFAIL`: Stop after N failures (default: `1`) +- `TEST_SCOPE`: Test scope (default: `implemented`) + - `implemented`: run only the `implemented_tests.txt` whitelist (the PR gate) + - `all`: run the entire upstream suite (used by the weekly full sweep) +- `MAXFAIL`: Stop after N failures, `0` = never stop (default: `1`) - `XDIST`: Enable parallel execution with N workers (default: `0`, disabled) +- `S3TESTS_REPO`: s3-tests repository URL (default: `https://github.com/ceph/s3-tests.git`) +- `S3TESTS_REV`: Pinned s3-tests commit for reproducible runs + - Bump deliberately: upstream changes can rename tests or change assertions, + so a bump usually requires reclassifying the test list files - `MARKEXPR`: pytest marker expression for filtering tests - Default: no marker filtering; file-based test lists control the selected tests - Can be customized to test specific marker groups @@ -246,6 +253,10 @@ Test results are saved in the `artifacts/s3tests-${TEST_MODE}/` directory (defau - `junit.xml`: Test results in JUnit format (compatible with CI/CD systems) - `pytest.log`: Detailed pytest logs with full test output +- `compat-report.md`: Classification report generated by `report_compat.py` — + regressions against `implemented_tests.txt`, promotion candidates (tests + that pass but are still listed as unimplemented/excluded), and tests missing + from every list - `rustfs-${TEST_MODE}/rustfs.log`: RustFS service logs - `rustfs-${TEST_MODE}/inspect.json`: Service metadata (PID, binary path, mode, etc.) @@ -316,8 +327,9 @@ The script will automatically install the following dependencies if missing (no - Linux: Automatically installs via `sudo apt-get install gettext-base` - **Note**: macOS installation may require manual intervention if brew fails -- **s3-tests repository**: Automatically cloned if not present - - Source: `https://github.com/ceph/s3-tests.git` +- **s3-tests repository**: Automatically fetched if not present + - Source: `https://github.com/ceph/s3-tests.git`, pinned to the commit in + `S3TESTS_REV` (see run.sh) for reproducible runs - Location: `${PROJECT_ROOT}/s3-tests` **Note**: The script adds `$HOME/.local/bin` to `PATH` automatically, so auto-installed Python tools are accessible. @@ -427,29 +439,23 @@ awscurl --service s3 --region us-east-1 \ ## Workflow Integration -This script mirrors the GitHub Actions workflow defined in `.github/workflows/e2e-s3tests.yml`. +This script is the single source of truth for running ceph s3-tests against +RustFS. Two GitHub Actions workflows delegate to it: -The script follows the same steps: +- **PR gate** (`.github/workflows/ci.yml`, job `s3-implemented-tests`): runs + the `implemented_tests.txt` whitelist against a single-node debug binary on + every pull request (`DEPLOY_MODE=binary`, `MAXFAIL=0`, `XDIST=4`). Any + failure blocks the PR. +- **Full sweep** (`.github/workflows/e2e-s3tests.yml`): weekly scheduled (and + manually dispatchable) run of the ENTIRE upstream suite (`TEST_SCOPE=all`) + against a Docker deployment — single node or a real 4-node distributed + cluster behind HAProxy. The sweep fails only on regressions in the + implemented whitelist; everything else is reported by `report_compat.py` + as promotion candidates or unclassified tests. -1. Check port availability (skip for existing mode) -2. Build/start RustFS service (varies by deployment mode) -3. Wait for service to be fully ready: - - Check process/container status - - Check port is listening - - Wait for "server started successfully" log message - - Verify S3 API is responding -4. Generate s3tests configuration from template -5. Provision alt user for s3-tests via admin API -6. Run ceph s3-tests with tox -7. Collect logs and results - -### Key Improvements Over Workflow - -- **Smart compilation**: Skips rebuild if binary is recent (< 30 minutes) -- **Better health checks**: Log-based readiness detection instead of blind waiting -- **Port conflict detection**: Prevents conflicts before starting service -- **Proxy handling**: Automatically disables proxy for localhost -- **Configurable paths**: All paths (data, configs, artifacts) can be customized +Keeping both workflows on this script means local runs, the PR gate, and the +scheduled sweep always execute tests the same way (same pinned s3-tests +revision, same config template, same user provisioning). ## See Also diff --git a/scripts/s3-tests/S3_COMPAT_WORKFLOW.md b/scripts/s3-tests/S3_COMPAT_WORKFLOW.md index f5e330012..c8e9d04b3 100644 --- a/scripts/s3-tests/S3_COMPAT_WORKFLOW.md +++ b/scripts/s3-tests/S3_COMPAT_WORKFLOW.md @@ -21,19 +21,14 @@ git merge upstream/main ### Step 2: Select candidate tests -Pick ~20 tests from `unimplemented_tests.txt` and write them to `selected_tests.txt`: - -```bash -TESTEXPR=$(scripts/s3-tests/build_testexpr.sh selected_tests.txt) \ - DEPLOY_MODE=build MAXFAIL=0 ./scripts/s3-tests/run.sh -``` - -Or run them directly: +Pick candidate tests from `unimplemented_tests.txt` and run them via a pytest `-k` expression: ```bash TESTEXPR="test_foo or test_bar" DEPLOY_MODE=build MAXFAIL=0 ./scripts/s3-tests/run.sh ``` +Alternatively, check the latest weekly full-sweep report (the `e2e-s3tests` workflow's job summary, or `artifacts/s3tests-single/compat-report.md`) — its "promotion candidates" section lists tests that already pass and only need reclassification. + Review `artifacts/s3tests-single/pytest.log` for results. ### Step 3: Triage failures diff --git a/scripts/s3-tests/non_standard_tests.txt b/scripts/s3-tests/non_standard_tests.txt deleted file mode 100644 index 0dd58a7b4..000000000 --- a/scripts/s3-tests/non_standard_tests.txt +++ /dev/null @@ -1,236 +0,0 @@ -# Legacy non-standard S3 test list (compatibility fallback) -# ========================================================= -# -# This file is kept for backward compatibility. -# Active exclusion classification now uses excluded_tests.txt. -# These tests use vendor-specific extensions not part of AWS S3 API. -# -# Exclusion reasons: -# - fails_on_aws marker: Ceph-specific features -# - X-RGW-* headers: Ceph proprietary headers -# - allowUnordered: Ceph-specific query parameter -# - Bucket Logging: Ceph-specific logging extensions -# - Lifecycle: Ceph-specific lifecycle expiration behavior -# - SSE-KMS: Ceph-specific KMS extensions -# - Error format differences: Minor response format variations - -test_account_usage -test_atomic_conditional_write_1mb -test_bucket_get_location -test_bucket_head_extended -test_bucket_header_acl_grants -test_bucket_list_return_data -test_bucket_list_return_data_versioning -test_bucket_logging_bucket_acl_required -test_bucket_logging_bucket_auth_type -test_bucket_logging_cleanup_bucket_concurrent_deletion_j -test_bucket_logging_cleanup_bucket_concurrent_deletion_j_single -test_bucket_logging_cleanup_bucket_concurrent_deletion_s -test_bucket_logging_cleanup_bucket_concurrent_deletion_s_single -test_bucket_logging_cleanup_bucket_deletion_j -test_bucket_logging_cleanup_bucket_deletion_j_single -test_bucket_logging_cleanup_bucket_deletion_s -test_bucket_logging_cleanup_bucket_deletion_s_single -test_bucket_logging_cleanup_concurrent_disabling_j -test_bucket_logging_cleanup_concurrent_disabling_j_single -test_bucket_logging_cleanup_concurrent_disabling_s -test_bucket_logging_cleanup_concurrent_disabling_s_single -test_bucket_logging_cleanup_concurrent_updating_j -test_bucket_logging_cleanup_concurrent_updating_j_single -test_bucket_logging_cleanup_concurrent_updating_s -test_bucket_logging_cleanup_concurrent_updating_s_single -test_bucket_logging_cleanup_disabling_j -test_bucket_logging_cleanup_disabling_j_single -test_bucket_logging_cleanup_disabling_s -test_bucket_logging_cleanup_disabling_s_single -test_bucket_logging_cleanup_updating_j -test_bucket_logging_cleanup_updating_j_single -test_bucket_logging_cleanup_updating_s -test_bucket_logging_cleanup_updating_s_single -test_bucket_logging_concurrent_flush_j -test_bucket_logging_concurrent_flush_j_single -test_bucket_logging_concurrent_flush_s -test_bucket_logging_concurrent_flush_s_single -test_bucket_logging_conf_concurrent_updating_pfx_j -test_bucket_logging_conf_concurrent_updating_pfx_s -test_bucket_logging_conf_concurrent_updating_roll_j -test_bucket_logging_conf_concurrent_updating_roll_s -test_bucket_logging_conf_updating_pfx_j -test_bucket_logging_conf_updating_pfx_s -test_bucket_logging_conf_updating_roll_j -test_bucket_logging_conf_updating_roll_s -test_bucket_logging_copy_objects -test_bucket_logging_copy_objects_bucket -test_bucket_logging_copy_objects_bucket_versioned -test_bucket_logging_copy_objects_versioned -test_bucket_logging_delete_objects -test_bucket_logging_delete_objects_versioned -test_bucket_logging_event_type_j -test_bucket_logging_event_type_s -test_bucket_logging_flush_empty -test_bucket_logging_flush_j -test_bucket_logging_flush_j_single -test_bucket_logging_flush_s -test_bucket_logging_flush_s_single -test_bucket_logging_get_objects -test_bucket_logging_get_objects_versioned -test_bucket_logging_head_objects -test_bucket_logging_head_objects_versioned -test_bucket_logging_key_filter_j -test_bucket_logging_key_filter_s -test_bucket_logging_mpu_copy -test_bucket_logging_mpu_copy_versioned -test_bucket_logging_mpu_j -test_bucket_logging_mpu_s -test_bucket_logging_mpu_versioned_j -test_bucket_logging_mpu_versioned_s -test_bucket_logging_mtime -test_bucket_logging_multi_delete -test_bucket_logging_multi_delete_versioned -test_bucket_logging_multiple_prefixes -test_bucket_logging_notupdating_j -test_bucket_logging_notupdating_j_single -test_bucket_logging_notupdating_s -test_bucket_logging_notupdating_s_single -test_bucket_logging_object_acl_required -test_bucket_logging_object_meta -test_bucket_logging_part_cleanup_concurrent_deletion_j -test_bucket_logging_part_cleanup_concurrent_deletion_s -test_bucket_logging_part_cleanup_concurrent_disabling_j -test_bucket_logging_part_cleanup_concurrent_disabling_s -test_bucket_logging_part_cleanup_concurrent_updating_j -test_bucket_logging_part_cleanup_concurrent_updating_s -test_bucket_logging_part_cleanup_deletion_j -test_bucket_logging_part_cleanup_deletion_s -test_bucket_logging_part_cleanup_disabling_j -test_bucket_logging_part_cleanup_disabling_s -test_bucket_logging_part_cleanup_updating_j -test_bucket_logging_part_cleanup_updating_s -test_bucket_logging_partitioned_key -test_bucket_logging_permission_change_j -test_bucket_logging_permission_change_s -test_bucket_logging_put_and_flush -test_bucket_logging_put_concurrency -test_bucket_logging_put_objects -test_bucket_logging_put_objects_versioned -test_bucket_logging_roll_time -test_bucket_logging_simple_key -test_bucket_logging_single_prefix -test_bucket_logging_target_cleanup_j -test_bucket_logging_target_cleanup_j_single -test_bucket_logging_target_cleanup_s -test_bucket_logging_target_cleanup_s_single -test_bucket_policy_get_obj_acl_existing_tag -test_bucket_policy_get_obj_existing_tag -test_bucket_policy_get_obj_tagging_existing_tag -test_bucket_policy_put_obj_copy_source -test_bucket_policy_put_obj_copy_source_meta -test_bucket_policy_put_obj_request_obj_tag -test_bucket_policy_put_obj_tagging_existing_tag -test_bucket_policy_set_condition_operator_end_with_IfExists -test_bucket_policy_upload_part_copy -test_bucket_recreate_new_acl -test_bucket_recreate_overwrite_acl -test_cors_presigned_get_object_tenant_v2 -test_cors_presigned_get_object_v2 -test_cors_presigned_put_object_tenant_v2 -test_cors_presigned_put_object_v2 -test_create_bucket_bucket_owner_enforced -test_create_bucket_bucket_owner_preferred -test_create_bucket_object_writer -test_delete_marker_expiration -test_delete_marker_nonversioned -test_delete_marker_suspended -test_delete_marker_versioned -test_delete_object_current_if_match -test_delete_object_current_if_match_last_modified_time -test_delete_object_current_if_match_size -test_delete_object_if_match -test_delete_object_if_match_last_modified_time -test_delete_object_if_match_size -test_delete_object_version_if_match -test_delete_object_version_if_match_last_modified_time -test_delete_object_version_if_match_size -test_delete_objects_current_if_match -test_delete_objects_current_if_match_last_modified_time -test_delete_objects_current_if_match_size -test_delete_objects_if_match -test_delete_objects_if_match_last_modified_time -test_delete_objects_if_match_size -test_delete_objects_version_if_match -test_delete_objects_version_if_match_last_modified_time -test_delete_objects_version_if_match_size -test_encryption_sse_c_multipart_invalid_chunks_1 -test_encryption_sse_c_multipart_invalid_chunks_2 -test_get_multipart_checksum_object_attributes -test_head_bucket_usage -test_lifecycle_cloud_multiple_transition -test_lifecycle_cloud_transition -test_lifecycle_cloud_transition_large_obj -test_lifecycle_deletemarker_expiration -test_lifecycle_deletemarker_expiration_with_days_tag -test_lifecycle_expiration -test_lifecycle_expiration_days0 -test_lifecycle_expiration_newer_noncurrent -test_lifecycle_expiration_noncur_tags1 -test_lifecycle_expiration_size_gt -test_lifecycle_expiration_size_lt -test_lifecycle_expiration_tags2 -test_lifecycle_expiration_versioned_tags2 -test_lifecycle_multipart_expiration -test_lifecycle_noncur_cloud_transition -test_lifecycle_noncur_expiration -test_lifecycle_noncur_transition -test_lifecycle_transition -test_lifecycle_transition_single_rule_multi_trans -test_lifecyclev2_expiration -test_list_buckets_anonymous -test_list_buckets_paginated -test_list_multipart_upload -test_list_multipart_upload_owner -test_multipart_get_part -test_multipart_single_get_part -test_multipart_sse_c_get_part -test_multipart_upload -test_multipart_upload_small -test_multipart_use_cksum_helper_crc32 -test_multipart_use_cksum_helper_crc32c -test_multipart_use_cksum_helper_crc64nvme -test_multipart_use_cksum_helper_sha1 -test_multipart_use_cksum_helper_sha256 -test_non_multipart_get_part -test_non_multipart_sse_c_get_part -test_object_header_acl_grants -test_object_raw_get_x_amz_expires_out_max_range -test_object_raw_get_x_amz_expires_out_positive_range -test_object_raw_put_authenticated_expired -test_object_read_unreadable -test_object_requestid_matches_header_on_error -test_object_set_get_unicode_metadata -test_object_write_with_chunked_transfer_encoding -test_post_object_invalid_date_format -test_post_object_invalid_request_field_value -test_post_object_missing_policy_condition -test_post_object_request_missing_policy_specified_field -test_post_object_set_key_from_filename -test_post_object_success_redirect_action -test_post_object_tags_anonymous_request -test_post_object_wrong_bucket -test_put_bucket_logging_account_j -test_put_bucket_logging_account_s -test_put_bucket_logging_extensions -test_put_bucket_logging_policy_wildcard_objects -test_put_bucket_logging_tenant_j -test_put_bucket_logging_tenant_s -test_put_bucket_ownership_bucket_owner_enforced -test_put_bucket_ownership_bucket_owner_preferred -test_put_bucket_ownership_object_writer -test_put_object_current_if_match -test_read_through -test_restore_noncur_obj -test_restore_object_permanent -test_restore_object_temporary -test_sse_kms_post_object_authenticated_request -test_versioned_object_acl_no_version_specified -test_versioning_multi_object_delete_with_marker_create -test_versioning_stack_delete_merkers diff --git a/scripts/s3-tests/report_compat.py b/scripts/s3-tests/report_compat.py new file mode 100644 index 000000000..90263302c --- /dev/null +++ b/scripts/s3-tests/report_compat.py @@ -0,0 +1,196 @@ +#!/usr/bin/env python3 +# Copyright 2024 RustFS Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Diff a ceph/s3-tests junit.xml result against the RustFS test classification lists. + +Classifies every executed test into: + - regressions: listed in implemented_tests.txt but failed/errored + - promotion candidates: passed but listed in unimplemented_tests.txt / excluded_tests.txt + - unclassified passes: passed but not present in any list (new upstream tests) + - unclassified failures: failed and not present in any list (new upstream tests) + +Writes a markdown report and prints a summary to stdout. Exit code is 0 unless +--fail-on-regression is given and at least one regression was found. +""" + +from __future__ import annotations + +import argparse +import pathlib +import sys +import xml.etree.ElementTree as ET + +LIST_FILES = { + "implemented": "implemented_tests.txt", + "unimplemented": "unimplemented_tests.txt", + "excluded": "excluded_tests.txt", +} + + +def load_list(path: pathlib.Path) -> set[str]: + names: set[str] = set() + if not path.is_file(): + return names + for line in path.read_text(encoding="utf-8").splitlines(): + line = line.strip() + if line and not line.startswith("#"): + names.add(line) + return names + + +def base_name(testcase_name: str) -> str: + """Strip pytest parametrization (test_foo[param]) to match list entries.""" + return testcase_name.split("[", 1)[0] + + +def parse_junit(path: pathlib.Path) -> dict[str, str]: + """Return {test name: status} with status in passed/failed/error/skipped. + + Parametrized cases collapse onto their base name; any failing variant marks + the whole test failed. + """ + results: dict[str, str] = {} + severity = {"skipped": 0, "passed": 1, "failed": 2, "error": 2} + root = ET.parse(path).getroot() + for case in root.iter("testcase"): + name = base_name(case.get("name", "")) + if not name: + continue + if case.find("failure") is not None: + status = "failed" + elif case.find("error") is not None: + status = "error" + elif case.find("skipped") is not None: + status = "skipped" + else: + status = "passed" + prev = results.get(name) + if prev is None or severity[status] > severity[prev]: + results[name] = status + return results + + +def render_section(title: str, rows: list[str], hint: str = "") -> list[str]: + lines = [f"## {title} ({len(rows)})", ""] + if hint: + lines += [hint, ""] + if rows: + lines += [f"- `{name}`" for name in sorted(rows)] + else: + lines.append("_none_") + lines.append("") + return lines + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--junit", required=True, type=pathlib.Path, help="junit.xml produced by pytest") + parser.add_argument( + "--lists-dir", + type=pathlib.Path, + default=pathlib.Path(__file__).resolve().parent, + help="directory containing the *_tests.txt classification files", + ) + parser.add_argument("--output", type=pathlib.Path, help="write the markdown report to this path") + parser.add_argument( + "--fail-on-regression", + action="store_true", + help="exit non-zero when a test from implemented_tests.txt failed", + ) + args = parser.parse_args() + + if not args.junit.is_file(): + print(f"[ERROR] junit file not found: {args.junit}", file=sys.stderr) + return 2 + + lists = {key: load_list(args.lists_dir / fname) for key, fname in LIST_FILES.items()} + results = parse_junit(args.junit) + + regressions: list[str] = [] + promotions: dict[str, list[str]] = {"unimplemented": [], "excluded": []} + unclassified_passed: list[str] = [] + unclassified_failed: list[str] = [] + counts = {"passed": 0, "failed": 0, "error": 0, "skipped": 0} + + for name, status in results.items(): + counts[status] += 1 + if status in ("failed", "error"): + if name in lists["implemented"]: + regressions.append(name) + elif name not in lists["unimplemented"] and name not in lists["excluded"]: + unclassified_failed.append(name) + elif status == "passed": + if name in lists["unimplemented"]: + promotions["unimplemented"].append(name) + elif name in lists["excluded"]: + promotions["excluded"].append(name) + elif name not in lists["implemented"]: + unclassified_passed.append(name) + + lines = [ + "# S3 compatibility report", + "", + f"Executed: {len(results)} tests — " + f"{counts['passed']} passed, {counts['failed']} failed, " + f"{counts['error']} errored, {counts['skipped']} skipped.", + "", + ] + lines += render_section( + "Regressions", + regressions, + "Listed in `implemented_tests.txt` but failing — these gate PRs and must be fixed.", + ) + lines += render_section( + "Promotion candidates (from unimplemented_tests.txt)", + promotions["unimplemented"], + "Now passing — move to `implemented_tests.txt` to lock in the coverage.", + ) + lines += render_section( + "Promotion candidates (from excluded_tests.txt)", + promotions["excluded"], + "Passing despite being excluded — re-evaluate the exclusion.", + ) + lines += render_section( + "Unclassified passes", + unclassified_passed, + "Passing but absent from every list — add to `implemented_tests.txt`.", + ) + lines += render_section( + "Unclassified failures", + unclassified_failed, + "Failing and absent from every list — triage into `unimplemented_tests.txt` or `excluded_tests.txt`.", + ) + + report = "\n".join(lines) + if args.output: + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(report, encoding="utf-8") + print(f"[INFO] Report written to {args.output}") + + print( + f"[INFO] {len(regressions)} regression(s), " + f"{len(promotions['unimplemented']) + len(promotions['excluded']) + len(unclassified_passed)} promotion candidate(s), " + f"{len(unclassified_failed)} unclassified failure(s)" + ) + for name in sorted(regressions): + print(f"[REGRESSION] {name}") + + if args.fail_on_regression and regressions: + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/s3-tests/run.sh b/scripts/s3-tests/run.sh index 4dd9c4511..76d1cfbce 100755 --- a/scripts/s3-tests/run.sh +++ b/scripts/s3-tests/run.sh @@ -43,6 +43,20 @@ S3_PORT="${S3_PORT:-9000}" TEST_MODE="${TEST_MODE:-single}" MAXFAIL="${MAXFAIL:-1}" XDIST="${XDIST:-0}" +# Test scope: +# "implemented" (default) - run only the implemented_tests.txt whitelist (PR gate) +# "all" - run the entire upstream suite (scheduled full sweep) +TEST_SCOPE="${TEST_SCOPE:-implemented}" +if [[ "${TEST_SCOPE}" != "implemented" && "${TEST_SCOPE}" != "all" ]]; then + echo "[ERROR] Invalid TEST_SCOPE: ${TEST_SCOPE} (must be \"implemented\" or \"all\")" >&2 + exit 1 +fi + +# Upstream ceph/s3-tests suite, pinned for reproducible runs. +# Bump S3TESTS_REV deliberately: upstream changes can rename tests or change +# assertions, so a bump usually requires reclassifying the test list files. +S3TESTS_REPO="${S3TESTS_REPO:-https://github.com/ceph/s3-tests.git}" +S3TESTS_REV="${S3TESTS_REV:-5522d1c351f75bc00ae0f64f742f3f095f5939d9}" # Compatibility default for the s3-tests harness: # this script provisions multiple local export directories on the same physical disk. @@ -146,7 +160,12 @@ fi # 2. Easy maintenance - edit txt files to add/remove tests # 3. Separation of concerns - test classification vs test execution # ============================================================================= -if [[ -z "${TESTEXPR:-}" ]]; then +TESTEXPR="${TESTEXPR:-}" +if [[ -n "${TESTEXPR}" ]]; then + log_info "Using custom TESTEXPR selection: ${TESTEXPR}" +elif [[ "${TEST_SCOPE}" == "all" ]]; then + log_info "TEST_SCOPE=all: running the entire upstream test suite (no whitelist)" +else if [[ -f "${IMPLEMENTED_TESTS_FILE}" ]]; then log_info "Loading test list from: ${IMPLEMENTED_TESTS_FILE}" load_testnodes_from_file "${IMPLEMENTED_TESTS_FILE}" @@ -240,8 +259,11 @@ Environment Variables: S3_ALT_ACCESS_KEY - Alt user access key (default: rustfsalt) S3_ALT_SECRET_KEY - Alt user secret key (default: rustfsalt) RUSTFS_SSE_S3_MASTER_KEY - Optional base64 32-byte key for local managed SSE fallback - MAXFAIL - Stop after N failures (default: 1) + MAXFAIL - Stop after N failures, 0 = never stop (default: 1) XDIST - Enable parallel execution with N workers (default: 0) + TEST_SCOPE - "implemented" (whitelist, default) or "all" (entire upstream suite) + S3TESTS_REPO - s3-tests repository URL (default: https://github.com/ceph/s3-tests.git) + S3TESTS_REV - Pinned s3-tests commit; bump deliberately and reclassify test lists MARKEXPR - pytest marker expression (default: no marker filtering) TESTEXPR - pytest -k expression (overrides implemented_tests.txt node list) S3TESTS_CONF_TEMPLATE - Path to s3tests config template (default: .github/s3tests/s3tests.conf) @@ -819,14 +841,23 @@ awscurl \ log_info "Alt user provisioned successfully" -# Step 8: Prepare s3-tests -log_info "Preparing s3-tests..." -if [ ! -d "${PROJECT_ROOT}/s3-tests" ]; then - git clone --depth 1 https://github.com/ceph/s3-tests.git "${PROJECT_ROOT}/s3-tests" || { - log_error "Failed to clone s3-tests" +# Step 8: Prepare s3-tests, pinned to S3TESTS_REV for reproducible runs +log_info "Preparing s3-tests at ${S3TESTS_REV}..." +if [ ! -d "${PROJECT_ROOT}/s3-tests/.git" ]; then + git init -q "${PROJECT_ROOT}/s3-tests" + git -C "${PROJECT_ROOT}/s3-tests" remote add origin "${S3TESTS_REPO}" +fi +if ! git -C "${PROJECT_ROOT}/s3-tests" cat-file -e "${S3TESTS_REV}^{commit}" 2>/dev/null; then + git -C "${PROJECT_ROOT}/s3-tests" fetch --depth 1 origin "${S3TESTS_REV}" || { + log_error "Failed to fetch s3-tests revision ${S3TESTS_REV} from ${S3TESTS_REPO}" exit 1 } fi +# -f discards local edits left by previous runs (e.g. pytest-xdist appended to requirements.txt) +git -C "${PROJECT_ROOT}/s3-tests" checkout -qf --detach "${S3TESTS_REV}" || { + log_error "Failed to checkout s3-tests revision ${S3TESTS_REV}" + exit 1 +} cd "${PROJECT_ROOT}/s3-tests" @@ -862,8 +893,11 @@ fi PYTEST_SELECTION_ARGS=() if [[ "${USE_FILE_TEST_NODES}" == "true" ]]; then PYTEST_SELECTION_ARGS=("${TEST_NODE_ARGS[@]}") -else +elif [[ -n "${TESTEXPR}" ]]; then PYTEST_SELECTION_ARGS=("${S3_TEST_FILE}" -k "${TESTEXPR}") +else + # TEST_SCOPE=all: the whole upstream suite + PYTEST_SELECTION_ARGS=("${S3_TEST_FILE}") fi # Run tests from s3tests/functional @@ -894,6 +928,16 @@ elif [ "${DEPLOY_MODE}" = "existing" ]; then echo "{\"host\": \"${S3_HOST}\", \"port\": ${S3_PORT}, \"mode\": \"existing\"}" > "${ARTIFACTS_DIR}/rustfs-${TEST_MODE}/inspect.json" || true fi +# Step 11: Classification report (informational, never fails the run) +REPORT_SCRIPT="${SCRIPT_DIR}/report_compat.py" +if [ -f "${REPORT_SCRIPT}" ] && [ -f "${ARTIFACTS_DIR}/junit.xml" ]; then + python3 "${REPORT_SCRIPT}" \ + --junit "${ARTIFACTS_DIR}/junit.xml" \ + --lists-dir "${SCRIPT_DIR}" \ + --output "${ARTIFACTS_DIR}/compat-report.md" \ + || log_warn "Compatibility report generation failed" +fi + # Summary if [ ${TEST_EXIT_CODE} -eq 0 ]; then log_info "Tests completed successfully!"