gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-26 08:18:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

ci(s3-tests): pin upstream suite, add weekly full sweep and compat report (#4204)

Browse Source 
Reworks the S3 compatibility test harness for reproducibility and faster
feedback:

- Pin ceph/s3-tests to a fixed commit (S3TESTS_REV, fetch-by-SHA) so the
  449-test PR gate is reproducible; previously every run cloned upstream
  master, letting test renames or assertion changes break CI silently.
- PR gate (ci.yml s3-implemented-tests): MAXFAIL=0 + XDIST=4 so a single
  CI round reports every failure in parallel instead of stopping at the
  first one serially.
- Add TEST_SCOPE=all to run.sh to run the entire upstream suite, and
  report_compat.py to diff junit results against the classification
  lists (regressions, promotion candidates, unclassified tests).
- Rewrite e2e-s3tests.yml: delegate execution to run.sh (single source
  of truth; also fixes the broken config generation that left S3_PORT
  empty), add a weekly scheduled full sweep that fails only on whitelist
  regressions, and fix the multi-node topology to a real distributed
  cluster (endpoint-style RUSTFS_VOLUMES) instead of four independent
  single-node stores behind a load balancer.
- Docs: rewrite stale .github/s3tests/README.md (marker-era strategy),
  update scripts/s3-tests/README.md, fix dead build_testexpr.sh
  reference in S3_COMPAT_WORKFLOW.md, drop legacy non_standard_tests.txt.

All 747 classified test names verified present at the pinned revision;
13 upstream tests are currently unclassified and will surface in the
first full-sweep report.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Zhengchao An
2026-07-02 23:43:19 +08:00
committed by GitHub
parent 31c026dd4f
commit 3eeb459ece
8 changed files with 472 additions and 639 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/s3tests/README.md
+51 -73
View File
@@ -1,103 +1,81 @@
# S3 Compatibility Tests Configuration
This directory contains the configuration for running [Ceph S3 compatibility tests](https://github.com/ceph/s3-tests) against RustFS.
This directory contains the configuration template for running
[Ceph S3 compatibility tests](https://github.com/ceph/s3-tests) against RustFS.
The test runner itself lives in [`scripts/s3-tests/`](../../scripts/s3-tests/)
— see its README for how tests are selected, executed, and reported. This
directory only holds the shared config template.
## Configuration File
The `s3tests.conf` file is based on the official `s3tests.conf.SAMPLE` from the ceph/s3-tests repository. It uses environment variable substitution via `envsubst` to configure the endpoint and credentials.
`s3tests.conf` is based on the official `s3tests.conf.SAMPLE` from the
ceph/s3-tests repository. It is a template: `scripts/s3-tests/run.sh` renders
it with `envsubst` before every run.
### Key Configuration Points
Required environment variables (all have defaults in `run.sh`):
- **Host**: Set via `${S3_HOST}` environment variable (e.g., `rustfs-single` for single-node, `lb` for multi-node)
- **Port**: 9000 (standard RustFS port)
- **Credentials**: Uses `${S3_ACCESS_KEY}` and `${S3_SECRET_KEY}` from workflow environment
- **TLS**: Disabled (`is_secure = False`)
| Variable | Meaning |
|----------|---------|
| `S3_HOST` | RustFS endpoint host |
| `S3_PORT` | RustFS endpoint port |
| `S3_ACCESS_KEY` / `S3_SECRET_KEY` | main user credentials |
| `S3_ALT_ACCESS_KEY` / `S3_ALT_SECRET_KEY` | alt user credentials (must differ from main) |
## Test Execution Strategy
TLS is disabled (`is_secure = False`).
### Network Connectivity Fix
## Test Selection Strategy
Tests run inside a Docker container on the `rustfs-net` network, which allows them to resolve and connect to the RustFS container hostnames. This fixes the "Temporary failure in name resolution" error that occurred when tests ran on the GitHub runner host.
Tests are selected by exact pytest node ids from classification list files in
`scripts/s3-tests/` — **not** by pytest markers:
### Performance Optimizations
- `implemented_tests.txt` — expected to pass; run as the per-PR gate
(`ci.yml`, job `s3-implemented-tests`)
- `unimplemented_tests.txt` — standard S3 features not yet implemented
- `excluded_tests.txt` — intentionally excluded (vendor-specific behavior or
product decisions)
1. **Parallel Execution**: Uses `pytest-xdist` with `-n 4` to run tests in parallel across 4 workers
2. **Load Distribution**: Uses `--dist=loadgroup` to distribute test groups across workers
3. **Fail-Fast**: Uses `--maxfail=50` to stop after 50 failures, saving time on catastrophic failures
The upstream s3-tests repository is pinned to a fixed commit (`S3TESTS_REV` in
`run.sh`) so results are reproducible; bump it deliberately and reclassify the
lists when upstream changes.
### Feature Filtering
## Where Tests Run
Tests are filtered using pytest markers (`-m`) to skip features not yet supported by RustFS:
- `lifecycle` - Bucket lifecycle policies
- `versioning` - Object versioning
- `s3website` - Static website hosting
- `bucket_logging` - Bucket logging
- `encryption` / `sse_s3` - Server-side encryption
- `cloud_transition` / `cloud_restore` - Cloud storage transitions
- `lifecycle_expiration` / `lifecycle_transition` - Lifecycle operations
This filtering:
1. Reduces test execution time significantly (from 1+ hour to ~10-15 minutes)
2. Focuses on features RustFS currently supports
3. Avoids hundreds of expected failures
- **Per PR**: `.github/workflows/ci.yml` job `s3-implemented-tests` runs the
implemented whitelist against a single-node debug binary. Any failure blocks
the PR.
- **Weekly + manual**: `.github/workflows/e2e-s3tests.yml` runs the full
upstream suite (`TEST_SCOPE=all`) against a Docker deployment (single node
or a 4-node distributed cluster behind HAProxy). It fails only on
regressions in the implemented whitelist and publishes a classification
report (`compat-report.md`, also shown in the job summary) listing promotion
candidates and unclassified tests.
## Running Tests Locally
### Single-Node Test
```bash
# Set credentials
export S3_ACCESS_KEY=rustfsadmin
export S3_SECRET_KEY=rustfsadmin
# Whitelist run against a locally built binary (the same thing the PR gate does)
./scripts/s3-tests/run.sh
# Start RustFS container
docker run -d --name rustfs-single \
--network rustfs-net \
-e RUSTFS_ADDRESS=0.0.0.0:9000 \
-e RUSTFS_ACCESS_KEY=$S3_ACCESS_KEY \
-e RUSTFS_SECRET_KEY=$S3_SECRET_KEY \
-e RUSTFS_VOLUMES="/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3" \
rustfs-ci
# Full sweep, never stop on failures, 4 pytest workers
TEST_SCOPE=all MAXFAIL=0 XDIST=4 ./scripts/s3-tests/run.sh
# Generate config
export S3_HOST=rustfs-single
envsubst < .github/s3tests/s3tests.conf > /tmp/s3tests.conf
# Run tests
docker run --rm \
--network rustfs-net \
-v /tmp/s3tests.conf:/etc/s3tests.conf:ro \
python:3.12-slim \
bash -c '
apt-get update -qq && apt-get install -y -qq git
git clone --depth 1 https://github.com/ceph/s3-tests.git /s3-tests
cd /s3-tests
pip install -q -r requirements.txt pytest-xdist
S3TEST_CONF=/etc/s3tests.conf pytest -v -n 4 \
s3tests/functional/test_s3.py \
-m "not lifecycle and not versioning and not s3website and not bucket_logging and not encryption and not sse_s3"
'
# A specific test against an already-running server
DEPLOY_MODE=existing TESTEXPR="test_bucket_list_empty" ./scripts/s3-tests/run.sh
```
## Test Results Interpretation
- **PASSED**: Test succeeded, feature works correctly
- **FAILED**: Test failed, indicates a potential bug or incompatibility
- **ERROR**: Test setup failed (e.g., network issues, missing dependencies)
- **SKIPPED**: Test skipped due to marker filtering
Results land in `artifacts/s3tests-single/` (`junit.xml`, `pytest.log`,
`compat-report.md`).
## Adding New Feature Support
When adding support for a new S3 feature to RustFS:
1. Remove the corresponding marker from the filter in `.github/workflows/e2e-s3tests.yml`
2. Run the tests to verify compatibility
3. Fix any failing tests
4. Update this README to reflect the newly supported feature
Follow [`scripts/s3-tests/S3_COMPAT_WORKFLOW.md`](../../scripts/s3-tests/S3_COMPAT_WORKFLOW.md).
In short: fix the behavior, verify with `TESTEXPR=...`, then move the test
from `unimplemented_tests.txt` to `implemented_tests.txt` so the PR gate locks
it in.
## References
- [Ceph S3 Tests Repository](https://github.com/ceph/s3-tests)
- [S3 API Compatibility](https://docs.aws.amazon.com/AmazonS3/latest/API/)
- [pytest-xdist Documentation](https://pytest-xdist.readthedocs.io/)
- [RustFS test runner](../../scripts/s3-tests/README.md)
.github/workflows/ci.yml
+2 -1
View File
@@ -429,7 +429,8 @@ jobs:
DEPLOY_MODE=binary \
RUSTFS_BINARY=./target/debug/rustfs \
TEST_MODE=single \
MAXFAIL=1 \
MAXFAIL=0 \
XDIST=4 \
S3_PORT="${S3_PORT}" \
DATA_ROOT="${RUN_ROOT}" \
S3TESTS_CONF=artifacts/s3tests-single/s3tests.conf \
.github/workflows/e2e-s3tests.yml
+133 -284
View File
@@ -12,31 +12,59 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Full ceph/s3-tests sweep against RustFS.
#
# The per-PR compatibility gate lives in ci.yml (job s3-implemented-tests) and
# runs only the implemented_tests.txt whitelist. This workflow complements it:
#
# - Scheduled weekly full sweep (TEST_SCOPE=all): runs the ENTIRE upstream
# suite and reports promotion candidates (tests that newly pass) and
# unclassified tests. The job fails only on regressions in the implemented
# whitelist or on infrastructure errors — expected failures from
# not-yet-implemented features do not turn the run red.
# - Manual runs (workflow_dispatch): same, with configurable mode/scope.
#
# All test execution is delegated to scripts/s3-tests/run.sh (single source of
# truth, also used locally and by the PR gate). This workflow only provisions
# the server topology: a single node, or a real 4-node distributed cluster
# (erasure-coded across nodes) behind an HAProxy round-robin load balancer.
name: e2e-s3tests
on:
workflow_dispatch:
inputs:
test-mode:
description: "Test mode to run"
description: "Server topology"
required: true
type: choice
default: "single"
options:
- single
- multi
scope:
description: "Test scope: full upstream suite or implemented whitelist"
required: true
type: choice
default: "all"
options:
- all
- implemented
xdist:
description: "Enable pytest-xdist (parallel). '0' to disable."
description: "pytest-xdist workers. '0' to disable."
required: false
default: "4"
maxfail:
description: "Stop after N failures. '0' to run everything."
required: false
default: "0"
maxfail:
description: "Stop after N failures (debug friendly)"
required: false
default: "1"
markexpr:
description: "pytest -m expression (feature filters)"
description: "Optional pytest -m expression"
required: false
default: "not lifecycle and not versioning and not s3website and not bucket_logging and not encryption"
default: ""
schedule:
# Weekly full sweep (Sunday 02:00 UTC): full suite, single-node topology.
- cron: "0 2 * * 0"
env:
# main user
@@ -47,11 +75,20 @@ env:
S3_ALT_SECRET_KEY: rustfsalt
S3_REGION: us-east-1
S3_HOST: 127.0.0.1
S3_PORT: "9000"
RUST_LOG: info
PLATFORM: linux/amd64
BUILDX_CACHE_SCOPE: rustfs-e2e-s3tests-source
# Scheduled runs have no inputs; default to a single-node full sweep.
TEST_MODE: ${{ github.event.inputs.test-mode || 'single' }}
TEST_SCOPE: ${{ github.event.inputs.scope || 'all' }}
XDIST: ${{ github.event.inputs.xdist || '4' }}
MAXFAIL: ${{ github.event.inputs.maxfail || '0' }}
MARKEXPR: ${{ github.event.inputs.markexpr || '' }}
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event.inputs['test-mode'] || 'single' }}
cancel-in-progress: true
@@ -61,10 +98,9 @@ defaults:
shell: bash
jobs:
s3tests-single:
if: github.event.inputs['test-mode'] == 'single'
runs-on: ubicloud-standard-2
timeout-minutes: 120
s3tests:
runs-on: ubicloud-standard-4
timeout-minutes: 180
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
@@ -93,222 +129,61 @@ jobs:
-t rustfs-ci \
-f Dockerfile.source .
- name: Create network
run: docker network inspect rustfs-net >/dev/null 2>&1 || docker network create rustfs-net
- name: Remove existing rustfs-single (if any)
run: docker rm -f rustfs-single >/dev/null 2>&1 || true
- name: Start single RustFS
if: env.TEST_MODE == 'single'
run: |
docker network inspect rustfs-net >/dev/null 2>&1 || docker network create rustfs-net
docker rm -f rustfs-single >/dev/null 2>&1 || true
docker run -d --name rustfs-single \
--network rustfs-net \
-p 9000:9000 \
-p ${S3_PORT}:9000 \
-e RUSTFS_ADDRESS=0.0.0.0:9000 \
-e RUSTFS_ACCESS_KEY=$S3_ACCESS_KEY \
-e RUSTFS_SECRET_KEY=$S3_SECRET_KEY \
-e RUSTFS_VOLUMES="/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3" \
-e RUSTFS_ACCESS_KEY=${S3_ACCESS_KEY} \
-e RUSTFS_SECRET_KEY=${S3_SECRET_KEY} \
-e RUSTFS_VOLUMES="/data/rustfs{0...3}" \
-v /tmp/rustfs-single:/data \
rustfs-ci
- name: Wait for RustFS ready
- name: Start 4-node distributed cluster
if: env.TEST_MODE == 'multi'
run: |
for i in {1..60}; do
if curl -sf http://127.0.0.1:9000/health >/dev/null 2>&1; then
echo "RustFS is ready"
exit 0
fi
# A real distributed deployment: every node lists all endpoints in
# RUSTFS_VOLUMES so data is erasure-coded ACROSS nodes. Do not use
# node-local volume paths here — that would create four independent
# single-node stores behind the load balancer.
cat > compose.yml <<EOF
x-rustfs-node: &rustfs-node
image: rustfs-ci
networks: [rustfs-net]
environment:
RUSTFS_ADDRESS: "0.0.0.0:9000"
RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
if [ "$(docker inspect -f '{{.State.Running}}' rustfs-single 2>/dev/null)" != "true" ]; then
echo "RustFS container not running" >&2
docker logs rustfs-single || true
exit 1
fi
sleep 2
done
echo "Health check timed out" >&2
docker logs rustfs-single || true
exit 1
- name: Generate s3tests config
run: |
export S3_HOST=127.0.0.1
envsubst < .github/s3tests/s3tests.conf > s3tests.conf
- name: Provision s3-tests alt user (required by suite)
run: |
# Admin API requires AWS SigV4 signing. awscurl is used by RustFS codebase as well.
awscurl \
--service s3 \
--region "${S3_REGION}" \
--access_key "${S3_ACCESS_KEY}" \
--secret_key "${S3_SECRET_KEY}" \
-X PUT \
-H 'Content-Type: application/json' \
-d '{"secretKey":"'"${S3_ALT_SECRET_KEY}"'","status":"enabled","policy":"readwrite"}' \
"http://127.0.0.1:9000/rustfs/admin/v3/add-user?accessKey=${S3_ALT_ACCESS_KEY}"
# Explicitly attach built-in policy via policy mapping.
# s3-tests relies on alt client being able to ListBuckets during setup cleanup.
awscurl \
--service s3 \
--region "${S3_REGION}" \
--access_key "${S3_ACCESS_KEY}" \
--secret_key "${S3_SECRET_KEY}" \
-X PUT \
"http://127.0.0.1:9000/rustfs/admin/v3/set-user-or-group-policy?policyName=readwrite&userOrGroup=${S3_ALT_ACCESS_KEY}&isGroup=false"
# Sanity check: alt user can list buckets (should not be AccessDenied).
awscurl \
--service s3 \
--region "${S3_REGION}" \
--access_key "${S3_ALT_ACCESS_KEY}" \
--secret_key "${S3_ALT_SECRET_KEY}" \
-X GET \
"http://127.0.0.1:9000/" >/dev/null
- name: Prepare s3-tests
run: |
git clone --depth 1 https://github.com/ceph/s3-tests.git s3-tests
- name: Run ceph s3-tests (debug friendly)
run: |
export PATH="$HOME/.local/bin:$PATH"
mkdir -p artifacts/s3tests-single
cd s3-tests
set -o pipefail
MAXFAIL="${{ github.event.inputs.maxfail }}"
if [ -z "$MAXFAIL" ]; then MAXFAIL="1"; fi
MARKEXPR="${{ github.event.inputs.markexpr }}"
if [ -z "$MARKEXPR" ]; then MARKEXPR="not lifecycle and not versioning and not s3website and not bucket_logging and not encryption"; fi
XDIST="${{ github.event.inputs.xdist }}"
if [ -z "$XDIST" ]; then XDIST="0"; fi
XDIST_ARGS=""
if [ "$XDIST" != "0" ]; then
# Add pytest-xdist to requirements.txt so tox installs it inside
# its virtualenv. Installing outside tox does NOT work.
echo "pytest-xdist" >> requirements.txt
XDIST_ARGS="-n $XDIST --dist=loadgroup"
fi
# Run tests from s3tests/functional (boto2+boto3 combined directory).
S3TEST_CONF=${GITHUB_WORKSPACE}/s3tests.conf \
tox -- \
-vv -ra --showlocals --tb=long \
--maxfail="$MAXFAIL" \
--junitxml=${GITHUB_WORKSPACE}/artifacts/s3tests-single/junit.xml \
$XDIST_ARGS \
s3tests/functional/test_s3.py \
-m "$MARKEXPR" \
2>&1 | tee ${GITHUB_WORKSPACE}/artifacts/s3tests-single/pytest.log
- name: Collect RustFS logs
if: always()
run: |
mkdir -p artifacts/rustfs-single
docker logs rustfs-single > artifacts/rustfs-single/rustfs.log 2>&1 || true
docker inspect rustfs-single > artifacts/rustfs-single/inspect.json || true
- name: Upload artifacts
if: always() && env.ACT != 'true'
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: s3tests-single
path: artifacts/**
s3tests-multi:
if: github.event_name == 'workflow_dispatch' && github.event.inputs['test-mode'] == 'multi'
runs-on: ubicloud-standard-2
timeout-minutes: 150
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Cache pip downloads
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-e2e-s3tests-${{ hashFiles('.github/workflows/e2e-s3tests.yml') }}
restore-keys: |
${{ runner.os }}-pip-e2e-s3tests-
- name: Install Python tools
run: |
python3 -m pip install --user --upgrade pip awscurl tox
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: Enable buildx
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
- name: Build RustFS image (source, cached)
run: |
DOCKER_BUILDKIT=1 docker buildx build --load \
--platform ${PLATFORM} \
--cache-from type=gha,scope=${BUILDX_CACHE_SCOPE} \
--cache-to type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE} \
-t rustfs-ci \
-f Dockerfile.source .
- name: Prepare cluster compose
run: |
cat > compose.yml <<'EOF'
services:
rustfs1:
image: rustfs-ci
<<: *rustfs-node
hostname: rustfs1
networks: [rustfs-net]
environment:
RUSTFS_ADDRESS: "0.0.0.0:9000"
RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
volumes:
- rustfs1-data:/data
volumes: [rustfs1-data:/data]
rustfs2:
image: rustfs-ci
<<: *rustfs-node
hostname: rustfs2
networks: [rustfs-net]
environment:
RUSTFS_ADDRESS: "0.0.0.0:9000"
RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
volumes:
- rustfs2-data:/data
volumes: [rustfs2-data:/data]
rustfs3:
image: rustfs-ci
<<: *rustfs-node
hostname: rustfs3
networks: [rustfs-net]
environment:
RUSTFS_ADDRESS: "0.0.0.0:9000"
RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
volumes:
- rustfs3-data:/data
volumes: [rustfs3-data:/data]
rustfs4:
image: rustfs-ci
<<: *rustfs-node
hostname: rustfs4
networks: [rustfs-net]
environment:
RUSTFS_ADDRESS: "0.0.0.0:9000"
RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
volumes:
- rustfs4-data:/data
volumes: [rustfs4-data:/data]
lb:
image: haproxy:2.9
hostname: lb
networks: [rustfs-net]
ports:
- "9000:9000"
- "${S3_PORT}:9000"
volumes:
- ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
networks:
@@ -340,104 +215,78 @@ jobs:
server s4 rustfs4:9000 check
EOF
- name: Launch cluster
run: docker compose -f compose.yml up -d
docker compose -f compose.yml up -d
- name: Wait for LB ready
- name: Wait for RustFS ready
run: |
for i in {1..90}; do
if curl -sf http://127.0.0.1:9000/health >/dev/null 2>&1; then
echo "Load balancer is ready"
for i in {1..120}; do
if curl -sf "http://${S3_HOST}:${S3_PORT}/health" >/dev/null 2>&1; then
echo "RustFS is ready"
exit 0
fi
sleep 2
done
echo "LB or backend not ready" >&2
docker compose -f compose.yml logs --tail=200 || true
echo "Health check timed out" >&2
if [ "${TEST_MODE}" = "single" ]; then
docker logs rustfs-single || true
else
docker compose -f compose.yml logs --tail=200 || true
fi
exit 1
- name: Generate s3tests config
- name: Run ceph s3-tests
run: |
export S3_HOST=127.0.0.1
envsubst < .github/s3tests/s3tests.conf > s3tests.conf
set +e
DEPLOY_MODE=existing \
TEST_MODE="${TEST_MODE}" \
TEST_SCOPE="${TEST_SCOPE}" \
XDIST="${XDIST}" \
MAXFAIL="${MAXFAIL}" \
MARKEXPR="${MARKEXPR}" \
./scripts/s3-tests/run.sh
RC=$?
set -e
- name: Provision s3-tests alt user (required by suite)
run: |
awscurl \
--service s3 \
--region "${S3_REGION}" \
--access_key "${S3_ACCESS_KEY}" \
--secret_key "${S3_SECRET_KEY}" \
-X PUT \
-H 'Content-Type: application/json' \
-d '{"secretKey":"'"${S3_ALT_SECRET_KEY}"'","status":"enabled","policy":"readwrite"}' \
"http://127.0.0.1:9000/rustfs/admin/v3/add-user?accessKey=${S3_ALT_ACCESS_KEY}"
awscurl \
--service s3 \
--region "${S3_REGION}" \
--access_key "${S3_ACCESS_KEY}" \
--secret_key "${S3_SECRET_KEY}" \
-X PUT \
"http://127.0.0.1:9000/rustfs/admin/v3/set-user-or-group-policy?policyName=readwrite&userOrGroup=${S3_ALT_ACCESS_KEY}&isGroup=false"
awscurl \
--service s3 \
--region "${S3_REGION}" \
--access_key "${S3_ALT_ACCESS_KEY}" \
--secret_key "${S3_ALT_SECRET_KEY}" \
-X GET \
"http://127.0.0.1:9000/" >/dev/null
- name: Prepare s3-tests
run: |
git clone --depth 1 https://github.com/ceph/s3-tests.git s3-tests
- name: Run ceph s3-tests (multi, debug friendly)
run: |
export PATH="$HOME/.local/bin:$PATH"
mkdir -p artifacts/s3tests-multi
cd s3-tests
set -o pipefail
MAXFAIL="${{ github.event.inputs.maxfail }}"
if [ -z "$MAXFAIL" ]; then MAXFAIL="1"; fi
MARKEXPR="${{ github.event.inputs.markexpr }}"
if [ -z "$MARKEXPR" ]; then MARKEXPR="not lifecycle and not versioning and not s3website and not bucket_logging and not encryption"; fi
XDIST="${{ github.event.inputs.xdist }}"
if [ -z "$XDIST" ]; then XDIST="0"; fi
XDIST_ARGS=""
if [ "$XDIST" != "0" ]; then
# Add pytest-xdist to requirements.txt so tox installs it inside
# its virtualenv. Installing outside tox does NOT work.
echo "pytest-xdist" >> requirements.txt
XDIST_ARGS="-n $XDIST --dist=loadgroup"
if [ "${TEST_SCOPE}" = "implemented" ]; then
# Whitelist run: every failure is a regression.
exit "${RC}"
fi
# Run tests from s3tests/functional (boto2+boto3 combined directory).
S3TEST_CONF=${GITHUB_WORKSPACE}/s3tests.conf \
tox -- \
-vv -ra --showlocals --tb=long \
--maxfail="$MAXFAIL" \
--junitxml=${GITHUB_WORKSPACE}/artifacts/s3tests-multi/junit.xml \
$XDIST_ARGS \
s3tests/functional/test_s3.py \
-m "$MARKEXPR" \
2>&1 | tee ${GITHUB_WORKSPACE}/artifacts/s3tests-multi/pytest.log
# Full sweep: failures outside the implemented whitelist are
# inventory (promotion candidates / unimplemented features), not a
# gate. Fail only on whitelist regressions or infrastructure errors.
JUNIT="artifacts/s3tests-${TEST_MODE}/junit.xml"
if [ ! -f "${JUNIT}" ]; then
echo "No junit.xml produced — infrastructure failure (exit ${RC})" >&2
exit "${RC}"
fi
python3 scripts/s3-tests/report_compat.py \
--junit "${JUNIT}" \
--lists-dir scripts/s3-tests \
--fail-on-regression
- name: Collect logs
- name: Publish compatibility report
if: always()
run: |
mkdir -p artifacts/cluster
docker compose -f compose.yml logs --no-color > artifacts/cluster/cluster.log 2>&1 || true
REPORT="artifacts/s3tests-${TEST_MODE}/compat-report.md"
if [ -f "${REPORT}" ]; then
cat "${REPORT}" >> "$GITHUB_STEP_SUMMARY"
fi
- name: Collect RustFS logs
if: always()
run: |
mkdir -p "artifacts/rustfs-${TEST_MODE}"
if [ "${TEST_MODE}" = "single" ]; then
docker logs rustfs-single > "artifacts/rustfs-${TEST_MODE}/rustfs.log" 2>&1 || true
docker inspect rustfs-single > "artifacts/rustfs-${TEST_MODE}/inspect.json" || true
else
docker compose -f compose.yml logs --no-color > "artifacts/rustfs-${TEST_MODE}/cluster.log" 2>&1 || true
fi
- name: Upload artifacts
if: always() && env.ACT != 'true'
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: s3tests-multi
name: s3tests-${{ env.TEST_MODE }}
path: artifacts/**
scripts/s3-tests/README.md
+35 -29
View File
@@ -22,7 +22,7 @@ The script will automatically:
4. **Prepare environment**:
- Generate s3tests configuration from template
- Provision alt user via admin API
- Clone s3-tests repository if missing
- Fetch s3-tests at the pinned revision (`S3TESTS_REV`)
- Install missing dependencies (awscurl, tox, gettext)
5. **Run tests**: Execute ceph s3-tests via tox with configured filters
6. **Collect results**: Save logs and test results in `artifacts/s3tests-${TEST_MODE}/`
@@ -48,7 +48,7 @@ DEPLOY_MODE=build ./scripts/s3-tests/run.sh
- Automatically compiles RustFS binary if it doesn't exist
- If binary exists and was compiled less than **30 minutes** ago, compilation is skipped (unless `--no-cache` is specified)
- Automatically starts the service after compilation
- Automatically clones s3-tests repository if missing
- Automatically fetches s3-tests at the pinned revision if missing
- Automatically installs missing dependencies (awscurl, tox, gettext)
- **Automatic Cleanup**: Process is automatically stopped when script exits
@@ -67,7 +67,7 @@ DEPLOY_MODE=binary RUSTFS_BINARY=./target/release/rustfs ./scripts/s3-tests/run.
**Behavior**:
- Uses existing binary file (must exist, script will not compile)
- Automatically starts the service using the specified binary
- Automatically clones s3-tests repository if missing
- Automatically fetches s3-tests at the pinned revision if missing
- Automatically installs missing dependencies (awscurl, tox, gettext)
- **Automatic Cleanup**: Process is automatically stopped when script exits
@@ -82,7 +82,7 @@ DEPLOY_MODE=docker ./scripts/s3-tests/run.sh
**Behavior**:
- Automatically builds Docker image using `Dockerfile.source`
- Creates Docker network (`rustfs-net`) if it doesn't exist
- Automatically clones s3-tests repository if missing
- Automatically fetches s3-tests at the pinned revision if missing
- Automatically installs missing dependencies (awscurl, tox, gettext)
- **Automatic Cleanup**: Container and network are automatically removed when script exits
@@ -100,7 +100,7 @@ DEPLOY_MODE=existing S3_HOST=192.168.1.100 S3_PORT=9000 ./scripts/s3-tests/run.s
**Behavior**:
- Skips service startup and port availability checks
- Connects directly to the specified service endpoint
- Automatically clones s3-tests repository if missing
- Automatically fetches s3-tests at the pinned revision if missing
- Automatically installs missing dependencies (awscurl, tox, gettext)
- **Note**: The service must already have the alt user (`rustfsalt`) provisioned, or the script will provision it automatically
@@ -141,8 +141,15 @@ DEPLOY_MODE=existing S3_HOST=192.168.1.100 S3_PORT=9000 ./scripts/s3-tests/run.s
### Test Parameters
- `TEST_MODE`: Test mode (default: `single`)
- `MAXFAIL`: Stop after N failures (default: `1`)
- `TEST_SCOPE`: Test scope (default: `implemented`)
- `implemented`: run only the `implemented_tests.txt` whitelist (the PR gate)
- `all`: run the entire upstream suite (used by the weekly full sweep)
- `MAXFAIL`: Stop after N failures, `0` = never stop (default: `1`)
- `XDIST`: Enable parallel execution with N workers (default: `0`, disabled)
- `S3TESTS_REPO`: s3-tests repository URL (default: `https://github.com/ceph/s3-tests.git`)
- `S3TESTS_REV`: Pinned s3-tests commit for reproducible runs
- Bump deliberately: upstream changes can rename tests or change assertions,
so a bump usually requires reclassifying the test list files
- `MARKEXPR`: pytest marker expression for filtering tests
- Default: no marker filtering; file-based test lists control the selected tests
- Can be customized to test specific marker groups
@@ -246,6 +253,10 @@ Test results are saved in the `artifacts/s3tests-${TEST_MODE}/` directory (defau
- `junit.xml`: Test results in JUnit format (compatible with CI/CD systems)
- `pytest.log`: Detailed pytest logs with full test output
- `compat-report.md`: Classification report generated by `report_compat.py` —
regressions against `implemented_tests.txt`, promotion candidates (tests
that pass but are still listed as unimplemented/excluded), and tests missing
from every list
- `rustfs-${TEST_MODE}/rustfs.log`: RustFS service logs
- `rustfs-${TEST_MODE}/inspect.json`: Service metadata (PID, binary path, mode, etc.)
@@ -316,8 +327,9 @@ The script will automatically install the following dependencies if missing (no
- Linux: Automatically installs via `sudo apt-get install gettext-base`
- **Note**: macOS installation may require manual intervention if brew fails
- **s3-tests repository**: Automatically cloned if not present
- Source: `https://github.com/ceph/s3-tests.git`
- **s3-tests repository**: Automatically fetched if not present
- Source: `https://github.com/ceph/s3-tests.git`, pinned to the commit in
`S3TESTS_REV` (see run.sh) for reproducible runs
- Location: `${PROJECT_ROOT}/s3-tests`
**Note**: The script adds `$HOME/.local/bin` to `PATH` automatically, so auto-installed Python tools are accessible.
@@ -427,29 +439,23 @@ awscurl --service s3 --region us-east-1 \
## Workflow Integration
This script mirrors the GitHub Actions workflow defined in `.github/workflows/e2e-s3tests.yml`.
This script is the single source of truth for running ceph s3-tests against
RustFS. Two GitHub Actions workflows delegate to it:
The script follows the same steps:
- **PR gate** (`.github/workflows/ci.yml`, job `s3-implemented-tests`): runs
the `implemented_tests.txt` whitelist against a single-node debug binary on
every pull request (`DEPLOY_MODE=binary`, `MAXFAIL=0`, `XDIST=4`). Any
failure blocks the PR.
- **Full sweep** (`.github/workflows/e2e-s3tests.yml`): weekly scheduled (and
manually dispatchable) run of the ENTIRE upstream suite (`TEST_SCOPE=all`)
against a Docker deployment — single node or a real 4-node distributed
cluster behind HAProxy. The sweep fails only on regressions in the
implemented whitelist; everything else is reported by `report_compat.py`
as promotion candidates or unclassified tests.
1. Check port availability (skip for existing mode)
2. Build/start RustFS service (varies by deployment mode)
3. Wait for service to be fully ready:
- Check process/container status
- Check port is listening
- Wait for "server started successfully" log message
- Verify S3 API is responding
4. Generate s3tests configuration from template
5. Provision alt user for s3-tests via admin API
6. Run ceph s3-tests with tox
7. Collect logs and results
### Key Improvements Over Workflow
- **Smart compilation**: Skips rebuild if binary is recent (< 30 minutes)
- **Better health checks**: Log-based readiness detection instead of blind waiting
- **Port conflict detection**: Prevents conflicts before starting service
- **Proxy handling**: Automatically disables proxy for localhost
- **Configurable paths**: All paths (data, configs, artifacts) can be customized
Keeping both workflows on this script means local runs, the PR gate, and the
scheduled sweep always execute tests the same way (same pinned s3-tests
revision, same config template, same user provisioning).
## See Also
scripts/s3-tests/S3_COMPAT_WORKFLOW.md
+3 -8
View File
@@ -21,19 +21,14 @@ git merge upstream/main
### Step 2: Select candidate tests
Pick ~20 tests from `unimplemented_tests.txt` and write them to `selected_tests.txt`:
```bash
TESTEXPR=$(scripts/s3-tests/build_testexpr.sh selected_tests.txt) \
DEPLOY_MODE=build MAXFAIL=0 ./scripts/s3-tests/run.sh
```
Or run them directly:
Pick candidate tests from `unimplemented_tests.txt` and run them via a pytest `-k` expression:
```bash
TESTEXPR="test_foo or test_bar" DEPLOY_MODE=build MAXFAIL=0 ./scripts/s3-tests/run.sh
```
Alternatively, check the latest weekly full-sweep report (the `e2e-s3tests` workflow's job summary, or `artifacts/s3tests-single/compat-report.md`) — its "promotion candidates" section lists tests that already pass and only need reclassification.
Review `artifacts/s3tests-single/pytest.log` for results.
### Step 3: Triage failures
scripts/s3-tests/non_standard_tests.txt
-236
View File
@@ -1,236 +0,0 @@
# Legacy non-standard S3 test list (compatibility fallback)
# =========================================================
#
# This file is kept for backward compatibility.
# Active exclusion classification now uses excluded_tests.txt.
# These tests use vendor-specific extensions not part of AWS S3 API.
#
# Exclusion reasons:
# - fails_on_aws marker: Ceph-specific features
# - X-RGW-* headers: Ceph proprietary headers
# - allowUnordered: Ceph-specific query parameter
# - Bucket Logging: Ceph-specific logging extensions
# - Lifecycle: Ceph-specific lifecycle expiration behavior
# - SSE-KMS: Ceph-specific KMS extensions
# - Error format differences: Minor response format variations
test_account_usage
test_atomic_conditional_write_1mb
test_bucket_get_location
test_bucket_head_extended
test_bucket_header_acl_grants
test_bucket_list_return_data
test_bucket_list_return_data_versioning
test_bucket_logging_bucket_acl_required
test_bucket_logging_bucket_auth_type
test_bucket_logging_cleanup_bucket_concurrent_deletion_j
test_bucket_logging_cleanup_bucket_concurrent_deletion_j_single
test_bucket_logging_cleanup_bucket_concurrent_deletion_s
test_bucket_logging_cleanup_bucket_concurrent_deletion_s_single
test_bucket_logging_cleanup_bucket_deletion_j
test_bucket_logging_cleanup_bucket_deletion_j_single
test_bucket_logging_cleanup_bucket_deletion_s
test_bucket_logging_cleanup_bucket_deletion_s_single
test_bucket_logging_cleanup_concurrent_disabling_j
test_bucket_logging_cleanup_concurrent_disabling_j_single
test_bucket_logging_cleanup_concurrent_disabling_s
test_bucket_logging_cleanup_concurrent_disabling_s_single
test_bucket_logging_cleanup_concurrent_updating_j
test_bucket_logging_cleanup_concurrent_updating_j_single
test_bucket_logging_cleanup_concurrent_updating_s
test_bucket_logging_cleanup_concurrent_updating_s_single
test_bucket_logging_cleanup_disabling_j
test_bucket_logging_cleanup_disabling_j_single
test_bucket_logging_cleanup_disabling_s
test_bucket_logging_cleanup_disabling_s_single
test_bucket_logging_cleanup_updating_j
test_bucket_logging_cleanup_updating_j_single
test_bucket_logging_cleanup_updating_s
test_bucket_logging_cleanup_updating_s_single
test_bucket_logging_concurrent_flush_j
test_bucket_logging_concurrent_flush_j_single
test_bucket_logging_concurrent_flush_s
test_bucket_logging_concurrent_flush_s_single
test_bucket_logging_conf_concurrent_updating_pfx_j
test_bucket_logging_conf_concurrent_updating_pfx_s
test_bucket_logging_conf_concurrent_updating_roll_j
test_bucket_logging_conf_concurrent_updating_roll_s
test_bucket_logging_conf_updating_pfx_j
test_bucket_logging_conf_updating_pfx_s
test_bucket_logging_conf_updating_roll_j
test_bucket_logging_conf_updating_roll_s
test_bucket_logging_copy_objects
test_bucket_logging_copy_objects_bucket
test_bucket_logging_copy_objects_bucket_versioned
test_bucket_logging_copy_objects_versioned
test_bucket_logging_delete_objects
test_bucket_logging_delete_objects_versioned
test_bucket_logging_event_type_j
test_bucket_logging_event_type_s
test_bucket_logging_flush_empty
test_bucket_logging_flush_j
test_bucket_logging_flush_j_single
test_bucket_logging_flush_s
test_bucket_logging_flush_s_single
test_bucket_logging_get_objects
test_bucket_logging_get_objects_versioned
test_bucket_logging_head_objects
test_bucket_logging_head_objects_versioned
test_bucket_logging_key_filter_j
test_bucket_logging_key_filter_s
test_bucket_logging_mpu_copy
test_bucket_logging_mpu_copy_versioned
test_bucket_logging_mpu_j
test_bucket_logging_mpu_s
test_bucket_logging_mpu_versioned_j
test_bucket_logging_mpu_versioned_s
test_bucket_logging_mtime
test_bucket_logging_multi_delete
test_bucket_logging_multi_delete_versioned
test_bucket_logging_multiple_prefixes
test_bucket_logging_notupdating_j
test_bucket_logging_notupdating_j_single
test_bucket_logging_notupdating_s
test_bucket_logging_notupdating_s_single
test_bucket_logging_object_acl_required
test_bucket_logging_object_meta
test_bucket_logging_part_cleanup_concurrent_deletion_j
test_bucket_logging_part_cleanup_concurrent_deletion_s
test_bucket_logging_part_cleanup_concurrent_disabling_j
test_bucket_logging_part_cleanup_concurrent_disabling_s
test_bucket_logging_part_cleanup_concurrent_updating_j
test_bucket_logging_part_cleanup_concurrent_updating_s
test_bucket_logging_part_cleanup_deletion_j
test_bucket_logging_part_cleanup_deletion_s
test_bucket_logging_part_cleanup_disabling_j
test_bucket_logging_part_cleanup_disabling_s
test_bucket_logging_part_cleanup_updating_j
test_bucket_logging_part_cleanup_updating_s
test_bucket_logging_partitioned_key
test_bucket_logging_permission_change_j
test_bucket_logging_permission_change_s
test_bucket_logging_put_and_flush
test_bucket_logging_put_concurrency
test_bucket_logging_put_objects
test_bucket_logging_put_objects_versioned
test_bucket_logging_roll_time
test_bucket_logging_simple_key
test_bucket_logging_single_prefix
test_bucket_logging_target_cleanup_j
test_bucket_logging_target_cleanup_j_single
test_bucket_logging_target_cleanup_s
test_bucket_logging_target_cleanup_s_single
test_bucket_policy_get_obj_acl_existing_tag
test_bucket_policy_get_obj_existing_tag
test_bucket_policy_get_obj_tagging_existing_tag
test_bucket_policy_put_obj_copy_source
test_bucket_policy_put_obj_copy_source_meta
test_bucket_policy_put_obj_request_obj_tag
test_bucket_policy_put_obj_tagging_existing_tag
test_bucket_policy_set_condition_operator_end_with_IfExists
test_bucket_policy_upload_part_copy
test_bucket_recreate_new_acl
test_bucket_recreate_overwrite_acl
test_cors_presigned_get_object_tenant_v2
test_cors_presigned_get_object_v2
test_cors_presigned_put_object_tenant_v2
test_cors_presigned_put_object_v2
test_create_bucket_bucket_owner_enforced
test_create_bucket_bucket_owner_preferred
test_create_bucket_object_writer
test_delete_marker_expiration
test_delete_marker_nonversioned
test_delete_marker_suspended
test_delete_marker_versioned
test_delete_object_current_if_match
test_delete_object_current_if_match_last_modified_time
test_delete_object_current_if_match_size
test_delete_object_if_match
test_delete_object_if_match_last_modified_time
test_delete_object_if_match_size
test_delete_object_version_if_match
test_delete_object_version_if_match_last_modified_time
test_delete_object_version_if_match_size
test_delete_objects_current_if_match
test_delete_objects_current_if_match_last_modified_time
test_delete_objects_current_if_match_size
test_delete_objects_if_match
test_delete_objects_if_match_last_modified_time
test_delete_objects_if_match_size
test_delete_objects_version_if_match
test_delete_objects_version_if_match_last_modified_time
test_delete_objects_version_if_match_size
test_encryption_sse_c_multipart_invalid_chunks_1
test_encryption_sse_c_multipart_invalid_chunks_2
test_get_multipart_checksum_object_attributes
test_head_bucket_usage
test_lifecycle_cloud_multiple_transition
test_lifecycle_cloud_transition
test_lifecycle_cloud_transition_large_obj
test_lifecycle_deletemarker_expiration
test_lifecycle_deletemarker_expiration_with_days_tag
test_lifecycle_expiration
test_lifecycle_expiration_days0
test_lifecycle_expiration_newer_noncurrent
test_lifecycle_expiration_noncur_tags1
test_lifecycle_expiration_size_gt
test_lifecycle_expiration_size_lt
test_lifecycle_expiration_tags2
test_lifecycle_expiration_versioned_tags2
test_lifecycle_multipart_expiration
test_lifecycle_noncur_cloud_transition
test_lifecycle_noncur_expiration
test_lifecycle_noncur_transition
test_lifecycle_transition
test_lifecycle_transition_single_rule_multi_trans
test_lifecyclev2_expiration
test_list_buckets_anonymous
test_list_buckets_paginated
test_list_multipart_upload
test_list_multipart_upload_owner
test_multipart_get_part
test_multipart_single_get_part
test_multipart_sse_c_get_part
test_multipart_upload
test_multipart_upload_small
test_multipart_use_cksum_helper_crc32
test_multipart_use_cksum_helper_crc32c
test_multipart_use_cksum_helper_crc64nvme
test_multipart_use_cksum_helper_sha1
test_multipart_use_cksum_helper_sha256
test_non_multipart_get_part
test_non_multipart_sse_c_get_part
test_object_header_acl_grants
test_object_raw_get_x_amz_expires_out_max_range
test_object_raw_get_x_amz_expires_out_positive_range
test_object_raw_put_authenticated_expired
test_object_read_unreadable
test_object_requestid_matches_header_on_error
test_object_set_get_unicode_metadata
test_object_write_with_chunked_transfer_encoding
test_post_object_invalid_date_format
test_post_object_invalid_request_field_value
test_post_object_missing_policy_condition
test_post_object_request_missing_policy_specified_field
test_post_object_set_key_from_filename
test_post_object_success_redirect_action
test_post_object_tags_anonymous_request
test_post_object_wrong_bucket
test_put_bucket_logging_account_j
test_put_bucket_logging_account_s
test_put_bucket_logging_extensions
test_put_bucket_logging_policy_wildcard_objects
test_put_bucket_logging_tenant_j
test_put_bucket_logging_tenant_s
test_put_bucket_ownership_bucket_owner_enforced
test_put_bucket_ownership_bucket_owner_preferred
test_put_bucket_ownership_object_writer
test_put_object_current_if_match
test_read_through
test_restore_noncur_obj
test_restore_object_permanent
test_restore_object_temporary
test_sse_kms_post_object_authenticated_request
test_versioned_object_acl_no_version_specified
test_versioning_multi_object_delete_with_marker_create
test_versioning_stack_delete_merkers
scripts/s3-tests/report_compat.py
+196
View File
@@ -0,0 +1,196 @@
#!/usr/bin/env python3
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Diff a ceph/s3-tests junit.xml result against the RustFS test classification lists.
Classifies every executed test into:
- regressions: listed in implemented_tests.txt but failed/errored
- promotion candidates: passed but listed in unimplemented_tests.txt / excluded_tests.txt
- unclassified passes: passed but not present in any list (new upstream tests)
- unclassified failures: failed and not present in any list (new upstream tests)
Writes a markdown report and prints a summary to stdout. Exit code is 0 unless
--fail-on-regression is given and at least one regression was found.
"""
from __future__ import annotations
import argparse
import pathlib
import sys
import xml.etree.ElementTree as ET
LIST_FILES = {
"implemented": "implemented_tests.txt",
"unimplemented": "unimplemented_tests.txt",
"excluded": "excluded_tests.txt",
}
def load_list(path: pathlib.Path) -> set[str]:
names: set[str] = set()
if not path.is_file():
return names
for line in path.read_text(encoding="utf-8").splitlines():
line = line.strip()
if line and not line.startswith("#"):
names.add(line)
return names
def base_name(testcase_name: str) -> str:
"""Strip pytest parametrization (test_foo[param]) to match list entries."""
return testcase_name.split("[", 1)[0]
def parse_junit(path: pathlib.Path) -> dict[str, str]:
"""Return {test name: status} with status in passed/failed/error/skipped.
Parametrized cases collapse onto their base name; any failing variant marks
the whole test failed.
"""
results: dict[str, str] = {}
severity = {"skipped": 0, "passed": 1, "failed": 2, "error": 2}
root = ET.parse(path).getroot()
for case in root.iter("testcase"):
name = base_name(case.get("name", ""))
if not name:
continue
if case.find("failure") is not None:
status = "failed"
elif case.find("error") is not None:
status = "error"
elif case.find("skipped") is not None:
status = "skipped"
else:
status = "passed"
prev = results.get(name)
if prev is None or severity[status] > severity[prev]:
results[name] = status
return results
def render_section(title: str, rows: list[str], hint: str = "") -> list[str]:
lines = [f"## {title} ({len(rows)})", ""]
if hint:
lines += [hint, ""]
if rows:
lines += [f"- `{name}`" for name in sorted(rows)]
else:
lines.append("_none_")
lines.append("")
return lines
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--junit", required=True, type=pathlib.Path, help="junit.xml produced by pytest")
parser.add_argument(
"--lists-dir",
type=pathlib.Path,
default=pathlib.Path(__file__).resolve().parent,
help="directory containing the *_tests.txt classification files",
)
parser.add_argument("--output", type=pathlib.Path, help="write the markdown report to this path")
parser.add_argument(
"--fail-on-regression",
action="store_true",
help="exit non-zero when a test from implemented_tests.txt failed",
)
args = parser.parse_args()
if not args.junit.is_file():
print(f"[ERROR] junit file not found: {args.junit}", file=sys.stderr)
return 2
lists = {key: load_list(args.lists_dir / fname) for key, fname in LIST_FILES.items()}
results = parse_junit(args.junit)
regressions: list[str] = []
promotions: dict[str, list[str]] = {"unimplemented": [], "excluded": []}
unclassified_passed: list[str] = []
unclassified_failed: list[str] = []
counts = {"passed": 0, "failed": 0, "error": 0, "skipped": 0}
for name, status in results.items():
counts[status] += 1
if status in ("failed", "error"):
if name in lists["implemented"]:
regressions.append(name)
elif name not in lists["unimplemented"] and name not in lists["excluded"]:
unclassified_failed.append(name)
elif status == "passed":
if name in lists["unimplemented"]:
promotions["unimplemented"].append(name)
elif name in lists["excluded"]:
promotions["excluded"].append(name)
elif name not in lists["implemented"]:
unclassified_passed.append(name)
lines = [
"# S3 compatibility report",
"",
f"Executed: {len(results)} tests — "
f"{counts['passed']} passed, {counts['failed']} failed, "
f"{counts['error']} errored, {counts['skipped']} skipped.",
"",
]
lines += render_section(
"Regressions",
regressions,
"Listed in `implemented_tests.txt` but failing — these gate PRs and must be fixed.",
)
lines += render_section(
"Promotion candidates (from unimplemented_tests.txt)",
promotions["unimplemented"],
"Now passing — move to `implemented_tests.txt` to lock in the coverage.",
)
lines += render_section(
"Promotion candidates (from excluded_tests.txt)",
promotions["excluded"],
"Passing despite being excluded — re-evaluate the exclusion.",
)
lines += render_section(
"Unclassified passes",
unclassified_passed,
"Passing but absent from every list — add to `implemented_tests.txt`.",
)
lines += render_section(
"Unclassified failures",
unclassified_failed,
"Failing and absent from every list — triage into `unimplemented_tests.txt` or `excluded_tests.txt`.",
)
report = "\n".join(lines)
if args.output:
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(report, encoding="utf-8")
print(f"[INFO] Report written to {args.output}")
print(
f"[INFO] {len(regressions)} regression(s), "
f"{len(promotions['unimplemented']) + len(promotions['excluded']) + len(unclassified_passed)} promotion candidate(s), "
f"{len(unclassified_failed)} unclassified failure(s)"
)
for name in sorted(regressions):
print(f"[REGRESSION] {name}")
if args.fail_on_regression and regressions:
return 1
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/s3-tests/run.sh
+52 -8
View File
@@ -43,6 +43,20 @@ S3_PORT="${S3_PORT:-9000}"
TEST_MODE="${TEST_MODE:-single}"
MAXFAIL="${MAXFAIL:-1}"
XDIST="${XDIST:-0}"
# Test scope:
# "implemented" (default) - run only the implemented_tests.txt whitelist (PR gate)
# "all" - run the entire upstream suite (scheduled full sweep)
TEST_SCOPE="${TEST_SCOPE:-implemented}"
if [[ "${TEST_SCOPE}" != "implemented" && "${TEST_SCOPE}" != "all" ]]; then
echo "[ERROR] Invalid TEST_SCOPE: ${TEST_SCOPE} (must be \"implemented\" or \"all\")" >&2
exit 1
fi
# Upstream ceph/s3-tests suite, pinned for reproducible runs.
# Bump S3TESTS_REV deliberately: upstream changes can rename tests or change
# assertions, so a bump usually requires reclassifying the test list files.
S3TESTS_REPO="${S3TESTS_REPO:-https://github.com/ceph/s3-tests.git}"
S3TESTS_REV="${S3TESTS_REV:-5522d1c351f75bc00ae0f64f742f3f095f5939d9}"
# Compatibility default for the s3-tests harness:
# this script provisions multiple local export directories on the same physical disk.
@@ -146,7 +160,12 @@ fi
# 2. Easy maintenance - edit txt files to add/remove tests
# 3. Separation of concerns - test classification vs test execution
# =============================================================================
if [[ -z "${TESTEXPR:-}" ]]; then
TESTEXPR="${TESTEXPR:-}"
if [[ -n "${TESTEXPR}" ]]; then
log_info "Using custom TESTEXPR selection: ${TESTEXPR}"
elif [[ "${TEST_SCOPE}" == "all" ]]; then
log_info "TEST_SCOPE=all: running the entire upstream test suite (no whitelist)"
else
if [[ -f "${IMPLEMENTED_TESTS_FILE}" ]]; then
log_info "Loading test list from: ${IMPLEMENTED_TESTS_FILE}"
load_testnodes_from_file "${IMPLEMENTED_TESTS_FILE}"
@@ -240,8 +259,11 @@ Environment Variables:
S3_ALT_ACCESS_KEY - Alt user access key (default: rustfsalt)
S3_ALT_SECRET_KEY - Alt user secret key (default: rustfsalt)
RUSTFS_SSE_S3_MASTER_KEY - Optional base64 32-byte key for local managed SSE fallback
MAXFAIL - Stop after N failures (default: 1)
MAXFAIL - Stop after N failures, 0 = never stop (default: 1)
XDIST - Enable parallel execution with N workers (default: 0)
TEST_SCOPE - "implemented" (whitelist, default) or "all" (entire upstream suite)
S3TESTS_REPO - s3-tests repository URL (default: https://github.com/ceph/s3-tests.git)
S3TESTS_REV - Pinned s3-tests commit; bump deliberately and reclassify test lists
MARKEXPR - pytest marker expression (default: no marker filtering)
TESTEXPR - pytest -k expression (overrides implemented_tests.txt node list)
S3TESTS_CONF_TEMPLATE - Path to s3tests config template (default: .github/s3tests/s3tests.conf)
@@ -819,14 +841,23 @@ awscurl \
log_info "Alt user provisioned successfully"
# Step 8: Prepare s3-tests
log_info "Preparing s3-tests..."
if [ ! -d "${PROJECT_ROOT}/s3-tests" ]; then
git clone --depth 1 https://github.com/ceph/s3-tests.git "${PROJECT_ROOT}/s3-tests" || {
log_error "Failed to clone s3-tests"
# Step 8: Prepare s3-tests, pinned to S3TESTS_REV for reproducible runs
log_info "Preparing s3-tests at ${S3TESTS_REV}..."
if [ ! -d "${PROJECT_ROOT}/s3-tests/.git" ]; then
git init -q "${PROJECT_ROOT}/s3-tests"
git -C "${PROJECT_ROOT}/s3-tests" remote add origin "${S3TESTS_REPO}"
fi
if ! git -C "${PROJECT_ROOT}/s3-tests" cat-file -e "${S3TESTS_REV}^{commit}" 2>/dev/null; then
git -C "${PROJECT_ROOT}/s3-tests" fetch --depth 1 origin "${S3TESTS_REV}" || {
log_error "Failed to fetch s3-tests revision ${S3TESTS_REV} from ${S3TESTS_REPO}"
exit 1
}
fi
# -f discards local edits left by previous runs (e.g. pytest-xdist appended to requirements.txt)
git -C "${PROJECT_ROOT}/s3-tests" checkout -qf --detach "${S3TESTS_REV}" || {
log_error "Failed to checkout s3-tests revision ${S3TESTS_REV}"
exit 1
}
cd "${PROJECT_ROOT}/s3-tests"
@@ -862,8 +893,11 @@ fi
PYTEST_SELECTION_ARGS=()
if [[ "${USE_FILE_TEST_NODES}" == "true" ]]; then
PYTEST_SELECTION_ARGS=("${TEST_NODE_ARGS[@]}")
else
elif [[ -n "${TESTEXPR}" ]]; then
PYTEST_SELECTION_ARGS=("${S3_TEST_FILE}" -k "${TESTEXPR}")
else
# TEST_SCOPE=all: the whole upstream suite
PYTEST_SELECTION_ARGS=("${S3_TEST_FILE}")
fi
# Run tests from s3tests/functional
@@ -894,6 +928,16 @@ elif [ "${DEPLOY_MODE}" = "existing" ]; then
echo "{\"host\": \"${S3_HOST}\", \"port\": ${S3_PORT}, \"mode\": \"existing\"}" > "${ARTIFACTS_DIR}/rustfs-${TEST_MODE}/inspect.json" || true
fi
# Step 11: Classification report (informational, never fails the run)
REPORT_SCRIPT="${SCRIPT_DIR}/report_compat.py"
if [ -f "${REPORT_SCRIPT}" ] && [ -f "${ARTIFACTS_DIR}/junit.xml" ]; then
python3 "${REPORT_SCRIPT}" \
--junit "${ARTIFACTS_DIR}/junit.xml" \
--lists-dir "${SCRIPT_DIR}" \
--output "${ARTIFACTS_DIR}/compat-report.md" \
|| log_warn "Compatibility report generation failed"
fi
# Summary
if [ ${TEST_EXIT_CODE} -eq 0 ]; then
log_info "Tests completed successfully!"