gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-02 11:29:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

ci(s3-tests): pin upstream suite, add weekly full sweep and compat report (#4204)

Browse Source 
Reworks the S3 compatibility test harness for reproducibility and faster
feedback:

- Pin ceph/s3-tests to a fixed commit (S3TESTS_REV, fetch-by-SHA) so the
  449-test PR gate is reproducible; previously every run cloned upstream
  master, letting test renames or assertion changes break CI silently.
- PR gate (ci.yml s3-implemented-tests): MAXFAIL=0 + XDIST=4 so a single
  CI round reports every failure in parallel instead of stopping at the
  first one serially.
- Add TEST_SCOPE=all to run.sh to run the entire upstream suite, and
  report_compat.py to diff junit results against the classification
  lists (regressions, promotion candidates, unclassified tests).
- Rewrite e2e-s3tests.yml: delegate execution to run.sh (single source
  of truth; also fixes the broken config generation that left S3_PORT
  empty), add a weekly scheduled full sweep that fails only on whitelist
  regressions, and fix the multi-node topology to a real distributed
  cluster (endpoint-style RUSTFS_VOLUMES) instead of four independent
  single-node stores behind a load balancer.
- Docs: rewrite stale .github/s3tests/README.md (marker-era strategy),
  update scripts/s3-tests/README.md, fix dead build_testexpr.sh
  reference in S3_COMPAT_WORKFLOW.md, drop legacy non_standard_tests.txt.

All 747 classified test names verified present at the pinned revision;
13 upstream tests are currently unclassified and will surface in the
first full-sweep report.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Zhengchao An
2026-07-02 23:43:19 +08:00
committed by GitHub
parent 31c026dd4f
commit 3eeb459ece
8 changed files with 472 additions and 639 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/s3tests/README.md
+51 -73
View File
@@ -1,103 +1,81 @@
# S3 Compatibility Tests Configuration
This directory contains the configuration for running [Ceph S3 compatibility tests](https://github.com/ceph/s3-tests) against RustFS.
This directory contains the configuration template for running
[Ceph S3 compatibility tests](https://github.com/ceph/s3-tests) against RustFS.
The test runner itself lives in [`scripts/s3-tests/`](../../scripts/s3-tests/)
— see its README for how tests are selected, executed, and reported. This
directory only holds the shared config template.
## Configuration File
The `s3tests.conf` file is based on the official `s3tests.conf.SAMPLE` from the ceph/s3-tests repository. It uses environment variable substitution via `envsubst` to configure the endpoint and credentials.
`s3tests.conf` is based on the official `s3tests.conf.SAMPLE` from the
ceph/s3-tests repository. It is a template: `scripts/s3-tests/run.sh` renders
it with `envsubst` before every run.
### Key Configuration Points
Required environment variables (all have defaults in `run.sh`):
- **Host**: Set via `${S3_HOST}` environment variable (e.g., `rustfs-single` for single-node, `lb` for multi-node)
- **Port**: 9000 (standard RustFS port)
- **Credentials**: Uses `${S3_ACCESS_KEY}` and `${S3_SECRET_KEY}` from workflow environment
- **TLS**: Disabled (`is_secure = False`)
| Variable | Meaning |
|----------|---------|
| `S3_HOST` | RustFS endpoint host |
| `S3_PORT` | RustFS endpoint port |
| `S3_ACCESS_KEY` / `S3_SECRET_KEY` | main user credentials |
| `S3_ALT_ACCESS_KEY` / `S3_ALT_SECRET_KEY` | alt user credentials (must differ from main) |
## Test Execution Strategy
TLS is disabled (`is_secure = False`).
### Network Connectivity Fix
## Test Selection Strategy
Tests run inside a Docker container on the `rustfs-net` network, which allows them to resolve and connect to the RustFS container hostnames. This fixes the "Temporary failure in name resolution" error that occurred when tests ran on the GitHub runner host.
Tests are selected by exact pytest node ids from classification list files in
`scripts/s3-tests/` — **not** by pytest markers:
### Performance Optimizations
- `implemented_tests.txt` — expected to pass; run as the per-PR gate
(`ci.yml`, job `s3-implemented-tests`)
- `unimplemented_tests.txt` — standard S3 features not yet implemented
- `excluded_tests.txt` — intentionally excluded (vendor-specific behavior or
product decisions)
1. **Parallel Execution**: Uses `pytest-xdist` with `-n 4` to run tests in parallel across 4 workers
2. **Load Distribution**: Uses `--dist=loadgroup` to distribute test groups across workers
3. **Fail-Fast**: Uses `--maxfail=50` to stop after 50 failures, saving time on catastrophic failures
The upstream s3-tests repository is pinned to a fixed commit (`S3TESTS_REV` in
`run.sh`) so results are reproducible; bump it deliberately and reclassify the
lists when upstream changes.
### Feature Filtering
## Where Tests Run
Tests are filtered using pytest markers (`-m`) to skip features not yet supported by RustFS:
- `lifecycle` - Bucket lifecycle policies
- `versioning` - Object versioning
- `s3website` - Static website hosting
- `bucket_logging` - Bucket logging
- `encryption` / `sse_s3` - Server-side encryption
- `cloud_transition` / `cloud_restore` - Cloud storage transitions
- `lifecycle_expiration` / `lifecycle_transition` - Lifecycle operations
This filtering:
1. Reduces test execution time significantly (from 1+ hour to ~10-15 minutes)
2. Focuses on features RustFS currently supports
3. Avoids hundreds of expected failures
- **Per PR**: `.github/workflows/ci.yml` job `s3-implemented-tests` runs the
implemented whitelist against a single-node debug binary. Any failure blocks
the PR.
- **Weekly + manual**: `.github/workflows/e2e-s3tests.yml` runs the full
upstream suite (`TEST_SCOPE=all`) against a Docker deployment (single node
or a 4-node distributed cluster behind HAProxy). It fails only on
regressions in the implemented whitelist and publishes a classification
report (`compat-report.md`, also shown in the job summary) listing promotion
candidates and unclassified tests.
## Running Tests Locally
### Single-Node Test
```bash
# Set credentials
export S3_ACCESS_KEY=rustfsadmin
export S3_SECRET_KEY=rustfsadmin
# Whitelist run against a locally built binary (the same thing the PR gate does)
./scripts/s3-tests/run.sh
# Start RustFS container
docker run -d --name rustfs-single \
--network rustfs-net \
-e RUSTFS_ADDRESS=0.0.0.0:9000 \
-e RUSTFS_ACCESS_KEY=$S3_ACCESS_KEY \
-e RUSTFS_SECRET_KEY=$S3_SECRET_KEY \
-e RUSTFS_VOLUMES="/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3" \
rustfs-ci
# Full sweep, never stop on failures, 4 pytest workers
TEST_SCOPE=all MAXFAIL=0 XDIST=4 ./scripts/s3-tests/run.sh
# Generate config
export S3_HOST=rustfs-single
envsubst < .github/s3tests/s3tests.conf > /tmp/s3tests.conf
# Run tests
docker run --rm \
--network rustfs-net \
-v /tmp/s3tests.conf:/etc/s3tests.conf:ro \
python:3.12-slim \
bash -c '
apt-get update -qq && apt-get install -y -qq git
git clone --depth 1 https://github.com/ceph/s3-tests.git /s3-tests
cd /s3-tests
pip install -q -r requirements.txt pytest-xdist
S3TEST_CONF=/etc/s3tests.conf pytest -v -n 4 \
s3tests/functional/test_s3.py \
-m "not lifecycle and not versioning and not s3website and not bucket_logging and not encryption and not sse_s3"
'
# A specific test against an already-running server
DEPLOY_MODE=existing TESTEXPR="test_bucket_list_empty" ./scripts/s3-tests/run.sh
```
## Test Results Interpretation
- **PASSED**: Test succeeded, feature works correctly
- **FAILED**: Test failed, indicates a potential bug or incompatibility
- **ERROR**: Test setup failed (e.g., network issues, missing dependencies)
- **SKIPPED**: Test skipped due to marker filtering
Results land in `artifacts/s3tests-single/` (`junit.xml`, `pytest.log`,
`compat-report.md`).
## Adding New Feature Support
When adding support for a new S3 feature to RustFS:
1. Remove the corresponding marker from the filter in `.github/workflows/e2e-s3tests.yml`
2. Run the tests to verify compatibility
3. Fix any failing tests
4. Update this README to reflect the newly supported feature
Follow [`scripts/s3-tests/S3_COMPAT_WORKFLOW.md`](../../scripts/s3-tests/S3_COMPAT_WORKFLOW.md).
In short: fix the behavior, verify with `TESTEXPR=...`, then move the test
from `unimplemented_tests.txt` to `implemented_tests.txt` so the PR gate locks
it in.
## References
- [Ceph S3 Tests Repository](https://github.com/ceph/s3-tests)
- [S3 API Compatibility](https://docs.aws.amazon.com/AmazonS3/latest/API/)
- [pytest-xdist Documentation](https://pytest-xdist.readthedocs.io/)
- [RustFS test runner](../../scripts/s3-tests/README.md)