mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-06 03:59:14 +00:00
test(e2e): add 4-node 4-disk distributed Actions suite
Add a nightly e2e-distributed lane that boots localhost 4-node clusters and covers S3, object lock, versioning, replication, quota, expand, decommission, rebalance, site replication, concurrency, and chaos. Co-authored-by: RustFS <hello@rustfs.com>
This commit is contained in:
Cursor Agent
@@ -12,10 +12,10 @@ Pick the lowest layer that can prove the change; add a higher-layer test only wh
|
||||
|---|---|---|---|
|
||||
| Unit & crate integration | Per-crate logic and in-process integration tests | `cargo nextest run --all --exclude e2e_test` (or `-p <crate>`); `make test` wraps it | Every PR, required (`Test and Lint`, `ci` profile) |
|
||||
| ecstore black-box | Erasure-coded read/write/recovery validation; profiles `quick` / `full` / `destructive` / `fuzz` | `scripts/run_ecstore_validation_suite.sh --profile quick` | Local and release validation only; not wired into any workflow. Contract: [ecstore-validation-suite-design.md](ecstore-validation-suite-design.md) |
|
||||
| e2e (`e2e_test` crate) | A real `rustfs` binary per test, driven over the S3, admin, and protocol APIs | `cargo nextest run --profile e2e-smoke -p e2e_test` | PR: `e2e-smoke` (report-only); merge queue / main push: `e2e-full`; nightly: `e2e-repl-nightly`, `e2e-nightly`, `e2e-protocols`. Guide: [`crates/e2e_test/README.md`](../../crates/e2e_test/README.md) |
|
||||
| e2e (`e2e_test` crate) | A real `rustfs` binary per test, driven over the S3, admin, and protocol APIs | `cargo nextest run --profile e2e-smoke -p e2e_test` | PR: `e2e-smoke` (report-only); merge queue / main push: `e2e-full`; nightly: `e2e-repl-nightly`, `e2e-nightly`, `e2e-protocols`, `e2e-distributed`. Guide: [`crates/e2e_test/README.md`](../../crates/e2e_test/README.md); 4-node 4-disk map: [distributed-e2e.md](distributed-e2e.md) |
|
||||
| s3s-e2e conformance | External S3 conformance tool against a live server | `./scripts/e2e-run.sh ./target/debug/rustfs <data-dir>` | PR, report-only (second half of the `End-to-End Tests` job) |
|
||||
| S3 compatibility | `ceph/s3-tests` (boto3; allow-list `scripts/s3-tests/implemented_tests.txt`) and MinIO `mint` | `scripts/s3-tests/run.sh`; mint via `.github/workflows/mint.yml` | s3-tests: PR report-only plus a weekly full sweep; mint: weekly, report-only |
|
||||
| Chaos / fault-injection | Single-node disk fault injection (`crates/e2e_test/src/chaos.rs`, `crates/e2e_test/src/fault_proxy.rs`) used by the reliability and heal e2e modules | Part of the e2e crate (`e2e-reliability` test-group) | With the `e2e-full` and nightly e2e lanes. A multi-node power-loss harness is not in tree |
|
||||
| Chaos / fault-injection | Single-node disk fault injection (`crates/e2e_test/src/chaos.rs`, `crates/e2e_test/src/fault_proxy.rs`) plus the 4-node kill/offline-drive/blackhole cases in `crates/e2e_test/src/distributed/chaos_test.rs` | Part of the e2e crate (`e2e-reliability` and `e2e-distributed`) | Reliability cases with `e2e-full`; 4-node chaos with the `e2e-distributed` nightly lane |
|
||||
| Fuzz | `cargo-fuzz` targets over untrusted parsing surfaces; isolated sub-workspace under `fuzz/` | `./scripts/fuzz/run.sh` (see [`fuzz/README.md`](../../fuzz/README.md)) | PR smoke on the paths listed in `.github/workflows/fuzz.yml`, plus nightly corpus |
|
||||
| Benchmarks | Criterion benches under each crate's `benches/` | `cargo bench -p <crate>` | On demand; never a gate |
|
||||
|
||||
@@ -61,6 +61,7 @@ All profiles are defined in `.config/nextest.toml`; its block comments hold the
|
||||
| `e2e-full` | Merge-queue / main-push single-node e2e lane |
|
||||
| `e2e-repl-nightly` | Nightly slow / cross-process replication lane |
|
||||
| `e2e-nightly` | Nightly serial multi-process cluster fault lane |
|
||||
| `e2e-distributed` | Nightly 4-node 4-disk S3 / lock / versioning / replication / quota / expand / decommission / rebalance / site-replication / chaos lane |
|
||||
| `e2e-protocols` | Nightly fixed-port FTPS/SFTP/WebDAV lane, run with `-j 1` |
|
||||
|
||||
Membership of each e2e profile is pinned by a digest in `.config/e2e-<profile>-selection.txt` and checked by `scripts/check_test_wiring.py --check-profile <profile>` before the lane runs. To list what a profile selects on your platform (the result is platform-dependent because some modules are linux-only):
|
||||
|
||||
@@ -74,6 +74,7 @@ Scheduled lanes never block a PR. Their workflow-local gate fails the run, sched
|
||||
| `ci.yml` (weekly) | full matrix, including the schedule/dispatch-only rio-v2 jobs `build-rustfs-debug-binary-rio-v2` and `e2e-tests-rio-v2` | per-job | yes | dispatch `ci.yml` |
|
||||
| `build.yml` (weekly) | `build-rustfs` over the six-target platform matrix in `prepare-platform-matrix` (four Linux, macOS aarch64, Windows x86_64) | build/package integrity | yes | dispatch `build.yml` with an exact platform set |
|
||||
| `e2e-replication-nightly.yml` (nightly) | `repl-nightly`, `cluster-nightly`, `protocols-nightly` | three independent gates; JUnit, membership listing, server logs | yes | `cargo nextest run --profile e2e-repl-nightly -p e2e_test`; `--profile e2e-nightly`; `-j 1 --profile e2e-protocols` |
|
||||
| `e2e-distributed.yml` (nightly) | `distributed` | 4-node 4-disk e2e gate; JUnit, membership listing, server logs | yes, with `never_ran_grace_until` | `cargo nextest run --profile e2e-distributed -p e2e_test` |
|
||||
| `e2e-s3tests.yml` (weekly) | `s3tests` (single and distributed, four shards each), `upstream-head-canary` | compatibility gate; report, JUnit, node IDs, server logs | yes | `scripts/s3-tests/run.sh` against an existing single or distributed target |
|
||||
| `fuzz.yml` (nightly) | `nightly-fuzz-corpus` per target | gate; corpus and crash artifacts | yes | `MAX_TOTAL_TIME=<seconds> ./scripts/fuzz/run.sh` |
|
||||
| `minio-interop.yml` (nightly) | `minio-interop` | EC + SSE read-parity gate | yes, with `never_ran_grace_until` | pinned Docker fixture steps in the workflow |
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
# Distributed 4-node 4-disk e2e
|
||||
|
||||
**Use this when:** adding or diagnosing GitHub Actions coverage for a 4-node cluster, or deciding whether a behaviour belongs in `e2e-distributed` versus the single-node `e2e-full` lane, the nightly cluster-fault lane, or the hardware functional chain.
|
||||
**Source of truth:** `crates/e2e_test/src/distributed/`, `[profile.e2e-distributed]` in `.config/nextest.toml`, `.github/workflows/e2e-distributed.yml`.
|
||||
|
||||
## Topology
|
||||
|
||||
The in-tree harness runs every node on `127.0.0.1` with a distinct port. That matches `RustFSTestClusterEnvironment` in `crates/e2e_test/src/common.rs`:
|
||||
|
||||
| Layout | Constructor | Use |
|
||||
|---|---|---|
|
||||
| 4 nodes × 4 drives, one pool | `ClusterTopology::single_pool_multidrive(4, 4)` | S3, object lock, versioning, quota, observability, concurrency, chaos |
|
||||
| 4 nodes × 1 drive, one pool | `ClusterTopology::single_pool(4)` | Two-site replication (8 processes total) |
|
||||
| 4 single-node pools × 4 drives | `ClusterTopology::per_node_pools(4, [[0],[1],[2],[3]])` | Decommission, rebalance, S3-during-move, integrity |
|
||||
| 2 single-node pools × 4 drives, then `append_single_node_pool` | expansion seed | Pool expand then rebalance |
|
||||
|
||||
A pool striped across several localhost ports is not expressible (`RUSTFS_VOLUMES` host ellipses would collide on disk paths). Multi-host striped pools remain the hardware functional-chain / backlog #1313 / #1314 lane.
|
||||
|
||||
## What this lane covers
|
||||
|
||||
`cargo nextest run --profile e2e-distributed -p e2e_test` selects `distributed::*`:
|
||||
|
||||
- S3 put / get / head / list / copy / rename / delete / presign / empty object
|
||||
- Object Lock COMPLIANCE, GOVERNANCE (with bypass), legal hold
|
||||
- Versioning, version GET, delete marker
|
||||
- Bucket replication between two 4-node clusters; hard quota
|
||||
- Health / admin info / storageinfo / audit target list
|
||||
- Pool expand, decommission, rebalance, checksum integrity, S3 during move
|
||||
- Site replication object convergence
|
||||
- High-concurrency PUT/GET; concurrent PUT during decommission
|
||||
- Node kill/restart, full process restart, drive offline, volume-proxy blackhole
|
||||
- Multipart and cross-node listing agreement
|
||||
|
||||
## Existing Actions gaps this lane does not replace
|
||||
|
||||
Those suites stay in place; this lane fills the in-tree 4×4 hole they leave.
|
||||
|
||||
| Existing lane | Gap |
|
||||
|---|---|
|
||||
| `rustfs-*-test.yml` functional chain | Clones private `rustfs/auto-testing`, runs on three shared VMs (`vm000`–`vm002`), `continue-on-error: true`, not a merge signal, not 4 nodes |
|
||||
| `e2e-smoke` / `e2e-full` | Almost all cases are single-node |
|
||||
| `e2e-nightly` | 4-node cluster faults and heal, not S3/lock/versioning/quota/decommission matrix |
|
||||
| `e2e-repl-nightly` | Site and bucket replication on 1–3 *single-node* processes |
|
||||
| `e2e-s3tests.yml` `multi` | Weekly ceph/s3-tests against Docker 4-node; not lock/WORM, decommission, chaos, or checksum integrity |
|
||||
| `crates/e2e_test/src/chaos.rs` | Single-node disk faults only |
|
||||
|
||||
Hardware power-loss, NIC pull, and real disk replacement still belong on the smoke-testing VMs. This lane simulates those with SIGKILL, directory rename, and `FaultProxy` blackhole.
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
cargo build -p rustfs --bins
|
||||
cargo nextest run --profile e2e-distributed -p e2e_test
|
||||
```
|
||||
|
||||
Membership is pinned by `.config/e2e-distributed-selection.txt`. Update it with `python3 ./scripts/check_test_wiring.py --update-profile e2e-distributed <listing.json> linux` after adding or renaming a case.
|
||||
Reference in New Issue
Block a user