docs(e2e): contributor guide for the e2e_test crate (backlog#1153 infra-10) (#4705)

docs(e2e): write contributor guide for e2e_test crate (backlog#1153 infra-10) Turn the e2e_test README skeleton into a dense, link-heavy contributor guide: crate overview + grouped module map, how to run (whole crate / one module / e2e-smoke profile / ILM serial lane / protocols suite), #[ignore] semantics as a pointer to live sources, how to add a test (RustFSTestEnvironment + RustFSTestClusterEnvironment, common.rs/chaos.rs helper inventory, isolation rules, #[serial]-vs-nextest reality), a CI subset map table, and a troubleshooting section (stale binary / RUSTFS_TEST_PORT / orphan processes). Keeps the ci-4 "CI smoke subset" section intact and restructures around it. Adds a reciprocal link from crates/e2e_test/AGENTS.md. Refs rustfs/backlog#1153 (infra-10), #1155.
2026-07-26 08:18:18 +00:00 · 2026-07-11 09:27:30 +08:00
parent 3f25426534
commit c41a813a31
2 changed files with 232 additions and 6 deletions
@@ -2,6 +2,11 @@

 Applies to `crates/e2e_test/`.

+Contributor guide (how to run suites, add a test, the harness/fixture
+inventory, `#[ignore]` classes, and the CI map) lives in
+[`README.md`](README.md). This file holds the crate rules that agents must
+follow.
+
 ## Test Reliability

 - Keep end-to-end tests deterministic and environment-aware.
@@ -1,12 +1,233 @@
 # e2e_test

-End-to-end test suite for RustFS. Tests spawn a real `rustfs` binary (see
-`src/common.rs`) and drive it with the AWS SDK.
+End-to-end test suite for RustFS. Each test spawns a **real `rustfs` binary**
+(built on demand from the workspace) and drives it over the network with the
+AWS SDK (`aws-sdk-s3`), raw HTTP (`reqwest` / `awscurl`), or a protocol client
+(FTPS / WebDAV / SFTP). This is the black-box integration layer: exhaustive
+end-to-end behavior lives here, unit behavior stays in the source crates
+(see [`AGENTS.md`](AGENTS.md)).

-> This README currently only documents the CI smoke subset (backlog#1149
-> ci-4). The full contributor guide — how to add a case, fixture helpers,
-> `#[ignore]` semantics, harness topologies — is tracked as backlog#1153
-> infra-10 and will extend this file.
+The harness lives in [`src/common.rs`](src/common.rs) (single-node +
+cluster environments, S3 client construction, `awscurl` helpers) and
+[`src/chaos.rs`](src/chaos.rs) (in-process disk fault injection). Crate-wide
+test conventions and environment-safety rules are in
+[`AGENTS.md`](AGENTS.md); this file is the contributor guide.
+
+## Module map (~50 modules)
+
+Registered in [`src/lib.rs`](src/lib.rs). Grouped by concern:
+
+| Group | Location | What it covers |
+| --- | --- | --- |
+| **functional** | top-level `*_test.rs` | S3 data plane: `list_objects_*`, `copy_object_*`, `delete_objects_versioning`, `head_object_*`, `checksum_upload`, `compression`, `content_encoding`, `special_chars`, `leading_slash_key`, `create_bucket_region`, `quota`, `data_usage`, `snowball_auto_extract`, `mc_mirror_small_bucket`, `archive_download_integrity`, `version_id_regression`, `delete_marker_migration_semantics` |
+| **object_lock** | [`src/object_lock/`](src/object_lock) | Retention / legal-hold / WORM semantics |
+| **kms** | [`src/kms/`](src/kms) | SSE-S3 / SSE-KMS / SSE-C, local + Vault backends, multipart encryption. Own guide: [`src/kms/README.md`](src/kms/README.md) |
+| **policy** | [`src/policy/`](src/policy), `existing_object_tag_policy_test`, `bucket_policy_check_test`, `anonymous_access_test`, `security_boundary_test`, `multipart_auth_test` | IAM / bucket-policy / STS session policy, policy variables, anonymous access, DoS/SSRF boundaries. Own guide: [`src/policy/README.md`](src/policy/README.md) |
+| **protocols** | [`src/protocols/`](src/protocols) | FTPS, WebDAV, SFTP compliance. Fixed ports, own guide: [`src/protocols/README.md`](src/protocols/README.md) |
+| **reliant** | [`src/reliant/`](src/reliant) | Tests that reuse an **externally started** server (SQL/select, conditional writes, lifecycle, deleted-object reads, node-interact). Run via [`scripts/run_e2e_tests.sh`](../../scripts/run_e2e_tests.sh); see [`src/reliant/README.md`](src/reliant/README.md) |
+| **cluster** | `cluster_concurrency_test`, `stale_multipart_cleanup_cluster_test`, `namespace_lock_quorum_test`, `admin_timeout_regression_test`, `object_lambda_test`, `replication_extension_test` | Multi-node scenarios via `RustFSTestClusterEnvironment` |
+| **chaos / reliability** | [`src/chaos.rs`](src/chaos.rs), `reliability_disk_fault_test`, `heal_erasure_disk_rebuild_test`, `server_startup_failfast_test` | Disk offline/replace/corrupt, EC rebuild, heal, fail-fast startup |
+
+## How to run
+
+All commands assume repo root. `cargo test` triggers an on-demand build of the
+`rustfs` binary from [`src/common.rs`](src/common.rs) (`rustfs_binary_path`) on
+first use — the first invocation is slow, later ones reuse the binary.
+
+```bash
+# Whole crate (default = ignored tests skipped)
+cargo nextest run -p e2e_test
+
+# One module
+cargo nextest run -p e2e_test -E 'test(list_objects_v2_pagination_test)'
+
+# PR smoke subset (see "CI smoke subset" below)
+cargo nextest run --profile e2e-smoke -p e2e_test
+
+# ILM serial lane — ignored lifecycle tests, single-threaded (mirrors CI)
+cargo nextest run -j1 --run-ignored ignored-only -p rustfs-scanner -p rustfs \
+  -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))'
+
+# Protocols suite — fixed ports, MUST be single-threaded, gated by build features
+RUSTFS_BUILD_FEATURES=ftps,webdav,sftp \
+  cargo test -p e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
+```
+
+The protocols suite has its own contract (fixed bind ports 9022–9301,
+`--test-threads=1`, feature-gated scheduling) documented in
+[`src/protocols/README.md`](src/protocols/README.md). `RUSTFS_BUILD_FEATURES`
+selects which features the spawned binary is built with; leave it unset to run
+every protocol entry.
+
+### `#[ignore]` semantics
+
+Ignored tests are excluded from the default `cargo nextest run` pass because
+they need something the default runner does not provide. **Do not maintain a
+static count here** — it rots (the set shrinks as ci-13 / ilm-3 activate
+suites). Read the live sources instead:
+
+```bash
+rg -n '#\[ignore' crates/e2e_test/src   # every ignore + its reason string
+```
+
+The reason string on each attribute is the classifier. Current classes:
+
+- **Needs a pre-started server** — `"requires running RustFS server at
+  localhost:9000"` / `"Connects to existing rustfs server"`. These are the
+  `reliant/*` and `policy/test_runner` tests; start a server first (e.g.
+  [`scripts/run_e2e_tests.sh`](../../scripts/run_e2e_tests.sh)) or use
+  `--run-ignored`.
+- **Heavy / external tool** — `"Starts a rustfs server; enable when running
+  full E2E"`, `"requires awscurl and spawns a real RustFS server"`. Spawn their
+  own server and/or need `awscurl` on `PATH`.
+- **Serial / global-state (ILM lane)** — lifecycle tests bind fixed ports and
+  share process-global singletons; run via the ILM serial lane above.
+
+## How to add a test
+
+### Single-node (the common case)
+
+Use `RustFSTestEnvironment` from [`src/common.rs`](src/common.rs). It picks a
+**random free port** and a **unique temp dir** per instance, so tests are
+parallel-safe by construction and clean up on `Drop`:
+
+```rust
+use crate::common::{RustFSTestEnvironment, TEST_BUCKET};
+
+#[tokio::test]
+async fn my_case() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;       // waits for readiness
+    let client = env.create_s3_client();          // aws-sdk-s3 Client
+    env.create_test_bucket(TEST_BUCKET).await?;
+    // ... drive `client` ...
+    Ok(())
+}
+```
+
+Register the module in [`src/lib.rs`](src/lib.rs) under `#[cfg(test)]`.
+
+### Cluster
+
+Use `RustFSTestClusterEnvironment::new(node_count)` then `.start()`; it spawns
+`node_count` servers over a shared erasure set and hands out per-node S3 clients
+via `create_s3_client(idx)` / `create_all_clients()`. See
+`cluster_concurrency_test.rs` and `namespace_lock_quorum_test.rs` for patterns.
+
+### Fixture / helper inventory (`src/common.rs`)
+
+| Helper | Purpose |
+| --- | --- |
+| `RustFSTestEnvironment::new` / `with_address` | Single-node env; random or fixed address |
+| `start_rustfs_server` / `_with_env` / `_without_cleanup` | Spawn the server (optional extra args / env vars / no pre-cleanup) |
+| `wait_for_server_ready` | Poll readiness before issuing requests |
+| `create_s3_client` / `create_test_bucket` / `delete_test_bucket` | aws-sdk-s3 client + bucket lifecycle |
+| `find_available_port` | Random free port (isolation primitive) |
+| `rustfs_binary_path` / `_with_features` | Locate/build the binary; honors `RUSTFS_BUILD_FEATURES` |
+| `requested_rustfs_build_features` / `rustfs_build_feature_enabled` | Feature-gate a test to what the binary was built with |
+| `awscurl_available` + `execute_awscurl` / `awscurl_post` / `_get` / `_put` / `_delete` / `awscurl_post_sts_form_urlencoded` | Admin/STS API calls via `awscurl` (skip gracefully when absent) |
+| `replication_fast_env` | Env vars that shrink replication timers (from repl-4); pass to `start_rustfs_server_with_env` |
+| `local_http_client` / `init_logging` | Loopback HTTP client; idempotent tracing init |
+| `RustFSTestClusterEnvironment` (`new`/`start`/`start_node`/`stop_node`/`create_all_clients`) | Multi-node harness |
+| Constants: `DEFAULT_ACCESS_KEY`, `DEFAULT_SECRET_KEY`, `TEST_BUCKET`, `ENV_RUSTFS_BUILD_FEATURES` | Shared credentials / bucket name / env-var name |
+
+Fault injectors live in [`src/chaos.rs`](src/chaos.rs): `DiskFaultHarness`
+(`take_disk_offline`, `bring_disk_online`, `replace_disk_with_empty`,
+`corrupt_object_shard`, `object_metadata_exists_on_disk`, `kill_server` /
+`restart_server`) plus `signed_admin_post`.
+
+### Isolation rules
+
+- **Port**: never hard-code a port for single-node tests — `new()` allocates a
+  random one. Fixed ports (protocols, ILM lane) force `--test-threads=1` / a
+  serial CI lane.
+- **Temp dir**: each env owns a temp dir cleaned on `Drop`; do not write under
+  a shared path.
+- **Orphans**: `RustFSTestEnvironment` kills its child on `Drop`, but a panicked
+  or `kill -9`'d run can leak a `rustfs` process holding a port — see
+  Troubleshooting.
+
+### `#[serial]` vs nextest reality
+
+`serial_test`'s `#[serial]` uses an **in-process** mutex. Under nextest each
+test runs in its **own process**, so `#[serial]` does **not** serialize across
+tests there — see the header of [`.config/nextest.toml`](../../.config/nextest.toml).
+Real cross-test serialization comes from a nextest test-group (`max-threads =
+1`) or a `-j1` CI lane. Single-node e2e tests should instead be parallel-safe by
+construction (random port + isolated temp dir) and need no serialization.
+
+## CI map
+
+`e2e_test` is **excluded** from the main `cargo nextest run --profile ci --all`
+pass ([`.github/workflows/ci.yml`](../../.github/workflows/ci.yml) line 158,
+`--exclude e2e_test`) — the whole crate is too slow to gate every PR. Subsets
+join CI through the nextest profile system only (never as ad-hoc jobs):
+
+| Suite | Runs where | Status |
+| --- | --- | --- |
+| Smoke subset (`e2e-smoke` profile) | `e2e-tests` job, every PR | **Active** (backlog#1149 ci-4) |
+| `s3s-e2e` black-box | `e2e-tests` + `e2e-tests-rio-v2` jobs | **Active** (external conformance tool) |
+| ILM / lifecycle (ignored) | `test-ilm-integration-serial` lane, `-j1` | **Active** (backlog#1148 ilm-1) |
+| KMS suite | — | Not in CI yet (backlog#1149 ci-5) |
+| Protocols (FTPS/WebDAV/SFTP) | — | Not in CI yet (backlog#1149 ci-7) |
+| Cluster / replication e2e | — | Not in CI yet (backlog#1147 repl-1) |
+| `reliant/*` (pre-started server) | — | Manual only |
+
+Links: [`ci.yml`](../../.github/workflows/ci.yml) `e2e-tests` (line 347),
+`test-ilm-integration-serial` (line 196). The `e2e-smoke` `default-filter` in
+[`.config/nextest.toml`](../../.config/nextest.toml) is the **single wiring
+mechanism** — extend that filter (or add a sibling profile) to admit more
+tests; do not add e2e jobs to `ci.yml`. repl-1 / ilm-3 are landing in parallel
+and may add lanes; keep the table above easy to extend.
+
+## Troubleshooting
+
+**Reproduce a CI failure locally** — run the exact profile/lane:
+
+```bash
+# Smoke (e2e-tests job)
+cargo nextest run --profile e2e-smoke -p e2e_test
+# ILM serial lane
+cargo nextest run -j1 --run-ignored ignored-only -p rustfs-scanner -p rustfs \
+  -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))'
+# s3s-e2e black box
+./scripts/e2e-run.sh ./target/debug/rustfs /tmp/rustfs-e2e-data
+```
+
+**Stale binary.** Tests build the `rustfs` binary once and reuse it. To avoid
+rebuilding while iterating on tests, `common.rs` reuses an existing binary when
+running *inside* the e2e test process even if sources changed
+(`can_reuse_inside_e2e`, [`src/common.rs`](src/common.rs) line 98). Downside: if
+you changed **server** code, force a rebuild with
+`cargo build -p rustfs` (or `touch` a source file outside the reuse window)
+before re-running, or CI's freshly built artifact will diverge from your local
+one.
+
+**Port already in use / orphan processes.** A hard-killed run can leak a
+`rustfs` child holding its port. Find and kill it:
+
+```bash
+pkill -f 'target/debug/rustfs' ; pkill -f 'target/release/rustfs'
+```
+
+The `s3s-e2e` CI job selects a random `RUSTFS_TEST_PORT` (see the `e2e-tests`
+job) to dodge this; local single-node tests already use random ports, so a
+lingering orphan is usually the cause of a spurious bind failure.
+
+**`awscurl` not found.** `awscurl`-dependent tests skip gracefully with a
+visible log line (`awscurl_available()`); install `awscurl` to actually run
+them.
+
+## Related
+
+- Crate rules & environment safety: [`AGENTS.md`](AGENTS.md)
+- Sub-suite guides: [`src/kms/README.md`](src/kms/README.md),
+  [`src/policy/README.md`](src/policy/README.md),
+  [`src/protocols/README.md`](src/protocols/README.md),
+  [`src/reliant/README.md`](src/reliant/README.md)
+- Authoritative per-module counts:
+  [`docs/testing/e2e-suite-inventory.md`](../../docs/testing/e2e-suite-inventory.md)
+- Test pyramid & flake policy: [`docs/testing/README.md`](../../docs/testing/README.md)

 ## CI smoke subset (`--profile e2e-smoke`)