diff --git a/crates/e2e_test/AGENTS.md b/crates/e2e_test/AGENTS.md index b23c827f3..032f40e63 100644 --- a/crates/e2e_test/AGENTS.md +++ b/crates/e2e_test/AGENTS.md @@ -2,6 +2,11 @@ Applies to `crates/e2e_test/`. +Contributor guide (how to run suites, add a test, the harness/fixture +inventory, `#[ignore]` classes, and the CI map) lives in +[`README.md`](README.md). This file holds the crate rules that agents must +follow. + ## Test Reliability - Keep end-to-end tests deterministic and environment-aware. diff --git a/crates/e2e_test/README.md b/crates/e2e_test/README.md index 8a304d2ff..f2c3507ab 100644 --- a/crates/e2e_test/README.md +++ b/crates/e2e_test/README.md @@ -1,12 +1,233 @@ # e2e_test -End-to-end test suite for RustFS. Tests spawn a real `rustfs` binary (see -`src/common.rs`) and drive it with the AWS SDK. +End-to-end test suite for RustFS. Each test spawns a **real `rustfs` binary** +(built on demand from the workspace) and drives it over the network with the +AWS SDK (`aws-sdk-s3`), raw HTTP (`reqwest` / `awscurl`), or a protocol client +(FTPS / WebDAV / SFTP). This is the black-box integration layer: exhaustive +end-to-end behavior lives here, unit behavior stays in the source crates +(see [`AGENTS.md`](AGENTS.md)). -> This README currently only documents the CI smoke subset (backlog#1149 -> ci-4). The full contributor guide — how to add a case, fixture helpers, -> `#[ignore]` semantics, harness topologies — is tracked as backlog#1153 -> infra-10 and will extend this file. +The harness lives in [`src/common.rs`](src/common.rs) (single-node + +cluster environments, S3 client construction, `awscurl` helpers) and +[`src/chaos.rs`](src/chaos.rs) (in-process disk fault injection). Crate-wide +test conventions and environment-safety rules are in +[`AGENTS.md`](AGENTS.md); this file is the contributor guide. + +## Module map (~50 modules) + +Registered in [`src/lib.rs`](src/lib.rs). Grouped by concern: + +| Group | Location | What it covers | +| --- | --- | --- | +| **functional** | top-level `*_test.rs` | S3 data plane: `list_objects_*`, `copy_object_*`, `delete_objects_versioning`, `head_object_*`, `checksum_upload`, `compression`, `content_encoding`, `special_chars`, `leading_slash_key`, `create_bucket_region`, `quota`, `data_usage`, `snowball_auto_extract`, `mc_mirror_small_bucket`, `archive_download_integrity`, `version_id_regression`, `delete_marker_migration_semantics` | +| **object_lock** | [`src/object_lock/`](src/object_lock) | Retention / legal-hold / WORM semantics | +| **kms** | [`src/kms/`](src/kms) | SSE-S3 / SSE-KMS / SSE-C, local + Vault backends, multipart encryption. Own guide: [`src/kms/README.md`](src/kms/README.md) | +| **policy** | [`src/policy/`](src/policy), `existing_object_tag_policy_test`, `bucket_policy_check_test`, `anonymous_access_test`, `security_boundary_test`, `multipart_auth_test` | IAM / bucket-policy / STS session policy, policy variables, anonymous access, DoS/SSRF boundaries. Own guide: [`src/policy/README.md`](src/policy/README.md) | +| **protocols** | [`src/protocols/`](src/protocols) | FTPS, WebDAV, SFTP compliance. Fixed ports, own guide: [`src/protocols/README.md`](src/protocols/README.md) | +| **reliant** | [`src/reliant/`](src/reliant) | Tests that reuse an **externally started** server (SQL/select, conditional writes, lifecycle, deleted-object reads, node-interact). Run via [`scripts/run_e2e_tests.sh`](../../scripts/run_e2e_tests.sh); see [`src/reliant/README.md`](src/reliant/README.md) | +| **cluster** | `cluster_concurrency_test`, `stale_multipart_cleanup_cluster_test`, `namespace_lock_quorum_test`, `admin_timeout_regression_test`, `object_lambda_test`, `replication_extension_test` | Multi-node scenarios via `RustFSTestClusterEnvironment` | +| **chaos / reliability** | [`src/chaos.rs`](src/chaos.rs), `reliability_disk_fault_test`, `heal_erasure_disk_rebuild_test`, `server_startup_failfast_test` | Disk offline/replace/corrupt, EC rebuild, heal, fail-fast startup | + +## How to run + +All commands assume repo root. `cargo test` triggers an on-demand build of the +`rustfs` binary from [`src/common.rs`](src/common.rs) (`rustfs_binary_path`) on +first use — the first invocation is slow, later ones reuse the binary. + +```bash +# Whole crate (default = ignored tests skipped) +cargo nextest run -p e2e_test + +# One module +cargo nextest run -p e2e_test -E 'test(list_objects_v2_pagination_test)' + +# PR smoke subset (see "CI smoke subset" below) +cargo nextest run --profile e2e-smoke -p e2e_test + +# ILM serial lane — ignored lifecycle tests, single-threaded (mirrors CI) +cargo nextest run -j1 --run-ignored ignored-only -p rustfs-scanner -p rustfs \ + -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' + +# Protocols suite — fixed ports, MUST be single-threaded, gated by build features +RUSTFS_BUILD_FEATURES=ftps,webdav,sftp \ + cargo test -p e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture +``` + +The protocols suite has its own contract (fixed bind ports 9022–9301, +`--test-threads=1`, feature-gated scheduling) documented in +[`src/protocols/README.md`](src/protocols/README.md). `RUSTFS_BUILD_FEATURES` +selects which features the spawned binary is built with; leave it unset to run +every protocol entry. + +### `#[ignore]` semantics + +Ignored tests are excluded from the default `cargo nextest run` pass because +they need something the default runner does not provide. **Do not maintain a +static count here** — it rots (the set shrinks as ci-13 / ilm-3 activate +suites). Read the live sources instead: + +```bash +rg -n '#\[ignore' crates/e2e_test/src # every ignore + its reason string +``` + +The reason string on each attribute is the classifier. Current classes: + +- **Needs a pre-started server** — `"requires running RustFS server at + localhost:9000"` / `"Connects to existing rustfs server"`. These are the + `reliant/*` and `policy/test_runner` tests; start a server first (e.g. + [`scripts/run_e2e_tests.sh`](../../scripts/run_e2e_tests.sh)) or use + `--run-ignored`. +- **Heavy / external tool** — `"Starts a rustfs server; enable when running + full E2E"`, `"requires awscurl and spawns a real RustFS server"`. Spawn their + own server and/or need `awscurl` on `PATH`. +- **Serial / global-state (ILM lane)** — lifecycle tests bind fixed ports and + share process-global singletons; run via the ILM serial lane above. + +## How to add a test + +### Single-node (the common case) + +Use `RustFSTestEnvironment` from [`src/common.rs`](src/common.rs). It picks a +**random free port** and a **unique temp dir** per instance, so tests are +parallel-safe by construction and clean up on `Drop`: + +```rust +use crate::common::{RustFSTestEnvironment, TEST_BUCKET}; + +#[tokio::test] +async fn my_case() -> Result<(), Box> { + let mut env = RustFSTestEnvironment::new().await?; + env.start_rustfs_server(vec![]).await?; // waits for readiness + let client = env.create_s3_client(); // aws-sdk-s3 Client + env.create_test_bucket(TEST_BUCKET).await?; + // ... drive `client` ... + Ok(()) +} +``` + +Register the module in [`src/lib.rs`](src/lib.rs) under `#[cfg(test)]`. + +### Cluster + +Use `RustFSTestClusterEnvironment::new(node_count)` then `.start()`; it spawns +`node_count` servers over a shared erasure set and hands out per-node S3 clients +via `create_s3_client(idx)` / `create_all_clients()`. See +`cluster_concurrency_test.rs` and `namespace_lock_quorum_test.rs` for patterns. + +### Fixture / helper inventory (`src/common.rs`) + +| Helper | Purpose | +| --- | --- | +| `RustFSTestEnvironment::new` / `with_address` | Single-node env; random or fixed address | +| `start_rustfs_server` / `_with_env` / `_without_cleanup` | Spawn the server (optional extra args / env vars / no pre-cleanup) | +| `wait_for_server_ready` | Poll readiness before issuing requests | +| `create_s3_client` / `create_test_bucket` / `delete_test_bucket` | aws-sdk-s3 client + bucket lifecycle | +| `find_available_port` | Random free port (isolation primitive) | +| `rustfs_binary_path` / `_with_features` | Locate/build the binary; honors `RUSTFS_BUILD_FEATURES` | +| `requested_rustfs_build_features` / `rustfs_build_feature_enabled` | Feature-gate a test to what the binary was built with | +| `awscurl_available` + `execute_awscurl` / `awscurl_post` / `_get` / `_put` / `_delete` / `awscurl_post_sts_form_urlencoded` | Admin/STS API calls via `awscurl` (skip gracefully when absent) | +| `replication_fast_env` | Env vars that shrink replication timers (from repl-4); pass to `start_rustfs_server_with_env` | +| `local_http_client` / `init_logging` | Loopback HTTP client; idempotent tracing init | +| `RustFSTestClusterEnvironment` (`new`/`start`/`start_node`/`stop_node`/`create_all_clients`) | Multi-node harness | +| Constants: `DEFAULT_ACCESS_KEY`, `DEFAULT_SECRET_KEY`, `TEST_BUCKET`, `ENV_RUSTFS_BUILD_FEATURES` | Shared credentials / bucket name / env-var name | + +Fault injectors live in [`src/chaos.rs`](src/chaos.rs): `DiskFaultHarness` +(`take_disk_offline`, `bring_disk_online`, `replace_disk_with_empty`, +`corrupt_object_shard`, `object_metadata_exists_on_disk`, `kill_server` / +`restart_server`) plus `signed_admin_post`. + +### Isolation rules + +- **Port**: never hard-code a port for single-node tests — `new()` allocates a + random one. Fixed ports (protocols, ILM lane) force `--test-threads=1` / a + serial CI lane. +- **Temp dir**: each env owns a temp dir cleaned on `Drop`; do not write under + a shared path. +- **Orphans**: `RustFSTestEnvironment` kills its child on `Drop`, but a panicked + or `kill -9`'d run can leak a `rustfs` process holding a port — see + Troubleshooting. + +### `#[serial]` vs nextest reality + +`serial_test`'s `#[serial]` uses an **in-process** mutex. Under nextest each +test runs in its **own process**, so `#[serial]` does **not** serialize across +tests there — see the header of [`.config/nextest.toml`](../../.config/nextest.toml). +Real cross-test serialization comes from a nextest test-group (`max-threads = +1`) or a `-j1` CI lane. Single-node e2e tests should instead be parallel-safe by +construction (random port + isolated temp dir) and need no serialization. + +## CI map + +`e2e_test` is **excluded** from the main `cargo nextest run --profile ci --all` +pass ([`.github/workflows/ci.yml`](../../.github/workflows/ci.yml) line 158, +`--exclude e2e_test`) — the whole crate is too slow to gate every PR. Subsets +join CI through the nextest profile system only (never as ad-hoc jobs): + +| Suite | Runs where | Status | +| --- | --- | --- | +| Smoke subset (`e2e-smoke` profile) | `e2e-tests` job, every PR | **Active** (backlog#1149 ci-4) | +| `s3s-e2e` black-box | `e2e-tests` + `e2e-tests-rio-v2` jobs | **Active** (external conformance tool) | +| ILM / lifecycle (ignored) | `test-ilm-integration-serial` lane, `-j1` | **Active** (backlog#1148 ilm-1) | +| KMS suite | — | Not in CI yet (backlog#1149 ci-5) | +| Protocols (FTPS/WebDAV/SFTP) | — | Not in CI yet (backlog#1149 ci-7) | +| Cluster / replication e2e | — | Not in CI yet (backlog#1147 repl-1) | +| `reliant/*` (pre-started server) | — | Manual only | + +Links: [`ci.yml`](../../.github/workflows/ci.yml) `e2e-tests` (line 347), +`test-ilm-integration-serial` (line 196). The `e2e-smoke` `default-filter` in +[`.config/nextest.toml`](../../.config/nextest.toml) is the **single wiring +mechanism** — extend that filter (or add a sibling profile) to admit more +tests; do not add e2e jobs to `ci.yml`. repl-1 / ilm-3 are landing in parallel +and may add lanes; keep the table above easy to extend. + +## Troubleshooting + +**Reproduce a CI failure locally** — run the exact profile/lane: + +```bash +# Smoke (e2e-tests job) +cargo nextest run --profile e2e-smoke -p e2e_test +# ILM serial lane +cargo nextest run -j1 --run-ignored ignored-only -p rustfs-scanner -p rustfs \ + -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' +# s3s-e2e black box +./scripts/e2e-run.sh ./target/debug/rustfs /tmp/rustfs-e2e-data +``` + +**Stale binary.** Tests build the `rustfs` binary once and reuse it. To avoid +rebuilding while iterating on tests, `common.rs` reuses an existing binary when +running *inside* the e2e test process even if sources changed +(`can_reuse_inside_e2e`, [`src/common.rs`](src/common.rs) line 98). Downside: if +you changed **server** code, force a rebuild with +`cargo build -p rustfs` (or `touch` a source file outside the reuse window) +before re-running, or CI's freshly built artifact will diverge from your local +one. + +**Port already in use / orphan processes.** A hard-killed run can leak a +`rustfs` child holding its port. Find and kill it: + +```bash +pkill -f 'target/debug/rustfs' ; pkill -f 'target/release/rustfs' +``` + +The `s3s-e2e` CI job selects a random `RUSTFS_TEST_PORT` (see the `e2e-tests` +job) to dodge this; local single-node tests already use random ports, so a +lingering orphan is usually the cause of a spurious bind failure. + +**`awscurl` not found.** `awscurl`-dependent tests skip gracefully with a +visible log line (`awscurl_available()`); install `awscurl` to actually run +them. + +## Related + +- Crate rules & environment safety: [`AGENTS.md`](AGENTS.md) +- Sub-suite guides: [`src/kms/README.md`](src/kms/README.md), + [`src/policy/README.md`](src/policy/README.md), + [`src/protocols/README.md`](src/protocols/README.md), + [`src/reliant/README.md`](src/reliant/README.md) +- Authoritative per-module counts: + [`docs/testing/e2e-suite-inventory.md`](../../docs/testing/e2e-suite-inventory.md) +- Test pyramid & flake policy: [`docs/testing/README.md`](../../docs/testing/README.md) ## CI smoke subset (`--profile e2e-smoke`)