rustfs/crates/e2e_test/README.md

# e2e_test

End-to-end test suite for RustFS. Each test spawns a **real `rustfs` binary**
(built and identified before the test invocation) and drives it over the network with the
AWS SDK (`aws-sdk-s3`), raw HTTP (`reqwest` / `awscurl`), or a protocol client
(FTPS / WebDAV / SFTP). This is the black-box integration layer: exhaustive
end-to-end behavior lives here, unit behavior stays in the source crates
(see [`AGENTS.md`](AGENTS.md)).

The harness lives in [`src/common.rs`](src/common.rs) (single-node +
cluster environments, S3 client construction, `awscurl` helpers) and
[`src/chaos.rs`](src/chaos.rs) (in-process disk fault injection). Crate-wide
test conventions and environment-safety rules are in
[`AGENTS.md`](AGENTS.md); this file is the contributor guide.

## Module map (~50 modules)

Registered in [`src/lib.rs`](src/lib.rs). Grouped by concern:

| Group | Location | What it covers |
| --- | --- | --- |
| **functional** | top-level `*_test.rs` | S3 data plane: `list_objects_*`, `copy_object_*`, `delete_objects_versioning`, `head_object_*`, `checksum_upload`, `compression`, `content_encoding`, `special_chars`, `leading_slash_key`, `create_bucket_region`, `quota`, `data_usage`, `snowball_auto_extract`, `mc_mirror_small_bucket`, `archive_download_integrity`, `version_id_regression`, `delete_marker_migration_semantics` |
| **object_lock** | [`src/object_lock/`](src/object_lock) | Retention / legal-hold / WORM semantics |
| **kms** | [`src/kms/`](src/kms) | SSE-S3 / SSE-KMS / SSE-C, local + Vault backends, multipart encryption. Own guide: [`src/kms/README.md`](src/kms/README.md) |
| **policy** | [`src/policy/`](src/policy), `existing_object_tag_policy_test`, `bucket_policy_check_test`, `anonymous_access_test`, `security_boundary_test`, `multipart_auth_test` | IAM / bucket-policy / STS session policy, policy variables, anonymous access, DoS/SSRF boundaries. Own guide: [`src/policy/README.md`](src/policy/README.md) |
| **protocols** | [`src/protocols/`](src/protocols) | FTPS, WebDAV, SFTP compliance. Fixed ports, own guide: [`src/protocols/README.md`](src/protocols/README.md) |
| **reliant** | [`src/reliant/`](src/reliant) | Tests that reuse an **externally started** server (SQL/select, conditional writes, lifecycle, deleted-object reads, node-interact). Run via [`scripts/run_e2e_tests.sh`](../../scripts/run_e2e_tests.sh); see [`src/reliant/README.md`](src/reliant/README.md) |
| **cluster** | `cluster_concurrency_test`, `stale_multipart_cleanup_cluster_test`, `namespace_lock_quorum_test`, `admin_timeout_regression_test`, `object_lambda_test`, `replication_extension_test`, `tier_stats_cluster_test` | Multi-node scenarios via `RustFSTestClusterEnvironment` |
| **chaos / reliability** | [`src/chaos.rs`](src/chaos.rs), `reliability_disk_fault_test`, `heal_erasure_disk_rebuild_test`, `server_startup_failfast_test` | Disk offline/replace/corrupt, EC rebuild, heal, fail-fast startup |
| **upgrade compatibility** | `upgrade_compatibility_test` | Pinned previous-release writes followed by current-build reads on the same data directory |

## How to run

All commands assume repo root and Python 3.9 or newer on Linux or macOS. Build the server once through the provenance entry point, then run the test command through the same script:

```bash
python3 scripts/e2e_binary.py build --features e2e-test-hooks

# Whole crate (ignored tests remain skipped)
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run -p e2e_test

# One module
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run -p e2e_test -E 'test(list_objects_v2_pagination_test)'

# PR smoke subset
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke -p e2e_test
```

`build` records the source contents, HEAD, resolved Cargo features, profile, toolchain, and binary SHA-256 beside the executable in `rustfs.e2e.json`. `run` validates that identity before and after the command, preserves command failures, and removes its temporary run receipt on completion. The Rust harness checks that receipt before starting each server; it never compiles a server inside a test process. Source or binary changes during a run invalidate the result, even when the test command succeeds. Use an isolated worktree and keep it unchanged until the command finishes.

The additional `--features` arguments must match between `build` and `run`; Cargo defaults remain enabled. The wrapper supplies `RUSTFS_BUILD_FEATURES` from Cargo's resolved feature list, including features enabled by `full`. Protocol helpers require a subset of that list. `CARGO_TARGET_DIR` and `--profile release` are supported. An in-workspace target directory must be Git-ignored; tracked files are always included in the source identity. `build --bins` preserves CI lanes that compile all RustFS binary targets. For a downloaded artifact, copy both the executable and its sidecar, then use `run`; do not generate a new identity for an arbitrary prebuilt binary. `CARGO_BIN_EXE_rustfs` cannot override the verified executable.

Each build/run holds an exclusive `rustfs.e2e.lock` marker beside the binary; concurrent wrappers fail immediately. Use a private target directory and do not run ordinary Cargo builds against it while tests are active: Cargo does not honor this marker. Interrupted runs fail and terminate their command group. After an uncatchable kill, inspect the PID recorded in a leftover marker and remove it only after confirming its owner has stopped. Embedded file symlinks are hashed through their target; embedded directory symlinks are rejected because their contents cannot be enumerated safely by this entry point.

The protocols suite has its own fixed-port and single-worker contract in [`src/protocols/README.md`](src/protocols/README.md). Use its command under [Troubleshooting](#troubleshooting).

### `#[ignore]` semantics

Ignored tests are excluded from the default `cargo nextest run` pass because
they need something the default runner does not provide. **Do not maintain a
static count here** — it rots (the set shrinks as ci-13 / ilm-3 activate
suites). Read the live sources instead:

```bash
rg -n '#\[ignore' crates/e2e_test/src   # every ignore + its reason string
```

The reason string on each attribute is the classifier. Current classes:

- **Needs a pre-started server** — `"requires running RustFS server at
  localhost:9000"` / `"Connects to existing rustfs server"`. These are the
  `reliant/*` tests; start a server first (e.g.
  [`scripts/run_e2e_tests.sh`](../../scripts/run_e2e_tests.sh)) or use
  `--run-ignored`.
- **Heavy / external tool** — `"Starts a rustfs server; enable when running
  full E2E"`, `"requires awscurl and spawns a real RustFS server"`. Spawn their
  own server and/or need `awscurl` on `PATH`.
- **Serial / global-state (ILM lane)** — lifecycle tests bind fixed ports and
  share process-global singletons; run via the ILM serial lane above.

## How to add a test

### Single-node (the common case)

Use `RustFSTestEnvironment` from [`src/common.rs`](src/common.rs). It picks a
**random free port** and a **unique temp dir** per instance, so tests are
parallel-safe by construction and clean up on `Drop`:

```rust
use crate::common::{RustFSTestEnvironment, TEST_BUCKET};

#[tokio::test]
async fn my_case() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    let mut env = RustFSTestEnvironment::new().await?;
    env.start_rustfs_server(vec![]).await?;       // waits for readiness
    let client = env.create_s3_client();          // aws-sdk-s3 Client
    env.create_test_bucket(TEST_BUCKET).await?;
    // ... drive `client` ...
    Ok(())
}
```

Register the module in [`src/lib.rs`](src/lib.rs) under `#[cfg(test)]`.

### Cluster

Use `RustFSTestClusterEnvironment::new(node_count)` then `.start()`; it spawns
`node_count` servers over a shared erasure set and hands out per-node S3 clients
via `create_s3_client(idx)` / `create_all_clients()`. See
`cluster_concurrency_test.rs` and `namespace_lock_quorum_test.rs` for patterns.

### Fixture / helper inventory (`src/common.rs`)

| Helper | Purpose |
| --- | --- |
| `RustFSTestEnvironment::new` / `with_address` | Single-node env; random or fixed address |
| `start_rustfs_server` / `_with_env` / `_without_cleanup` | Spawn the server (optional extra args / env vars / no pre-cleanup) |
| `wait_for_server_ready` | Poll readiness before issuing requests |
| `create_s3_client` / `create_test_bucket` / `delete_test_bucket` | aws-sdk-s3 client + bucket lifecycle |
| `find_available_port` | Random free port (isolation primitive) |
| `rustfs_binary_path` / `_with_features` | Verify this run's binary receipt and required feature subset |
| `requested_rustfs_build_features` / `rustfs_build_feature_enabled` | Feature-gate a test to what the binary was built with |
| `execute_awscurl` / `awscurl_post` / `_get` / `_put` / `_delete` / `awscurl_post_sts_form_urlencoded` | Admin/STS API calls via `awscurl`; missing binaries are test failures |
| `replication_fast_env` | Env vars that shrink replication timers (from repl-4); pass to `start_rustfs_server_with_env` |
| `local_http_client` / `init_logging` | Loopback HTTP client; idempotent tracing init |
| `RustFSTestClusterEnvironment` (`new`/`start`/`start_node`/`stop_node`/`create_all_clients`) | Multi-node harness |
| Constants: `DEFAULT_ACCESS_KEY`, `DEFAULT_SECRET_KEY`, `TEST_BUCKET`, `ENV_RUSTFS_BUILD_FEATURES` | Shared credentials / bucket name / env-var name |

Fault injectors live in [`src/chaos.rs`](src/chaos.rs): `DiskFaultHarness`
(`take_disk_offline`, `bring_disk_online`, `replace_disk_with_empty`,
`corrupt_object_shard`, `object_metadata_exists_on_disk`, `kill_server` /
`restart_server`) plus `signed_admin_post`.

### Isolation rules

- **Port**: never hard-code a port for single-node tests — `new()` allocates a
  random one. Fixed ports (protocols, ILM lane) force `--test-threads=1` / a
  serial CI lane.
- **Temp dir**: each env owns a temp dir cleaned on `Drop`; do not write under
  a shared path.
- **Orphans**: `RustFSTestEnvironment` kills its child on `Drop`, but a panicked
  or `kill -9`'d run can leak a `rustfs` process holding a port — see
  Troubleshooting.

### `#[serial]` vs nextest reality

`serial_test`'s `#[serial]` uses an **in-process** mutex. Under nextest each
test runs in its **own process**, so `#[serial]` does **not** serialize across
tests there — see the header of [`.config/nextest.toml`](../../.config/nextest.toml).
Real cross-test serialization comes from a nextest test-group (`max-threads =
1`) or a `-j1` CI lane. Single-node e2e tests should instead be parallel-safe by
construction (random port + isolated temp dir) and need no serialization.

## CI map

`e2e_test` is **excluded** from the main `cargo nextest run --profile ci --all`
pass (`--exclude e2e_test`) — the whole crate is too slow to gate every PR.
Subsets join CI through nextest profiles; the fixed-port protocol suite uses
the same profile for membership and execution with one nightly worker.

| Suite | Runs where | Status |
| --- | --- | --- |
| Smoke subset (`e2e-smoke` profile) | `e2e-tests` job, every PR | **Active** (backlog#1149 ci-4) |
| Full single-node suite (`e2e-full` profile) | `e2e-full` job, merge queue + main | **Active** (backlog#1149 ci-5) |
| `s3s-e2e` black-box | `e2e-tests` + `e2e-tests-rio-v2` jobs | **Active** (external conformance tool) |
| ILM / lifecycle (ignored) | `test-ilm-integration-serial` lane, `-j1` | **Active** (backlog#1148 ilm-1) |
| KMS suite | `e2e-full` job, merge queue + main | **Active** |
| Direct and mixed-version rolling upgrades from pinned previous release | `e2e-upgrade.yml`, storage-sensitive PRs + release tags + weekly | **Active** |
| Cluster faults (`e2e-nightly` profile) | consolidated nightly workflow | **Active** (backlog#1149 ci-7) |
| Protocols (FTPS/WebDAV/SFTP) | consolidated nightly workflow, serial | **Active** (backlog#1149 ci-7) |
| Replication (fast subset) | `e2e-smoke` profile, `e2e-tests` job, every PR | **Active** (backlog#1147 repl-1) |
| Replication (slow + multi-node) | `e2e-repl-nightly` profile, consolidated nightly workflow | **Active** (backlog#1147 repl-1) |
| `reliant/*` | 19 tests in PR smoke; remaining default tests in `e2e-full` | **Active** except `#[ignore]` |

The profile filters in [`.config/nextest.toml`](../../.config/nextest.toml) are
the wiring source of truth. Committed test-ID digests under
`.config/e2e-*-selection.txt` make every membership change explicit.

## Troubleshooting

**Reproduce a CI failure locally** — run the exact profile/lane:

```bash
# Smoke, full, and cluster lanes share a server with fault-test hooks.
python3 scripts/e2e_binary.py build --features e2e-test-hooks
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke -p e2e_test
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-full -p e2e_test
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-nightly -p e2e_test

# Replication nightly uses the default server; awscurl is required for STS.
python3 scripts/e2e_binary.py build
python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-repl-nightly -p e2e_test

# Protocol nightly owns fixed ports.
python3 scripts/e2e_binary.py build --features ftps,webdav,sftp
python3 scripts/e2e_binary.py run --features ftps,webdav,sftp -- cargo nextest run -j 1 --profile e2e-protocols -p e2e_test --no-capture

# The ILM serial lane does not use this server harness.
cargo nextest run -j1 --run-ignored ignored-only -p rustfs-scanner -p rustfs \
  -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))'
```

**Stale or unverified binary.** Re-run the matching `build` command after changing source or features, then invoke tests through `run`. A missing receipt, copied old executable, or mismatched build identity is a prerequisite failure. Bare Cargo invocations that start a server deliberately fail; unit tests that do not start a server can still run directly.

**Port already in use / orphan processes.** A hard-killed run can leak a
`rustfs` child holding its port. Find and kill it:

```bash
pkill -f 'target/debug/rustfs' ; pkill -f 'target/release/rustfs'
```

The `s3s-e2e` CI job selects a random `RUSTFS_TEST_PORT` (see the `e2e-tests`
job) to dodge this; local single-node tests already use random ports, so a
lingering orphan is usually the cause of a spurious bind failure.

**`awscurl` not found.** `awscurl`-dependent tests fail closed with a process
spawn error. Install the pinned CI version before running their profiles.

## Related

- Crate rules & environment safety: [`AGENTS.md`](AGENTS.md)
- Sub-suite guides: [`src/kms/README.md`](src/kms/README.md),
  [`src/policy/README.md`](src/policy/README.md),
  [`src/protocols/README.md`](src/protocols/README.md),
  [`src/reliant/README.md`](src/reliant/README.md)
- Per-module counts: `cargo nextest list -p e2e_test --profile <profile>`
  (one-liner in [`docs/testing/README.md`](../../docs/testing/README.md))
- Test pyramid & flake policy: [`docs/testing/README.md`](../../docs/testing/README.md)

## CI smoke subset (`--profile e2e-smoke`)

A subset of this crate runs on every PR via the `e2e-tests` job:

```bash
cargo nextest run --profile e2e-smoke -p e2e_test
```

The selection lives in `.config/nextest.toml` under `[profile.e2e-smoke]`
(`default-filter`). That filter is the **single wiring mechanism** for e2e
tests in CI — extend it (or add a sibling profile) instead of adding new e2e
jobs to `ci.yml`.

### Admission criteria for the smoke subset

A test module may join the smoke filter only if every test in it is:

1. **Fast** — single-digit seconds per test; the whole subset must keep the
   `e2e-tests` job ≤ 20 minutes.
2. **Single-node** — spawns its own server via
   `RustFSTestEnvironment`/`start_rustfs_server` on a random port with an
   isolated temp dir. No `RustFSTestClusterEnvironment`, no fixed ports.
3. **Hermetic dependencies** — no pre-started server at `localhost:9000`, no
   Vault, and no fixed protocol ports. Any required CLI must be pinned and
   installed by the workflow; a missing CLI must fail the test.
4. **Not `#[ignore]`** — ignored tests are activation work (backlog#1149
   ci-13 / backlog#1148 ilm-3), not smoke candidates.

Note on `#[serial]`: nextest runs each test in its own process, so
`serial_test`'s in-process mutex does **not** serialize across tests there
(see the header of `.config/nextest.toml`). Smoke tests must therefore be
parallel-safe by construction (random port + isolated temp dir), which the
current subset is.

### Test inventory

Per-module counts are not committed; list them with
`cargo nextest list -p e2e_test --profile <profile>` (the result is
platform-dependent because some modules are linux-only; the `jq` one-liner is
in `docs/testing/README.md`). When a profile membership change is
intentional, review its JSON listing before updating the matching
`.config/e2e-*-selection.txt` test-ID digest. Update only the platform that
produced the listing:

```bash
python3 scripts/check_test_wiring.py --update-profile e2e-full /path/to/listing.json linux
```