rustfs/crates/e2e_test

e2e_test

End-to-end test suite for RustFS. Each test spawns a real rustfs binary (built and identified before the test invocation) and drives it over the network with the AWS SDK (aws-sdk-s3), raw HTTP (reqwest / awscurl), or a protocol client (FTPS / WebDAV / SFTP). This is the black-box integration layer: exhaustive end-to-end behavior lives here, unit behavior stays in the source crates (see AGENTS.md).

The harness lives in src/common.rs (single-node + cluster environments, S3 client construction, awscurl helpers) and src/chaos.rs (in-process disk fault injection). Crate-wide test conventions and environment-safety rules are in AGENTS.md; this file is the contributor guide.

Module map (~50 modules)

Registered in src/lib.rs. Grouped by concern:

Group	Location	What it covers
functional	top-level *_test.rs	S3 data plane: list_objects_*, copy_object_*, delete_objects_versioning, head_object_*, checksum_upload, compression, content_encoding, special_chars, leading_slash_key, create_bucket_region, quota, data_usage, snowball_auto_extract, mc_mirror_small_bucket, archive_download_integrity, version_id_regression, delete_marker_migration_semantics
object_lock	src/object_lock/	Retention / legal-hold / WORM semantics
kms	src/kms/	SSE-S3 / SSE-KMS / SSE-C, local + Vault backends, multipart encryption. Own guide: src/kms/README.md
policy	src/policy/, existing_object_tag_policy_test, bucket_policy_check_test, anonymous_access_test, security_boundary_test, multipart_auth_test	IAM / bucket-policy / STS session policy, policy variables, anonymous access, DoS/SSRF boundaries. Own guide: src/policy/README.md
protocols	src/protocols/	FTPS, WebDAV, SFTP compliance. Fixed ports, own guide: src/protocols/README.md
reliant	src/reliant/	Tests that reuse an externally started server (SQL/select, conditional writes, lifecycle, deleted-object reads, node-interact). Run via scripts/run_e2e_tests.sh; see src/reliant/README.md
cluster	cluster_concurrency_test, stale_multipart_cleanup_cluster_test, namespace_lock_quorum_test, admin_timeout_regression_test, object_lambda_test, replication_extension_test, tier_stats_cluster_test	Multi-node scenarios via RustFSTestClusterEnvironment
distributed 4×4	src/distributed/	Storage-sensitive PR and nightly e2e-distributed lane: S3, object lock/WORM, versioning, bucket/site replication, quota, expand/decommission/rebalance, concurrency, chaos, 4-node upgrade of historical data and IAM AK/SK. Map: docs/testing/distributed-e2e.md
chaos / reliability	src/chaos.rs, reliability_disk_fault_test, heal_erasure_disk_rebuild_test, server_startup_failfast_test	Disk offline/replace/corrupt, EC rebuild, heal, fail-fast startup
upgrade compatibility	upgrade_compatibility_test	Pinned previous-release writes followed by current-build reads on the same data directory

The external-tool storage_metric_ownership_test validates the OTLP/Collector/Prometheus path, including a rolling upgrade and node failures. See the storage metrics guide for its required binaries and focused command.

How to run

All commands assume repo root and Python 3.9 or newer on Linux or macOS. Build the server once through the provenance entry point, then run the test command through the same script:

python3 scripts/e2e_binary.py build --features e2e-test-hooks

# Whole crate (ignored tests remain skipped)
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run -p e2e_test

# One module
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run -p e2e_test -E 'test(list_objects_v2_pagination_test)'

# PR smoke subset
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke -p e2e_test

Root-heal interruption scenarios use a test-only commit barrier, so build and run them with e2e-test-hooks:

python3 scripts/e2e_binary.py build --features e2e-test-hooks
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run -p e2e_test -E 'test(heal_erasure_disk_rebuild_test)'

build records the source contents, HEAD, resolved Cargo features, profile, toolchain, and binary SHA-256 beside the executable in rustfs.e2e.json. run validates that identity before and after the command, preserves command failures, and removes its temporary run receipt on completion. The Rust harness checks that receipt before starting each server; it never compiles a server inside a test process. Source or binary changes during a run invalidate the result, even when the test command succeeds. Use an isolated worktree and keep it unchanged until the command finishes.

The additional --features arguments must match between build and run; Cargo defaults remain enabled. The wrapper supplies RUSTFS_BUILD_FEATURES from Cargo's resolved feature list, including features enabled by full. Protocol helpers require a subset of that list. CARGO_TARGET_DIR and --profile release are supported. An in-workspace target directory must be Git-ignored; tracked files are always included in the source identity. build --bins preserves CI lanes that compile all RustFS binary targets. For a downloaded artifact, copy both the executable and its sidecar, then use run; do not generate a new identity for an arbitrary prebuilt binary. CARGO_BIN_EXE_rustfs cannot override the verified executable.

Each build/run holds an exclusive rustfs.e2e.lock marker beside the binary; concurrent wrappers fail immediately. Use a private target directory and do not run ordinary Cargo builds against it while tests are active: Cargo does not honor this marker. Interrupted runs fail and terminate their command group. After an uncatchable kill, inspect the PID recorded in a leftover marker and remove it only after confirming its owner has stopped. Embedded file symlinks are hashed through their target; embedded directory symlinks are rejected because their contents cannot be enumerated safely by this entry point.

The protocols suite has its own fixed-port and single-worker contract in src/protocols/README.md. Use its command under Troubleshooting.

#[ignore] semantics

Ignored tests are excluded from the default cargo nextest run pass because they need something the default runner does not provide. Do not maintain a static count here — it rots (the set shrinks as ci-13 / ilm-3 activate suites). Read the live sources instead:

rg -n '#\[ignore' crates/e2e_test/src   # every ignore + its reason string

The reason string on each attribute is the classifier. Current classes:

• Needs a pre-started server — "requires running RustFS server at localhost:9000" / "Connects to existing rustfs server". These are the reliant/* tests; start a server first (e.g. scripts/run_e2e_tests.sh) or use --run-ignored.
• Heavy / external tool — "Starts a rustfs server; enable when running full E2E", "requires awscurl and spawns a real RustFS server". Spawn their own server and/or need awscurl on PATH.
• Serial / global-state (ILM lane) — lifecycle tests bind fixed ports and share process-global singletons; run via the ILM serial lane above.

How to add a test

Single-node (the common case)

Use RustFSTestEnvironment from src/common.rs. It picks a random free port and a unique temp dir per instance, so tests are parallel-safe by construction and clean up on Drop:

use crate::common::{RustFSTestEnvironment, TEST_BUCKET};

#[tokio::test]
async fn my_case() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    let mut env = RustFSTestEnvironment::new().await?;
    env.start_rustfs_server(vec![]).await?;       // waits for readiness
    let client = env.create_s3_client();          // aws-sdk-s3 Client
    env.create_test_bucket(TEST_BUCKET).await?;
    // ... drive `client` ...
    Ok(())
}

Register the module in src/lib.rs under #[cfg(test)].

Cluster

Use RustFSTestClusterEnvironment::new(node_count) then .start(); it spawns node_count servers over a shared erasure set and hands out per-node S3 clients via create_s3_client(idx) / create_all_clients(). See cluster_concurrency_test.rs and namespace_lock_quorum_test.rs for patterns.

Fixture / helper inventory (src/common.rs)

Helper	Purpose
RustFSTestEnvironment::new / with_address	Single-node env; random or fixed address
start_rustfs_server / _with_env / _without_cleanup	Spawn the server (optional extra args / env vars / no pre-cleanup)
wait_for_server_ready	Poll readiness before issuing requests
create_s3_client / create_test_bucket / delete_test_bucket	aws-sdk-s3 client + bucket lifecycle
find_available_port	Random free port (isolation primitive)
rustfs_binary_path / _with_features	Verify this run's binary receipt and required feature subset
requested_rustfs_build_features / rustfs_build_feature_enabled	Feature-gate a test to what the binary was built with
execute_awscurl / awscurl_post / _get / _put / _delete / awscurl_post_sts_form_urlencoded	Admin/STS API calls via awscurl; missing binaries are test failures
replication_fast_env	Env vars that shrink replication timers (from repl-4); pass to start_rustfs_server_with_env
local_http_client / init_logging	Loopback HTTP client; idempotent tracing init
RustFSTestClusterEnvironment (new/start/start_node/stop_node/create_all_clients)	Multi-node harness
Constants: DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, TEST_BUCKET, ENV_RUSTFS_BUILD_FEATURES	Shared credentials / bucket name / env-var name

Fault injectors live in src/chaos.rs: DiskFaultHarness (take_disk_offline, bring_disk_online, replace_disk_with_empty, corrupt_object_shard, object_metadata_exists_on_disk, kill_server / restart_server) plus signed_admin_post.

Isolation rules

• Port: never hard-code a port for single-node tests — new() allocates a random one. Fixed ports (protocols, ILM lane) force --test-threads=1 / a serial CI lane.
• Temp dir: each env owns a temp dir cleaned on Drop; do not write under a shared path.
• Orphans: RustFSTestEnvironment kills its child on Drop, but a panicked or kill -9'd run can leak a rustfs process holding a port — see Troubleshooting.

#[serial] vs nextest reality

serial_test's #[serial] uses an in-process mutex. Under nextest each test runs in its own process, so #[serial] does not serialize across tests there — see the header of .config/nextest.toml. Real cross-test serialization comes from a nextest test-group (max-threads = 1) or a -j1 CI lane. Single-node e2e tests should instead be parallel-safe by construction (random port + isolated temp dir) and need no serialization.

CI map

e2e_test is excluded from the main cargo nextest run --profile ci --all pass (--exclude e2e_test) — the whole crate is too slow to gate every PR. Subsets join CI through nextest profiles; the fixed-port protocol suite uses the same profile for membership and execution with one nightly worker.

Suite	Runs where	Status
Smoke subset (e2e-smoke profile)	e2e-tests job, every PR	Active (backlog#1149 ci-4)
Full single-node suite (e2e-full profile)	e2e-full job, merge queue + main	Active (backlog#1149 ci-5)
s3s-e2e black-box	e2e-tests + e2e-tests-rio-v2 jobs	Active (external conformance tool)
ILM / lifecycle (ignored)	test-ilm-integration-serial lane, -j1	Active (backlog#1148 ilm-1)
KMS suite	e2e-full job, merge queue + main	Active
Direct and mixed-version rolling upgrades from pinned previous release	e2e-upgrade.yml, storage-sensitive PRs + release tags + weekly	Active
Cluster faults (e2e-nightly profile)	consolidated nightly workflow	Active (backlog#1149 ci-7)
Distributed 4-node 4-disk (e2e-distributed profile)	.github/workflows/e2e-distributed.yml	Active (storage-sensitive PR / nightly / dispatch)
Protocols (FTPS/WebDAV/SFTP)	consolidated nightly workflow, serial	Active (backlog#1149 ci-7)
Replication (fast subset)	e2e-smoke profile, e2e-tests job, every PR	Active (backlog#1147 repl-1)
Replication (slow + multi-node)	e2e-repl-nightly profile, consolidated nightly workflow	Active (backlog#1147 repl-1)
reliant/*	19 tests in PR smoke; remaining default tests in e2e-full	Active except #[ignore]

The profile filters in .config/nextest.toml are the wiring source of truth. Committed test-ID digests under .config/e2e-*-selection.txt make every membership change explicit.

Troubleshooting

Endpoint blackhole scenario skipped — heal_erasure_disk_rebuild_test::tests::test_cluster_root_heal_recovers_after_target_endpoint_blackhole installs a loopback iptables DROP rule and therefore needs CAP_NET_ADMIN (root or passwordless sudo -n iptables). A host where iptables is missing or cannot read the OUTPUT chain (typical inside an unprivileged container, where the nf_tables backend reports "Permission denied" even under sudo) logs a heal_interruption_skipped warning and returns without exercising heal. Set RUSTFS_E2E_REQUIRE_NET_FAULT_INJECTION=1 on lanes that do provision the capability so a broken runner fails instead of skipping.

Reproduce a CI failure locally — run the exact profile/lane:

# Smoke, full, and cluster lanes share a server with fault-test hooks.
python3 scripts/e2e_binary.py build --features e2e-test-hooks
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke -p e2e_test
python3 scripts/e2e_binary.py run --binary "$RUSTFS_E2E_STARTUP_CAS_BINARY" --features e2e-test-hooks -- cargo nextest run --profile e2e-full -p e2e_test
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-nightly -p e2e_test

# Distributed 4-node 4-disk lane uses the default server.
# Upgrade cases require RUSTFS_UPGRADE_SOURCE_BINARY and fail closed without it.
python3 scripts/e2e_binary.py build
python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-distributed -p e2e_test

# Replication nightly uses the default server; awscurl is required for STS.
python3 scripts/e2e_binary.py build
python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-repl-nightly -p e2e_test

# Protocol nightly owns fixed ports.
python3 scripts/e2e_binary.py build --features ftps,webdav,sftp
python3 scripts/e2e_binary.py run --features ftps,webdav,sftp -- cargo nextest run -j 1 --profile e2e-protocols -p e2e_test --no-capture

# The ILM serial lane does not use this server harness.
cargo nextest run -j1 --run-ignored ignored-only -p rustfs-scanner -p rustfs \
  -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))'

The full lane also requires the startup-CAS build manifest generated by the Build debug binary step in .github/workflows/ci.yml. Preserve that binary and both sidecars as its Preserve startup CAS binary input step does, and use the same RUSTFS_E2E_STARTUP_CAS_* environment as Run e2e full suite. A generic local build alone does not supply that fixture evidence.

Stale or unverified binary. Re-run the matching build command after changing source or features, then invoke tests through run. A missing receipt, copied old executable, or mismatched build identity is a prerequisite failure. Bare Cargo invocations that start a server deliberately fail; unit tests that do not start a server can still run directly.

Port already in use / orphan processes. A hard-killed run can leak a rustfs child holding its port. Find and kill it:

pkill -f 'target/debug/rustfs' ; pkill -f 'target/release/rustfs'

The s3s-e2e CI job selects a random RUSTFS_TEST_PORT (see the e2e-tests job) to dodge this; local single-node tests already use random ports, so a lingering orphan is usually the cause of a spurious bind failure.

awscurl not found. awscurl-dependent tests fail closed with a process spawn error. Install the pinned CI version before running their profiles.

Related

• Crate rules & environment safety: AGENTS.md
• Sub-suite guides: src/kms/README.md, src/policy/README.md, src/protocols/README.md, src/reliant/README.md
• Per-module counts: cargo nextest list -p e2e_test --profile <profile> (one-liner in docs/testing/README.md)
• Test pyramid & flake policy: docs/testing/README.md

CI smoke subset (--profile e2e-smoke)

A subset of this crate runs on every PR via the e2e-tests job:

python3 scripts/e2e_binary.py build --features e2e-test-hooks
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke -p e2e_test

The selection lives in .config/nextest.toml under [profile.e2e-smoke] (default-filter). That filter is the single wiring mechanism for e2e tests in CI — extend it (or add a sibling profile) instead of adding new e2e jobs to ci.yml.

Admission criteria for the smoke subset

A test module may join the smoke filter only if every test in it is:

1. Fast — single-digit seconds per test; the whole subset must keep the e2e-tests job ≤ 20 minutes.
2. Single-node — spawns its own server via RustFSTestEnvironment/start_rustfs_server on a random port with an isolated temp dir. No RustFSTestClusterEnvironment, no fixed ports.
3. Hermetic dependencies — no pre-started server at localhost:9000, no Vault, and no fixed protocol ports. Any required CLI must be pinned and installed by the workflow; a missing CLI must fail the test.
4. Not #[ignore] — ignored tests are activation work (backlog#1149 ci-13 / backlog#1148 ilm-3), not smoke candidates.

Note on #[serial]: nextest runs each test in its own process, so serial_test's in-process mutex does not serialize across tests there (see the header of .config/nextest.toml). Smoke tests must therefore be parallel-safe by construction (random port + isolated temp dir), which the current subset is.

Test inventory

Per-module counts are not committed; list them with cargo nextest list -p e2e_test --profile <profile> (the result is platform-dependent because some modules are linux-only; the jq one-liner is in docs/testing/README.md). When a profile membership change is intentional, review its JSON listing before updating the matching .config/e2e-*-selection.txt test-ID digest. Update only the platform that produced the listing:

python3 scripts/check_test_wiring.py --update-profile e2e-full /path/to/listing.json linux