唐小鸭
d20476a66c
SSE-C ciphertext passthrough replicates via X-Rustfs-Replication-* transport
headers. A MinIO/generic-S3 target silently discards them, storing bare
ciphertext with no decryption material — yet the PUT succeeded, so the object
reported COMPLETED with a silently unreadable replica (backlog#1675 N2).
Fail-closed design:
- SsecPassthroughCapability {Unknown, Supported, Unsupported} cached in
BucketTargetSys per target ARN with a recording timestamp. Entries reset
whenever the target is rebuilt, edited, or removed (arn_remotes_map
lifecycle) and expire after SSEC_PASSTHROUGH_CAPABILITY_TTL (10 minutes):
an expired verdict in either direction is re-earned through the audit, so
an Unsupported target recovers automatically after an upgrade (at most one
wasted PUT+HEAD audit per bad target per TTL window) and a Supported
verdict cannot outlive a backend swapped behind the same endpoint.
- Replication worker (replicate_object and replicate_all): fresh Unsupported
targets never receive the PUT — the attempt fails immediately into the
normal MRF retry channel with a "run ?replication-check to re-probe" hint.
Unknown or expired verdicts are audited: after the PUT the worker HEADs
the replica back through the replication-check channel (source version id
mapped through resolve_read_api_version_id, so null-version objects audit
correctly) and requires SSE-C evidence (the echoed customer-algorithm
header); missing evidence records Unsupported and fails the attempt.
Convergence HEADs are audited the same way, so a broken ciphertext replica
from an earlier attempt can never launder itself into COMPLETED via an
ETag match. The gate/evidence policy is pure (replication_target_boundary,
staleness folded in as an input) for the M2 worker migration.
- replication-check grows an SsecPassthrough probe phase: a probe PUT
carrying the live transport-header shape, HEAD-back for evidence, and a
machine-readable Code BucketRemoteSsecPassthroughUnsupported on failure.
The probe verdict is synced into the runtime capability cache. Unlike
VersionFidelity, a failed SsecPassthrough phase does NOT fail the target
overall — it is a capability limit, not a broken replication contract,
and a plaintext-only deployment against such a target must not turn red.
- fake_s3_target: default mode now models a RustFS target (stores the
transport headers, echoes SSE-C evidence); the new
drop_unlisted_replication_headers mode models MinIO. The journal records
whether a request carried transport headers.
Receiver-echo verification: the replication-check HEAD exemption only skips
SSE-C key validation; the response has always built sse-customer-algorithm
from stored metadata (rustfs/src/app/object_usecase.rs), so no receiver
change was needed — pinned end to end by the replication-check e2e against
a real RustFS target.
Rolling-upgrade constraint: RustFS targets older than the replication-check
HEAD exemption (#5898) answer the audit HEAD without SSE-C evidence (or fail
it outright), so SSE-C replication to such targets reports FAILED. This is
deliberate — FAILED-and-retryable beats a silently undecryptable replica —
and self-heals: once the target is upgraded, the next TTL expiry (or a
manual ?replication-check re-probe) re-audits and records Supported.
Plaintext and managed-SSE replication are unaffected. The capability cache
is per-node; each node audits independently.
Known limitations:
- The audit judges evidence from the echoed customer-algorithm header only.
A hypothetical target that preserves that one header while dropping other
transport headers (partial-drop) would pass the audit; no known target
behaves this way — observed targets drop the whole unknown-header family.
- A mixed-version target cluster can flap the verdict between audits routed
to different target nodes until the rollout completes; the TTL bounds how
long each stale verdict persists.
New e2e (backlog#1675 C1 + N2, red-first): fail-closed against a
header-dropping fake (FAILED + no second PUT via the capability cache,
journal-asserted; red run showed the old COMPLETED), replication-check
reports the SsecPassthrough phase Code while the target stays OK overall,
SSE-C heal convergence after a real target outage, and SSE-C
existing-object resync landing a REPLICA readable with the customer key.
TTL expiry in both directions is pinned at the cache and gate seams.
e2e_test
End-to-end test suite for RustFS. Each test spawns a real rustfs binary
(built on demand from the workspace) and drives it over the network with the
AWS SDK (aws-sdk-s3), raw HTTP (reqwest / awscurl), or a protocol client
(FTPS / WebDAV / SFTP). This is the black-box integration layer: exhaustive
end-to-end behavior lives here, unit behavior stays in the source crates
(see AGENTS.md).
The harness lives in src/common.rs (single-node +
cluster environments, S3 client construction, awscurl helpers) and
src/chaos.rs (in-process disk fault injection). Crate-wide
test conventions and environment-safety rules are in
AGENTS.md; this file is the contributor guide.
Module map (~50 modules)
Registered in src/lib.rs. Grouped by concern:
| Group | Location | What it covers |
|---|---|---|
| functional | top-level *_test.rs |
S3 data plane: list_objects_*, copy_object_*, delete_objects_versioning, head_object_*, checksum_upload, compression, content_encoding, special_chars, leading_slash_key, create_bucket_region, quota, data_usage, snowball_auto_extract, mc_mirror_small_bucket, archive_download_integrity, version_id_regression, delete_marker_migration_semantics |
| object_lock | src/object_lock/ |
Retention / legal-hold / WORM semantics |
| kms | src/kms/ |
SSE-S3 / SSE-KMS / SSE-C, local + Vault backends, multipart encryption. Own guide: src/kms/README.md |
| policy | src/policy/, existing_object_tag_policy_test, bucket_policy_check_test, anonymous_access_test, security_boundary_test, multipart_auth_test |
IAM / bucket-policy / STS session policy, policy variables, anonymous access, DoS/SSRF boundaries. Own guide: src/policy/README.md |
| protocols | src/protocols/ |
FTPS, WebDAV, SFTP compliance. Fixed ports, own guide: src/protocols/README.md |
| reliant | src/reliant/ |
Tests that reuse an externally started server (SQL/select, conditional writes, lifecycle, deleted-object reads, node-interact). Run via scripts/run_e2e_tests.sh; see src/reliant/README.md |
| cluster | cluster_concurrency_test, stale_multipart_cleanup_cluster_test, namespace_lock_quorum_test, admin_timeout_regression_test, object_lambda_test, replication_extension_test |
Multi-node scenarios via RustFSTestClusterEnvironment |
| chaos / reliability | src/chaos.rs, reliability_disk_fault_test, heal_erasure_disk_rebuild_test, server_startup_failfast_test |
Disk offline/replace/corrupt, EC rebuild, heal, fail-fast startup |
How to run
All commands assume repo root. cargo test triggers an on-demand build of the
rustfs binary from src/common.rs (rustfs_binary_path) on
first use — the first invocation is slow, later ones reuse the binary.
# Whole crate (default = ignored tests skipped)
cargo nextest run -p e2e_test
# One module
cargo nextest run -p e2e_test -E 'test(list_objects_v2_pagination_test)'
# PR smoke subset (see "CI smoke subset" below)
cargo nextest run --profile e2e-smoke -p e2e_test
# ILM serial lane — ignored lifecycle tests, single-threaded (mirrors CI)
cargo nextest run -j1 --run-ignored ignored-only -p rustfs-scanner -p rustfs \
-E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))'
# Protocols suite — fixed ports, MUST be single-threaded, gated by build features
RUSTFS_BUILD_FEATURES=ftps,webdav,sftp \
cargo test -p e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
The protocols suite has its own contract (fixed bind ports 9022–9301,
--test-threads=1, feature-gated scheduling) documented in
src/protocols/README.md. RUSTFS_BUILD_FEATURES
selects which features the spawned binary is built with; leave it unset to run
every protocol entry.
#[ignore] semantics
Ignored tests are excluded from the default cargo nextest run pass because
they need something the default runner does not provide. Do not maintain a
static count here — it rots (the set shrinks as ci-13 / ilm-3 activate
suites). Read the live sources instead:
rg -n '#\[ignore' crates/e2e_test/src # every ignore + its reason string
The reason string on each attribute is the classifier. Current classes:
- Needs a pre-started server —
"requires running RustFS server at localhost:9000"/"Connects to existing rustfs server". These are thereliant/*andpolicy/test_runnertests; start a server first (e.g.scripts/run_e2e_tests.sh) or use--run-ignored. - Heavy / external tool —
"Starts a rustfs server; enable when running full E2E","requires awscurl and spawns a real RustFS server". Spawn their own server and/or needawscurlonPATH. - Serial / global-state (ILM lane) — lifecycle tests bind fixed ports and share process-global singletons; run via the ILM serial lane above.
How to add a test
Single-node (the common case)
Use RustFSTestEnvironment from src/common.rs. It picks a
random free port and a unique temp dir per instance, so tests are
parallel-safe by construction and clean up on Drop:
use crate::common::{RustFSTestEnvironment, TEST_BUCKET};
#[tokio::test]
async fn my_case() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?; // waits for readiness
let client = env.create_s3_client(); // aws-sdk-s3 Client
env.create_test_bucket(TEST_BUCKET).await?;
// ... drive `client` ...
Ok(())
}
Register the module in src/lib.rs under #[cfg(test)].
Cluster
Use RustFSTestClusterEnvironment::new(node_count) then .start(); it spawns
node_count servers over a shared erasure set and hands out per-node S3 clients
via create_s3_client(idx) / create_all_clients(). See
cluster_concurrency_test.rs and namespace_lock_quorum_test.rs for patterns.
Fixture / helper inventory (src/common.rs)
| Helper | Purpose |
|---|---|
RustFSTestEnvironment::new / with_address |
Single-node env; random or fixed address |
start_rustfs_server / _with_env / _without_cleanup |
Spawn the server (optional extra args / env vars / no pre-cleanup) |
wait_for_server_ready |
Poll readiness before issuing requests |
create_s3_client / create_test_bucket / delete_test_bucket |
aws-sdk-s3 client + bucket lifecycle |
find_available_port |
Random free port (isolation primitive) |
rustfs_binary_path / _with_features |
Locate/build the binary; honors RUSTFS_BUILD_FEATURES |
requested_rustfs_build_features / rustfs_build_feature_enabled |
Feature-gate a test to what the binary was built with |
awscurl_available + execute_awscurl / awscurl_post / _get / _put / _delete / awscurl_post_sts_form_urlencoded |
Admin/STS API calls via awscurl (skip gracefully when absent) |
replication_fast_env |
Env vars that shrink replication timers (from repl-4); pass to start_rustfs_server_with_env |
local_http_client / init_logging |
Loopback HTTP client; idempotent tracing init |
RustFSTestClusterEnvironment (new/start/start_node/stop_node/create_all_clients) |
Multi-node harness |
Constants: DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, TEST_BUCKET, ENV_RUSTFS_BUILD_FEATURES |
Shared credentials / bucket name / env-var name |
Fault injectors live in src/chaos.rs: DiskFaultHarness
(take_disk_offline, bring_disk_online, replace_disk_with_empty,
corrupt_object_shard, object_metadata_exists_on_disk, kill_server /
restart_server) plus signed_admin_post.
Isolation rules
- Port: never hard-code a port for single-node tests —
new()allocates a random one. Fixed ports (protocols, ILM lane) force--test-threads=1/ a serial CI lane. - Temp dir: each env owns a temp dir cleaned on
Drop; do not write under a shared path. - Orphans:
RustFSTestEnvironmentkills its child onDrop, but a panicked orkill -9'd run can leak arustfsprocess holding a port — see Troubleshooting.
#[serial] vs nextest reality
serial_test's #[serial] uses an in-process mutex. Under nextest each
test runs in its own process, so #[serial] does not serialize across
tests there — see the header of .config/nextest.toml.
Real cross-test serialization comes from a nextest test-group (max-threads = 1) or a -j1 CI lane. Single-node e2e tests should instead be parallel-safe by
construction (random port + isolated temp dir) and need no serialization.
CI map
e2e_test is excluded from the main cargo nextest run --profile ci --all
pass (.github/workflows/ci.yml line 158,
--exclude e2e_test) — the whole crate is too slow to gate every PR. Subsets
join CI through the nextest profile system only (never as ad-hoc jobs):
| Suite | Runs where | Status |
|---|---|---|
Smoke subset (e2e-smoke profile) |
e2e-tests job, every PR |
Active (backlog#1149 ci-4) |
s3s-e2e black-box |
e2e-tests + e2e-tests-rio-v2 jobs |
Active (external conformance tool) |
| ILM / lifecycle (ignored) | test-ilm-integration-serial lane, -j1 |
Active (backlog#1148 ilm-1) |
| KMS suite | — | Not in CI yet (backlog#1149 ci-5) |
| Protocols (FTPS/WebDAV/SFTP) | — | Not in CI yet (backlog#1149 ci-7) |
| Replication (fast subset) | e2e-smoke profile, e2e-tests job, every PR |
Active (backlog#1147 repl-1) |
| Replication (slow + dual-node) | e2e-repl-nightly profile, scheduled workflow |
Active (backlog#1147 repl-1) |
reliant/* (pre-started server) |
— | Manual only |
Links: ci.yml e2e-tests (line 347),
test-ilm-integration-serial (line 196). The e2e-smoke default-filter in
.config/nextest.toml is the single wiring
mechanism — extend that filter (or add a sibling profile) to admit more
tests; do not add e2e jobs to ci.yml. repl-1 / ilm-3 are landing in parallel
and may add lanes; keep the table above easy to extend.
Troubleshooting
Reproduce a CI failure locally — run the exact profile/lane:
# Smoke (e2e-tests job) — includes the 20 fast replication tests
cargo nextest run --profile e2e-smoke -p e2e_test
# Replication nightly lane (16 slow + dual-node tests; install awscurl for the
# STS dual-node test, else it skips gracefully)
cargo nextest run --profile e2e-repl-nightly -p e2e_test
# ILM serial lane
cargo nextest run -j1 --run-ignored ignored-only -p rustfs-scanner -p rustfs \
-E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))'
# s3s-e2e black box
./scripts/e2e-run.sh ./target/debug/rustfs /tmp/rustfs-e2e-data
Stale binary. Tests build the rustfs binary once and reuse it. To avoid
rebuilding while iterating on tests, common.rs reuses an existing binary when
running inside the e2e test process even if sources changed
(can_reuse_inside_e2e, src/common.rs line 98). Downside: if
you changed server code, force a rebuild with
cargo build -p rustfs (or touch a source file outside the reuse window)
before re-running, or CI's freshly built artifact will diverge from your local
one.
Port already in use / orphan processes. A hard-killed run can leak a
rustfs child holding its port. Find and kill it:
pkill -f 'target/debug/rustfs' ; pkill -f 'target/release/rustfs'
The s3s-e2e CI job selects a random RUSTFS_TEST_PORT (see the e2e-tests
job) to dodge this; local single-node tests already use random ports, so a
lingering orphan is usually the cause of a spurious bind failure.
awscurl not found. awscurl-dependent tests skip gracefully with a
visible log line (awscurl_available()); install awscurl to actually run
them.
Related
- Crate rules & environment safety:
AGENTS.md - Sub-suite guides:
src/kms/README.md,src/policy/README.md,src/protocols/README.md,src/reliant/README.md - Authoritative per-module counts:
docs/testing/e2e-suite-inventory.md - Test pyramid & flake policy:
docs/testing/README.md
CI smoke subset (--profile e2e-smoke)
A subset of this crate runs on every PR via the e2e-tests job:
cargo nextest run --profile e2e-smoke -p e2e_test
The selection lives in .config/nextest.toml under [profile.e2e-smoke]
(default-filter). That filter is the single wiring mechanism for e2e
tests in CI — extend it (or add a sibling profile) instead of adding new e2e
jobs to ci.yml.
Admission criteria for the smoke subset
A test module may join the smoke filter only if every test in it is:
- Fast — single-digit seconds per test; the whole subset must keep the
e2e-testsjob ≤ 20 minutes. - Single-node — spawns its own server via
RustFSTestEnvironment/start_rustfs_serveron a random port with an isolated temp dir. NoRustFSTestClusterEnvironment, no fixed ports. - Dependency-free — no pre-started server at
localhost:9000, no Vault, no fixed protocol ports. Tools that may be absent on the runner (e.g.awscurl) are acceptable only when the test skips gracefully with a visible log line (seebucket_policy_check_test.rs). - Not
#[ignore]— ignored tests are activation work (backlog#1149 ci-13 / backlog#1148 ilm-3), not smoke candidates.
Note on #[serial]: nextest runs each test in its own process, so
serial_test's in-process mutex does not serialize across tests there
(see the header of .config/nextest.toml). Smoke tests must therefore be
parallel-safe by construction (random port + isolated temp dir), which the
current subset is.
Authoritative test inventory
docs/testing/e2e-suite-inventory.md records the per-module test counts as
listed by cargo nextest list -p e2e_test. Regenerate it when adding or
moving e2e tests so acceptance numbers in the test-strategy issues
(backlog#1147–#1155) stay auditable.