houseme
1553dc3f62
* fix(obs): open cleaner compression source with O_NOFOLLOW The compressor opened the source log via File::open, which follows a symlink at the final path component. Between the scanner selecting a regular file and this open, an attacker with write access to the log directory could swap the entry for a symlink (TOCTOU) pointing at, say, /etc/shadow, whose contents would then be copied into an archive. Open the source with O_NOFOLLOW on Unix so such a swap fails with ELOOP; the temp/archive path already refused symlinks, this closes the source side. Refs rustfs/backlog#1210 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): recompress instead of trusting leftover cleaner archives archive_header_ok only checked the first 2-4 magic bytes before treating an existing .gz/.zst as a completed prior result and letting the caller delete the source log. A file with valid magic but a truncated or forged body passes that check, so an attacker with write access to the log directory (or a crashed prior run) could plant such a stub and make the cleaner delete the real log without ever producing a usable archive — silent audit-data loss. Chosen fix: stop trusting cross-process leftovers entirely and always recompress the source in this pass, rather than fully decoding every leftover to validate it. Full-decode validation would add real CPU cost and decode-bug surface for a rare crash-recovery case; the existing atomic create_new+rename already overwrites whatever sits at the archive path (a planted symlink is replaced, never followed) with a freshly written, fsync'd archive, so a partial/forged leftover can never gate source deletion. This is the lowest-regression option. Refs rustfs/backlog#1211 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(object-data-cache): cap memory-gate reservation at cache growth headroom The memory gate subtracts `admitted_since_refresh` from the snapshot's available bytes so a burst arriving faster than the 5 s refresh cannot over-allocate. That counter is GROSS: it only rolls over on the refresh and never rolls back when a fill is later evicted, cancelled, or loses the invalidation race. Under sustained high-throughput churn (net footprint flat and far below `max_capacity`) the raw counter balloons past the memory the cache actually holds, so `effective_available` collapses and the gate reports false memory pressure — skipping the hottest fills with SkippedMemoryPressure until the next 5 s refresh. This only lowers hit rate; it never returns wrong data and self-heals each refresh. Fix direction 1 (minimal regression): cap the reservation deduction at the cache's own growth headroom (`max_capacity - weighted_size()`) instead of letting the unbounded gross counter shrink the system-available budget. The cache can never hold more than `max_capacity`, so a burst adds at most that headroom of real memory before moka evicts to stay bounded (net-zero churn beyond that point) — capping the deduction there keeps the reservation honest without treating gross churn as growth. Chosen over net-accounting (direction 2, releasing bytes on every failure/cancel/eviction path) because that only plugs the leak on failed fills and would not address the core defect: churn of *successful* insert/evict fills over the 5 s window. It also touches only the gate plus one call site rather than every failure path in moka_backend. The cap only ever raises `effective_available`, so real memory pressure (a low snapshot at refresh) still suppresses fills; when the cache is at capacity the headroom is 0 and the deduction vanishes, correctly reflecting net-zero churn. `MokaBackend` now stores `max_capacity` and passes the live headroom into `allows_fill`. Adds targeted gate tests: gross churn far above headroom no longer falsely suppresses, yet the reservation still bounds a burst while the cache can genuinely grow. Refs rustfs/backlog#1212 Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): assert native O_DIRECT path runs in uring read test uring_preserves_o_direct_for_eligible_reads only compared bytes through LocalDisk::read_file_mmap_copy. On a filesystem that rejects O_DIRECT the read silently degrades to the buffered StdBackend fallback and the byte check still passes, so the test could go green without the native read_at_direct path ever executing -- a vacuous pass. Add a per-disk native_direct_reads counter on UringBackend, incremented only when pread_uring_direct completes, and rebuild the test to drive a real UringBackend's pread_bytes and assert the counter is non-zero (every eligible read went through the native tier). When io_uring or O_DIRECT is unavailable on the host filesystem (restricted CI runners, tmpfs), the test skips loudly via eprintln instead of asserting a tautology, while still checking byte-correctness on whatever tier served the read. The counter also gives a gray release a positive signal that the O_DIRECT tier is serving reads, not just a fallback count. Refs rustfs/backlog#1213 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): warn + count read-time EINVAL on native O_DIRECT reads classify_direct_read_error is only reached from the read side: the O_DIRECT open in pread_uring_direct already succeeded (an open-time refusal is handled earlier as DirectOpenError::ODirectRefused). So an EINVAL/EOPNOTSUPP arriving here is a read-time error on an fd the kernel accepted for O_DIRECT -- far more likely an alignment bug in the aligned read path than an unsupported filesystem. The old code latched the disk's native path off with only a once-per-disk debug trace, hiding a potential correctness regression behind a silent buffered-read downgrade. Diagnostics only: the fallback behaviour is unchanged (the native path is still latched off and the caller still reads via StdBackend). This adds a rustfs_io_uring_direct_read_einval_total counter and promotes the once-per-disk trace from debug to warn so an operator can see an alignment regression instead of an unexplained latency/CPU shift. Refs rustfs/backlog#1214 Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ecstore): document data-blocks-first default and its tail-latency cost DEFAULT_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP is true and must stay true: deferred-parity is the deliberate, already-rolled-out full-object GET default from backlog#1159/#923. Flipping it back to false in code would silently revert that rollout for every deployment that has not set the env var, so this commit only documents -- no behaviour change. The added notes explain what data-blocks-first does (schedule data shards up front, engage parity lazily on a missing/corrupt data shard), the known trade-off (parity is engaged late, so a slow-but-not-dead data drive raises GET p99 because the faster parity shards are not raced against it until a data shard is declared missing), and the operational rollback switch (RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP=false), which is intentionally an env override rather than a code default change. No metric was added: the low-risk observability hook for "slow data drive engaged deferred parity" would live at the deferred-stripe engage point, which is out of this file's scope; this change stays documentation-only to avoid touching the hot GET path. Refs rustfs/backlog#1215 Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ecstore): document wide-directory walk stall hazard and tuning list_dir enumerates a whole directory in one os::read_dir call (count = -1), and the walk caller bounds that entire enumeration with the per-read stall budget (default 5s) as if it were a single read. For a wide, flat prefix -- one directory holding millions of immediate children -- a single readdir can exceed the budget on a healthy disk, trip DiskError::Timeout, and surface as a ListObjects 500 quorum failure though the drive is fine (a #2999 sub-class). This is documented, not rewritten: turning the one-shot readdir into a streaming/batched enumeration that refreshes the stall deadline between chunks is an architecture-level change with high regression surface (ordering, the count contract, quorum merge) and belongs in a separate follow-up. The supported mitigation today is operational, so the comments point wide-directory deployments at RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS and the high-latency drive-timeout profile, which widen the budget with no code change. Notes were added at list_dir, the scan_dir call site, and get_drive_walkdir_stall_timeout. No behaviour change. Refs rustfs/backlog#1216 Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ecstore): document consumer-peek vs producer-stall coupling In list_path_raw the consumer's peek_timeout is drawn from the same source and same value (walkdir_stall_timeout, default 5s) as the producer-side walk stall budget, but the two measure different things: the producer stall bounds a single drive read, while the consumer peek bounds the gap between two ADJACENT entries arriving from a reader. Because they share a value, the consumer cannot wait meaningfully longer for the next entry than the producer is allowed to spend producing one. Walking a region dense with non-listable internal items can make a HEALTHY drive miss the budget between visible entries; the consumer then declares it stalled and detaches it, dropping a good drive from the merge and capping the "large prefix succeeds" guarantee. Documented, not decoupled: giving the consumer peek an independent, strictly-larger budget would cut these false detaches but equally delays detaching a genuinely dead drive and shifts listing tail-latency semantics, so it wants soak data before changing the default. The comment records the invariant any such follow-up must keep -- consumer peek >= producer stall, never stricter -- so it can never fail a drive before the producer would. No behaviour change. Refs rustfs/backlog#1217 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(io-metrics): add time-based trigger for low-IOPS latency percentiles Percentiles were recomputed only every 128 IOs and seeded to 0, so a low-traffic deployment exported p95/p99 = 0/stale for a long time after startup. Add a 10s wall-clock trigger alongside the count throttle so the first recompute can fire before 128 samples accrue. Hot-path per-op mean update is unchanged. Refs rustfs/backlog#1218 Co-Authored-By: heihutu <heihutu@gmail.com> * test(e2e): cover codec-streaming parity under fault injection and NoSuchKey The codec-streaming compat A/B previously ran only against a healthy 4-disk EC set with successful full GETs: the DiskFaultHarness was constructed but never faulted, the error path was untested, and the range assertion silently compared legacy-vs-legacy (ranges always fall back to the duplex path), overstating what it proved. Add two genuinely-failable scenarios reusing the existing harness and fixtures: - Parity reconstruction A/B: take one data disk offline and re-run the full object matrix on both phases while the EC 2+2 set rebuilds each large object from the surviving shards. Assert codec == legacy byte-for-byte (sha256) and header-for-header, and assert the codec phase served the reconstructed objects with zero duplex-pipe fallback (the reader gate is drive-health-independent, so the codec fast path is really exercised through reconstruction). - NoSuchKey negative path: compare the HTTP status + S3 error code of a missing-key GET across the legacy and codec phases and require them to be identical (404/NoSuchKey), guarding against the codec env perturbing the error path. Also clarify the range-phase comment so it is not misread as codec-range correctness coverage: both sides are served by the same legacy range path, so the assertion only proves ranges keep working and keep falling back to legacy with the gates open. Verified: cargo check/--no-run pass and the test passes locally (1 passed; dup_codec=0 confirms the codec path ran). Refs rustfs/backlog#1219 Co-Authored-By: heihutu <heihutu@gmail.com> * ci(ecstore): exercise native O_DIRECT read path on an ext4 loopback The uring-integration leg ran on the runner's default TMPDIR, which may sit on tmpfs/overlayfs where open(O_DIRECT) fails and the native read_at_direct path silently latches off to the aligned StdBackend fallback. Mount a dedicated ext4 loopback and point TMPDIR at it so the real io_uring dep (bumped git->0.1.0->0.2.0->0.2.1) and the native O_DIRECT read path are actually covered rather than validated only by signature diffing. Refs rustfs/backlog#1220 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>
e2e_test
End-to-end test suite for RustFS. Each test spawns a real rustfs binary
(built on demand from the workspace) and drives it over the network with the
AWS SDK (aws-sdk-s3), raw HTTP (reqwest / awscurl), or a protocol client
(FTPS / WebDAV / SFTP). This is the black-box integration layer: exhaustive
end-to-end behavior lives here, unit behavior stays in the source crates
(see AGENTS.md).
The harness lives in src/common.rs (single-node +
cluster environments, S3 client construction, awscurl helpers) and
src/chaos.rs (in-process disk fault injection). Crate-wide
test conventions and environment-safety rules are in
AGENTS.md; this file is the contributor guide.
Module map (~50 modules)
Registered in src/lib.rs. Grouped by concern:
| Group | Location | What it covers |
|---|---|---|
| functional | top-level *_test.rs |
S3 data plane: list_objects_*, copy_object_*, delete_objects_versioning, head_object_*, checksum_upload, compression, content_encoding, special_chars, leading_slash_key, create_bucket_region, quota, data_usage, snowball_auto_extract, mc_mirror_small_bucket, archive_download_integrity, version_id_regression, delete_marker_migration_semantics |
| object_lock | src/object_lock/ |
Retention / legal-hold / WORM semantics |
| kms | src/kms/ |
SSE-S3 / SSE-KMS / SSE-C, local + Vault backends, multipart encryption. Own guide: src/kms/README.md |
| policy | src/policy/, existing_object_tag_policy_test, bucket_policy_check_test, anonymous_access_test, security_boundary_test, multipart_auth_test |
IAM / bucket-policy / STS session policy, policy variables, anonymous access, DoS/SSRF boundaries. Own guide: src/policy/README.md |
| protocols | src/protocols/ |
FTPS, WebDAV, SFTP compliance. Fixed ports, own guide: src/protocols/README.md |
| reliant | src/reliant/ |
Tests that reuse an externally started server (SQL/select, conditional writes, lifecycle, deleted-object reads, node-interact). Run via scripts/run_e2e_tests.sh; see src/reliant/README.md |
| cluster | cluster_concurrency_test, stale_multipart_cleanup_cluster_test, namespace_lock_quorum_test, admin_timeout_regression_test, object_lambda_test, replication_extension_test |
Multi-node scenarios via RustFSTestClusterEnvironment |
| chaos / reliability | src/chaos.rs, reliability_disk_fault_test, heal_erasure_disk_rebuild_test, server_startup_failfast_test |
Disk offline/replace/corrupt, EC rebuild, heal, fail-fast startup |
How to run
All commands assume repo root. cargo test triggers an on-demand build of the
rustfs binary from src/common.rs (rustfs_binary_path) on
first use — the first invocation is slow, later ones reuse the binary.
# Whole crate (default = ignored tests skipped)
cargo nextest run -p e2e_test
# One module
cargo nextest run -p e2e_test -E 'test(list_objects_v2_pagination_test)'
# PR smoke subset (see "CI smoke subset" below)
cargo nextest run --profile e2e-smoke -p e2e_test
# ILM serial lane — ignored lifecycle tests, single-threaded (mirrors CI)
cargo nextest run -j1 --run-ignored ignored-only -p rustfs-scanner -p rustfs \
-E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))'
# Protocols suite — fixed ports, MUST be single-threaded, gated by build features
RUSTFS_BUILD_FEATURES=ftps,webdav,sftp \
cargo test -p e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
The protocols suite has its own contract (fixed bind ports 9022–9301,
--test-threads=1, feature-gated scheduling) documented in
src/protocols/README.md. RUSTFS_BUILD_FEATURES
selects which features the spawned binary is built with; leave it unset to run
every protocol entry.
#[ignore] semantics
Ignored tests are excluded from the default cargo nextest run pass because
they need something the default runner does not provide. Do not maintain a
static count here — it rots (the set shrinks as ci-13 / ilm-3 activate
suites). Read the live sources instead:
rg -n '#\[ignore' crates/e2e_test/src # every ignore + its reason string
The reason string on each attribute is the classifier. Current classes:
- Needs a pre-started server —
"requires running RustFS server at localhost:9000"/"Connects to existing rustfs server". These are thereliant/*andpolicy/test_runnertests; start a server first (e.g.scripts/run_e2e_tests.sh) or use--run-ignored. - Heavy / external tool —
"Starts a rustfs server; enable when running full E2E","requires awscurl and spawns a real RustFS server". Spawn their own server and/or needawscurlonPATH. - Serial / global-state (ILM lane) — lifecycle tests bind fixed ports and share process-global singletons; run via the ILM serial lane above.
How to add a test
Single-node (the common case)
Use RustFSTestEnvironment from src/common.rs. It picks a
random free port and a unique temp dir per instance, so tests are
parallel-safe by construction and clean up on Drop:
use crate::common::{RustFSTestEnvironment, TEST_BUCKET};
#[tokio::test]
async fn my_case() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?; // waits for readiness
let client = env.create_s3_client(); // aws-sdk-s3 Client
env.create_test_bucket(TEST_BUCKET).await?;
// ... drive `client` ...
Ok(())
}
Register the module in src/lib.rs under #[cfg(test)].
Cluster
Use RustFSTestClusterEnvironment::new(node_count) then .start(); it spawns
node_count servers over a shared erasure set and hands out per-node S3 clients
via create_s3_client(idx) / create_all_clients(). See
cluster_concurrency_test.rs and namespace_lock_quorum_test.rs for patterns.
Fixture / helper inventory (src/common.rs)
| Helper | Purpose |
|---|---|
RustFSTestEnvironment::new / with_address |
Single-node env; random or fixed address |
start_rustfs_server / _with_env / _without_cleanup |
Spawn the server (optional extra args / env vars / no pre-cleanup) |
wait_for_server_ready |
Poll readiness before issuing requests |
create_s3_client / create_test_bucket / delete_test_bucket |
aws-sdk-s3 client + bucket lifecycle |
find_available_port |
Random free port (isolation primitive) |
rustfs_binary_path / _with_features |
Locate/build the binary; honors RUSTFS_BUILD_FEATURES |
requested_rustfs_build_features / rustfs_build_feature_enabled |
Feature-gate a test to what the binary was built with |
awscurl_available + execute_awscurl / awscurl_post / _get / _put / _delete / awscurl_post_sts_form_urlencoded |
Admin/STS API calls via awscurl (skip gracefully when absent) |
replication_fast_env |
Env vars that shrink replication timers (from repl-4); pass to start_rustfs_server_with_env |
local_http_client / init_logging |
Loopback HTTP client; idempotent tracing init |
RustFSTestClusterEnvironment (new/start/start_node/stop_node/create_all_clients) |
Multi-node harness |
Constants: DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, TEST_BUCKET, ENV_RUSTFS_BUILD_FEATURES |
Shared credentials / bucket name / env-var name |
Fault injectors live in src/chaos.rs: DiskFaultHarness
(take_disk_offline, bring_disk_online, replace_disk_with_empty,
corrupt_object_shard, object_metadata_exists_on_disk, kill_server /
restart_server) plus signed_admin_post.
Isolation rules
- Port: never hard-code a port for single-node tests —
new()allocates a random one. Fixed ports (protocols, ILM lane) force--test-threads=1/ a serial CI lane. - Temp dir: each env owns a temp dir cleaned on
Drop; do not write under a shared path. - Orphans:
RustFSTestEnvironmentkills its child onDrop, but a panicked orkill -9'd run can leak arustfsprocess holding a port — see Troubleshooting.
#[serial] vs nextest reality
serial_test's #[serial] uses an in-process mutex. Under nextest each
test runs in its own process, so #[serial] does not serialize across
tests there — see the header of .config/nextest.toml.
Real cross-test serialization comes from a nextest test-group (max-threads = 1) or a -j1 CI lane. Single-node e2e tests should instead be parallel-safe by
construction (random port + isolated temp dir) and need no serialization.
CI map
e2e_test is excluded from the main cargo nextest run --profile ci --all
pass (.github/workflows/ci.yml line 158,
--exclude e2e_test) — the whole crate is too slow to gate every PR. Subsets
join CI through the nextest profile system only (never as ad-hoc jobs):
| Suite | Runs where | Status |
|---|---|---|
Smoke subset (e2e-smoke profile) |
e2e-tests job, every PR |
Active (backlog#1149 ci-4) |
s3s-e2e black-box |
e2e-tests + e2e-tests-rio-v2 jobs |
Active (external conformance tool) |
| ILM / lifecycle (ignored) | test-ilm-integration-serial lane, -j1 |
Active (backlog#1148 ilm-1) |
| KMS suite | — | Not in CI yet (backlog#1149 ci-5) |
| Protocols (FTPS/WebDAV/SFTP) | — | Not in CI yet (backlog#1149 ci-7) |
| Replication (fast subset) | e2e-smoke profile, e2e-tests job, every PR |
Active (backlog#1147 repl-1) |
| Replication (slow + dual-node) | e2e-repl-nightly profile, scheduled workflow |
Active (backlog#1147 repl-1) |
reliant/* (pre-started server) |
— | Manual only |
Links: ci.yml e2e-tests (line 347),
test-ilm-integration-serial (line 196). The e2e-smoke default-filter in
.config/nextest.toml is the single wiring
mechanism — extend that filter (or add a sibling profile) to admit more
tests; do not add e2e jobs to ci.yml. repl-1 / ilm-3 are landing in parallel
and may add lanes; keep the table above easy to extend.
Troubleshooting
Reproduce a CI failure locally — run the exact profile/lane:
# Smoke (e2e-tests job) — includes the 20 fast replication tests
cargo nextest run --profile e2e-smoke -p e2e_test
# Replication nightly lane (16 slow + dual-node tests; install awscurl for the
# STS dual-node test, else it skips gracefully)
cargo nextest run --profile e2e-repl-nightly -p e2e_test
# ILM serial lane
cargo nextest run -j1 --run-ignored ignored-only -p rustfs-scanner -p rustfs \
-E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))'
# s3s-e2e black box
./scripts/e2e-run.sh ./target/debug/rustfs /tmp/rustfs-e2e-data
Stale binary. Tests build the rustfs binary once and reuse it. To avoid
rebuilding while iterating on tests, common.rs reuses an existing binary when
running inside the e2e test process even if sources changed
(can_reuse_inside_e2e, src/common.rs line 98). Downside: if
you changed server code, force a rebuild with
cargo build -p rustfs (or touch a source file outside the reuse window)
before re-running, or CI's freshly built artifact will diverge from your local
one.
Port already in use / orphan processes. A hard-killed run can leak a
rustfs child holding its port. Find and kill it:
pkill -f 'target/debug/rustfs' ; pkill -f 'target/release/rustfs'
The s3s-e2e CI job selects a random RUSTFS_TEST_PORT (see the e2e-tests
job) to dodge this; local single-node tests already use random ports, so a
lingering orphan is usually the cause of a spurious bind failure.
awscurl not found. awscurl-dependent tests skip gracefully with a
visible log line (awscurl_available()); install awscurl to actually run
them.
Related
- Crate rules & environment safety:
AGENTS.md - Sub-suite guides:
src/kms/README.md,src/policy/README.md,src/protocols/README.md,src/reliant/README.md - Authoritative per-module counts:
docs/testing/e2e-suite-inventory.md - Test pyramid & flake policy:
docs/testing/README.md
CI smoke subset (--profile e2e-smoke)
A subset of this crate runs on every PR via the e2e-tests job:
cargo nextest run --profile e2e-smoke -p e2e_test
The selection lives in .config/nextest.toml under [profile.e2e-smoke]
(default-filter). That filter is the single wiring mechanism for e2e
tests in CI — extend it (or add a sibling profile) instead of adding new e2e
jobs to ci.yml.
Admission criteria for the smoke subset
A test module may join the smoke filter only if every test in it is:
- Fast — single-digit seconds per test; the whole subset must keep the
e2e-testsjob ≤ 20 minutes. - Single-node — spawns its own server via
RustFSTestEnvironment/start_rustfs_serveron a random port with an isolated temp dir. NoRustFSTestClusterEnvironment, no fixed ports. - Dependency-free — no pre-started server at
localhost:9000, no Vault, no fixed protocol ports. Tools that may be absent on the runner (e.g.awscurl) are acceptable only when the test skips gracefully with a visible log line (seebucket_policy_check_test.rs). - Not
#[ignore]— ignored tests are activation work (backlog#1149 ci-13 / backlog#1148 ilm-3), not smoke candidates.
Note on #[serial]: nextest runs each test in its own process, so
serial_test's in-process mutex does not serialize across tests there
(see the header of .config/nextest.toml). Smoke tests must therefore be
parallel-safe by construction (random port + isolated temp dir), which the
current subset is.
Authoritative test inventory
docs/testing/e2e-suite-inventory.md records the per-module test counts as
listed by cargo nextest list -p e2e_test. Regenerate it when adding or
moving e2e tests so acceptance numbers in the test-strategy issues
(backlog#1147–#1155) stay auditable.