The background tier free-version recovery walk pinned a hardcoded 60s
total wall-clock timeout that overrides every operator knob, so any
bucket whose healthy full walk exceeds 60s fails forever; the failed
run's duration was also subtracted from the next 60s tick, restarting
the walk immediately and pinning CPU and disk I/O.
- Drop the total wall-clock budget on the recovery walk
(walkdir_timeout: Duration::ZERO) and inherit the operator-tunable
drive stall budget (RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS) for
per-call progress, so hung disks still fail fast.
- Back off failed runs from completion time: 60s doubling to a 600s
cap, reset on success; never subtract the failed run's duration.
- Add RUSTFS_TIER_FREE_VERSION_RECOVERY_ENABLED (default true) to opt
out of the recovery worker on deployments with no remote tiers;
invalid values warn and fail open.
Fixes#5130
Co-authored-by: claude <claude@ehdtn.com>
A PUT that fans out erasure shards to remote peers awaited every shard writer to completion on both the per-block write and the final shutdown, and the remote HttpWriter had no progress deadline. A peer that accepts the TCP connection but never drains the request body (or never sends a response) therefore wedges the writer forever once the bounded buffers fill, pinning an otherwise-healthy write quorum indefinitely — a cluster-level write-availability hazard triggered by a single bad peer (rustfs/backlog#1319, https://github.com/rustfs/backlog/issues/1319).
MultiWriter now wraps each shard write and each shard-writer shutdown in a forward-progress deadline. The budget is re-armed on every block, so it bounds a stall rather than the total transfer time of a large object: a slow-but-honest writer that keeps completing shards is never killed, while a writer that makes no progress within the budget is failed and its disk dropped before commit. An optional absolute per-object cap (disabled by default) backstops a slow-drip peer that dribbles just enough progress to reset the per-block timer without ever converging; it is off by default so a legitimate large upload over a slow link is not killed on total time alone. Both knobs come from RUSTFS_OBJECT_DISK_WRITE_STALL_TIMEOUT (default 30s) and RUSTFS_OBJECT_DISK_WRITE_ABSOLUTE_CAP (default 0 = disabled); setting the stall timeout to 0 restores the previous wait-forever behavior for a conservative rollback.
The deadline enforcement lives in MultiWriter (writer-agnostic), so it covers local and remote writers alike and keeps the existing control-flow shape: a timed-out shard is marked failed (Error::Timeout, which is not an ignored error) and excluded from the write quorum exactly like any other shard write failure, and the unchanged nil_count/quorum check then continues on quorum or fails cleanly. This deliberately stays out of the MultiWriter lifecycle / commit-coordinator territory owned by rustfs/backlog#1312.
When a stalled writer is dropped to fail its shard, the remote HttpWriter must stop holding the connection and its buffered body. HttpWriter previously left its spawned request task running on drop; it now aborts that background task in Drop (it is no longer pin-projected, since every field is Unpin and the AsyncWrite impl already used get_mut). Bytes already handed to the transport cannot be unsent, but they land only in this upload's unique tmp path and are reclaimed by tmp GC — they never touch a committed object.
Tests, all on a paused virtual clock so they are deterministic and non-flaky:
- one black-hole writer still meets a 3/4 write quorum without hanging; two black holes fail the quorum cleanly (both for the per-block write and the shutdown paths).
- a slow-but-honest writer that keeps making progress within the stall budget is never failed across many blocks.
- the absolute cap bounds a slow-drip writer within a finite budget while the healthy writers keep quorum.
- the default policy is armed by default and honors 0 as disabled.
- HttpWriter aborts its background request task on drop against a hanging peer.
The toxiproxy/black-hole 4x4 end-to-end acceptance depends on black-box test facilities from rustfs/backlog#1325, which are not built yet; that acceptance is deferred to #1325 and intentionally not faked here.
* fix(obs): open cleaner compression source with O_NOFOLLOW
The compressor opened the source log via File::open, which follows a
symlink at the final path component. Between the scanner selecting a
regular file and this open, an attacker with write access to the log
directory could swap the entry for a symlink (TOCTOU) pointing at, say,
/etc/shadow, whose contents would then be copied into an archive. Open
the source with O_NOFOLLOW on Unix so such a swap fails with ELOOP; the
temp/archive path already refused symlinks, this closes the source side.
Refs rustfs/backlog#1210
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): recompress instead of trusting leftover cleaner archives
archive_header_ok only checked the first 2-4 magic bytes before treating
an existing .gz/.zst as a completed prior result and letting the caller
delete the source log. A file with valid magic but a truncated or forged
body passes that check, so an attacker with write access to the log
directory (or a crashed prior run) could plant such a stub and make the
cleaner delete the real log without ever producing a usable archive —
silent audit-data loss.
Chosen fix: stop trusting cross-process leftovers entirely and always
recompress the source in this pass, rather than fully decoding every
leftover to validate it. Full-decode validation would add real CPU cost
and decode-bug surface for a rare crash-recovery case; the existing
atomic create_new+rename already overwrites whatever sits at the archive
path (a planted symlink is replaced, never followed) with a freshly
written, fsync'd archive, so a partial/forged leftover can never gate
source deletion. This is the lowest-regression option.
Refs rustfs/backlog#1211
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): cap memory-gate reservation at cache growth headroom
The memory gate subtracts `admitted_since_refresh` from the snapshot's
available bytes so a burst arriving faster than the 5 s refresh cannot
over-allocate. That counter is GROSS: it only rolls over on the refresh and
never rolls back when a fill is later evicted, cancelled, or loses the
invalidation race. Under sustained high-throughput churn (net footprint flat
and far below `max_capacity`) the raw counter balloons past the memory the
cache actually holds, so `effective_available` collapses and the gate reports
false memory pressure — skipping the hottest fills with SkippedMemoryPressure
until the next 5 s refresh. This only lowers hit rate; it never returns wrong
data and self-heals each refresh.
Fix direction 1 (minimal regression): cap the reservation deduction at the
cache's own growth headroom (`max_capacity - weighted_size()`) instead of
letting the unbounded gross counter shrink the system-available budget. The
cache can never hold more than `max_capacity`, so a burst adds at most that
headroom of real memory before moka evicts to stay bounded (net-zero churn
beyond that point) — capping the deduction there keeps the reservation honest
without treating gross churn as growth. Chosen over net-accounting (direction
2, releasing bytes on every failure/cancel/eviction path) because that only
plugs the leak on failed fills and would not address the core defect: churn of
*successful* insert/evict fills over the 5 s window. It also touches only the
gate plus one call site rather than every failure path in moka_backend.
The cap only ever raises `effective_available`, so real memory pressure (a low
snapshot at refresh) still suppresses fills; when the cache is at capacity the
headroom is 0 and the deduction vanishes, correctly reflecting net-zero churn.
`MokaBackend` now stores `max_capacity` and passes the live headroom into
`allows_fill`. Adds targeted gate tests: gross churn far above headroom no
longer falsely suppresses, yet the reservation still bounds a burst while the
cache can genuinely grow.
Refs rustfs/backlog#1212
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): assert native O_DIRECT path runs in uring read test
uring_preserves_o_direct_for_eligible_reads only compared bytes through
LocalDisk::read_file_mmap_copy. On a filesystem that rejects O_DIRECT the
read silently degrades to the buffered StdBackend fallback and the byte
check still passes, so the test could go green without the native
read_at_direct path ever executing -- a vacuous pass.
Add a per-disk native_direct_reads counter on UringBackend, incremented
only when pread_uring_direct completes, and rebuild the test to drive a
real UringBackend's pread_bytes and assert the counter is non-zero (every
eligible read went through the native tier). When io_uring or O_DIRECT is
unavailable on the host filesystem (restricted CI runners, tmpfs), the
test skips loudly via eprintln instead of asserting a tautology, while
still checking byte-correctness on whatever tier served the read.
The counter also gives a gray release a positive signal that the O_DIRECT
tier is serving reads, not just a fallback count.
Refs rustfs/backlog#1213
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): warn + count read-time EINVAL on native O_DIRECT reads
classify_direct_read_error is only reached from the read side: the
O_DIRECT open in pread_uring_direct already succeeded (an open-time
refusal is handled earlier as DirectOpenError::ODirectRefused). So an
EINVAL/EOPNOTSUPP arriving here is a read-time error on an fd the kernel
accepted for O_DIRECT -- far more likely an alignment bug in the aligned
read path than an unsupported filesystem. The old code latched the disk's
native path off with only a once-per-disk debug trace, hiding a potential
correctness regression behind a silent buffered-read downgrade.
Diagnostics only: the fallback behaviour is unchanged (the native path is
still latched off and the caller still reads via StdBackend). This adds a
rustfs_io_uring_direct_read_einval_total counter and promotes the
once-per-disk trace from debug to warn so an operator can see an alignment
regression instead of an unexplained latency/CPU shift.
Refs rustfs/backlog#1214
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(ecstore): document data-blocks-first default and its tail-latency cost
DEFAULT_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP is true and must stay
true: deferred-parity is the deliberate, already-rolled-out full-object
GET default from backlog#1159/#923. Flipping it back to false in code
would silently revert that rollout for every deployment that has not set
the env var, so this commit only documents -- no behaviour change.
The added notes explain what data-blocks-first does (schedule data shards
up front, engage parity lazily on a missing/corrupt data shard), the known
trade-off (parity is engaged late, so a slow-but-not-dead data drive
raises GET p99 because the faster parity shards are not raced against it
until a data shard is declared missing), and the operational rollback
switch (RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP=false), which is
intentionally an env override rather than a code default change.
No metric was added: the low-risk observability hook for "slow data drive
engaged deferred parity" would live at the deferred-stripe engage point,
which is out of this file's scope; this change stays documentation-only to
avoid touching the hot GET path.
Refs rustfs/backlog#1215
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(ecstore): document wide-directory walk stall hazard and tuning
list_dir enumerates a whole directory in one os::read_dir call (count =
-1), and the walk caller bounds that entire enumeration with the per-read
stall budget (default 5s) as if it were a single read. For a wide, flat
prefix -- one directory holding millions of immediate children -- a single
readdir can exceed the budget on a healthy disk, trip DiskError::Timeout,
and surface as a ListObjects 500 quorum failure though the drive is fine
(a #2999 sub-class).
This is documented, not rewritten: turning the one-shot readdir into a
streaming/batched enumeration that refreshes the stall deadline between
chunks is an architecture-level change with high regression surface
(ordering, the count contract, quorum merge) and belongs in a separate
follow-up. The supported mitigation today is operational, so the comments
point wide-directory deployments at RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS
and the high-latency drive-timeout profile, which widen the budget with no
code change. Notes were added at list_dir, the scan_dir call site, and
get_drive_walkdir_stall_timeout. No behaviour change.
Refs rustfs/backlog#1216
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(ecstore): document consumer-peek vs producer-stall coupling
In list_path_raw the consumer's peek_timeout is drawn from the same source
and same value (walkdir_stall_timeout, default 5s) as the producer-side
walk stall budget, but the two measure different things: the producer
stall bounds a single drive read, while the consumer peek bounds the gap
between two ADJACENT entries arriving from a reader. Because they share a
value, the consumer cannot wait meaningfully longer for the next entry
than the producer is allowed to spend producing one. Walking a region
dense with non-listable internal items can make a HEALTHY drive miss the
budget between visible entries; the consumer then declares it stalled and
detaches it, dropping a good drive from the merge and capping the "large
prefix succeeds" guarantee.
Documented, not decoupled: giving the consumer peek an independent,
strictly-larger budget would cut these false detaches but equally delays
detaching a genuinely dead drive and shifts listing tail-latency
semantics, so it wants soak data before changing the default. The comment
records the invariant any such follow-up must keep -- consumer peek >=
producer stall, never stricter -- so it can never fail a drive before the
producer would. No behaviour change.
Refs rustfs/backlog#1217
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(io-metrics): add time-based trigger for low-IOPS latency percentiles
Percentiles were recomputed only every 128 IOs and seeded to 0, so a
low-traffic deployment exported p95/p99 = 0/stale for a long time after
startup. Add a 10s wall-clock trigger alongside the count throttle so the
first recompute can fire before 128 samples accrue. Hot-path per-op mean
update is unchanged.
Refs rustfs/backlog#1218
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): cover codec-streaming parity under fault injection and NoSuchKey
The codec-streaming compat A/B previously ran only against a healthy
4-disk EC set with successful full GETs: the DiskFaultHarness was
constructed but never faulted, the error path was untested, and the
range assertion silently compared legacy-vs-legacy (ranges always fall
back to the duplex path), overstating what it proved.
Add two genuinely-failable scenarios reusing the existing harness and
fixtures:
- Parity reconstruction A/B: take one data disk offline and re-run the
full object matrix on both phases while the EC 2+2 set rebuilds each
large object from the surviving shards. Assert codec == legacy
byte-for-byte (sha256) and header-for-header, and assert the codec
phase served the reconstructed objects with zero duplex-pipe fallback
(the reader gate is drive-health-independent, so the codec fast path
is really exercised through reconstruction).
- NoSuchKey negative path: compare the HTTP status + S3 error code of a
missing-key GET across the legacy and codec phases and require them to
be identical (404/NoSuchKey), guarding against the codec env
perturbing the error path.
Also clarify the range-phase comment so it is not misread as
codec-range correctness coverage: both sides are served by the same
legacy range path, so the assertion only proves ranges keep working and
keep falling back to legacy with the gates open.
Verified: cargo check/--no-run pass and the test passes locally
(1 passed; dup_codec=0 confirms the codec path ran).
Refs rustfs/backlog#1219
Co-Authored-By: heihutu <heihutu@gmail.com>
* ci(ecstore): exercise native O_DIRECT read path on an ext4 loopback
The uring-integration leg ran on the runner's default TMPDIR, which may sit
on tmpfs/overlayfs where open(O_DIRECT) fails and the native read_at_direct
path silently latches off to the aligned StdBackend fallback. Mount a
dedicated ext4 loopback and point TMPDIR at it so the real io_uring dep
(bumped git->0.1.0->0.2.0->0.2.1) and the native O_DIRECT read path are
actually covered rather than validated only by signature diffing.
Refs rustfs/backlog#1220
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(scanner): scope long walk timeouts
* fix(scanner): bound IAM config walks
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
* fix(ecstore): make delete_volume non-recursive by default to prevent bucket-heal wipe (backlog#799 B1)
`delete_volume` unconditionally `remove_dir_all`'d the whole bucket tree, and the
bucket-heal "remove" branch called it fire-and-forget on every local disk. A
mis-classified "dangling" bucket (or a non-force S3 DeleteBucket on a populated
bucket) was therefore recursively wiped — a potential whole-bucket data loss.
The `VolumeNotEmpty` -> recreate/`BucketNotEmpty` handling already present in
both delete_bucket paths was dead code because the primitive never refused.
Add an explicit `force_delete` flag to `DiskAPI::delete_volume` and default the
non-force path to a non-recursive `remove_dir` (rmdir), which fails atomically
with `VolumeNotEmpty` if the bucket still holds any object data. Only an explicit
force delete (S3 force bucket delete) removes recursively. Mirrors MinIO's
`xlStorage.DeleteVol` (`Remove` vs `RemoveAll`).
- Trait + all impls (local behavior, dispatch, disk_store, remote RPC) take the
flag; the gRPC `DeleteVolumeRequest` gains a `force` field (proto3 default
false → old peers get the safe non-recursive behavior on rolling upgrade).
- Heal remove branch passes `false` and no longer discards the result: a
`VolumeNotEmpty` refusal is logged (the bucket is not dangling) instead of
wiping data.
- Both `delete_bucket` paths pass `opts.force`, activating the previously-dead
`VolumeNotEmpty` -> `BucketNotEmpty`/recreate handling (correct S3 semantics).
Adds a regression test: non-force delete of a non-empty bucket returns
VolumeNotEmpty and preserves the data; force delete removes it.
Design converged by two independent expert reviews (MinIO-fidelity +
defense-in-depth) referencing MinIO xl-storage.go. Refs backlog#799 (B1),
issue rustfs/backlog#850. The safety expert's deeper hardening (typed
capability instead of a bool, trash-instead-of-in-place for force, quorum
re-verification of dangling) is noted on #850 as follow-up.
* fix(ecstore): reword 'mis-classified' -> 'misclassified' to satisfy typos (backlog#799 B1)
* fix(rustfs): thread force_delete through StorageDiskRpcExt::delete_volume + test literal (backlog#799 B1)
* fix(iam): prevent transient IAM walk timeout from crashing startup
IAM startup performs a blocking full metadata walk on `.rustfs.sys/config/iam/`.
When that distributed walk times out (e.g. disk pressure after cluster reboot),
the old code treated the failure as fatal and exited the process, causing a
systemd restart loop.
Changes:
- Add `startup_iam.rs`: attempt IAM init, enter degraded mode on failure,
spawn background retry task with exponential backoff (5s→10s→20s→30s cap)
- Log level escalates to ERROR after 12 retries (~5 min) to aid diagnosis
- `/health/ready` returns 503 until IAM recovers; IAM-dependent ops return
`IamSysNotInitialized` (existing fail-closed behavior preserved)
- Fix admin path boundary matching: `/minio/administrator` no longer falsely
matches as admin prefix
- Normalize Content-Length: 0 for admin GET requests with empty body
Fixes#3175
* fix(iam): move constant assertion into const block
Fixes clippy::assertions-on-constants warning on
IAM_RETRY_ESCALATION_THRESHOLD assertion.
* fix(iam): address PR review comments
- Replace OnceLock with AtomicU64 sentinel for test isolation;
add reset_test_failure_counter() for integration tests
- Use u32::try_from() instead of `as u32` narrowing cast in
compute_backoff_interval
- Rename misleading test; update to verify finalize retry behavior
- Restructure spawn_iam_recovery_task into init-retry and
finalize-retry phases so transient readiness failures are retried
instead of leaving the server permanently degraded
* fix(iam): gate test hooks behind debug_assertions
- reset_test_failure_counter() now stores sentinel (u64::MAX) to
correctly trigger env var re-read on next call
- RUSTFS_TEST_IAM_FAIL_INIT_ATTEMPTS only honored in debug builds
- RUSTFS_TEST_IAM_RETRY_INTERVAL_MS only honored in debug builds
* test(iam): cover deferred bootstrap recovery
Add a dedicated embedded deferred-IAM integration test in a separate test binary to avoid process-global startup collisions.
Strengthen startup IAM recovery coverage with focused unit tests and keep the existing embedded smoke test isolated while carrying the manual test license header update in the same change set.
* fix(startup): tighten deferred IAM recovery path
Adopt follow-up review feedback by silencing misleading app-context warnings after IAM recovery, reusing boundary-aware path prefix checks in the readiness gate, and tying deferred IAM recovery retries to server shutdown tokens.
Keep the deferred IAM embedded integration coverage and startup recovery unit coverage green after the follow-up hardening.
* refactor(startup): simplify IAM recovery task
Collapse the deferred IAM recovery implementation back to a concrete production flow instead of keeping boxed callback seams in the runtime path.
Keep only stable backoff unit coverage in startup_iam and rely on the embedded deferred bootstrap integration test for end-to-end recovery behavior.
* refactor(startup): trim IAM recovery test scaffolding
Keep the concrete deferred IAM recovery path intact while removing bulky test-only async loop scaffolding from startup_iam.
Retain the stable backoff unit checks and rely on the embedded deferred bootstrap integration test for end-to-end recovery coverage.
* fix: apply code review improvements from PR #3188 review
- Simplify RecoveryFuture type alias by removing unnecessary lifetime
- Fix finalize_iam_recovery to return Err if app context unavailable
- Update bootstrap_or_defer_iam_init doc comment to reflect Err case
- Use boundary-aware has_path_prefix for admin path matching in utils.rs
- Add test for adminx boundary rejection in utils.rs and layer.rs
- Improve embedded deferred IAM test with timeout wrapper
* style: merge has_path_prefix import into existing use block
* fix(iam): address final review follow-ups
- fix main startup readiness publication to pass ServiceStateManager correctly
- centralize IAM test env keys in rustfs_config and reuse them in runtime/tests
- keep deferred IAM bootstrap validation aligned with the final review fixes
* fix: isolate listing timeouts from drive health
Keep walk_dir scanner timeouts request-scoped instead of marking local drives faulty.
Add regression coverage for follow-up bucket info, set-level list_path, and system-prefix listings after prior walk timeouts.
* test(iam): gate deferred bootstrap test to debug
Align the deferred IAM embedded integration test with debug-only IAM fault injection hooks so release-profile runs do not assert deferred bootstrap behavior that cannot be triggered.
* test(ecstore): bound prior walk timeout regressions
- set walk_dir stall timeout explicitly in prior-timeout listing tests
- keep the system-prefix follow-up listing scoped to the same base dir
- assert the expected directory entry so the timeout regression test stays fast and stable
* fmt
* fix: keep scanner walk timeouts from offlining drives
Scanner walk operations can time out on large or slow directory listings without proving the backing drive is faulty. Keep the timeout error local to the scan while preserving failure marking for ordinary disk operations.
Constraint: Scanner walk_dir can include listing work that exceeds the drive timeout under slow storage.
Rejected: Disable timeout failure marking globally | real stuck disk operations must still affect drive health
Confidence: high
Scope-risk: narrow
Directive: Do not route scanner/listing timeout back into drive offline state without reproducing issue #2651
Tested: cargo test -p rustfs-ecstore timeout
Tested: cargo fmt --all --check
Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings
Tested: make pre-commit with en_US.UTF-8 locale
Related: https://github.com/rustfs/rustfs/issues/2651
* fix: keep remote scanner walks from offlining drives
Remote walk_dir uses a streaming request that can hit total or stall timeouts during large scans without proving the remote drive is faulty. Route that scanner path through an explicit ignore action while preserving failure marking for ordinary remote operations.
Also treat Duration::ZERO as no operation timeout for remote health tracking, matching the existing local disk wrapper contract while still marking network-like errors on default paths.
Constraint: Scanner walk_dir streams can be slow because of directory size or consumer backpressure.
Rejected: Disable remote timeout failure marking globally | normal remote disk operations still need to evict failed connections and update drive health
Confidence: high
Scope-risk: narrow
Directive: Do not reintroduce remote scanner timeout failure marking without reproducing issue #2651 against remote disks.
Tested: cargo test -p rustfs-ecstore execute_with_timeout
Tested: cargo test -p rustfs-ecstore timeout
Tested: cargo fmt --all --check
Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings
Tested: make pre-commit with en_US.UTF-8 locale
Related: https://github.com/rustfs/rustfs/issues/2651
* test: prove scanner walk backpressure keeps drives online
Add a local scanner walk regression test where the output writer never makes progress. The test confirms the walk timeout returns without marking the drive faulty, covering a stall source that is not caused by object count or network transfer speed.
Constraint: Scanner walk can block on downstream writer backpressure as well as disk or network IO.\nConfidence: high\nScope-risk: narrow\nTested: cargo test -p rustfs-ecstore walk_dir_writer_backpressure_timeout_does_not_mark_drive_failure\nTested: cargo test -p rustfs-ecstore timeout\nTested: cargo fmt --all --check\nTested: cargo clippy --workspace --all-features --all-targets -- -D warnings\nTested: make pre-commit
---------
Co-authored-by: houseme <housemecn@gmail.com>
cxymds