* fix(ecstore): stop scan_dir emitting entries past a limit hit inside a subdirectory
scan_dir's flush loop recurses into a pending subdirectory when the
current sibling entry's page limit is reached mid-recursion, but kept
writing the current (later-sorting) entry regardless. gather_results
then builds the next page's continuation marker from that later entry,
which permanently skips the still-unscanned tail of the subdirectory
on resume instead of just deferring it to the next page.
Add a limit re-check right after the flush loop, before the current
entry is written, so scan_dir stops cleanly at the true last-written
key. Reproduces and fixes the rc.5 recursive ListObjectsV2 data-loss
report (7826/7881 keys, contiguous 55-key block silently dropped).
Adds scan_dir_does_not_emit_entries_past_a_limit_hit_inside_a_subdirectory.
* fix(ecstore): re-check the page limit on every dir_stack flush iteration
The flush loop that drains dir_stack can pop and recurse into more than
one pending subdirectory per outer iteration (whenever more than one
stack entry sorts below the current sibling entry). The limit re-check
added in the previous commit only ran once, after the whole flush loop
exited - so if the first recursive scan_dir call already exhausted the
page limit, the loop's next pop+recurse still went ahead and scanned
(and emitted entries for) another subdirectory beyond where the page
was supposed to stop.
Confirmed against production data: a bucket with ~1.17M objects under
one prefix still cut a recursive ListObjectsV2 listing short (825 of an
expected much larger next page, IsTruncated=false) even with the first
fix deployed, at a two-level-nested subdirectory. Move the check inside
the while loop so it runs before every pop, not just once after.
---------
Co-authored-by: Claude Agent <agent@local>
* fix(ilm): enqueue committed tier free versions
* fix(ilm): stabilize causal cleanup CI coverage
* test(ilm): make expire GET race deterministic
* test(ilm): synchronize expiry with active GET
refactor(ecstore): make store-to-disk error narrowing a named fallible operation
Backlog#1845 step 4. The blanket impl From<StorageError> for DiskError let ? silently push store-only errors (locks, buckets, quotas) across the disk boundary into DiskError::other, where the rendered message fragments reduce_errs quorum buckets. Same story for the blanket From<StorageError> for rustfs_filemeta::Error and its other() catch-all.
Both impls are replaced by named, fallible methods: StorageError::narrow_to_disk() and StorageError::narrow_to_filemeta(). Variants with an identity on the far side map across unchanged - including the two documented lossy collapses (SlowDown -> TooManyOpenFiles, StorageFull -> DiskFull) that the round-trip tests pin - and everything else returns Err(self) so the call site decides what crossing the boundary means. Removing the impls let the compiler enumerate every conversion site; the census that scoped this issue had found 5, the compiler found 33.
Call sites keep their existing behavior: the io identity bridge and the generic sites fold Err into the io-backed other() exactly as the old catch-all did (identity still recoverable by downcast), listing paths use one shared to_filemeta_err helper, and the two sites that relied on the SlowDown collapse now construct DiskError::TooManyOpenFiles directly so the loss is visible where it happens. No behavior change intended anywhere; the io::Error bridge itself is untouched by design.
Ref rustfs/backlog#1845
Route strict inline rename_data dst-parent fsync through the default-off group-commit helper when enabled while preserving the namespace file-sync limited path by default.
Co-authored-by: heihutu <heihutu@gmail.com>
Use Linux openat2 with RESOLVE_BENEATH and RESOLVE_NO_SYMLINKS for LocalDisk I/O path validation while keeping the existing lstat walk as the public-path and unsupported-kernel fallback. Add focused regression coverage for traversal, symlink swaps, missing leaves, recreated parents, high-cardinality prefixes, final symlink leaves, and concurrent validation.
Co-authored-by: heihutu <heihutu@gmail.com>
Add default-off PUT stage attribution for the rename_data sync tail so strict durability probes can split queue wait, fdatasync, directory fsync, rename, per-disk wait, and quorum wait without changing commit ordering or S3-visible behavior.
Co-authored-by: heihutu <heihutu@gmail.com>
Replace every bare `#[allow(dead_code)]` in ecstore with either a deletion or a per-item allow carrying a `reason`. Blanket allows at module, struct, and impl level silence the lint for future members too, so each is narrowed to the members that are actually dead.
Delete the dead cluster in `config/heal.rs` (`Config`, its three methods, `RUSTFS_BITROT_CYCLE_IN_MONTHS`, `parse_bitrot_config`) rather than annotate it: it has no callers and is unreachable outside the crate, and `parse_bitrot_config` would panic on its disabled path via `Duration::from_secs_f64(-1.0)`. `DEFAULT_KVS` stays, since the config registry uses it.
Correct two `reason` strings on `Checksum::new` and `PutObjReader::md5_current_hex_string`, which are methods but carried a field-only rationale.
Refs backlog#1823
Co-authored-by: houseme <housemecn@gmail.com>
* chore(ecstore): drop the disk dead_code blanket
Removing the blanket exposes 36 items in the lowest storage layer: 7 deleted, 29 kept with reasoned item-level allows. That is the smallest deletion share of this burn-down, and the reason is a verification limit rather than a judgement call.
disk/local.rs carries 141 `#[cfg(target_os = "linux")]` sites — the densest platform gating in the tree, because O_DIRECT and io_uring only exist there. The direct-I/O cluster (six ENV_RUSTFS_OBJECT_DIRECT_IO_* constants plus is_direct_io_read_enabled, is_direct_io_write_enabled, get_direct_io_read_threshold, direct_write_staging_capacity, direct_write_tail_split and DIRECT_WRITE_STAGING_BYTES) reads as dead on macOS purely because its production callers at local.rs:1766, 3114 and 4605 sit inside Linux-gated blocks. direct_write_staging_capacity even documents itself as "Platform-independent (no O_DIRECT), so it is unit-tested on any host".
Deleting those would leave every local check green — 4096 tests pass, clippy is clean, make pre-commit exits 0 — and break the Linux build in CI, because all four local lanes compile for aarch64-apple-darwin. Cross-checking locally is not available either: cargo check --target x86_64-unknown-linux-gnu fails in the aws-lc-sys build script for want of a Linux C cross-compiler. Their allows name the platform reason so the next reader on a non-Linux host does not repeat the investigation.
Deleted, all in files with no target_os gating at all (os.rs, disk_store.rs):
- HealthDiskCtxKey and HealthDiskCtxValue with its private log_success. Note that DiskHealthTracker::log_success is a different method of the same name and is live from cluster/rpc/peer_s3_client.rs and remote_disk.rs — the two have to be told apart by type, not by name.
- LocalDiskWrapper::new_with_health and check_id.
- os.rs file_exists and lock_destination_directory_for_path_access.
Kept with allows: DiskHealthTracker's set_faulty, mark_offline, waiting_count and last_success have test callers in remote_disk.rs, so they only look dead in the lib target. to_disk_error, remove_all and sync_dir_files are asserted by their own files' tests. The reclaim, mmap and path-cache field groups are written but never read back.
Placement follows the same rule as the earlier roots: per-method allows inside impl DiskHealthTracker and impl LocalDisk, since both are mostly live and a block-level allow would be a smaller version of the blanket this issue removes. Struct-level allows are used only where the warning covers that struct's own fields. The three cached_read_env! functions take their allow inside the macro invocation, before the fn line, because the macro forwards $(#[$meta:meta])* onto the generated item.
Verification, four lanes warning-free: default, --tests, --features rio-v2 --tests, --features test-util --tests. cargo nextest run -p rustfs-ecstore 4096 passed; clippy --lib --tests -D warnings clean; make pre-commit exit 0. The Linux lane is not covered locally and is left to CI.
Ref rustfs/backlog#1823 (step 2).
* chore(ecstore): correct two dead_code reasons in the disk root
check_valid_path and reject_symlink_components have no caller at all -
not even a test - so 'asserted by this file's tests' misreads them as
covered. Both are method wrappers over live free functions; say that
instead.
Ref rustfs/backlog#1823.
Record local read_version path resolution, path length check, xl.meta read, and metadata decode durations through the existing GET stage metrics channel. The new samples are gated by GET stage metrics so metrics-off reads avoid timer and recorder work.
Co-authored-by: heihutu <heihutu@gmail.com>
* perf(metrics): attribute PUT stage costs
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): move PUT metadata during shuffle
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(s3): reuse PUT object lock state
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): trim PUT metadata fanout clones
Build per-disk PUT metadata only for committed writer slots, move the response metadata out of the fanout vector, and preserve fresh FileInfo shuffle semantics.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(metrics): make PUT stage attribution opt-in
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): commit inline PUT shards directly
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): streamline rename staging cleanup
Use the directory-specific removal operation for rename_data staging parents. This avoids a guaranteed failed file-removal probe on Unix-like hosts and lets Windows remove the empty directory directly while preserving best-effort non-empty handling.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): cover inline PUT rename failures
Cache the detailed stage metrics gate once per PUT and exercise exact-quorum and quorum-minus-one failures after inline shard encoding.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
fix(get): give UringBackend the only fd cache for its disk (#1801)
#1801 made `StdBackend::new` always build a descriptor cache. `UringBackend`
wraps a `StdBackend` (`inner`), so under io_uring a disk ended up with TWO
`FdCache`s: the wrapper's and the inner's. `UringBackend::pread_bytes`
delegates to `inner.pread_bytes` on four fallback paths (latch-off, O_DIRECT
unsupported / error, buffered-read error), which populated `inner.fd_cache` —
but `UringBackend`'s invalidation only touches its own cache, so the inner
cache was never invalidated. For up to `FD_CACHE_TTL` (5s) after a heal/rename/
delete, a fallback read could serve the pre-mutation inode: exactly the
stale-descriptor hazard `FdCache`'s generation guard exists to close
(rustfs/backlog#1176). It also double-counted `FD_CACHE_CAPACITY` (512 fds)
against `RLIMIT_NOFILE` per disk (backlog#1178).
Fix: `UringBackend` now constructs its inner `StdBackend` with the new
`StdBackend::new_without_fd_cache`, so the wrapper owns the only cache for the
disk. The inner backend opens per read on fallback, leaving nothing
unguarded. `StdBackend::new` (standalone default) is unchanged; a private
`build(root, build_fd_cache)` holds the shared construction.
- Default (non-io_uring) path: byte-for-byte unchanged.
- io_uring path: one cache per disk, fully covered by the wrapper's
invalidation; halves the per-disk fd budget under `RLIMIT_NOFILE`.
- `RUSTFS_IO_URING_FD_CACHE` / `RUSTFS_LOCAL_FD_CACHE` semantics preserved.
- Regression test pins `new_without_fd_cache` -> no cache.
Found by a post-merge re-review of the Wave 1 GET PRs. cargo check/clippy
clean; Linux compile + the io_uring fd-cache suite deferred to CI.
Co-authored-by: heihutu <heihutu@gmail.com>
StdBackend opened, stat'd, and access-checked the shard file on every
positioned read, so each small-object GET paid N x (open + access + fstat)
syscalls even for hot shards. UringBackend already caches descriptors
behind a generation-guarded, rlimit-bounded moka cache with full
rename/delete/heal invalidation; StdBackend had no equivalent.
Port that cache to the default backend:
- StdBackend gains an `fd_cache: Option<FdCache>` (Linux only, mirroring
UringBackend), built in `new()` behind `RUSTFS_LOCAL_FD_CACHE` (default
on) and the same `rlimit_allows_fd_cache` guard.
- pread_bytes consults the cache on the buffered path: a hit reuses the
descriptor via `dup` (one syscall, no path resolution or permission
re-check) and skips volume access; a miss opens as before, snapshots the
invalidation generation, and hands the freshly opened descriptor back
for `insert_if_fresh`, which refuses to cache if a heal/delete bumped the
generation mid-open (rustfs/backlog#1176). O_DIRECT reads keep opening
their own aligned descriptors.
- The DirectReadCopy branch switches from seek+read_exact to
`FileExt::read_exact_at`: a `dup`'d cached descriptor shares the source
descriptor's open-file offset, so a positioned read (like the mmap path's
offset argument) keeps concurrent cache hits on the same shard correct.
- The four `LocalIoBackend` invalidation methods now drop stale entries on
StdBackend. LocalDisk already calls them on rename_data/rename_file/
delete/delete_volume/close, so no new call sites are needed.
- Two tests mirror the io_uring ones: a heal rename must be hidden until
invalidate_cached_fds_under runs, and a repeated read caches exactly one
descriptor that prefix invalidation drops.
Behavior is byte-for-byte unchanged on a miss and on non-Linux; the cache
is auto-disabled only when RLIMIT_NOFILE is too low. macOS cargo check
--lib and --tests pass; Linux compile deferred to CI.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ecstore): publish multipart parts on Windows
* test(ecstore): pin Windows multipart durability
---------
Co-authored-by: houseme <housemecn@gmail.com>
* fix(ecstore): anchor Windows rename publication
* fix(ecstore): complete Windows rename confinement
* test(ecstore): retain Windows retry assertion path
* fix(ecstore): accept configured Windows root paths
* fix(ecstore): size Windows rename buffers correctly
* fix(ecstore): use native relative rename on Windows
* fix(ecstore): preserve Windows rename parent guards
* fix(ecstore): reuse guarded Windows rename trees
* fix(ecstore): compile Windows publication helpers
* fix(ecstore): preserve configured Windows disk roots
* fix(ecstore): flush Windows shards with write access
* fix(ecstore): stage Windows rollback backup replacement
* fix(ecstore): defer Windows staged file cleanup
* fix(ecstore): type Windows staged write result
* fix(ecstore): retry Windows sharing violations
* fix(ecstore): share Windows staged deletes
* fix(ecstore): split Windows staged publication handles
* fix(ecstore): close Windows staged writer before rename
* fix(ecstore): share Windows staged publication deletes
* fix(ecstore): allow guarded Windows child publication
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
* fix(ecstore): purge the stale destination data dir on healing rename_data commits
Heal commits reuse the version's existing data_dir, so when repairing
in-place corruption (bitrot) the destination directory still exists and
holds the corrupt shard files. rename(2) cannot replace a non-empty
directory (EEXIST on XFS, ENOTEMPTY on ext4), so the commit failed on
every attempt — including all scheduler retries — and in-place bitrot was
detected and reconstructed but never repaired.
Purge the stale destination data dir (move_to_trash) before the commit
rename, for healing commits only: fresh PUTs mint a new data_dir and can
never collide, and a non-healing collision keeps failing loudly. Adds the
FileInfo::is_healing() reader for the marker set_healing() already writes.
* style(ecstore): emit the heal purge failure as a structured event
The new warning was the only sentence-style log in `rename_data`'s commit
path — it sat ten lines above `info!(event = EVENT_DISK_LOCAL_RENAME_REJECTED,
component = ..., subsystem = ...)` and interpolated its values into the
message instead of carrying them as fields, so it is invisible to any operator
query keyed on `event`.
Give it the shape the rest of the file uses: a named
`EVENT_DISK_LOCAL_HEAL_PURGE_FAILED`, `component`/`subsystem`, `dst_path` and
`error` as fields, and a short label as the message. Level stays `warn` — the
purge is best effort and the rename below fails closed — and the condition,
the branch, and the control flow are unchanged.
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
#5724 reclaimed the synthetic inline-rollback dir after a committed rename with delete_data_dir(recursive: true), which has no notion of object metadata: for unversioned objects the synthetic UUID is a fixed, publicly-known constant, so object/<rollback-dir> can simultaneously be a legitimate child key's directory, and recursively deleting it reopens the authorization bypass #5703 closed (PutObject on K destroying K/<uuid> without DeleteObject permission).
Replace the recursive pass with a file-precise one: after quorum commit, delete exactly object/<rollback>/xl.meta.bkp with a non-recursive delete on every disk whose rollback dir is not also the cleanup dir. The parent-rmdir walk removes the dir only when the backup was its sole content, so the BucketNotEmpty leak fix is preserved (#5724's regression test passes unchanged) while a child key at the same path keeps its metadata. The undo path's restore_metadata_backup now also reclaims the emptied synthetic dir, mirroring restore_delete_rollback.
Collapse per-shard blocking-pool round-trips into one spawn_blocking
pread batch when all online shards are local and mmap-read is enabled.
Co-authored-by: ba <ba@ubuntu-server.alpha30.bos16>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
A streaming GET holds snapshot leases on the object's data directories,
and DeleteObjects defers their physical cleanup until the leases are
released. A DeleteBucket issued inside that window passes the xl.meta
emptiness check but fails closed in the non-force delete_volume tree
removal on the leftover part files, returning BucketNotEmpty for a
logically empty bucket.
This is what intermittently failed the s3tests
test_encryption_sse_c_multipart_bad_download teardown in CI: the test
never reads its 30MiB GET body, so the server-side stream (and its
leases) stays alive until the connection drops, racing the teardown's
DeleteObjects + DeleteBucket sequence. With the body held open the
failure reproduces 5/5 locally; after this change it passes 20/20, and
the real s3-tests case passes 20 consecutive runs.
delete_volume now executes the registry-tracked pending deferred
deletions for the volume before removing the directory tree. Only data
dirs whose logical delete already committed are touched; unknown files
still fail closed with VolumeNotEmpty.
Record local read_dir entry counts and duration for live-walker ListObjects scans so wide root/prefix amplification can be measured below the cross-set merge layer.
Add a focused LocalDisk scan_dir test showing a page limit of one still observes the whole parent directory enumeration, plus metric helper coverage.
Co-authored-by: heihutu <heihutu@gmail.com>
Sergey Shlukov