A streaming GET holds snapshot leases on the object's data directories,
and DeleteObjects defers their physical cleanup until the leases are
released. A DeleteBucket issued inside that window passes the xl.meta
emptiness check but fails closed in the non-force delete_volume tree
removal on the leftover part files, returning BucketNotEmpty for a
logically empty bucket.
This is what intermittently failed the s3tests
test_encryption_sse_c_multipart_bad_download teardown in CI: the test
never reads its 30MiB GET body, so the server-side stream (and its
leases) stays alive until the connection drops, racing the teardown's
DeleteObjects + DeleteBucket sequence. With the body held open the
failure reproduces 5/5 locally; after this change it passes 20/20, and
the real s3-tests case passes 20 consecutive runs.
delete_volume now executes the registry-tracked pending deferred
deletions for the volume before removing the directory tree. Only data
dirs whose logical delete already committed are touched; unknown files
still fail closed with VolumeNotEmpty.
rename_all_missing_source_still_warns and rename_all_real_failure_still_warns
assert that `warn_reliable_rename_failure` emitted its WARN, but that is a
single production callsite shared with tests that call rename_all *without*
installing a subscriber — rename_all_missing_source_returns_file_not_found,
two tests above, is one of them.
tracing caches each callsite's Interest process-globally and the first thread
to reach a callsite fixes that value; while at most one dispatcher is
registered, tracing-core derives it from the registering thread's own
subscriber, and registration is once-only. When the subscriber-less sibling
wins, the callsite is cached as Interest::never() and the WARN never fires,
so the assertion sees empty output:
ordinary missing-source failures must keep the WARN, got:
Reproduced at 3/25 with `disk::os::tests::rename_all_missing_source` (both
tests), against 0/20 for the victim alone. Fixed by pinning callsite interest
inside warn_capture(), so every current and future user of that helper is
covered rather than just the two tests that happen to fail today.
pin_callsite_interest_for_test() moves from cluster::rpc::background_monitor
to a new crate-level test_tracing module: it is domain-neutral and now has
consumers in two unrelated subsystems, and disk::os should not have to reach
into a cluster::rpc test helper.
Verified: repro filter 0/30 (was 3/25); disk::os:: 0/12; cluster::rpc:: 0/12
and its poisoner pair 0/15, confirming the moved helper still holds.
Follow-up to #5438. Closes the last item in #5439.
The background tier free-version recovery walk pinned a hardcoded 60s
total wall-clock timeout that overrides every operator knob, so any
bucket whose healthy full walk exceeds 60s fails forever; the failed
run's duration was also subtracted from the next 60s tick, restarting
the walk immediately and pinning CPU and disk I/O.
- Drop the total wall-clock budget on the recovery walk
(walkdir_timeout: Duration::ZERO) and inherit the operator-tunable
drive stall budget (RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS) for
per-call progress, so hung disks still fail fast.
- Back off failed runs from completion time: 60s doubling to a 600s
cap, reset on success; never subtract the failed run's duration.
- Add RUSTFS_TIER_FREE_VERSION_RECOVERY_ENABLED (default true) to opt
out of the recovery worker on deployments with no remote tiers;
invalid values warn and fail open.
Fixes#5130
Co-authored-by: claude <claude@ehdtn.com>
Record local read_dir entry counts and duration for live-walker ListObjects scans so wide root/prefix amplification can be measured below the cross-set merge layer.
Add a focused LocalDisk scan_dir test showing a page limit of one still observes the whole parent directory enumeration, plus metric helper coverage.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(filemeta): add state-aware file info validation
* fix(filemeta): validate shard arithmetic and delete paths
* fix(ecstore): add fallible erasure construction
* fix(ecstore): resolve storage parity per pool
* fix(storage): report heterogeneous erasure layouts
* fix(admin): publish prepared storage config atomically
* fix(storage): harden per-pool parity boundaries
* fix(storage): address pre-PR validation findings
* test(ci): fix strict-topology validation fixtures
* fix(heal): preserve delete markers during repair
* refactor(filemeta): drop unused ValidatedFileInfo witness
ValidatedFileInfo wrapped an unread `_file_info` reference alongside an `Option<ValidatedErasureLayout>`, but only the layout was ever consumed. Return the layout directly from `FileInfo::validate` so the sole production consumer (`LocalDisk::check_parts`) and the two unit tests read it without the extra witness type and lifetime.
No behavior change.
* fix(filemeta): keep compressed and MinIO-migrated tiered objects readable
The new decode-path validation rejected several legitimate on-disk shapes that older RustFS and MinIO-migrated data carry, turning readable objects into FileCorrupt:
- Compressed objects written with an unknown upload size persist a negative per-part actual_size (the documented "unknown size" sentinel that ObjectInfo::get_actual_size already tolerates). validate_collection_contents rejected it via usize::try_from; now a negative actual_size skips shard validation and only real, non-negative sizes are checked.
- MinIO-migrated objects transitioned to a versioned remote tier store the tier version id as a UUID string, not 16 raw bytes. MetaObject::into_fileinfo returned FileCorrupt (main tolerated it as None), making all versions of the object unreadable; MetaDeleteMarker free-version records took a Some(nil) sentinel path with the same effect, which also breaks free-version expiry (remote-tier leak). Both now decode through a shared transitioned_version_id_from_meta_sys helper: 16 raw bytes or a UUID string are accepted, anything else is tolerated as None instead of failing the read.
Regression tests updated to assert the readable/compat behavior, with new tests covering MinIO string-form recovery.
* fix(scanner): build the delete-marker test fixture without erasure geometry
get_size_counts_delete_markers_separately_from_versions built its delete marker with `FileInfo::new(object, 1, 1)`, which attaches erasure geometry (data=1/parity=1/distribution). This PR classifies versions by shape via `is_storage_delete_marker()` (no geometry) rather than the raw `deleted` flag, so a geometry-bearing "delete marker" is correctly serialized as a purge-pending payload Object and counted as a version — CI saw summary.versions=3, expected 2.
Real delete markers carry no erasure geometry (delete paths build them as `FileInfo { deleted: true, ..Default::default() }`), so construct the fixture the same way. It then classifies as a storage delete marker and the counts (versions=2, delete_markers=1) hold. This keeps the PR's more-correct classification, which prevents a purge-pending object's geometry from being dropped when serialized as a bare delete marker.
* docs(changelog): note per-pool parity fix and storage-class startup upgrade caveat
Records the #4801 per-pool erasure parity fix under Fixed, and documents the upgrade behavior where a persisted storage class that a small or heterogeneous pool cannot satisfy now fails startup — with the RUSTFS_STORAGE_CLASS_STANDARD recovery steps. Docs-only; covers R4 from the on-disk compatibility audit.
* fix(heal): report parity from erasure geometry, not is_valid()
heal_object set HealResultItem.parity_blocks via `if lfi.is_valid()`, which was missed by the migration of the other quorum/metadata predicates. With the new `is_valid()` semantics (full payload validation; delete markers now return false), a delete marker or a geometry-bearing version with a benign collection quirk would misreport parity as the pool default instead of its own. Use `has_valid_erasure_geometry()` — the narrow "does this carry erasure geometry" predicate the rest of the migration uses — so reporting matches the object's actual layout. Reporting-only; no data-path change.
* fix(filemeta): do not silently serialize a non-canonical deleted FileInfo as an Object
`From<FileInfo> for FileMetaVersion` classifies by `is_storage_delete_marker()` (shape), which correctly routes canonical delete markers to Delete and purge-pending payloads (deleted=true with real erasure geometry) to Object. But a `deleted` FileInfo that is neither a canonical marker nor a valid erasure payload would silently serialize as a zero-geometry MetaObject that later fails `validate_for_metadata_read`. Write paths validate first (`validate_for_erasure_write` / `validate_for_metadata_read`), so this is a caller bug; `From` is infallible, so surface it with a structured `warn!` on the malformed branch instead of writing corrupt metadata silently. Legitimate purge-pending objects (valid geometry) are unaffected — the guard only fires for `deleted && !has_valid_erasure_geometry()`.
* test(filemeta): assert real historical xl.meta versions pass metadata-read validation
Empirical companion to the code-reasoned decode-tolerance invariants (docs/architecture/erasure-coding.md §11) and the rolling-upgrade / MinIO-migration compatibility concern: the tightened `validate_for_metadata_read` runs on every local disk read and peer-RPC-decoded FileInfo, so it must accept every version of real historically-written xl.meta, never reject it as FileCorrupt.
Loads five real fixtures — MinIO small-inline, MinIO versioned (two object versions + a delete marker), MinIO large multipart, a legacy V1 (xl.json-derived) object, and a legacy meta_ver 2 object — decodes every version with parts materialized, and asserts validate_for_metadata_read() is Ok for each. Reverting the tolerant handling (delete-marker shape, legacy per-part checksums, string/short transitioned-versionID, negative actual_size) turns this red.
* fix(ci): remove duplicate storage test re-exports
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(ecstore): add fallible erasure construction
(cherry picked from commit bd148b20f7)
* fix(ecstore): resolve storage parity per pool
(cherry picked from commit c05c2cb24b)
* fix(ecstore): keep carved per-pool parity core self-contained on main
Fixups so the cherry-picked fallible-erasure + per-pool-parity core builds standalone on current main without the excluded scope-creep commits:
- runtime/sources.rs: re-add backend_storage_class_parities (removed by the per-pool commit; its rebalance caller was updated in an unrelated reporting commit that was left out). Reimplemented over the snapshot API, behavior-identical.
- config/mod.rs: rename the storage-class publish test module (main independently added a mod tests, so the cherry-pick collided).
- rustfs storage_api.rs + startup_storage.rs: route the storage-class ENV consts through the startup storage facade and use a local const for the erasure-set-drive-count env name, satisfying the layer/facade guardrail (main's guardrail is stricter than when the core was authored).
* fix(ecstore): use struct-init in erasure test helper to satisfy clippy field_reassign_with_default
The cherry-picked fallible-erasure commit's `erasure_with_invalid_dimensions` test helper built `Erasure` via `default()` then reassigned fields, which trips `clippy::field_reassign_with_default` under `-D warnings` (only surfaced by `--all-targets`, which lints test code). #4977 fixed this in a later commit that was not part of the carved core. Use struct-init with `..Default::default()`, matching #4977's final form.
* fix(rustfs): gate the test-only storage-class ENV facade re-export behind cfg(test)
The ENV constants (INLINE_BLOCK_ENV/OPTIMIZE_ENV/RRS_ENV/STANDARD_ENV) re-exported through the startup storage facade are only consumed by a #[cfg(test)] test in startup_storage.rs, so in a non-test lib build the re-export is unused and trips -D unused-imports under clippy --all-targets. Gate it with #[cfg(test)], matching #4977's final form.
---------
Co-authored-by: cxymds <cxymds@gmail.com>
POSIX lets rmdir report a non-empty directory as either ENOTEMPTY or EEXIST. Linux/macOS/Windows use ENOTEMPTY (ErrorKind::DirectoryNotEmpty); illumos/Solaris return EEXIST (errno 17), which Rust surfaces as ErrorKind::AlreadyExists and which a DirectoryNotEmpty match never catches.
LocalDisk::delete_file removes xl.meta and then recurses to rmdir the object directory, tolerating only NotFound and DirectoryNotEmpty. Since #4300 (transactional delete rollback-staging, new in beta9) the object directory still holds the rollback backup dir when that rmdir runs — the caller removes it only after write quorum is confirmed — so the rmdir legitimately reports "not empty". On Linux that is tolerated; on Solaris it is EEXIST, which fell through to the catch-all arm and became FileAccessDeniedWithContext. That failed the delete commit, rolled the metadata back, and left the object undeletable, so the client retried indefinitely with a spurious FileAccessDenied and no EACCES anywhere (rustfs/rustfs#4978). The same Linux-errno assumption also broke non-force DeleteBucket on a populated bucket on Solaris.
Add a portable is_dir_not_empty_error classifier (DirectoryNotEmpty kind plus raw ENOTEMPTY/EEXIST), mirroring MinIO's isSysErrNotEmpty, and use it at the two directory-removal sites via is_benign_object_rmdir_error (delete_file) and classify_delete_volume_error (delete_volume). The classifier is applied only at rmdir/remove_dir_all sites, where EEXIST unambiguously means "not empty", so EEXIST keeps its normal meaning everywhere else. rmdir never returns EEXIST on Linux/macOS/Windows, so the new raw-errno branch is unreachable there and the change is a strict no-op off illumos/Solaris.
Adds unit tests for the classifier (DirectoryNotEmpty/ENOTEMPTY/EEXIST match, EACCES/ENOENT reject, real non-empty rmdir against the host errno) and call-site decision tests that make a Solaris EEXIST regression detectable on Linux CI.
Fixes#4978
A PUT that fans out erasure shards to remote peers awaited every shard writer to completion on both the per-block write and the final shutdown, and the remote HttpWriter had no progress deadline. A peer that accepts the TCP connection but never drains the request body (or never sends a response) therefore wedges the writer forever once the bounded buffers fill, pinning an otherwise-healthy write quorum indefinitely — a cluster-level write-availability hazard triggered by a single bad peer (rustfs/backlog#1319, https://github.com/rustfs/backlog/issues/1319).
MultiWriter now wraps each shard write and each shard-writer shutdown in a forward-progress deadline. The budget is re-armed on every block, so it bounds a stall rather than the total transfer time of a large object: a slow-but-honest writer that keeps completing shards is never killed, while a writer that makes no progress within the budget is failed and its disk dropped before commit. An optional absolute per-object cap (disabled by default) backstops a slow-drip peer that dribbles just enough progress to reset the per-block timer without ever converging; it is off by default so a legitimate large upload over a slow link is not killed on total time alone. Both knobs come from RUSTFS_OBJECT_DISK_WRITE_STALL_TIMEOUT (default 30s) and RUSTFS_OBJECT_DISK_WRITE_ABSOLUTE_CAP (default 0 = disabled); setting the stall timeout to 0 restores the previous wait-forever behavior for a conservative rollback.
The deadline enforcement lives in MultiWriter (writer-agnostic), so it covers local and remote writers alike and keeps the existing control-flow shape: a timed-out shard is marked failed (Error::Timeout, which is not an ignored error) and excluded from the write quorum exactly like any other shard write failure, and the unchanged nil_count/quorum check then continues on quorum or fails cleanly. This deliberately stays out of the MultiWriter lifecycle / commit-coordinator territory owned by rustfs/backlog#1312.
When a stalled writer is dropped to fail its shard, the remote HttpWriter must stop holding the connection and its buffered body. HttpWriter previously left its spawned request task running on drop; it now aborts that background task in Drop (it is no longer pin-projected, since every field is Unpin and the AsyncWrite impl already used get_mut). Bytes already handed to the transport cannot be unsent, but they land only in this upload's unique tmp path and are reclaimed by tmp GC — they never touch a committed object.
Tests, all on a paused virtual clock so they are deterministic and non-flaky:
- one black-hole writer still meets a 3/4 write quorum without hanging; two black holes fail the quorum cleanly (both for the per-block write and the shutdown paths).
- a slow-but-honest writer that keeps making progress within the stall budget is never failed across many blocks.
- the absolute cap bounds a slow-drip writer within a finite budget while the healthy writers keep quorum.
- the default policy is armed by default and honors 0 as disabled.
- HttpWriter aborts its background request task on drop against a hanging peer.
The toxiproxy/black-hole 4x4 end-to-end acceptance depends on black-box test facilities from rustfs/backlog#1325, which are not built yet; that acceptance is deferred to #1325 and intentionally not faked here.
* fix(obs): open cleaner compression source with O_NOFOLLOW
The compressor opened the source log via File::open, which follows a
symlink at the final path component. Between the scanner selecting a
regular file and this open, an attacker with write access to the log
directory could swap the entry for a symlink (TOCTOU) pointing at, say,
/etc/shadow, whose contents would then be copied into an archive. Open
the source with O_NOFOLLOW on Unix so such a swap fails with ELOOP; the
temp/archive path already refused symlinks, this closes the source side.
Refs rustfs/backlog#1210
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): recompress instead of trusting leftover cleaner archives
archive_header_ok only checked the first 2-4 magic bytes before treating
an existing .gz/.zst as a completed prior result and letting the caller
delete the source log. A file with valid magic but a truncated or forged
body passes that check, so an attacker with write access to the log
directory (or a crashed prior run) could plant such a stub and make the
cleaner delete the real log without ever producing a usable archive —
silent audit-data loss.
Chosen fix: stop trusting cross-process leftovers entirely and always
recompress the source in this pass, rather than fully decoding every
leftover to validate it. Full-decode validation would add real CPU cost
and decode-bug surface for a rare crash-recovery case; the existing
atomic create_new+rename already overwrites whatever sits at the archive
path (a planted symlink is replaced, never followed) with a freshly
written, fsync'd archive, so a partial/forged leftover can never gate
source deletion. This is the lowest-regression option.
Refs rustfs/backlog#1211
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): cap memory-gate reservation at cache growth headroom
The memory gate subtracts `admitted_since_refresh` from the snapshot's
available bytes so a burst arriving faster than the 5 s refresh cannot
over-allocate. That counter is GROSS: it only rolls over on the refresh and
never rolls back when a fill is later evicted, cancelled, or loses the
invalidation race. Under sustained high-throughput churn (net footprint flat
and far below `max_capacity`) the raw counter balloons past the memory the
cache actually holds, so `effective_available` collapses and the gate reports
false memory pressure — skipping the hottest fills with SkippedMemoryPressure
until the next 5 s refresh. This only lowers hit rate; it never returns wrong
data and self-heals each refresh.
Fix direction 1 (minimal regression): cap the reservation deduction at the
cache's own growth headroom (`max_capacity - weighted_size()`) instead of
letting the unbounded gross counter shrink the system-available budget. The
cache can never hold more than `max_capacity`, so a burst adds at most that
headroom of real memory before moka evicts to stay bounded (net-zero churn
beyond that point) — capping the deduction there keeps the reservation honest
without treating gross churn as growth. Chosen over net-accounting (direction
2, releasing bytes on every failure/cancel/eviction path) because that only
plugs the leak on failed fills and would not address the core defect: churn of
*successful* insert/evict fills over the 5 s window. It also touches only the
gate plus one call site rather than every failure path in moka_backend.
The cap only ever raises `effective_available`, so real memory pressure (a low
snapshot at refresh) still suppresses fills; when the cache is at capacity the
headroom is 0 and the deduction vanishes, correctly reflecting net-zero churn.
`MokaBackend` now stores `max_capacity` and passes the live headroom into
`allows_fill`. Adds targeted gate tests: gross churn far above headroom no
longer falsely suppresses, yet the reservation still bounds a burst while the
cache can genuinely grow.
Refs rustfs/backlog#1212
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): assert native O_DIRECT path runs in uring read test
uring_preserves_o_direct_for_eligible_reads only compared bytes through
LocalDisk::read_file_mmap_copy. On a filesystem that rejects O_DIRECT the
read silently degrades to the buffered StdBackend fallback and the byte
check still passes, so the test could go green without the native
read_at_direct path ever executing -- a vacuous pass.
Add a per-disk native_direct_reads counter on UringBackend, incremented
only when pread_uring_direct completes, and rebuild the test to drive a
real UringBackend's pread_bytes and assert the counter is non-zero (every
eligible read went through the native tier). When io_uring or O_DIRECT is
unavailable on the host filesystem (restricted CI runners, tmpfs), the
test skips loudly via eprintln instead of asserting a tautology, while
still checking byte-correctness on whatever tier served the read.
The counter also gives a gray release a positive signal that the O_DIRECT
tier is serving reads, not just a fallback count.
Refs rustfs/backlog#1213
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): warn + count read-time EINVAL on native O_DIRECT reads
classify_direct_read_error is only reached from the read side: the
O_DIRECT open in pread_uring_direct already succeeded (an open-time
refusal is handled earlier as DirectOpenError::ODirectRefused). So an
EINVAL/EOPNOTSUPP arriving here is a read-time error on an fd the kernel
accepted for O_DIRECT -- far more likely an alignment bug in the aligned
read path than an unsupported filesystem. The old code latched the disk's
native path off with only a once-per-disk debug trace, hiding a potential
correctness regression behind a silent buffered-read downgrade.
Diagnostics only: the fallback behaviour is unchanged (the native path is
still latched off and the caller still reads via StdBackend). This adds a
rustfs_io_uring_direct_read_einval_total counter and promotes the
once-per-disk trace from debug to warn so an operator can see an alignment
regression instead of an unexplained latency/CPU shift.
Refs rustfs/backlog#1214
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(ecstore): document data-blocks-first default and its tail-latency cost
DEFAULT_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP is true and must stay
true: deferred-parity is the deliberate, already-rolled-out full-object
GET default from backlog#1159/#923. Flipping it back to false in code
would silently revert that rollout for every deployment that has not set
the env var, so this commit only documents -- no behaviour change.
The added notes explain what data-blocks-first does (schedule data shards
up front, engage parity lazily on a missing/corrupt data shard), the known
trade-off (parity is engaged late, so a slow-but-not-dead data drive
raises GET p99 because the faster parity shards are not raced against it
until a data shard is declared missing), and the operational rollback
switch (RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP=false), which is
intentionally an env override rather than a code default change.
No metric was added: the low-risk observability hook for "slow data drive
engaged deferred parity" would live at the deferred-stripe engage point,
which is out of this file's scope; this change stays documentation-only to
avoid touching the hot GET path.
Refs rustfs/backlog#1215
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(ecstore): document wide-directory walk stall hazard and tuning
list_dir enumerates a whole directory in one os::read_dir call (count =
-1), and the walk caller bounds that entire enumeration with the per-read
stall budget (default 5s) as if it were a single read. For a wide, flat
prefix -- one directory holding millions of immediate children -- a single
readdir can exceed the budget on a healthy disk, trip DiskError::Timeout,
and surface as a ListObjects 500 quorum failure though the drive is fine
(a #2999 sub-class).
This is documented, not rewritten: turning the one-shot readdir into a
streaming/batched enumeration that refreshes the stall deadline between
chunks is an architecture-level change with high regression surface
(ordering, the count contract, quorum merge) and belongs in a separate
follow-up. The supported mitigation today is operational, so the comments
point wide-directory deployments at RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS
and the high-latency drive-timeout profile, which widen the budget with no
code change. Notes were added at list_dir, the scan_dir call site, and
get_drive_walkdir_stall_timeout. No behaviour change.
Refs rustfs/backlog#1216
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(ecstore): document consumer-peek vs producer-stall coupling
In list_path_raw the consumer's peek_timeout is drawn from the same source
and same value (walkdir_stall_timeout, default 5s) as the producer-side
walk stall budget, but the two measure different things: the producer
stall bounds a single drive read, while the consumer peek bounds the gap
between two ADJACENT entries arriving from a reader. Because they share a
value, the consumer cannot wait meaningfully longer for the next entry
than the producer is allowed to spend producing one. Walking a region
dense with non-listable internal items can make a HEALTHY drive miss the
budget between visible entries; the consumer then declares it stalled and
detaches it, dropping a good drive from the merge and capping the "large
prefix succeeds" guarantee.
Documented, not decoupled: giving the consumer peek an independent,
strictly-larger budget would cut these false detaches but equally delays
detaching a genuinely dead drive and shifts listing tail-latency
semantics, so it wants soak data before changing the default. The comment
records the invariant any such follow-up must keep -- consumer peek >=
producer stall, never stricter -- so it can never fail a drive before the
producer would. No behaviour change.
Refs rustfs/backlog#1217
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(io-metrics): add time-based trigger for low-IOPS latency percentiles
Percentiles were recomputed only every 128 IOs and seeded to 0, so a
low-traffic deployment exported p95/p99 = 0/stale for a long time after
startup. Add a 10s wall-clock trigger alongside the count throttle so the
first recompute can fire before 128 samples accrue. Hot-path per-op mean
update is unchanged.
Refs rustfs/backlog#1218
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): cover codec-streaming parity under fault injection and NoSuchKey
The codec-streaming compat A/B previously ran only against a healthy
4-disk EC set with successful full GETs: the DiskFaultHarness was
constructed but never faulted, the error path was untested, and the
range assertion silently compared legacy-vs-legacy (ranges always fall
back to the duplex path), overstating what it proved.
Add two genuinely-failable scenarios reusing the existing harness and
fixtures:
- Parity reconstruction A/B: take one data disk offline and re-run the
full object matrix on both phases while the EC 2+2 set rebuilds each
large object from the surviving shards. Assert codec == legacy
byte-for-byte (sha256) and header-for-header, and assert the codec
phase served the reconstructed objects with zero duplex-pipe fallback
(the reader gate is drive-health-independent, so the codec fast path
is really exercised through reconstruction).
- NoSuchKey negative path: compare the HTTP status + S3 error code of a
missing-key GET across the legacy and codec phases and require them to
be identical (404/NoSuchKey), guarding against the codec env
perturbing the error path.
Also clarify the range-phase comment so it is not misread as
codec-range correctness coverage: both sides are served by the same
legacy range path, so the assertion only proves ranges keep working and
keep falling back to legacy with the gates open.
Verified: cargo check/--no-run pass and the test passes locally
(1 passed; dup_codec=0 confirms the codec path ran).
Refs rustfs/backlog#1219
Co-Authored-By: heihutu <heihutu@gmail.com>
* ci(ecstore): exercise native O_DIRECT read path on an ext4 loopback
The uring-integration leg ran on the runner's default TMPDIR, which may sit
on tmpfs/overlayfs where open(O_DIRECT) fails and the native read_at_direct
path silently latches off to the aligned StdBackend fallback. Mount a
dedicated ext4 loopback and point TMPDIR at it so the real io_uring dep
(bumped git->0.1.0->0.2.0->0.2.1) and the native O_DIRECT read path are
actually covered rather than validated only by signature diffing.
Refs rustfs/backlog#1220
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* Change Rust toolchain channel to stable
Signed-off-by: houseme <housemecn@gmail.com>
* style: apply clippy --fix and cargo fix lint suggestions
Run `cargo clippy --fix --all-targets --all-features` and
`cargo fix --lib --all-targets` across the workspace, then resolve the
remaining warnings by hand:
- collapse needless borrows in `format!` args, prefer `?` over explicit
early returns, and use `.values()` / `.flatten()` iterator adapters
- rewrite the `Md5` scan loop via `manual_flatten` and re-indent the
`select!` macro body (rustfmt skips macro interiors)
- annotate the intentional dead-code `Md5` inherent methods (constructed
only by the test factory) with `#[allow(dead_code)]`
Behavior is unchanged.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(obs): prevent rolling log stdout aliasing
* fix(ecstore): bound hot-path tracing payloads
* test(logging): guard service and disk log invariants
* fix(obs): silence useless_conversion on st_dev for Linux clippy
rustix's Stat.st_dev is u64 on Linux/glibc, making u64::try_from a no-op
that trips clippy::useless_conversion under -D warnings. The conversion is
still needed on macOS/BSD where st_dev is a signed dev_t, so suppress the
lint on that line rather than dropping the portable fallible conversion.
* fix(logging): keep stdout sink validation portable (#4769)
* fix(obs): keep stdout device conversion portable
* docs(logging): update single-writer plan status
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
fix(test): use canonicalized disk root for file_sync_probe in rename_data test
On macOS, tempfile::tempdir returns /var/folders/... while LocalDisk
resolves the root to /private/var/folders/... via dunce::canonicalize.
The file_sync_probe::enter() path check uses starts_with(), so passing
the non-canonical tempdir path caused the probe to never activate,
making wait_for_active() hang indefinitely.
Use disk.root (already canonicalized) for the probe instead.
Bound large shard-directory syncs per disk and process while preserving small-directory and durability behavior.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
`record_uring_fallback` is called at multiple sites in the io_uring read
path whenever a read falls back to `StdBackend`. It formatted
`self.root.display().to_string()` on every call, heap-allocating a
`String` from a `Path` that never changes after construction — pure
per-read waste when io_uring is degraded.
Cache the label once in `UringBackend::try_new` as a `String` field
(`root_label`) and clone it per emission. The metric name and the
`"root"` label value are unchanged; only the redundant `Path` formatting
is removed. The clone is a single alloc of an already-short string.
Refs: rustfs/backlog#1185
Co-authored-by: heihutu <heihutu@gmail.com>
* perf(metrics): drop needless per-emission work on hot metric paths
Audit of the metrics hot paths surfaced four low-risk wins where emission did
work it did not need to:
- `record_file_cache_reclaim_success/error` (disk/local.rs) called `.to_string()`
on `kind` (already `&'static str`) and on the `"ok"`/`"err"` literals, heap-
allocating up to four `String`s per page-cache reclaim window — which runs per
read-stream reclaim. The `metrics` macros accept `&'static str` label values
directly, so pass them as-is.
- `record_read_repair_dedup` (set_disk/core/io_primitives.rs) likewise
`.to_string()`-ed an already-`&'static str` `reason`.
- `SetDisks::get_object_reader` (set_disk/ops/object.rs) captured `Instant::now()`
and emitted the `rustfs.lock.acquire.*` counter and histogram unconditionally
on every GET, right beside an already-gated stage timer. Gate them behind
`get_stage_metrics_enabled()` too, so an inactive observability config pays no
per-GET clock read or recorder lookups.
- The per-response-body-chunk counter in server/http.rs re-ran the `counter!`
registry lookup on every chunk (a streamed GET emits many). Resolve the
label-less handle once into a `LazyLock<metrics::Counter>`; the global recorder
is installed at startup before any response streams, so the cached handle binds
to the final recorder.
No metric names or label values change. The only behavior change is that the
`rustfs.lock.acquire.*` GET-path metrics now follow the GET stage-metrics flag,
consistent with the neighbouring stage timings.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(metrics): gate page-cache reclaim metrics behind metrics_enabled()
`record_file_cache_reclaim_success/error` run per read-stream reclaim window on
large-object reads and emitted unconditionally. When general metrics are
disabled the `counter!`/`histogram!` macros still construct three metric keys
per call for nothing. Skip the emission behind `rustfs_io_metrics::metrics_enabled()`,
matching how the io-metrics free functions self-gate. The serial reclaim-metrics
test now enables the flag (save/restore) alongside the existing stage gate.
Left ungated deliberately: `record_read_repair_dedup` (rare read-repair path,
and its non-serial test would need a global-flag toggle), and the HTTP body-chunk
counter (its cached handle already makes the disabled case a no-op increment).
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Close the three test-completeness items in rustfs/backlog#1180 that the earlier
hardening (rustfs/rustfs#4726, #4729) left unpinned; the other two of the five
(sharded cancel routing, bailout-handle error) already landed in rustfs/uring.
- rename_data end-to-end invalidation: drive the real `LocalDisk::rename_data`
commit (non-inline part, so `invalidate_part_paths` is non-empty) and assert
the destination part descriptor is dropped — not merely `rename_file`/`delete`.
Both production `rename_data` call sites share this invalidation, so a
"fix one copy, miss the other" regression is now caught. The test is
non-vacuous: it seeds the cache, removes the on-disk data dir out of band (the
cached fd keeps the old inode alive and clears the path for the directory
rename `rename_data` performs), and asserts a read still returns the OLD bytes
before the commit — which fails outright if the cache is off, so it cannot pass
without a live cache.
- FD_CACHE_TTL backstop: an injected short TTL proves the cache self-evicts a
descriptor with no explicit invalidation; a static check pins the 5s value.
- zero-length read bounds parity on the cache-HIT path: a `length == 0` read
past EOF must be rejected identically to the miss path and StdBackend, pinning
the #1173 fix against regression.
Refactors `FdCache::new` to delegate to a private `with_ttl(ttl)` helper so the
TTL backstop can be exercised with a short TTL instead of a multi-second wait.
Verified: `cargo test -p rustfs-ecstore` on Linux with real io_uring
(seccomp=unconfined, RLIMIT_NOFILE raised, RUSTFS_URING_TESTS_MUST_RUN=1 so a
degraded skip fails rather than passing vacuously).
Co-authored-by: heihutu <heihutu@gmail.com>
Two behavior-preserving cleanups to the io_uring local read backend that
accumulated as later work layered onto it:
- Reunite the `UringBackend` struct doc comment. The backlog#1145 fd-cache
constants were inserted into the middle of the struct's doc block, so its
opening sentences were orphaned onto `ENV_RUSTFS_IO_URING_FD_CACHE` and the
struct itself was documented by a sentence fragment starting mid-clause with
"through rustfs-uring's...". Move the opening back onto the struct and give
the constant its own one-line doc.
- Fold the fd-cache handle and its lookup key into a single
`Option<(&FdCache, FdKey)>` in `pread_uring`, so the get and the
insert_if_fresh sites stop re-deriving `self.fd_cache.as_ref()` and
re-matching presence. Semantics are identical, including the backlog#1176
generation guard (`gen_at_open` snapshot before open, insert only when the
generation is unchanged).
No functional change. The borrow/move pattern was validated against a
host-compilable reduction (the cfg(linux) path cannot be built from macOS);
CI compiles the Linux backend.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ecstore): close the fd-cache open-then-insert race with a generation guard (rustfs/backlog#1176)
pread_uring's miss path opened a descriptor on the blocking pool and only then
inserted it into the moka cache. moka's invalidations cover only entries present
at call time, so a heal/delete commit that invalidated between the open and the
insert could not stop the just-opened stale inode from being cached afterwards —
serving the pre-heal/pre-delete inode for up to the TTL and defeating the heal.
Add an invalidation generation to FdCache, bumped by invalidate_exact and
invalidate_under before they touch moka. The read path snapshots the generation
before opening and inserts via insert_if_fresh, which refuses the insert if the
generation moved during the open and, with a post-insert re-check, removes the
entry if an invalidation raced the insert itself. Reads that never miss are
unaffected.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): invalidate the fd cache on the primary object-delete paths (rustfs/backlog#1175)
The fd-cache invalidation contract was only wired into DiskAPI::delete,
rename_file and rename_data, but object deletion almost never goes through
LocalDisk::delete — DeleteObject(s) reach delete_version, delete_versions ->
delete_versions_internal, and delete_paths, all of which remove a version's data
dir (move_to_trash / rename_all staging) with no invalidation. A cached io_uring
descriptor kept the deleted part.N inode readable for up to the TTL, so a GET in
that window could still return deleted data.
Invalidate every cached fd under the removed data dir at each site: in
delete_version and delete_versions_internal the data_dir uuid and object path are
in hand (invalidate_cached_fds_under(volume, "{path}/{uuid}")); delete_paths
invalidates under each removed path. A later rollback that restores a data dir
just causes the next read to re-open it.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): close remaining fd-cache invalidation gaps (rustfs/backlog#1177)
Three residual paths could keep serving a stale descriptor:
- delete_volume removed the whole bucket tree (remove_dir_all/remove_dir) with no
invalidation, and the cache-hit read path skips the volume-access check, so a
cached fd kept a removed object readable. Add invalidate_cached_fds_for_volume
(a per-volume moka predicate) and call it after the bucket is removed.
- A retired LocalDisk instance (renew_disk on reconnect builds a fresh one) kept
its populated cache alive while still referenced by in-flight ops, so
invalidations through the new instance never reached it. close() now clears the
backend's cache via clear_cached_fds.
- rename_data's post-commit rollback (a commit-metadata fsync failure under
strict durability) restored the old data dir without dropping fds cached during
the committed window; the streaming branch now invalidates the dst part fds on
those rollback paths. The inline branch's rollback runs inside spawn_blocking
and is left to the TTL backstop.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): narrow the io_uring latch classes to match StdBackend (rustfs/backlog#1171)
The runtime degradation classification reused the probe-time restriction errnos,
which the driver's C7 contract explicitly warns against, so a single per-file
error could latch a whole disk off io_uring:
- is_io_uring_unsupported no longer includes EACCES: at read time on an
already-open fd it is per-file (an LSM hooks security_file_permission on every
read) and StdBackend hits the same denial, so falling back masks nothing and a
full-disk latch would be wrong. ENOSYS and EPERM (seccomp/LSM applied after
startup) remain. EOPNOTSUPP is now classified per-path by the caller.
- pread_uring_direct's read-error arm now mirrors StdBackend: an O_DIRECT-shape
error (EINVAL/EOPNOTSUPP) latches only direct_uring.supported so eligible reads
take StdBackend's aligned path, instead of over-latching the whole io_uring
backend or never latching a read-side EINVAL at all.
- try_new only negative-caches genuine restriction-class probe failures in
URING_UNSUPPORTED_DISKS; an unexpected (possibly transient) probe failure now
falls back without latching, so the next reconnect re-probes.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ecstore): log when a disk latches io_uring off at runtime (rustfs/backlog#1172)
A probe-gated gray release was flying blind: the permanent per-disk `active`
latch flipped with no log and no metric, so the only message operators ever saw
was the startup "io_uring read backend enabled" line — which stayed true on
dashboards even after the very first read latched the disk back to StdBackend
forever.
Add latch_active_off, which flips the latch with `swap` and logs the true->false
transition exactly once at warn with a dedicated event constant, disk root, and
errno. Both the buffered and O_DIRECT read paths use it. A fallback/latch metric
counter and periodic export of the driver StatsSnapshot (cq_overflow,
cancel_already) remain as follow-ups that need rustfs_io_metrics plumbing.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(audit): correct the stale rustfs-uring license-allow rationale (rustfs/backlog#1181)
The dependency-review allow said rustfs-uring is "pulled as a git dependency",
but ecstore now pins it from crates.io. Update the rationale and scope the allow
to the exact pinned version (pkg:cargo/rustfs-uring@0.1.0) so a future version
bump forces a conscious re-review of the license/provenance claim instead of
being waved through on an outdated justification.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): offload io_uring driver teardown off the tokio worker (rustfs/backlog#1170)
UringBackend held Arc<UringDriver> and had no Drop, so when the last LocalDisk
reference dropped in async context (disk reconnect via renew_disk, or shutdown),
UringDriver's own Drop ran on that thread — sending Shutdown and joining each
shard thread, which can block up to the bounded-drain timeout (5s) on a hung /
D-state disk, stalling a tokio worker.
Wrap the driver in ManuallyDrop (deref is transparent, so read call sites are
unchanged) and add a Drop that takes the Arc and, when a runtime is present,
drops it on a blocking thread so the potentially-blocking join never runs on a
runtime worker. Off-runtime it drops inline.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): restore StdBackend read parity on the uring paths (rustfs/backlog#1173)
Two byte-for-byte parity breaks against StdBackend on the io_uring read paths:
- A zero-length read on an fd-cache hit returned Ok(empty) without any bounds
check, while StdBackend and the uring miss path return FileCorrupt for an
offset past EOF. Fstat the cached descriptor on the length==0 path and match.
- reclaim_read_range fadvise(DONTNEED)'d the raw unaligned [offset, offset+len)
range, but fadvise only drops fully-covered pages, so the head partial page
stayed resident — whereas StdBackend's mmap path reclaims the page-aligned
superset. Bitrot shards' 32-byte block headers keep offsets off page
boundaries, so this diverged on the common case. Page-align the reclaim window
to match the mmap path exactly.
(The third parity item from the audit — a failed reclaim fadvise failing the
read — is already parity: StdBackend's mmap path propagates the same fadvise
error with `?`, so no change is needed.)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): bound worst-case in-flight memory by chunking huge uring reads (rustfs/backlog#1174)
The driver's backpressure permits count operations, not bytes, and it zero-fills
a full-size buffer per op, so a single unbounded read could pin ~length bytes per
permit (128 permits x shards x up to ~2 GiB). ecstore passes a whole part's shard
range as one pread_bytes with no upstream chunking.
On the buffered path, split reads larger than URING_MAX_OP_LEN (128 MiB) into
sequential chunks, awaited one at a time, so worst-case in-flight memory is
bounded by permits x URING_MAX_OP_LEN per shard. The threshold is high enough
that ordinary shard reads keep the single-op, zero-copy fast path unchanged. The
O_DIRECT path (opt-in, alignment-constrained) is left for a follow-up.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): gate the io_uring fd cache on RLIMIT_NOFILE headroom (rustfs/backlog#1178)
The fd cache holds up to FD_CACHE_CAPACITY (512) descriptors per disk, but
try_new cannot know the disk count and nothing checked the process fd budget. On
a bare-metal / non-systemd run with the common 1024 soft RLIMIT_NOFILE, two disks
would already exhaust fds with EMFILE surfacing on reads and probes.
Check the soft limit at try_new: enable the cache only with ample headroom
(>= 16384), otherwise log a warning once and fall back to open-per-read. The
packaged systemd unit sets 1,048,576, so tuned deployments are unaffected.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): make io_uring test skips visible and gate non-vacuity (rustfs/backlog#1179)
The ecstore io_uring tests degrade to a silent pass when io_uring is unavailable
(bare `return`s or plain eprintlns), so a CI leg on a restricted runner never
exercises the real UringBackend/FdCache/latch paths yet still goes green — an
integration regression could merge unseen.
Add uring_test_skip: it emits a grep-able `SKIP <name>` line and, when
RUSTFS_URING_TESTS_MUST_RUN is set (a CI leg that guarantees io_uring, e.g. a
seccomp=unconfined container), panics instead of skipping. Route the silent-skip
sites through it. Wiring a dedicated CI leg that sets that env on a capable
runner is tracked in the issue; this provides the enforcement mechanism.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): cover delete_paths fd-cache invalidation (rustfs/backlog#1180)
Add an end-to-end test that seeds the descriptor cache with a read, removes the
part via disk.delete_paths (one of the primary object-delete entry points that
does not go through LocalDisk::delete), and asserts the next read no longer
returns the removed inode — pinning the invalidation added in #1175. The sharded
cancel-routing half of #1180 is covered in the rustfs-uring PR.
Co-Authored-By: heihutu <heihutu@gmail.com>
* io_uring audit follow-ups: O_DIRECT chunking, inline invalidation, metrics, CI leg (backlog#1160) (#4729)
* fix(ecstore): chunk large O_DIRECT reads too, bounding in-flight memory (rustfs/backlog#1174)
The buffered read path already splits reads above URING_MAX_OP_LEN into
sequential chunks; do the same for the O_DIRECT path, which was left for a
follow-up. read_at_direct aligns each chunk's sub-range internally, and chunk
sizes are a multiple of URING_MAX_OP_LEN so a boundary re-read is at most one
block. Extract classify_direct_read_error so the single-op and chunked paths
share one copy of the EINVAL/EOPNOTSUPP-vs-subsystem latch classification rather
than duplicating it.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): invalidate cached fds on the inline rename_data rollback (rustfs/backlog#1177)
The streaming rename_data branch invalidates cached part fds on its post-commit
rollback paths, but the inline branch runs its commit and rollback inside a
single spawn_blocking closure where the async invalidate cannot be called, so it
was left to the TTL backstop.
Capture the closure's result instead of `??`-propagating it: on error (a
commit-metadata fsync failure under strict durability rolls the committed rename
back), invalidate the dst part paths at the async level before returning. Inline
objects keep their data in xl.meta rather than separate part inodes, so this is
largely defensive, but it removes the caveat and keeps the two branches
consistent.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ecstore): export io_uring latch/fallback and driver stats metrics (rustfs/backlog#1172)
Complete the gray-release observability. Beyond the warn log added earlier, emit
metrics so a dashboard can answer "how much traffic is on io_uring vs falling
back, and is any disk degrading":
- rustfs_io_uring_latch_off_total — a disk latching io_uring off at runtime.
- rustfs_io_uring_read_fallback_total — each io_uring -> StdBackend read
fallback (latched-off short-circuit, O_DIRECT error, buffered error).
- a low-frequency per-disk exporter of the driver StatsSnapshot as gauges
(in_flight, cq_overflow, cancel_already), spawned in try_new. It holds only a
Weak reference so it never keeps the driver alive, and drops any temporary
strong reference on the blocking pool so a last-reference UringDriver::Drop
join never runs on an async worker (rustfs/backlog#1170).
submit_errors is deliberately not exported yet: it is a field added in the
unreleased rustfs-uring 0.2.0, and ecstore still pins 0.1.0. It lands once the
dependency is bumped (rustfs/backlog#1181).
Co-Authored-By: heihutu <heihutu@gmail.com>
* ci: add a real-io_uring integration leg on ubuntu-latest (rustfs/backlog#1179)
The existing self-hosted sm-standard runners cannot guarantee io_uring is
available (a container seccomp filter can block io_uring_setup), so the ecstore
uring tests degrade to a silent skip and never exercise the real
UringBackend/FdCache/latch paths in CI.
Add a job on GitHub-hosted ubuntu-latest, which runs a recent kernel with no
container seccomp filter, running the uring-named ecstore tests with
RUSTFS_IO_URING_READ_ENABLE=true and RUSTFS_URING_TESTS_MUST_RUN=1 — the
non-vacuity gate makes the leg fail rather than skip if io_uring is unavailable,
so an integration regression can no longer merge green behind a vacuous pass.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
The rename_data crash-consistency harness (backlog#935/#878) only armed
pre-commit crash points (AfterDataRename, AfterBackupBeforeMetaCommit),
which assert the *old* version survives. The other half of the
"partial commit -> old or new, never mixed" invariant — a hard power loss
*after* the xl.meta commit rename must leave the *new* version — had no
crash-point coverage. The existing after-metadata-commit failpoint only
exercises the graceful in-process rollback (restores old), a different
path from a hard crash (no rollback runs, commit stays on disk).
Add a RenameDataCrashPoint::AfterMetaCommit injection right after the
commit rename (cfg(test), compiles to a const-false no-op in production,
like the existing points) and overwrite/fresh x strict/relaxed tests
asserting the object reads back as the new version. The fresh case pins
that the point is genuinely post-commit: a pre-commit misplacement would
leave no readable object and fail the assertion.
cargo test -p rustfs-ecstore crash_consistency: 9 passed. clippy -D
warnings clean; fmt and arch guards clean.
`pread_uring` and `pread_uring_direct` accumulated across seven rounds
(#4632/#4635/#4645/#4649/#4653/#4658/#4662) and each carried its own copy of
the same open preamble: resolve the bucket path, run the volume access check,
resolve the object path, and check the path length. The only real difference
is what happens after — one opens buffered and errors as `DiskError`, the
other opens `O_DIRECT` and errors as `DirectOpenError` so it can latch an
O_DIRECT refusal.
Extract that shared sequence into `resolve_uring_object_path`, a blocking
helper called inside both `spawn_blocking` closures. Each caller keeps exactly
what differs — its open flags, its error type, its metadata/bounds/align
handling — and no longer restates the resolution. No behavior change: the same
checks run in the same order and produce the same errors (the O_DIRECT path
still maps them through `DirectOpenError::Disk`).
Net -12 lines. Verified on a real Linux host (16-core, real io_uring):
clippy -p rustfs-ecstore --all-targets -D warnings clean; disk::local tests
144 passed, 0 failed — including the io_uring, O_DIRECT, fd-cache, and
page-cache-reclaim cases that exercise both read paths.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): roll back delete on disks that staged then errored (backlog#1158)
#4300 rolls back a failed delete only on disks that returned Ok, skipping any
disk that staged its rollback backup, applied the delete, and then errored --
leaving that disk deleted while its peers are restored. On rollback, fan the
undo out to every online disk instead; the disk-side restore_delete_rollback
is already idempotent (Ok no-op when nothing was staged), so unstaged disks
are unaffected. The err.is_some() skip now applies only to the success/cleanup
path. Covers both single-object and batch delete.
Refs backlog#1158.
fix(ecstore): io_uring reads must reclaim the page cache like StdBackend (backlog#1145)
`StdBackend::pread_bytes` calls `fadvise(DONTNEED)` over the range it just
read whenever `should_reclaim_file_cache_after_read(length)` holds —
`RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE` (on by default) above a 4 MiB
threshold. Large object reads are usually cold, and leaving them resident
evicts everything else, so this is a deliberate policy rather than a side
effect of how StdBackend happens to read.
`pread_uring` and `pread_uring_direct` never did it. Enabling io_uring on a
disk therefore turned the policy off silently: shard reads at or above the
threshold stayed resident and grew the page cache without bound. This is the
same class of mistake as the O_DIRECT loss fixed earlier — copying the read
itself while dropping the policy attached to it.
It also badly distorts any benchmark. An end-to-end warp GET A/B on a
16-core host (4 MiB objects, conc 64) showed io_uring at 8782 MiB/s against
StdBackend's 116 MiB/s. That is not a 76x speedup: with io_uring the run
issued *zero* device reads and grew the page cache, while StdBackend read
2950 MiB from the device and shrank it. The two legs were not running the
same policy. (Disabling the mmap read path changed nothing — 1.01x — so the
mmap-vs-pread difference was not the cause either.)
Both io_uring read paths now reclaim the range they read, with the same gate,
the same range, and the same error handling as StdBackend. O_DIRECT should
leave nothing resident anyway, but a filesystem that quietly buffered the read
still has to honour the policy, so `pread_uring_direct` reclaims too.
The test measures the policy, not the call: it reads an 8 MiB file through
each backend and asks `mincore(2)` how much stayed resident. Crucially it also
asserts the reclaim-disabled case leaves the range resident — without that
gate, a backend that never reclaimed would pass silently. Verified: with the
fix removed, the test fails with "uring: reclaim is on ... 2048/2048 pages
remain"; with the fix it passes.
Verified on a real Linux host (16-core, real io_uring): clippy --tests
-D warnings clean; disk::local tests 143 passed, 0 failed.
Co-authored-by: heihutu <heihutu@gmail.com>
houseme