* chore(deps): remove redundant dependency features
Remove manifest feature entries that are implied by other requested features in the same dependency declaration.
Verified that the resolved Cargo feature graph is unchanged after the cleanup.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(deps): narrow tokio and reqwest features
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Move workspace-level dependency feature lists into the member crates that consume each dependency while keeping required default-features flags at the workspace root.
Also refresh starshard to 2.2.2 via cargo update and cargo upgrade --exclude ratelimit.
Co-authored-by: heihutu <heihutu@gmail.com>
* test(ecstore): extract shared MockWarmBackend into a test-util feature (backlog#1148 ilm-6)
The tier/lifecycle integration tests carried two byte-for-byte copies of an
in-memory WarmBackend mock — one in crates/scanner/tests and one in
rustfs/src/app — plus duplicated register_mock_tier and polling helpers. Both
implemented the same ecstore WarmBackend trait.
Consolidate them into ecstore behind a new `test-util` feature, exposed via the
`rustfs_ecstore::api::tier::test_util` facade:
- MockWarmBackend: in-memory WarmBackend with an operation log (for ordering
assertions such as "local delete precedes remote remove") and fault injection
(FaultConfig): unreachable, HTTP 5xx, credential rejection, injected latency,
plus external_remove to simulate an out-of-band remote deletion.
- register_mock_tier / register_mock_tier_backend: register the mock into any
TierConfigMgr handle (the global manager used by scanner tests or a
per-instance one used by the app tests).
- xl.meta transition assertion helpers: read_transition_meta,
assert_transition_meta_consistent (cross-shard consistency of the
status/tier/remote-key/remote-version-id tuple plus free-version count), and
free_version_count.
- polling helpers: wait_for_remote_absence, wait_for_object_count,
wait_for_free_version_absence.
Both existing copies now consume this single definition; `rg 'struct
MockWarmBackend'` collapses to one. The feature is enabled only from
[dev-dependencies], so it never links into the production binary (resolver 3).
Designed for downstream ilm-8 (restore lifecycle) and ilm-11 (tier fault
injection matrix). Coordinates with #4706 (ilm-2), which adds op-logging to the
scanner mock — that op-logging is now part of this shared surface, so #4706
should rebase onto it.
Refs rustfs/backlog#1148 (ilm-6), rustfs/backlog#1155.
* test(ecstore): fix shared MockWarmBackend usage after main merge
- Access stored objects via MockWarmBackend::contains() instead of the now
private inner objects map (fixes E0609 after the shared test-util refactor).
- Drop dead ReadCloser/ReaderImpl/DiskAPI imports and the unused
transition_api test re-exports the mock extraction left behind.
- Reword the scanner/rustfs test-util dependency comments so they no longer
embed the literal rustfs_ecstore:: path that trips the ECStore
architecture-migration guard.
chore(deps): bump rustfs-uring to 0.2.1 from crates.io
rustfs-uring 0.2.1 routes the driver's runtime diagnostics through `tracing`
with structured fields instead of `eprintln!` (rustfs/uring#13). No public API
change — UringDriver::probe_and_start_sharded, read_at, and read_at_direct keep
their signatures — and the read path / cancel-safety ownership model are
untouched, so this is a drop-in patch bump of the version requirement plus the
lockfile entry.
0.2.1 adds a `tracing` dependency; it is recorded in the rustfs-uring lock
entry. tracing is already in the workspace graph, so nothing new is pulled in.
The Cargo.lock change is scoped to the rustfs-uring package only; unrelated
lockfile drift is left for its own change.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ecstore): close the fd-cache open-then-insert race with a generation guard (rustfs/backlog#1176)
pread_uring's miss path opened a descriptor on the blocking pool and only then
inserted it into the moka cache. moka's invalidations cover only entries present
at call time, so a heal/delete commit that invalidated between the open and the
insert could not stop the just-opened stale inode from being cached afterwards —
serving the pre-heal/pre-delete inode for up to the TTL and defeating the heal.
Add an invalidation generation to FdCache, bumped by invalidate_exact and
invalidate_under before they touch moka. The read path snapshots the generation
before opening and inserts via insert_if_fresh, which refuses the insert if the
generation moved during the open and, with a post-insert re-check, removes the
entry if an invalidation raced the insert itself. Reads that never miss are
unaffected.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): invalidate the fd cache on the primary object-delete paths (rustfs/backlog#1175)
The fd-cache invalidation contract was only wired into DiskAPI::delete,
rename_file and rename_data, but object deletion almost never goes through
LocalDisk::delete — DeleteObject(s) reach delete_version, delete_versions ->
delete_versions_internal, and delete_paths, all of which remove a version's data
dir (move_to_trash / rename_all staging) with no invalidation. A cached io_uring
descriptor kept the deleted part.N inode readable for up to the TTL, so a GET in
that window could still return deleted data.
Invalidate every cached fd under the removed data dir at each site: in
delete_version and delete_versions_internal the data_dir uuid and object path are
in hand (invalidate_cached_fds_under(volume, "{path}/{uuid}")); delete_paths
invalidates under each removed path. A later rollback that restores a data dir
just causes the next read to re-open it.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): close remaining fd-cache invalidation gaps (rustfs/backlog#1177)
Three residual paths could keep serving a stale descriptor:
- delete_volume removed the whole bucket tree (remove_dir_all/remove_dir) with no
invalidation, and the cache-hit read path skips the volume-access check, so a
cached fd kept a removed object readable. Add invalidate_cached_fds_for_volume
(a per-volume moka predicate) and call it after the bucket is removed.
- A retired LocalDisk instance (renew_disk on reconnect builds a fresh one) kept
its populated cache alive while still referenced by in-flight ops, so
invalidations through the new instance never reached it. close() now clears the
backend's cache via clear_cached_fds.
- rename_data's post-commit rollback (a commit-metadata fsync failure under
strict durability) restored the old data dir without dropping fds cached during
the committed window; the streaming branch now invalidates the dst part fds on
those rollback paths. The inline branch's rollback runs inside spawn_blocking
and is left to the TTL backstop.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): narrow the io_uring latch classes to match StdBackend (rustfs/backlog#1171)
The runtime degradation classification reused the probe-time restriction errnos,
which the driver's C7 contract explicitly warns against, so a single per-file
error could latch a whole disk off io_uring:
- is_io_uring_unsupported no longer includes EACCES: at read time on an
already-open fd it is per-file (an LSM hooks security_file_permission on every
read) and StdBackend hits the same denial, so falling back masks nothing and a
full-disk latch would be wrong. ENOSYS and EPERM (seccomp/LSM applied after
startup) remain. EOPNOTSUPP is now classified per-path by the caller.
- pread_uring_direct's read-error arm now mirrors StdBackend: an O_DIRECT-shape
error (EINVAL/EOPNOTSUPP) latches only direct_uring.supported so eligible reads
take StdBackend's aligned path, instead of over-latching the whole io_uring
backend or never latching a read-side EINVAL at all.
- try_new only negative-caches genuine restriction-class probe failures in
URING_UNSUPPORTED_DISKS; an unexpected (possibly transient) probe failure now
falls back without latching, so the next reconnect re-probes.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ecstore): log when a disk latches io_uring off at runtime (rustfs/backlog#1172)
A probe-gated gray release was flying blind: the permanent per-disk `active`
latch flipped with no log and no metric, so the only message operators ever saw
was the startup "io_uring read backend enabled" line — which stayed true on
dashboards even after the very first read latched the disk back to StdBackend
forever.
Add latch_active_off, which flips the latch with `swap` and logs the true->false
transition exactly once at warn with a dedicated event constant, disk root, and
errno. Both the buffered and O_DIRECT read paths use it. A fallback/latch metric
counter and periodic export of the driver StatsSnapshot (cq_overflow,
cancel_already) remain as follow-ups that need rustfs_io_metrics plumbing.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(audit): correct the stale rustfs-uring license-allow rationale (rustfs/backlog#1181)
The dependency-review allow said rustfs-uring is "pulled as a git dependency",
but ecstore now pins it from crates.io. Update the rationale and scope the allow
to the exact pinned version (pkg:cargo/rustfs-uring@0.1.0) so a future version
bump forces a conscious re-review of the license/provenance claim instead of
being waved through on an outdated justification.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): offload io_uring driver teardown off the tokio worker (rustfs/backlog#1170)
UringBackend held Arc<UringDriver> and had no Drop, so when the last LocalDisk
reference dropped in async context (disk reconnect via renew_disk, or shutdown),
UringDriver's own Drop ran on that thread — sending Shutdown and joining each
shard thread, which can block up to the bounded-drain timeout (5s) on a hung /
D-state disk, stalling a tokio worker.
Wrap the driver in ManuallyDrop (deref is transparent, so read call sites are
unchanged) and add a Drop that takes the Arc and, when a runtime is present,
drops it on a blocking thread so the potentially-blocking join never runs on a
runtime worker. Off-runtime it drops inline.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): restore StdBackend read parity on the uring paths (rustfs/backlog#1173)
Two byte-for-byte parity breaks against StdBackend on the io_uring read paths:
- A zero-length read on an fd-cache hit returned Ok(empty) without any bounds
check, while StdBackend and the uring miss path return FileCorrupt for an
offset past EOF. Fstat the cached descriptor on the length==0 path and match.
- reclaim_read_range fadvise(DONTNEED)'d the raw unaligned [offset, offset+len)
range, but fadvise only drops fully-covered pages, so the head partial page
stayed resident — whereas StdBackend's mmap path reclaims the page-aligned
superset. Bitrot shards' 32-byte block headers keep offsets off page
boundaries, so this diverged on the common case. Page-align the reclaim window
to match the mmap path exactly.
(The third parity item from the audit — a failed reclaim fadvise failing the
read — is already parity: StdBackend's mmap path propagates the same fadvise
error with `?`, so no change is needed.)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): bound worst-case in-flight memory by chunking huge uring reads (rustfs/backlog#1174)
The driver's backpressure permits count operations, not bytes, and it zero-fills
a full-size buffer per op, so a single unbounded read could pin ~length bytes per
permit (128 permits x shards x up to ~2 GiB). ecstore passes a whole part's shard
range as one pread_bytes with no upstream chunking.
On the buffered path, split reads larger than URING_MAX_OP_LEN (128 MiB) into
sequential chunks, awaited one at a time, so worst-case in-flight memory is
bounded by permits x URING_MAX_OP_LEN per shard. The threshold is high enough
that ordinary shard reads keep the single-op, zero-copy fast path unchanged. The
O_DIRECT path (opt-in, alignment-constrained) is left for a follow-up.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): gate the io_uring fd cache on RLIMIT_NOFILE headroom (rustfs/backlog#1178)
The fd cache holds up to FD_CACHE_CAPACITY (512) descriptors per disk, but
try_new cannot know the disk count and nothing checked the process fd budget. On
a bare-metal / non-systemd run with the common 1024 soft RLIMIT_NOFILE, two disks
would already exhaust fds with EMFILE surfacing on reads and probes.
Check the soft limit at try_new: enable the cache only with ample headroom
(>= 16384), otherwise log a warning once and fall back to open-per-read. The
packaged systemd unit sets 1,048,576, so tuned deployments are unaffected.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): make io_uring test skips visible and gate non-vacuity (rustfs/backlog#1179)
The ecstore io_uring tests degrade to a silent pass when io_uring is unavailable
(bare `return`s or plain eprintlns), so a CI leg on a restricted runner never
exercises the real UringBackend/FdCache/latch paths yet still goes green — an
integration regression could merge unseen.
Add uring_test_skip: it emits a grep-able `SKIP <name>` line and, when
RUSTFS_URING_TESTS_MUST_RUN is set (a CI leg that guarantees io_uring, e.g. a
seccomp=unconfined container), panics instead of skipping. Route the silent-skip
sites through it. Wiring a dedicated CI leg that sets that env on a capable
runner is tracked in the issue; this provides the enforcement mechanism.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): cover delete_paths fd-cache invalidation (rustfs/backlog#1180)
Add an end-to-end test that seeds the descriptor cache with a read, removes the
part via disk.delete_paths (one of the primary object-delete entry points that
does not go through LocalDisk::delete), and asserts the next read no longer
returns the removed inode — pinning the invalidation added in #1175. The sharded
cancel-routing half of #1180 is covered in the rustfs-uring PR.
Co-Authored-By: heihutu <heihutu@gmail.com>
* io_uring audit follow-ups: O_DIRECT chunking, inline invalidation, metrics, CI leg (backlog#1160) (#4729)
* fix(ecstore): chunk large O_DIRECT reads too, bounding in-flight memory (rustfs/backlog#1174)
The buffered read path already splits reads above URING_MAX_OP_LEN into
sequential chunks; do the same for the O_DIRECT path, which was left for a
follow-up. read_at_direct aligns each chunk's sub-range internally, and chunk
sizes are a multiple of URING_MAX_OP_LEN so a boundary re-read is at most one
block. Extract classify_direct_read_error so the single-op and chunked paths
share one copy of the EINVAL/EOPNOTSUPP-vs-subsystem latch classification rather
than duplicating it.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): invalidate cached fds on the inline rename_data rollback (rustfs/backlog#1177)
The streaming rename_data branch invalidates cached part fds on its post-commit
rollback paths, but the inline branch runs its commit and rollback inside a
single spawn_blocking closure where the async invalidate cannot be called, so it
was left to the TTL backstop.
Capture the closure's result instead of `??`-propagating it: on error (a
commit-metadata fsync failure under strict durability rolls the committed rename
back), invalidate the dst part paths at the async level before returning. Inline
objects keep their data in xl.meta rather than separate part inodes, so this is
largely defensive, but it removes the caveat and keeps the two branches
consistent.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ecstore): export io_uring latch/fallback and driver stats metrics (rustfs/backlog#1172)
Complete the gray-release observability. Beyond the warn log added earlier, emit
metrics so a dashboard can answer "how much traffic is on io_uring vs falling
back, and is any disk degrading":
- rustfs_io_uring_latch_off_total — a disk latching io_uring off at runtime.
- rustfs_io_uring_read_fallback_total — each io_uring -> StdBackend read
fallback (latched-off short-circuit, O_DIRECT error, buffered error).
- a low-frequency per-disk exporter of the driver StatsSnapshot as gauges
(in_flight, cq_overflow, cancel_already), spawned in try_new. It holds only a
Weak reference so it never keeps the driver alive, and drops any temporary
strong reference on the blocking pool so a last-reference UringDriver::Drop
join never runs on an async worker (rustfs/backlog#1170).
submit_errors is deliberately not exported yet: it is a field added in the
unreleased rustfs-uring 0.2.0, and ecstore still pins 0.1.0. It lands once the
dependency is bumped (rustfs/backlog#1181).
Co-Authored-By: heihutu <heihutu@gmail.com>
* ci: add a real-io_uring integration leg on ubuntu-latest (rustfs/backlog#1179)
The existing self-hosted sm-standard runners cannot guarantee io_uring is
available (a container seccomp filter can block io_uring_setup), so the ecstore
uring tests degrade to a silent skip and never exercise the real
UringBackend/FdCache/latch paths in CI.
Add a job on GitHub-hosted ubuntu-latest, which runs a recent kernel with no
container seccomp filter, running the uring-named ecstore tests with
RUSTFS_IO_URING_READ_ENABLE=true and RUSTFS_URING_TESTS_MUST_RUN=1 — the
non-vacuity gate makes the leg fail rather than skip if io_uring is unavailable,
so an integration regression can no longer merge green behind a vacuous pass.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
chore(deps): bump rustfs-uring to 0.2.0 from crates.io
rustfs-uring 0.2.0 is published on crates.io. It carries the read-driver
hardening from the backlog#1160 adversarial audit (cancel-safety and
graceful-degradation fixes on the Linux io_uring read path). The public
API ecstore consumes is unchanged — UringDriver::probe_and_start_sharded,
read_at, read_at_direct all keep their 0.1.0 signatures — so this is a
drop-in bump of the version requirement plus the lockfile entry.
The Cargo.lock change is intentionally scoped to the rustfs-uring package
only; unrelated lockfile drift is left for its own change.
Co-authored-by: heihutu <heihutu@gmail.com>
* chore(ecstore): switch rustfs-uring from the git rev to crates.io 0.1.0
Prepared ahead of the rustfs-uring 0.1.0 crates.io release (rustfs/uring):
flip ecstore's dependency from the pinned git revision to the published
`0.1.0`, and drop the now-unneeded git-source allowance in deny.toml.
DO NOT MERGE until rustfs-uring 0.1.0 is on crates.io. Until then cargo
cannot resolve `rustfs-uring = "0.1.0"`, so this branch will not build and
CI will be red — that is expected, not a defect in the change.
After the crate is published:
1. `cargo update -p rustfs-uring --precise 0.1.0` to regenerate Cargo.lock
(deliberately not touched here — the registry entry, with its checksum,
cannot be written before the crate exists);
2. push the Cargo.lock;
3. CI goes green; merge.
No code change. The guard `scripts/check_no_tokio_io_uring.sh` is unaffected
(it bans only tokio's io-uring runtime feature, not an explicit rustfs-uring
dependency). `audit.yml`'s `allow-dependencies-licenses` for rustfs-uring is
left in place — it is a first-party Apache-2.0 crate either way.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore: refresh rustfs-uring lockfile
---------
Co-authored-by: heihutu <heihutu@gmail.com>
A buffered read that hits the page cache completes inline inside
`io_uring_enter`, so the thread driving a ring performs that read's
memcpy. One ring per disk therefore capped cache-hit reads at a single
core's memory bandwidth: measured on a 16-core host, one driver thread sat
pinned at 100% CPU while throughput stayed flat at ~5 GB/s regardless of
read size, against 50 GB/s for the blocking-pool baseline.
rustfs/uring#6 taught the driver to hold N independent rings, each with its
own thread, pending table, backpressure semaphore, and eventfd. Wire it up:
`UringBackend::try_new` now calls `probe_and_start_sharded`, and
`RUSTFS_IO_URING_SHARDS` selects the count per disk.
The default is a quarter of the available parallelism clamped to `1..=4`,
because the cost is `disks × shards` driver threads (each normally blocked
in `poll(2)`). Any override is clamped to `1..=16`, so a mistyped value can
neither disable the driver (0) nor spawn threads without bound; an
unparseable value falls back to the default.
Effect (warm page cache, 16-core, rustfs/uring's concurrent_pread_bench):
1 MiB, conc 8: 1 shard 4911 MB/s -> 8 shards 47361 MB/s (9.6x);
the blocking-pool baseline is 50662 MB/s
64 KiB, conc 32: StdBackend 153678 IOPS, p999 3030 us
8 shards 345402 IOPS, p999 897 us
64 KiB, conc 128: StdBackend 135155 IOPS, p999 10716 us
8 shards 389047 IOPS, p999 4092 us
Sharding removes the throughput deficit *and* keeps io_uring's tail-latency
advantage, rather than trading one for the other.
Unchanged: io_uring read stays gray-off by default
(`RUSTFS_IO_URING_READ_ENABLE`), reads are byte-for-byte identical to
StdBackend, the per-disk degradation latches and probe cache (backlog#1101)
and the O_DIRECT tiered fallback (backlog#1102) all still apply. Rings stay
per-disk, so a stalled disk cannot starve another disk's rings
(backlog#1055). Bumps the rustfs-uring pin to the merged #6 commit.
Verified on a real Linux host (16-core, real io_uring): cargo clippy
--tests -D warnings clean; disk::local tests 132 passed, 0 failed —
including the existing io_uring and O_DIRECT cases now running on the
sharded driver, plus a new test covering the shard-count default, override,
and clamping.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): keep O_DIRECT for eligible reads via native io_uring (rustfs/backlog#1102)
Wire ecstore's io_uring read backend to rustfs-uring's native aligned
O_DIRECT read (`read_at_direct`, merged in rustfs/uring#3), so an
O_DIRECT-eligible read keeps BOTH io_uring's async submission AND
O_DIRECT's page-cache bypass instead of trading one for the other.
`UringBackend::pread_bytes` now selects the read shape by a tiered,
per-disk-latched ladder — never a blanket downgrade:
1. io_uring latched off for this disk (backlog#1101) -> StdBackend.
2. O_DIRECT-eligible + native path usable -> pread_uring_direct: open
the file O_DIRECT, probe the device alignment once (cached), and
read via read_at_direct. Best path: async + no cache pollution.
3. O_DIRECT-eligible but the fs refused O_DIRECT earlier (latched)
-> StdBackend aligned path (which itself degrades to buffered).
4. non-O_DIRECT read -> buffered pread_uring.
Latching keeps a failure from being re-attempted per-read: an fs that
refuses O_DIRECT (EINVAL/EOPNOTSUPP on open) latches the native direct
path off for that disk; a restriction-class errno from the read latches
io_uring off wholesale (backlog#1101). Any other error falls back for
that one read without masking a genuine data problem as a permanent
downgrade. The alignment comes from the existing probe_direct_io_align
(statx STATX_DIOALIGN, default 4096) and is cached in a per-disk
OnceLock so the probe runs at most once.
This replaces the interim guard that routed every O_DIRECT-eligible read
to StdBackend (which disabled io_uring on the hottest shard reads); the
native path is now the default and StdBackend is only a fallback tier.
Bumps the rustfs-uring pin to the merged #3 commit.
Verified on a real Linux host (16-core, real io_uring + O_DIRECT):
cargo check --tests, clippy -D warnings, and disk::local tests all pass
(128 passed, 0 failed), including uring_preserves_o_direct_for_eligible_reads
across unaligned ranges.
Co-authored-by: heihutu <heihutu@gmail.com>
feat(ecstore): add runtime-probed io_uring read backend, gray-off by default (rustfs/backlog#1104)
Wire the standalone rustfs-uring crate (https://github.com/rustfs/uring) into
LocalIoBackend as an opt-in read backend. When RUSTFS_IO_URING_READ_ENABLE is
set AND the per-disk io_uring probe succeeds, positioned reads go through the
cancel-safe UringDriver; otherwise — default, or on a restricted host, or on
any per-read driver error — reads fall back to StdBackend byte-for-byte, so the
default build is unchanged.
- Cargo.toml: rustfs-uring as a Linux-only git dependency (pinned rev). The
guard scripts/check_no_tokio_io_uring.sh allows an explicit io-uring
integration; only the tokio "io-uring" runtime feature is banned.
- UringBackend mirrors StdBackend::pread_bytes's resolution/access/bounds
preamble and only swaps the raw byte read; the other three trait methods
delegate to the inner StdBackend.
- Backend selection at LocalDisk construction via build_local_io_backend.
- Differential test uring_backend_reads_match_std: with the flag set, every
positioned read returns byte-identical data (driver-served when io_uring is
available, StdBackend fallback otherwise).
Verified: cargo check -p rustfs-ecstore on Linux; the differential test passes
under real io_uring (seccomp=unconfined). The runtime errno degradation latch,
per-disk probe cache, and productionization are tracked in
rustfs/backlog#1101/#1102.
Co-authored-by: heihutu <heihutu@gmail.com>
Merge the hotpath-rs wall-time instrumentation from the backlog#936 analysis worktree behind an opt-in 'hotpath' cargo feature, keeping the default build at zero overhead and zero dependency.
- hotpath is an optional dependency everywhere (dep:hotpath feature syntax); the default dependency tree contains no hotpath crate at all
- 40+ measurement points across S3 handlers, ECStore/SetDisks object and multipart ops, erasure encode/decode, bitrot, LocalDisk I/O, FileMeta codec, and HashReader
- attribute sites use #[cfg_attr(feature = "hotpath", hotpath::measure)]; async_trait bodies use per-crate hp_guard! macros (ecstore + rustfs bin); rio gates measure_block! behind hp_measure_block!
- feature chain: rustfs -> rustfs-ecstore -> rustfs-rio / rustfs-filemeta, each crate owning its own gate
- hotpath-alloc is intentionally not wired up (hotpath 0.21.x TLS panic on cross-thread guard drop under tokio, see backlog#935); mimalloc stays the unconditional global allocator
- docs/development/hotpath-profiling.md documents building, HOTPATH_* env vars, SIGTERM report flow, and how to reproduce the backlog#936 timing reports
Refs: https://github.com/rustfs/backlog/issues/935 (HP-14, item 2), https://github.com/rustfs/backlog/issues/936
Co-authored-by: heihutu <heihutu@gmail.com>
Drop the [target.'cfg(target_os = "linux")'.dependencies] tokio
"io-uring" feature from 6 crates and the rustfs binary. Because
.cargo/config.toml enables --cfg tokio_unstable globally, this feature
switched every Linux build's file I/O onto tokio's io_uring runtime
backend. Restricted Linux environments (Docker default seccomp, gVisor,
proot, old kernels) reject io_uring_setup with EACCES/ENOSYS, which
tokio surfaced as PermissionDenied and RustFS reported as
DiskAccessDenied at startup.
Add scripts/check_no_tokio_io_uring.sh so the feature cannot silently
return: it fails on any tokio dependency line enabling "io-uring", while
still allowing a future application-level io-uring crate dependency.
Wire it into make pre-commit/pre-pr/dev-check and CI.
Tracking: rustfs/backlog#890 (parent rustfs/backlog#897)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(migration): decrypt MinIO IAM & server config on drop-in migration
MinIO encrypts IAM identity/service-account files and the server config at
rest with a key derived from the root credentials. The drop-in migration
paths read those blobs from the legacy `.minio.sys` bucket and parsed them
as plaintext JSON, so any encrypted blob failed to parse and was silently
skipped with "incompatible format". This is why users migrating from MinIO
kept their buckets/objects/policies but lost users and access keys (#2212).
The IAM load path already knows how to decrypt these blobs (RustFS master
keys plus MinIO-compatible legacy keys derived from the root credentials),
but that logic lived behind a private method and was never used by the
migration paths. Expose it as `rustfs_iam::try_decrypt_iam_blob` and inject
it into both migration paths via a `LegacyBlobDecryptFn` callback (ecstore
cannot depend on the IAM crate, so the closure is wired in the binary crate).
When a blob cannot be decrypted the raw bytes are used as-is, preserving the
previous plaintext-only behavior with no regression.
Also improve object-layer migration observability without changing control
flow: `try_migrate_format` now distinguishes "no legacy format" (a normal
fresh install) from "legacy format present but incompatible", and the caller
logs a loud error before initializing a fresh format that would leave the
existing MinIO objects unreadable. Topology/version skip reasons are promoted
from debug to warn.
Fixes a pre-existing test isolation race by marking
`test_recovery_falls_back_to_default_config_when_blob_stays_corrupt` serial,
since it reads a process-wide env var toggled by a sibling test.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(migration): box FormatV3 in LegacyFormatOutcome to satisfy clippy
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): stabilize concurrent multipart resend lock timeout
concurrent_resend_same_part_commits_one_generation spawns 6 same-part
resends whose cross-disk commits serialize on the per-uploadId commit
lock. Under the full nextest suite the parallel disk load pushes those
serialized commits past the small default lock-acquire timeout (5s),
producing a spurious `Lock(Timeout ...)` unrelated to the property under
test (observed on CI at 5.775s vs ~0.5s in isolation).
Raise RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT to the production default (30s)
for the concurrent-commit section via temp_env, so the regression guard
reflects correctness (exactly one intact generation) rather than disk
latency under CI load. The meaningful assertions are unchanged, and
#[serial] keeps the process-wide env override isolated.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(lock): bound fast-lock notification wait to prevent lost-wakeup stall
The real cause of the concurrent_resend_same_part_commits_one_generation
failures was a lost wakeup in the fast-lock slow path, not disk latency:
raising the acquire timeout to 30s only delayed the failure (it then timed
out at 30s), proving a genuine stall rather than overload.
In acquire_lock_slow_path a waiter that reaches the notification phase did a
single `timeout(remaining, wait_for_write())` spanning the whole acquire
budget, and treated that wait's elapse as a hard `Timeout`. But the release
path only notifies when `writer_waiters > 0`, so if the holder releases in
the gap after the waiter's `try_acquire` fails and before it registers as a
waiter, no notification (and no stored permit, since the pooled `Notify` is
gated) is produced. The waiter then blocks until the deadline even though the
lock is free and stays free — a spurious lock-acquire timeout. The shared
process-wide notify pool makes it worse: a wakeup can be consumed by a waiter
of a different lock hashing to the same slot.
Bound each notification wait (NOTIFY_WAIT_CAP = 50ms) and, on elapse, loop
back and re-`try_acquire` instead of returning `Timeout`; the deadline check
at the top of the loop is the single source of truth for timing out. A
lost/stolen wakeup now degrades to bounded re-polling (acquire within ~50ms
of the lock becoming free) instead of stalling for the whole timeout.
Correctness (mutual exclusion) is unchanged — acquisition still only happens
via `try_acquire_*`.
Add a regression test that reproduces the stall (holder + late waiter across
many keys): it times out without the fix and passes in ~1s with it. Revert
the earlier acquire-timeout workaround in the multipart test now that the
underlying stall is fixed, so it runs under the default timeout again.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
test: add insta snapshot test for storage error display format
Add snapshot test to detect unexpected changes in StorageError
display format. This catches output format regressions that
traditional assert tests might miss.
Refs #740
fix(ecstore): replace unbounded metadata cache with moka
Replace the manual Arc<RwLock<HashMap>> metadata cache with
moka::future::Cache, which provides:
- Built-in LRU eviction when max_capacity is reached
- Automatic TTL expiry via time_to_live (250ms)
- Lock-free concurrent reads
- Non-blocking invalidation
Fixes the memory leak risk from unbounded HashMap and the
all-or-nothing eviction logic that cleared all entries at once.
Closes#743
Co-authored-by: houseme <housemecn@gmail.com>
* feat(replication): support insecure https bucket targets
* feat(replication): support custom ca bucket targets
* test(replication): satisfy e2e clippy for TLS helpers
* fix(replication): avoid native root panic for custom trust stores
* test(replication): decouple private IP target test from TLS roots
* test(replication): use target TLS client in private IP unit test
The session watchdog now selects its detection method per
platform. It previously probed kernel TCP state through a
Linux-only procfs path, so on macOS every healthy idle session
was killed within a minute, and on Windows the watchdog never
spawned at all, leaving wedged sessions with no cleanup. Linux
keeps its fast-kill watchdog unchanged. Other platforms get a
silence-only backstop that kills a session only at the
documented 30-minute idle ceiling.
The host-key loader now has a Windows arm. It loads OpenSSH
format host keys from the configured directory and logs a
one-time warning to restrict NTFS ACLs on the key directory,
the same operator-managed approach FTPS, WebDAV, KMS, and IAM
already use on Windows. Startup previously aborted with
UnsupportedPlatform because the Unix mode-bit permission check
has no Windows equivalent. tokio's io-uring feature is now
enabled only in Linux builds. io-uring is a Linux kernel
interface and enabling it unconditionally broke the Windows
build.
Co-authored-by: houseme <housemecn@gmail.com>
* chore(deps): refresh workspace deps and linux fs_type gating
- refresh workspace dependency pins and lockfile updates
- remove now-unused crate dependency entries in multiple Cargo.toml files
- enable profiling export defaults in config and scripts/run.sh
- gate os::fs_type module/function/tests to Linux to avoid non-Linux dead_code warnings
* fix(utils): simplify fs_type linux gating
- keep fs_type module-level linux cfg in os::mod
- remove redundant linux cfg on get_fs_type and test module
* chore(deps): bump s3s git revision
- update workspace s3s dependency to rev 507e1312b211c3ddc214b03875d6fabd15d22ed5
- refresh Cargo.lock source entry for s3s
* chore(dev): allow mysql_async git source and env overrides
- allow mysql_async git source in deny.toml allow-git list
- make scripts/run.sh core env vars overrideable via existing shell env
* fix(utils): import get_fs_type in fs_type tests
- add explicit super::get_fs_type import in fs_type test module
- fix Linux E0425 unresolved function errors in unit tests
* chore(dev): tune run script observability defaults
- make profiling export env overrideable in scripts/run.sh
- set RUSTFS_OBS_SAMPLE_RATIO default from 2.0 to 1.0
- update allow-git review window comments in deny.toml
* test(obs): stabilize profiling env alias tests
houseme