Each erasure block runs its Reed-Solomon encode through
tokio::task::block_in_place on the multi-threaded runtime. That parks the
worker and asks the scheduler to relocate other tasks, but the encode itself
is only ~110µs per 1MiB block (p99 ~542µs) — the profiling in backlog#932
flagged the scheduling disturbance as comparable to the compute it guards.
Call the encode closure inline on the multi-threaded runtime instead. The
CurrentThread (and any other) flavor keeps spawn_blocking so the sole executor
thread is never blocked and block_in_place's multi-thread-only requirement is
respected. Applied to both ingest paths (encode_block / Vec and
encode_block_bytes_mut / BytesMut); no change to encode output, quorum,
shutdown, or error handling.
Adds encode_works_on_multi_thread_runtime to cover the previously-untested
multi-threaded arm for both ingest paths, asserting streaming and batched
encode produce identical shard bytes.
This is the low-risk, correctness-neutral item that backlog#932's adversarial
verification recommended splitting out and doing first; the larger per-writer
pipeline restructure it belongs to stays gated on a Linux multi-disk baseline.
Refs: rustfs/backlog#932 (HP-11), rustfs/backlog#936
Co-authored-by: heihutu <heihutu@gmail.com>
The rename_data commit sequence is the highest-risk durability path in the
store, and the HP-1/HP-4/HP-5 work (fsync coalescing, group commit, relaxed
durability tiers) all need a standing gate that proves a power loss mid-commit
can never leave a mixed or corrupt object. There was one graceful-rollback
failpoint but no crash-consistency coverage.
Add a deterministic harness that models a hard power loss — the commit
sequence stops dead at the armed step with no in-process cleanup — and then
reopens the disk to assert the raw on-disk state is coherent:
- Two pre-commit injection points, RenameDataCrashPoint::{AfterDataRename,
AfterBackupBeforeMetaCommit}, constructed at the real commit-path call sites
but gated so the production build compiles them to a const-false no-op
(mirrors the existing should_fail_before_old_metadata_backup pattern).
- A parameterized scenario over {strict, relaxed} durability x {both crash
points} x {overwrite, fresh object}: seed, stage a replacement, inject,
reopen, and assert the object reads back as exactly the old version (or does
not exist when there was no old version) with its data dir intact, never the
half-committed new one. The un-injected run asserts the commit makes the new
version visible. Relaxed is held to the same old-or-new invariant as strict
(only the durability window widens), which is exactly the property the
durability-relaxation work must not break (rustfs/backlog#878 hard rule).
- Wire an explicit `ecstore-crash-consistency` gate into the destructive
profile of run_ecstore_validation_suite.sh so it runs in the standing suite
(coordinates with the #878 destructive profile rather than building a
parallel one).
Production behavior is unchanged: the injection guards are no-ops outside
tests, and the existing rename_data rollback tests still pass.
Refs: rustfs/backlog#935 (HP-14 crash-consistency harness), rustfs/backlog#896
(test plan), rustfs/backlog#878 (ECStore validation suite), rustfs/backlog#936
Co-authored-by: heihutu <heihutu@gmail.com>
* perf(ecstore): read bitrot hash+data in one pass on the shard path
BitrotReader::read issued two reads per block: one read_exact for the
32-byte hash, then a separate loop for the shard data. On the streaming
disk reader (a raw tokio File whose every read is a spawn_blocking
round-trip) that is two dispatches per block. Since the on-disk layout is
a contiguous [hash][data] run, pull both in a single pass into a reused
scratch buffer and split afterwards, halving the per-block dispatch count
on the streaming path. The no-hash path still reads straight into the
caller buffer with no extra copy, and an in-memory Cursor (inline/mmap)
just does a slice copy.
All existing invariants are preserved: a short read of either the hash or
the data maps to UnexpectedEof before and independent of the hash check
(backlog#799 B2), and the hash-mismatch / InvalidData semantics are
unchanged.
Correctness-only change; the per-dispatch latency win is platform
dependent and its default reliance is left to the warp size-bucket
benchmark gate tracked by backlog#935, per the backlog#933 acceptance
note.
Refs: rustfs/backlog#933 (HP-12 item 2), rustfs/backlog#936
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs: reword bitrot test comment to satisfy typos check
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Phase 5 Slice 4 (backlog#939): move the S3 region — a write-once identity
scalar — out of the GLOBAL_REGION process static into the per-instance
InstanceContext, so two instances can serve different regions.
- InstanceContext gains `region: OnceLock<Region>` with `set_region()` /
`region()`. `set_region` keeps the write-once fail-fast contract: a second
write panics, exactly as the process global did (not downgraded to a warn).
- global.rs `set_global_region` / `get_global_region` keep their signatures and
forward to the current instance's context; the GLOBAL_REGION static is
removed. Single-instance: startup writes the bootstrap context (which the
ECStore adopts), so reads are unchanged.
Tests: set/get round-trip, two contexts hold distinct regions, and a second
set_region panics (fail-fast preserved).
Verification: cargo test -p rustfs-ecstore (9 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 4). Stacked on Slice 3 (#4417).
* fix(scanner): scope long walk timeouts
* fix(scanner): bound IAM config walks
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
fix(ecstore): stop ListMultipartUploads from returning one upload past max-uploads (backlog#954)
The result-collection loop in `SetDisks::list_multipart_uploads` pushed an
upload into the page and only then checked `ret_uploads.len() > max_uploads`,
so each page returned up to max_uploads + 1 uploads and pointed
`next_upload_id_marker` at that surplus entry, violating the S3
ListMultipartUploads max-uploads contract.
Move the cap check before the push (MinIO-style) so a page never exceeds
max_uploads, and derive `is_truncated` from the post-marker cursor position
(`upload_idx < uploads.len()`) rather than comparing the page length against
the full listing length. The old length comparison mis-reported truncation on
the final page whenever a marker had skipped earlier entries, which prevented
marker-based pagination from terminating.
Add a regression test that starts more in-progress uploads than a page holds
and asserts a single page returns exactly max_uploads with a correct
next-marker, that the exact-boundary case is not marked truncated, and that
paginating one upload at a time enumerates every upload once with no loss,
duplication, or non-termination.
Incidental: add the missing `ctx` field to the `new_multipart_lock_test_store`
cfg(test) helper so the ecstore test build compiles after the #4413/#4437
merge collision.
Refs: https://github.com/rustfs/backlog/issues/954
`create_file` writes erasure shard / multipart part data through the page
cache, so the commit-point `sync_dir_files` fdatasync in `rename_data` has to
flush the whole shard's dirty pages (profiling: create_file avg 67.52ms vs
1.51ms nosync; BitrotWriter::write p95 40.86ms), and concurrent PUTs on one
device stall each other's writeback inside the rename critical section.
Add an opt-in Linux O_DIRECT streaming writer that reuses the direct-io read
infrastructure (statx DIOALIGN probe, aligned bounce buffer, supported latch,
one-time fallback log). Incoming bytes are staged into an aligned buffer;
full aligned batches and the shutdown tail are flushed on the blocking pool
(same offloading posture as the buffered writer it replaces). The
sub-alignment tail is written buffered after clearing O_DIRECT via fcntl
(MinIO's recipe). Shard bytes reach the device during the write phase, so the
unchanged commit-point `sync_dir_files` fdatasync degrades to a cheap
metadata/device FLUSH instead of flushing ~2MiB of dirty pages.
Gated by `RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE` (default false), Linux-only,
and durability-neutral: `sync_dir_files` still fdatasyncs every shard at the
commit point, so the strict/relaxed/none durability tiers keep their exact
guarantees. Filesystems that reject O_DIRECT (tmpfs, overlayfs, 9p) latch the
path off and fall back to buffered writes; an O_DIRECT open/write EINVAL is
never surfaced as `InvalidInput` (which `to_file_error` maps to `FileNotFound`
and would masquerade as a missing shard, triggering a spurious EC rebuild).
Unit tests cover the env gate, the alignment/staging-capacity and tail-split
math, an end-to-end `create_file` round trip across sizes crossing the
alignment and multi-batch boundaries (buffered fallback on macOS / O_DIRECT on
a block device), and the writer state machine over a plain file on Linux. Real
O_DIRECT throughput on Linux NVMe (ext4/xfs) is deferred to the azure 4-node
bench per the issue's GO/NO-GO gate; the development host is macOS and cannot
exercise O_DIRECT.
Refs: rustfs/backlog#927, rustfs/backlog#936
Co-authored-by: heihutu <heihutu@gmail.com>
perf(ecstore): per-bucket durability tier overrides (HP-5 phase 2)
Let a bucket override the process-wide RUSTFS_DURABILITY_MODE with its own
strict/relaxed/none tier, stored as a durability.json extension entry in the
bucket metadata file and resolved at commit points via effective_durability.
System-critical buckets (.rustfs.sys, .minio.sys) can never carry an override
and stay pinned to strict; the legacy full-off switch keeps its historical
semantics and per-bucket overrides do not apply under it. Overrides are
published and cleared through the existing bucket metadata cache-invalidation
path, and an admin GET/PUT handler exposes the configuration. Default behavior
is unchanged: with no override a bucket follows the global mode, which defaults
to strict and stays byte-for-byte identical to before.
Refs: https://github.com/rustfs/backlog/issues/938, https://github.com/rustfs/backlog/issues/936
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(swift): replace assert!(false) with panic! in expiration_worker test
clippy::assertions_on_constants fails the swift clippy CI job under -D warnings.
* fix(ecstore): add missing ctx field to multipart lock test store
The multipart list-parts lock test (#4437) constructs ECStore without the
ctx field added by the Phase 5 InstanceContext work (backlog#939). Both
landed on main independently, leaving a semantic conflict that breaks the
ecstore lib-test build (E0063). Adopt the process bootstrap context,
matching the existing bootstrap_ctx() test in store/mod.rs.
fix(ecstore): validate erasure distribution values to avoid shuffle index panic (backlog#949)
The element values of `erasure.distribution` read from `xl.meta` were never
range-checked. `FileInfo::is_valid()` and `MetaObjectV1::valid()` only verified
`distribution.len()` and the `erasure.index` bound, not that each distribution
value is a valid 1-based slot in `[1, N]`. The metadata shuffle helpers then use
these values directly as `distribution[k] - 1` indices, so a corrupt or
adversarial `xl.meta` carrying a `0` (usize underflow) or a value greater than N
(out-of-bounds) triggers a panic in the shuffle path, turning bad-disk metadata
that erasure coding is meant to tolerate into a request/task crash.
Fix, two layers:
- Validate distribution values at metadata acceptance: `is_valid_distribution`
now requires the distribution to be a permutation of `1..=N` (correct length,
every value in range, no duplicates). `FileInfo::is_valid()` and
`MetaObjectV1::valid()` use it, so `find_file_info_in_quorum` rejects corrupt
metadata and it surfaces as a clean `ErasureReadQuorum` error instead of an
index path.
- Defensive indexing in the shuffle helpers (`shuffle_disks_and_parts_metadata`,
`_by_index`, `_by_index_owned`, `shuffle_parts_metadata`, `shuffle_disks`,
`shuffle_check_parts`): out-of-range distribution values are skipped via
`checked_sub(1)` + bounds-checked slot access instead of a bare `idx - 1`
index, matching the existing pattern in
`collect_inline_data_shard_fileinfos_by_index`.
Regression tests: `is_valid_distribution`/`is_valid`/`valid` reject distributions
containing `0`, values greater than N, duplicates, and wrong length while
accepting valid permutations; the shuffle helpers no longer panic on corrupt
distributions and preserve output length.
Refs: https://github.com/rustfs/backlog/issues/949
* refactor(ecstore): add per-instance InstanceContext, migrate erasure setup type
Phase 5 of the global-singleton consolidation (backlog#939): begin moving
runtime identity state out of process globals so multiple ECStore instances
can coexist in one process. Isolation is carried by the object graph
(ECStore -> Sets -> SetDisks holding an Arc<InstanceContext>), not a
task-local, which does not propagate across the many internal tokio::spawn
boundaries in the data/background paths.
This first slice migrates the erasure setup type -- previously three
independent process-global bools -- into a single per-instance
RwLock<SetupType> that derives is_erasure / is_dist_erasure / is_erasure_sd,
removing a triple source of truth that could drift out of sync.
- New runtime::instance module: InstanceContext + process bootstrap context.
- The legacy free-function facade (is_erasure/update_erasure_type/...) keeps
its signatures and forwards to the current instance's context, falling back
to the bootstrap context before a store is published.
- ECStore gains a pub(crate) ctx field and setup_is_* accessors; its
constructors adopt the bootstrap context (never mint a fresh one) so startup
writes and post-construction reads share one cell -- single-instance
behavior is byte-for-byte unchanged.
Tests: erasure predicate derivation vs the legacy behavior, object-graph
carrier isolation across two ECStore instances, and bootstrap adoption.
Refs: backlog#939 (Phase 5, Slice 1), backlog#653 (item 8)
* refactor(ecstore): thread InstanceContext down the object graph (Phase 5 Slice 2) (#4415)
* refactor(ecstore): source the namespace lock manager per-instance (#4417)
refactor(ecstore): source the namespace lock manager per-instance (Phase 5 Slice 3)
Phase 5 Slice 3 (backlog#939): give each instance its own lock namespace by
sourcing SetDisks' lock manager from the instance context instead of the
process singleton. This removes the false cross-instance mutual exclusion (and
attendant ABBA risk) that a shared GlobalLockManager would cause once multiple
instances coexist.
- InstanceContext gains a `lock_manager: Arc<GlobalLockManager>`. `new()` mints
a fresh manager (independent per-instance); `bootstrap_ctx()` aliases the
process singleton via get_global_lock_manager(), so a single-instance
deployment keeps exactly one shared namespace.
- SetDisks::new sources `local_lock_manager` from `ctx.lock_manager()` (the ctx
it already adopts), not `runtime_sources::global_lock_manager()`. Single
instance: same Arc as before, so behavior is unchanged.
- Remove the now-unused `runtime_sources::global_lock_manager()` wrapper.
Tests: bootstrap lock manager aliases the process singleton; two fresh contexts
own distinct managers; a SetDisks' lock manager is the one from its context and
aliases the global singleton in a single-instance build.
Verification: cargo test -p rustfs-ecstore (10 Phase 5 + set_disk locking
regressions green), cargo clippy -p rustfs-ecstore --all-targets (clean),
make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 3). Stacked on #4415 (Slice 2).
fix(lock): fence write commit on lock loss (backlog#899 Phase 2)
Phase 0+1 (#4388) made object write locks refreshable and marks the guard
lost when the heartbeat can no longer refresh a quorum, but does not act on
it. Under a partition a long write's lock can expire on an unreachable node
and be reclaimed, letting a third party re-acquire it; the original writer
keeps going and both commit -- a double write.
Expose the loss signal through NamespaceLockGuard::is_lock_lost() and
ObjectLockDiagGuard::is_lock_lost(), and fence the commit in put_object and
complete_multipart_upload: immediately before rename_data (the atomic commit
point), abort with a retryable NamespaceLockQuorumUnavailable (503) if the
lock was lost. In multipart the check precedes cleanup_multipart_path so a
lost lock leaves the upload intact and retryable. A write that already
reached rename_data Ok is durable and never aborted.
The loss criterion is unchanged (reacts to Phase 1's signal). Heal and the
long-GET read side are deferred follow-ups.
Replication deletes dropped the S3 `versionId` query parameter for every
replication request, conveying the version only via the internal
x-*-source-version-id header. A generic (non-MinIO/RustFS) S3 target ignores
that header, so a version purge degenerated into a delete-marker creation while
the source still stamped VersionPurgeStatus=Complete — a silent divergence.
Only omit the query `versionId` when propagating a delete-marker CREATION
(the target must mint its own marker); version purges, delete-marker purges and
force deletes now address the exact version. Extracted into
resolve_delete_api_version_id with unit coverage.
Fixesrustfs/backlog#857
* fix(ecstore): make post-commit old data dir cleanup best-effort (backlog#898)
A write is authoritatively committed once rename_data returns Ok (the new
version is durable on >= write_quorum disks and immediately readable). The
subsequent reclamation of the now-dereferenced old object/<data_dir> is pure
space reclamation, yet commit_rename_data_dir propagated a below-quorum GC
failure via `?` into ErasureWriteQuorum -> 503, producing a false-negative
ACK for an already-persisted write. This is a deliberate divergence from
MinIO (erasure-object.go:1577), which couples the two; the divergence is
justified by durability semantics, not parity.
Changes:
- commit_rename_data_dir now returns a structured OldDataDirCleanup receipt
and never returns Err. Adds an old==committed-dir anti-misdelete guard and
a committed_data_dir parameter. Classification is extracted into pure
functions (classify_old_data_dir_cleanup / map_cleanup_join_result /
is_cleanup_not_found) so it is unit-testable. Task panic/cancel is mapped to
a non-ignored DiskError::other (never DiskNotFound), and not-found is
normalized to reclaimed.
- object.rs / multipart.rs consume the receipt instead of `?`. The result
reverts to Ok, so the invalidate_get_object_metadata_cache self-heal and the
capacity/compression accounting that a `?` early-return previously skipped
now run on the cleanup-failure path too.
- On residue, report_old_data_dir_cleanup emits leak metrics and enqueues an
object heal over the existing heal channel (disk-health signal replacing the
503). heal_object -> reclaim_orphan_data_dirs already reclaims unreferenced
local data dirs, closing the loop end to end.
- Adds rustfs_old_data_dir_* counters (attempted/reclaimed/leaked/below_quorum)
as the operator-visible backstop for leaked residue.
- Adds a test-only (#[cfg(test)]) delete fault-injection seam; in production it
inlines to a no-op None and has no behavioral effect.
Tests: pure-function A/C group + join-error mapping + actions decision; A5/A5b
real-disk guard/reclaim integration; end-to-end overwrite returning 200 while
old-data-dir cleanup fails. #864 rollback guard test remains green.
* fix(ecstore): resolve merge conflicts with origin/main in io_primitives.rs
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Merge the hotpath-rs wall-time instrumentation from the backlog#936 analysis worktree behind an opt-in 'hotpath' cargo feature, keeping the default build at zero overhead and zero dependency.
- hotpath is an optional dependency everywhere (dep:hotpath feature syntax); the default dependency tree contains no hotpath crate at all
- 40+ measurement points across S3 handlers, ECStore/SetDisks object and multipart ops, erasure encode/decode, bitrot, LocalDisk I/O, FileMeta codec, and HashReader
- attribute sites use #[cfg_attr(feature = "hotpath", hotpath::measure)]; async_trait bodies use per-crate hp_guard! macros (ecstore + rustfs bin); rio gates measure_block! behind hp_measure_block!
- feature chain: rustfs -> rustfs-ecstore -> rustfs-rio / rustfs-filemeta, each crate owning its own gate
- hotpath-alloc is intentionally not wired up (hotpath 0.21.x TLS panic on cross-thread guard drop under tokio, see backlog#935); mimalloc stays the unconditional global allocator
- docs/development/hotpath-profiling.md documents building, HOTPATH_* env vars, SIGTERM report flow, and how to reproduce the backlog#936 timing reports
Refs: https://github.com/rustfs/backlog/issues/935 (HP-14, item 2), https://github.com/rustfs/backlog/issues/936
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(ecstore): add stripe-advance handles for deferred bitrot readers
Give DeferredObjectReader a shared pending state and expose a
DeferredReaderStripeHandle that advances the still-unopened source by whole
bitrot blocks using the same bitrot_encoded_range geometry the reader was
created with (identity mapping when hash_size == 0). This lets the GET decode
path open a parity shard aligned to the stripe where a data shard failed
instead of reading every parity shard on every stripe (backlog#923).
An already-opened (or failed) reader rejects the advance so callers retire it
rather than engage it out of alignment; bitrot verification after an advance
checks the advanced stripe's block against that stripe's stored hash.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): read only data shards on healthy lockstep GET behind opt-in gate
PR #4289's lockstep fix made every reconstruction-verifying GET read all
data+parity shards per stripe; the parity blocks are read, bitrot-hashed and
then discarded, a deterministic 2x read-bytes/IOPS/hash-CPU amplification on
healthy 2+2 objects (backlog#923). With the new opt-in gate
RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE=true (default: false, behavior
identical to main):
- read_lockstep keeps only the data slots engaged while the object is
healthy; parity slots stay unopened deferred readers.
- When a data shard is missing or dies at stripe k, parity readers are
engaged mid-object by advancing their deferred stripe handle to stripe k,
preserving the lockstep alignment invariant from backlog#832.
- Degraded stripes engage one parity beyond the decode quorum so
reconstruction verification keeps an extra source to check against
(erasure.rs only verifies when available > data shards); an engaged parity
reader that errors is retired for the rest of the object like any other,
and a parity reader that cannot be realigned is retired instead of being
read out of position.
- fill_deferred_bitrot_readers records stripe handles for deferred slots and,
gate-on only, swaps eagerly opened parity readers for unopened deferred
ones so they remain engageable mid-object; ready/error bookkeeping used by
quorum decisions is untouched.
- Both GET paths (legacy duplex via Erasure::decode_with_stripe_handles,
codec streaming via ParallelReader::with_deferred_parity_handles) carry the
handles from reader setup.
Short-read -> UnexpectedEof -> whole-object retirement and the
inconsistent-source rejection are unchanged in both gate modes; tests lock
the healthy-path data-shards-only call counts, the default read-all-shards
behavior, mid-object parity engagement for streaming and hash_size==0
formats, and mid-stream inconsistent-parity rejection.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
fix(lock): renew distributed locks via heartbeat; wire server refresh (backlog#899)
A held distributed write lock had a fixed 30s TTL and was never renewed, so
any operation exceeding it got its per-node lease reclaimed and stolen by a
contender, causing split-brain writes. This implements Phase 0 + Phase 1 of the
#899 design (Phase 2 abort-on-loss is deferred and tracked in code comments).
Phase 0 (server refresh wiring, P0):
- node_service handle_refresh was a no-op stub that parsed args then always
returned success=true. Extract a testable `refresh_lock` free function that
actually delegates to the node's lock backend. Not-found maps to
success=false with no error_info, so RemoteClient::refresh keeps its
Ok(resp.success) semantics and yields a real not_found signal to the
coordinator heartbeat. Without this, client heartbeats were silently no-ops.
Phase 1 (client heartbeat + safe interval + observability):
- DistributedLockGuard spawns a heartbeat that refreshes every per-client lease
on a derived interval; interval is derived without Duration::clamp
(entries<=1 or interval>=ttl => no spawn), fixing the sub-second-ttl panic.
- Add LockLostSignal: declare the lock lost when not_found exceeds
entries.len() - refresh_quorum; RPC errors are not counted (absorbed by the
ttl > interval margin). Expose is_lock_lost()/lock_lost() for observers.
- disarm(), release(), and Drop now abort the heartbeat before releasing so no
refresh races the unlock (refresh only extends, never creates, a lease).
- Reclaim path stays behaviorally unchanged but now warns with owner/resource/
lease age and records a metric (#698 scavenger preserved).
- Add DEFAULT_LOCK_REFRESH_INTERVAL, LockRequest.refresh_interval (serde default
for RPC back-compat) + builder, and lock lifecycle metrics.
Open questions adopt the design's documented defaults (marked TODO in code):
refresh not-found -> Ok(false)/error_info=None; lost-quorum base entries.len();
DEFAULT_LOCK_REFRESH_INTERVAL=10s.
Tests: heartbeat keepalive/quorum-loss/jitter/boundary/disarm (lock crate),
server refresh delegation (rustfs), and end-to-end survives-past-ttl plus
crashed-owner-reclaim regressions (namespace).
* perf(ecstore): move speculative tmp cleanup off the PUT hot path
On a successful PUT, rename_data has already moved the data dir out of the tmp workspace, so the delete_all(RUSTFS_META_TMP_BUCKET) at the end of SetDisks::put_object is a speculative no-op safety net. It was awaited inline on the response path, where profiling (backlog#924 / HP-3) showed the same-disk queueing behind fsync-heavy load turns a ~49us no-op into ~9ms average (p99 77ms, macOS F_FULLFSYNC amplified) added to every PUT.
Run that cleanup on a spawned task instead, keeping it as a real backstop (rename_data's remove_std only removes empty dirs and silently ignores failures). The failure path (quorum loss / rollback) keeps the cleanup inline so a failed PUT never returns with tmp shards still on disk. If the process dies before the spawned task runs, cleanup_stale_tmp_objects (24h expiry, 5-minute loop) reclaims the entry.
Scope note: ops/multipart.rs delete_all on RUSTFS_META_MULTIPART_BUCKET is intentionally untouched; it removes real leftovers and deferring it would widen CompleteMultipartUpload/Abort races.
Regression tests (hermetic SetDisks on formatted local disks, no global state): PUT success drains the tmp workspace (polling the spawned task), and PUT failure (missing bucket volume, rename_data quorum error after tmp shards were written) cleans the workspace inline before returning.
Ref: https://github.com/rustfs/backlog/issues/924
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): do not retry NotFound in reliable_rename_inner
reliable_rename_inner blindly retried the rename once on any error. A NotFound retry cannot succeed: nothing recreates the missing source or parent directory between attempts, so the second rename fails identically and speculative cleanup renames (e.g. move_to_trash on an already-removed tmp path) always paid for two syscalls.
Extract the retry decision into should_retry_rename: NotFound returns immediately, any other error keeps the existing single retry. This helper is shared by the rename_data commit path via rename_all, so behavior there is covered by a new rename_all success regression test alongside the retry-predicate tests.
Ref: https://github.com/rustfs/backlog/issues/924
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Step 1 of 4 for rustfs/backlog#922 (HP-1): write_all_internal previously
coupled file-content durability (fdatasync) with directory-entry
durability (parent dir fsync) behind a single bool. For tmp files that
are immediately renamed away, the tmp parent dir fsync contributes
nothing to crash consistency: the safe-rename recipe only needs file
content fdatasync -> rename -> fsync of the destination parent, because
the rename removes the tmp directory entry and a crash before the
rename means the PUT was never acknowledged.
Replace the bool with a SyncMode enum (None / FileAndDir / FileOnly)
and use FileOnly at exactly the two write-then-rename tmp write points:
the tmp xl.meta write in the non-inline rename_data path and the tmp
write inside write_all_meta. write_all_public (format.json etc.) and
the old-metadata rollback backup keep FileAndDir since those files stay
where they are written. The rest of the commit sequence (shard
sync_dir_files, rename, destination parent fsync) is untouched, and
behavior with drive sync disabled is unchanged.
This saves one directory fsync per disk per non-inline PUT (4 on a
4-drive set). Unit tests assert, via a test-only fsync-dir recorder,
that tmp write points no longer fsync their parent while the public
write point, the rollback backup, and the commit-rename destination
parent still do.
Co-authored-by: heihutu <heihutu@gmail.com>
Under a full nextest run on loaded machines the legitimately serialized
cross-disk commits in concurrent_resend_same_part_commits_one_generation
exceed the acquire deadline by themselves: observed at the 5s default,
at the 30s production default (#4370), and on CI even at 60s — which is
a hard ceiling, because fast_lock clamps every requested timeout to
MAX_ACQUIRE_TIMEOUT (60s) while the Timeout error still reports the
requested value, so raising the env override higher is a no-op.
Request the full 60s ceiling explicitly and cap the queue depth at three
concurrent resends, bounding the last waiter to two serialized commits
(~12s each on the slowest observed CI runner). Three resends still race
the streaming phase and contend on the commit lock, which is all the
generation-mixing regression (backlog#853) needs; every correctness
assertion is unchanged.
Co-authored-by: heihutu <heihutu@gmail.com>
MetaObject::into_fileinfo indexed part_sizes[i]/part_actual_sizes[i] by
part_numbers.len() without checking the arrays are the same length,
unlike the adjacent part_etags/part_indices which are length-guarded.
decode_from pushes the three arrays independently and the xl.meta CRC
only covers bytes, so a CRC-valid but internally inconsistent xl.meta
(foreign writer / MinIO interop) triggers an out-of-bounds panic on the
GET/HEAD/LIST decode path.
Guard the three arrays for equal length and return Err(FileCorrupt) so a
divergent shard is skipped and quorum uses the other disks, instead of
panicking the request task. Cascade into_fileinfo to Result across its
callers, and fix io_primitives early-return to derive the version id from
the merged header and fall into the per-disk loop (single-disk survival +
heal). The 2118 merge-first path is left as a documented follow-up.
Refs backlog#900 (filemeta-01).
test(interop): real-MinIO read + migration parity, Phase 1/2 (backlog#580)
Capture authentic on-disk fixtures from MinIO RELEASE.2025-07-23 (a bucket with
versioning, object-lock, lifecycle, tagging, quota, a public policy, SSE-S3
encryption, a webhook notification target, and a replication rule, plus inline /
versioned / multipart objects and a delete marker) and prove RustFS reads and
migrates them losslessly:
- filemeta parses_real_minio_object_xlmeta: small inline, two-object-version +
delete marker, and multipart object xl.meta parse to the expected FileInfo.
- ecstore parses_real_minio_bucket_metadata_blob_without_loss: the MinIO
.metadata.bin msgpack decodes via the PascalCase field names and
parse_all_configs loads all ten config types present (policy, lifecycle incl.
<ExpiryUpdatedAt>, object-lock, versioning, tagging, quota, notification,
encryption/SSE-S3, replication incl. DeleteMarkerReplication /
ExistingObjectReplication) without loss.
- ecstore reads_minio_inline_bucket_metadata_via_bitrot: MinIO inlines an object
body as [HighwayHash256 32B][body]; RustFS's BitrotReader with HighwayHash256S
verifies and yields the exact blob (the "inline_data 前缀不同" is that prefix).
- ecstore migrates_real_minio_bucket_metadata_end_to_end: on a throwaway 4-drive
local ECStore, a real MinIO .metadata.bin seeded under .minio.sys is migrated
into .rustfs.sys byte-identically for every config, exercising the Phase 2
source adapter (MIGRATING_META_BUCKET = ".minio.sys") through the object layer.
All four run as ordinary crate tests (nextest CI). Phase 4 (MinIO re-reading a
RustFS drive) is documented as out of scope for one-way migration.
Refs rustfs/backlog#580
* refactor(ecstore): extract LocalIoBackend trait behind LocalDisk
Model the LocalDisk per-file I/O hot path as a LocalIoBackend trait
(pread_bytes / open_read_stream / open_full_read / open_write) and move
the existing method bodies verbatim into a default StdBackend. LocalDisk
holds Arc<dyn LocalIoBackend> and the DiskAPI methods forward to it.
This is a behavior-preserving refactor: no logic, error-mapping, metrics,
or cfg-branch changes. It creates the seam for an alternative
runtime-probed io_uring backend (rustfs/backlog#894) without touching
DiskAPI callers. Commit-point durability (fdatasync -> rename ->
fsync-dir in rename_data) deliberately stays outside the trait.
Add a differential test asserting all four read shapes return identical
bytes across page-boundary and file-tail ranges.
Tracking: rustfs/backlog#891 (parent rustfs/backlog#897)
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ecstore): true O_DIRECT read path with per-disk graceful fallback (#4366)
* feat(ecstore): true O_DIRECT read path with per-disk graceful fallback
Implement a real O_DIRECT positioned read inside StdBackend::pread_bytes
(Linux only) and wire up the previously dead RUSTFS_OBJECT_DIRECT_IO_READ_*
knobs, which had zero call sites.
Open with rustix OFlags::DIRECT, probe the DIO alignment once per disk via
statx STATX_DIOALIGN (4096 fallback), read the aligned superset into an
alignment-allocated bounce buffer with a short-read loop, and slice out the
exact logical range so padding never reaches BitrotReader.
Any O_DIRECT failure falls back to the buffered read methods; EINVAL or
EOPNOTSUPP (tmpfs, overlayfs, 9p) latches the path off per disk with one
warning. O_DIRECT errors never surface: EINVAL maps to FileNotFound in
to_file_error and would trigger spurious EC rebuilds.
Default behavior is unchanged (knob off). macOS keeps F_NOCACHE.
Tracking: rustfs/backlog#892 (parent rustfs/backlog#897)
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): add P1.5 benchmark gate harness for O_DIRECT reads (#4369)
* test(ecstore): add P1.5 benchmark gate harness for O_DIRECT reads
Add an ignored, Linux-only release-mode test (direct_read_bench_gate)
that measures DiskAPI::read_file_mmap_copy through a real LocalDisk with
cold-cache enforcement (fadvise DONTNEED between rounds), reporting
p50/p95/p99/mean latency, wall time, process CPU time, and throughput as
one JSON line for A/B diffing between the buffered baseline and the
O_DIRECT candidate selected by the production env knobs.
Content is verified byte-for-byte before any timing starts.
Tracking: rustfs/backlog#893 (parent rustfs/backlog#897)
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): add concurrency dimension to P1.5 bench harness
Model the EC GET shape (FuturesUnordered over concurrent shard reads)
via RUSTFS_BENCH_CONCURRENCY (default 1, sequential as before). Needed
to evaluate blocking-pool pressure for the io_uring gate decision.
Tracking: rustfs/backlog#893
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* ci: retrigger after cancelled required check
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): use is_multiple_of in O_DIRECT bench cache-drop check
clippy's manual_is_multiple_of (rust 1.96) fails -D warnings on the benchmark helper's `done % file_count == 0`.
Verification:
- cargo clippy -p rustfs-ecstore --all-targets
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
A node could show OFFLINE in the console with zero logs explaining why
(rustfs/backlog#888, reported in rustfs#4304): the consecutive-failure
threshold crossing in handle_server_info_failure marked the peer offline
silently, the per-call warnings only existed on the observing node and
never named the transition, and the PeerRestClient offline flag plus its
background recovery monitor ran without any start/success log.
Logging-only change, no behavior change:
- handle_server_info_failure / handle_storage_info_failure: WARN
event="peer_marked_offline" exactly once at the threshold crossing
(later failures while already offline stay DEBUG), and DEBUG
event="peer_probe_failure" while returning cached state below the
threshold.
- update_server_info_cache / update_storage_info_cache: INFO
event="peer_recovered_online" when a probe succeeds after the peer had
been reported offline.
- PeerRestClient::mark_offline_and_spawn_recovery: WARN
event="peer_connection_marked_offline" when the offline flag is first
set (guarded by the recovery_running CAS so repeated failures do not
spam), and INFO event="peer_connection_recovered" with the attempt
count when connectivity is restored.
An "offline then back" episode now leaves a complete, correlatable
trace: N probe failures -> peer_marked_offline -> recovery monitor ->
peer_recovered_online.
Verification:
- cargo test -p rustfs-ecstore --lib (notification + peer_rest suites)
- make pre-commit
Ref: rustfs/backlog#888, rustfs#4304
Co-authored-by: heihutu <heihutu@gmail.com>
concurrent_resend_same_part_commits_one_generation keeps failing on CI
with Lock(Timeout ... 5s) even after the fast-lock lost-wakeup fix
(NOTIFY_WAIT_CAP re-polling) landed — observed on rustfs#4365's Test and
Lint runs. Two independent causes exist and both are real:
1. the lost/stolen wakeup stall — fixed in fast_lock::shard; and
2. the six legitimately serialized cross-disk commits exceeding the
small 5s default acquire timeout under full-suite CI disk load.
The lost-wakeup fix reverted the earlier 30s test override on the
assumption that cause 1 was the whole story; CI shows cause 2 stands on
its own. Restore the temp_env override (production-default 30s) around
the concurrent section so the regression guard asserts the correctness
property (exactly one intact generation), not CI disk latency. The
NOTIFY_WAIT_CAP fix stays untouched.
Verification:
- cargo test -p rustfs-ecstore --lib concurrent_resend_same_part_commits_one_generation
Ref: rustfs/backlog#882
Co-authored-by: heihutu <heihutu@gmail.com>
Drop the [target.'cfg(target_os = "linux")'.dependencies] tokio
"io-uring" feature from 6 crates and the rustfs binary. Because
.cargo/config.toml enables --cfg tokio_unstable globally, this feature
switched every Linux build's file I/O onto tokio's io_uring runtime
backend. Restricted Linux environments (Docker default seccomp, gVisor,
proot, old kernels) reject io_uring_setup with EACCES/ENOSYS, which
tokio surfaced as PermissionDenied and RustFS reported as
DiskAccessDenied at startup.
Add scripts/check_no_tokio_io_uring.sh so the feature cannot silently
return: it fails on any tokio dependency line enabling "io-uring", while
still allowing a future application-level io-uring crate dependency.
Wire it into make pre-commit/pre-pr/dev-check and CI.
Tracking: rustfs/backlog#890 (parent rustfs/backlog#897)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(migration): decrypt MinIO IAM & server config on drop-in migration
MinIO encrypts IAM identity/service-account files and the server config at
rest with a key derived from the root credentials. The drop-in migration
paths read those blobs from the legacy `.minio.sys` bucket and parsed them
as plaintext JSON, so any encrypted blob failed to parse and was silently
skipped with "incompatible format". This is why users migrating from MinIO
kept their buckets/objects/policies but lost users and access keys (#2212).
The IAM load path already knows how to decrypt these blobs (RustFS master
keys plus MinIO-compatible legacy keys derived from the root credentials),
but that logic lived behind a private method and was never used by the
migration paths. Expose it as `rustfs_iam::try_decrypt_iam_blob` and inject
it into both migration paths via a `LegacyBlobDecryptFn` callback (ecstore
cannot depend on the IAM crate, so the closure is wired in the binary crate).
When a blob cannot be decrypted the raw bytes are used as-is, preserving the
previous plaintext-only behavior with no regression.
Also improve object-layer migration observability without changing control
flow: `try_migrate_format` now distinguishes "no legacy format" (a normal
fresh install) from "legacy format present but incompatible", and the caller
logs a loud error before initializing a fresh format that would leave the
existing MinIO objects unreadable. Topology/version skip reasons are promoted
from debug to warn.
Fixes a pre-existing test isolation race by marking
`test_recovery_falls_back_to_default_config_when_blob_stays_corrupt` serial,
since it reads a process-wide env var toggled by a sibling test.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(migration): box FormatV3 in LegacyFormatOutcome to satisfy clippy
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): stabilize concurrent multipart resend lock timeout
concurrent_resend_same_part_commits_one_generation spawns 6 same-part
resends whose cross-disk commits serialize on the per-uploadId commit
lock. Under the full nextest suite the parallel disk load pushes those
serialized commits past the small default lock-acquire timeout (5s),
producing a spurious `Lock(Timeout ...)` unrelated to the property under
test (observed on CI at 5.775s vs ~0.5s in isolation).
Raise RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT to the production default (30s)
for the concurrent-commit section via temp_env, so the regression guard
reflects correctness (exactly one intact generation) rather than disk
latency under CI load. The meaningful assertions are unchanged, and
#[serial] keeps the process-wide env override isolated.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(lock): bound fast-lock notification wait to prevent lost-wakeup stall
The real cause of the concurrent_resend_same_part_commits_one_generation
failures was a lost wakeup in the fast-lock slow path, not disk latency:
raising the acquire timeout to 30s only delayed the failure (it then timed
out at 30s), proving a genuine stall rather than overload.
In acquire_lock_slow_path a waiter that reaches the notification phase did a
single `timeout(remaining, wait_for_write())` spanning the whole acquire
budget, and treated that wait's elapse as a hard `Timeout`. But the release
path only notifies when `writer_waiters > 0`, so if the holder releases in
the gap after the waiter's `try_acquire` fails and before it registers as a
waiter, no notification (and no stored permit, since the pooled `Notify` is
gated) is produced. The waiter then blocks until the deadline even though the
lock is free and stays free — a spurious lock-acquire timeout. The shared
process-wide notify pool makes it worse: a wakeup can be consumed by a waiter
of a different lock hashing to the same slot.
Bound each notification wait (NOTIFY_WAIT_CAP = 50ms) and, on elapse, loop
back and re-`try_acquire` instead of returning `Timeout`; the deadline check
at the top of the loop is the single source of truth for timing out. A
lost/stolen wakeup now degrades to bounded re-polling (acquire within ~50ms
of the lock becoming free) instead of stalling for the whole timeout.
Correctness (mutual exclusion) is unchanged — acquisition still only happens
via `try_acquire_*`.
Add a regression test that reproduces the stall (holder + late waiter across
many keys): it times out without the fix and passes in ~1s with it. Revert
the earlier acquire-timeout workaround in the multipart test now that the
underlying stall is fixed, so it runs under the default timeout again.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(server): make startup readiness wait configurable and raise default (#4264)
The startup runtime-readiness wait was a hardcoded 30s constant with no env
override. On slow multi-node cold starts (Docker/K8s/Synology NAS) this window
is shorter than the internal startup budgets it depends on — the endpoint
DNS-retry window (~90s) and the format-load retry loop (~100s worst case) — so
readiness times out and the node exits with
`startup readiness timed out after 30s: storage_ready=false, lock_quorum_ready=false`
before storage/lock quorum can converge, feeding the restart storm reported in
the issue.
- Add `RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS` (default 120s), documented in
rustfs-config health constants.
- Resolve the wait at runtime via `startup_runtime_readiness_max_wait()`; a
value of `0` falls back to the default instead of timing out instantly.
- Repoint `STARTUP_RUNTIME_READINESS_MAX_WAIT` at the shared config default so
there is a single source of truth, and cover the getter with unit tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): stop peer/disk background monitors on graceful shutdown (#4264)
Long-lived peer health/recovery and remote-disk monitors are detached
`tokio::spawn` tasks that each hold a `tracing::Span` via `.instrument(..)` for
their whole lifetime. Nothing cancelled them at shutdown, so on the normal
return path the Tokio runtime was dropped while they were still alive and their
`Span`s were dropped during worker-thread thread-local-storage (TLS)
destruction. At that point `tracing-subscriber`'s fmt `on_close` can touch an
already-destroyed TLS slot and panic with
`cannot access a Thread Local Storage value during or after destruction`, which
escalates to a panic-during-panic abort (SIGILL / exit 132) — the crash
reported on Synology in issue #4264, amplified by the restart storm.
- Add `cluster::rpc::background_monitor` with a process-global shutdown token,
`spawn_background_monitor()` (races the monitor future against that token so
its span drops while the runtime is alive), and public
`shutdown_background_monitors()`.
- Route every span-holding peer_s3 / peer_rest / remote_disk monitor spawn
through `spawn_background_monitor` instead of `tokio::spawn(..).instrument()`.
- Expose `rustfs_ecstore::shutdown_background_monitors()` and call it from the
graceful shutdown sequence (right after `ctx.cancel()`, before runtime
teardown) via the `storage_api` compatibility boundary.
Existing recovery-probe span-context tests still pass, confirming log
correlation is preserved.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* perf(ecstore): replace linear merge scan with k-way heap and drop clone-to-parse (backlog#874 backlog#875)
merge_entry_channels advanced the k-way merge with a linear scan over all
channel heads (O(entries x channels)) and allocated two fresh Strings per
pairwise comparison via path::clean. Every step also cloned MetaCacheEntry
values, including entry.clone().xl_meta() clone-to-parse calls.
- Introduce MergeHead with a cached cleaned name (allocated only when the
raw name is not already clean) and drive the merge with a BinaryHeap of
boxed heads: O(log channels) per entry, allocation-free comparisons.
- Move entries through the merge instead of cloning; the winner is sent
without an intermediate copy.
- Remove the dead merge_file_meta_versions block: it only ran for
prefix-dir groups whose entries have empty metadata, so xl_meta() always
failed; cross-drive version merging happens in the resolve path.
- Keep legacy same-name semantics (dir groups collapse, objects shadow
prefix dirs, later object candidate wins) and add regression tests for
interleaved ordering, dir/object precedence, uncleaned-name grouping,
and prefix-dir collapse.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): honor ascending versions_sort in ListObjects walk (#4348)
* fix(ecstore): honor ascending versions_sort in walk and document ordering invariant (backlog#876)
The walk loop carried a bare `//TODO: SORT` inside the
`WalkVersionsSortOrder::Ascending` branch, so the requested ascending
order was silently ignored and versions streamed newest-first (the raw
FileMeta order). WalkOptions defaults to Ascending, so every default
walker -- notably replication resync, which replays versions and needs
oldest-first to preserve the version-stack order -- received the exact
opposite of the contract.
FileMeta maintains versions newest-first (sort_by_mod_time is
descending) and into_file_info_versions preserves that order, so
ascending emission is the exact reverse of file_info_versions output.
Reverse in place when ascending is requested and add a regression test
locking the newest-first invariant plus the reversal contract.
Key-ordering audit result (no gap found): per-disk walkers emit sorted
streams, merge_entry_channels performs an ordered k-way merge, and
gather_results only filters by marker/limit, so ListObjects key order is
guaranteed upstream and needs no post-sort.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): enable GET metadata early-stop by default (#4349)
* perf(ecstore): enable GET metadata early-stop by default with env opt-out (backlog#872)
The metadata early-stop fanout (read_all_fileinfo_early_stop) has been
implemented and instrumented for a while but stayed behind an opt-in
flag, so default GETs always waited for every disk to answer the
metadata read even after quorum agreement was reached.
Flip RUSTFS_GET_METADATA_EARLY_STOP_ENABLE to default-on. The gate stays
conservative: should_allow_metadata_early_stop only admits metadata-only
reads (read_data=false) without version_id, healing, or free-version
requirements, everything else falls back to the full-wait fanout, and
setting the env var to false restores the old behavior entirely. The
version-aware gate (RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE)
remains opt-in because versioned reads carry a higher stale-selection
risk profile.
Also replace the stale "optimize concurrency" TODO in
get_object_fileinfo with a pointer to the early-stop implementation and
add regression tests for the new default plus the explicit opt-out path.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): lazily construct codec streaming multipart readers (#4350)
* perf(ecstore): lazily construct codec streaming multipart part readers (backlog#871)
get_object_decode_reader_with_fileinfo opened shard readers for every
part of a multipart object before returning the streaming reader, so
TTFB paid for parts x disks file opens up front and an early client
disconnect wasted the setup work for every unread part.
Replace the eager loop with LazyMultipartCodecStreamingReader: the first
part is still built eagerly so the dominant fallback conditions (missing
shards / read quorum) are detected before any byte is streamed and the
whole request can fall back to the legacy duplex path exactly as before.
Each subsequent part is built on demand -- when the previous part hits
EOF -- via a spawned task handle owned by the reader; dropping the
reader aborts an in-flight build so disconnects stop all further IO.
If a later part hits a fallback condition mid-stream (a shard vanished
after the request started), the reader surfaces an explicit read error
with a pipeline-failure metric instead of silently degrading; the
client's retry then detects the condition on the eager first-part setup
and takes the legacy path cleanly.
Adds unit tests for in-order streaming across lazy boundaries, deferred
construction (no build when the client stops within part 1), and the
mid-stream fallback error path.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): prefetch next multipart part reader setup during decode (#4351)
* perf(ecstore): prefetch next multipart part reader setup during decode (backlog#870)
get_object_with_fileinfo processed multipart parts strictly serially:
the next part's bitrot reader setup (file opens + read-quorum wait
across all disks) only started after the current part finished
decoding, so large multipart reads paid full setup latency between
every part.
Overlap the two stages with a depth-one pipeline: right after the
current part's readers are obtained, the next part's setup is spawned
(shared inputs behind Arc) and joined when the loop reaches that part.
The shared setup_multipart_part_readers helper keeps stage-duration
metrics semantics identical for both paths; a failed or stale prefetch
falls back to the synchronous setup, and the PrefetchedReaderSetup
guard aborts the in-flight task on error returns, early breaks, or
caller drop so disconnects stop background disk IO.
Gate: RUSTFS_GET_MULTIPART_READER_SETUP_PREFETCH (default on, env
opt-out). Adds a three-part end-to-end read test covering the prefetch
hit path and cross-part content ordering.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): move FileInfo through GET shuffle instead of cloning (#4352)
perf(ecstore): move FileInfo entries through the GET shuffle instead of cloning (backlog#873)
shuffle_disks_and_parts_metadata_by_index deep-cloned every valid
FileInfo (parts, erasure info, metadata map) once per disk on each GET.
Add an ownership-taking variant that runs the same by-index consistency
check as a read-only first pass and then moves entries into their
shuffled slots with mem::take, and switch get_object_with_fileinfo to
it -- that call site already owned the parts metadata vector. Disk
handles are Arc clones and stay cheap.
Scope notes from the backlog#873 audit:
- get_object_fileinfo's disks.clone() stays: DiskStore is Arc<Disk>, so
the clone is per-slot refcounting and correctly avoids holding the
RwLock read guard across the metadata fanout awaits.
- get_object_decode_reader_with_fileinfo keeps the borrowing shuffle:
its caller must retain files/disks for the legacy fallback path, so an
owned variant would just shift the same clone upstream.
- The metadata-cache hit path still clones parts_metadata; sharing the
cached entry via Arc changes the read-path return types and is left
as a follow-up.
Equivalence tests cover both the by-index placement and the mod-time
fallback against the borrowing variant.
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ecstore): gate merge emission on cleaned key and clear clippy redundant_clone
Address review + CI findings on the ListObjects/GET optimization PR:
- merge_entry_channels gated emission on the raw entry name while the heap
orders by the cleaned key, so entries whose cleaned order and raw byte order
disagree (e.g. redundant slashes) could be dropped. Gate on the same cleaned
sort key the heap uses; add a regression test (`a//c` after `a/b`).
- Drop three redundant `.clone()` calls in test code flagged by
clippy::redundant_clone (owned-shuffle equivalence tests and the walk
ascending-versions contract test) that failed the CI clippy gate.
- Document the known mid-stream fallback limitation of the opt-in multipart
codec streaming reader (default off) and mark the in-place per-part legacy
degradation as a follow-up.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): force full metadata fanout for object tagging writes (backlog#872)
put_object_tags reads the object fileinfo with read_data=false and then
writes the updated tags to the online-disk set that read returned. With
metadata early-stop enabled by default, that read now returns as soon as
read quorum is reached, so the online-disk set is only a read-quorum
subset. Writing tags to that subset fails write quorum -> ErasureWriteQuorum
-> S3 SlowDown, which is exactly the s3-tests tagging failures
(PutObjectTagging/DeleteObjectTagging, reached max retries).
Thread a caller-controlled `allow_early_stop` gate through
read_all_fileinfo_observed/_inner and add get_object_fileinfo_gated;
put_object_tags calls it with allow_early_stop=false so the metadata read
does the full quorum fanout and returns the complete online-disk set as
the write target. Pure-read callers (GET/HEAD/tag read) keep the
early-stop fast path unchanged.
Extract metadata_early_stop_permitted() as the single gate and add a unit
test locking the invariant: caller opt-out (and observe=false, and data
reads) never early-stop even with the env flags on.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(ecstore): reclaim orphaned data dirs during heal (backlog #3231/#3191)
Pre-#3510, an unversioned overwrite leaked one UUID-named data dir per PUT.
The write path now cleans up going forward, but pre-existing strays stay on
disk forever: heal's dangling logic only removes whole objects whose data is
missing, never surplus data dirs of an otherwise-healthy object. That leaked
space is what eventually made ListObjects scan tens of GB and time out.
Add SetDisks::reclaim_orphan_data_dirs, a fail-closed, quorum-safe sweep that
mirrors purge_orphan_dir_object:
- referenced set is the UNION of get_data_dirs() across every online replica's
xl.meta, so a dir named by any replica is kept;
- if a disk holds the object dir but its xl.meta is missing or unparseable the
object is treated as degraded and nothing is removed;
- only UUID-named subdirectories are ever considered.
Wire it into heal_object's healthy tail (under the object write lock) as a
best-effort step so the background heal scanner and admin heal recover the
leaked space automatically; a reclaim failure never fails the heal.
Covered by four unit tests: removal of an unreferenced dir, no-op when all dirs
are referenced, fail-closed abort on missing metadata, and cross-replica union
preservation.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(ecstore): fix typo unparseable -> unparsable
Satisfies the repository typos CI check.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Zhengchao An