Workers::give() decrements in_use to 0 before calling semaphore.add_permits(1), while a queued Workers::take() task only increments in_use after its woken future is scheduled and polled, allowing Workers::wait() to return prematurely while a woken worker is still in flight.
Root Cause:
In crates/concurrency/src/workers.rs, Workers::give() decrements in_use inside self.in_use.send_modify() before calling self.semaphore.add_permits(1), while Workers::take() only increments in_use after self.semaphore.acquire().await returns and the woken task is scheduled and polled. During the handoff window when the last active worker calls give() and transfers its semaphore permit to a queued take() waiter (so semaphore.available_permits() < limit), in_use is 0, causing Workers::wait() (rx.wait_for(|&held| held == 0)) to complete prematurely while work is still in flight and Workers::available() to over-report available capacity.
Fix:
Require both held == 0 and self.semaphore.available_permits() == self.limit in Workers::wait(), notify self.in_use watchers after self.semaphore.add_permits(1) in Workers::give() and via an AcquireCancelGuard in Workers::take(), and bound Workers::available() by self.semaphore.available_permits().
Signed-off-by: czcbangkai <czcbangkai@google.com>
Co-authored-by: Hauser <housemecn@gmail.com>
* perf(ecstore): unify per-disk blocking workers with spin keep-alive and coalesce DELETE hops
Unify per-disk blocking I/O and fsync workers under a 3-lane DiskBlockingQueue
(Foreground, Isolated, Fsync) with MAX_NAMESPACE_WORKERS_PER_DISK = 1 and a
bounded user-space spin/yield keep-alive (WORKER_IDLE_SPIN_ITERS = 256), and
coalesce redundant blocking dispatches along the DELETE hot path:
- Expand DiskBlockingQueue to 3 lanes (Foreground, Isolated, Fsync) with 1
warm worker per disk/lane and a 256-iteration spin/yield keep-alive before
exiting spawn_blocking, preventing Tokio BlockingPool shared.idle futex
wake/wait thrashing between back-to-back stages of PUT and DELETE.
- Route fsync_dir, fsync_dir_with_owner, and namespace file-sync barriers
through the per-disk Fsync lane on FSYNC_RUNTIME, and replace LocalDisk's
duplicate MetadataReadQueue with LocalDisk::run_metadata_read backed by
self.blocking_queue.
- Route write_all_internal(Owned), open_file(O_TRUNC), remove_file_with_owner,
and remove_dir_with_owner through DiskBlockingQueue with pre-execution
cancellation checks.
- Coalesce DELETE directory scans in commit_reserved_version_delete and
finish_version_delete into single blocking passes with periodic cancellation
polling, batch rollback file writes in write_delete_rollback_files, batch
rollback unlinks + directory fsync in remove_files_and_fsync_with_owner, and
coalesce post-rename directory fsyncs in activate_delete_preparation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
* perf(ecstore): isolate durable work and reduce idle queue locking
Route coalesced writes and delete settlement through the fsync lane while preserving namespace leases, cancellation ownership, and durability ordering. Keep relaxed work on the foreground lane and replace idle mutex polling with an advisory atomic hint; dequeue and worker retirement remain mutex-protected.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): yield disk workers across shared blocking pools
Limit each draining task to 64 completed jobs and reuse the worker guard to reschedule remaining work behind other disks. Preserve per-disk FIFO, worker limits, and per-operation ownership. Add a deterministic two-disk regression with one blocking thread.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): release queued jobs when workers cannot start
Construct the worker guard before submission and track whether execution began. Release pending jobs outside the lane mutex when the last worker is rejected, so waiters, resource units, and self-owning namespace leases cannot remain stranded. Cover closed runtimes and shutdown during batch handoff.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(connect): synchronize restart with observed inventory
Wait for the received POST instead of treating transport backoff as proof of delivery. Inject a pre-HTTP connection drop and use a verified loopback IP fixture without changing timeout or replay assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(ecstore): unify disk worker lifecycle and drain loop
Represent submitted, running, and retired workers explicitly. Share dequeue, execution, idle recheck, and retirement in one loop while preserving bounded handoff and rejected-task ownership cleanup.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(storage): separate real I/O from paused-clock assertions
Complete durable fixtures before pausing virtual time, drive the immediate scanner tick before freezing time during filesystem waits, and keep checkpoint-backed retry accounting on a real clock. Preserve default fsync isolation, disk deadlines, retry counts, and all outcome assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
A delete-all lifecycle expiry is a prefix delete that takes the bucket
lifecycle write lock, while the rebalance entry holds that lock for read
through the inline apply and its bucket incarnation fence. The write
waited out the lock timeout, stalled every lifecycle read in the bucket,
and failed the entry on every retry.
Like decommission (#8777), rebalance now moves such a version live and
lifecycle expires it on the target.
Refs rustfs/backlog#2873
The listing pipeline signalled "this page exhausted the namespace" by
setting `MetaCacheEntriesSortedResult.err = Some(Error::Unexpected)` and
had every consumer filter that variant, so any future `Unexpected` on the
producer path would have been read as a complete listing. A second truth
source, the shared `producer_limit_reached` atomic, was last-writer-wins
across the disks of a page and was never serialized to remote peers.
Collectors now return `ListPage { entries, exhausted }`, errors surface as
errors, and the early empty-listing exits return an exhausted empty page.
The producer's stop reason travels in the metacache stream itself:
`MetacacheWriter::close_with_limit_reached` appends it after the `false`
terminator as repeated terminator markers (the only trailing bytes a
deployed reader tolerates, because `list_path_raw_inner` keeps polling
readers after EOF), `MetacacheReader::limit_reached` decodes it, and
`list_path_raw_inner` folds every reader's trailer into a
`ListPathRawOutcome` that reaches the collector. Remote disks therefore
report their per-disk limit stops as well, and the `producer_limit_reached`
fields on `WalkDirOptions`, `ListPathRawOptions` and `ListPathOptions` are
removed.
* test(e2e): tighten scoped multipart fault acceptance
Target Abort root deletion instead of child transaction cleanup, require
actual metadata peer hits, and keep faulted APIs single-attempt. Validate
bounded ordinary recovery and safe quorum refusal for partial mutations,
with the recovery enhancement tracked in backlog#2940.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(e2e): drive concurrent TLS reads independently
Give each coordinator its own supervised task and cooperative polling
budget while retaining the existing workload, deadline and byte/hash
assertions. Preserve node logs after fixture teardown and report the
operation, response phase and received bytes when a transfer times out.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(listing): honor persisted erasure quorum
Separate metadata admission and primary-reader sizing from each version's
persisted erasure requirements. Propagate agreed-entry uncertainty and
report LIST read-quorum failures as storage unavailability.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(listing): borrow metadata during quorum resolution
Keep validated metadata inspection and agreed-version selection in
filemeta. Resolve independent reader votes through borrowed slots and
retain owned buffers until a filtered result is required.
Preserve fixed-baseline legacy comparisons, cached latest fast paths,
quorum policy, and disk fan-out. Add ownership regressions and paired
release benchmarks for cold and cached version histories.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): remove redundant fallback clone
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Run two cargo update and cargo upgrade rounds from origin/main.
Update mimalloc to 0.7.2, hotpath to 0.29.1, and ZIP to 9.1.0,
with compatible crypto, compression, and smallvec updates.
Retain the generic-array version required by crypto-common 0.1.7.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scripts): collect scanner host telemetry on macOS
Use platform-supported iostat arguments and expose optional collector
failures without interrupting scanner-status collection.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): reuse validated metadata read path buffers
Share the checked bucket/object tuple across metadata read consumers and borrow clean normalization inputs without skipping filesystem validation. Keep Windows reconstruction and the second bucket check intact.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scripts): finalize scanner telemetry for failed runs
Initialize host telemetry artifacts before external captures so reused output directories cannot retain prior successes. Share normal and error-path finalization, reap collectors on request failure, and preserve the primary exit status even if status recording fails.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): distinguish unreadable recovery inventories
Treat only the exact system-prefix read-quorum result as incomplete during bounded polling. Preserve strict final reads and the shutdown/drain assertions, record failed observations, and cover interrupted empty streaks and unrelated errors.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(rio): ZigZag-encode signed delta varints and widen scratch buffer in compress_index
Index::append_to passes signed delta residuals (c_off and u_off, which are negative whenever a compressed block is smaller than est_block_uncomp / 2) to write_varint(buf, v: i64), which checks while v >= 0x80 without ZigZag encoding and truncates every negative delta to a single byte, while tmp = [0u8; 8] panics out-of-bounds on 9-10 byte varints.
Root Cause:
In crates/rio/src/compress_index.rs, Index::append_to delta-encodes u_off and c_off (c_off -= prev.compressed_offset + c_predict, where c_predict starts at est_block_uncomp / 2) as signed i64 residuals and passes them to write_varint(buf, v: i64). However, write_varint checks while v >= 0x80 on the signed i64 without ZigZag encoding (unlike Go binary.PutVarint / binary.Varint in the S2 index format): for any negative v, v >= 0x80 is false, so write_varint writes a single truncated byte v as u8 without continuation bits, either silently corrupting compressed_offset on Index::load (when (v as u8) < 0x80) or desynchronizing the varint stream (when (v as u8) >= 0x80). Additionally, tmp = [0u8; 8] at line 223 panics with an out-of-bounds index when encoding 9-10 byte varints, and read_varint lacks an overflow check on 10-byte inputs.
Fix:
Use ZigZag encoding and decoding in write_varint and read_varint, reject 64-bit varint overflows in read_varint, and widen the scratch buffer tmp in Index::append_to from [0u8; 8] to [0u8; 10].
Signed-off-by: czcbangkai <czcbangkai@google.com>
* test(rio): use est_block_uncomp field and compare IndexInfo fields in compress_index test
---------
Signed-off-by: czcbangkai <czcbangkai@google.com>
`LocalDisk::walk_dir` stored its own `limit_reached` into the
`producer_limit_reached` flag that every producer of a page shares, so a
walk reaching EOF after a sibling stopped at its per-disk limit cleared the
signal and the collector reported a complete page. Use `fetch_or` so the
flag answers "did any bounded walk stop early", whatever the finish order.
`MetaCacheEntry::is_in_dir` stripped the directory prefix repeatedly, so
`a/a/b` counted as a direct child of `a/`. `MetaCacheEntriesSorted::
forward_past` kept every entry when none sorted after the marker and split
on an index computed over the flattened entries. Both are masked today by
the collector's pre-filtering; fix them and pin the contracts with tests.
* fix(multipart): allow abort after partial commit
Read staging metadata at read quorum during Abort while retaining the
delete write quorum and generation checks. Bound post-fault lock recovery,
isolate quorum rows from peer breaker history, and cover the metadata and
deletion boundaries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(multipart): release topology guards before metadata I/O
Read multipart metadata from owned disk snapshots so slow I/O does not
block registry renewal. Cover read and write lookups with a deterministic
paused-I/O regression.
Reuse one cluster per quorum matrix after observing fresh responses from
every peer. Preserve fault-hit and quorum assertions without repeatedly
starting clusters or disabling peer request breakers.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): avoid head-of-line blocking in delete recovery
Recover independent keys through a bounded rolling window while preserving per-key hint generations, admission budgets, cancellation retry, and the existing transaction decision engine.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): bound durable delete preparation and bootstrap recovery
Prepare independent Linux delete transactions with bounded concurrency while retaining per-key result order and serial fallback for overlapping, rollback, and legacy requests. Preserve startup discovery until bucket metadata is initialized, and add release performance and process-crash acceptance runners.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): verify clean exits in recovery acceptance
Reject forced or OOM exits when measuring clean startup and verify original ETags across mixed-layout crash recovery and a second restart.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): distinguish prepared and decided batch crashes
Observe real preparation and receipt states separately, retain non-vacuous acknowledged-delete sentinels, and require read-only physical convergence before S3 reads can trigger cleanup.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): move final delete recovery options
Move the options value into the batch-ordering regression's final case
instead of cloning it. Keep the earlier clone needed for reuse.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): replay delete receipts after interrupted proof GC
Retire only the exact empty transaction leaf before journal activation. Preserve occupied directories and prefix children, and cover partial cross-disk proof GC without a fresh delete. Tighten crash evidence to distinguish directory-only remnants from retained content.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bound implicit startup delay and publish config early
Limit warm-cache startup to the smaller of the cycle interval and one
minute before jitter, while preserving explicit startup settings.
Publish configuration gauges when applying runtime settings and retain
timeout diagnostics until an actual new cycle begins.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): unify startup waits and skip unchanged configuration
Bound read-only startup inspection and make startup and pre-leadership
retries respond to cancellation and configuration changes. Preserve one
startup deadline and discover replication behind lifecycle-only buckets.
Avoid irrelevant bucket listings and repeated per-disk reconfiguration.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): retain effective startup policy and fallback metrics
Keep the sampled startup deadline when configuration changes leave its
normalized policy unchanged. Initialize retained runtime metrics after
an invalid first refresh without suppressing the validation error or
replacing a newer accepted configuration.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
get_online_offline_disks_stats() counts and demotes all root_disk entries regardless of drive state, underflowing online_disks[ep] (0usize - 1) and double-counting offline_disks[ep] when a root_disk drive is offline or unknown alongside healthy non-root drives.
Root Cause:
In crates/ecstore/src/diagnostics/admin_server_info.rs: (1) root_disk_count counts every drive with root_disk = true even if the drive was already classified into offline_disks or unknown_disks; (2) the subsequent demotion loop unconditionally decrements online_disks[ep] and increments offline_disks[ep] for every disk.root_disk without checking whether disk.state was classified as online (ok or unformatted). When an endpoint has an offline or unknown root_disk drive and zero online drives, *online_disks.get_mut(ep) -= 1 underflows 0usize (panicking in debug/checked builds or wrapping to usize::MAX) and inflates offline_disks.
Fix:
Introduce is_online_drive_state() and only count and demote root_disk drives whose state was classified as online (ok or unformatted), preserving the offline and unknown bucket tallies and keeping online + offline + unknown equal to disks_info.len().
Signed-off-by: czcbangkai <czcbangkai@google.com>
* perf(rpc): remove redundant boxing from read streams
Keep the capped read stream concrete until StreamingBlob owns its type erasure. Add verified four-node TLS coverage for full, unaligned range and concurrent reads, with bounded client memory and explicit integrity checks.
Document protocol comparison commands and the old-node workaround during rolling upgrades.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): preserve TLS trust for alternate credentials
Derive credential-specific clients from the cluster configuration so they retain the generated CA. Exercise successful trusted reads and reject an invalid replacement identity instead of inheriting root credentials.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): collapse acceptance report path guard
Combine the optional report path and existing-file checks into a let chain
without changing the report or PID sidecar collision behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
HttpCompressionConfig::extract_filename_from_content_disposition() calls header_value.to_lowercase() and slices &header_value[start..] using the byte index from lower.find("filename="), panicking on multi-byte Unicode characters whose UTF-8 byte length changes under case mapping and matching unanchored *filename= parameter suffixes.
Root Cause:
In rustfs/src/server/compress.rs, extract_filename_from_content_disposition() computes lower = header_value.to_lowercase() and uses the byte offset idx from lower.find("filename=") to slice &header_value[start..]. Because Unicode case mapping can expand or shrink UTF-8 byte lengths (e.g., U+0130 "İ" is 2 bytes in UTF-8 but lowercases to 3 bytes; U+212A "K" is 3 bytes in UTF-8 but lowercases to 1 byte), start can land inside a multi-byte UTF-8 codepoint or past the end of header_value, causing a runtime panic. In addition, lower.find("filename=") matches any parameter ending in filename= (such as x-filename=).
Fix:
Use header_value.to_ascii_lowercase() so byte offsets remain 1:1 with header_value, and require filename= to appear either at the start of the header or after a semicolon parameter delimiter.
Signed-off-by: czcbangkai <czcbangkai@google.com>
StdBackend::pread_bytes caches FdCacheEntry { file, len } and skips fstat on cache hits, passing cached len directly to unsafe { mmap_options.map(file) }. However, StdBackend::open_write / UringBackend::open_write (create_file, append_file), LocalDisk::write_all_private (write_all), LocalDisk::rename_part, and directory LocalDisk::rename_file fail to invalidate FdCache, causing mmap past EOF (SIGBUS), false DiskError::FileCorrupt on grown files, or stale reads after renames.
Root Cause:
In crates/ecstore/src/disk/local.rs, StdBackend::pread_bytes caches open File handles and their initial fstat byte length in FdCache and uses the cached len directly for bounds checks and unsafe { mmap_options.map(file) }. While LocalDisk::rename_file (non-directory) and LocalDisk::delete invalidate FdCache, five mutation paths omit invalidation: (1) StdBackend::open_write (called by create_file with O_TRUNC and append_file), (2) LocalDisk::write_all_private (called by write_all with O_TRUNC), (3) LocalDisk::rename_part, (4) LocalDisk::rename_file when src_is_dir is true, and (5) UringBackend::open_write / unlink / rename which fail to forward invalidations to self.inner. Truncating a cached file across a page boundary causes mmap to map beyond EOF and trigger SIGBUS; growing a cached file causes false DiskError::FileCorrupt.
Fix:
Invalidate FdCache in StdBackend::open_write, UringBackend::open_write/unlink/rename, LocalDisk::write_all_private, LocalDisk::rename_part (both src and dst), and directory LocalDisk::rename_file (via invalidate_prefix on src and dst).
Signed-off-by: czcbangkai <czcbangkai@google.com>
* test(e2e): add MPU RPC quorum and recovery acceptance
Exercise separate storage and lock RPC delays across multipart phases,
including quorum boundaries, cancellation, bucket recreation, and bounded
eight-node load. Honor TMPDIR for isolated cluster drive directories.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(lock): probe healthy voters behind silent peers
Spread additional probes across the existing acquisition attempt budget
when the initial quorum-plus-spare wave stalls. Preserve quorum thresholds,
fast-path fan-out, and existing ownership and late-response cleanup.
Cover every four-node read-voter ordering, a stalled eight-node write
quorum, and normal fan-out. Extend MPU acceptance to EC12+4 at twelve and
eleven metadata acknowledgements.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
overtrue