Collapse per-shard blocking-pool round-trips into one spawn_blocking
pread batch when all online shards are local and mmap-read is enabled.
Co-authored-by: ba <ba@ubuntu-server.alpha30.bos16>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* perf(ecstore): slice in-memory shards instead of copying them twice (backlog#1159)
The GET path reads a shard out of the page cache into a `Bytes`, then
`open_disk_reader` erased it behind `Box<dyn AsyncRead>` by wrapping it in
a `Cursor`. Downstream, `BitrotReader` could only get it back by copying:
once out of the `Cursor` into its scratch buffer, and once from there into
the caller's buffer. CPU profiling of a cached 1 MiB GET (device reads = 0)
attributed 8.23% of the whole server to `Cursor::poll_read` alone — a copy
of data that was already sitting in memory.
Keep the source concrete instead of erasing it. `ShardReader` is an enum of
`InMemory(Cursor<Bytes>)` and `Stream(Box<dyn AsyncRead ...>)`, and the new
`ShardSource::try_take_block(n)` lets an in-memory source hand over the
`[hash][data]` block as a slice. `read_appending` uses it to verify the hash
on the slice and `extend_from_slice` the shard straight into the caller's
buffer: one copy instead of two.
`try_take_block` defaults to `None`, so a streaming source keeps the old
path byte for byte, along with its short-read and EOF semantics. A source
that cannot serve `n` bytes declines rather than truncating, so a partial
block still becomes UnexpectedEof rather than a short shard. The hash is
still checked before anything is appended, so a corrupt shard never reaches
the caller's buffer on either path. The deferred parity reader opens its
source lazily and stays on the streaming path; parity is only read when a
data shard fails.
Tests gate equivalence and non-vacuity:
* `try_take_block` fires for `Cursor<Bytes>`, advances the position exactly
as a read of the same length would, declines when fewer than `n` bytes
remain, and returns `None` for a non-`Bytes` source — without this the
equivalence test below would silently compare one path against itself;
* both paths return identical bytes for the same shard;
* a corrupt shard fails on the fast path too, appending nothing.
Verified: clippy --tests -D warnings clean; `erasure::` 215 passed, 0 failed;
`set_disk::core::io_primitives` 49 and `io_support::` 22 pass.
Stacked on #4681 (`read_appending`), which this builds on.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): implement ShardSource for Cursor<&[u8]> used by the erasure bench
`crates/ecstore/benches/erasure_benchmark.rs` builds
`BitrotReader<Cursor<&[u8]>>`, which the new `ShardSource` bound on
`ParallelReader`/`decode` does not accept. `cargo clippy --tests` does not
compile bench targets, so this only surfaced in CI's `--all-targets` run.
A borrowed slice carries no `Bytes` to hand out, so it takes the default
`try_take_block` and keeps the old streaming copy path — no behavior change.
Verified with the same target set CI uses:
`cargo clippy -p rustfs-ecstore --all-targets -- -D warnings` clean.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(ecstore): add stripe-advance handles for deferred bitrot readers
Give DeferredObjectReader a shared pending state and expose a
DeferredReaderStripeHandle that advances the still-unopened source by whole
bitrot blocks using the same bitrot_encoded_range geometry the reader was
created with (identity mapping when hash_size == 0). This lets the GET decode
path open a parity shard aligned to the stripe where a data shard failed
instead of reading every parity shard on every stripe (backlog#923).
An already-opened (or failed) reader rejects the advance so callers retire it
rather than engage it out of alignment; bitrot verification after an advance
checks the advanced stripe's block against that stripe's stored hash.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ecstore): read only data shards on healthy lockstep GET behind opt-in gate
PR #4289's lockstep fix made every reconstruction-verifying GET read all
data+parity shards per stripe; the parity blocks are read, bitrot-hashed and
then discarded, a deterministic 2x read-bytes/IOPS/hash-CPU amplification on
healthy 2+2 objects (backlog#923). With the new opt-in gate
RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE=true (default: false, behavior
identical to main):
- read_lockstep keeps only the data slots engaged while the object is
healthy; parity slots stay unopened deferred readers.
- When a data shard is missing or dies at stripe k, parity readers are
engaged mid-object by advancing their deferred stripe handle to stripe k,
preserving the lockstep alignment invariant from backlog#832.
- Degraded stripes engage one parity beyond the decode quorum so
reconstruction verification keeps an extra source to check against
(erasure.rs only verifies when available > data shards); an engaged parity
reader that errors is retired for the rest of the object like any other,
and a parity reader that cannot be realigned is retired instead of being
read out of position.
- fill_deferred_bitrot_readers records stripe handles for deferred slots and,
gate-on only, swaps eagerly opened parity readers for unopened deferred
ones so they remain engageable mid-object; ready/error bookkeeping used by
quorum decisions is untouched.
- Both GET paths (legacy duplex via Erasure::decode_with_stripe_handles,
codec streaming via ParallelReader::with_deferred_parity_handles) carry the
handles from reader setup.
Short-read -> UnexpectedEof -> whole-object retirement and the
inconsistent-source rejection are unchanged in both gate modes; tests lock
the healthy-path data-shards-only call counts, the default read-all-shards
behavior, mid-object parity engagement for streaming and hash_size==0
formats, and mid-stream inconsistent-parity rejection.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
rdiperri-wasabi