gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-20 19:42:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

perf(ecstore): slice in-memory shards instead of copying them twice (#4687)

Browse Source 
* perf(ecstore): slice in-memory shards instead of copying them twice (backlog#1159)

The GET path reads a shard out of the page cache into a `Bytes`, then
`open_disk_reader` erased it behind `Box<dyn AsyncRead>` by wrapping it in
a `Cursor`. Downstream, `BitrotReader` could only get it back by copying:
once out of the `Cursor` into its scratch buffer, and once from there into
the caller's buffer. CPU profiling of a cached 1 MiB GET (device reads = 0)
attributed 8.23% of the whole server to `Cursor::poll_read` alone — a copy
of data that was already sitting in memory.

Keep the source concrete instead of erasing it. `ShardReader` is an enum of
`InMemory(Cursor<Bytes>)` and `Stream(Box<dyn AsyncRead ...>)`, and the new
`ShardSource::try_take_block(n)` lets an in-memory source hand over the
`[hash][data]` block as a slice. `read_appending` uses it to verify the hash
on the slice and `extend_from_slice` the shard straight into the caller's
buffer: one copy instead of two.

`try_take_block` defaults to `None`, so a streaming source keeps the old
path byte for byte, along with its short-read and EOF semantics. A source
that cannot serve `n` bytes declines rather than truncating, so a partial
block still becomes UnexpectedEof rather than a short shard. The hash is
still checked before anything is appended, so a corrupt shard never reaches
the caller's buffer on either path. The deferred parity reader opens its
source lazily and stays on the streaming path; parity is only read when a
data shard fails.

Tests gate equivalence and non-vacuity:
  * `try_take_block` fires for `Cursor<Bytes>`, advances the position exactly
    as a read of the same length would, declines when fewer than `n` bytes
    remain, and returns `None` for a non-`Bytes` source — without this the
    equivalence test below would silently compare one path against itself;
  * both paths return identical bytes for the same shard;
  * a corrupt shard fails on the fast path too, appending nothing.

Verified: clippy --tests -D warnings clean; `erasure::` 215 passed, 0 failed;
`set_disk::core::io_primitives` 49 and `io_support::` 22 pass.

Stacked on #4681 (`read_appending`), which this builds on.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ecstore): implement ShardSource for Cursor<&[u8]> used by the erasure bench

`crates/ecstore/benches/erasure_benchmark.rs` builds
`BitrotReader<Cursor<&[u8]>>`, which the new `ShardSource` bound on
`ParallelReader`/`decode` does not accept. `cargo clippy --tests` does not
compile bench targets, so this only surfaced in CI's `--all-targets` run.

A borrowed slice carries no `Bytes` to hand out, so it takes the default
`try_take_block` and keeps the old streaming copy path — no behavior change.

Verified with the same target set CI uses:
`cargo clippy -p rustfs-ecstore --all-targets -- -D warnings` clean.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
2026-07-10 23:57:10 +08:00
committed by GitHub
parent b9e5e818a9
commit c2362bca14
5 changed files with 235 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/ecstore/src/erasure/coding/decode.rs
+18 -12
View File
@@ -34,7 +34,6 @@ use std::io;
use std::io::ErrorKind;
use std::pin::Pin;
use std::time::{Duration, Instant};
use tokio::io::AsyncRead;
use tokio::io::AsyncWrite;
use tokio::io::AsyncWriteExt;
use tracing::{error, warn};
@@ -270,7 +269,7 @@ fn read_shard<'a, R>(
metrics_path: Option<&'static str>,
) -> ShardReadFuture<'a>
where
R: AsyncRead + Unpin + Send + Sync + 'a,
R: crate::erasure::coding::ShardSource + 'a,
{
let role = shard_role(index, data_shards);
if let Some(reader) = reader {
@@ -395,7 +394,7 @@ pub(crate) struct ParallelReader<R> {
impl<R> ParallelReader<R>
where
R: AsyncRead + Unpin + Send + Sync,
R: crate::erasure::coding::ShardSource,
{
// Readers should handle disk errors before being passed in, ensuring each reader reaches the available number of BitrotReaders
pub fn new(readers: Vec<Option<BitrotReader<R>>>, e: Erasure, offset: usize, total_length: usize) -> Self {
@@ -686,7 +685,7 @@ fn record_scheduled_read_cost(
impl<R> ParallelReader<R>
where
R: AsyncRead + Unpin + Send + Sync,
R: crate::erasure::coding::ShardSource,
{
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub async fn read(&mut self) -> (Vec<Option<Vec<u8>>>, Vec<Option<Error>>) {
@@ -1247,7 +1246,7 @@ where
#[async_trait::async_trait]
impl<R> ShardStripeSource for ParallelReader<R>
where
R: AsyncRead + Unpin + Send + Sync,
R: crate::erasure::coding::ShardSource,
{
async fn read_next_stripe(&mut self) -> StripeReadState {
let read_quorum = self.data_shards;
@@ -1275,7 +1274,7 @@ async fn read_stripe_timed<R>(
stage_metrics_enabled: bool,
) -> (Vec<Option<Vec<u8>>>, Vec<Option<Error>>)
where
R: AsyncRead + Unpin + Send + Sync,
R: crate::erasure::coding::ShardSource,
{
let stripe_read_stage_start = get_stage_timer_if_enabled(stage_metrics_enabled);
let out = reader.read().await;
@@ -1422,7 +1421,7 @@ impl Erasure {
) -> (usize, Option<std::io::Error>)
where
W: AsyncWrite + Send + Sync + Unpin,
R: AsyncRead + Unpin + Send + Sync,
R: crate::erasure::coding::ShardSource,
{
self.decode_inner(writer, readers, offset, length, total_length, None, Vec::new())
.await
@@ -1439,7 +1438,7 @@ impl Erasure {
) -> (usize, Option<std::io::Error>)
where
W: AsyncWrite + Send + Sync + Unpin,
R: AsyncRead + Unpin + Send + Sync,
R: crate::erasure::coding::ShardSource,
{
self.decode_inner(writer, readers, offset, length, total_length, Some(read_costs), Vec::new())
.await
@@ -1461,7 +1460,7 @@ impl Erasure {
) -> (usize, Option<std::io::Error>)
where
W: AsyncWrite + Send + Sync + Unpin,
R: AsyncRead + Unpin + Send + Sync,
R: crate::erasure::coding::ShardSource,
{
self.decode_inner(writer, readers, offset, length, total_length, read_costs, deferred_handles)
.await
@@ -1582,7 +1581,7 @@ impl Erasure {
) -> (usize, Option<std::io::Error>)
where
W: AsyncWrite + Send + Sync + Unpin,
R: AsyncRead + Unpin + Send + Sync,
R: crate::erasure::coding::ShardSource,
{
if readers.len() != self.data_shards + self.parity_shards {
record_get_object_pipeline_failure(GET_STAGE_RANGE, GetObjectFailureReason::RangeOrLengthInvalid);
@@ -1806,10 +1805,11 @@ mod tests {
atomic::{AtomicUsize, Ordering},
};
use std::task::{Context, Poll};
use tokio::io::AsyncRead;
use tokio::io::ReadBuf;
use tokio::time::{Instant as TokioInstant, Sleep};
type BoxedShardReader = Box<dyn AsyncRead + Send + Sync + Unpin>;
type BoxedShardReader = crate::io_support::bitrot::ShardReader;
/// Counts the raw bytes pulled from a shard stream, to prove which shards
/// a decode path actually touches (backlog#923 call-count evidence).
@@ -1818,6 +1818,10 @@ mod tests {
bytes_read: Arc<AtomicUsize>,
}
/// Streaming test source: no in-memory block, so it exercises the same path a
/// real disk stream takes.
impl crate::erasure::coding::ShardSource for CountingShardReader {}
impl AsyncRead for CountingShardReader {
fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
let before = buf.filled().len();
@@ -1853,7 +1857,7 @@ mod tests {
.map(|(_, len)| buf[..*len].to_vec())
.unwrap_or_else(|| buf.clone());
readers.push(Some(BitrotReader::new(
Box::new(Cursor::new(bytes)) as BoxedShardReader,
crate::io_support::bitrot::ShardReader::Stream(Box::new(Cursor::new(bytes))),
shard_size,
hash_algo.clone(),
false,
@@ -1893,6 +1897,8 @@ mod tests {
TimedOut,
}
impl crate::erasure::coding::ShardSource for TestShardReader {}
impl AsyncRead for TestShardReader {
fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
match &mut *self {