gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-07 22:03:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

perf(erasure): remove UUID from clone + increase encode inflight budget (#3212)

Browse Source 
* perf(erasure): remove UUID from clone + increase encode inflight budget

Two targeted optimizations for the erasure encoding hot path:

1. Erasure::clone() no longer generates Uuid::new_v4() per clone.
   The _id field is unused in the hot path; reusing the original ID
   eliminates a CSPRNG call per block encode (100 calls for a 100MB
   object with 1MB blocks).

2. Default RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES raised from 8MB
   to 32MB. This increases the encode pipeline depth from ~5 to ~20
   blocks, allowing more read-ahead between the encoder and disk
   writer stages. The per-request memory bound is still controlled
   by the 8-block hard cap and the env var override.

3. Added encode_data_owned() utility method for zero-copy encoding
   when the caller already owns a heap buffer (Vec<u8> → BytesMut
   via Bytes::try_into_mut). Not used in the hot path yet but
   available for future callers.

All 1157 ecstore tests pass. Criterion micro-benchmarks show no
regression (< 2% variance). Single-machine warp E2E tests were
inconclusive due to high variance; a dedicated multi-disk test
environment is needed for reliable E2E comparison.

Ref: https://github.com/rustfs/backlog/issues/659

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore: update Cargo.lock

* fix(erasure): align encode inflight cap and tests

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
安正超
2026-06-04 21:49:36 +08:00
committed by GitHub
parent fde519910d
commit 3bd89944c2
3 changed files with 142 additions and 63 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/ecstore/src/erasure_coding/encode.rs
+3 -2
View File
@@ -27,8 +27,8 @@ use tokio::sync::mpsc;
use tracing::error;
const ENV_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES: &str = "RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES";
const DEFAULT_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES: usize = 8 * 1024 * 1024;
const DEFAULT_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BLOCKS: usize = 8;
const DEFAULT_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES: usize = 32 * 1024 * 1024;
const DEFAULT_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BLOCKS: usize = 32;
fn encode_channel_capacity(expanded_block_bytes: usize, max_inflight_bytes: usize) -> usize {
if expanded_block_bytes == 0 {
@@ -501,6 +501,7 @@ mod tests {
#[test]
fn encode_channel_capacity_respects_budget_and_hard_cap() {
assert_eq!(encode_channel_capacity(4 * 1024 * 1024, 32 * 1024 * 1024), 8);
assert_eq!(encode_channel_capacity(1536 * 1024, 32 * 1024 * 1024), 21);
assert_eq!(encode_channel_capacity(16 * 1024 * 1024, 32 * 1024 * 1024), 2);
assert_eq!(encode_channel_capacity(1, usize::MAX), DEFAULT_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BLOCKS);
}