mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-30 00:47:13 +00:00
61914ac4ad31155ab0aeab2df5349a46df031416
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 houseme
|
0c4c1caef8 |
perf(ecstore): add Vec<u8> buffer pool for EC operations (#6538)
* feat(mimalloc): add arena diagnostics and configuration Based on mimalloc maintainer feedback (microsoft/mimalloc#1372), add diagnostics to check mimalloc arena configuration at runtime. Changes: - Add rustfs-mimalloc-sys to workspace dependencies - Add log_mimalloc_diagnostics() function to check: - arena_max_object_size - pagemap_commit status - mimalloc version - Add memory_observability module with mimalloc diagnostics This helps diagnose why allocations might be going outside arenas, which is the suspected root cause of futex contention. Ref: rustfs/backlog#2005 Ref: microsoft/mimalloc#1372 Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): add Vec<u8> buffer pool for EC operations Add a general-purpose buffer pool to reduce Vec<u8> allocations in hot paths like EC encoding/decoding. Changes: - Add BufferPool struct in crates/ecstore/src/erasure/codec/buffer_pool.rs - Thread-safe pool with capacity-based bucketing (power-of-two) - Global EC_BUFFER_POOL instance with 16 buffers per bucket - Add buffer_pool module to codec/mod.rs Expected impact: - Reduce heap allocations in EC encode/decode paths - Avoid memzero overhead (proven 4.8% CPU saving in ShardBufferPool) - Reduce mimalloc lock contention Note: Main bottleneck remains mimalloc internal synchronization (futex 98.64% time). Buffer pool provides modest improvement (+2-5%). Ref: rustfs/backlog#2005 Co-Authored-By: heihutu <heihutu@gmail.com> * style: apply cargo fmt to buffer pool and related files Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): add #[allow(dead_code)] to buffer pool The BufferPool infrastructure is ready but not yet integrated into the EC hot paths. Add #[allow(dead_code)] with clear documentation about integration status. Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): integrate BufferPool into bitrot verify path Replace vec![0; shard_size] with get_ec_buffer() in the bitrot verification hot path to reduce heap allocations and avoid memzero. Co-Authored-By: heihutu <heihutu@gmail.com> * style: apply cargo fmt to buffer pool and bitrot changes Co-Authored-By: heihutu <heihutu@gmail.com> * refactor(ecstore): clean up buffer pool code - Remove unnecessary #[allow(dead_code)] attributes - Update module documentation to reflect current integration status - Simplify code structure Co-Authored-By: heihutu <heihutu@gmail.com> * perf(runtime): cap default worker threads at 16 Testing showed 16 worker threads outperforms 32+ for 1KiB PUT workloads due to reduced mimalloc lock contention. A/B test results (testing 4-node cluster, c=64): - worker_threads=32: 740 obj/s (baseline) - worker_threads=16: 785 obj/s (+6.1%) The default was detect_cores() which returned 32 on our testing nodes. Cap at 16 for optimal small-object performance. Ref: rustfs/backlog#2005 Co-Authored-By: heihutu <heihutu@gmail.com> * style: apply cargo fmt to buffer pool and runtime changes Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): remove unused BufferPool::new() function The new() function was never used since EC_BUFFER_POOL initializes directly with with_limits(16). Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): update buffer_pool tests to use with_limits Replace BufferPool::new() with BufferPool::with_limits(16) in tests since new() was removed in favor of with_limits(). Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: hector <hetor@rustfs.com> Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
eb41f45175 |
chore(ecstore): drop the cluster and erasure dead_code blankets (#6088)
Removing both blankets exposes 23 items, of which only four are deleted. The ratio is the point: close to the core data path the blankets were hiding test assertions and migration seams, not dead code. A cfg-split function is the reason two symbols in the internode transport look dead when neither is. build_internode_data_transport_from_env has two bodies, one under #[cfg(test)] that calls build_internode_data_transport directly and one under #[cfg(not(test))] that goes through the INTERNODE_DATA_TRANSPORT static so tests do not share process-global transport state. Each half's helper is live in exactly one build, and because cargo check --tests compiles both the lib target and the test harness, both symbols appear in one warning list. Deleting either one breaks the other lane. Both are kept with allows naming their half. Three deletion candidates were withdrawn after a per-name grep: ParallelReader::new, ErasureDecodeReader::new and SyncErasureDecodeReader::new all have test callers. The last two are exactly the shape of the dead wrapper deleted in #6084 — a thin forward to a new_with_metrics_path sibling — except that sibling is live in production (set_disk/read.rs) and the wrappers are used by tests. Deleted: - RemotePeerS3Client::get_addr and RemoteLocker::from_url, neither with a consumer in any lane. - RemotePeerS3Client's node field, which new writes after using it to derive addr and nothing ever reads. Its only other writer was a test helper that built a whole Node solely to fill the field; that block goes too. - ParallelReader::can_decode, superseded by an inlined copy. The copy's comment named the method it replaced, so deleting the method alone would have left a dangling reference; the comment now describes the check instead of pointing at a method that no longer exists. Kept with allows: the erasure items are decode/encode invariants asserted by their own files' tests (shard_read_launch_order, decode_with_read_costs, emit_data_shards, queued_block_bytes, the engine trait facets, the ParallelReader and decode-reader constructors, encode_stream_callback_async). On the cluster side, peer_replay_state, heal_bucket_local and clone_drives are test-only, InternodeDataTransportCapabilities and tcp_http are constructed only by transport test doubles, and the InternodeDataTransport trait's name/capabilities pair is an unused capability-negotiation facet kept for the transport split (backlog#1350) — six impls provide them and no caller negotiates on them yet. Verification, four lanes warning-free: default, --tests, --features rio-v2 --tests, --features test-util --tests. cargo nextest run -p rustfs-ecstore 4041 passed; clippy --lib --tests -D warnings clean; make pre-commit exit 0. Ref rustfs/backlog#1823 (step 2). |
||
|
houseme
|
d2b1003612 |
perf(storage): converge Wave 2 hot-path optimizations (#6065)
* perf(get): share inline shards and lock clients Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): converge PUT encoding on contiguous blocks Co-Authored-By: heihutu <heihutu@gmail.com> * perf(get): cache codec streaming gate config Co-Authored-By: heihutu <heihutu@gmail.com> * fix(sse): redact projected customer headers Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): collapse GET metadata snapshots Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): reuse decode stripe scratch Co-Authored-By: heihutu <heihutu@gmail.com> * refactor(ecstore): trim decode scratch adapters Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): adapt transition checks to metadata snapshots Co-Authored-By: heihutu <heihutu@gmail.com> * perf(get): release metadata snapshots at ownership boundary Co-Authored-By: heihutu <heihutu@gmail.com> * refactor(ecstore): close cumulative fast-path findings Co-Authored-By: heihutu <heihutu@gmail.com> * fix(storage): preserve lock and header invariants Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): adapt cumulative paths after rebase Co-Authored-By: heihutu <heihutu@gmail.com> * fix(rio-v2): adapt generated metadata fixture Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
5f1a475c56 |
perf(ecstore): stop zeroing pooled shard buffers on the GET path (backlog#1159) (#4681)
`ShardBufferPool::take` handed out a `resize(len, 0)`-ed buffer, and the
reader then overwrote every byte of it. CPU profiling of a cached 1 MiB
GET (device reads = 0, so all cost is CPU) attributed 4.81% of the whole
server to that memset — a buffer pool exists to reuse an allocation, and
memsetting it gives the saving straight back.
The zeroing was load-bearing only because `BitrotReader::read` takes
`&mut [u8]`, which must be initialized. But the reader never reads what
the caller put there, and never returns a partially filled buffer: both
the hashed and the no-hash path either fill the whole shard or fail with
UnexpectedEof, and a hash mismatch is an error rather than a short read.
So the initialization bought nothing observable.
Add `BitrotReader::read_appending(&mut Vec<u8>, want)`, which appends into
the buffer's spare capacity instead of demanding initialized bytes:
* hashed path — unchanged single copy, `extend_from_slice(data)` in place
of `copy_from_slice` into a pre-zeroed buffer, and only after the hash
verifies, so corrupt bytes never reach the caller's buffer;
* no-hash path — `read_buf` writes straight into the spare capacity and
advances the length only over bytes the reader actually wrote, so an
uninitialized tail can never be exposed.
`ShardBufferPool::take` now yields an empty buffer with capacity, and
`read_shard` no longer needs to `truncate`. `read` keeps its old signature
for the remaining callers.
Four tests gate the contract rather than the call:
* `read_appending` is byte-for-byte identical to `read` on both paths;
* a truncated shard is UnexpectedEof, never a partially filled buffer;
* bytes that fail the bitrot hash never reach the caller's buffer;
* `want > shard_size` is rejected;
plus the pool test now asserts the allocation is reused (same pointer) and
never zeroed.
Verified: `erasure::` 213 passed, 0 failed; on a real Linux host
`erasure::` 209 and `disk::local::` 143 pass serially, and the failures
seen in a parallel full-suite run reproduce identically on unmodified main
(they are ENOSPC from a full root filesystem plus pre-existing flakes).
Not claimed: an end-to-end throughput number. The A/B on the bench host was
too noisy to attribute (one rep pair was not fully cached, and its root
filesystem filled mid-run); what is measured is that the removed memset was
4.81% of GET CPU in the pre-change profile.
Co-authored-by: heihutu <heihutu@gmail.com>
|
||
|
Zhengchao An
|
ed81d2f6b8 |
test(ecstore): complete EC validation coverage gate
* test(ecstore): complete EC validation coverage gate * test(ecstore): stabilize validation suite after rebase * test(ecstore): fix rio-v2 clippy lint |
||
|
houseme
|
4715dbed01 | fix(ecstore): stop rejecting recoverable degraded reads in codec parity verification (#4399) | ||
|
houseme
|
46d7f9e1f2 |
feat(get): harden codec streaming rollout (#3981)
* feat(get): consolidate GET performance optimization Consolidated implementation of all GET performance optimizations into a single, well-organized commit replacing the previous patch-on-patch approach. ## Changes ### Configuration (set_disk/mod.rs) - Consolidated all GET optimization flags into a single organized section - Enabled by default: codec streaming, metadata early-stop, page cache reclaim - Added codec streaming multipart flag (default: disabled) - Added version-aware early-stop flag (default: disabled) - Added adaptive duplex buffer sizing based on object size - All flags use OnceLock caching with rollout percentage support ### Metadata Early-Stop (set_disk/read.rs) - Delete marker early-stop when quorum agrees - Version-aware early-stop for versioned GET requests - MetadataQuorumAccumulator enhanced with: - delete_marker_votes tracking - requested_version_id and matching_version_votes tracking - version_early_stop_decision() method - 6 new tests for version early-stop scenarios ### Codec Streaming (erasure/coding/decode_reader.rs) - DualInFlight (2-stripe lookahead) enabled by default ### Decode Pipeline (erasure/coding/decode.rs) - Stripe prefetch count configuration - Bitrot-decode overlap configuration ### Disk Layer (disk/local.rs) - O_DIRECT read configuration constants (preparation) ### Metrics (io-metrics/lib.rs) - BytesPool acquisition/return metrics - Metadata phase duration with early-stop label - Total duration with reader_path label ### Diagnostics (diagnostics/) - Early-stop reason constants - Pool tier/outcome label constants ### Observability (.docker/observability/) - 3 Grafana dashboards for GET optimization monitoring - Prometheus alert rules (6 alerts: 3 critical, 3 warning) - Updated README.md and README_ZH.md with usage docs ### Config (config/src/constants/runtime.rs) - Page cache reclaim read enabled by default ## Environment Variables | Variable | Default | Description | |----------|---------|-------------| | RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag | | RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % | | RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming | | RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag | | RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % | | RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop | | RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim | | RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) | | RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch | | RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap | | RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes | ## Rollback All optimizations can be disabled via environment variables: RUSTFS_GET_CODEC_STREAMING_ENABLE=false RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false Co-Authored-By: heihutu <heihutu@gmail.com> * test(get): add stress test scripts for GET optimization validation - quick-validate-get-optimization.sh: Quick 5-minute validation - stress-test-get-optimization.sh: Full 30+ minute stress test - README-stress-test.md: Usage documentation Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): align file cache reclaim defaults * chore(deps): update redis and erasure codec * test(ecstore): align decode fill policy default * fix(get): wire codec streaming rollout gate * perf(get): skip metrics-off codec timers * test(get): capture codec streaming diagnostics * test(get): add multipart fallback probe * test(get): add encrypted fallback probe * test(get): add compressed fallback probe * test(get): add degraded read fallback probe * test(get): cover remote fallback probe * test(get): report warp request p99 * test(get): capture OTLP metric deltas * perf(get): align codec streaming inflight default * perf(get): reuse codec reader output buffers * test(get): count codec reader fill starts * perf(get): reuse codec reader fill worker * perf(get): lazy init rustfs codec reconstruct * test(get): cover rustfs codec source faults * docs(get): record rustfs codec fallback scope * feat(get): add multipart codec reader opt-in * test(get): add multipart codec smoke option * test(get): cover multipart codec degraded fallback * perf(get): bound multipart codec eager setup * test(get): satisfy codec hardening PR gate --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
GatewayJ
|
675597ec16 |
fix(ecstore): handle stalled recovery reads and listings (#3790)
* fix(ecstore): handle stalled recovery reads and listings * fix(rio): start HTTP stall timeout on read * fix(ecstore): handle stalled reads and partial lists * fix(ecstore): retire stalled shards and list errors * fix(ecstore): preserve list merge lookahead entries * fix(ecstore): bound zero-copy shard reads * fix(ecstore): hedge stalled shard reads * fix(ecstore): retire abandoned shard reads * fix(ecstore): include part identity in metadata quorum * fix(ecstore): validate heal shard sources * fix(ecstore): verify reconstructed read shards * chore(ecstore): log slow object read stages * fix(heal): throttle auto heal during recovery * fix(scanner): yield to foreground reads * fix(scanner): track streaming object reads * fix(ecstore): avoid false read heal fanout * fix(ecstore): verify codec streaming reconstruction sources * fix(ecstore): preserve quorum progress on slow shards * fix(storage): restore read timeout facade * fix(ecstore): retain fallback readers after quorum * chore: allow decode helper argument lists --------- Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Zhengchao An
|
0a5b1b1b3a |
refactor: consolidate ecstore owner module layout (#3934)
* refactor: shrink ecstore root owner facades * refactor: remove ecstore core store root shims * refactor: move ecstore erasure owner modules * refactor: remove ecstore root rpc facade * refactor: move ecstore services domain modules |