mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-04 12:27:43 +00:00
houseme
d232a46b4d
perf(ecstore): wire legacy decode stripe prefetch / bitrot-decode overlap (backlog#930 HP-9 step 2) The legacy GET decode duplex loop (default GET path + every Range request) was strictly serial: read a stripe, reconstruct it, emit it, and only then begin reading the next stripe. Two switches introduced by PR#3972 to overlap this work — RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT and RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE — were config-only with zero call sites since introduction. Wire both into the loop as a depth-1 stripe prefetch: while the current stripe is reconstructed and emitted, the next stripe's shard reads (including bitrot verification) run concurrently, hiding read latency under the emit / duplex-backpressure stage. ParallelReader::read is inherently serial (it takes &mut self and advances a shared stripe cursor), so at most one read can be in flight; a prefetch count above 1 therefore collapses to the same single-stripe-ahead pipeline rather than reading several stripes ahead. Both switches feed one gate, legacy_stripe_prefetch_enabled(). Default (count == 1, overlap == false) keeps the loop on the pre-existing strictly-serial path, byte for byte: the prefetch pipeline is a separate branch and the serial branch is unchanged. The reconstruct/emit body is factored into a shared emit_decoded_stripe helper used by both branches so error attribution, read-quorum handling, reconstruction verification and stage metrics cannot drift between them. Correctness guarantees preserved under prefetch: - A speculatively prefetched read for stripe N+1 is only consumed when the loop reaches N+1; if stripe N stops the loop, the in-flight read is awaited and dropped, so its errors never surface against stripe N. - Bitrot (HighwayHash) verification runs inside each stripe read and is not bypassed or reordered; a corrupt shard is still rejected and, when unrecoverable, the read fails without emitting garbage. - Shard buffers are recycled only after the overlapping next read has claimed its own — one extra stripe of memory (double buffering) with buffer reuse preserved at a one-stripe lag. - Per-stripe exact length advance (backlog#799 B2), lockstep reconstruction verification (backlog#832) and the hash_size == 0 pass-through are unchanged. Adds regression tests exercising serial-default, count>1 and overlap configs: full/range reads byte-exact, degraded (missing-shard) reconstruction, corrupt shard rejected-but-recovered, unrecoverable corruption erroring with no output, late-stripe failure attributed correctly with stripe 0 still emitted, hash_size == 0 pass-through, and the gate defaulting off. Co-authored-by: heihutu <heihutu@gmail.com>
RustFS ECStore - Erasure Coding Storage
High-performance erasure coding storage engine for RustFS distributed object storage
📖 Documentation
· 🐛 Bug Reports
· 💬 Discussions
📖 Overview
RustFS ECStore provides erasure coding storage capabilities for the RustFS distributed object storage system. For the complete RustFS experience, please visit the main RustFS repository.
✨ Features
- Reed-Solomon erasure coding implementation
- Configurable redundancy levels (N+K schemes)
- Automatic data healing and reconstruction
- Multi-drive support with intelligent placement
- Parallel encoding/decoding for performance
- Efficient disk space utilization
📚 Documentation
For comprehensive documentation, examples, and usage guides, please visit the main RustFS repository.
📄 License
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
Copyright 2024 RustFS Team
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
RustFS is a trademark of RustFS, Inc.
All other trademarks are the property of their respective owners.
Made with ❤️ by the RustFS Storage Team