mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-07 13:53:12 +00:00
Zhengchao An
92402a3bde
perf(get,heal): land verified fixes for backlog #800-#804 Five fixes from the GET-path performance audit and scanner/heal completeness audit (rustfs/backlog#800..#804), each verified locally: - backlog#800 (heal checkpoint O(N^2)): ResumeCheckpoint object sets are now HashSet (Vec::contains was O(n) per healed object; 1.5ms at n=1M vs 11ns measured), and per-object checkpoint/resume-state persistence is batched (1000 mutations / 5s) instead of rewriting the whole file per object. complete_page() prunes the sets at page boundaries so memory stays bounded; positions still persist unconditionally and legacy Vec-format checkpoints still deserialize. - backlog#801 (DiskInfo.healing never set): erasure-set heal now writes a healing marker (.rustfs.sys/healing.bin) on the disks it rebuilds (endpoints plumbed via HealRequest/HealTask.heal_endpoints from the auto disk scanner) and clears it on success. LocalDisk::disk_info surfaces the marker, so scanner heal coordination, lock selection and admin/metrics healing counts see the rebuild. - backlog#802 (cache probe after data read): new GetObjectBodyCacheHook in ecstore lets the app-layer object data cache serve the body inside get_object_reader, after metadata quorum resolution (etag known) but before the erasure shard read/decode. Previously a hit still paid the full disk read. Hook is None/no-op when the cache is disabled. - backlog#803 (GET hot-path redundant work): ObjectInfo is cloned for event notification only when an event will actually be built (GET and HEAD paths; events are currently suppressed so the clone was pure waste); get_opts/put_opts/del_opts resolve bucket versioning with one metadata-sys lookup instead of two; skip_verify_bitrot and get_lock_acquire_timeout env reads are cached via OnceLock; the io-priority metric is no longer double-counted; GetObject input fields are cloned selectively instead of cloning the whole input. - backlog#804 (disk permit starvation): the disk-read permit wait is now bounded (RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT, default 5s, 0 = previous unbounded behavior); on timeout the GET proceeds without a permit and the bypass is counted. DiskReadPermitReader also releases the permit at body EOF instead of holding it until the client drops the stream. Verification: make pre-commit; cargo clippy -D warnings on the four changed crates; full rustfs lib suite (2096 tests) green; rustfs-heal lib suite green with new unit tests for checkpoint pruning/legacy format/throttle, permit EOF release, and the cache hook (hit + SSE skip). The heal_integration_test and one set_disk listing test fail identically on unmodified main (pre-existing global-state ordering flakes, verified via git stash A/B). Co-authored-by: houseme <housemecn@gmail.com>
RustFS ECStore - Erasure Coding Storage
High-performance erasure coding storage engine for RustFS distributed object storage
📖 Documentation
· 🐛 Bug Reports
· 💬 Discussions
📖 Overview
RustFS ECStore provides erasure coding storage capabilities for the RustFS distributed object storage system. For the complete RustFS experience, please visit the main RustFS repository.
✨ Features
- Reed-Solomon erasure coding implementation
- Configurable redundancy levels (N+K schemes)
- Automatic data healing and reconstruction
- Multi-drive support with intelligent placement
- Parallel encoding/decoding for performance
- Efficient disk space utilization
📚 Documentation
For comprehensive documentation, examples, and usage guides, please visit the main RustFS repository.
📄 License
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
Copyright 2024 RustFS Team
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
RustFS is a trademark of RustFS, Inc.
All other trademarks are the property of their respective owners.
Made with ❤️ by the RustFS Storage Team