mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-11 07:36:53 +00:00
Zhengchao An
a7f035a8c3
The LoadBucketMetadata peer-notification handler loaded bucket metadata with the fabricating loader (ConfigNotFound -> BucketMetadata::new) and unconditionally cached the result. On a transient read-quorum dip during a reload notification, a peer cached an authoritative "no Object Lock" default for a lock-enabled bucket, disabling the batch-delete retention gate (object_lock_delete_check_required) on that node until the next refresh, and wiping its bucket-target/durability sync state. Production changes: - New BucketMetadataSys::reload_from_store (metadata_sys:: reload_bucket_metadata): the peer reload path uses the presence-aware loader and installs only metadata actually read from persisted storage. A load miss returns an error (surfaced to the notifying peer as success=false) and leaves the cache untouched; deletion still propagates only through the dedicated DeleteBucketMetadata notification. - The reload runs under the outer metadata-sys write guard, load included, mirroring update(): every other cache installer holds that lock, so a reload snapshot can never land after - and roll back - a newer concurrent install (the stale-load lost-update from the review), and the install-plus-registry-sync sequence stays atomic against concurrent removes and reloads (previously only the set call was write-guarded, with the load outside any lock). - The peer-visible miss error is a fixed string: the notifying peer substring-matches error text against network-failure needles (is_network_like_error), so interpolating a bucket name (e.g. a legal bucket literally named "unavailable") could mark a healthy peer offline. - get_config's lazy insert routes through set(), picking up the negative-cache invalidation. An earlier draft instead guarded set() with a per-config updated_at freshness comparison. Adversarial validation rejected it (three roles independently): update_config stamps with the handling node's wall clock, so within the skew the cluster already tolerates (+/-300s RPC auth window) a config rewritten with an earlier stamp - e.g. revoking a public-read policy through a second node, or any same-field rewrite after an NTP step-back - would be skipped by every peer forever, silently pinning the revoked permissive config with no re-convergence path (the 15-minute refresh also routed through the guard). Race staleness is second-scale while skew is minute-scale, so no tolerance bound can separate them; the write-guard serialization closes the same race without clocks and preserves the refresh loop's unconditional converge-to-disk property, which is the cluster's self-healing mechanism. Startup audit (BucketMetadataSys::init): concurrent_load's insert-if-vacant still installs a fabricated default when a transient miss hits at boot - indistinguishable from a legacy bucket without a metadata file at this layer - bounded by the next successful persisted load. Making the object-lock gate fail closed on such entries is filed as a follow-up, alongside the bare "unavailable" needle in is_network_like_error and the Swift cache-only metadata writes. Tests: - bucket::metadata_sys::tests:: peer_reload_never_caches_fabricated_defaults_as_authoritative: miss installs nothing / miss keeps the existing entry intact (asserting the dedicated non-persisted error) / persisted reload converges the cache over a stale entry. - node_service::tests:: test_load_bucket_metadata_failure_skips_scanner_maintenance: a failed reload reports failure and does not advance scanner maintenance activity (previously recorded even on a miss). - The handler success path stays uncovered at the RPC layer (needs an isolated global object layer, like the pre-existing ignored test); the composition is pinned at the sys level instead. Verification: - cargo fmt --check and cargo clippy --lib --tests clean on rustfs-ecstore and rustfs. - Targeted suites green; full cargo test -p rustfs-ecstore --lib: 3198/3200 with two parallelism-sensitive lock-test flakes from the known baseline (pass in isolation; a different pair flakes per run). - Adversarial validation (high-risk tier, all seven roles as independent parallel reviewers) run per AGENTS.md; all findings fixed or rebutted with evidence, three out-of-scope findings filed as follow-up tasks. Co-authored-by: houseme <housemecn@gmail.com>
RustFS ECStore - Erasure Coding Storage
High-performance erasure coding storage engine for RustFS distributed object storage
📖 Documentation
· 🐛 Bug Reports
· 💬 Discussions
📖 Overview
RustFS ECStore provides erasure coding storage capabilities for the RustFS distributed object storage system. For the complete RustFS experience, please visit the main RustFS repository.
✨ Features
- Reed-Solomon erasure coding implementation
- Configurable redundancy levels (N+K schemes)
- Automatic data healing and reconstruction
- Multi-drive support with intelligent placement
- Parallel encoding/decoding for performance
- Efficient disk space utilization
📚 Documentation
For comprehensive documentation, examples, and usage guides, please visit the main RustFS repository.
📄 License
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
Copyright 2024 RustFS Team
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
RustFS is a trademark of RustFS, Inc.
All other trademarks are the property of their respective owners.
Made with ❤️ by the RustFS Storage Team