mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-16 18:08:21 +00:00
唐小鸭
c1f66969d7
* test(site-replication): pin ILM expiry merge contract for incoming lc-config Red-light evidence for backlog#1675 P1-1: the lc-config receiver overwrites the whole local lifecycle config with whatever the peer sends (and deletes it wholesale on peer delete), so an expiry-only document erases the receiver's local tier/transition rules, and peer transition rules get installed across sites. The new tests pin the MinIO mergeWithCurrentLCConfig semantics plus RustFS hardening: - incoming expiry documents merge with (never replace) local rules - local transition sides are authoritative for same-id rules - incoming transition fields are discarded at the trust boundary - dropped expiry rules strip the expiry side but keep transitions; pure-expiry rules are removed - delete merges with the empty set instead of dropping the config - disabled rules survive; abort-mpu-only rules stay site-local - deterministic order (idempotent re-delivery) and expiry_updated_at stamping for the staleness axis All fail against the current overwrite implementation (identity extraction of merge_incoming_lifecycle_config). * fix(site-replication): merge incoming ILM expiry documents instead of overwriting The lc-config receiver replaced the whole local lifecycle config with the peer's document (and deleted it wholesale on peer delete), so an expiry-only update erased the receiver's local tier/transition rules, and a peer's transition rules were installed across sites (backlog#1675 P1-1). Receiver (apply_bucket_meta_item): - lc-config now merges via merge_incoming_lifecycle_config, mirroring MinIO's mergeWithCurrentLCConfig with a trust-boundary hardening: incoming transition fields are discarded outright; the local transition side of a same-id rule is authoritative. A peer delete merges with the empty set — pure-expiry rules go away, transition rules survive with their expiry side cleared, and only an empty result deletes the config file. - Staleness moves to the expiry axis (config.expiry_updated_at): lifecycle_config_updated_at also moves on local transition-only edits, which shadowed newer peer expiry updates. - Receiver-side replicateILMExpiry gate, symmetric with the sender hook (previously any peer could install expiry rules while the option was off). - Rule order is deterministic (local order, incoming-new appended), so re-delivering the same document is byte-stable and does not rewrite bucket metadata per broadcast. Sender: - Both admin choke points — the bucket-meta hook and the SRInfo bucket entry feeding bootstrap/repair and consistency views — now emit only the expiry subset (transition fields stripped, non-expiry rules dropped). MinIO receivers install incoming rules verbatim, so transition rules must never leave the site. An unparseable local config is forwarded unfiltered rather than degraded to a delete. Not covered here (follow-up): a two-site e2e with a real tier backend to exercise transition-rule preservation end to end; receiver-side validate_transition_tier for merged configs. * fix(site-replication): close ILM merge review findings Adversarial review of the lc-config merge surfaced four real defects, all fixed here: - Deletion tombstone regression: with the staleness axis moved to the in-config expiry_updated_at, a deleted lifecycle config fell back to UNIX_EPOCH and any delayed stale broadcast could resurrect deleted expiry rules. The axis now falls back to the whole-config write time (which survives deletion in bucket metadata as the deletion's lower bound), also covering legacy configs that predate the axis field. - MinIO zero-rule documents: MinIO's delete tombstone / transition-only state marshals a lifecycle document with no <Rule>, which the strict s3s deserializer rejects — the receiver now recognizes it as the 'no expiry rules here' statement (delete semantics) instead of erroring on every MinIO heal pass. - Inflated expiry axis at the sender: PutBucketLifecycle stamped expiry_updated_at unconditionally, so a transition-only edit advanced the axis and let this site's stale expiry subset shadow and roll back newer peer expiry edits fleet-wide. The stamp is now conditional (expiry subset present before or after the edit, MinIO parity), the hook item travels with the config's expiry axis (UNIX_EPOCH when the site has none), and the SRInfo bucket entry feeds bootstrap/repair the same axis instead of the whole-config write time. - Del-marker parity: MinIO's CloneNonTransition never emits del-marker or abort-mpu fields, so treating del_marker_expiration as traveling expiry let a MinIO broadcast delete this site's del-marker-only rules. Both fields are now site-local on every edge: stripped from outbound subsets and inbound rules, restored from the local side on same-id merges, and never a deletion criterion. Receiver-side validation of merged configs (object-lock / tier constraints, MinIO runs finalLcCfg.Validate) remains a follow-up. * fix(site-replication): close the second ILM review round - Missed-delete repair: a deleted expiry state now travels through bootstrap/repair as an explicit timestamped lc-config delete item (lifecycle_expiry_statement distinguishes deletion — whole-config write time advanced past the created backfill — from never-configured buckets and from transition-only configs without an expiry axis, which say nothing). A peer that missed the live delete converges on repair; the receiver's staleness guard protects newer peer state. - Strict tombstone recognition: only a well-delimited zero-rule <LifecycleConfiguration> document maps to delete semantics; truncated or foreign payloads that fail the strict deserializer are rejected instead of being treated as a delete that erases local expiry rules. - Staleness fallback axis narrowed: the whole-config write time is used only for deleted or legacy-with-expiry state. A present transition-only config without an expiry axis compares at epoch — its whole-config time moves on transition edits and must not shadow or block independent peer expiry updates and same-timestamp repairs. * fix(site-replication): validate tombstone children structurally Second review round: a well-delimited root could still smuggle malformed content — e.g. <LifecycleConfiguration><ExpiryUpdatedAt> </LifecycleConfiguration> passed the no-<Rule check and was applied as a delete. The tombstone body must now be a sequence of well-formed simple children (matching open/close or self-closing, no nested markup, no stray text, none named Rule); anything else surfaces InvalidRequest. Malformed-child cases pinned in the recognition test. * fix(site-replication): serialize lifecycle merges --------- Co-authored-by: overtrue <anzhengchao@gmail.com>
RustFS ECStore - Erasure Coding Storage
High-performance erasure coding storage engine for RustFS distributed object storage
📖 Documentation
· 🐛 Bug Reports
· 💬 Discussions
📖 Overview
RustFS ECStore provides erasure coding storage capabilities for the RustFS distributed object storage system. For the complete RustFS experience, please visit the main RustFS repository.
✨ Features
- Reed-Solomon erasure coding implementation
- Configurable redundancy levels (N+K schemes)
- Automatic data healing and reconstruction
- Multi-drive support with intelligent placement
- Parallel encoding/decoding for performance
- Efficient disk space utilization
📚 Documentation
For comprehensive documentation, examples, and usage guides, please visit the main RustFS repository.
📄 License
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
Copyright 2024 RustFS Team
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
RustFS is a trademark of RustFS, Inc.
All other trademarks are the property of their respective owners.
Made with ❤️ by the RustFS Storage Team