mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-16 18:08:21 +00:00
唐小鸭
c1f66969d7
* test(site-replication): pin ILM expiry merge contract for incoming lc-config Red-light evidence for backlog#1675 P1-1: the lc-config receiver overwrites the whole local lifecycle config with whatever the peer sends (and deletes it wholesale on peer delete), so an expiry-only document erases the receiver's local tier/transition rules, and peer transition rules get installed across sites. The new tests pin the MinIO mergeWithCurrentLCConfig semantics plus RustFS hardening: - incoming expiry documents merge with (never replace) local rules - local transition sides are authoritative for same-id rules - incoming transition fields are discarded at the trust boundary - dropped expiry rules strip the expiry side but keep transitions; pure-expiry rules are removed - delete merges with the empty set instead of dropping the config - disabled rules survive; abort-mpu-only rules stay site-local - deterministic order (idempotent re-delivery) and expiry_updated_at stamping for the staleness axis All fail against the current overwrite implementation (identity extraction of merge_incoming_lifecycle_config). * fix(site-replication): merge incoming ILM expiry documents instead of overwriting The lc-config receiver replaced the whole local lifecycle config with the peer's document (and deleted it wholesale on peer delete), so an expiry-only update erased the receiver's local tier/transition rules, and a peer's transition rules were installed across sites (backlog#1675 P1-1). Receiver (apply_bucket_meta_item): - lc-config now merges via merge_incoming_lifecycle_config, mirroring MinIO's mergeWithCurrentLCConfig with a trust-boundary hardening: incoming transition fields are discarded outright; the local transition side of a same-id rule is authoritative. A peer delete merges with the empty set — pure-expiry rules go away, transition rules survive with their expiry side cleared, and only an empty result deletes the config file. - Staleness moves to the expiry axis (config.expiry_updated_at): lifecycle_config_updated_at also moves on local transition-only edits, which shadowed newer peer expiry updates. - Receiver-side replicateILMExpiry gate, symmetric with the sender hook (previously any peer could install expiry rules while the option was off). - Rule order is deterministic (local order, incoming-new appended), so re-delivering the same document is byte-stable and does not rewrite bucket metadata per broadcast. Sender: - Both admin choke points — the bucket-meta hook and the SRInfo bucket entry feeding bootstrap/repair and consistency views — now emit only the expiry subset (transition fields stripped, non-expiry rules dropped). MinIO receivers install incoming rules verbatim, so transition rules must never leave the site. An unparseable local config is forwarded unfiltered rather than degraded to a delete. Not covered here (follow-up): a two-site e2e with a real tier backend to exercise transition-rule preservation end to end; receiver-side validate_transition_tier for merged configs. * fix(site-replication): close ILM merge review findings Adversarial review of the lc-config merge surfaced four real defects, all fixed here: - Deletion tombstone regression: with the staleness axis moved to the in-config expiry_updated_at, a deleted lifecycle config fell back to UNIX_EPOCH and any delayed stale broadcast could resurrect deleted expiry rules. The axis now falls back to the whole-config write time (which survives deletion in bucket metadata as the deletion's lower bound), also covering legacy configs that predate the axis field. - MinIO zero-rule documents: MinIO's delete tombstone / transition-only state marshals a lifecycle document with no <Rule>, which the strict s3s deserializer rejects — the receiver now recognizes it as the 'no expiry rules here' statement (delete semantics) instead of erroring on every MinIO heal pass. - Inflated expiry axis at the sender: PutBucketLifecycle stamped expiry_updated_at unconditionally, so a transition-only edit advanced the axis and let this site's stale expiry subset shadow and roll back newer peer expiry edits fleet-wide. The stamp is now conditional (expiry subset present before or after the edit, MinIO parity), the hook item travels with the config's expiry axis (UNIX_EPOCH when the site has none), and the SRInfo bucket entry feeds bootstrap/repair the same axis instead of the whole-config write time. - Del-marker parity: MinIO's CloneNonTransition never emits del-marker or abort-mpu fields, so treating del_marker_expiration as traveling expiry let a MinIO broadcast delete this site's del-marker-only rules. Both fields are now site-local on every edge: stripped from outbound subsets and inbound rules, restored from the local side on same-id merges, and never a deletion criterion. Receiver-side validation of merged configs (object-lock / tier constraints, MinIO runs finalLcCfg.Validate) remains a follow-up. * fix(site-replication): close the second ILM review round - Missed-delete repair: a deleted expiry state now travels through bootstrap/repair as an explicit timestamped lc-config delete item (lifecycle_expiry_statement distinguishes deletion — whole-config write time advanced past the created backfill — from never-configured buckets and from transition-only configs without an expiry axis, which say nothing). A peer that missed the live delete converges on repair; the receiver's staleness guard protects newer peer state. - Strict tombstone recognition: only a well-delimited zero-rule <LifecycleConfiguration> document maps to delete semantics; truncated or foreign payloads that fail the strict deserializer are rejected instead of being treated as a delete that erases local expiry rules. - Staleness fallback axis narrowed: the whole-config write time is used only for deleted or legacy-with-expiry state. A present transition-only config without an expiry axis compares at epoch — its whole-config time moves on transition edits and must not shadow or block independent peer expiry updates and same-timestamp repairs. * fix(site-replication): validate tombstone children structurally Second review round: a well-delimited root could still smuggle malformed content — e.g. <LifecycleConfiguration><ExpiryUpdatedAt> </LifecycleConfiguration> passed the no-<Rule check and was applied as a delete. The tombstone body must now be a sequence of well-formed simple children (matching open/close or self-closing, no nested markup, no stray text, none named Rule); anything else surfaces InvalidRequest. Malformed-child cases pinned in the recognition test. * fix(site-replication): serialize lifecycle merges --------- Co-authored-by: overtrue <anzhengchao@gmail.com>
RustFS is a high-performance distributed object storage software built using Rust
Getting Started · Docs · Bug reports · Discussions
English | Simplified Chinese
RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. Along with MinIO, it shares a range of advantages such as simplicity, broad S3 API compatibility for supported features, open-source nature, support for data lakes, AI, and big data. Furthermore, it has a better and more user-friendly open-source license in comparison to other storage systems, being constructed under the Apache license. As Rust serves as its foundation, RustFS provides faster speed and safer distributed features for high-performance object storage.
Features
- High Performance: Built with Rust, ensuring speed and efficiency.
- Distributed Architecture: Scalable and fault-tolerant design for large-scale deployments.
- S3 Compatibility: Integration with common S3-compatible applications; current coverage is tracked in the S3 compatibility matrix.
- Data Lake Support: Optimized for big data and AI workloads.
- Open Source: Licensed under Apache 2.0, encouraging community contributions and transparency.
- User-Friendly: Designed with simplicity in mind, making it easy to deploy and manage.
RustFS vs MinIO
Stress test server parameters
| Type | parameter | Remark |
|---|---|---|
| CPU | 2 Core | Intel Xeon(Sapphire Rapids) Platinum 8475B , 2.7/3.2 GHz |
| Memory | 4GB | |
| Network | 15Gbp | |
| Driver | 40GB x 4 | IOPS 3800 / Driver |
https://github.com/user-attachments/assets/2e4979b5-260c-4f2c-ac12-c87fd558072a
RustFS vs Other object storage
| RustFS | Other object storage |
|---|---|
| Powerful Console | Simple and useless Console |
| Developed based on Rust language, memory is safer | Developed in Go or C, with potential issues like memory GC/leaks |
| Does not report logs to third-party countries | Reporting logs to other third countries may violate national security laws |
| Licensed under Apache, more business-friendly | AGPL V3 License and other License, polluted open source and License traps, infringement of intellectual property rights |
| S3-compatible core, with coverage tracked in the compatibility matrix | Variable S3 support and local cloud vendor coverage |
| Rust-based development, strong support for secure and innovative devices | Poor support for edge gateways and secure innovative devices |
| Stable commercial prices, free community support | High pricing, with costs up to $250,000 for 1PiB |
| No risk | Intellectual property risks and risks of prohibited uses |
Quickstart
To get started with RustFS, follow these steps:
-
One-click installation script (Option 1)
curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh -
Docker Quick Start (Option 2)
# Docker Hub (recommended)
docker run -d -p 9000:9000 -v /data:/data rustfs/rustfs:latest
# Alternative using Podman
podman run -d -p 9000:9000 -v /data:/data rustfs/rustfs:latest
- Access the Console: Open your web browser and navigate to
http://localhost:9001to access the RustFS console, default username and password isrustfsadmin. - Create a Bucket: Use the console to create a new bucket for your objects.
- Upload Objects: You can upload files directly through the console or use S3-compatible APIs to interact with your RustFS instance.
Documentation
For detailed documentation, including configuration options, API references, and advanced usage, please visit our Documentation.
Getting Help
If you have any questions or need assistance, you can:
- Check the FAQ for common issues and solutions.
- Join our GitHub Discussions to ask questions and share your experiences.
- Open an issue on our GitHub Issues page for bug reports or feature requests.
Links
- Documentation - The manual you should read
- Changelog - What we broke and fixed
- GitHub Discussions - Where the community lives
Contact
- Bugs: GitHub Issues
- Business: hello@rustfs.com
- Jobs: jobs@rustfs.com
- General Discussion: GitHub Discussions
- Contributing: CONTRIBUTING.md
Contributors
RustFS is a community-driven project, and we appreciate all contributions. Check out the Contributors page to see the amazing people who have helped make RustFS better.
License
RustFS is a trademark of RustFS, Inc. All other trademarks are the property of their respective owners.