mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-27 16:48:58 +00:00
abdullahnah92
cb37a64a81
* fix(site-repl): clamp replication_cfg_mismatch to owning deployments and add resync status branch
Fix 3: replication_cfg_mismatch was set on ALL deployments for a bucket when
any replication-config discrepancy existed, even on deployments that simply have
no config. mc computes "in sync" as max_buckets minus per-deployment mismatch
entries; with N deployments all flagged for 1 bucket, the result underflows to
1-N = -1. Now only deployments that own a replication config are flagged.
Fix 4: SiteReplicationResyncOpHandler accepted "start" and "cancel" but returned
an empty body (and later a parse error on the client) for any other operation
string. Add a SITE_REPL_RESYNC_STATUS="status" arm that returns the stored
SRResyncOpStatus or an explicit {"status":"not-found"} object so mc never sees
an unexpected end of JSON input.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(site-repl): derive sync_state from reachability and replication rule completeness
PeerInfo.sync_state was set to SyncStatus::Unknown at every construction site
and never updated from real signals. build_status_info now tracks which peers
were reachable during the metainfo fetch phase and, after all bucket stats are
merged, derives sync_state as:
- Enable : reachable AND no replication_cfg_mismatch for any bucket
- Disable : reachable BUT at least one bucket has incomplete/missing rules
- Unknown : unreachable (fetch failed)
The "Sync" column in mc admin replicate status will now reflect actual state
rather than always showing Unknown.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(site-repl): add SRRotateServiceAccountHandler for split-brain svc-acct repair
When site-replicator-0 gets desynced (different secret on different peers), calls
via the old service account return 403, blocking remove and replicate operations
with no in-band recovery path.
SiteReplicationRemoveHandler already purges local state unconditionally before
sending peer notifications, so force-remove works locally even when peers reject
the 403. The new POST /v3/site-replication/rotate-svc-acct endpoint provides the
missing recovery path: it generates a fresh service-account secret, applies it
locally, and pushes a peer/join to every member. Each peer's SRPeerJoinHandler
accepts the join idempotently (update if exists, create otherwise), repairing the
desynced credential without a full teardown + restart.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(site-repl): back-fill pre-existing buckets and objects on replicate add
SiteReplicationAddHandler and SRPeerJoinHandler both called persist_site_replication_state
and returned success without doing anything for buckets that existed before the
sites were linked. Only buckets created after the link (via site_replication_make_bucket_hook)
ever replicated.
Introduce backfill_existing_buckets_after_add which, after persisting the new
state, iterates every local bucket and:
1. Ensures versioning is enabled (required by replication).
2. Reconciles bucket targets so a target entry exists for each remote peer.
3. Reconciles the replication config so a rule pointing to each peer is present.
4. Broadcasts a make-bucket-hook to peers (idempotent) so they create the bucket.
5. Kicks start_site_bucket_resync toward every remote peer so pre-existing
objects travel across.
Errors per bucket are logged but never abort the overall add — manual resync
remains available as a fallback. Both the initiator and the receiving (join) side
run the backfill so convergence happens regardless of which side held data first.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(site-repl): reconcile replication config to restore bidirectional replication
Root cause: ensure_site_replication_bucket_replication_config bailed with Ok(())
the moment any replication config was found on the bucket. When bucket B was
propagated to site-2 via the make-with-versioning bucket-op, site-2's
configure-replication step loaded the freshly-written config and immediately
returned, never adding the reverse-direction rule pointing back to site-1. Result:
objects uploaded to site-2 failed with "replication head_object fallback failed
... service error" because no rule targeted the originating site.
Fix: drop the early-return. Instead load the existing rules, build the full
desired config via build_site_replication_config, and MERGE — adding only the
rules that are absent (identified by their "site-repl-<deployment_id>" id). Re-
number priorities after each merge to avoid conflicts. Existing non-site-repl
rules are preserved. The write is skipped entirely when all desired rules are
already present, so repeat calls remain cheap.
Together with Fix 1 (backfill), any file written to ANY member now replicates to
ALL members. The offline-and-recover case also benefits: when a peer returns,
the per-bucket rules are complete and the resyncer can catch up.
Also register the new rotate-svc-acct route in route_policy and
route_registration_test so the route inventory assertions stay green.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(replication): remove broken resync worker-signal gate
The resync_bucket task opened a new broadcast receiver via
worker_rx.resubscribe(), which positions the receiver at the current
write-head of the ring buffer — past all 10 bootstrap signals written
in ReplicationResyncer::new(). Every spawned resync task therefore
blocked on recv() forever, making `mc admin replicate resync start`
report "started" while no objects ever moved.
Remove the dead wait entirely. Each resync_bucket call is already
spawned on-demand (tokio::spawn in start_bucket_resync / load_resync),
so no additional gate is needed. The per-object concurrency limit is
already enforced by the inner mpsc worker channels (line ~877). Also
remove the now-dead worker_tx/worker_rx fields, bootstrap loop, and
signal-send in resync_bucket_mark_status.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* docs(site-repl): document rotate-svc-acct idempotency and partial-failure behavior
* style: cargo fmt and remove redundant clones
* fix(site-repl): preserve object-lock state when back-filling existing buckets
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: houseme <housemecn@gmail.com>
RustFS ECStore - Erasure Coding Storage
High-performance erasure coding storage engine for RustFS distributed object storage
📖 Documentation
· 🐛 Bug Reports
· 💬 Discussions
📖 Overview
RustFS ECStore provides erasure coding storage capabilities for the RustFS distributed object storage system. For the complete RustFS experience, please visit the main RustFS repository.
✨ Features
- Reed-Solomon erasure coding implementation
- Configurable redundancy levels (N+K schemes)
- Automatic data healing and reconstruction
- Multi-drive support with intelligent placement
- Parallel encoding/decoding for performance
- Efficient disk space utilization
📚 Documentation
For comprehensive documentation, examples, and usage guides, please visit the main RustFS repository.
📄 License
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
Copyright 2024 RustFS Team
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
RustFS is a trademark of RustFS, Inc.
All other trademarks are the property of their respective owners.
Made with ❤️ by the RustFS Storage Team