fix(site-replication): bidirectional sync, pre-existing data back-fill, and related fixes (#3401)

* fix(site-repl): clamp replication_cfg_mismatch to owning deployments and add resync status branch

Fix 3: replication_cfg_mismatch was set on ALL deployments for a bucket when
any replication-config discrepancy existed, even on deployments that simply have
no config. mc computes "in sync" as max_buckets minus per-deployment mismatch
entries; with N deployments all flagged for 1 bucket, the result underflows to
1-N = -1. Now only deployments that own a replication config are flagged.

Fix 4: SiteReplicationResyncOpHandler accepted "start" and "cancel" but returned
an empty body (and later a parse error on the client) for any other operation
string. Add a SITE_REPL_RESYNC_STATUS="status" arm that returns the stored
SRResyncOpStatus or an explicit {"status":"not-found"} object so mc never sees
an unexpected end of JSON input.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(site-repl): derive sync_state from reachability and replication rule completeness

PeerInfo.sync_state was set to SyncStatus::Unknown at every construction site
and never updated from real signals. build_status_info now tracks which peers
were reachable during the metainfo fetch phase and, after all bucket stats are
merged, derives sync_state as:
  - Enable  : reachable AND no replication_cfg_mismatch for any bucket
  - Disable : reachable BUT at least one bucket has incomplete/missing rules
  - Unknown : unreachable (fetch failed)

The "Sync" column in mc admin replicate status will now reflect actual state
rather than always showing Unknown.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(site-repl): add SRRotateServiceAccountHandler for split-brain svc-acct repair

When site-replicator-0 gets desynced (different secret on different peers), calls
via the old service account return 403, blocking remove and replicate operations
with no in-band recovery path.

SiteReplicationRemoveHandler already purges local state unconditionally before
sending peer notifications, so force-remove works locally even when peers reject
the 403. The new POST /v3/site-replication/rotate-svc-acct endpoint provides the
missing recovery path: it generates a fresh service-account secret, applies it
locally, and pushes a peer/join to every member. Each peer's SRPeerJoinHandler
accepts the join idempotently (update if exists, create otherwise), repairing the
desynced credential without a full teardown + restart.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(site-repl): back-fill pre-existing buckets and objects on replicate add

SiteReplicationAddHandler and SRPeerJoinHandler both called persist_site_replication_state
and returned success without doing anything for buckets that existed before the
sites were linked. Only buckets created after the link (via site_replication_make_bucket_hook)
ever replicated.

Introduce backfill_existing_buckets_after_add which, after persisting the new
state, iterates every local bucket and:
  1. Ensures versioning is enabled (required by replication).
  2. Reconciles bucket targets so a target entry exists for each remote peer.
  3. Reconciles the replication config so a rule pointing to each peer is present.
  4. Broadcasts a make-bucket-hook to peers (idempotent) so they create the bucket.
  5. Kicks start_site_bucket_resync toward every remote peer so pre-existing
     objects travel across.

Errors per bucket are logged but never abort the overall add — manual resync
remains available as a fallback. Both the initiator and the receiving (join) side
run the backfill so convergence happens regardless of which side held data first.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(site-repl): reconcile replication config to restore bidirectional replication

Root cause: ensure_site_replication_bucket_replication_config bailed with Ok(())
the moment any replication config was found on the bucket. When bucket B was
propagated to site-2 via the make-with-versioning bucket-op, site-2's
configure-replication step loaded the freshly-written config and immediately
returned, never adding the reverse-direction rule pointing back to site-1. Result:
objects uploaded to site-2 failed with "replication head_object fallback failed
... service error" because no rule targeted the originating site.

Fix: drop the early-return. Instead load the existing rules, build the full
desired config via build_site_replication_config, and MERGE — adding only the
rules that are absent (identified by their "site-repl-<deployment_id>" id). Re-
number priorities after each merge to avoid conflicts. Existing non-site-repl
rules are preserved. The write is skipped entirely when all desired rules are
already present, so repeat calls remain cheap.

Together with Fix 1 (backfill), any file written to ANY member now replicates to
ALL members. The offline-and-recover case also benefits: when a peer returns,
the per-bucket rules are complete and the resyncer can catch up.

Also register the new rotate-svc-acct route in route_policy and
route_registration_test so the route inventory assertions stay green.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(replication): remove broken resync worker-signal gate

The resync_bucket task opened a new broadcast receiver via
worker_rx.resubscribe(), which positions the receiver at the current
write-head of the ring buffer — past all 10 bootstrap signals written
in ReplicationResyncer::new().  Every spawned resync task therefore
blocked on recv() forever, making `mc admin replicate resync start`
report "started" while no objects ever moved.

Remove the dead wait entirely.  Each resync_bucket call is already
spawned on-demand (tokio::spawn in start_bucket_resync / load_resync),
so no additional gate is needed.  The per-object concurrency limit is
already enforced by the inner mpsc worker channels (line ~877).  Also
remove the now-dead worker_tx/worker_rx fields, bootstrap loop, and
signal-send in resync_bucket_mark_status.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs(site-repl): document rotate-svc-acct idempotency and partial-failure behavior

* style: cargo fmt and remove redundant clones

* fix(site-repl): preserve object-lock state when back-filling existing buckets

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: houseme <housemecn@gmail.com>

crates/ecstore/src/bucket/replication/replication_resyncer.rs

@@ -98,7 +98,6 @@ const EVENT_RESYNC_OBJECT_PROCESSED: &str = "replication_resync_object_processed
 const EVENT_RESYNC_RUNTIME_SKIPPED: &str = "replication_resync_runtime_skipped";
 const EVENT_REPLICATION_DELETE_SKIPPED: &str = "replication_delete_skipped";
 const EVENT_REPLICATION_FORCE_DELETE_SKIPPED: &str = "replication_force_delete_skipped";
-const EVENT_RESYNC_WORKER_SIGNAL_FAILED: &str = "replication_resync_worker_signal_failed";
 const EVENT_RESYNC_TASK_FAILED: &str = "replication_resync_task_failed";
 const EVENT_RESYNC_TARGET_OPERATION_FAILED: &str = "replication_resync_target_operation_failed";
 const EVENT_RESYNC_RUNTIME_CHANNEL_FAILED: &str = "replication_resync_runtime_channel_failed";
@@ -459,33 +458,14 @@ pub struct ReplicationResyncer {
     pub status_map: Arc<RwLock<HashMap<String, BucketReplicationResyncStatus>>>,
     pub worker_size: usize,
     pub cancel_tokens: Arc<RwLock<HashMap<String, CancellationToken>>>,
-    pub worker_tx: tokio::sync::broadcast::Sender<()>,
-    pub worker_rx: tokio::sync::broadcast::Receiver<()>,
 }
 
 impl ReplicationResyncer {
     pub async fn new() -> Self {
-        let (worker_tx, worker_rx) = tokio::sync::broadcast::channel(RESYNC_WORKER_COUNT);
-
-        for _ in 0..RESYNC_WORKER_COUNT {
-            if let Err(err) = worker_tx.send(()) {
-                error!(
-                    event = EVENT_RESYNC_WORKER_SIGNAL_FAILED,
-                    component = LOG_COMPONENT_ECSTORE,
-                    subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
-                    reason = "worker_bootstrap_signal_failed",
-                    error = %err,
-                    "Failed to signal replication resync worker"
-                );
-            }
-        }
-
         Self {
             status_map: Arc::new(RwLock::new(HashMap::new())),
             worker_size: RESYNC_WORKER_COUNT,
             cancel_tokens: Arc::new(RwLock::new(HashMap::new())),
-            worker_tx,
-            worker_rx,
         }
     }
 
@@ -686,18 +666,6 @@ impl ReplicationResyncer {
                 "Failed to update resync status"
             );
         }
-        if let Err(err) = self.worker_tx.send(()) {
-            error!(
-                event = EVENT_RESYNC_WORKER_SIGNAL_FAILED,
-                component = LOG_COMPONENT_ECSTORE,
-                subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
-                bucket = %opts.bucket,
-                arn = %opts.arn,
-                reason = "worker_release_signal_failed",
-                error = %err,
-                "Failed to signal replication resync worker"
-            );
-        }
         // TODO: Metrics
     }
 
@@ -709,14 +677,18 @@ impl ReplicationResyncer {
         heal: bool,
         opts: ResyncOpts,
     ) {
-        let mut worker_rx = self.worker_rx.resubscribe();
-
-        tokio::select! {
-            _ = cancellation_token.cancelled() => {
-                return;
-            }
-
-            _ = worker_rx.recv() => {}
+        // Check cancellation before starting the scan.
+        // NOTE: the previous design waited here on `worker_rx.resubscribe().recv()` to
+        // throttle concurrent resyncs, but `resubscribe()` positions the new receiver at
+        // the current write-head of the broadcast ring buffer, so all pre-sent bootstrap
+        // signals (written in `ReplicationResyncer::new`) are invisible to it.  Every
+        // spawned task therefore blocked forever, which is why `resync start` reported
+        // "started" yet objects never moved.  Throttling at this level is also incorrect
+        // for broadcast channels (one send unblocks ALL receivers).  The inner
+        // per-object worker pool (mpsc channels, line ~877) already provides the right
+        // concurrency limit.
+        if cancellation_token.is_cancelled() {
+            return;
         }
 
         let cfg = match get_replication_config(&opts.bucket).await {