fix(site-replication): keep reverse direction after config broadcast (#5292)

* fix(site-replication): keep reverse direction after config broadcast

`site-repl-*` rules encode the sender's outbound direction: their
destination ARN names the receiver. `apply_bucket_meta_item` wrote an
incoming rule set verbatim over the receiver's, leaving the receiver with
a rule whose ARN is its own deployment ID. `reconcile_site_replication_bucket_targets`
skips the local peer, so no bucket target can back that ARN and every
object was dropped; the follow-up call reconciled targets only, so
nothing rebuilt the lost reverse rule. Replication went one-directional
after any PutBucketReplication broadcast — the console's Save button,
`mc replicate import`, a metadata import, or `/site-replication/repair`.

Only operator-authored rules now travel between sites; each site owns its
`site-repl-*` rules and rebuilds them from the current peer set.

Four defects kept that invisible or unrecoverable:

- `update_all_targets` discarded target-client build errors silently, and
  `replicate_object` logged the resulting missing-target drop at debug
  while every other failure there logs at error. Both now report.
- `site_replication_rule_complete` never checked that a rule's
  destination named a remote site, so two sites holding identical
  configs — the post-clobber state — passed as in sync.
- `update_service_account` cannot rewrite `parent_user`, and IAM records
  encrypted with a previous root secret decode as "no such account".
  Startup now reconciles the account, reseeding from the secret every
  site-replication bucket target already stores, and refuses the
  delete-then-create sequence when the parent cannot back an account.
  Bucket rules are reconciled too, so an already-broken site heals on
  upgrade.
- A joined site never verified it could reach the initiator, whose
  endpoint is derived from the Host header of the admin request that
  created the topology. The join now probes each peer and reports through
  `initial_sync_error_message`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(site-replication): report unreachable targets and reconcile on a timer

Rule-shape checking cannot see an unreachable peer. A `site-repl-*` rule
can be perfectly formed while the endpoint recorded for its peer is one
this site cannot reach: `update_all_targets` then builds no client and
`replicate_object` drops every object against that ARN, yet the rule set
still reads as correct and the bucket reports in sync.

Each site now reports whether all of its `site-repl-*` rules resolve to a
live target (`SRBucketInfo.replicationTargetsOnline`, read from the
already-resolved client map so the status path stays cheap), and the
status aggregation treats an offline report as a mismatch. The field is
additive and optional: peers that omit it are "unknown", never a fault,
so a mixed-version topology does not flip every bucket to out of sync.

The reconcilers also run on a 10-minute timer instead of at startup only,
so drift is repaired without waiting for a restart. Both are no-ops when
the wiring already matches — the bucket pass compares serialized targets
and the rule set before writing. The tick takes the site-replication
lifecycle lock with a non-blocking try_acquire and skips the round when
an add/remove/endpoint-refresh holds it: those run in phases, and
rebuilding rules between two of them would resurrect what the operation
just tore down.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* refactor(site-replication): invert reconcile dependency to satisfy layers

`startup_services.rs` sits in the infra layer and was calling the
reconcilers in `admin::handlers::site_replication`, which is interface —
a reverse dependency that check_layer_dependencies.sh rejects.

Moving the reconcilers down is not viable in this change: they rest on
the site-replication state core (`SiteReplicationState` alone has 107
in-file uses, `load_site_replication_state` 38, the state lock 33), so
relocating it would move ~2000 lines and ~200 call sites through a
bug-fix PR.

Invert the direction instead. A new infra module owns the contract and
the schedule; the admin layer registers its reconciler from
`register_site_replication_route`, which runs while the admin router is
built — `init_startup_http_servers` awaits that before
`init_startup_runtime_services` reconciles, so the hook is always
installed in time. No logic moves and no baseline entry is added: the
dependency genuinely reverses.

The lifecycle guard now wraps both reconcilers in one round rather than
each separately, closing the window between them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(site-replication): harden reconcile per review feedback

Addresses the automated review on #5292.

Security: secret recovery from bucket targets accepted any target carrying
the `site-replicator-0` access key. Bucket targets are writable by anyone
holding `admin:SetBucketTarget`, so such a principal could plant a secret
and have reconciliation recreate the broadly privileged replication
account with it. A target must now name a peer in the persisted state and
point at that peer's recorded endpoint, disagreeing targets abort the
recovery, and only missing/unreadable-account errors may trigger it at
all — a transient store failure no longer rewrites a live account.

Durability: the repair no longer deletes before creating. A readable
account is rebound in place through a new `parent_user` field on
`UpdateServiceAccountOpts`, gated to `site-replicator-0` under
`allow_site_replicator_account` exactly as the account itself is. The
parent also lives in the session-token claims, and
`prepare_service_account_auth` denies the account when the two disagree,
so both move together.

Availability: the reconcile scheduler no longer requires an inline IAM
bootstrap. Deferred IAM recovers in the background with no callback into
the scheduler, which left a recovered node with self-pointing rules until
the next restart. It now starts unconditionally and returns early while
IAM or the object store are unavailable. Its first pass runs inside the
task, so walking every bucket no longer delays startup.

Correctness: an endpoint refresh commits bucket targets and peer state in
separate steps without holding the lifecycle lock, so a tick landing
between them rewrote targets from the stale endpoint; the reconciler now
also skips while any pending marker is set. Rule repair preserves an
operator-authored `role` and clears only sender-owned site-replication
ARNs, matching the merge path.

Hot path: the per-object missing-target message returns to debug. The
condition is reported once per bucket per reconcile pass instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>

rustfs/src/admin/handlers/service_account.rs

@@ -652,6 +652,7 @@ impl Operation for UpdateServiceAccount {
             description: new_description.clone(),
             expiration: new_expiration,
             session_policy: sp,
+            parent_user: None,
             allow_site_replicator_account: false,
         };
 

rustfs/src/admin/handlers/user.rs

@@ -1198,6 +1198,7 @@ impl Operation for ImportIam {
                                         description: None,
                                         expiration: None,
                                         status: Some(status),
+                                        parent_user: None,
                                         allow_site_replicator_account: false,
                                     },
                                 )

rustfs/src/lib.rs

@@ -90,6 +90,7 @@ pub mod protocols;
 pub mod runtime_capabilities;
 pub(crate) mod runtime_sources;
 pub mod server;
+pub(crate) mod site_replication_reconcile;
 pub(crate) mod startup_audit;
 pub(crate) mod startup_auth;
 pub(crate) mod startup_background;

rustfs/src/site_replication_reconcile.rs

@@ -0,0 +1,81 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Startup and periodic reconciliation of site-replication wiring.
+//!
+//! The reconcilers themselves live in `admin::handlers::site_replication`, next to the
+//! site-replication state they repair. Startup runs in this (infra) layer and must not
+//! reach up into the admin (interface) layer, so the direction is inverted here: this
+//! module owns the schedule and the contract, and the admin layer registers its
+//! implementations into it from `register_site_replication_route`. That registration
+//! happens while the admin router is built, which `init_startup_http_servers` does before
+//! `init_startup_runtime_services` calls in — so the hooks are always installed by the
+//! time startup reconciles.
+
+use std::future::Future;
+use std::pin::Pin;
+use std::sync::OnceLock;
+use std::time::Duration;
+
+use tokio_util::sync::CancellationToken;
+use tracing::warn;
+
+const RECONCILE_INTERVAL: Duration = Duration::from_secs(600);
+
+/// A reconciler reports its own failures; the outcome carries no value because neither
+/// caller can act on one — a site that cannot repair its replication wiring still serves S3.
+type ReconcileHook = fn() -> Pin<Box<dyn Future<Output = ()> + Send>>;
+
+static RECONCILER: OnceLock<ReconcileHook> = OnceLock::new();
+
+/// Install the admin layer's reconciler. Idempotent: a second call is ignored, which keeps
+/// repeated router construction (tests, the embedded server) from panicking.
+pub(crate) fn register_site_replication_reconciler(reconcile: ReconcileHook) {
+    let _ = RECONCILER.set(reconcile);
+}
+
+/// Repair drifted site-replication wiring, immediately and then on a timer.
+///
+/// The first pass runs inside the spawned task rather than on the caller's path: it walks
+/// every bucket, so awaiting it would delay the rest of startup in proportion to bucket
+/// count. It is also spawned unconditionally, including when IAM bootstrap was deferred —
+/// a node that recovers IAM in the background would otherwise keep self-pointing rules
+/// until someone restarted it. The reconciler itself returns early while IAM or the object
+/// store are still unavailable, so an early tick is silent rather than noisy.
+///
+/// Later passes are no-ops once the wiring matches — the bucket pass compares the serialized
+/// targets and rule set before writing — so a healthy deployment reads and writes nothing.
+pub(crate) fn spawn_site_replication_reconcile_task(ctx: CancellationToken) {
+    if RECONCILER.get().is_none() {
+        warn!("site replication reconciler is not registered; periodic reconcile disabled");
+        return;
+    }
+
+    tokio::spawn(async move {
+        let mut ticker = tokio::time::interval(RECONCILE_INTERVAL);
+        ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
+
+        loop {
+            tokio::select! {
+                _ = ctx.cancelled() => break,
+                // The first tick fires immediately, which is the startup repair pass.
+                _ = ticker.tick() => {
+                    if let Some(reconcile) = RECONCILER.get() {
+                        reconcile().await;
+                    }
+                }
+            }
+        }
+    });
+}

rustfs/src/startup_services.rs

@@ -12,6 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+use crate::site_replication_reconcile::spawn_site_replication_reconcile_task;
 use crate::storage_api::startup::services::{ECStore, EndpointServerPools, ServerContextSlot};
 use crate::{
     config::Config,
@@ -79,6 +80,11 @@ pub(crate) async fn init_startup_runtime_services(
 
     let buckets = init_bucket_metadata_runtime(store.clone(), ctx.clone()).await?;
     let iam_bootstrap = init_iam_runtime(store.clone(), ctx.clone(), readiness, state_manager, server_ctx).await?;
+    // Unconditional: deferred IAM recovers in the background and has no callback into this
+    // scheduler, so gating on the inline disposition would leave a recovered node with
+    // self-pointing replication rules until the next restart. The task waits for IAM and
+    // bucket metadata itself, and its first pass runs off this path.
+    spawn_site_replication_reconcile_task(ctx.clone());
     init_auth_integrations().await?;
     init_notification_runtime(endpoint_pools, buckets).await?;
     let enable_scanner = init_background_service_runtime(store.clone()).await?;