rustfs/docs/operations/site-replication-operations.md

Site Replication Operations

Use this when: operating a site-replication deployment, diagnosing a peer outage or incomplete topology change, changing a site's address, pairing sites that already contain data, or planning an upgrade.

Source of truth: rustfs/src/admin/handlers/site_replication.rs, rustfs/src/site_replication/, and the bucket-replication worker under crates/ecstore/src/bucket/replication/.

Site replication combines two different convergence paths:

• the control plane replicates buckets, bucket metadata, IAM, and topology;
• ordinary bucket replication moves object versions and delete operations.

An enabled: true response only says that a site has more than one configured peer. It does not prove that every peer is reachable or caught up. Always read pendingOperation, retryStats, PeerErrors, and Metrics as well.

Routine checks

Run these commands from an admin workstation with one alias per site:

mc admin replicate info site-a
mc admin replicate status site-a

Check more than one site. A partition can leave each side with a different but locally valid view.

replicate info is the compact control-plane view:

Field	Interpretation
enabled	More than one site is configured; this is not a health verdict.
sites	The locally persisted topology. Compare deployment IDs and endpoints on every site.
retryStats.pending	Collapsed peer deliveries waiting to be retried.
retryStats.failed	Deliveries that crossed the escalation threshold and require attention.
retryStats.lastError	A redacted summary of the most recent delivery failure.
pendingOperation	A durable multi-step topology operation described below. Absence is the healthy steady state.

replicate status adds detailed convergence state:

Field	Interpretation
Sites / PeerStates	Configured peers and derived reachability/configuration state.
PeerErrors	A peer could not be queried. Its detailed counters may be absent; do not read zeros as success.
BucketStats	Per-bucket presence and versioning, replication, lifecycle, Object Lock, and metadata mismatches.
PolicyStats, UserStats, GroupStats	IAM inventory mismatches.
RetryStats	Durable control-plane retry backlog and escalation count.
Metrics.replMetrics	Per-destination online state, downtime, replicated counts/bytes, and failed totals/windows.
Metrics.queued / Metrics.inProgress	Object work waiting or active on the responding node.
Metrics.errors	Node-level object-replication failures. When only queue statistics are available, RustFS synthesizes a node entry and preserves this counter rather than reporting zero.
Metrics.retries	Redeliveries. Always zero today: a failed object is not retried by an event, it waits for the scanner pass described below. Read errors instead.

Healthy means: the same topology is visible on all sites, no pending operation, no peer error, no failed retry escalation, required bucket/IAM state is in sync, and queue/error counters are stable or falling. Counters are cumulative; alert on their rate and on a backlog that does not drain, not merely on a non-zero total.

Pending operations and recovery

pendingOperation contains operation, an opaque id, pendingPeers, and ackedPeers. Do not edit the site-replication state object by hand. The marker is the crash-recovery journal and removing it can make a partially applied operation look complete.

The heavyweight reconciler runs once at startup and every 600 seconds. The lightweight retry drain runs every 30 seconds. A restart is therefore a valid way to cause an immediate heavyweight pass after the underlying fault has been fixed, but it is not a substitute for fixing connectivity, credentials, TLS, or the remote endpoint.

remove

The original topology and each peer acknowledgement are persisted before the operation finalizes. While peers remain in pendingPeers, restore access to them and wait for reconciliation. If a peer is permanently gone, a new remove request may remove all currently active unacknowledged peers; RustFS permits that request and then finalizes against the remaining topology. Removing the local site or all sites is also an explicit completion path.

Do not re-add a site merely to hide this marker. First compare the topology on all reachable peers. If the same operation ID makes no progress for more than one heavyweight interval, collect PeerErrors, RetryStats, and the site-replication logs before retrying the remove.

rotate-svc-acct

Service-account rotation keeps the candidate secrets and peer acknowledgements until every current remote peer accepts the rotation. Restore the failing peer and allow the reconciler to resume it. Do not manually delete either candidate credential during this window: doing so can remove the only credential that a not-yet-acknowledged peer accepts.

After the marker clears, verify replicate status from every site, then retire any separately retained old credential material according to local policy.

endpoint-refresh

An endpoint, CA, or TLS-verification edit first refreshes the replication target on every active peer and records acknowledgements. On startup and every heavyweight pass, RustFS probes peer capability, uses the endpoint-refresh API when supported (or the legacy peer-edit fallback), refreshes local bucket targets, and commits the edit only after every still-active peer acknowledges.

If this marker is stuck:

1. Confirm that the proposed endpoint and CA are correct and reachable from every site, not only from the admin workstation.
2. Restore the site-replication service account and TLS trust path.
3. Wait for one 600-second pass or restart one healthy node to trigger the startup pass.
4. Re-run the identical edit only if the operation remains visible; a different endpoint edit is rejected while the existing refresh is pending. The journal pins the edit's payload, so a re-run without --replicate-ilm-expiry keeps the value the first attempt recorded, and a re-run asking for a different value is rejected. Finish or remove the pending refresh before changing it.

A peer removed from the topology no longer blocks completion. A remove request is accepted when it removes every active unacknowledged peer.

While this marker is present, control-plane retry replay to the other peers keeps running, but bucket wiring reconciliation waits: it rewrites the same targets the refresh is changing. Expect bucket-level drift on this site to persist until the refresh settles.

Outage recovery and convergence time

Control-plane retry begins on the 30-second drain, while heavyweight snapshots, pending topology operations, and bucket wiring are revisited on the 600-second pass. Object MRF entries are persisted every 10 seconds by default and target health is probed every 5 seconds. These are scheduling bounds, not delivery SLAs: network timeouts and the amount of queued work add to them.

Objects that must be rediscovered by the scanner have this conservative upper bound before discovery:

RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES
    × max(RUSTFS_SCANNER_CYCLE, actual duration of one scanner cycle)

The defaults re-descend a compacted directory every 16 cycles. A practical production starting point for a tighter recovery objective is RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES=4; 1 forces re-descent every cycle. Measure the additional disk and metadata load before lowering it further or tuning the scanner cadence. For an immediate operator-driven recovery, start a site resync with rc admin replicate resync start and monitor it with rc admin replicate resync status (see Changing a site's address for the arguments; mc admin replicate resync does not work against RustFS). Transfer time after discovery remains proportional to backlog size, bandwidth, worker capacity, and target latency. Use queue depth and the rate of Metrics.errors rather than the formula alone to decide whether convergence is progressing.

Changing a site's address

Every site stores each peer's endpoint, its own included. The local entry is recorded once, when the site joins, from the address the setup request used to reach it. Nothing re-derives it: after a server's IP or port changes, every site keeps the old endpoint, and the bucket targets the other sites derive from it keep pointing there. Replication toward the moved site stops; replication from it keeps working.

Reads can hide the break. A HEAD or GET on the moved site may still succeed because it proxies to a peer that holds the object. Confirm with replicate status (Metrics.replMetrics online state and failed counters) or by listing a recently written key on the moved site.

An endpoint edit must reach every other peer at its stored endpoint before it changes anything. The edited site itself is tried at its stored endpoint and then at the new one, so only that one site may be unreachable at its old address. When another peer cannot be reached, the edit fails with probe endpoint target refresh on peer ... failed and leaves every site's state unchanged.

Planned change: one site at a time

The commands below use the RustFS CLI rc (admin replicate edit and admin replicate resync exist since rc 0.1.30). site-a is the moved site and site-b stands for each other site. An alias for the moved site must use its new address: rc alias set connects to the server before saving the alias. --site takes the exact deployment ID or a site name that is unique in rc admin replicate info, and every change runs only with --yes.

Repeat for each site:

1. Change the server address and start the site on the new address.

2. Update its endpoint from any site, including the moved site itself:

   rc alias set site-a https://<new-address>:9000 <ACCESS_KEY> <SECRET_KEY>
   rc admin replicate info site-a
   rc admin replicate edit site-a --site <ID-or-name> --endpoint https://<new-address>:9000 --yes
   

   In the Console, use Edit Site on that site's row in Site Replication. Consoles that predate local-site editing only offer it on remote rows; use a peer site's Console or rc.

   mc admin replicate update <alias> --deployment-id <ID> --endpoint <URL> also changes the endpoint, but it sends no site name, and RustFS then renames the site after the new endpoint's host. Restore the name afterwards with rc admin replicate edit <alias> --site <ID> --name <name> --yes.

3. Check rc --json admin replicate info on every site: all of them list the new endpoint and none reports pendingOperation.

4. From every other site, resync toward the moved site to recover the writes that missed it while its stored endpoint was stale:

   rc admin replicate resync start site-b --site <ID-or-name-of-site-a> --yes
   rc admin replicate resync status site-b --site <ID-or-name-of-site-a>
   

   The alias is the site that sends the objects; rc refuses a resync toward the alias's own site. Consoles with local-site editing also offer Start Resync in the dialog shown after Edit Site saves a new peer endpoint. mc admin replicate resync cannot be used: it reads the target's deployment ID from admin info, which RustFS nests under info, so mc reports that the alias is not part of the replication cluster.

Only then move the next site.

Every site already moved

When all sites changed address together, each edit needs the other peers' old addresses, so every edit fails. Make the old addresses reachable again for the duration of the repair, for example by keeping the old IP as a secondary address on each host or by forwarding the old address and port to the new one. Then, from any one site, update every site's endpoint (its own included) with rc admin replicate edit, confirm that all sites agree, remove the temporary addresses, and resync from each site toward every other site.

Do not change a site-replication bucket target directly with bucket-level remote-target edits (mc admin bucket remote edit or the Console's bucket replication form). Those targets are derived from the site topology; RustFS rejects a connection change on them, and the endpoint must be updated through the site edit above.

Pairing sites that already contain data

When more than one requested site is non-empty, preflight considers each bucket name held by more than one site:

• versioning must be Enabled on every site holding the shared bucket;
• Object Lock enablement must be identical on every holder.

A bucket present on only one site is safe: post-add backfill creates it on the other peers. A shared unversioned bucket is rejected because merging can overwrite the only copy of an object. An Object Lock mismatch is rejected because lock enablement cannot be changed after bucket creation and convergence could otherwise strip a WORM guarantee.

If preflight rejects the pair, keep the authoritative copy, delete the conflicting bucket (or its contents) from all other sites, run replicate add again, and then start replicate resync from the surviving site. Back up and validate the authoritative data before deleting anything.

IAM convergence and repair boundary

Ordinary IAM changes are delivered to each peer. A successful bulk IAM import also schedules one collapsed full-IAM snapshot per remote peer. A failed IAM deletion is replayed before that snapshot so the snapshot cannot re-create a principal or grant that was already revoked.

The safety state has two bounds:

• deletion high-water marks are retained for 30 days;
• deletion replay bodies are capped at 256 distinct entities per peer.

Repeated deletion of the same entity replaces its saved body. When the per-peer cap is exceeded or the body cannot be serialized, the retry entry remains escalated rather than pretending the deletion is replayable. An item from an older sender without a source timestamp cannot install the 30-day high-water mark, so verify it explicitly after a prolonged split. A successful drain clears replay bodies; removing the peer prunes its bodies. For an escalated IAM retry, use the site-replication repair workflow for the affected peer and IAM family, then verify users, service accounts, groups, policies, and mappings on both sides. Repair is the operator's explicit accountability transfer and clears the saved deletion bodies only after the IAM repair succeeds.

A group's status converges in one direction. An explicit disable is applied everywhere, including through a snapshot, but a membership change never carries an enable - it would otherwise re-enable a group frozen on the receiving site. If a group ended up disabled on one site only, re-enable it there explicitly with mc admin group enable; a snapshot or repair will not do it.

Treat IAM divergence as a security incident: a user deleted on one site can remain usable on an unreachable peer until replay or repair completes. A peer whose IAM entry is escalated does not receive scheduled snapshots either - including the one a bulk import schedules - until the repair settles it.

Encrypted objects

Source form	Replication behavior	Fail-closed condition
SSE-S3	The source decrypts the object; the request sends only AES256 intent; the destination encrypts with its own KMS. Source envelope material never leaves the site.	The destination cannot satisfy the encryption request, or the source metadata is incomplete/unsupported. The replica is FAILED; plaintext is not silently stored.
SSE-KMS	The source decrypts the object; the request sends aws:kms intent without the source-local key ID; the destination selects its own configured KMS key.	Either side cannot decrypt/encrypt, or the metadata mixes incompatible encryption evidence.
SSE-C	Stored ciphertext and the required SSE-C replication transport metadata pass through. RustFS verifies target evidence before accepting the replica.	The target does not echo the customer-algorithm evidence, required material/layout is absent, or the metadata is ambiguous.

Unknown MinIO/RustFS encryption markers are never forwarded as ordinary user metadata. They fail replication so an operator must migrate or repair the object with a supported format.

Rolling upgrades and rollback

Keep every node in one site on the same version whenever possible. Upgrade all nodes of one site consecutively, verify its startup reconciliation and status, then move to the next site. Do not intentionally leave a site mixed-version: admin requests can land on different nodes, and an older node may not resume a new pending-operation shape or expose its health fields.

Current state additions are optional and defaulted, so older readers ignore them. The target-version ledger is stored as dual-prefixed internal object metadata and is also ignored by older readers; rollback does not corrupt the object format, but older code loses the assigned-version routing improvement.

Before rolling back across the fix that retains the data directory of a version awaiting purge replication (rustfs/rustfs#7307), ensure no version purge is pending. Older code can free that retained version's data directory before the remote purge is acknowledged, leaving unreadable metadata and blocking bucket deletion. Drain or repair replication and take a metadata/data backup first.

Runtime knobs

These values are read when the owning background task starts. Restart the server after changing them. The millisecond intervals have a 10 ms floor; invalid values fall back to the default with a warning.

Variable	Default	Effect
RUSTFS_REPL_HEALTH_CHECK_INTERVAL_MS	5000	Remote-target health probe interval. Lowering it increases outbound probes.
RUSTFS_REPL_MRF_FLUSH_INTERVAL_MS	10000	Maximum periodic interval between MRF persistence flushes; 1,000 new entries also trigger a flush.
RUSTFS_REPL_RESYNC_POLL_MAX_MS	60000	Upper bound for randomized resync retry-poll sleep.
RUSTFS_REPL_RESYNC_MAX_JOBS	2	Concurrent resync jobs; values are bounded to 1..=32.

Transport-specific controls and target behavior are documented in Replication outbound transport. Validate a new destination with Replication target check, and read Replication object size limits before moving large objects.