Preserve metadata replication operations in the durable MRF and route tagging, retention, and legal-hold updates through the existing full-object replication transport. Keep ACL propagation outside the contract because the current object model has no durable object ACL state.
Refs #1616
Add RAII guards for replication runtime backlog tickets so active worker and queue counters unwind on every terminal path.
Expose node-local MRF pending, dropped, missed, and flush-failure metrics through the bucket replication Prometheus collector while keeping the existing current backlog and durable MRF gauges additive.
Update durable MRF summary maintenance to aggregate incrementally during the persister loop, avoiding repeated full-entry scans on each successful flush.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(site-replication): keep reverse direction after config broadcast
`site-repl-*` rules encode the sender's outbound direction: their
destination ARN names the receiver. `apply_bucket_meta_item` wrote an
incoming rule set verbatim over the receiver's, leaving the receiver with
a rule whose ARN is its own deployment ID. `reconcile_site_replication_bucket_targets`
skips the local peer, so no bucket target can back that ARN and every
object was dropped; the follow-up call reconciled targets only, so
nothing rebuilt the lost reverse rule. Replication went one-directional
after any PutBucketReplication broadcast — the console's Save button,
`mc replicate import`, a metadata import, or `/site-replication/repair`.
Only operator-authored rules now travel between sites; each site owns its
`site-repl-*` rules and rebuilds them from the current peer set.
Four defects kept that invisible or unrecoverable:
- `update_all_targets` discarded target-client build errors silently, and
`replicate_object` logged the resulting missing-target drop at debug
while every other failure there logs at error. Both now report.
- `site_replication_rule_complete` never checked that a rule's
destination named a remote site, so two sites holding identical
configs — the post-clobber state — passed as in sync.
- `update_service_account` cannot rewrite `parent_user`, and IAM records
encrypted with a previous root secret decode as "no such account".
Startup now reconciles the account, reseeding from the secret every
site-replication bucket target already stores, and refuses the
delete-then-create sequence when the parent cannot back an account.
Bucket rules are reconciled too, so an already-broken site heals on
upgrade.
- A joined site never verified it could reach the initiator, whose
endpoint is derived from the Host header of the admin request that
created the topology. The join now probes each peer and reports through
`initial_sync_error_message`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(site-replication): report unreachable targets and reconcile on a timer
Rule-shape checking cannot see an unreachable peer. A `site-repl-*` rule
can be perfectly formed while the endpoint recorded for its peer is one
this site cannot reach: `update_all_targets` then builds no client and
`replicate_object` drops every object against that ARN, yet the rule set
still reads as correct and the bucket reports in sync.
Each site now reports whether all of its `site-repl-*` rules resolve to a
live target (`SRBucketInfo.replicationTargetsOnline`, read from the
already-resolved client map so the status path stays cheap), and the
status aggregation treats an offline report as a mismatch. The field is
additive and optional: peers that omit it are "unknown", never a fault,
so a mixed-version topology does not flip every bucket to out of sync.
The reconcilers also run on a 10-minute timer instead of at startup only,
so drift is repaired without waiting for a restart. Both are no-ops when
the wiring already matches — the bucket pass compares serialized targets
and the rule set before writing. The tick takes the site-replication
lifecycle lock with a non-blocking try_acquire and skips the round when
an add/remove/endpoint-refresh holds it: those run in phases, and
rebuilding rules between two of them would resurrect what the operation
just tore down.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(site-replication): invert reconcile dependency to satisfy layers
`startup_services.rs` sits in the infra layer and was calling the
reconcilers in `admin::handlers::site_replication`, which is interface —
a reverse dependency that check_layer_dependencies.sh rejects.
Moving the reconcilers down is not viable in this change: they rest on
the site-replication state core (`SiteReplicationState` alone has 107
in-file uses, `load_site_replication_state` 38, the state lock 33), so
relocating it would move ~2000 lines and ~200 call sites through a
bug-fix PR.
Invert the direction instead. A new infra module owns the contract and
the schedule; the admin layer registers its reconciler from
`register_site_replication_route`, which runs while the admin router is
built — `init_startup_http_servers` awaits that before
`init_startup_runtime_services` reconciles, so the hook is always
installed in time. No logic moves and no baseline entry is added: the
dependency genuinely reverses.
The lifecycle guard now wraps both reconcilers in one round rather than
each separately, closing the window between them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(site-replication): harden reconcile per review feedback
Addresses the automated review on #5292.
Security: secret recovery from bucket targets accepted any target carrying
the `site-replicator-0` access key. Bucket targets are writable by anyone
holding `admin:SetBucketTarget`, so such a principal could plant a secret
and have reconciliation recreate the broadly privileged replication
account with it. A target must now name a peer in the persisted state and
point at that peer's recorded endpoint, disagreeing targets abort the
recovery, and only missing/unreadable-account errors may trigger it at
all — a transient store failure no longer rewrites a live account.
Durability: the repair no longer deletes before creating. A readable
account is rebound in place through a new `parent_user` field on
`UpdateServiceAccountOpts`, gated to `site-replicator-0` under
`allow_site_replicator_account` exactly as the account itself is. The
parent also lives in the session-token claims, and
`prepare_service_account_auth` denies the account when the two disagree,
so both move together.
Availability: the reconcile scheduler no longer requires an inline IAM
bootstrap. Deferred IAM recovers in the background with no callback into
the scheduler, which left a recovered node with self-pointing rules until
the next restart. It now starts unconditionally and returns early while
IAM or the object store are unavailable. Its first pass runs inside the
task, so walking every bucket no longer delays startup.
Correctness: an endpoint refresh commits bucket targets and peer state in
separate steps without holding the lifecycle lock, so a tick landing
between them rewrote targets from the stale endpoint; the reconciler now
also skips while any pending marker is set. Rule repair preserves an
operator-authored `role` and clears only sender-owned site-replication
ARNs, matching the merge path.
Hot path: the per-object missing-target message returns to debug. The
condition is reported once per bucket per reconcile pass instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
test(replication): lock outbound checksum consistency for new algorithms (T3)
Adds a consistency test at the replication put-options boundary confirming that
the AWS 2026-04 additional checksum algorithms (XXHash3/64/128, SHA-512, MD5) are
forwarded into replication user_metadata identically to the classic five. The
outbound replication path routes a stored object checksum through the
algorithm-agnostic decrypt_checksums -> user_metadata flow, so the new algorithms
(already covered by rustfs-rio read_checksums) need no new-algorithm-specific
handling. This locks that behavior against regressions.
Investigation summary (no code change needed on the outbound side): the
per-algorithm ChecksumMode selection path is dormant (opts.checksum is never set
to a specific algorithm; tiering uses Content-MD5; the add_crc bool is dead
code), so extending ChecksumMode was unnecessary.
Co-authored-by: heihutu <heihutu@gmail.com>
* Change Rust toolchain channel to stable
Signed-off-by: houseme <housemecn@gmail.com>
* style: apply clippy --fix and cargo fix lint suggestions
Run `cargo clippy --fix --all-targets --all-features` and
`cargo fix --lib --all-targets` across the workspace, then resolve the
remaining warnings by hand:
- collapse needless borrows in `format!` args, prefer `?` over explicit
early returns, and use `.values()` / `.flatten()` iterator adapters
- rewrite the `Md5` scan loop via `manual_flatten` and re-indent the
`select!` macro body (rustfmt skips macro interiors)
- annotate the intentional dead-code `Md5` inherent methods (constructed
only by the test factory) with `#[allow(dead_code)]`
Behavior is unchanged.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(replication): allow loopback replication targets under an explicit test opt-in
Commit 5c7c757a3 (#4712) activated the previously-dormant replication e2e
suite (they had never run anywhere). All 9 fast tests then failed on main
because the SSRF egress guard rejects the 127.0.0.1 targets the e2e harness
configures: `target endpoint is not allowed: outbound URL host '127.0.0.1'
is not allowed: loopback address`. The whole harness runs on loopback, so
every replication test hit this before reaching its actual assertion.
Loopback is a genuine SSRF vector and must stay rejected in production, so
this does not relax the guard. Instead `validate_replication_target_endpoint`
gains an off-by-default opt-in (`RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET`)
that re-enables loopback targets (127.0.0.1 / ::1 / localhost) for single-host
multi-instance dev and the e2e harness. Private addresses stay unconditionally
allowed as before; the opt-in does not widen into link-local or the cloud
metadata endpoint. The e2e harness sets the env for every server it spawns
(single-node and cluster paths), overridable via extra_env.
Verified end-to-end: all 9 previously-failing replication_extension_test
smoke tests pass against a locally built binary. New unit tests in
bucket_target_sys pin the matrix — public/private always allowed, loopback
gated on the opt-in in both IP and hostname forms, and metadata/link-local
still rejected even with the opt-in on.
Refs: backlog#1147
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(replication): rename optin -> opt_in to satisfy typos check
Pure rename of three unit-test function names; no behaviour change.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
fix(object-data-cache): make GET body cache key write-unique and dedup lookups
Address four object-data-cache GET-path findings (backlog#1107 batch):
ODC-06 (backlog#1111): the cache key was content-unique, not write-unique.
Extend ObjectDataCacheKey with the resolved version's modification time
(i128 unix nanoseconds, None -> 0), derived once in the shared planner so the
ecstore hook and the usecase layer produce an identical key. An unversioned
overwrite advances mod_time, so a stale node can no longer serve old bytes for
up to the TTL under an MD5 collision; etag + size stay as belt-and-braces.
ODC-16 (backlog#1121): every cacheable GET planned and looked up twice (once in
the ecstore hook, once in the usecase layer), double-counting hits, hit_bytes
and lookups. GetObjectReader now carries a GetObjectBodySource marker
(Unprobed / HookMissed / HookServed); the hook stamps it, and
build_get_object_body_with_cache serves a hook-served body directly and skips
its lookup whenever the hook already probed. One hook-served GET now records
exactly one lookup.
ODC-19 (backlog#1124): ENABLE=true with no explicit mode defaulted to HitOnly,
which never fills and keeps a permanent 0% hit rate. Default to
FillBufferedOnly, log the resolved mode at startup, and warn when HitOnly is
selected explicitly.
ODC-24 (backlog#1129): max_entry_bytes above the in-memory GET fill limits was
silently inert. Clamp the planner's size eligibility to
min(max_entry_bytes, seek-support threshold, 64 MiB buffer cap) so ineligible
sizes plan SkipTooLarge instead of being reported eligible, and warn at startup
when the excess is inert.
Co-authored-by: heihutu <heihutu@gmail.com>
Introduce RUSTFS_REPL_HEALTH_CHECK_INTERVAL_MS, RUSTFS_REPL_MRF_FLUSH_INTERVAL_MS
and RUSTFS_REPL_RESYNC_POLL_MAX_MS so tests and operators can shorten the
replication background loops. Defaults are unchanged; invalid values fall
back with a warn and values below 10ms are clamped to avoid busy-spin.
Add replication_fast_env() e2e helper (backlog#1147 repl-4).
Phase 5 Slice 11 (backlog#939): move the background replication pool and stats —
the last service handles, and the only async ones — out of the process statics
into the per-instance InstanceContext.
- InstanceContext gains `replication_stats: OnceCell<Arc<ReplicationStats>>` and
`replication_pool: OnceCell<Arc<DynReplicationPool>>` (tokio async OnceCell),
with sync read accessors (`replication_stats`/`replication_pool`/
`replication_initialized`) and pub(crate) cell accessors for the async init.
- `init_background_replication` initializes the current instance's cells via the
same `get_or_init(async {…}).await` (workers still spawned once on first
init). The lifecycle owner helpers and the runtime-source accessors keep their
signatures and route through the current instance's context; the two statics
(and the now-unused lazy_static import) are removed. The replication-boundary
arch guard still passes.
Single-instance: init materializes one shared pool/stats via the bootstrap
context — byte-for-byte the same as the eager statics.
Tests: replication state is None until set and independent across instances.
Verification: cargo test -p rustfs-ecstore (22 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 11). Stacked on Slice 10 (#4494).
* fix(scanner): scope long walk timeouts
* fix(scanner): bound IAM config walks
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
fix(replication): persist original mtime in MRF entries (backlog#867)
MRF delete entries did not persist the original delete-marker mtime, so
after a restart the recovery replay path reconstructed the delete without
a source timestamp. Downstream the replica delete-marker was stamped with
the replay time (now()) instead of the source mtime, causing delete-marker
timestamp divergence across clusters.
Extend the MrfReplicateEntry disk format with an optional deleteMarkerMtime
field (persisted as Unix nanoseconds) in both duplicate struct definitions
(rustfs-replication and rustfs-filemeta). DeletedObjectReplicationInfo now
persists delete_marker_mtime, and start_mrf_processor restores it onto the
reconstructed delete so the replica keeps the source timestamp.
Backward compatibility: the new key uses skip_serializing_if + serde
default, so historical MRF files without it decode to None and replay
falls back to the current time (pre-#867 behaviour). No panic or entry
loss on old files.
Closesrustfs/backlog#867
fix(replication): don't silently swallow resync status persistence failure (backlog#799 B23)
After a resync computes a new replication status, it persists it via
`put_object_metadata` but discarded the `Err` case (`if let Ok(u) = ...`). A
failure left the object's on-disk replication status disagreeing with the resync
result with no signal at all. Log the failure at warn level instead.
Refs backlog#799 (B23), tracked in rustfs/backlog#863.
MRF delete replay reconstructed the delete with `..Default::default()`, leaving
`replication_state = None`. `replicate_delete` derives its target set purely
from `replication_state.replicate_decision_str`, so an empty state produced an
empty decision -> zero targets: the replayed delete contacted no remote at all,
a silent no-op that left replicas permanently diverged after a restart.
The MRF entry doesn't persist the decision and the source object is already
gone, so re-derive it from the live bucket config at replay time via
`check_replicate_delete` — mirroring the object heal path
(`get_heal_replicate_object_info`) — and set it on the reconstructed delete's
`replication_state`. This is a contained fix with no change to the on-disk MRF
format.
Refs backlog#799 (B9).
Note: the source delete-marker mtime is still not persisted in the MRF entry,
so a replayed delete marker is stamped with the replay time on the target. That
is a separate, minor consistency nuance (the delete now propagates correctly)
and can be addressed by extending the MRF entry format in a follow-up.
The MRF persister accumulated overflow entries in `pending`, flushed them with
`flush_mrf_to_disk`, and cleared `pending` on success. But `flush_mrf_to_disk`
*overwrites* the whole MRF file with exactly the entries passed. After flushing
batch A (file = A) and clearing, the next flush wrote batch B and thereby
overwrote the file to contain only B — and the MRF file is only replayed (and
cleared) at startup, never during the run, so batch A's entries were silently
lost. A crash after the B flush lost all of batch A's pending replications.
Keep `pending` cumulative (the file must hold the full set of overflow entries
for the run) and rewrite the whole set on each flush instead of clearing after
success:
- flush eagerly once 1 000 *new* entries accumulate since the last write
(measured against the flushed length, so a large backlog isn't rewritten on
every add), and on the 10s tick when dirty;
- bound the in-memory/on-disk backlog with `MRF_PENDING_CAP` (200 000) and log
once when the cap is hit rather than growing without limit.
Refs backlog#799 (B10).
The resync result verification HEADed the target after replicating and counted
the outcome with inverted error handling:
- for a delete marker, ANY HEAD error (timeout, 5xx, auth, malformed) was
counted as replicated (success);
- for a versioned object against an AWS-style target, HEAD was sent with the
RustFS UUID versionId, which AWS rejects with 400, so a well-replicated
object was counted as failed.
Classify the error before counting:
- delete marker: only a definitive 404/NoSuchKey or 405/MethodNotAllowed
confirms the marker propagated (`is_retryable_delete_replication_head_error`
== false); any retryable/ambiguous error now counts as failed;
- versioned object with a version-id-format rejection: re-verify via
`head_object_fallback` (versionId-less HEAD) before deciding — present ->
replicated, absent/error -> failed;
- all other errors: failed, as before.
Reuses the existing, unit-tested classifier helpers. Verified against the
existing resyncer suite (24 tests).
Refs backlog#799 (B13).
During resync against an AWS-style target that rejects RustFS UUID versionIds,
the code retries HEAD without a versionId and compares ETags. On a match it set
`replication_action = None` ("already in sync, nothing to copy") but did not
return, so control fell through to `if replication_action != ReplicationAction::All`
— a branch meant only for the unsupported metadata-only case — and stamped the
object FAILED with "metadata-only replication is not implemented". The target
already held an identical object, yet the source recorded FAILED forever, so
AWS-style targets never converged and the MRF kept re-queuing.
Handle `ReplicationAction::None` explicitly before that branch: record it as
Completed (with the resync timestamp/`replication_resynced` bookkeeping for
ExistingObject + reset_id, mirroring the HEAD-success None path) and return.
Only `ReplicationAction::Metadata` now takes the metadata-unsupported failure
branch; `All` still proceeds to the copy. This path is the only way `None`
reaches that point (the HEAD-success None case already returns earlier).
Refs backlog#799 (B11).
cxymds