* refactor(replication): split four oversized hot-path functions into focused helpers
Pure-move decomposition of the four oversized functions flagged by the
replication compatibility review (P1-18), unblocking migration milestone
M2 which requires resyncer moves to stay mechanical:
- resync_bucket (522 lines -> 61-line step sequence): leader lock,
target resolution, walk/collector/worker spawning, and dispatch loop
extracted into focused helpers; pure decision helpers (DTO builders,
HEAD-result classification) separated from IO orchestration.
- replicate_all (411 lines -> 113-line main body): initial target-info
seeding, read/stat option builders, skip-path notes, target HEAD
action resolution, and the multipart/single-put payload transport
extracted as private free functions.
- start_mrf_processor (306 lines -> 46-line spawn body): recovery guard,
ledger load, per-entry replay (delete/object/metadata), and retained
entry resolution extracted; retry bookkeeping semantics preserved
exactly (inner continue-paths push inside helpers, outer Missed push
stays in the loop).
- apply_iam_item (255 lines -> match dispatch skeleton): one helper per
IAM item type.
No behavior change: log texts, error paths, event emissions, and metric
counts are byte-identical; existing tests unchanged and green (238
ecstore replication/mrf/resync + 232 rustfs site-replication).
* feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets (#6172)
* feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets
Implements the MinIO active-active read-proxy protocol (P1-5 of the
replication compatibility review): when a GET/HEAD/GetObjectTagging/
PutObjectTagging/DeleteObjectTagging request fails locally with
not-found and the bucket has replication targets, the request is proxied
to the targets in rule order, mirroring bucket-replication.go
proxyGetToReplicationTarget/proxyHeadToRepTarget/proxyTaggingToRepTarget.
Protocol surface:
- Anti-loop: inbound {x-rustfs-,x-minio-}source-proxy-request is parsed
into ObjectOptions (proxy_request + proxy_header_set, matching MinIO
ProxyRequest/ProxyHeaderSet); a request carrying the marker with ANY
value is never re-proxied. Outbound client proxy calls send the marker
as "true"; replication worker convergence HEADs send it as "false" so
a peer's proxy layer cannot answer a convergence check by proxying
back to the source (which would fake Completed without a PUT).
- Target selection: new replication_proxy.rs get_proxy_targets — empty
when the marker is set, versioning is suspended, or no replication
config; otherwise filter_target_arns -> TargetClient lookup, skipping
targets with proxying disabled.
- TargetClient gains head_object_for_proxy/get_object (streaming) and
the three tagging calls. Proxy calls never send the replication-check
SSE-C exemption header; customer SSE-C keys are forwarded verbatim so
the target performs real decryption. Conditional (If-*) headers are
not forwarded (MinIO parity); Range and part_number are, with
parts_count/tag_count/storage_class/expiration passed through.
- Metrics: proxy counters now count only real client proxy traffic,
MinIO-aligned (one total per proxied request, one failed when no
target served it). The previous misattributed counters — replication
worker HEAD/PUT (#2672) and local tagging operations (#2682) — are
removed; ReplProxyMetric now maps the tagging counters instead of
dropping them.
e2e (fake_s3_target extended with tagging + header journaling): proxied
GET body + outbound header contract (marker present, no
replication-check, SSE-C passthrough), HEAD, anti-loop 404 with zero
outbound requests, GetObjectTagging, and metric mapping unit tests.
Rolling note: proxying only activates for buckets with replication
targets; requests carrying the marker keep pre-upgrade behavior.
Refs rustfs/backlog#1675 (P1-5)
* fix(replication): fail SSE-C passthrough closed on targets that drop transport headers (#6178)
SSE-C ciphertext passthrough replicates via X-Rustfs-Replication-* transport
headers. A MinIO/generic-S3 target silently discards them, storing bare
ciphertext with no decryption material — yet the PUT succeeded, so the object
reported COMPLETED with a silently unreadable replica (backlog#1675 N2).
Fail-closed design:
- SsecPassthroughCapability {Unknown, Supported, Unsupported} cached in
BucketTargetSys per target ARN with a recording timestamp. Entries reset
whenever the target is rebuilt, edited, or removed (arn_remotes_map
lifecycle) and expire after SSEC_PASSTHROUGH_CAPABILITY_TTL (10 minutes):
an expired verdict in either direction is re-earned through the audit, so
an Unsupported target recovers automatically after an upgrade (at most one
wasted PUT+HEAD audit per bad target per TTL window) and a Supported
verdict cannot outlive a backend swapped behind the same endpoint.
- Replication worker (replicate_object and replicate_all): fresh Unsupported
targets never receive the PUT — the attempt fails immediately into the
normal MRF retry channel with a "run ?replication-check to re-probe" hint.
Unknown or expired verdicts are audited: after the PUT the worker HEADs
the replica back through the replication-check channel (source version id
mapped through resolve_read_api_version_id, so null-version objects audit
correctly) and requires SSE-C evidence (the echoed customer-algorithm
header); missing evidence records Unsupported and fails the attempt.
Convergence HEADs are audited the same way, so a broken ciphertext replica
from an earlier attempt can never launder itself into COMPLETED via an
ETag match. The gate/evidence policy is pure (replication_target_boundary,
staleness folded in as an input) for the M2 worker migration.
- replication-check grows an SsecPassthrough probe phase: a probe PUT
carrying the live transport-header shape, HEAD-back for evidence, and a
machine-readable Code BucketRemoteSsecPassthroughUnsupported on failure.
The probe verdict is synced into the runtime capability cache. Unlike
VersionFidelity, a failed SsecPassthrough phase does NOT fail the target
overall — it is a capability limit, not a broken replication contract,
and a plaintext-only deployment against such a target must not turn red.
- fake_s3_target: default mode now models a RustFS target (stores the
transport headers, echoes SSE-C evidence); the new
drop_unlisted_replication_headers mode models MinIO. The journal records
whether a request carried transport headers.
Receiver-echo verification: the replication-check HEAD exemption only skips
SSE-C key validation; the response has always built sse-customer-algorithm
from stored metadata (rustfs/src/app/object_usecase.rs), so no receiver
change was needed — pinned end to end by the replication-check e2e against
a real RustFS target.
Rolling-upgrade constraint: RustFS targets older than the replication-check
HEAD exemption (#5898) answer the audit HEAD without SSE-C evidence (or fail
it outright), so SSE-C replication to such targets reports FAILED. This is
deliberate — FAILED-and-retryable beats a silently undecryptable replica —
and self-heals: once the target is upgraded, the next TTL expiry (or a
manual ?replication-check re-probe) re-audits and records Supported.
Plaintext and managed-SSE replication are unaffected. The capability cache
is per-node; each node audits independently.
Known limitations:
- The audit judges evidence from the echoed customer-algorithm header only.
A hypothetical target that preserves that one header while dropping other
transport headers (partial-drop) would pass the audit; no known target
behaves this way — observed targets drop the whole unknown-header family.
- A mixed-version target cluster can flap the verdict between audits routed
to different target nodes until the rollout completes; the TTL bounds how
long each stale verdict persists.
New e2e (backlog#1675 C1 + N2, red-first): fail-closed against a
header-dropping fake (FAILED + no second PUT via the capability cache,
journal-asserted; red run showed the old COMPLETED), replication-check
reports the SsecPassthrough phase Code while the target stays OK overall,
SSE-C heal convergence after a real target outage, and SSE-C
existing-object resync landing a REPLICA readable with the customer key.
TTL expiry in both directions is pinned at the cache and gate seams.
* refactor(replication): move resyncer pure decision logic into rustfs-replication (M2) (#6180)
* refactor(replication): move resyncer pure decision logic into rustfs-replication (M2)
Pure-move milestone M2 of the ECStore replication split (backlog#1675
P1-17): relocate the resyncer's IO-free decision helpers, with their unit
tests, into the crates they already belong to by type ownership. No
behavior change.
Moved into crates/replication:
- resync.rs: resync_status_duration
- delete.rs: resync_existing_delete_replication_info,
replicate_delete_outcome, target_delete_version_id,
delete_marker_purge_version_id, delete_marker_purge_mrf_entry
- object.rs: version_identity_drifted, is_replication_target_offline_error,
SsecPassthroughCapability, SsecPassthroughGate, ssec_passthrough_gate,
ssec_passthrough_evidence_present (param-demoted to the echoed
customer-algorithm string; ECStore keeps the HeadObjectOutput adapter)
- filemeta.rs: NULL_VERSION_ID wire literal (crate-owned copy per the
filemeta-independence contract)
ECStore rewiring (Rule #14: imports stay in *_boundary.rs):
- resync/object-decision/target boundaries re-export the moved symbols;
resyncer call sites are unchanged
- bucket_target_sys keeps only the verdict cache + TTL and re-exports the
capability enum so existing consumer paths keep compiling
Not moved (signatures carry ECStore or aws-sdk types):
verify_resync_head_result, resync_target_error_detail, the SdkError
classifiers, the replicate_all_* option/info builders, and the env-coupled
bounded_resync_max_jobs admission clamp. README milestone table updated.
* chore(replication): retire the datatypes.rs relay early
README sanctions retiring datatypes.rs ahead of M4. The module was a
pure relay (resync boundary -> datatypes -> mod.rs facade) with no
external consumer importing it directly, so the facade now re-exports
ResyncStatusType from replication_resync_boundary and the relay file is
deleted. Consumers stay behind the ECStore facade, keeping Migration
Rule #15 intact — the original retirement wording ("consumers import
through rustfs-replication directly") conflicted with that rule and is
corrected in the README.
* chore(arch): extend migration guards to the M2-moved decision contracts
The adversarial review of the M2 move found the per-symbol ratchet in
check_architecture_migration_rules.sh was not extended for the moved
symbols, leaving them free to be redefined in ECStore or imported past
their boundary without CI noticing:
- resync definition pin + boundary fences gain resync_status_duration;
- the object-decision boundary fences gain the five delete-family
helpers (delete_marker_purge_mrf_entry, delete_marker_purge_version_id,
replicate_delete_outcome, resync_existing_delete_replication_info,
target_delete_version_id);
- the target-boundary fence gains the SSE-C gate family, the offline
classifier, and version_identity_drifted;
- a new definition pin rejects ECStore redefinitions of the M2-moved
fns/enums (ssec_passthrough_evidence_present deliberately excluded:
ECStore keeps a thin HeadObjectOutput adapter under that name).
Mutation-verified: a probe fn ssec_passthrough_gate under
crates/ecstore/src/bucket/replication trips the new pin.
Also anchors the intentionally-duplicated NULL_VERSION_ID wire literal
from the filemeta side and tightens the M2 README note on
bounded_resync_max_jobs.
* chore(ecstore): drop the bucket dead_code blanket
The last blanket of the backlog#1823 burn-down, and the largest: 71 items across lifecycle, replication, metadata, quota, object lock and bucket utils. Four are deleted.
Deleted, all trivial:
- check_valid_object_name and check_valid_object_name_prefix, a pair that only calls into each other with no external caller. Worth stating plainly so nobody reads this as a validation gap: object names are validated through check_object_name_for_length_and_slash, which is live; this pair is a second, unwired entry point.
- DEFAULT_HEALTH_CHECK_RELOAD_DURATION, a lone unused constant.
- The LifecycleReplicationConfig alias, which orphaned a re-export in replication/mod.rs that goes with it.
Everything else is kept, in four groups, because the blanket here was hiding structure rather than rot:
Windows platform gating. WINDOWS_RESERVED_NAMES, the two reason constants and object_name_has_windows_incompatible_segment are called from inside the #[cfg(target_os = "windows")] block in check_object_name_for_length_and_slash (utils.rs:228-255), so they only read as dead on non-Windows hosts. As with the Linux gating in the disk root, this cannot be adjudicated locally: cargo check for both x86_64-pc-windows-msvc and x86_64-unknown-linux-gnu fails in the aws-lc-sys build script for want of a cross C toolchain. CI covers both.
Declared boundary surface. The *_boundary.rs and *_bridge.rs files carry the replication split plan's contracts, which scripts/check_architecture_migration_rules.sh pins through the EcstoreReplicationBoundaryImports section of the split-plan doc. Their unused items are declarations, not leftovers.
test-util seams. ConfigWriteLockProbe with install/wait_until_attempted follows the same pattern as the barriers in the services and set_disk roots.
MinIO-parity tier/lifecycle entry points that this port never wired: apply_lifecycle_action, get_transitioned_object_reader, recover_tier_free_versions, delete_object_from_remote_tier, abort_tier_delete_journal_entry and the replication pool's worker-management surface. These are complete, substantial machinery with no caller — the same shape as data_usage's local_snapshot feature. Removing them is a product decision, so they are made explicit here rather than deleted.
Verification, four lanes warning-free: default, --tests, --features rio-v2 --tests, --features test-util --tests. cargo nextest run -p rustfs-ecstore 4096 passed; clippy --lib --tests -D warnings clean; make pre-commit exit 0. Note that clippy is what caught the orphaned re-export above: cargo check and pre-commit both treat unused_imports as a warning.
Ref rustfs/backlog#1823 (step 2, final root).
* chore(ecstore): correct inaccurate dead_code reasons in the bucket root
Six items were labelled 'asserted by this file's tests' or as MinIO-parity
entry points while having no caller at all - free get_bucket_acl_config and
created_at only reach their own live methods (production goes through
created_at_in), BucketVersioningSys::get_in, utils::serialize_content and
ServiceType have no reference anywhere, and with_transition_queue_env_async
is an unused test fixture, not a tier entry point. Name what each one is so
the next reader does not assume coverage that is not there.
Ref rustfs/backlog#1823.
* test(replication): pin the version-fidelity probe contract (red)
P1-19 (rustfs/backlog#1675 B2): the supported replication contract is
targets that adopt the source version id — a target that mints its own ids
silently breaks every version-addressed operation that follows (version
deletes, heal re-drives never match), diverging the two sides with no
signal. replication-check already captures the probe PUT's response version
id but never compares it.
Red evidence (current main): against a FakeS3Target with
assign_own_version_ids enabled, ?replication-check returns Status "OK" —
the drift is invisible.
test_replication_check_flags_version_minting_target expects a
VersionFidelity phase that fails with the machine-readable code
BucketRemoteTargetVersionMismatch, skips the later mutation phases, and
still cleans up the probe via the version id the target actually assigned.
Test infra: FakeS3Target gains assign_own_version_ids (models a generic S3
service; validated-but-not-mirrored source version headers) and a
prefix+max-keys ListObjectVersions implementation (the probe key allocation
requires it); stored_versions accessor duplicated from the P1-21 branch
(identical code, resolves clean on merge).
* fix(replication): probe the version-identity contract in replication-check
P1-19 (rustfs/backlog#1675 B2, plan B). Replication only converges on
targets that adopt the source version id: version-addressed deletes and
heal re-drives address the source id, so a target that mints its own ids
silently diverges — nothing surfaced this. replication-check already
captured the probe PUT's response version id but never compared it.
- The probe PUT now carries the source version as `?versionId=` (the exact
shape live replication uses since P0-5, and the only shape MinIO
consumes; the internal source-version-id header alone would let the
probe pass against targets the real data path drifts on). Reuses
ecstore's append_version_id_query through the api facade.
- New VersionFidelity phase: the probe PUT's response version id must
equal the sent source id. On mismatch the phase fails with the
machine-readable extension key `"Code": "BucketRemoteTargetVersionMismatch"`
(new optional Code field on phase statuses; Go decoders ignore unknown
keys), the overall target fails, the later version-addressed mutation
phases are skipped, and cleanup still removes the probe via the id the
target actually assigned (with the existing list-based sweep as backstop
when the target returns no version id at all).
- Runtime half: TargetClient::put_object now returns the assigned version
id (mirroring remove_object), and the replication PUT path audits it —
every drifting PUT increments
rustfs_replication_version_identity_drift_total and the first drift per
target ARN logs a structured warning pointing at ?replication-check.
The drift judgment is a pure function with an exemption-matrix test
(empty / literal "null" / nil-uuid sources carry no contract).
- docs/operations/replication-check.md documents the phase and the code.
Red -> green: test_replication_check_flags_version_minting_target (fake
target with assign_own_version_ids; on main the check reported Status
"OK"). The probe's query shape is pinned by a journal assertion (revert
of the query hunk alone fails it), probe-level unit tests cover the
mismatch/mirror matrix including cleanup addressing the minted id, and
the existing success e2e now asserts VersionFidelity OK against a RustFS
target. Adversarial review (seven roles): non-blocking; noted follow-ups
are the multipart runtime audit (the probe phase already pins the
contract) and per-target re-warning after reconfiguration.
* fix(e2e): stop the fake target self-deadlocking on version-id minting
The assign_own_version_ids flag was read with a fresh `lock(&self.store)`
inside two paths that already hold that guard — delete_object's
marker-creation branch and create_multipart_upload — and the store mutex
is not reentrant, so both hung forever (CI: the fake target's own
multipart and delete-marker tests ran >1560s until the job was
cancelled). Read the flag from the live guard instead.
The replication e2e paths did not catch this: a version-addressed purge
DELETE never mints an id, and the probe PUT reads the flag before taking
the guard.
* chore(test): refresh the nextest replication count invariant
The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata
(authority: `cargo nextest list`); refresh it to this branch's
post-rebase total.
* test(replication): pin delayed delete-marker purge failure handling (red)
P1-21 (rustfs/backlog#1675 B2): two failing e2e tests that pin the missing
failure handling of the delayed delete-marker purge:
- test_delayed_delete_marker_purge_retries_after_transient_target_failure:
four scripted 503s outlast every existing channel (version-purge
replication + its in-process MRF fast retries + the watcher's single
attempt = 3 target DELETEs, all faulted in the recorded run); the
replicated marker is stranded on the target forever.
- test_delayed_delete_marker_purge_exhaustion_persists_to_mrf_and_replays_on_restart:
exhausted purge intents never reach the durable MRF journal, so a restart
replays nothing (recorded run: 3 faulted attempts, zero post-restart).
Red-light evidence (current main):
- Test A: FAILED, journal shows 3x DeleteObject fault=Status(503), no clean
attempt, target marker still present after 15s.
- Test B: FAILED after 468s, same 3 faulted attempts, no purge DELETE after
restart, marker still present.
Test infra: FakeS3Target::stored_versions() exposes per-key version state so
purge tests assert target state instead of inferring it from the journal;
nextest count comments 36->38 nightly / 56->58 total.
* fix(replication): retry, persist and replay failed delete-marker purges
P1-21 (rustfs/backlog#1675 B2). The delayed delete-marker purge was
fire-and-forget: the target DELETE discarded its result (`let _ =`), a
missing target client was silently skipped, and nothing recorded the intent
— one transient target error stranded the replicated marker on the target
forever. Separately, `replicate_delete_with_outcome` held its outcome
hostage to `!requires_delayed_purge`, pinning every delete-marker MRF entry
to Missed so the durable backlog retained them permanently.
Changes:
- `replicate_delete_marker_purge_to_targets` now reports per-target
results (warn + metrics on failure, including `target_client_missing`),
supports retrying only the failed targets, and treats a target-side
NoSuchKey/NoSuchVersion as purge success (strict-404 targets must not
retain the intent forever).
- The delayed watcher (`watch_and_purge_source_delete_marker`) retries
failed targets across its 5x1s watch window; on exhaustion it persists
the purge intent to the durable MRF journal via the new
`ReplicationPoolTrait::persist_mrf_entry` (journal-only on purpose: live
re-dispatch would loop unboundedly against a down target). Intent entries
are shaped as marker-creation deletes so replay funnels into the stale-
marker branch.
- The stale-marker branch (source marker already gone) now purges the
targets instead of silently returning success — closing a latent leak —
and reports the purge result as the replay outcome. Heal callers retry
for the full window (the startup MRF processor runs before target
clients initialize); live callers attempt once and fall back to a fresh
durable intent, so a down target cannot pin a replication worker.
- The outcome formula (extracted as `replicate_delete_outcome` and pinned
by a unit test) no longer includes the delayed purge, so successfully
replayed delete-marker entries are acknowledged instead of retained
forever.
Verification: red -> green e2e pair (transient-failure retry; exhaustion ->
durable MRF -> restart replay -> second-restart zero-replay ack) plus unit
tests; `make pre-commit`, logging guardrails, clippy (ecstore + e2e_test)
all clean; full ecstore lib suite 3729 passed (3 pre-existing local-DNS
kubernetes endpoint failures reproduce without this change).
Adversarial validation (7 roles): no blocking findings after adding the
outcome-formula guard test. Known residuals recorded in the PR: watcher
shutdown window (intent not yet persisted), rolling-downgrade replay acks
without purging (equals pre-fix behavior), and replay falling back to the
source version id on targets that mint their own version ids (P1-19).
* chore(test): refresh the nextest replication count invariant
The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata
(authority: `cargo nextest list`); refresh it to this branch's
post-rebase total.
* fix(replication): purge the marker version the target actually assigned
Review follow-up (#5864), two real defects:
- The delayed purge watcher was spawned with the pre-merge `dobj`, so the
per-target marker version ids this round recorded were invisible to it.
Against a target that mints its own ids the purge fell back to a
source-derived id, the target answered the versioned DELETE with an
idempotent 204, and that "success" cleared the retry set while the real
marker stayed behind. The watcher now receives the merged replication
state (`drs`), which folds this round's target-assigned ids in.
- A target whose recorded version metadata is inconsistent was skipped
without entering `failed_arns`, so an empty result made both the watcher
and the MRF replay treat a purge that issued no DELETE as successful and
drop the intent. The refusal is now a per-target failure (own metric
label): the leak stays visible and the intent is retained instead of
being acknowledged. The version decision also moved ahead of the client
lookup, so the refusal is decided from metadata alone.
Tests: a new e2e drives a fake target with `assign_own_version_ids`, which
ignores the forwarded source-version header for both objects and delete
markers, and asserts the replicated marker is really gone; a unit test
pins the corrupt-metadata refusal as a failed outcome without any target
client registered. The detached-watcher shutdown window is documented at
the watcher as a known non-durable window with the write-ahead follow-up
spelled out.
Complete the encrypted-object replication series (backlog#1783, PR-C of
3, after #5872 and #5885): SSE-C objects replicate as ciphertext
passthrough — the source holds no customer key, so the stored bytes and
their encryption metadata travel verbatim and the replica decrypts only
with the original customer key, single-part and multipart.
- Sender: SSE-C objects read raw (raw_data_movement_read), transfer at
ciphertext size, and range multipart parts over stored part sizes.
- Receiver: authorized replication PUTs restore the stored SSE-C keys
from the transport headers (exact lowercase forms - the read-path
check is case-sensitive), set ObjectOptions.preserve_ciphertext, and
skip compression, bucket-default SSE, and sse_encryption behind one
restore-derived gate. Multipart uses an internal session marker to
store parts verbatim and strips it on complete.
- Convergence: the replication HEAD sends
x-rustfs-source-replication-check; the target authorizes it as
ReplicateObjectAction and skips SSE-C read validation for that
request only, so keyless convergence HEADs see etag/size/mtime
instead of 400 and SSE-C replicas stop re-driving forever.
- e2e: SSE-C contract flips to a key-gated readable replica (no-key and
wrong-key GETs fail - the direct silent-plaintext detector); new
multipart passthrough contract with ETag/marker/stability assertions.
Open the managed-SSE replication gate (backlog#1783, PR-B of 3, after
#5872): the replication reader already decrypts through the injected
object-encryption resolver, so the source sends plaintext plus an
encryption intent header (AES256 / aws:kms, never the source key id) and
the target re-encrypts on its normal PUT path with its own KMS. No DEK
crosses sites.
- replication_put_object_options: fail closed only on Unsupported;
insert the SSE intent after the strip loop.
- TargetClient::create_multipart_upload sends the full opts.header()
set, fixing multipart replicas losing content-type/user metadata
(plaintext included).
- Preserve source ETag and mtime on replicas (authorized replication
only): receiver wires x-rustfs-source-etag into preserve_etag for PUT
and CompleteMultipartUpload, resolve_complete_etag consumes it, and
complete options carry source_etag/source_mtime (absent mtime
degrades to epoch, not now_utc). Without this every replication HEAD
comparison re-drives re-encrypted objects forever.
- e2e: managed SSE contracts flip to success on an independent-KMS
dual-process pair (byte-identical plain GET proves target-owned
envelopes; ETag/mtime preserved; version stable across scanner
cycles; resync converges; multipart keeps structure and metadata);
new target-without-KMS fail-closed contract; SSE-C stays FAILED.
Co-authored-by: houseme <housemecn@gmail.com>
Both per-target replication methods assign the optimistic Completed status to rinfo before building put options, and the Err branch of replication_put_object_options returned rinfo unchanged. Since #5633 made the source encryption classification case-insensitive, managed SSE sources are rejected at this gate, and the rejection was reported as successful replication: the source object was marked COMPLETED, ObjectReplicationComplete was emitted, and nothing existed on the target.
Set replication_status = Failed (and record the error in the replicate_object branch) so the composite status, the OperationFailedReplication event, and MRF retries reflect the fail-closed outcome. This restores the contract pinned by test_bucket_replication_sse_kms_failure_contract, which timed out in the e2e-replication-nightly runs on 2026-08-03 and 2026-08-04.
* feat(replication): purge delete markers by the target's own version id
When a delete marker is replicated, the target assigns it a version id. The
purge that follows derived one from the *source* uuid instead, which is only
correct when the target mirrors source version ids. A generic S3 target does
not: the derived id addresses a version that does not exist there, so the
purge is a no-op and the replica keeps a marker the source has already
removed. Same failure class as #4401.
Record the id the target reports and address it directly on purge.
Data path, all of it driven by the object's internal metadata rather than the
`ReplicationState` wire form, which encodes positionally and cannot carry a
map:
- `rustfs-utils`: the `replication-delete-marker-version-<arn>` key family,
plus `strip_internal_prefix_preserving_case` — ARNs are case-sensitive and
the existing `strip_internal_prefix` lowercases.
- `ReplicationState` gains the map and a `..._corrupt` flag, both
`#[serde(skip)]`; `ReplicatedTargetInfo` carries the per-target id.
- `persist_target_delete_marker_versions` is merge-only. A delete arriving
over internode RPC has an empty map, so treating it as authoritative would
let a remote disk erase an id the local disk still holds.
- `delete_object_version` copies the map into `fi.metadata` before dispatch,
so the durable carrier crosses the wire even though the field does not.
- The keys are folded into the quorum hash through their normalized form:
the dual internal prefixes carrying one mapping share an identity, while a
genuine disagreement between disks still shows up as a quorum difference.
- `corrupt` (the prefixes disagreed) fails closed: skip the purge and warn
rather than guess an id and risk destroying a live version on the target.
Ported from the rc.1 branch, which cannot merge as a whole: its MRF replay
rewrite collides with #5659/#5671/#5672/#5673 and regressed
`MRF_PENDING_CAP`. main's MRF machinery is kept; only this capability moves
across. It touches no MRF code.
Two things did not survive the port, deliberately. The branch's
`missing_is_complete` purge regression does not exist here — it came from its
own HEAD-precheck rewrite, and main's simpler path never had it. And the
branch's `MrfReplicateEntry` ordering fields are MRF-redesign scope, left
behind.
Verification: cargo fmt --all --check, git diff --check,
cargo check --workspace --all-targets, and the suites for the four touched
crates — 4070 tests, 2 pre-existing failures unrelated to this change
(`system_resolver_negative_result_reaches_the_dns_allowlist`,
`test_resolve_domain_preserves_system_resolver_error_provenance`; both are
the sandbox DNS interception, they fail on a clean checkout too).
* fix(replication): keep the layer guard happy
scripts/check_architecture_migration_rules.sh matches on text, so the doc
comments naming `rustfs_filemeta::` read as a cross-layer dependency even
though nothing imports it. Reword them; the guard passes.
* fix(replication): make the target-version cap deterministic
Two defects in this PR, both found in review.
The cap was applied while iterating a `HashMap`, so *which* 1000 entries
survived depended on iteration order. Two disks decoding the same oversized
metadata could keep different subsets, hash differently, and lose quorum —
instead of both reporting the same corruption. Collect first, then truncate
in `BTreeMap` order, which is total and identical everywhere.
And `persist_target_delete_marker_versions` discarded the `corrupt` flag from
the RPC carrier, committing a delete-marker update that looked clean while the
exact remote marker identity was unknown. It now declines to merge a corrupt
carrier. Because the helper only ever inserts, declining leaves the durable
keys already on the object untouched, which is strictly safer than writing a
mapping we cannot trust.
Residual, stated rather than papered over: corruption confined to the RPC
carrier is not persisted as a sentinel, so a later reader of an object that
carried no durable keys still sees "legacy, no mapping" rather than "corrupt".
Persisting that would need a wire-format addition; the consumer already fails
closed on any corruption it can observe.
New test: `target_delete_marker_versions_cap_is_deterministic_across_decodes`
decodes the same 1050-entry map twice and asserts both the corrupt flag and
the retained subset agree.
* fix(replication): preserve multipart source mtime (#5669)
* fix(kms): repair unopenable ciphertext and cover the Vault backends (#5668)
* Add black-box behavior tests for KMS resilience and serialization
* fix(kms): repair unopenable ciphertext across backends
Black-box testing of the KMS crate surfaced several defects that make
encrypted data permanently unreadable.
Symmetric envelopes. The Local and Vault Transit backends returned raw
cipher output from `encrypt` while `decrypt` parsed a JSON envelope, so
anything sealed through the master-key path could never be opened again.
Local also discarded the AES-GCM nonce. Both now emit the same envelope
`decrypt` consumes, matching the Static backend.
Deterministic AAD. The object layer derived AEAD additional data by
serializing a `HashMap` directly. Iteration order differs per instance,
so a context rebuilt from storage produced different AAD bytes than the
one used to seal and the object stopped opening. Ordering by key removes
that dependency, matching the Static backend's existing `context_aad`.
Objects written with the default single-key context are unaffected,
since a one-entry map has only one serialization.
Cipher in the header projection. `metadata_to_headers` recorded the SSE
mode (`AES256` / `aws:kms`), which cannot represent ChaCha20-Poly1305,
so a ChaCha-sealed object came back claiming `aws:kms` and was opened
with the wrong cipher. The cipher now travels in
`x-rustfs-encryption-algorithm` — the header the storage layer already
reads but nothing ever wrote. Objects without it fall back as before.
Also: the Static backend ignored `key_spec` and always issued 256-bit
data keys; Local `list_keys` hardcoded `truncated: false`, ignored
`marker`, and paginated over unordered `read_dir`, so a paginating
client silently saw a partial key list; and Local and Vault KV2 reported
`key_id: "unknown"` from `decrypt` despite the envelope naming the
master key.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(kms): cover both Vault backends and key rotation
The behavior suite ran only against Local and Static, and its own harness
documented the gap: the Vault backends had no business-capability
coverage at all. Setting `RUSTFS_KMS_VAULT_TOKEN` now adds Vault KV2 and
Vault Transit to every `for_each_backend` spec against a live server.
That lane is what surfaced the Transit envelope defect fixed in the
previous commit.
`rotate` and `versioning` are advertised only by the Vault backends, so
until now every capability-gated branch for them took the
`UnsupportedCapability` side and the working half was never asserted — a
rotation that dropped prior key versions would have gone green. The new
`behavior_rotation.rs` pins that half: material sealed before a rotation
still opens after it, repeated rotations accumulate versions rather than
overwriting a single spare, and the history survives a restart.
Two test defects fixed. `objects_round_trip_across_sizes_and_algorithms`
asserted a 1-byte object differs from its own ciphertext, which collides
once every 256 runs; the assertion now applies only where a collision is
not realistic, and small objects stay covered by the tag check and the
decrypt round-trip. `test_from_env_selects_token_file` depended on
`RUSTFS_KMS_VAULT_TOKEN` being absent from the caller's environment and
now clears it explicitly.
The snapshots directory was also removed from `.gitignore`: insta
snapshots are the assertions themselves, so leaving them untracked gives
CI nothing to compare against. Only `.snap.new` scratch files are
ignored now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(kms): adapt behavior suite to current key APIs
Rebasing onto main brought four API changes the suite predates.
`DeleteKeyRequest` gained `confirm_key_id`, and immediate deletion is now
gated on the server's `allow_immediate_deletion`. Scheduled deletions pass
`None`; the four specs that destroy a key outright echo the key id back
and opt the harness config in, which is what the gate asks of a real
caller.
`LocalBackupExportRequest` gained `sanitized_config`. These specs cover
the key-material path, so they seal no configuration and pass `None`.
`KmsCacheStats` became a named struct with real hit, miss, and eviction
counters. `cache_stats_returns_an_entry_count_and_no_hit_or_miss_data`
existed to pin the old placeholder behavior — that the second tuple
element was always zero — which main has since fixed, so it is now
`cache_stats_reports_hits_and_misses_separately` and asserts the counters
actually move.
Starting the service provisions the reserved probe key, so it shows up in
listings and backup bundles. Exact-set assertions filter it through a new
`without_probe_key` helper rather than naming it, keeping those specs
about the keys they seeded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(kms): bind the AAD to the stored context bytes
Review caught that canonicalizing the AAD on decrypt breaks objects sealed
before canonicalization existed, and it was right. The AAD is the
*serialization* of the encryption context, and `x-rustfs-encryption-context`
stores that exact byte sequence: `encrypt_object` fed one `HashMap` to the
AEAD and then moved the same map into the metadata the header is written
from, so the stored string is byte-identical to the AAD the object was
sealed under. Those objects are therefore recoverable — but only while
nothing round-trips the value through a `HashMap` and re-serializes it.
Recomputing sorted AAD on decrypt would have turned a readable object into
a permanently unreadable one. The previous behavior was worse than the
first analysis credited: it did not merely fail intermittently, it made
the failure deterministic.
`EncryptionMetadata` now carries `context_aad`, the bytes the object was
actually sealed with. Encryption records what it fed the AEAD, the header
projection stores those bytes verbatim (and preserves a legacy ordering
across a re-projection rather than rewriting it into sorted form), and
`headers_to_metadata` carries the stored string through untouched. Both
decrypt paths, SSE-KMS and SSE-C, prefer it and fall back to canonical
serialization only when no stored serialization exists. Canonicalization
still applies to everything newly sealed, so the original ordering bug
cannot recur.
Two tests pin this: a legacy record whose sealed bytes are non-canonical
must survive a full header round trip unchanged, and a context header
rewritten to an equivalent-but-reordered serialization must fail
authentication rather than silently re-deriving a working AAD. Both were
mutation-checked against the reinstated bug on each side.
Also from review: the lifecycle churn test asserted only that every
request was accounted for, which holds whether the state gate exists or
not, so both branches are now pinned deterministically after the churn
(asserting `refused > 0` on the concurrent phase would only trade the hole
for a scheduling flake). And the Local and Vault KV2 envelopes compare
`encryption_context` without authenticating it — `DekCrypto` seals only
the plaintext — which is now documented at both sites; closing it needs a
versioned envelope, since existing ciphertext was sealed without AAD.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: ccccpj <ccccpj@outlook.com>
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Preserve metadata replication operations in the durable MRF and route tagging, retention, and legal-hold updates through the existing full-object replication transport. Keep ACL propagation outside the contract because the current object model has no durable object ACL state.
Refs #1616
* fix(site-replication): keep reverse direction after config broadcast
`site-repl-*` rules encode the sender's outbound direction: their
destination ARN names the receiver. `apply_bucket_meta_item` wrote an
incoming rule set verbatim over the receiver's, leaving the receiver with
a rule whose ARN is its own deployment ID. `reconcile_site_replication_bucket_targets`
skips the local peer, so no bucket target can back that ARN and every
object was dropped; the follow-up call reconciled targets only, so
nothing rebuilt the lost reverse rule. Replication went one-directional
after any PutBucketReplication broadcast — the console's Save button,
`mc replicate import`, a metadata import, or `/site-replication/repair`.
Only operator-authored rules now travel between sites; each site owns its
`site-repl-*` rules and rebuilds them from the current peer set.
Four defects kept that invisible or unrecoverable:
- `update_all_targets` discarded target-client build errors silently, and
`replicate_object` logged the resulting missing-target drop at debug
while every other failure there logs at error. Both now report.
- `site_replication_rule_complete` never checked that a rule's
destination named a remote site, so two sites holding identical
configs — the post-clobber state — passed as in sync.
- `update_service_account` cannot rewrite `parent_user`, and IAM records
encrypted with a previous root secret decode as "no such account".
Startup now reconciles the account, reseeding from the secret every
site-replication bucket target already stores, and refuses the
delete-then-create sequence when the parent cannot back an account.
Bucket rules are reconciled too, so an already-broken site heals on
upgrade.
- A joined site never verified it could reach the initiator, whose
endpoint is derived from the Host header of the admin request that
created the topology. The join now probes each peer and reports through
`initial_sync_error_message`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(site-replication): report unreachable targets and reconcile on a timer
Rule-shape checking cannot see an unreachable peer. A `site-repl-*` rule
can be perfectly formed while the endpoint recorded for its peer is one
this site cannot reach: `update_all_targets` then builds no client and
`replicate_object` drops every object against that ARN, yet the rule set
still reads as correct and the bucket reports in sync.
Each site now reports whether all of its `site-repl-*` rules resolve to a
live target (`SRBucketInfo.replicationTargetsOnline`, read from the
already-resolved client map so the status path stays cheap), and the
status aggregation treats an offline report as a mismatch. The field is
additive and optional: peers that omit it are "unknown", never a fault,
so a mixed-version topology does not flip every bucket to out of sync.
The reconcilers also run on a 10-minute timer instead of at startup only,
so drift is repaired without waiting for a restart. Both are no-ops when
the wiring already matches — the bucket pass compares serialized targets
and the rule set before writing. The tick takes the site-replication
lifecycle lock with a non-blocking try_acquire and skips the round when
an add/remove/endpoint-refresh holds it: those run in phases, and
rebuilding rules between two of them would resurrect what the operation
just tore down.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(site-replication): invert reconcile dependency to satisfy layers
`startup_services.rs` sits in the infra layer and was calling the
reconcilers in `admin::handlers::site_replication`, which is interface —
a reverse dependency that check_layer_dependencies.sh rejects.
Moving the reconcilers down is not viable in this change: they rest on
the site-replication state core (`SiteReplicationState` alone has 107
in-file uses, `load_site_replication_state` 38, the state lock 33), so
relocating it would move ~2000 lines and ~200 call sites through a
bug-fix PR.
Invert the direction instead. A new infra module owns the contract and
the schedule; the admin layer registers its reconciler from
`register_site_replication_route`, which runs while the admin router is
built — `init_startup_http_servers` awaits that before
`init_startup_runtime_services` reconciles, so the hook is always
installed in time. No logic moves and no baseline entry is added: the
dependency genuinely reverses.
The lifecycle guard now wraps both reconcilers in one round rather than
each separately, closing the window between them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(site-replication): harden reconcile per review feedback
Addresses the automated review on #5292.
Security: secret recovery from bucket targets accepted any target carrying
the `site-replicator-0` access key. Bucket targets are writable by anyone
holding `admin:SetBucketTarget`, so such a principal could plant a secret
and have reconciliation recreate the broadly privileged replication
account with it. A target must now name a peer in the persisted state and
point at that peer's recorded endpoint, disagreeing targets abort the
recovery, and only missing/unreadable-account errors may trigger it at
all — a transient store failure no longer rewrites a live account.
Durability: the repair no longer deletes before creating. A readable
account is rebound in place through a new `parent_user` field on
`UpdateServiceAccountOpts`, gated to `site-replicator-0` under
`allow_site_replicator_account` exactly as the account itself is. The
parent also lives in the session-token claims, and
`prepare_service_account_auth` denies the account when the two disagree,
so both move together.
Availability: the reconcile scheduler no longer requires an inline IAM
bootstrap. Deferred IAM recovers in the background with no callback into
the scheduler, which left a recovered node with self-pointing rules until
the next restart. It now starts unconditionally and returns early while
IAM or the object store are unavailable. Its first pass runs inside the
task, so walking every bucket no longer delays startup.
Correctness: an endpoint refresh commits bucket targets and peer state in
separate steps without holding the lifecycle lock, so a tick landing
between them rewrote targets from the stale endpoint; the reconciler now
also skips while any pending marker is set. Rule repair preserves an
operator-authored `role` and clears only sender-owned site-replication
ARNs, matching the merge path.
Hot path: the per-object missing-target message returns to debug. The
condition is reported once per bucket per reconcile pass instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
* Change Rust toolchain channel to stable
Signed-off-by: houseme <housemecn@gmail.com>
* style: apply clippy --fix and cargo fix lint suggestions
Run `cargo clippy --fix --all-targets --all-features` and
`cargo fix --lib --all-targets` across the workspace, then resolve the
remaining warnings by hand:
- collapse needless borrows in `format!` args, prefer `?` over explicit
early returns, and use `.values()` / `.flatten()` iterator adapters
- rewrite the `Md5` scan loop via `manual_flatten` and re-indent the
`select!` macro body (rustfmt skips macro interiors)
- annotate the intentional dead-code `Md5` inherent methods (constructed
only by the test factory) with `#[allow(dead_code)]`
Behavior is unchanged.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(replication): allow loopback replication targets under an explicit test opt-in
Commit 5c7c757a3 (#4712) activated the previously-dormant replication e2e
suite (they had never run anywhere). All 9 fast tests then failed on main
because the SSRF egress guard rejects the 127.0.0.1 targets the e2e harness
configures: `target endpoint is not allowed: outbound URL host '127.0.0.1'
is not allowed: loopback address`. The whole harness runs on loopback, so
every replication test hit this before reaching its actual assertion.
Loopback is a genuine SSRF vector and must stay rejected in production, so
this does not relax the guard. Instead `validate_replication_target_endpoint`
gains an off-by-default opt-in (`RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET`)
that re-enables loopback targets (127.0.0.1 / ::1 / localhost) for single-host
multi-instance dev and the e2e harness. Private addresses stay unconditionally
allowed as before; the opt-in does not widen into link-local or the cloud
metadata endpoint. The e2e harness sets the env for every server it spawns
(single-node and cluster paths), overridable via extra_env.
Verified end-to-end: all 9 previously-failing replication_extension_test
smoke tests pass against a locally built binary. New unit tests in
bucket_target_sys pin the matrix — public/private always allowed, loopback
gated on the opt-in in both IP and hostname forms, and metadata/link-local
still rejected even with the opt-in on.
Refs: backlog#1147
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(replication): rename optin -> opt_in to satisfy typos check
Pure rename of three unit-test function names; no behaviour change.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(scanner): scope long walk timeouts
* fix(scanner): bound IAM config walks
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
fix(replication): don't silently swallow resync status persistence failure (backlog#799 B23)
After a resync computes a new replication status, it persists it via
`put_object_metadata` but discarded the `Err` case (`if let Ok(u) = ...`). A
failure left the object's on-disk replication status disagreeing with the resync
result with no signal at all. Log the failure at warn level instead.
Refs backlog#799 (B23), tracked in rustfs/backlog#863.
The resync result verification HEADed the target after replicating and counted
the outcome with inverted error handling:
- for a delete marker, ANY HEAD error (timeout, 5xx, auth, malformed) was
counted as replicated (success);
- for a versioned object against an AWS-style target, HEAD was sent with the
RustFS UUID versionId, which AWS rejects with 400, so a well-replicated
object was counted as failed.
Classify the error before counting:
- delete marker: only a definitive 404/NoSuchKey or 405/MethodNotAllowed
confirms the marker propagated (`is_retryable_delete_replication_head_error`
== false); any retryable/ambiguous error now counts as failed;
- versioned object with a version-id-format rejection: re-verify via
`head_object_fallback` (versionId-less HEAD) before deciding — present ->
replicated, absent/error -> failed;
- all other errors: failed, as before.
Reuses the existing, unit-tested classifier helpers. Verified against the
existing resyncer suite (24 tests).
Refs backlog#799 (B13).
During resync against an AWS-style target that rejects RustFS UUID versionIds,
the code retries HEAD without a versionId and compares ETags. On a match it set
`replication_action = None` ("already in sync, nothing to copy") but did not
return, so control fell through to `if replication_action != ReplicationAction::All`
— a branch meant only for the unsupported metadata-only case — and stamped the
object FAILED with "metadata-only replication is not implemented". The target
already held an identical object, yet the source recorded FAILED forever, so
AWS-style targets never converged and the MRF kept re-queuing.
Handle `ReplicationAction::None` explicitly before that branch: record it as
Completed (with the resync timestamp/`replication_resynced` bookkeeping for
ExistingObject + reset_id, mirroring the HEAD-success None path) and return.
Only `ReplicationAction::Metadata` now takes the metadata-unsupported failure
branch; `All` still proceeds to the copy. This path is the only way `None`
reaches that point (the HEAD-success None case already returns earlier).
Refs backlog#799 (B11).
* fix(core-storage): fix critical correctness defects from core-storage audit
Fixes verified defects found in a deep audit of the core storage path
(erasure coding, disk persistence, quorum, heal, replication resync):
- ecstore/disk: rewrite live xl.meta atomically (temp+rename) in
delete_versions_internal and write_metadata instead of in-place
truncate, which exposed torn metadata to concurrent readers and
crashes on the DeleteObjects hot path
- ecstore/erasure: allow heal to reconstruct from exactly data_shards
bitrot-verified sources; requiring data_shards+1 made objects
permanently unhealable after losing parity_shards disks
- ecstore/set_disk: direct-memory inline GET applied the erasure
distribution permutation twice (shuffled inputs re-indexed through
distribution), concatenating wrong shards into the response body in
degraded reads; collect from canonical disk-ordered inputs
- ecstore/set_disk: heal now preserves the committed inline layout
instead of recomputing it with a hardcoded unversioned threshold,
which split quorum identity of healed replicas and caused endless
re-heal churn
- ecstore/replication: resync results channel switched from
broadcast(1) to mpsc; a lagged broadcast receiver ended the stats
collector and every subsequent failure went uncounted, letting
failed resyncs be marked completed
- ecstore/replication: ignore an empty persisted resync checkpoint;
resuming with one skipped every object and marked the resync
completed without replicating anything
- ecstore/replication: fix inverted not-found error classification in
replicate_object/replicate_delete logging paths
- ecstore/erasure: guard decode paths against zero block_size or
data_shards from corrupt on-disk metadata (divide-by-zero panic)
- ecstore/disk: os::read_dir no longer consumes the entry limit on
entries it does not return (is_empty_dir misjudgment); create_file
opens with O_TRUNC to avoid stale trailing bytes
- filemeta: treat Some(nil) version id as a null version in
matches_not_strict; disk-loaded headers never store None, so the
mod_time quorum guard for unversioned overwrites never fired and an
interrupted overwrite could displace the committed version in merge
- filemeta: fix msgpack skip lengths for fixext (missed the ext type
byte) and ext16/32 (over-skipped) unknown fields
- filemeta: return FileCorrupt instead of usize underflow when
xl.meta is truncated inside the CRC trailer
- filemeta: surface delete-marker insertion failure in delete_version
instead of reporting success when the data dir is shared
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(replication): drop duplicate cfg(test) etag import from boundary module
The test module already imports content_matches_by_etag locally, so the
top-level cfg(test) import is unused under -D warnings and fails clippy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
houseme