Treat recovery-directory lookup on a replacement endpoint already deferred as replacement_path_unavailable as an expected debug diagnostic instead of a durable generation conflict.
Keep real survivor recovery conflicts and corrupt records on the existing warning/blocking path.
Co-authored-by: heihutu <heihutu@gmail.com>
* test(heal): add privileged replacement rebuild e2e
Add ignored Linux-only 3x4 automatic replacement coverage for EC8+4 and EC6+6. The tests use real tmpfs mounts in an isolated mount namespace, wait for scanner-driven replacement recovery status, and verify the replacement target with per-version xl.meta and part.N physical census without invoking Admin deep heal.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(test): avoid unsafe in privileged replacement e2e
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(heal): harden privileged replacement e2e
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(heal): prove absent replacement recovery witness
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): prove absent replacement observation
Stop the target node before detaching the test mount so RustFS releases its mount lease instead of continuing to serve the old tmpfs through an open fd. Restart the node with the endpoint absent and wait for the scanner's real readiness rejection in that node's log.
Assert the absent window has no replacement intent, completion proof, checkpoint, healing marker, or Admin v4 durable record for the target before mounting the blank replacement and waiting for automatic recovery plus physical shard census.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): streamline cluster log capture
Move cluster-node log capture out of ClusterNode and into per-node cluster launch configuration so the privileged replacement E2E uses an explicit harness API instead of mutating node identity data.
Reuse the same stdout/stderr capture helper for single-node and cluster processes, and pin the per-node capture behavior with a focused common test.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): harden privileged replacement proof
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* test(site-replication): pin retry-event lost-update against locked RMW (red)
P1-15 (rustfs/backlog#1675 B2): the site-replication retry-event writers
(enqueue/dequeue, which hang off every hook broadcast path) perform a
load -> mutate -> persist without taking SITE_REPLICATION_STATE_LOCK, so
a single process can lose a concurrent lock-holding writer's update; the
service-side reload path is equally unlocked, and no writer holds a
distributed lock across the read-modify-write, so multi-node RMW loses
updates even where the process lock is held.
Red evidence (current main): replaying enqueue's exact three steps around
a completed mark_pending_rotation_peer_acked commit wipes the rotation
ack — the final state holds the retry event but not the ack.
* fix(site-replication): route state RMW through one locked transaction
P1-15 PR1 (rustfs/backlog#1675 B2). The site-replication state object
(config/site-replication/state.json, which also carries the retry-event
queue) was mutated through read-modify-write sequences with inconsistent
locking: the retry-event writers on every hook broadcast path and the
RPC-driven service reload took no lock at all (single-process lost
updates, pinned by the red commit), and no writer held a distributed lock
across the whole RMW (cross-node lost updates everywhere).
- New admin/site_replication_state module: the state transaction boundary
`with_site_replication_state_lock[_on]` — process mutex plus the
distributed config-object write lock (the pattern proven by the repair
state), with the shared path constant. The process mutex is transitional
until PR2 migrates the remaining ~26 call sites.
- handlers: typed `update_site_replication_state` (no-lock load /
persist-or-clear inside the boundary; normalizes the peer map exactly
once, retiring the double-clone/double-normalize persist path, P2-22).
Migrated: retry-event enqueue (always-write), dequeue (lock-free probe,
transaction on hit), mark_pending_rotation/remove_peer_acked.
- service reload: the tolerant byte-level read->normalize->save now runs
inside the same boundary via no-lock IO — a cluster-wide reload fan-out
can no longer overwrite a concurrent state writer. Normalization
semantics untouched (all six service-side tests unchanged and green).
- Add/PeerJoin/Edit handlers release the state guard before their peer
fan-out: the transport helpers' retry-event bookkeeping now re-enters
the state transaction and must not nest inside the guard (the
adversarial review caught this as a re-entrancy deadlock; the fix
mirrors the Remove/Rotate handlers' existing scope). The Edit non-
refresh branch commits before fanning out — the old fanout-first order
recorded retry events pointing at a state the local site had not saved.
- ecstore: delete_config_no_lock (+ facade/bridge exports) so the clear
half of persist-or-clear works under the held object lock.
Red -> green: the red commit pinned the deterministic lost-update
interleaving (stale retry-event persist wiping a committed rotation ack);
the test now drives the real functions concurrently for 8 rounds and
asserts every retry event and every ack survives. Full
handlers/service site-replication unit suites green (171 + 6); dual-node
site-replication e2e (state edit fresh/stale, object replication) green;
fmt / clippy / logging guardrails clean.
Adversarial review: one blocking finding (the re-entrancy deadlock above)
fixed and re-verified by a full second pass over all 30 lock sites and
the Add/Join/Edit call graphs. Non-blocking notes recorded for PR2:
mark_* now persists on miss (persist-or-clear semantics; a miss-skip
return is a cheap follow-up), Add still holds the guard across the peer
join probe (pre-existing availability debt), and a timeout-guarded
unreachable-peer regression test for the fan-out paths.
* fix(site-replication): keep the state mutex behind an owner helper
CI's architecture migration guard lists SITE_REPLICATION_STATE_LOCK as an
owner-local static, so it may not be `pub(crate)`. Keep it private to the
new module and let the not-yet-migrated RMW call sites take it through
`site_replication_state_process_guard()` — the sanctioned owner-helper
pattern; the helper disappears with the mutex in PR2.
* fix(site-replication): keep peer-edit delivery under the state guard
Review follow-up (#5882).
Releasing the guard before the fan-out (my deadlock fix) traded the
ordering the guard used to provide: edit A could commit and stall while
edit B committed and reached a peer first, then A arrived last and won.
The peer edit handler applies whatever arrives — it has no generation or
updated-at fence — and a successful stale delivery is not repaired by the
retry queue, so the sites diverge silently.
The fan-out is back under the guard. What actually could not run there is
the retry-event bookkeeping, which re-enters the state transaction, so the
edit branch now delivers with the plain transport and settles the retry
queue after the guard is released: successes dequeue, the first failure
enqueues and is returned. Ordering and bookkeeping both preserved. The add
handler keeps its peer-edit finalize fan-out under the guard for the same
reason and releases only before bootstrap/back-fill, which send bucket-ops
(not peer edits) through retry-event transports.
The concurrency test could not tell the two guards apart — both writers
took both locks, so it passed with either removed. Replaced by two tests
that isolate one guard each, both verified by mutation:
- a process-only legacy writer (the shape the not-yet-migrated call sites
still use) racing the transaction: fails when the transaction stops
taking the process mutex;
- two writers that bypass the process mutex, as separate nodes do, driving
the production object-lock path (`with_site_replication_state_object_lock`
factored out for exactly this): fails when the distributed lock is
removed.
Verification: handlers 173 + service 6 unit tests green; site-replication
dual-node and three-node edit e2e green; arch/layer/logging guardrails,
fmt and clippy clean.
* fix(site-replication): fence peer-edit delivery by generation
Review follow-up on the two remaining holes in the edit path.
Ordering was only process-local. `SITE_REPLICATION_STATE_LOCK` is per
node, so holding it across the fan-out orders the edits ONE node accepts
and nothing else: two nodes of the same site can both commit and reach a
peer in the opposite order, and the peer edit handler applied whatever
arrived last. Each edit now takes a generation from
`SiteReplicationState::edit_generation`, allocated in the same commit as
the edit itself — i.e. under the distributed state-object lock, so two
nodes can never share one. The generation rides the peer-edit request as
query parameters and the receiver rejects (acks without applying) a
delivery at or below the mark it already applied for that origin site,
recording the mark in the same commit as the edit it fences. Peers that
predate the fence send no parameters and are applied as before.
Retry settlement could discard a newer failure. After the guard is
released, a success for edit A removed every retry event for
(peer, peer-edit): if edit B committed, failed its own delivery and
enqueued while A was in flight, A erased it — local state B, peer on A,
nothing queued to converge them. Settlement now only removes events whose
recorded generation is not newer than the one being settled, and a later
failure never lowers the fence. Broadcast paths carry no generation and
settle unconditionally as before; their events live under their own
paths and cannot collide with a peer-edit delivery.
A departed peer's mark is dropped on load: a site that leaves drops below
two peers, which clears its state object and restarts its counter at
zero, so a leftover mark would reject every edit it sends after it
rejoins.
Tests: two-node generation uniqueness (drop the object lock and the two
nodes collide), the receiver's staleness predicate and its wiring, the
settlement interleaving (drop the fence and B's retry is erased), and the
rejoin reset.
Refs: rustfs/backlog#1675 (P1-15)
P1-20 (rustfs/backlog#1675 B2, test-only). No prior test wrote objects
BEFORE the replication rule arrived, leaving the scanner's existing-object
resync pass — the only channel for such objects — without end-to-end
coverage, and the enqueue truth table partially unpinned at unit level.
e2e (both negative cells are contracts, asserted over multiple fast-scanner
cycles next to a replicated control key that proves the scanner and the
live path are running):
- test_scanner_compensates_existing_objects_across_write_paths: plain PUT,
CopyObject and Snowball auto-extract products written pre-rule all
converge via scanner compensation; a null-version object (PUT before the
bucket became versioned) is pinned as never compensated (the scanner heal
gate skips nil-version objects).
- test_scanner_never_compensates_when_existing_object_replication_disabled:
ExistingObjectReplication=Disabled is a contract, not a delay — existing
keys stay absent while post-rule writes replicate normally.
Unit truth-table pins (crates/replication):
- queue.rs: an empty replicate decision (Disabled existing-object, inbound
REPLICA) skips heal queueing for every status; Completed without a resync
decision skips.
- operation.rs: existing-object resync without a reset replicates exactly
the never-replicated (Empty) objects.
Helper: put_bucket_replication_with_statuses parameterizes the previously
hardcoded ExistingObjectReplication status; the nextest count comments are
refreshed to the post-rebase totals.
Increase the replay cache resource model so 16 CPU / 31-32 GiB field nodes auto-size to the 32M cap without an env override.
Co-authored-by: heihutu <heihutu@gmail.com>
* test(replication): pin the version-fidelity probe contract (red)
P1-19 (rustfs/backlog#1675 B2): the supported replication contract is
targets that adopt the source version id — a target that mints its own ids
silently breaks every version-addressed operation that follows (version
deletes, heal re-drives never match), diverging the two sides with no
signal. replication-check already captures the probe PUT's response version
id but never compares it.
Red evidence (current main): against a FakeS3Target with
assign_own_version_ids enabled, ?replication-check returns Status "OK" —
the drift is invisible.
test_replication_check_flags_version_minting_target expects a
VersionFidelity phase that fails with the machine-readable code
BucketRemoteTargetVersionMismatch, skips the later mutation phases, and
still cleans up the probe via the version id the target actually assigned.
Test infra: FakeS3Target gains assign_own_version_ids (models a generic S3
service; validated-but-not-mirrored source version headers) and a
prefix+max-keys ListObjectVersions implementation (the probe key allocation
requires it); stored_versions accessor duplicated from the P1-21 branch
(identical code, resolves clean on merge).
* fix(replication): probe the version-identity contract in replication-check
P1-19 (rustfs/backlog#1675 B2, plan B). Replication only converges on
targets that adopt the source version id: version-addressed deletes and
heal re-drives address the source id, so a target that mints its own ids
silently diverges — nothing surfaced this. replication-check already
captured the probe PUT's response version id but never compared it.
- The probe PUT now carries the source version as `?versionId=` (the exact
shape live replication uses since P0-5, and the only shape MinIO
consumes; the internal source-version-id header alone would let the
probe pass against targets the real data path drifts on). Reuses
ecstore's append_version_id_query through the api facade.
- New VersionFidelity phase: the probe PUT's response version id must
equal the sent source id. On mismatch the phase fails with the
machine-readable extension key `"Code": "BucketRemoteTargetVersionMismatch"`
(new optional Code field on phase statuses; Go decoders ignore unknown
keys), the overall target fails, the later version-addressed mutation
phases are skipped, and cleanup still removes the probe via the id the
target actually assigned (with the existing list-based sweep as backstop
when the target returns no version id at all).
- Runtime half: TargetClient::put_object now returns the assigned version
id (mirroring remove_object), and the replication PUT path audits it —
every drifting PUT increments
rustfs_replication_version_identity_drift_total and the first drift per
target ARN logs a structured warning pointing at ?replication-check.
The drift judgment is a pure function with an exemption-matrix test
(empty / literal "null" / nil-uuid sources carry no contract).
- docs/operations/replication-check.md documents the phase and the code.
Red -> green: test_replication_check_flags_version_minting_target (fake
target with assign_own_version_ids; on main the check reported Status
"OK"). The probe's query shape is pinned by a journal assertion (revert
of the query hunk alone fails it), probe-level unit tests cover the
mismatch/mirror matrix including cleanup addressing the minted id, and
the existing success e2e now asserts VersionFidelity OK against a RustFS
target. Adversarial review (seven roles): non-blocking; noted follow-ups
are the multipart runtime audit (the probe phase already pins the
contract) and per-target re-warning after reconfiguration.
* fix(e2e): stop the fake target self-deadlocking on version-id minting
The assign_own_version_ids flag was read with a fresh `lock(&self.store)`
inside two paths that already hold that guard — delete_object's
marker-creation branch and create_multipart_upload — and the store mutex
is not reentrant, so both hung forever (CI: the fake target's own
multipart and delete-marker tests ran >1560s until the job was
cancelled). Read the flag from the live guard instead.
The replication e2e paths did not catch this: a version-addressed purge
DELETE never mints an id, and the probe PUT reads the flag before taking
the guard.
* chore(test): refresh the nextest replication count invariant
The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata
(authority: `cargo nextest list`); refresh it to this branch's
post-rebase total.
* fix(ecstore): publish multipart parts on Windows
* test(ecstore): pin Windows multipart durability
---------
Co-authored-by: houseme <housemecn@gmail.com>
Keep opt-in bounded GET data-read fanout from waiting on a single pending ReadVersion response when an unscheduled spare disk can satisfy quorum. Add a deterministic 2+2 regression that pauses the third scheduled metadata read and verifies the spare is started before returning.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Assert the prepared GET metadata path keeps the default full data-read fanout after PR #5929 made bounded data-read fanout opt-in.
Co-authored-by: heihutu <heihutu@gmail.com>
The SNI preservation test is the only object-lambda test doing a real
TLS handshake; the shared helper's 2s whole-request timeout turns
concurrent fsync-heavy TestECStoreEnv neighbors into a deterministic
TimedOut when the per-build nextest schedule overlaps them. The test
verifies SNI, not latency, so widen its budget to a still-bounded 30s.
* test(replication): pin delayed delete-marker purge failure handling (red)
P1-21 (rustfs/backlog#1675 B2): two failing e2e tests that pin the missing
failure handling of the delayed delete-marker purge:
- test_delayed_delete_marker_purge_retries_after_transient_target_failure:
four scripted 503s outlast every existing channel (version-purge
replication + its in-process MRF fast retries + the watcher's single
attempt = 3 target DELETEs, all faulted in the recorded run); the
replicated marker is stranded on the target forever.
- test_delayed_delete_marker_purge_exhaustion_persists_to_mrf_and_replays_on_restart:
exhausted purge intents never reach the durable MRF journal, so a restart
replays nothing (recorded run: 3 faulted attempts, zero post-restart).
Red-light evidence (current main):
- Test A: FAILED, journal shows 3x DeleteObject fault=Status(503), no clean
attempt, target marker still present after 15s.
- Test B: FAILED after 468s, same 3 faulted attempts, no purge DELETE after
restart, marker still present.
Test infra: FakeS3Target::stored_versions() exposes per-key version state so
purge tests assert target state instead of inferring it from the journal;
nextest count comments 36->38 nightly / 56->58 total.
* fix(replication): retry, persist and replay failed delete-marker purges
P1-21 (rustfs/backlog#1675 B2). The delayed delete-marker purge was
fire-and-forget: the target DELETE discarded its result (`let _ =`), a
missing target client was silently skipped, and nothing recorded the intent
— one transient target error stranded the replicated marker on the target
forever. Separately, `replicate_delete_with_outcome` held its outcome
hostage to `!requires_delayed_purge`, pinning every delete-marker MRF entry
to Missed so the durable backlog retained them permanently.
Changes:
- `replicate_delete_marker_purge_to_targets` now reports per-target
results (warn + metrics on failure, including `target_client_missing`),
supports retrying only the failed targets, and treats a target-side
NoSuchKey/NoSuchVersion as purge success (strict-404 targets must not
retain the intent forever).
- The delayed watcher (`watch_and_purge_source_delete_marker`) retries
failed targets across its 5x1s watch window; on exhaustion it persists
the purge intent to the durable MRF journal via the new
`ReplicationPoolTrait::persist_mrf_entry` (journal-only on purpose: live
re-dispatch would loop unboundedly against a down target). Intent entries
are shaped as marker-creation deletes so replay funnels into the stale-
marker branch.
- The stale-marker branch (source marker already gone) now purges the
targets instead of silently returning success — closing a latent leak —
and reports the purge result as the replay outcome. Heal callers retry
for the full window (the startup MRF processor runs before target
clients initialize); live callers attempt once and fall back to a fresh
durable intent, so a down target cannot pin a replication worker.
- The outcome formula (extracted as `replicate_delete_outcome` and pinned
by a unit test) no longer includes the delayed purge, so successfully
replayed delete-marker entries are acknowledged instead of retained
forever.
Verification: red -> green e2e pair (transient-failure retry; exhaustion ->
durable MRF -> restart replay -> second-restart zero-replay ack) plus unit
tests; `make pre-commit`, logging guardrails, clippy (ecstore + e2e_test)
all clean; full ecstore lib suite 3729 passed (3 pre-existing local-DNS
kubernetes endpoint failures reproduce without this change).
Adversarial validation (7 roles): no blocking findings after adding the
outcome-formula guard test. Known residuals recorded in the PR: watcher
shutdown window (intent not yet persisted), rolling-downgrade replay acks
without purging (equals pre-fix behavior), and replay falling back to the
source version id on targets that mint their own version ids (P1-19).
* chore(test): refresh the nextest replication count invariant
The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata
(authority: `cargo nextest list`); refresh it to this branch's
post-rebase total.
* fix(replication): purge the marker version the target actually assigned
Review follow-up (#5864), two real defects:
- The delayed purge watcher was spawned with the pre-merge `dobj`, so the
per-target marker version ids this round recorded were invisible to it.
Against a target that mints its own ids the purge fell back to a
source-derived id, the target answered the versioned DELETE with an
idempotent 204, and that "success" cleared the retry set while the real
marker stayed behind. The watcher now receives the merged replication
state (`drs`), which folds this round's target-assigned ids in.
- A target whose recorded version metadata is inconsistent was skipped
without entering `failed_arns`, so an empty result made both the watcher
and the MRF replay treat a purge that issued no DELETE as successful and
drop the intent. The refusal is now a per-target failure (own metric
label): the leak stays visible and the intent is retained instead of
being acknowledged. The version decision also moved ahead of the client
lookup, so the refusal is decided from metadata alone.
Tests: a new e2e drives a fake target with `assign_own_version_ids`, which
ignores the forwarded source-version header for both objects and delete
markers, and asserts the replicated marker is really gone; a unit test
pins the corrupt-metadata refusal as a failed outcome without any target
client registered. The detached-watcher shutdown window is documented at
the watcher as a known non-durable window with the write-ahead follow-up
spelled out.
Keep GET data-read metadata early-stop and bounded fanout behind explicit environment switches so the default path preserves full fanout read-failure tolerance.
Retain the focused opt-in A/B coverage and the invalid parity full-fanout guard for heterogeneous set layouts.
Co-authored-by: heihutu <heihutu@gmail.com>
The RPM build step fails because fpm's --config-files flag requires
/etc/default/rustfs to exist in the staging area, but unlike the DEB
build (which creates it in its package directory structure), the fpm
command has no prior step creating this file.
Create the config file in a temporary directory and pass it to fpm
via a source=dest mapping, matching the DEB build's behavior.
Track accepted replay cache records by gRPC operation and split Lock/Unlock and ReadVersion methods out of grpc_other so hotpath validation can attribute nonce pressure without changing replay protection semantics.
Co-authored-by: heihutu <heihutu@gmail.com>
Add a replacement recovery peer RPC so Admin v4 can distinguish definitive cluster proofs from unsupported, unavailable, or conflicting peer state without extending the existing background heal v3/v1 status protocol.
Co-authored-by: heihutu <heihutu@gmail.com>
The DEB version substitution used ${VERSION/-/~} which caused bash
to expand ~ to $HOME (e.g. /home/runner), producing an invalid
version string like '1.0.0/home/runnerrc.1'.
Store ~ in a variable first to prevent tilde expansion.
Add a v4 admin status endpoint for local durable automatic replacement recovery records without changing the v3 background heal status or peer v1 payloads.
Co-authored-by: heihutu <heihutu@gmail.com>
Three Kubernetes endpoint-identity tests read the real kernel hostname
and panicked when it is an IP literal (e.g. macOS without a static
HostName, where DHCP/reverse-DNS sets the kernel hostname to an address
like 192.168.1.11).
Add a cfg(test) override seam (force_kernel_hostname_for_test, mirroring
the existing force_local_host_resolution_timeout_for_test pattern) and
route the production read through kernel_hostname_for_endpoint_identity()
so the tests inject deterministic hostnames instead of depending on the
host environment. Production behavior is unchanged.
Complete the encrypted-object replication series (backlog#1783, PR-C of
3, after #5872 and #5885): SSE-C objects replicate as ciphertext
passthrough — the source holds no customer key, so the stored bytes and
their encryption metadata travel verbatim and the replica decrypts only
with the original customer key, single-part and multipart.
- Sender: SSE-C objects read raw (raw_data_movement_read), transfer at
ciphertext size, and range multipart parts over stored part sizes.
- Receiver: authorized replication PUTs restore the stored SSE-C keys
from the transport headers (exact lowercase forms - the read-path
check is case-sensitive), set ObjectOptions.preserve_ciphertext, and
skip compression, bucket-default SSE, and sse_encryption behind one
restore-derived gate. Multipart uses an internal session marker to
store parts verbatim and strips it on complete.
- Convergence: the replication HEAD sends
x-rustfs-source-replication-check; the target authorizes it as
ReplicateObjectAction and skips SSE-C read validation for that
request only, so keyless convergence HEADs see etag/size/mtime
instead of 400 and SSE-C replicas stop re-driving forever.
- e2e: SSE-C contract flips to a key-gated readable replica (no-key and
wrong-key GETs fail - the direct silent-plaintext detector); new
multipart passthrough contract with ETag/marker/stability assertions.
* fix(ecstore): anchor Windows rename publication
* fix(ecstore): complete Windows rename confinement
* test(ecstore): retain Windows retry assertion path
* fix(ecstore): accept configured Windows root paths
* fix(ecstore): size Windows rename buffers correctly
* fix(ecstore): use native relative rename on Windows
* fix(ecstore): preserve Windows rename parent guards
* fix(ecstore): reuse guarded Windows rename trees
* fix(ecstore): compile Windows publication helpers
* fix(ecstore): preserve configured Windows disk roots
* fix(ecstore): flush Windows shards with write access
* fix(ecstore): stage Windows rollback backup replacement
* fix(ecstore): defer Windows staged file cleanup
* fix(ecstore): type Windows staged write result
* fix(ecstore): retry Windows sharing violations
* fix(ecstore): share Windows staged deletes
* fix(ecstore): split Windows staged publication handles
* fix(ecstore): close Windows staged writer before rename
* fix(ecstore): share Windows staged publication deletes
* fix(ecstore): allow guarded Windows child publication
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Install tzdata in both published runtime variants and verify IANA timezone resolution during image builds.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Open the managed-SSE replication gate (backlog#1783, PR-B of 3, after
#5872): the replication reader already decrypts through the injected
object-encryption resolver, so the source sends plaintext plus an
encryption intent header (AES256 / aws:kms, never the source key id) and
the target re-encrypts on its normal PUT path with its own KMS. No DEK
crosses sites.
- replication_put_object_options: fail closed only on Unsupported;
insert the SSE intent after the strip loop.
- TargetClient::create_multipart_upload sends the full opts.header()
set, fixing multipart replicas losing content-type/user metadata
(plaintext included).
- Preserve source ETag and mtime on replicas (authorized replication
only): receiver wires x-rustfs-source-etag into preserve_etag for PUT
and CompleteMultipartUpload, resolve_complete_etag consumes it, and
complete options carry source_etag/source_mtime (absent mtime
degrades to epoch, not now_utc). Without this every replication HEAD
comparison re-drives re-encrypted objects forever.
- e2e: managed SSE contracts flip to success on an independent-KMS
dual-process pair (byte-identical plain GET proves target-owned
envelopes; ETag/mtime preserved; version stable across scanner
cycles; resync converges; multipart keeps structure and metadata);
new target-without-KMS fail-closed contract; SSE-C stays FAILED.
Co-authored-by: houseme <housemecn@gmail.com>
Henry Guo