* test(replication): pin the version-fidelity probe contract (red)
P1-19 (rustfs/backlog#1675 B2): the supported replication contract is
targets that adopt the source version id — a target that mints its own ids
silently breaks every version-addressed operation that follows (version
deletes, heal re-drives never match), diverging the two sides with no
signal. replication-check already captures the probe PUT's response version
id but never compares it.
Red evidence (current main): against a FakeS3Target with
assign_own_version_ids enabled, ?replication-check returns Status "OK" —
the drift is invisible.
test_replication_check_flags_version_minting_target expects a
VersionFidelity phase that fails with the machine-readable code
BucketRemoteTargetVersionMismatch, skips the later mutation phases, and
still cleans up the probe via the version id the target actually assigned.
Test infra: FakeS3Target gains assign_own_version_ids (models a generic S3
service; validated-but-not-mirrored source version headers) and a
prefix+max-keys ListObjectVersions implementation (the probe key allocation
requires it); stored_versions accessor duplicated from the P1-21 branch
(identical code, resolves clean on merge).
* fix(replication): probe the version-identity contract in replication-check
P1-19 (rustfs/backlog#1675 B2, plan B). Replication only converges on
targets that adopt the source version id: version-addressed deletes and
heal re-drives address the source id, so a target that mints its own ids
silently diverges — nothing surfaced this. replication-check already
captured the probe PUT's response version id but never compared it.
- The probe PUT now carries the source version as `?versionId=` (the exact
shape live replication uses since P0-5, and the only shape MinIO
consumes; the internal source-version-id header alone would let the
probe pass against targets the real data path drifts on). Reuses
ecstore's append_version_id_query through the api facade.
- New VersionFidelity phase: the probe PUT's response version id must
equal the sent source id. On mismatch the phase fails with the
machine-readable extension key `"Code": "BucketRemoteTargetVersionMismatch"`
(new optional Code field on phase statuses; Go decoders ignore unknown
keys), the overall target fails, the later version-addressed mutation
phases are skipped, and cleanup still removes the probe via the id the
target actually assigned (with the existing list-based sweep as backstop
when the target returns no version id at all).
- Runtime half: TargetClient::put_object now returns the assigned version
id (mirroring remove_object), and the replication PUT path audits it —
every drifting PUT increments
rustfs_replication_version_identity_drift_total and the first drift per
target ARN logs a structured warning pointing at ?replication-check.
The drift judgment is a pure function with an exemption-matrix test
(empty / literal "null" / nil-uuid sources carry no contract).
- docs/operations/replication-check.md documents the phase and the code.
Red -> green: test_replication_check_flags_version_minting_target (fake
target with assign_own_version_ids; on main the check reported Status
"OK"). The probe's query shape is pinned by a journal assertion (revert
of the query hunk alone fails it), probe-level unit tests cover the
mismatch/mirror matrix including cleanup addressing the minted id, and
the existing success e2e now asserts VersionFidelity OK against a RustFS
target. Adversarial review (seven roles): non-blocking; noted follow-ups
are the multipart runtime audit (the probe phase already pins the
contract) and per-target re-warning after reconfiguration.
* fix(e2e): stop the fake target self-deadlocking on version-id minting
The assign_own_version_ids flag was read with a fresh `lock(&self.store)`
inside two paths that already hold that guard — delete_object's
marker-creation branch and create_multipart_upload — and the store mutex
is not reentrant, so both hung forever (CI: the fake target's own
multipart and delete-marker tests ran >1560s until the job was
cancelled). Read the flag from the live guard instead.
The replication e2e paths did not catch this: a version-addressed purge
DELETE never mints an id, and the probe PUT reads the flag before taking
the guard.
* chore(test): refresh the nextest replication count invariant
The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata
(authority: `cargo nextest list`); refresh it to this branch's
post-rebase total.
* test(replication): pin delayed delete-marker purge failure handling (red)
P1-21 (rustfs/backlog#1675 B2): two failing e2e tests that pin the missing
failure handling of the delayed delete-marker purge:
- test_delayed_delete_marker_purge_retries_after_transient_target_failure:
four scripted 503s outlast every existing channel (version-purge
replication + its in-process MRF fast retries + the watcher's single
attempt = 3 target DELETEs, all faulted in the recorded run); the
replicated marker is stranded on the target forever.
- test_delayed_delete_marker_purge_exhaustion_persists_to_mrf_and_replays_on_restart:
exhausted purge intents never reach the durable MRF journal, so a restart
replays nothing (recorded run: 3 faulted attempts, zero post-restart).
Red-light evidence (current main):
- Test A: FAILED, journal shows 3x DeleteObject fault=Status(503), no clean
attempt, target marker still present after 15s.
- Test B: FAILED after 468s, same 3 faulted attempts, no purge DELETE after
restart, marker still present.
Test infra: FakeS3Target::stored_versions() exposes per-key version state so
purge tests assert target state instead of inferring it from the journal;
nextest count comments 36->38 nightly / 56->58 total.
* fix(replication): retry, persist and replay failed delete-marker purges
P1-21 (rustfs/backlog#1675 B2). The delayed delete-marker purge was
fire-and-forget: the target DELETE discarded its result (`let _ =`), a
missing target client was silently skipped, and nothing recorded the intent
— one transient target error stranded the replicated marker on the target
forever. Separately, `replicate_delete_with_outcome` held its outcome
hostage to `!requires_delayed_purge`, pinning every delete-marker MRF entry
to Missed so the durable backlog retained them permanently.
Changes:
- `replicate_delete_marker_purge_to_targets` now reports per-target
results (warn + metrics on failure, including `target_client_missing`),
supports retrying only the failed targets, and treats a target-side
NoSuchKey/NoSuchVersion as purge success (strict-404 targets must not
retain the intent forever).
- The delayed watcher (`watch_and_purge_source_delete_marker`) retries
failed targets across its 5x1s watch window; on exhaustion it persists
the purge intent to the durable MRF journal via the new
`ReplicationPoolTrait::persist_mrf_entry` (journal-only on purpose: live
re-dispatch would loop unboundedly against a down target). Intent entries
are shaped as marker-creation deletes so replay funnels into the stale-
marker branch.
- The stale-marker branch (source marker already gone) now purges the
targets instead of silently returning success — closing a latent leak —
and reports the purge result as the replay outcome. Heal callers retry
for the full window (the startup MRF processor runs before target
clients initialize); live callers attempt once and fall back to a fresh
durable intent, so a down target cannot pin a replication worker.
- The outcome formula (extracted as `replicate_delete_outcome` and pinned
by a unit test) no longer includes the delayed purge, so successfully
replayed delete-marker entries are acknowledged instead of retained
forever.
Verification: red -> green e2e pair (transient-failure retry; exhaustion ->
durable MRF -> restart replay -> second-restart zero-replay ack) plus unit
tests; `make pre-commit`, logging guardrails, clippy (ecstore + e2e_test)
all clean; full ecstore lib suite 3729 passed (3 pre-existing local-DNS
kubernetes endpoint failures reproduce without this change).
Adversarial validation (7 roles): no blocking findings after adding the
outcome-formula guard test. Known residuals recorded in the PR: watcher
shutdown window (intent not yet persisted), rolling-downgrade replay acks
without purging (equals pre-fix behavior), and replay falling back to the
source version id on targets that mint their own version ids (P1-19).
* chore(test): refresh the nextest replication count invariant
The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata
(authority: `cargo nextest list`); refresh it to this branch's
post-rebase total.
* fix(replication): purge the marker version the target actually assigned
Review follow-up (#5864), two real defects:
- The delayed purge watcher was spawned with the pre-merge `dobj`, so the
per-target marker version ids this round recorded were invisible to it.
Against a target that mints its own ids the purge fell back to a
source-derived id, the target answered the versioned DELETE with an
idempotent 204, and that "success" cleared the retry set while the real
marker stayed behind. The watcher now receives the merged replication
state (`drs`), which folds this round's target-assigned ids in.
- A target whose recorded version metadata is inconsistent was skipped
without entering `failed_arns`, so an empty result made both the watcher
and the MRF replay treat a purge that issued no DELETE as successful and
drop the intent. The refusal is now a per-target failure (own metric
label): the leak stays visible and the intent is retained instead of
being acknowledged. The version decision also moved ahead of the client
lookup, so the refusal is decided from metadata alone.
Tests: a new e2e drives a fake target with `assign_own_version_ids`, which
ignores the forwarded source-version header for both objects and delete
markers, and asserts the replicated marker is really gone; a unit test
pins the corrupt-metadata refusal as a failed outcome without any target
client registered. The detached-watcher shutdown window is documented at
the watcher as a known non-durable window with the write-ahead follow-up
spelled out.
唐小鸭