fix(replication): retry, persist and replay failed delete-marker purges (#5864)

* test(replication): pin delayed delete-marker purge failure handling (red) P1-21 (rustfs/backlog#1675 B2): two failing e2e tests that pin the missing failure handling of the delayed delete-marker purge: - test_delayed_delete_marker_purge_retries_after_transient_target_failure: four scripted 503s outlast every existing channel (version-purge replication + its in-process MRF fast retries + the watcher's single attempt = 3 target DELETEs, all faulted in the recorded run); the replicated marker is stranded on the target forever. - test_delayed_delete_marker_purge_exhaustion_persists_to_mrf_and_replays_on_restart: exhausted purge intents never reach the durable MRF journal, so a restart replays nothing (recorded run: 3 faulted attempts, zero post-restart). Red-light evidence (current main): - Test A: FAILED, journal shows 3x DeleteObject fault=Status(503), no clean attempt, target marker still present after 15s. - Test B: FAILED after 468s, same 3 faulted attempts, no purge DELETE after restart, marker still present. Test infra: FakeS3Target::stored_versions() exposes per-key version state so purge tests assert target state instead of inferring it from the journal; nextest count comments 36->38 nightly / 56->58 total. * fix(replication): retry, persist and replay failed delete-marker purges P1-21 (rustfs/backlog#1675 B2). The delayed delete-marker purge was fire-and-forget: the target DELETE discarded its result (`let _ =`), a missing target client was silently skipped, and nothing recorded the intent — one transient target error stranded the replicated marker on the target forever. Separately, `replicate_delete_with_outcome` held its outcome hostage to `!requires_delayed_purge`, pinning every delete-marker MRF entry to Missed so the durable backlog retained them permanently. Changes: - `replicate_delete_marker_purge_to_targets` now reports per-target results (warn + metrics on failure, including `target_client_missing`), supports retrying only the failed targets, and treats a target-side NoSuchKey/NoSuchVersion as purge success (strict-404 targets must not retain the intent forever). - The delayed watcher (`watch_and_purge_source_delete_marker`) retries failed targets across its 5x1s watch window; on exhaustion it persists the purge intent to the durable MRF journal via the new `ReplicationPoolTrait::persist_mrf_entry` (journal-only on purpose: live re-dispatch would loop unboundedly against a down target). Intent entries are shaped as marker-creation deletes so replay funnels into the stale- marker branch. - The stale-marker branch (source marker already gone) now purges the targets instead of silently returning success — closing a latent leak — and reports the purge result as the replay outcome. Heal callers retry for the full window (the startup MRF processor runs before target clients initialize); live callers attempt once and fall back to a fresh durable intent, so a down target cannot pin a replication worker. - The outcome formula (extracted as `replicate_delete_outcome` and pinned by a unit test) no longer includes the delayed purge, so successfully replayed delete-marker entries are acknowledged instead of retained forever. Verification: red -> green e2e pair (transient-failure retry; exhaustion -> durable MRF -> restart replay -> second-restart zero-replay ack) plus unit tests; `make pre-commit`, logging guardrails, clippy (ecstore + e2e_test) all clean; full ecstore lib suite 3729 passed (3 pre-existing local-DNS kubernetes endpoint failures reproduce without this change). Adversarial validation (7 roles): no blocking findings after adding the outcome-formula guard test. Known residuals recorded in the PR: watcher shutdown window (intent not yet persisted), rolling-downgrade replay acks without purging (equals pre-fix behavior), and replay falling back to the source version id on targets that mint their own version ids (P1-19). * chore(test): refresh the nextest replication count invariant The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata (authority: `cargo nextest list`); refresh it to this branch's post-rebase total. * fix(replication): purge the marker version the target actually assigned Review follow-up (#5864), two real defects: - The delayed purge watcher was spawned with the pre-merge `dobj`, so the per-target marker version ids this round recorded were invisible to it. Against a target that mints its own ids the purge fell back to a source-derived id, the target answered the versioned DELETE with an idempotent 204, and that "success" cleared the retry set while the real marker stayed behind. The watcher now receives the merged replication state (`drs`), which folds this round's target-assigned ids in. - A target whose recorded version metadata is inconsistent was skipped without entering `failed_arns`, so an empty result made both the watcher and the MRF replay treat a purge that issued no DELETE as successful and drop the intent. The refusal is now a per-target failure (own metric label): the leak stays visible and the intent is retained instead of being acknowledged. The version decision also moved ahead of the client lookup, so the refusal is decided from metadata alone. Tests: a new e2e drives a fake target with `assign_own_version_ids`, which ignores the forwarded source-version header for both objects and delete markers, and asserts the replicated marker is really gone; a unit test pins the corrupt-metadata refusal as a failed outcome without any target client registered. The detached-watcher shutdown window is documented at the watcher as a known non-durable window with the write-ahead follow-up spelled out.
2026-08-12 16:16:55 +00:00 · 2026-08-10 22:16:21 +08:00
parent fe2516ee86
commit 3c31eaf06f
5 changed files with 791 additions and 46 deletions
@@ -2567,6 +2567,12 @@ pub trait ReplicationPoolTrait: std::fmt::Debug {
    async fn queue_replica_task(&self, ri: ReplicateObjectInfo) -> ReplicationQueueAdmission;
    async fn queue_replica_delete_task(&self, ri: DeletedObjectReplicationInfo) -> ReplicationQueueAdmission;
    async fn queue_replica_delete_batch(&self, deletes: &[DeletedObjectReplicationInfo]) -> ReplicationBatchAdmission;
+    /// Persist one entry straight to the durable MRF journal, bypassing the
+    /// live worker queues. For failures whose source state is already gone —
+    /// e.g. exhausted delete-marker purges — where only a startup replay can
+    /// retry, and live re-dispatch would loop unboundedly against a down
+    /// target.
+    async fn persist_mrf_entry(&self, entry: MrfReplicateEntry) -> ReplicationQueueAdmission;
    async fn resize(&self, priority: ReplicationPriority, max_workers: usize, max_l_workers: usize);
    async fn get_bucket_resync_status(&self, bucket: &str) -> Result<BucketReplicationResyncStatus, EcstoreError>;
    async fn cancel_bucket_resync(&self, opts: ResyncOpts) -> Result<(), EcstoreError>;
@@ -2607,6 +2613,10 @@ impl<S: ReplicationStorage> ReplicationPoolTrait for ReplicationPool<S> {
        self.queue_replica_delete_batch(deletes).await
    }

+    async fn persist_mrf_entry(&self, entry: MrfReplicateEntry) -> ReplicationQueueAdmission {
+        self.queue_mrf_save_admission(entry, "delete_marker_purge").await
+    }
+
    async fn resize(&self, priority: ReplicationPriority, max_workers: usize, max_l_workers: usize) {
        self.resize(priority, max_workers, max_l_workers).await;
    }
@@ -18,9 +18,10 @@ use super::replication_config_store::ReplicationConfigStore;
 use super::replication_error_boundary::{Result, is_err_object_not_found, is_err_version_not_found};
 use super::replication_event_sink::{EventArgs, send_event, send_local_event};
 use super::replication_filemeta_boundary::{
-    NULL_VERSION_ID, REPLICATE_EXISTING, REPLICATE_EXISTING_DELETE, ReplicateDecision, ReplicateObjectInfo, ReplicatedInfos,
-    ReplicatedTargetInfo, ReplicationAction, ReplicationState, ReplicationStatusType, ReplicationType, VersionPurgeStatusType,
-    get_replication_state, parse_replicate_decision, replication_statuses_map, target_reset_header, version_purge_statuses_map,
+    MrfReplicateEntry, NULL_VERSION_ID, REPLICATE_EXISTING, REPLICATE_EXISTING_DELETE, ReplicateDecision, ReplicateObjectInfo,
+    ReplicatedInfos, ReplicatedTargetInfo, ReplicationAction, ReplicationState, ReplicationStatusType, ReplicationType,
+    ReplicationWorkerOperation, VersionPurgeStatusType, get_replication_state, parse_replicate_decision,
+    replication_statuses_map, target_reset_header, version_purge_statuses_map,
 };
 use super::replication_lock_boundary::ReplicationLockTiming;
 use super::replication_logging::{EVENT_RESYNC_CONFIG_LOOKUP_SKIPPED, LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_REPLICATION_RESYNC};
@@ -33,7 +34,7 @@ use super::replication_object_decision_boundary::{
    is_retryable_delete_replication_head_error, is_version_delete_replication, replication_etags_match,
    replication_multipart_complete_actual_size, replication_multipart_part_plan, should_retry_delete_marker_purge,
 };
-use super::replication_queue_boundary::DeletedObjectReplicationInfo;
+use super::replication_queue_boundary::{DeletedObjectReplicationInfo, ReplicationQueueAdmission};
 use super::replication_resync_boundary::ResyncStatusType;
 use super::replication_resync_boundary::{
    BucketReplicationResyncStatus, ResyncOpts, TargetReplicationResyncStatus, encode_resync_file, is_version_id_mismatch,
@@ -63,6 +64,7 @@ use futures::stream::StreamExt;
 use http::HeaderMap;
 use http_body::Frame;
 use http_body_util::StreamBody;
+use metrics::counter;
 #[cfg(test)]
 use rmp_serde;
 use rustfs_s3_types::EventName;
@@ -100,6 +102,9 @@ const EVENT_REPLICATION_FORCE_DELETE_SKIPPED: &str = "replication_force_delete_s
 const EVENT_RESYNC_TASK_FAILED: &str = "replication_resync_task_failed";
 const EVENT_RESYNC_TARGET_OPERATION_FAILED: &str = "replication_resync_target_operation_failed";
 const EVENT_RESYNC_RUNTIME_CHANNEL_FAILED: &str = "replication_resync_runtime_channel_failed";
+const EVENT_DELETE_MARKER_PURGE_FAILED: &str = "replication_delete_marker_purge_failed";
+const EVENT_DELETE_MARKER_PURGE_MRF: &str = "replication_delete_marker_purge_mrf";
+const METRIC_DELETE_MARKER_PURGE_TOTAL: &str = "rustfs_replication_delete_marker_purge_total";
 const REPLICATION_TARGET_OFFLINE_ERROR_MARKERS: &[&str] = &[
    "dispatch failure",
    "timeouterror",
@@ -1271,7 +1276,12 @@ pub(crate) async fn replicate_delete_with_outcome<S: ReplicationStorage>(
                    reason = "source_version_missing",
                    "Skipping stale delete-marker replication"
                );
-                return true;
+                // The marker is gone at the source, but a replica of it may
+                // already exist on the targets (a live race, or an MRF
+                // purge-intent replay landing here on purpose). Purge instead
+                // of just skipping; the result decides whether an MRF replay
+                // may acknowledge the entry.
+                return purge_stale_delete_marker_targets(&bucket, &dobj).await;
            }
            Err(err) => {
                source_state_verified = false;
@@ -1485,29 +1495,6 @@ pub(crate) async fn replicate_delete_with_outcome<S: ReplicationStorage>(
    let is_version_purge = is_version_delete_replication(&dobj.delete_object);

    let requires_delayed_purge = should_retry_delete_marker_purge(&dobj.delete_object);
-    if requires_delayed_purge {
-        let bucket_clone = bucket.clone();
-        let dobj_clone = dobj.clone();
-        let dsc_clone = dsc.clone();
-        let storage_clone = storage.clone();
-        tokio::spawn(async move {
-            for _ in 0..5 {
-                if let Some(delete_marker_version_id) = dobj_clone.delete_object.delete_marker_version_id
-                    && source_delete_marker_missing(
-                        &*storage_clone,
-                        &bucket_clone,
-                        &dobj_clone.delete_object.object_name,
-                        delete_marker_version_id,
-                    )
-                    .await
-                {
-                    replicate_delete_marker_purge_to_targets(&bucket_clone, &dobj_clone, &dsc_clone).await;
-                    break;
-                }
-                tokio::time::sleep(TokioDuration::from_secs(1)).await;
-            }
-        });
-    }

    let (replication_status, prev_status) = if !is_version_purge {
        (
@@ -1550,6 +1537,24 @@ pub(crate) async fn replicate_delete_with_outcome<S: ReplicationStorage>(
        drs.replication_timestamp = Some(OffsetDateTime::now_utc());
    }

+    if requires_delayed_purge {
+        // Hand the watcher the MERGED replication state: `drs` folds this
+        // round's per-target results into the previous state, including the
+        // version ids the targets assigned to the markers they just created.
+        // Spawning with the pre-merge `dobj` made the purge fall back to a
+        // source-derived id, which a target that mints its own ids answers
+        // with an idempotent 204 — the intent was then dropped while the
+        // real marker stayed behind.
+        let bucket_clone = bucket.clone();
+        let mut dobj_clone = dobj.clone();
+        dobj_clone.delete_object.replication_state = Some(drs.clone());
+        let dsc_clone = dsc.clone();
+        let storage_clone = storage.clone();
+        tokio::spawn(async move {
+            watch_and_purge_source_delete_marker(bucket_clone, dobj_clone, dsc_clone, storage_clone).await;
+        });
+    }
+
    let event_name = if replication_status == ReplicationStatusType::Completed {
        EventName::ObjectReplicationComplete.to_string()
    } else {
@@ -1608,12 +1613,36 @@ pub(crate) async fn replicate_delete_with_outcome<S: ReplicationStorage>(
        }
    };

+    replicate_delete_outcome(
+        expected_targets,
+        rinfos.targets.len(),
+        state_persisted,
+        source_state_verified,
+        &replication_status,
+    )
+}
+
+/// Whether a delete replication fully succeeded — the MRF replay acknowledges
+/// (drops) an entry exactly when this returns true.
+///
+/// The delayed purge is deliberately NOT an input: holding the outcome hostage
+/// to it (`&& !requires_delayed_purge`) forced `false` for every delete-marker
+/// entry and retained them all in the durable MRF journal forever. Purge
+/// failures persist their own purge-intent entry instead
+/// (`watch_and_purge_source_delete_marker`), and replays of those entries
+/// report purge success through `purge_stale_delete_marker_targets`.
+fn replicate_delete_outcome(
+    expected_targets: usize,
+    replicated_targets: usize,
+    state_persisted: bool,
+    source_state_verified: bool,
+    replication_status: &ReplicationStatusType,
+) -> bool {
    expected_targets > 0
-        && rinfos.targets.len() == expected_targets
+        && replicated_targets == expected_targets
        && state_persisted
        && source_state_verified
-        && !requires_delayed_purge
-        && replication_status == ReplicationStatusType::Completed
+        && *replication_status == ReplicationStatusType::Completed
 }

 async fn source_delete_marker_missing<S: EcstoreObjectOperations>(
@@ -1663,48 +1692,286 @@ fn delete_marker_purge_version_id(
    })
 }

-async fn replicate_delete_marker_purge_to_targets(bucket: &str, dobj: &DeletedObjectReplicationInfo, dsc: &ReplicateDecision) {
+/// One purge pass over the eligible targets. Returns the ARNs that must be
+/// retried: the remote DELETE failed, or the target client was unavailable
+/// (e.g. a runtime cache miss). Inconsistent recorded version mappings are a
+/// deliberate refusal — retrying cannot make guessing a version id safe — so
+/// they are logged and excluded from the retry set.
+async fn replicate_delete_marker_purge_to_targets(
+    bucket: &str,
+    dobj: &DeletedObjectReplicationInfo,
+    dsc: &ReplicateDecision,
+    retry_arns: Option<&[String]>,
+) -> Vec<String> {
    let Some(delete_marker_version_id) = dobj.delete_object.delete_marker_version_id else {
-        return;
+        return Vec::new();
    };

+    let target_arns = dobj.admitted_target_arns();
+    let mut failed_arns = Vec::new();
    for tgt_entry in dsc.targets_map.values() {
        if !tgt_entry.replicate {
            continue;
        }
-        let target_arns = dobj.admitted_target_arns();
        if !target_arns.is_empty() && !target_arns.iter().any(|arn| arn == &tgt_entry.arn) {
            continue;
        }
-        let Some(tgt_client) = ReplicationTargetStore::remote_target_client(bucket, &tgt_entry.arn).await else {
+        if let Some(retry_arns) = retry_arns
+            && !retry_arns.iter().any(|arn| arn == &tgt_entry.arn)
+        {
            continue;
-        };
-
+        }
+        // Decide the version first: refusing to guess is a per-target
+        // FAILURE, not a silent skip. Reporting it as success would let the
+        // watcher and the MRF replay drop the purge intent while the marker
+        // is still on the target — the leak stays visible instead (the
+        // entry is retained and keeps warning) until an operator repairs
+        // the metadata.
        let Some(purge_version_id) = delete_marker_purge_version_id(
            dobj.delete_object.replication_state.as_ref(),
            &tgt_entry.arn,
            delete_marker_version_id,
        ) else {
            warn!(
+                event = EVENT_DELETE_MARKER_PURGE_FAILED,
                component = LOG_COMPONENT_ECSTORE,
                subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
                bucket,
                object = dobj.delete_object.object_name,
                arn = tgt_entry.arn,
-                "Skipping delete-marker purge: recorded target version metadata is inconsistent"
+                reason = "recorded_target_version_inconsistent",
+                "Delete-marker purge refused: recorded target version metadata is inconsistent"
            );
+            counter!(METRIC_DELETE_MARKER_PURGE_TOTAL, "state" => "refused").increment(1);
+            failed_arns.push(tgt_entry.arn.clone());
            continue;
        };

-        let _ = tgt_client
+        let Some(tgt_client) = ReplicationTargetStore::remote_target_client(bucket, &tgt_entry.arn).await else {
+            warn!(
+                event = EVENT_DELETE_MARKER_PURGE_FAILED,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
+                bucket,
+                object = dobj.delete_object.object_name,
+                arn = tgt_entry.arn,
+                reason = "target_client_missing",
+                "Delete-marker purge attempt failed"
+            );
+            counter!(METRIC_DELETE_MARKER_PURGE_TOTAL, "state" => "failed").increment(1);
+            failed_arns.push(tgt_entry.arn.clone());
+            continue;
+        };
+
+        match tgt_client
            .remove_object(
                &tgt_client.bucket,
                &dobj.delete_object.object_name,
                purge_version_id,
                replication_delete_marker_purge_remove_options(dobj.delete_object.delete_marker_mtime),
            )
-            .await;
+            .await
+        {
+            Ok(_) => {
+                counter!(METRIC_DELETE_MARKER_PURGE_TOTAL, "state" => "purged").increment(1);
+            }
+            // The marker version is already gone on the target: the purge goal
+            // is met. Strict S3 targets 404 here (RustFS/MinIO answer 204);
+            // treating it as a failure would retain the intent entry forever.
+            Err(error) if matches!(error.code.as_deref(), Some("NoSuchKey" | "NoSuchVersion")) => {
+                counter!(METRIC_DELETE_MARKER_PURGE_TOTAL, "state" => "purged").increment(1);
+            }
+            Err(error) => {
+                warn!(
+                    event = EVENT_DELETE_MARKER_PURGE_FAILED,
+                    component = LOG_COMPONENT_ECSTORE,
+                    subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
+                    bucket,
+                    object = dobj.delete_object.object_name,
+                    arn = tgt_entry.arn,
+                    error = %error,
+                    reason = "target_delete_failed",
+                    "Delete-marker purge attempt failed"
+                );
+                counter!(METRIC_DELETE_MARKER_PURGE_TOTAL, "state" => "failed").increment(1);
+                mark_replication_target_offline_if_needed(&tgt_client, &error).await;
+                failed_arns.push(tgt_entry.arn.clone());
+            }
+        }
    }
+    failed_arns
+}
+
+const DELETE_MARKER_PURGE_WATCH_ROUNDS: usize = 5;
+const DELETE_MARKER_PURGE_WATCH_INTERVAL: TokioDuration = TokioDuration::from_secs(1);
+
+/// Watch the source delete marker for a short window after its replication.
+///
+/// KNOWN NON-DURABLE WINDOW: this task is detached, so a process exit inside
+/// the watch window loses an intent that has not been persisted yet. The
+/// window predates this code (the previous implementation had no durable
+/// channel at all, and no replay half either), so nothing regresses — closing
+/// it needs a write-ahead intent recorded before the parent delete is
+/// acknowledged, which is tracked as follow-up rather than done here: every
+/// delete-marker replication would pay a journal write for a purge that
+/// almost never happens.
+///
+/// If the marker disappears (deleted before or while the replica landed),
+/// purge the replicated marker from the targets, retrying failed targets on
+/// later rounds. When the window drains with targets still dirty, persist the
+/// purge intent as a durable MRF entry so the next startup replays it through
+/// `purge_stale_delete_marker_targets`.
+async fn watch_and_purge_source_delete_marker<S: ReplicationStorage>(
+    bucket: String,
+    dobj: DeletedObjectReplicationInfo,
+    dsc: ReplicateDecision,
+    storage: Arc<S>,
+) {
+    let Some(delete_marker_version_id) = dobj.delete_object.delete_marker_version_id else {
+        return;
+    };
+
+    // `pending` is None until the source marker is observed missing; after the
+    // first purge pass it holds the targets that still need a successful purge.
+    let mut pending: Option<Vec<String>> = None;
+    for round in 0..DELETE_MARKER_PURGE_WATCH_ROUNDS {
+        pending = match pending.take() {
+            None => {
+                if source_delete_marker_missing(&*storage, &bucket, &dobj.delete_object.object_name, delete_marker_version_id)
+                    .await
+                {
+                    Some(replicate_delete_marker_purge_to_targets(&bucket, &dobj, &dsc, None).await)
+                } else {
+                    None
+                }
+            }
+            Some(failed_arns) => Some(replicate_delete_marker_purge_to_targets(&bucket, &dobj, &dsc, Some(&failed_arns)).await),
+        };
+        if matches!(pending.as_deref(), Some([])) {
+            return;
+        }
+        if round + 1 < DELETE_MARKER_PURGE_WATCH_ROUNDS {
+            tokio::time::sleep(DELETE_MARKER_PURGE_WATCH_INTERVAL).await;
+        }
+    }
+    if let Some(failed_arns) = pending.filter(|failed_arns| !failed_arns.is_empty()) {
+        enqueue_delete_marker_purge_mrf(&dobj, failed_arns).await;
+    }
+}
+
+/// Shape an exhausted purge intent as a marker-creation delete entry. Replay
+/// reconstructs it with `delete_marker: true`, finds the source marker gone,
+/// and funnels into the stale-marker branch of `replicate_delete_with_outcome`
+/// — which re-runs the purge without touching source state and reports purge
+/// success as the replay outcome.
+fn delete_marker_purge_mrf_entry(dobj: &DeletedObjectReplicationInfo, failed_arns: Vec<String>) -> MrfReplicateEntry {
+    let mut entry = dobj.to_mrf_entry();
+    entry.delete_marker = true;
+    entry.version_id = None;
+    entry.retry_count = 0;
+    entry.target_arns = failed_arns;
+    entry
+}
+
+async fn enqueue_delete_marker_purge_mrf(dobj: &DeletedObjectReplicationInfo, failed_arns: Vec<String>) {
+    let arns = failed_arns.join(",");
+    let miss_reason = match runtime_sources::replication_pool() {
+        None => Some("replication_pool_unavailable"),
+        Some(pool) => match pool.persist_mrf_entry(delete_marker_purge_mrf_entry(dobj, failed_arns)).await {
+            ReplicationQueueAdmission::Queued => None,
+            _ => Some("mrf_save_unavailable"),
+        },
+    };
+    match miss_reason {
+        None => {
+            warn!(
+                event = EVENT_DELETE_MARKER_PURGE_MRF,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
+                bucket = dobj.bucket,
+                object = dobj.delete_object.object_name,
+                arns,
+                state = "queued",
+                "Delete-marker purge exhausted its watch window; intent persisted to the MRF journal"
+            );
+            counter!(METRIC_DELETE_MARKER_PURGE_TOTAL, "state" => "mrf_queued").increment(1);
+        }
+        Some(reason) => {
+            warn!(
+                event = EVENT_DELETE_MARKER_PURGE_MRF,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
+                bucket = dobj.bucket,
+                object = dobj.delete_object.object_name,
+                arns,
+                state = "missed",
+                reason,
+                "Delete-marker purge intent could not be persisted for retry"
+            );
+            counter!(METRIC_DELETE_MARKER_PURGE_TOTAL, "state" => "mrf_missed").increment(1);
+        }
+    }
+}
+
+/// The marker vanished at the source while its replication was still pending
+/// (a live race), or this is an MRF purge-intent replay. Any marker already
+/// replicated to a target must still be purged; run bounded retry passes and
+/// report the result so an MRF replay only acknowledges the entry once every
+/// target is clean. Live callers persist a fresh purge intent on failure;
+/// replay callers (`ReplicationType::Heal`) rely on Missed retention instead,
+/// so the journal does not accumulate duplicate entries.
+///
+/// Heal callers retry for the full watch window because the startup MRF
+/// processor runs before bucket metadata (and thus target clients) finishes
+/// initializing — the first pass can see `target_client_missing` and a later
+/// round resolves the client; the replay loop is serial and startup-only, so
+/// blocking it for up to the window per dirty entry is acceptable. Live
+/// callers run on replication workers where a down target would pin a worker
+/// for the whole window, so they attempt once and lean on the durable intent
+/// entry instead.
+async fn purge_stale_delete_marker_targets(bucket: &str, dobj: &DeletedObjectReplicationInfo) -> bool {
+    let decision_str = dobj
+        .delete_object
+        .replication_state
+        .as_ref()
+        .map(|state| state.replicate_decision_str.clone())
+        .unwrap_or_default();
+    let dsc = match parse_replicate_decision(bucket, &decision_str) {
+        Ok(dsc) => dsc,
+        Err(error) => {
+            warn!(
+                event = EVENT_DELETE_MARKER_PURGE_FAILED,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
+                bucket,
+                object = dobj.delete_object.object_name,
+                error = %error,
+                reason = "replicate_decision_parse_failed",
+                "Delete-marker purge attempt failed"
+            );
+            return false;
+        }
+    };
+    let rounds = if dobj.op_type == ReplicationType::Heal {
+        DELETE_MARKER_PURGE_WATCH_ROUNDS
+    } else {
+        1
+    };
+    let mut failed_arns = replicate_delete_marker_purge_to_targets(bucket, dobj, &dsc, None).await;
+    for _ in 1..rounds {
+        if failed_arns.is_empty() {
+            break;
+        }
+        tokio::time::sleep(DELETE_MARKER_PURGE_WATCH_INTERVAL).await;
+        failed_arns = replicate_delete_marker_purge_to_targets(bucket, dobj, &dsc, Some(&failed_arns)).await;
+    }
+    if failed_arns.is_empty() {
+        return true;
+    }
+    if dobj.op_type != ReplicationType::Heal {
+        enqueue_delete_marker_purge_mrf(dobj, failed_arns).await;
+    }
+    false
 }

 async fn replicate_force_delete_to_targets<S: ReplicationStorage>(dobj: &DeletedObjectReplicationInfo, storage: Arc<S>) -> bool {
@@ -3218,6 +3485,7 @@ async fn replicate_object_with_multipart<S: ReplicationObjectIO>(ctx: MultipartR

 #[cfg(test)]
 mod tests {
+    use super::super::replication_filemeta_boundary::ReplicateTargetDecision;
    use super::super::replication_target_boundary::{BucketTarget, BucketTargets};
    use super::*;
    use s3s::dto::{
@@ -3581,6 +3849,101 @@ mod tests {
        );
    }

+    /// P1-21 regression guard for the outcome formula. A fully successful
+    /// delete-marker replication must acknowledge its MRF entry: the formula
+    /// once carried `&& !requires_delayed_purge`, which pinned every
+    /// delete-marker entry to Missed and retained the whole backlog forever.
+    /// (Deterministically staging a marker-creation entry in the durable
+    /// journal from e2e would require saturating the worker queues, so the
+    /// formula is pinned here instead; the purge-intent replay half is pinned
+    /// by the delayed-purge e2e pair.)
+    #[test]
+    fn test_replicate_delete_outcome_is_not_held_hostage_by_the_delayed_purge() {
+        assert!(
+            replicate_delete_outcome(1, 1, true, true, &ReplicationStatusType::Completed),
+            "a completed delete-marker replication must be acknowledgeable even though a delayed purge watch is pending"
+        );
+        assert!(!replicate_delete_outcome(0, 0, true, true, &ReplicationStatusType::Completed));
+        assert!(!replicate_delete_outcome(2, 1, true, true, &ReplicationStatusType::Completed));
+        assert!(!replicate_delete_outcome(1, 1, false, true, &ReplicationStatusType::Completed));
+        assert!(!replicate_delete_outcome(1, 1, true, false, &ReplicationStatusType::Completed));
+        assert!(!replicate_delete_outcome(1, 1, true, true, &ReplicationStatusType::Failed));
+    }
+
+    /// P1-21 review follow-up: a target whose recorded marker version is
+    /// inconsistent must be reported as a per-target FAILURE. Treating the
+    /// refusal as success let the watcher and the MRF replay drop the purge
+    /// intent while the marker was still on the target.
+    #[tokio::test]
+    async fn test_delete_marker_purge_reports_corrupt_recorded_version_as_failure() {
+        let arn = format!("arn:rustfs:replication:us-east-1:corrupt:{}", Uuid::new_v4());
+        let mut dsc = ReplicateDecision::new();
+        dsc.set(ReplicateTargetDecision::new(arn.clone(), true, false));
+
+        let mut state = ReplicationState {
+            target_delete_marker_version_ids_corrupt: true,
+            ..Default::default()
+        };
+        state.targets.insert(arn.clone(), ReplicationStatusType::Completed);
+
+        let dobj = DeletedObjectReplicationInfo {
+            delete_object: ReplicationDeletedObject {
+                object_name: "doc.txt".to_string(),
+                delete_marker: true,
+                delete_marker_version_id: Some(Uuid::new_v4()),
+                replication_state: Some(state),
+                ..Default::default()
+            },
+            bucket: "bucket-a".to_string(),
+            ..Default::default()
+        };
+
+        // No target client is registered: the refusal must be decided from
+        // the recorded metadata alone, before any remote call is attempted.
+        let failed = replicate_delete_marker_purge_to_targets("bucket-a", &dobj, &dsc, None).await;
+
+        assert_eq!(
+            failed,
+            vec![arn],
+            "a refused purge must stay in the failed set so the intent is never acknowledged"
+        );
+    }
+
+    #[test]
+    fn test_delete_marker_purge_mrf_entry_replays_through_the_stale_marker_branch() {
+        let delete_marker_version_id = Uuid::new_v4();
+        let dobj = DeletedObjectReplicationInfo {
+            delete_object: ReplicationDeletedObject {
+                object_name: "doc.txt".to_string(),
+                // A version-purge flavored source event: the entry must still
+                // be reshaped as a marker-creation delete so replay funnels
+                // into the stale-marker branch instead of re-running the full
+                // delete replication (whose source-state stamping would fail
+                // against the already-purged version).
+                delete_marker: false,
+                version_id: Some(Uuid::new_v4()),
+                delete_marker_version_id: Some(delete_marker_version_id),
+                ..Default::default()
+            },
+            bucket: "bucket-a".to_string(),
+            ..Default::default()
+        };
+
+        let entry = delete_marker_purge_mrf_entry(&dobj, vec!["arn:a".to_string()]);
+
+        assert!(entry.delete_marker, "purge intents must replay as marker-creation deletes");
+        assert_eq!(entry.version_id, None, "the purged data version must not leak into the replay");
+        assert_eq!(entry.delete_marker_version_id, Some(delete_marker_version_id));
+        assert_eq!(
+            entry.target_arns,
+            vec!["arn:a".to_string()],
+            "only the targets whose purge failed may be retried"
+        );
+        assert_eq!(entry.retry_count, 0);
+        assert_eq!(entry.bucket, "bucket-a");
+        assert_eq!(entry.object, "doc.txt");
+    }
+
    #[test]
    fn test_is_retryable_delete_replication_head_error_allows_delete_marker_head_responses() {
        assert!(