fix(replication): retry, persist and replay failed delete-marker purges (#5864)

* test(replication): pin delayed delete-marker purge failure handling (red) P1-21 (rustfs/backlog#1675 B2): two failing e2e tests that pin the missing failure handling of the delayed delete-marker purge: - test_delayed_delete_marker_purge_retries_after_transient_target_failure: four scripted 503s outlast every existing channel (version-purge replication + its in-process MRF fast retries + the watcher's single attempt = 3 target DELETEs, all faulted in the recorded run); the replicated marker is stranded on the target forever. - test_delayed_delete_marker_purge_exhaustion_persists_to_mrf_and_replays_on_restart: exhausted purge intents never reach the durable MRF journal, so a restart replays nothing (recorded run: 3 faulted attempts, zero post-restart). Red-light evidence (current main): - Test A: FAILED, journal shows 3x DeleteObject fault=Status(503), no clean attempt, target marker still present after 15s. - Test B: FAILED after 468s, same 3 faulted attempts, no purge DELETE after restart, marker still present. Test infra: FakeS3Target::stored_versions() exposes per-key version state so purge tests assert target state instead of inferring it from the journal; nextest count comments 36->38 nightly / 56->58 total. * fix(replication): retry, persist and replay failed delete-marker purges P1-21 (rustfs/backlog#1675 B2). The delayed delete-marker purge was fire-and-forget: the target DELETE discarded its result (`let _ =`), a missing target client was silently skipped, and nothing recorded the intent — one transient target error stranded the replicated marker on the target forever. Separately, `replicate_delete_with_outcome` held its outcome hostage to `!requires_delayed_purge`, pinning every delete-marker MRF entry to Missed so the durable backlog retained them permanently. Changes: - `replicate_delete_marker_purge_to_targets` now reports per-target results (warn + metrics on failure, including `target_client_missing`), supports retrying only the failed targets, and treats a target-side NoSuchKey/NoSuchVersion as purge success (strict-404 targets must not retain the intent forever). - The delayed watcher (`watch_and_purge_source_delete_marker`) retries failed targets across its 5x1s watch window; on exhaustion it persists the purge intent to the durable MRF journal via the new `ReplicationPoolTrait::persist_mrf_entry` (journal-only on purpose: live re-dispatch would loop unboundedly against a down target). Intent entries are shaped as marker-creation deletes so replay funnels into the stale- marker branch. - The stale-marker branch (source marker already gone) now purges the targets instead of silently returning success — closing a latent leak — and reports the purge result as the replay outcome. Heal callers retry for the full window (the startup MRF processor runs before target clients initialize); live callers attempt once and fall back to a fresh durable intent, so a down target cannot pin a replication worker. - The outcome formula (extracted as `replicate_delete_outcome` and pinned by a unit test) no longer includes the delayed purge, so successfully replayed delete-marker entries are acknowledged instead of retained forever. Verification: red -> green e2e pair (transient-failure retry; exhaustion -> durable MRF -> restart replay -> second-restart zero-replay ack) plus unit tests; `make pre-commit`, logging guardrails, clippy (ecstore + e2e_test) all clean; full ecstore lib suite 3729 passed (3 pre-existing local-DNS kubernetes endpoint failures reproduce without this change). Adversarial validation (7 roles): no blocking findings after adding the outcome-formula guard test. Known residuals recorded in the PR: watcher shutdown window (intent not yet persisted), rolling-downgrade replay acks without purging (equals pre-fix behavior), and replay falling back to the source version id on targets that mint their own version ids (P1-19). * chore(test): refresh the nextest replication count invariant The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata (authority: `cargo nextest list`); refresh it to this branch's post-rebase total. * fix(replication): purge the marker version the target actually assigned Review follow-up (#5864), two real defects: - The delayed purge watcher was spawned with the pre-merge `dobj`, so the per-target marker version ids this round recorded were invisible to it. Against a target that mints its own ids the purge fell back to a source-derived id, the target answered the versioned DELETE with an idempotent 204, and that "success" cleared the retry set while the real marker stayed behind. The watcher now receives the merged replication state (`drs`), which folds this round's target-assigned ids in. - A target whose recorded version metadata is inconsistent was skipped without entering `failed_arns`, so an empty result made both the watcher and the MRF replay treat a purge that issued no DELETE as successful and drop the intent. The refusal is now a per-target failure (own metric label): the leak stays visible and the intent is retained instead of being acknowledged. The version decision also moved ahead of the client lookup, so the refusal is decided from metadata alone. Tests: a new e2e drives a fake target with `assign_own_version_ids`, which ignores the forwarded source-version header for both objects and delete markers, and asserts the replicated marker is really gone; a unit test pins the corrupt-metadata refusal as a failed outcome without any target client registered. The detached-watcher shutdown window is documented at the watcher as a known non-durable window with the write-ahead follow-up spelled out.
2026-08-13 08:36:54 +00:00 · 2026-08-10 22:16:21 +08:00
parent fe2516ee86
commit 3c31eaf06f
5 changed files with 791 additions and 46 deletions
@@ -141,6 +141,7 @@ struct ControlState {

 #[derive(Default)]
 struct StoreState {
+    assign_own_version_ids: bool,
    buckets: HashMap<String, BucketState>,
    uploads: HashMap<String, MultipartState>,
    total_bytes: usize,
@@ -383,6 +384,12 @@ impl FakeS3Target {
            .is_some_and(|version| !version.delete_marker)
    }

+    /// Make the target mint its own version ids instead of mirroring the
+    /// forwarded source version id — models a generic S3 service.
+    pub fn assign_own_version_ids(&self, enabled: bool) {
+        lock(&self.backend.store).assign_own_version_ids = enabled;
+    }
+
    /// Queue `times` copies of a fault for one operation.
    pub fn inject(&self, operation: Operation, action: FaultAction, times: usize) {
        if times == 0 {
@@ -433,6 +440,25 @@ impl FakeS3Target {
        lock(&self.control).requests.drain(..).collect()
    }

+    /// Stored versions for one key as `(version_id, is_delete_marker)`, oldest
+    /// first. Empty when the bucket or key does not exist. Lets purge tests
+    /// assert on the target's actual state instead of inferring it from the
+    /// request journal (a versioned DELETE is a silent no-op for missing ids).
+    pub fn stored_versions(&self, bucket: &str, key: &str) -> Vec<(String, bool)> {
+        let state = lock(&self.backend.store);
+        state
+            .buckets
+            .get(bucket)
+            .and_then(|bucket_state| bucket_state.objects.get(key))
+            .map(|versions| {
+                versions
+                    .iter()
+                    .map(|version| (version.version_id.clone(), version.delete_marker))
+                    .collect()
+            })
+            .unwrap_or_default()
+    }
+
    pub async fn shutdown(mut self) {
        let _ = self.shutdown.send(true);
        if let Some(task) = self.task.take() {
@@ -689,10 +715,17 @@ fn validate_retained_identifier(value: String, field: &str) -> S3Result<String>
    }
 }

-fn new_version_id(headers: &HeaderMap) -> S3Result<String> {
+/// `assign_own` models a target that mints its own version ids (a generic S3
+/// service): the forwarded source-version-id header is validated but NOT
+/// mirrored into the stored version.
+fn new_version_id(headers: &HeaderMap, assign_own: bool) -> S3Result<String> {
    let Some(value) = header_value(headers, &SOURCE_VERSION_ID_HEADERS) else {
        return Ok(Uuid::new_v4().to_string());
    };
+    if assign_own {
+        validate_retained_identifier(value.trim().to_owned(), "source version ID")?;
+        return Ok(Uuid::new_v4().to_string());
+    }
    let value = validate_retained_identifier(value.trim().to_owned(), "source version ID")?;
    let version_id = Uuid::parse_str(&value).map_err(|_| s3s::s3_error!(InvalidArgument, "source version ID must be a UUID"))?;
    Ok(version_id.to_string())
@@ -1078,7 +1111,8 @@ impl S3 for FakeBackend {
        let input = req.input;
        let body = collect_stream(input.body, input.content_length, fault.as_ref(), &self.control).await?;
        validate_stored_metadata(&input.content_type, &input.metadata)?;
-        let version_id = new_version_id(&headers)?;
+        let assign_own = lock(&self.store).assign_own_version_ids;
+        let version_id = new_version_id(&headers, assign_own)?;
        let e_tag = match source_etag(&headers)? {
            Some(value) => value,
            None => {
@@ -1212,7 +1246,9 @@ impl S3 for FakeBackend {
            ));
        }

-        let version_id = new_version_id(&headers)?;
+        // `state` is the live store guard: read the flag from it. Re-locking
+        // would self-deadlock (the store mutex is not reentrant).
+        let version_id = new_version_id(&headers, state.assign_own_version_ids)?;
        upsert_version(
            &mut state,
            &input.bucket,
@@ -1252,12 +1288,15 @@ impl S3 for FakeBackend {
        ensure_upload_budget(&state)?;
        validate_stored_metadata(&input.content_type, &input.metadata)?;
        let upload_id = Uuid::new_v4().to_string();
+        // Read the flag before the mutable borrow of `state.uploads` below
+        // (and never re-lock the store: the mutex is not reentrant).
+        let version_id = new_version_id(&headers, state.assign_own_version_ids)?;
        state.uploads.insert(
            upload_id.clone(),
            MultipartState {
                bucket: input.bucket.clone(),
                key: input.key.clone(),
-                version_id: new_version_id(&headers)?,
+                version_id,
                content_type: input.content_type,
                metadata: input.metadata,
                parts: BTreeMap::new(),
@@ -18,6 +18,7 @@ use crate::common::{
 };
 use crate::fake_s3_target::{
    FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, FaultAction as FakeTargetFault, Operation as FakeTargetOperation,
+    RequestRecord,
 };
 use crate::kms::common::{create_key_with_specific_id, sse_customer_key_md5_base64};
 use crate::storage_api::replication_extension::BucketTargetSys;
@@ -7487,3 +7488,335 @@ async fn test_scanner_never_cascades_inbound_replicas() -> TestResult {

    Ok(())
 }
+
+// --- P1-21 (backlog#1675): delayed delete-marker purge failure handling ---
+//
+// The fixtures below wire a versioned source bucket to a FakeS3Target with the
+// default replication shape: DeleteMarkerReplication=Enabled and
+// DeleteReplication omitted. With version-delete replication unconfigured,
+// purging the source marker version emits no replication event, and the data
+// scanner cannot see a source version that is gone — the delayed purge watcher
+// spawned by the marker replication is the ONLY channel that can remove the
+// replicated marker from the target.
+
+const DELAYED_PURGE_KEY: &str = "doc.txt";
+
+fn delayed_purge_process_env() -> Vec<(&'static str, &'static str)> {
+    let mut env = replication_fast_env();
+    env.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
+    env.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
+    env
+}
+
+async fn start_delayed_purge_fixture(
+    source_bucket: &str,
+    target_bucket: &str,
+) -> Result<(FakeS3Target, RustFSTestEnvironment, Client), Box<dyn Error + Send + Sync>> {
+    let target = FakeS3Target::start().await?;
+    target.create_bucket(target_bucket);
+
+    let mut source_env = RustFSTestEnvironment::new().await?;
+    source_env
+        .start_rustfs_server_with_env(vec![], &delayed_purge_process_env())
+        .await?;
+
+    let source_client = source_env.create_s3_client();
+    source_client.create_bucket().bucket(source_bucket).send().await?;
+    enable_bucket_versioning(&source_env, source_bucket).await?;
+
+    let target_arn = set_replication_target_with_options(
+        &source_env,
+        source_bucket,
+        ReplicationTargetOptions {
+            endpoint: &target.address(),
+            access_key: FAKE_ACCESS_KEY,
+            secret_key: FAKE_SECRET_KEY,
+            target_bucket,
+            secure: false,
+            skip_tls_verify: false,
+            ca_cert_pem: None,
+        },
+    )
+    .await?;
+    put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
+
+    Ok((target, source_env, source_client))
+}
+
+/// PUT an object, stack a delete marker on it, and wait until the fake target
+/// stores the marker replica. Returns the source marker version id — the fake
+/// target mirrors it because delete replication forwards
+/// `x-*-source-version-id`.
+///
+/// Timing budget for callers: the delayed purge watcher only observes the
+/// source for ~4s after the marker replication completes, so the source-side
+/// marker-version DELETE must be issued promptly after this returns (the
+/// 100ms journal poll below keeps the detection latency small).
+async fn replicate_delete_marker(
+    target: &FakeS3Target,
+    target_bucket: &str,
+    source_client: &Client,
+    source_bucket: &str,
+) -> Result<String, Box<dyn Error + Send + Sync>> {
+    source_client
+        .put_object()
+        .bucket(source_bucket)
+        .key(DELAYED_PURGE_KEY)
+        .body(ByteStream::from_static(b"delayed purge payload"))
+        .send()
+        .await?;
+
+    let delete = source_client
+        .delete_object()
+        .bucket(source_bucket)
+        .key(DELAYED_PURGE_KEY)
+        .send()
+        .await?;
+    assert_eq!(delete.delete_marker(), Some(true), "unversioned DELETE must create a marker");
+    let marker_version = delete
+        .version_id()
+        .ok_or("source DELETE omitted the marker version ID")?
+        .to_string();
+
+    // Wait for ANY delete marker: a target that mints its own version ids
+    // does not mirror the source one.
+    let deadline = tokio::time::Instant::now() + Duration::from_secs(30);
+    loop {
+        let replicated = target
+            .stored_versions(target_bucket, DELAYED_PURGE_KEY)
+            .iter()
+            .any(|(_, delete_marker)| *delete_marker);
+        if replicated {
+            return Ok(marker_version);
+        }
+        if tokio::time::Instant::now() >= deadline {
+            return Err(
+                format!("fake target never stored the replicated delete marker; journal: {:?}", target.requests()).into(),
+            );
+        }
+        sleep(Duration::from_millis(100)).await;
+    }
+}
+
+/// Journal records of purge attempts: target DELETE calls addressing the marker
+/// version explicitly. The marker-creation replica DELETE carries no
+/// `versionId` query, so the version id is an exact discriminator.
+fn delayed_purge_attempts(target: &FakeS3Target, marker_version: &str) -> Vec<RequestRecord> {
+    target
+        .requests()
+        .into_iter()
+        .filter(|record| {
+            record.operation == FakeTargetOperation::DeleteObject
+                && record.key.as_deref() == Some(DELAYED_PURGE_KEY)
+                && record.version_id.as_deref() == Some(marker_version)
+        })
+        .collect()
+}
+
+async fn wait_for_target_marker_purged(
+    target: &FakeS3Target,
+    target_bucket: &str,
+    max_wait: Duration,
+) -> Result<(), Box<dyn Error + Send + Sync>> {
+    let deadline = tokio::time::Instant::now() + max_wait;
+    loop {
+        let marker_present = target
+            .stored_versions(target_bucket, DELAYED_PURGE_KEY)
+            .iter()
+            .any(|(_, delete_marker)| *delete_marker);
+        if !marker_present {
+            return Ok(());
+        }
+        if tokio::time::Instant::now() >= deadline {
+            return Err(format!(
+                "target delete marker was never purged; target state: {:?}",
+                target.stored_versions(target_bucket, DELAYED_PURGE_KEY)
+            )
+            .into());
+        }
+        sleep(Duration::from_millis(200)).await;
+    }
+}
+
+/// P1-21: the delayed purge's single target DELETE currently swallows failures
+/// (`let _ =`), so one transient target error strands the replicated marker on
+/// the target forever. Contract under test: a failed purge attempt is retried
+/// within the watch window and converges once the fault clears.
+#[tokio::test]
+#[serial]
+async fn test_delayed_delete_marker_purge_retries_after_transient_target_failure() -> TestResult {
+    init_logging();
+    let source_bucket = "delayed-purge-retry-src";
+    let target_bucket = "delayed-purge-retry-dst";
+    let (target, _source_env, source_client) = start_delayed_purge_fixture(source_bucket, target_bucket).await?;
+
+    let marker_version = replicate_delete_marker(&target, target_bucket, &source_client, source_bucket).await?;
+
+    // Four scripted failures. Fault budget accounting (each journal record
+    // consumes one fault, including the SDK's own per-request retries):
+    // deleting the marker version fans out over the version-purge replication
+    // channel (initial attempt + its fast in-memory MRF retries) plus the
+    // delayed purge watcher's single pre-fix attempt — three target DELETE calls
+    // in total today, empirically (see the exhaustion test's journal). Four
+    // faults outlast all of them, so only a delayed-purge retry in a later
+    // watch round can converge. If the SDK retry configuration ever changes,
+    // re-derive this budget from a fresh journal capture.
+    target.inject(
+        FakeTargetOperation::DeleteObject,
+        FakeTargetFault::Status(StatusCode::SERVICE_UNAVAILABLE),
+        4,
+    );
+
+    // Purge the marker at the source. The watcher spawned when the marker
+    // replication completed moments ago observes the source marker vanish
+    // within its watch window and drives the target purge.
+    source_client
+        .delete_object()
+        .bucket(source_bucket)
+        .key(DELAYED_PURGE_KEY)
+        .version_id(&marker_version)
+        .send()
+        .await?;
+
+    // Tight window on purpose: a fixed delayed purge retries on 1s rounds and
+    // converges within ~5s, while any straggling backoff retry from the other
+    // channels would land later and must not be what turns this test green.
+    wait_for_target_marker_purged(&target, target_bucket, Duration::from_secs(15)).await?;
+
+    let attempts = delayed_purge_attempts(&target, &marker_version);
+    assert!(
+        attempts.len() >= 2,
+        "expected the faulted purge attempt plus at least one retry, got: {attempts:?}"
+    );
+    assert!(
+        attempts.iter().any(|record| record.fault.is_none()),
+        "expected a clean purge attempt after the fault script drained, got: {attempts:?}"
+    );
+
+    target.shutdown().await;
+    Ok(())
+}
+
+/// P1-21 review follow-up: the watcher must purge the version the TARGET
+/// assigned to the replicated marker, not one derived from the source uuid.
+/// A target that mints its own version ids answers a source-derived purge
+/// with an idempotent 204, which used to look like success and strand the
+/// real marker on the target forever.
+#[tokio::test]
+#[serial]
+async fn test_delayed_delete_marker_purge_uses_target_assigned_version() -> TestResult {
+    init_logging();
+    let source_bucket = "delayed-purge-mint-src";
+    let target_bucket = "delayed-purge-mint-dst";
+    let (target, _source_env, source_client) = start_delayed_purge_fixture(source_bucket, target_bucket).await?;
+    // The target ignores the forwarded source-version-id header and mints its
+    // own ids for both the object and the replicated delete marker.
+    target.assign_own_version_ids(true);
+
+    let marker_version = replicate_delete_marker(&target, target_bucket, &source_client, source_bucket).await?;
+
+    source_client
+        .delete_object()
+        .bucket(source_bucket)
+        .key(DELAYED_PURGE_KEY)
+        .version_id(&marker_version)
+        .send()
+        .await?;
+
+    // The replicated marker carries a target-minted version id, so nothing
+    // but the recorded mapping can address it.
+    wait_for_target_marker_purged(&target, target_bucket, Duration::from_secs(25)).await?;
+
+    target.shutdown().await;
+    Ok(())
+}
+
+/// P1-21: when every watch-window purge attempt fails, the purge intent must
+/// survive as a durable MRF entry and replay on the next startup; once the
+/// replayed purge succeeds, the entry must be acknowledged instead of being
+/// retained as Missed forever.
+#[tokio::test]
+#[serial]
+async fn test_delayed_delete_marker_purge_exhaustion_persists_to_mrf_and_replays_on_restart() -> TestResult {
+    init_logging();
+    let source_bucket = "delayed-purge-mrf-src";
+    let target_bucket = "delayed-purge-mrf-dst";
+    let (target, mut source_env, source_client) = start_delayed_purge_fixture(source_bucket, target_bucket).await?;
+
+    let marker_version = replicate_delete_marker(&target, target_bucket, &source_client, source_bucket).await?;
+
+    // Outlast the whole watch window: every in-process purge attempt fails.
+    target.inject(
+        FakeTargetOperation::DeleteObject,
+        FakeTargetFault::Status(StatusCode::SERVICE_UNAVAILABLE),
+        64,
+    );
+
+    source_client
+        .delete_object()
+        .bucket(source_bucket)
+        .key(DELAYED_PURGE_KEY)
+        .version_id(&marker_version)
+        .send()
+        .await?;
+
+    // Let the watch window drain before restarting. The wall-clock length is
+    // not 5x1s: every faulted attempt embeds the SDK's own per-request 503
+    // retries (a few seconds each), so instead of a fixed sleep, wait until
+    // the faulted attempts stop arriving (the watcher exhausted its rounds and
+    // persisted the purge intent), then give the MRF persister its 100ms
+    // flush interval.
+    let deadline = tokio::time::Instant::now() + Duration::from_secs(60);
+    let mut last_seen = delayed_purge_attempts(&target, &marker_version).len();
+    let mut quiet_since = tokio::time::Instant::now();
+    loop {
+        sleep(Duration::from_millis(500)).await;
+        let seen = delayed_purge_attempts(&target, &marker_version).len();
+        if seen != last_seen {
+            last_seen = seen;
+            quiet_since = tokio::time::Instant::now();
+        }
+        if last_seen > 0 && quiet_since.elapsed() >= Duration::from_secs(5) {
+            break;
+        }
+        if tokio::time::Instant::now() >= deadline {
+            return Err(format!("purge attempts never quiesced (saw {last_seen}); journal: {:?}", target.requests()).into());
+        }
+    }
+    sleep(Duration::from_secs(1)).await;
+
+    let marker_survives_faults = target
+        .stored_versions(target_bucket, DELAYED_PURGE_KEY)
+        .iter()
+        .any(|(_, delete_marker)| *delete_marker);
+    assert!(marker_survives_faults, "scripted faults must have blocked every in-process purge attempt");
+
+    target.clear_faults();
+    let attempts_before_restart = delayed_purge_attempts(&target, &marker_version).len();
+
+    // Startup MRF replay must re-drive the purge and clean the target.
+    source_env
+        .restart_server_preserving_data(vec![], &delayed_purge_process_env())
+        .await?;
+    wait_for_target_marker_purged(&target, target_bucket, Duration::from_secs(30)).await?;
+    let attempts_after_replay = delayed_purge_attempts(&target, &marker_version).len();
+    assert!(
+        attempts_after_replay > attempts_before_restart,
+        "the restart replay must have issued the purge DELETE"
+    );
+
+    // The successful replay must acknowledge the MRF entry: another restart may
+    // not re-drive the purge again.
+    source_env
+        .restart_server_preserving_data(vec![], &delayed_purge_process_env())
+        .await?;
+    sleep(Duration::from_secs(5)).await;
+    assert_eq!(
+        delayed_purge_attempts(&target, &marker_version).len(),
+        attempts_after_replay,
+        "acknowledged purge-intent MRF entries must not replay again"
+    );
+
+    target.shutdown().await;
+    Ok(())
+}