fix(replication): probe the version-identity contract in replication-check (#5881)

* test(replication): pin the version-fidelity probe contract (red)

P1-19 (rustfs/backlog#1675 B2): the supported replication contract is
targets that adopt the source version id — a target that mints its own ids
silently breaks every version-addressed operation that follows (version
deletes, heal re-drives never match), diverging the two sides with no
signal. replication-check already captures the probe PUT's response version
id but never compares it.

Red evidence (current main): against a FakeS3Target with
assign_own_version_ids enabled, ?replication-check returns Status "OK" —
the drift is invisible.

test_replication_check_flags_version_minting_target expects a
VersionFidelity phase that fails with the machine-readable code
BucketRemoteTargetVersionMismatch, skips the later mutation phases, and
still cleans up the probe via the version id the target actually assigned.

Test infra: FakeS3Target gains assign_own_version_ids (models a generic S3
service; validated-but-not-mirrored source version headers) and a
prefix+max-keys ListObjectVersions implementation (the probe key allocation
requires it); stored_versions accessor duplicated from the P1-21 branch
(identical code, resolves clean on merge).

* fix(replication): probe the version-identity contract in replication-check

P1-19 (rustfs/backlog#1675 B2, plan B). Replication only converges on
targets that adopt the source version id: version-addressed deletes and
heal re-drives address the source id, so a target that mints its own ids
silently diverges — nothing surfaced this. replication-check already
captured the probe PUT's response version id but never compared it.

- The probe PUT now carries the source version as `?versionId=` (the exact
  shape live replication uses since P0-5, and the only shape MinIO
  consumes; the internal source-version-id header alone would let the
  probe pass against targets the real data path drifts on). Reuses
  ecstore's append_version_id_query through the api facade.
- New VersionFidelity phase: the probe PUT's response version id must
  equal the sent source id. On mismatch the phase fails with the
  machine-readable extension key `"Code": "BucketRemoteTargetVersionMismatch"`
  (new optional Code field on phase statuses; Go decoders ignore unknown
  keys), the overall target fails, the later version-addressed mutation
  phases are skipped, and cleanup still removes the probe via the id the
  target actually assigned (with the existing list-based sweep as backstop
  when the target returns no version id at all).
- Runtime half: TargetClient::put_object now returns the assigned version
  id (mirroring remove_object), and the replication PUT path audits it —
  every drifting PUT increments
  rustfs_replication_version_identity_drift_total and the first drift per
  target ARN logs a structured warning pointing at ?replication-check.
  The drift judgment is a pure function with an exemption-matrix test
  (empty / literal "null" / nil-uuid sources carry no contract).
- docs/operations/replication-check.md documents the phase and the code.

Red -> green: test_replication_check_flags_version_minting_target (fake
target with assign_own_version_ids; on main the check reported Status
"OK"). The probe's query shape is pinned by a journal assertion (revert
of the query hunk alone fails it), probe-level unit tests cover the
mismatch/mirror matrix including cleanup addressing the minted id, and
the existing success e2e now asserts VersionFidelity OK against a RustFS
target. Adversarial review (seven roles): non-blocking; noted follow-ups
are the multipart runtime audit (the probe phase already pins the
contract) and per-target re-warning after reconfiguration.

* fix(e2e): stop the fake target self-deadlocking on version-id minting

The assign_own_version_ids flag was read with a fresh `lock(&self.store)`
inside two paths that already hold that guard — delete_object's
marker-creation branch and create_multipart_upload — and the store mutex
is not reentrant, so both hung forever (CI: the fake target's own
multipart and delete-marker tests ran >1560s until the job was
cancelled). Read the flag from the live guard instead.

The replication e2e paths did not catch this: a version-addressed purge
DELETE never mints an id, and the probe PUT reads the flag before taking
the guard.

* chore(test): refresh the nextest replication count invariant

The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata
(authority: `cargo nextest list`); refresh it to this branch's
post-rebase total.

rustfs/src/admin/router.rs

@@ -17,7 +17,7 @@ use super::storage_api::bucket::metadata_sys;
 use super::storage_api::bucket::replication::{self, BucketReplicationResyncStatus, BucketStats, ReplicationStatusType};
 use super::storage_api::bucket::target::{BucketTarget, BucketTargetType, BucketTargets};
 use super::storage_api::bucket::target_sys::{
-    BucketTargetSys, PutObjectOptions, RemoveObjectOptions, S3ClientError, TargetClient,
+    BucketTargetSys, PutObjectOptions, RemoveObjectOptions, S3ClientError, TargetClient, append_version_id_query,
 };
 use super::storage_api::bucket::versioning_sys::BucketVersioningSys;
 use super::storage_api::bucket::{AdminReplicationConfigExt as _, AdminVersioningConfigExt as _};
@@ -42,6 +42,7 @@ use crate::server::{
 };
 use crate::storage::storage_api::lock_bucket_targets_metadata;
 use aws_sdk_s3::primitives::ByteStream as AwsByteStream;
+use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
 use bytes::Bytes;
 use futures::{Stream, StreamExt};
 use http::HeaderValue;
@@ -206,6 +207,9 @@ struct ReplicationResetStatusTarget {
 
 const REPLICATION_CHECK_PROBE_PREFIX: &str = ".rustfs.sys/replication-check/";
 const REPLICATION_CHECK_ERROR_MAX_BYTES: usize = 512;
+/// RustFS extension code (no madmin analogue): the target does not adopt the
+/// source version id, breaking the version-identity replication contract.
+const REPLICATION_CHECK_CODE_VERSION_MISMATCH: &str = "BucketRemoteTargetVersionMismatch";
 
 #[derive(Debug, Clone, serde::Serialize)]
 struct ReplicationCheckResponse {
@@ -245,6 +249,8 @@ struct ReplicationCheckPhases {
     object_lock: ReplicationCheckPhaseStatus,
     #[serde(rename = "Put")]
     put: ReplicationCheckPhaseStatus,
+    #[serde(rename = "VersionFidelity")]
+    version_fidelity: ReplicationCheckPhaseStatus,
     #[serde(rename = "DeleteMarker")]
     delete_marker: ReplicationCheckPhaseStatus,
     #[serde(rename = "VersionDelete")]
@@ -259,6 +265,11 @@ struct ReplicationCheckPhaseStatus {
     status: &'static str,
     #[serde(rename = "Error", skip_serializing_if = "Option::is_none")]
     error: Option<String>,
+    /// Machine-readable failure code (RustFS extension key; Go decoders
+    /// ignore unknown keys). Only set for failures that a caller is expected
+    /// to branch on, e.g. `BucketRemoteTargetVersionMismatch`.
+    #[serde(rename = "Code", skip_serializing_if = "Option::is_none")]
+    code: Option<&'static str>,
 }
 
 impl Default for ReplicationCheckPhaseStatus {
@@ -266,6 +277,7 @@ impl Default for ReplicationCheckPhaseStatus {
         Self {
             status: "SKIPPED",
             error: None,
+            code: None,
         }
     }
 }
@@ -275,6 +287,7 @@ impl ReplicationCheckPhaseStatus {
         Self {
             status: "OK",
             error: None,
+            code: None,
         }
     }
 
@@ -282,6 +295,14 @@ impl ReplicationCheckPhaseStatus {
         Self {
             status: "FAILED",
             error: Some(bound_replication_check_error(error.into())),
+            code: None,
+        }
+    }
+
+    fn failed_with_code(error: impl Into<String>, code: &'static str) -> Self {
+        Self {
+            code: Some(code),
+            ..Self::failed(error)
         }
     }
 }
@@ -2046,12 +2067,39 @@ fn fail_replication_check_target(result: &mut ReplicationCheckTargetStatus, erro
     }
 }
 
+/// The probe PUT reports both sides of the version-identity contract: the
+/// source version id it sent (header + `?versionId=` query, the exact shape
+/// live replication uses) and the version id the target answered with.
+struct ReplicationProbePutOutcome {
+    sent_version_id: String,
+    response_version_id: Option<String>,
+}
+
+struct ReplicationProbeMultipartError {
+    primary: S3ClientError,
+    cleanup_error: Option<String>,
+}
+
+impl From<S3ClientError> for ReplicationProbeMultipartError {
+    fn from(primary: S3ClientError) -> Self {
+        Self {
+            primary,
+            cleanup_error: None,
+        }
+    }
+}
+
 #[async_trait::async_trait]
 trait ReplicationProbeOperations {
-    async fn put(&mut self) -> Result<Option<String>, S3ClientError>;
+    async fn put(&mut self) -> Result<ReplicationProbePutOutcome, S3ClientError>;
+    /// Multipart decides the target version at initiate time and only reports
+    /// it on completion, so the identity contract has to be probed separately
+    /// there: a target can adopt PutObject version ids and still mint its own
+    /// for CreateMultipartUpload.
+    async fn multipart_put(&mut self) -> Result<ReplicationProbePutOutcome, ReplicationProbeMultipartError>;
     async fn create_delete_marker(&mut self, version_id: Option<&str>) -> Result<Option<String>, S3ClientError>;
     async fn delete_version(&mut self, version_id: Option<&str>) -> Result<(), S3ClientError>;
-    async fn cleanup(&mut self, known_version_ids: [Option<&str>; 2]) -> Result<(), String>;
+    async fn cleanup(&mut self, known_version_ids: [Option<&str>; 3]) -> Result<(), String>;
 }
 
 struct RemoteReplicationProbeOperations<'a> {
@@ -2063,10 +2111,14 @@ struct RemoteReplicationProbeOperations<'a> {
 
 #[async_trait::async_trait]
 impl ReplicationProbeOperations for RemoteReplicationProbeOperations<'_> {
-    async fn put(&mut self) -> Result<Option<String>, S3ClientError> {
+    async fn put(&mut self) -> Result<ReplicationProbePutOutcome, S3ClientError> {
         put_replication_probe_object(self.client, self.bucket, self.key, self.time).await
     }
 
+    async fn multipart_put(&mut self) -> Result<ReplicationProbePutOutcome, ReplicationProbeMultipartError> {
+        multipart_put_replication_probe_object(self.client, self.bucket, self.key, self.time).await
+    }
+
     async fn create_delete_marker(&mut self, version_id: Option<&str>) -> Result<Option<String>, S3ClientError> {
         delete_replication_probe_object(
             self.client,
@@ -2090,20 +2142,49 @@ impl ReplicationProbeOperations for RemoteReplicationProbeOperations<'_> {
         .map(|_| ())
     }
 
-    async fn cleanup(&mut self, known_version_ids: [Option<&str>; 2]) -> Result<(), String> {
+    async fn cleanup(&mut self, known_version_ids: [Option<&str>; 3]) -> Result<(), String> {
         cleanup_replication_probe(self.client, self.bucket, self.key, known_version_ids).await
     }
 }
 
+/// `None` when the target adopted the source version id on this path.
+fn version_fidelity_error(api: &str, outcome: &ReplicationProbePutOutcome) -> Option<String> {
+    if outcome.response_version_id.as_deref() == Some(outcome.sent_version_id.as_str()) {
+        return None;
+    }
+    Some(format!(
+        "target assigned version id {} instead of adopting the source version id {} on {api}; \
+         version-addressed replication (version deletes, heal) cannot converge on this target",
+        outcome.response_version_id.as_deref().unwrap_or("<none>"),
+        outcome.sent_version_id,
+    ))
+}
+
 async fn execute_replication_probe(result: &mut ReplicationCheckTargetStatus, operations: &mut impl ReplicationProbeOperations) {
     let mut probe_version_id = None;
+    let mut multipart_probe_version_id = None;
     let mut delete_marker_version_id = None;
     let mut cleanup_required = true;
+    let mut multipart_cleanup_error = None;
 
     match operations.put().await {
-        Ok(version_id) => {
-            probe_version_id = version_id;
+        Ok(outcome) => {
             result.phases.put = ReplicationCheckPhaseStatus::passed();
+            // P1-19 version-identity contract: replication only converges on
+            // targets that adopt the source version id — version-addressed
+            // deletes and heal re-drives never match a minted id. Judge it
+            // from the probe PUT's own response; on mismatch the later
+            // mutation phases are pointless (they address by version id), but
+            // cleanup still runs against whatever id the target assigned.
+            match version_fidelity_error("PutObject", &outcome) {
+                None => result.phases.version_fidelity = ReplicationCheckPhaseStatus::passed(),
+                Some(error) => {
+                    result.phases.version_fidelity =
+                        ReplicationCheckPhaseStatus::failed_with_code(&error, REPLICATION_CHECK_CODE_VERSION_MISMATCH);
+                    fail_replication_check_target(result, error);
+                }
+            }
+            probe_version_id = outcome.response_version_id;
         }
         Err(err) => {
             let error = format_replication_check_client_error(&err, ReplicationCheckFailureContext::ReplicateObject);
@@ -2115,7 +2196,29 @@ async fn execute_replication_probe(result: &mut ReplicationCheckTargetStatus, op
         }
     }
 
-    if result.phases.put.status == "OK" {
+    // The multipart path fixes the target version at initiate and only
+    // reports it on completion, so a target can adopt PutObject ids and still
+    // mint its own here — probe it before declaring the contract met.
+    if result.phases.version_fidelity.status == "OK" {
+        match operations.multipart_put().await {
+            Ok(outcome) => {
+                multipart_probe_version_id = outcome.response_version_id.clone();
+                if let Some(error) = version_fidelity_error("CreateMultipartUpload", &outcome) {
+                    result.phases.version_fidelity =
+                        ReplicationCheckPhaseStatus::failed_with_code(&error, REPLICATION_CHECK_CODE_VERSION_MISMATCH);
+                    fail_replication_check_target(result, error);
+                }
+            }
+            Err(err) => {
+                let error = format_replication_check_client_error(&err.primary, ReplicationCheckFailureContext::ReplicateObject);
+                result.phases.version_fidelity = ReplicationCheckPhaseStatus::failed(&error);
+                fail_replication_check_target(result, error);
+                multipart_cleanup_error = err.cleanup_error;
+            }
+        }
+    }
+
+    if result.phases.put.status == "OK" && result.phases.version_fidelity.status == "OK" {
         match operations.create_delete_marker(probe_version_id.as_deref()).await {
             Ok(version_id) => {
                 delete_marker_version_id = version_id;
@@ -2138,19 +2241,31 @@ async fn execute_replication_probe(result: &mut ReplicationCheckTargetStatus, op
         }
     }
 
-    if cleanup_required {
-        match operations
-            .cleanup([probe_version_id.as_deref(), delete_marker_version_id.as_deref()])
+    let cleanup_result = if cleanup_required {
+        operations
+            .cleanup([
+                probe_version_id.as_deref(),
+                multipart_probe_version_id.as_deref(),
+                delete_marker_version_id.as_deref(),
+            ])
             .await
-        {
-            Ok(()) => result.phases.cleanup = ReplicationCheckPhaseStatus::passed(),
-            Err(error) => {
-                result.phases.cleanup = ReplicationCheckPhaseStatus::failed(&error);
-                fail_replication_check_target(result, format!("probe cleanup failed: {error}"));
-            }
-        }
     } else {
+        Ok(())
+    };
+
+    let mut cleanup_errors = Vec::new();
+    if let Some(error) = multipart_cleanup_error {
+        cleanup_errors.push(error);
+    }
+    if let Err(error) = cleanup_result {
+        cleanup_errors.push(error);
+    }
+    if cleanup_errors.is_empty() {
         result.phases.cleanup = ReplicationCheckPhaseStatus::passed();
+    } else {
+        let error = cleanup_errors.join("; ");
+        result.phases.cleanup = ReplicationCheckPhaseStatus::failed(&error);
+        fail_replication_check_target(result, format!("probe cleanup failed: {error}"));
     }
 }
 
@@ -2225,13 +2340,7 @@ fn build_replication_probe_remove_options(now: OffsetDateTime, replication_delet
     }
 }
 
-async fn put_replication_probe_object(
-    target_client: &TargetClient,
-    target_bucket: &str,
-    probe_key: &str,
-    now: OffsetDateTime,
-) -> Result<Option<String>, S3ClientError> {
-    let options = build_replication_probe_put_options(now);
+fn build_replication_probe_headers(options: &PutObjectOptions) -> HeaderMap {
     let mut headers = HeaderMap::new();
     insert_header(&mut headers, SUFFIX_SOURCE_VERSION_ID, &options.internal.source_version_id);
     insert_header(
@@ -2245,8 +2354,166 @@ async fn put_replication_probe_object(
         HeaderName::from_static("x-amz-replication-status"),
         HeaderValue::from_static(ReplicationStatusType::Replica.as_str()),
     );
+    headers
+}
 
-    target_client
+/// Probe the identity contract on the multipart path: initiate carrying the
+/// source version as `?versionId=` (where the target fixes the version),
+/// upload one small part, and read the version the completion reports.
+async fn multipart_put_replication_probe_object(
+    target_client: &TargetClient,
+    target_bucket: &str,
+    probe_key: &str,
+    now: OffsetDateTime,
+) -> Result<ReplicationProbePutOutcome, ReplicationProbeMultipartError> {
+    let options = build_replication_probe_put_options(now);
+    let sent_version_id = options.internal.source_version_id.clone();
+    let headers = build_replication_probe_headers(&options);
+
+    let initiate_headers = headers.clone();
+    let initiate_version_id = sent_version_id.clone();
+    let created = target_client
+        .client
+        .create_multipart_upload()
+        .bucket(target_bucket)
+        .key(probe_key)
+        .customize()
+        .map_request(move |mut req| {
+            for (key, value) in initiate_headers.clone() {
+                req.headers_mut().insert(key.expect("operation should succeed"), value);
+            }
+            let uri = append_version_id_query(req.uri(), &initiate_version_id);
+            req.set_uri(uri).map_err(std::io::Error::other)?;
+            Result::<_, std::io::Error>::Ok(req)
+        })
+        .send()
+        .await
+        .map_err(S3ClientError::from)
+        .map_err(ReplicationProbeMultipartError::from)?;
+    let upload_id = created
+        .upload_id()
+        .ok_or_else(|| S3ClientError::new("target multipart initiate returned no upload id"))
+        .map_err(ReplicationProbeMultipartError::from)?
+        .to_string();
+
+    let uploaded = match target_client
+        .client
+        .upload_part()
+        .bucket(target_bucket)
+        .key(probe_key)
+        .upload_id(&upload_id)
+        .part_number(1)
+        .content_length(8)
+        .body(AwsByteStream::from_static(b"aaaaaaaa"))
+        .send()
+        .await
+    {
+        Ok(uploaded) => uploaded,
+        Err(error) => {
+            return Err(abort_failed_replication_probe_multipart(
+                target_client,
+                target_bucket,
+                probe_key,
+                &upload_id,
+                S3ClientError::from(error),
+            )
+            .await);
+        }
+    };
+
+    let completed_part = CompletedPart::builder()
+        .part_number(1)
+        .set_e_tag(uploaded.e_tag().map(ToOwned::to_owned))
+        .build();
+    let complete_headers = headers.clone();
+    let completed = match target_client
+        .client
+        .complete_multipart_upload()
+        .bucket(target_bucket)
+        .key(probe_key)
+        .upload_id(&upload_id)
+        .multipart_upload(
+            CompletedMultipartUpload::builder()
+                .set_parts(Some(vec![completed_part]))
+                .build(),
+        )
+        .customize()
+        .map_request(move |mut req| {
+            for (key, value) in complete_headers.clone() {
+                req.headers_mut().insert(key.expect("operation should succeed"), value);
+            }
+            Result::<_, std::io::Error>::Ok(req)
+        })
+        .send()
+        .await
+    {
+        Ok(completed) => completed,
+        Err(error) => {
+            return Err(abort_failed_replication_probe_multipart(
+                target_client,
+                target_bucket,
+                probe_key,
+                &upload_id,
+                S3ClientError::from(error),
+            )
+            .await);
+        }
+    };
+
+    Ok(ReplicationProbePutOutcome {
+        sent_version_id,
+        response_version_id: completed.version_id().map(ToOwned::to_owned),
+    })
+}
+
+async fn abort_failed_replication_probe_multipart(
+    target_client: &TargetClient,
+    target_bucket: &str,
+    probe_key: &str,
+    upload_id: &str,
+    primary_error: S3ClientError,
+) -> ReplicationProbeMultipartError {
+    match target_client
+        .client
+        .abort_multipart_upload()
+        .bucket(target_bucket)
+        .key(probe_key)
+        .upload_id(upload_id)
+        .send()
+        .await
+    {
+        Ok(_) => ReplicationProbeMultipartError::from(primary_error),
+        Err(error) => {
+            let abort_error = S3ClientError::from(error);
+            if abort_error.code.as_deref() == Some("NoSuchUpload") {
+                ReplicationProbeMultipartError::from(primary_error)
+            } else {
+                ReplicationProbeMultipartError {
+                    primary: primary_error,
+                    cleanup_error: Some("failed to abort multipart replication probe".to_string()),
+                }
+            }
+        }
+    }
+}
+
+async fn put_replication_probe_object(
+    target_client: &TargetClient,
+    target_bucket: &str,
+    probe_key: &str,
+    now: OffsetDateTime,
+) -> Result<ReplicationProbePutOutcome, S3ClientError> {
+    let options = build_replication_probe_put_options(now);
+    let sent_version_id = options.internal.source_version_id.clone();
+    let headers = build_replication_probe_headers(&options);
+
+    // Carry the source version as `?versionId=` exactly like a live
+    // replication PUT (P0-5 shape): the probe must exercise the query the
+    // real data path relies on, and the response tells us whether the target
+    // adopts the id. The probe id is always a fresh non-nil UUID, so the
+    // null-version mapping in the live path does not apply here.
+    let query_version_id = sent_version_id.clone();
+    let response = target_client
         .client
         .put_object()
         .bucket(target_bucket)
@@ -2259,12 +2526,18 @@ async fn put_replication_probe_object(
             for (key, value) in headers.clone() {
                 req.headers_mut().insert(key.expect("operation should succeed"), value);
             }
+            let uri = append_version_id_query(req.uri(), &query_version_id);
+            req.set_uri(uri).map_err(std::io::Error::other)?;
             Result::<_, std::io::Error>::Ok(req)
         })
         .send()
         .await
-        .map(|output| output.version_id().map(ToOwned::to_owned))
-        .map_err(S3ClientError::from)
+        .map_err(S3ClientError::from)?;
+
+    Ok(ReplicationProbePutOutcome {
+        sent_version_id,
+        response_version_id: response.version_id().map(ToOwned::to_owned),
+    })
 }
 
 async fn delete_replication_probe_object(
@@ -3431,6 +3704,12 @@ mod tests {
     #[derive(Default)]
     struct ScriptedReplicationProbe {
         put_error: Option<&'static str>,
+        /// Version id the scripted target answers with on PUT; None models a
+        /// mirroring target that echoes the sent source version id.
+        minted_version_id: Option<&'static str>,
+        /// Same, for the multipart leg: a target may mirror PutObject ids and
+        /// still mint its own at CreateMultipartUpload.
+        minted_multipart_version_id: Option<&'static str>,
         delete_marker_error: Option<&'static str>,
         version_delete_error: Option<&'static str>,
         cleanup_error: Option<&'static str>,
@@ -3449,14 +3728,25 @@ mod tests {
 
     #[async_trait::async_trait]
     impl ReplicationProbeOperations for ScriptedReplicationProbe {
-        async fn put(&mut self) -> Result<Option<String>, S3ClientError> {
+        async fn put(&mut self) -> Result<ReplicationProbePutOutcome, S3ClientError> {
             self.calls.push("put");
             match self.put_error {
                 Some(code) => Err(scripted_probe_error(code)),
-                None => Ok(Some("object-version".to_string())),
+                None => Ok(ReplicationProbePutOutcome {
+                    sent_version_id: "object-version".to_string(),
+                    response_version_id: Some(self.minted_version_id.unwrap_or("object-version").to_string()),
+                }),
             }
         }
 
+        async fn multipart_put(&mut self) -> Result<ReplicationProbePutOutcome, ReplicationProbeMultipartError> {
+            self.calls.push("multipart-put");
+            Ok(ReplicationProbePutOutcome {
+                sent_version_id: "multipart-version".to_string(),
+                response_version_id: Some(self.minted_multipart_version_id.unwrap_or("multipart-version").to_string()),
+            })
+        }
+
         async fn create_delete_marker(&mut self, _version_id: Option<&str>) -> Result<Option<String>, S3ClientError> {
             self.calls.push("delete-marker");
             match self.delete_marker_error {
@@ -3473,7 +3763,7 @@ mod tests {
             }
         }
 
-        async fn cleanup(&mut self, known_version_ids: [Option<&str>; 2]) -> Result<(), String> {
+        async fn cleanup(&mut self, known_version_ids: [Option<&str>; 3]) -> Result<(), String> {
             self.calls.push("cleanup");
             self.cleanup_ids = known_version_ids
                 .into_iter()
@@ -3486,6 +3776,44 @@ mod tests {
         }
     }
 
+    /// P1-19: a target that mints its own version ids must fail the
+    /// VersionFidelity phase with the machine-readable mismatch code, skip
+    /// the version-addressed mutation phases (they cannot mean anything on a
+    /// drifting target), and still clean up using the id the target actually
+    /// assigned — the source-derived id would never match.
+    #[tokio::test]
+    async fn replication_probe_flags_version_minting_target() {
+        let mut result = replication_check_target("arn:a", "OK", None);
+        let mut operations = ScriptedReplicationProbe {
+            minted_version_id: Some("target-minted-version"),
+            ..Default::default()
+        };
+
+        execute_replication_probe(&mut result, &mut operations).await;
+
+        assert_eq!(operations.calls, ["put", "cleanup"]);
+        assert_eq!(result.status, "FAILED");
+        assert_eq!(result.phases.put.status, "OK");
+        assert_eq!(result.phases.version_fidelity.status, "FAILED");
+        assert_eq!(result.phases.version_fidelity.code, Some(REPLICATION_CHECK_CODE_VERSION_MISMATCH));
+        assert_eq!(result.phases.delete_marker.status, "SKIPPED");
+        assert_eq!(result.phases.version_delete.status, "SKIPPED");
+        assert_eq!(result.phases.cleanup.status, "OK");
+        assert_eq!(operations.cleanup_ids, [Some("target-minted-version".to_string()), None, None]);
+    }
+
+    #[tokio::test]
+    async fn replication_probe_passes_version_fidelity_for_mirroring_target() {
+        let mut result = replication_check_target("arn:a", "OK", None);
+        let mut operations = ScriptedReplicationProbe::default();
+
+        execute_replication_probe(&mut result, &mut operations).await;
+
+        assert_eq!(result.status, "OK");
+        assert_eq!(result.phases.version_fidelity.status, "OK");
+        assert_eq!(result.phases.version_fidelity.code, None);
+    }
+
     #[tokio::test]
     async fn replication_probe_attempts_cleanup_after_ambiguous_put_failure() {
         let mut result = replication_check_target("arn:a", "OK", None);
@@ -3514,8 +3842,15 @@ mod tests {
 
         execute_replication_probe(&mut result, &mut operations).await;
 
-        assert_eq!(operations.calls, ["put", "delete-marker", "version-delete", "cleanup"]);
-        assert_eq!(operations.cleanup_ids, [Some("object-version".to_string()), None]);
+        assert_eq!(operations.calls, ["put", "multipart-put", "delete-marker", "version-delete", "cleanup"]);
+        assert_eq!(
+            operations.cleanup_ids,
+            [
+                Some("object-version".to_string()),
+                Some("multipart-version".to_string()),
+                None
+            ]
+        );
         assert_eq!(result.phases.delete_marker.status, "FAILED");
         assert_eq!(result.phases.version_delete.status, "OK");
         assert_eq!(result.phases.cleanup.status, "OK");
@@ -3532,10 +3867,14 @@ mod tests {
 
         execute_replication_probe(&mut result, &mut operations).await;
 
-        assert_eq!(operations.calls, ["put", "delete-marker", "version-delete", "cleanup"]);
+        assert_eq!(operations.calls, ["put", "multipart-put", "delete-marker", "version-delete", "cleanup"]);
         assert_eq!(
             operations.cleanup_ids,
-            [Some("object-version".to_string()), Some("marker-version".to_string())]
+            [
+                Some("object-version".to_string()),
+                Some("multipart-version".to_string()),
+                Some("marker-version".to_string())
+            ]
         );
         assert_eq!(result.phases.version_delete.status, "FAILED");
         assert_eq!(result.phases.cleanup.status, "FAILED");

rustfs/src/admin/storage_api.rs

@@ -183,6 +183,7 @@ pub(crate) mod bandwidth {
 }
 
 pub(crate) mod bucket_target_sys {
+    pub(crate) use super::ecstore_bucket::bucket_target_sys::append_version_id_query;
     pub(crate) type AdvancedPutOptions = super::ecstore_bucket::bucket_target_sys::AdvancedPutOptions;
     pub(crate) type BucketTargetError = super::ecstore_bucket::bucket_target_sys::BucketTargetError;
     pub(crate) type BucketTargetSys = super::ecstore_bucket::bucket_target_sys::BucketTargetSys;