fix(replication): fail SSE-C passthrough closed on targets that drop transport headers

SSE-C ciphertext passthrough replicates via X-Rustfs-Replication-* transport headers. A MinIO/generic-S3 target silently discards them, storing bare ciphertext with no decryption material — yet the PUT succeeded, so the object reported COMPLETED with a silently unreadable replica (backlog#1675 N2). Fail-closed design: - SsecPassthroughCapability {Unknown, Supported, Unsupported} cached in BucketTargetSys per target ARN with a recording timestamp. Entries reset whenever the target is rebuilt, edited, or removed (arn_remotes_map lifecycle) and expire after SSEC_PASSTHROUGH_CAPABILITY_TTL (10 minutes): an expired verdict in either direction is re-earned through the audit, so an Unsupported target recovers automatically after an upgrade (at most one wasted PUT+HEAD audit per bad target per TTL window) and a Supported verdict cannot outlive a backend swapped behind the same endpoint. - Replication worker (replicate_object and replicate_all): fresh Unsupported targets never receive the PUT — the attempt fails immediately into the normal MRF retry channel with a "run ?replication-check to re-probe" hint. Unknown or expired verdicts are audited: after the PUT the worker HEADs the replica back through the replication-check channel (source version id mapped through resolve_read_api_version_id, so null-version objects audit correctly) and requires SSE-C evidence (the echoed customer-algorithm header); missing evidence records Unsupported and fails the attempt. Convergence HEADs are audited the same way, so a broken ciphertext replica from an earlier attempt can never launder itself into COMPLETED via an ETag match. The gate/evidence policy is pure (replication_target_boundary, staleness folded in as an input) for the M2 worker migration. - replication-check grows an SsecPassthrough probe phase: a probe PUT carrying the live transport-header shape, HEAD-back for evidence, and a machine-readable Code BucketRemoteSsecPassthroughUnsupported on failure. The probe verdict is synced into the runtime capability cache. Unlike VersionFidelity, a failed SsecPassthrough phase does NOT fail the target overall — it is a capability limit, not a broken replication contract, and a plaintext-only deployment against such a target must not turn red. - fake_s3_target: default mode now models a RustFS target (stores the transport headers, echoes SSE-C evidence); the new drop_unlisted_replication_headers mode models MinIO. The journal records whether a request carried transport headers. Receiver-echo verification: the replication-check HEAD exemption only skips SSE-C key validation; the response has always built sse-customer-algorithm from stored metadata (rustfs/src/app/object_usecase.rs), so no receiver change was needed — pinned end to end by the replication-check e2e against a real RustFS target. Rolling-upgrade constraint: RustFS targets older than the replication-check HEAD exemption (#5898) answer the audit HEAD without SSE-C evidence (or fail it outright), so SSE-C replication to such targets reports FAILED. This is deliberate — FAILED-and-retryable beats a silently undecryptable replica — and self-heals: once the target is upgraded, the next TTL expiry (or a manual ?replication-check re-probe) re-audits and records Supported. Plaintext and managed-SSE replication are unaffected. The capability cache is per-node; each node audits independently. Known limitations: - The audit judges evidence from the echoed customer-algorithm header only. A hypothetical target that preserves that one header while dropping other transport headers (partial-drop) would pass the audit; no known target behaves this way — observed targets drop the whole unknown-header family. - A mixed-version target cluster can flap the verdict between audits routed to different target nodes until the rollout completes; the TTL bounds how long each stale verdict persists. New e2e (backlog#1675 C1 + N2, red-first): fail-closed against a header-dropping fake (FAILED + no second PUT via the capability cache, journal-asserted; red run showed the old COMPLETED), replication-check reports the SsecPassthrough phase Code while the target stays OK overall, SSE-C heal convergence after a real target outage, and SSE-C existing-object resync landing a REPLICA readable with the customer key. TTL expiry in both directions is pinned at the cache and gate seams.
2026-08-18 18:46:17 +00:00 · 2026-08-18 00:54:26 +08:00
parent 9baa92563a
commit d20476a66c
8 changed files with 1300 additions and 31 deletions
@@ -90,6 +90,13 @@ const SOURCE_LEGALHOLD_TIMESTAMP_HEADERS: [&str; 2] = [
    "x-rustfs-source-replication-legalhold-timestamp",
    "x-minio-source-replication-legalhold-timestamp",
 ];
+/// Wire prefix of the SSE-C passthrough replication transport headers
+/// (`X-Rustfs-Replication-*`). In the default mode the fake stores them like a
+/// RustFS target and echoes SSE-C evidence back on HEAD/GET; with
+/// [`FakeS3Target::drop_unlisted_replication_headers`] it models MinIO /
+/// generic S3, which silently discard unknown x-* headers.
+const REPLICATION_SSE_TRANSPORT_PREFIX: &str = "x-rustfs-replication-";
+const REPLICATION_SSEC_ALGORITHM_TRANSPORT_HEADER: &str = "x-rustfs-replication-ssec-algorithm";
 const RESERVED_BUCKET_PREFIXES: [&str; 3] = ["xn--", "sthree-", "amzn-s3-demo-"];
 const RESERVED_BUCKET_SUFFIXES: [&str; 6] = ["-s3alias", "--ol-s3", ".mrap", "--x-s3", "--table-s3", "-an"];

@@ -166,6 +173,10 @@ pub struct ProxyHeaderSnapshot {
    pub ssec_algorithm: Option<String>,
    pub ssec_key_present: bool,
    pub ssec_key_md5: Option<String>,
+    /// Whether the request carried any `X-Rustfs-Replication-*` SSE-C
+    /// passthrough transport header, so fail-closed tests can assert the
+    /// sender really shipped the material a dropping target discarded.
+    pub ssec_transport_present: bool,
 }

 impl ProxyHeaderSnapshot {
@@ -179,6 +190,9 @@ impl ProxyHeaderSnapshot {
                .map(bounded_journal_value),
            ssec_key_present: headers.contains_key("x-amz-server-side-encryption-customer-key"),
            ssec_key_md5: header_value(headers, &["x-amz-server-side-encryption-customer-key-md5"]).map(bounded_journal_value),
+            ssec_transport_present: headers
+                .keys()
+                .any(|name| name.as_str().starts_with(REPLICATION_SSE_TRANSPORT_PREFIX)),
        }
    }
 }
@@ -213,6 +227,10 @@ struct ControlState {
 struct StoreState {
    assign_own_version_ids: bool,
    assign_own_multipart_version_ids: bool,
+    /// MinIO-like mode: silently discard non-whitelisted replication
+    /// transport headers instead of storing them (see
+    /// [`REPLICATION_SSE_TRANSPORT_PREFIX`]).
+    drop_unlisted_replication_headers: bool,
    buckets: HashMap<String, BucketState>,
    uploads: HashMap<String, MultipartState>,
    total_bytes: usize,
@@ -237,6 +255,9 @@ struct ObjectVersion {
    /// Object tags as ordered key/value pairs (PutObjectTagging replaces the
    /// whole set, DeleteObjectTagging clears it).
    tags: Vec<(String, String)>,
+    /// SSE-C passthrough transport headers stored with the version (RustFS
+    /// target behavior); empty when the drop mode discarded them.
+    replication_sse_headers: Vec<(String, String)>,
 }

 #[derive(Clone)]
@@ -246,6 +267,7 @@ struct MultipartState {
    version_id: String,
    content_type: Option<String>,
    metadata: Option<HashMap<String, String>>,
+    replication_sse_headers: Vec<(String, String)>,
    parts: BTreeMap<i32, MultipartPart>,
 }

@@ -466,6 +488,15 @@ impl FakeS3Target {

    /// Mint own version ids for the multipart path only — models a target
    /// that adopts PutObject version ids but not CreateMultipartUpload ones.
+    /// MinIO-like mode: silently drop every `X-Rustfs-Replication-*` SSE-C
+    /// passthrough transport header instead of storing it. The default (off)
+    /// models a RustFS target, which preserves the headers and echoes SSE-C
+    /// evidence (`x-amz-server-side-encryption-customer-algorithm`) on
+    /// HEAD/GET of the replica.
+    pub fn drop_unlisted_replication_headers(&self, enabled: bool) {
+        lock(&self.backend.store).drop_unlisted_replication_headers = enabled;
+    }
+
    pub fn assign_own_multipart_version_ids(&self, enabled: bool) {
        lock(&self.backend.store).assign_own_multipart_version_ids = enabled;
    }
@@ -842,6 +873,29 @@ fn new_version_id(headers: &HeaderMap, assign_own: bool) -> S3Result<String> {
    Ok(version_id.to_string())
 }

+/// Capture the SSE-C passthrough transport headers a replication PUT carried.
+/// Returns an empty set in the MinIO-like drop mode.
+fn captured_replication_sse_headers(headers: &HeaderMap, drop_unlisted: bool) -> Vec<(String, String)> {
+    if drop_unlisted {
+        return Vec::new();
+    }
+    headers
+        .iter()
+        .filter(|(name, _)| name.as_str().starts_with(REPLICATION_SSE_TRANSPORT_PREFIX))
+        .filter_map(|(name, value)| Some((name.as_str().to_string(), value.to_str().ok()?.to_string())))
+        .collect()
+}
+
+/// SSE-C evidence a RustFS-like target echoes for a stored passthrough
+/// replica: the customer algorithm restored from the transport headers.
+fn stored_sse_customer_algorithm(version: &ObjectVersion) -> Option<String> {
+    version
+        .replication_sse_headers
+        .iter()
+        .find(|(name, _)| name == REPLICATION_SSEC_ALGORITHM_TRANSPORT_HEADER)
+        .map(|(_, value)| value.clone())
+}
+
 fn source_etag(headers: &HeaderMap) -> S3Result<Option<String>> {
    header_value(headers, &SOURCE_ETAG_HEADERS)
        .map(|value| validate_retained_identifier(value, "source ETag").map(|value| normalize_etag(&value)))
@@ -1312,7 +1366,10 @@ impl S3 for FakeBackend {
        let input = req.input;
        let body = collect_stream(input.body, input.content_length, fault.as_ref(), &self.control).await?;
        validate_stored_metadata(&input.content_type, &input.metadata)?;
-        let assign_own = lock(&self.store).assign_own_version_ids;
+        let (assign_own, drop_unlisted) = {
+            let state = lock(&self.store);
+            (state.assign_own_version_ids, state.drop_unlisted_replication_headers)
+        };
        let version_id = new_version_id(&headers, assign_own)?;
        let e_tag = match source_etag(&headers)? {
            Some(value) => value,
@@ -1330,6 +1387,7 @@ impl S3 for FakeBackend {
            content_type: input.content_type,
            metadata: input.metadata,
            tags: Vec::new(),
+            replication_sse_headers: captured_replication_sse_headers(&headers, drop_unlisted),
        };
        upsert_version(&mut lock(&self.store), &input.bucket, input.key, version)?;
        Ok(apply_response_fault(
@@ -1350,6 +1408,7 @@ impl S3 for FakeBackend {
            let state = lock(&self.store);
            find_version(&state, &input.bucket, &input.key, input.version_id.as_deref())?
        };
+        let sse_customer_algorithm = stored_sse_customer_algorithm(&version);
        Ok(apply_response_fault(
            S3Response::new(GetObjectOutput {
                body: Some(StreamingBlob::new(Body::from(version.body.clone()))),
@@ -1359,6 +1418,7 @@ impl S3 for FakeBackend {
                e_tag: Some(ETag::Strong(version.e_tag)),
                last_modified: Some(version.last_modified.clone()),
                version_id: Some(version.version_id),
+                sse_customer_algorithm,
                ..Default::default()
            }),
            fault.as_ref(),
@@ -1373,6 +1433,7 @@ impl S3 for FakeBackend {
            let state = lock(&self.store);
            find_version(&state, &input.bucket, &input.key, input.version_id.as_deref())?
        };
+        let sse_customer_algorithm = stored_sse_customer_algorithm(&version);
        Ok(apply_response_fault(
            S3Response::new(HeadObjectOutput {
                content_length: Some(version.body.len() as i64),
@@ -1381,6 +1442,7 @@ impl S3 for FakeBackend {
                e_tag: Some(ETag::Strong(version.e_tag)),
                last_modified: Some(version.last_modified.clone()),
                version_id: Some(version.version_id),
+                sse_customer_algorithm,
                ..Default::default()
            }),
            fault.as_ref(),
@@ -1530,6 +1592,7 @@ impl S3 for FakeBackend {
                content_type: None,
                metadata: None,
                tags: Vec::new(),
+                replication_sse_headers: Vec::new(),
            },
        )?;
        Ok(apply_response_fault(
@@ -1557,9 +1620,10 @@ impl S3 for FakeBackend {
        ensure_upload_budget(&state)?;
        validate_stored_metadata(&input.content_type, &input.metadata)?;
        let upload_id = Uuid::new_v4().to_string();
-        // Read the flag before the mutable borrow of `state.uploads` below
+        // Read the flags before the mutable borrow of `state.uploads` below
        // (and never re-lock the store: the mutex is not reentrant).
        let mint_own = state.assign_own_version_ids || state.assign_own_multipart_version_ids;
+        let drop_unlisted = state.drop_unlisted_replication_headers;
        let version_id = new_version_id(&headers, mint_own)?;
        state.uploads.insert(
            upload_id.clone(),
@@ -1569,6 +1633,7 @@ impl S3 for FakeBackend {
                version_id,
                content_type: input.content_type,
                metadata: input.metadata,
+                replication_sse_headers: captured_replication_sse_headers(&headers, drop_unlisted),
                parts: BTreeMap::new(),
            },
        );
@@ -1706,6 +1771,7 @@ impl S3 for FakeBackend {
                    version_id: upload.version_id.clone(),
                    content_type: upload.content_type.clone(),
                    metadata: upload.metadata.clone(),
+                    replication_sse_headers: upload.replication_sse_headers.clone(),
                    parts: BTreeMap::new(),
                },
                selected,
@@ -1733,6 +1799,7 @@ impl S3 for FakeBackend {
            content_type: upload.content_type,
            metadata: upload.metadata,
            tags: Vec::new(),
+            replication_sse_headers: upload.replication_sse_headers,
        };
        let mut state = lock(&self.store);
        let current = state
@@ -1937,6 +2004,65 @@ mod tests {
        Ok(())
    }

+    /// Default mode is RustFS-like: SSE-C passthrough transport headers are
+    /// stored and the customer algorithm is echoed on HEAD/GET. Drop mode is
+    /// MinIO-like: the headers are silently discarded, so no evidence comes
+    /// back — the exact difference the N2 fail-closed audit keys on. Both
+    /// modes journal that the sender shipped the transport headers.
+    #[tokio::test]
+    async fn ssec_passthrough_headers_echo_and_drop_modes() -> Result<(), BoxError> {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket("target-bucket");
+        let client = client(&target);
+
+        let put_with_transport_headers = |key: &'static str| {
+            client
+                .put_object()
+                .bucket("target-bucket")
+                .key(key)
+                .body(ByteStream::from_static(b"ciphertext"))
+                .customize()
+                .map_request(move |mut request| {
+                    let headers = request.headers_mut();
+                    headers.insert("x-rustfs-replication-ssec-algorithm", "AES256");
+                    headers.insert("x-rustfs-replication-ssec-key-md5", "AAAAAAAAAAAAAAAAAAAAAA==");
+                    Ok::<_, std::convert::Infallible>(request)
+                })
+                .send()
+        };
+
+        put_with_transport_headers("kept").await?;
+        let head = client.head_object().bucket("target-bucket").key("kept").send().await?;
+        assert_eq!(head.sse_customer_algorithm(), Some("AES256"));
+        let get = client.get_object().bucket("target-bucket").key("kept").send().await?;
+        assert_eq!(get.sse_customer_algorithm(), Some("AES256"));
+
+        target.drop_unlisted_replication_headers(true);
+        put_with_transport_headers("dropped").await?;
+        let head = client.head_object().bucket("target-bucket").key("dropped").send().await?;
+        assert_eq!(head.sse_customer_algorithm(), None, "drop mode must discard SSE-C evidence");
+
+        let requests = target.requests();
+        for key in ["kept", "dropped"] {
+            let record = requests
+                .iter()
+                .find(|record| record.operation == Operation::PutObject && record.key.as_deref() == Some(key))
+                .expect("PUT must be journaled");
+            assert!(
+                record.proxy_headers.ssec_transport_present,
+                "the journal must prove the sender shipped the transport headers for {key}"
+            );
+        }
+        let plain_head = requests
+            .iter()
+            .find(|record| record.operation == Operation::HeadObject)
+            .expect("HEAD must be journaled");
+        assert!(!plain_head.proxy_headers.ssec_transport_present);
+
+        target.shutdown().await;
+        Ok(())
+    }
+
    macro_rules! assert_sdk_error {
        ($error:expr, $status:expr, $code:expr) => {{
            let error = &$error;
@@ -3202,6 +3328,7 @@ mod tests {
                        version_id: index.to_string(),
                        content_type: None,
                        metadata: None,
+                        replication_sse_headers: Vec::new(),
                        parts: BTreeMap::new(),
                    },
                );
@@ -2610,17 +2610,20 @@ async fn test_replication_check_succeeds_with_remote_target() -> Result<(), Box<

    assert_eq!(response.status(), StatusCode::OK);
    let payload: serde_json::Value = response.json().await?;
-    assert_eq!(payload["Status"], "OK");
+    assert_eq!(payload["Status"], "OK", "{payload}");
    assert_eq!(payload["ActiveMutation"], true);
    assert_eq!(payload["Targets"].as_array().map(Vec::len), Some(1));
-    assert_eq!(payload["Targets"][0]["Status"], "OK");
-    assert_eq!(payload["Targets"][0]["Phases"]["Put"]["Status"], "OK");
+    assert_eq!(payload["Targets"][0]["Status"], "OK", "{payload}");
+    assert_eq!(payload["Targets"][0]["Phases"]["Put"]["Status"], "OK", "{payload}");
    // A RustFS target adopts the source version id, so the P1-19
    // version-identity probe passes.
-    assert_eq!(payload["Targets"][0]["Phases"]["VersionFidelity"]["Status"], "OK");
-    assert_eq!(payload["Targets"][0]["Phases"]["DeleteMarker"]["Status"], "OK");
-    assert_eq!(payload["Targets"][0]["Phases"]["VersionDelete"]["Status"], "OK");
-    assert_eq!(payload["Targets"][0]["Phases"]["Cleanup"]["Status"], "OK");
+    assert_eq!(payload["Targets"][0]["Phases"]["VersionFidelity"]["Status"], "OK", "{payload}");
+    // A RustFS target preserves the SSE-C passthrough transport headers and
+    // echoes the customer algorithm on the replication-check HEAD (N2).
+    assert_eq!(payload["Targets"][0]["Phases"]["SsecPassthrough"]["Status"], "OK", "{payload}");
+    assert_eq!(payload["Targets"][0]["Phases"]["DeleteMarker"]["Status"], "OK", "{payload}");
+    assert_eq!(payload["Targets"][0]["Phases"]["VersionDelete"]["Status"], "OK", "{payload}");
+    assert_eq!(payload["Targets"][0]["Phases"]["Cleanup"]["Status"], "OK", "{payload}");

    let target_client = target_env.create_s3_client();
    let versions = target_client
@@ -4649,6 +4652,410 @@ async fn test_bucket_replication_sse_c_multipart_passthrough() -> TestResult {
    Ok(())
 }

+/// N2 (backlog#1675 P1-22): SSE-C passthrough replication to a target that
+/// silently drops the `X-Rustfs-Replication-*` transport headers (MinIO-like
+/// behavior, modeled by the fake target's drop mode) used to report COMPLETED
+/// while the replica had irrecoverably lost its decryption material — the red
+/// light this test was born failing on. Fail-closed contract now under test:
+/// the first attempt PUTs, HEAD-backs the replica, finds no SSE-C evidence,
+/// records the target Unsupported and reports FAILED; a second SSE-C object
+/// fails without any PUT reaching the target (capability cache, proven from
+/// the target journal); plaintext objects still replicate COMPLETED.
+#[tokio::test]
+#[serial]
+async fn test_ssec_replication_fails_closed_when_target_drops_passthrough_headers() -> TestResult {
+    init_logging();
+
+    let target = FakeS3Target::start().await?;
+    let target_bucket = "ssec-drop-dst";
+    target.create_bucket(target_bucket);
+    target.drop_unlisted_replication_headers(true);
+
+    let mut source_env = RustFSTestEnvironment::new().await?;
+    let mut env_vars = replication_fast_env();
+    env_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
+    env_vars.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
+    source_env.start_rustfs_server_with_env(vec![], &env_vars).await?;
+
+    let source_bucket = "ssec-drop-src";
+    let source_client = source_env.create_s3_client();
+    source_client.create_bucket().bucket(source_bucket).send().await?;
+    enable_bucket_versioning(&source_env, source_bucket).await?;
+    let target_arn = set_replication_target_with_options(
+        &source_env,
+        source_bucket,
+        ReplicationTargetOptions {
+            endpoint: &target.address(),
+            access_key: FAKE_ACCESS_KEY,
+            secret_key: FAKE_SECRET_KEY,
+            target_bucket,
+            secure: false,
+            skip_tls_verify: false,
+            ca_cert_pem: None,
+        },
+    )
+    .await?;
+    put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
+
+    let customer_key = BASE64_STANDARD.encode(REPL17_SSEC_KEY);
+    let customer_key_md5 = sse_customer_key_md5_base64(REPL17_SSEC_KEY);
+    let put_ssec = |key: &'static str| {
+        source_client
+            .put_object()
+            .bucket(source_bucket)
+            .key(key)
+            .body(ByteStream::from_static(b"ssec fail-closed payload"))
+            .sse_customer_algorithm("AES256")
+            .sse_customer_key(&customer_key)
+            .sse_customer_key_md5(&customer_key_md5)
+            .send()
+    };
+
+    // First SSE-C object: the audit must catch the dropped material.
+    put_ssec("ssec-first.txt").await?;
+    wait_for_source_replication_status(&source_client, source_bucket, "ssec-first.txt", "FAILED", true).await?;
+
+    let requests = target.take_requests();
+    let first_put = requests
+        .iter()
+        .find(|record| record.operation == FakeTargetOperation::PutObject && record.key.as_deref() == Some("ssec-first.txt"))
+        .ok_or("the first SSE-C object must have been PUT (capability was Unknown)")?;
+    assert!(
+        first_put.proxy_headers.ssec_transport_present,
+        "the replication PUT must have shipped the SSE-C transport headers the target then dropped"
+    );
+    assert!(
+        requests.iter().any(|record| {
+            record.operation == FakeTargetOperation::HeadObject
+                && record.key.as_deref() == Some("ssec-first.txt")
+                && record.sequence > first_put.sequence
+                && record.proxy_headers.replication_check.as_deref() == Some("true")
+        }),
+        "the post-PUT HEAD-back audit must have run through the replication-check channel; journal: {requests:?}"
+    );
+
+    // Second SSE-C object: the cached Unsupported verdict fails it closed
+    // before any PUT — including MRF retries of the first object.
+    put_ssec("ssec-second.txt").await?;
+    wait_for_source_replication_status(&source_client, source_bucket, "ssec-second.txt", "FAILED", true).await?;
+    assert!(
+        !target.requests().iter().any(|record| {
+            record.operation == FakeTargetOperation::PutObject
+                && record.key.as_deref() != Some("plain-control.txt")
+                && record.proxy_headers.ssec_transport_present
+        }),
+        "no further SSE-C ciphertext may reach a target recorded Unsupported; journal: {:?}",
+        target.requests()
+    );
+
+    // The gate is scoped to SSE-C: plaintext replication keeps working.
+    source_client
+        .put_object()
+        .bucket(source_bucket)
+        .key("plain-control.txt")
+        .body(ByteStream::from_static(b"plaintext control payload"))
+        .send()
+        .await?;
+    wait_for_source_replication_status(&source_client, source_bucket, "plain-control.txt", "COMPLETED", false).await?;
+    assert!(target.has_object(target_bucket, "plain-control.txt"));
+
+    target.shutdown().await;
+    Ok(())
+}
+
+/// N2 (backlog#1675 P1-22): the admin replication-check must expose the same
+/// verdict operators would otherwise only learn from failing SSE-C objects —
+/// an SsecPassthrough probe phase that fails with the machine-readable
+/// `BucketRemoteSsecPassthroughUnsupported` code against a header-dropping
+/// target, with no probe residue left behind. The target's overall status
+/// stays OK: unlike version-identity drift, dropped passthrough headers are
+/// a capability limit, and a plaintext-only deployment against a MinIO-like
+/// target must not turn red.
+#[tokio::test]
+#[serial]
+async fn test_replication_check_flags_ssec_passthrough_dropping_target() -> TestResult {
+    init_logging();
+
+    let target = FakeS3Target::start().await?;
+    let target_bucket = "ssec-check-dst";
+    target.create_bucket(target_bucket);
+    target.drop_unlisted_replication_headers(true);
+
+    let mut source_env = RustFSTestEnvironment::new().await?;
+    let mut env_vars = replication_fast_env();
+    env_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
+    env_vars.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
+    source_env.start_rustfs_server_with_env(vec![], &env_vars).await?;
+
+    let source_bucket = "ssec-check-src";
+    let source_client = source_env.create_s3_client();
+    source_client.create_bucket().bucket(source_bucket).send().await?;
+    enable_bucket_versioning(&source_env, source_bucket).await?;
+    let target_arn = set_replication_target_with_options(
+        &source_env,
+        source_bucket,
+        ReplicationTargetOptions {
+            endpoint: &target.address(),
+            access_key: FAKE_ACCESS_KEY,
+            secret_key: FAKE_SECRET_KEY,
+            target_bucket,
+            secure: false,
+            skip_tls_verify: false,
+            ca_cert_pem: None,
+        },
+    )
+    .await?;
+    put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
+
+    let response = run_replication_check(&source_env, source_bucket).await?;
+    assert_eq!(response.status(), StatusCode::OK);
+    let payload: serde_json::Value = response.json().await?;
+
+    assert_eq!(
+        payload["Status"], "OK",
+        "a capability-only SSE-C failure must not fail the check overall: {payload}"
+    );
+    let target_report = &payload["Targets"][0];
+    assert_eq!(target_report["Status"], "OK", "{payload}");
+    let ssec = &target_report["Phases"]["SsecPassthrough"];
+    assert_eq!(ssec["Status"], "FAILED", "SsecPassthrough phase must fail: {payload}");
+    assert_eq!(
+        ssec["Code"], "BucketRemoteSsecPassthroughUnsupported",
+        "the failure must carry the machine-readable code: {payload}"
+    );
+    // Basic replication of plaintext objects works on this target: every other
+    // phase passes, so the code is the discriminator operators branch on.
+    assert_eq!(target_report["Phases"]["Put"]["Status"], "OK", "{payload}");
+    assert_eq!(target_report["Phases"]["VersionFidelity"]["Status"], "OK", "{payload}");
+    assert_eq!(target_report["Phases"]["DeleteMarker"]["Status"], "OK", "{payload}");
+    assert_eq!(target_report["Phases"]["VersionDelete"]["Status"], "OK", "{payload}");
+    assert_eq!(target_report["Phases"]["Cleanup"]["Status"], "OK", "{payload}");
+
+    // The SSE-C probe PUT must have shipped the real transport header names —
+    // a mangled or missing header set would fail the phase for the wrong
+    // reason and mask a working target.
+    let requests = target.requests();
+    assert!(
+        requests
+            .iter()
+            .any(|record| record.operation == FakeTargetOperation::PutObject && record.proxy_headers.ssec_transport_present),
+        "the SSE-C probe PUT must carry the X-Rustfs-Replication-* transport headers; journal: {requests:?}"
+    );
+
+    // No probe residue, including the SSE-C probe version.
+    let probe_put = requests
+        .into_iter()
+        .find(|record| record.operation == FakeTargetOperation::PutObject)
+        .ok_or("the probe PUT never reached the fake target")?;
+    let probe_key = probe_put.key.ok_or("probe PUT journal record has no key")?;
+    assert!(
+        target.stored_versions(target_bucket, &probe_key).is_empty(),
+        "all probe versions must be cleaned up"
+    );
+
+    target.shutdown().await;
+    Ok(())
+}
+
+/// C1 (backlog#1675 P1-22): heal-path convergence for SSE-C. An SSE-C object
+/// whose live replication failed during a target outage must converge through
+/// the scanner/heal compensation once the target returns — passing the N2
+/// HEAD-back audit against the recovered RustFS target — and the replica must
+/// be readable with the customer key.
+#[tokio::test]
+#[serial]
+async fn test_bucket_replication_sse_c_heals_after_target_outage() -> TestResult {
+    init_logging();
+
+    let (source_env, mut target_env, source_bucket, target_bucket) =
+        build_sse_replication_pair("ssec-heal", false, false).await?;
+    let source_client = source_env.create_s3_client();
+    let key = "ssec-heal-contract.txt";
+    let body = b"repl-22 ssec heal payload".to_vec();
+    let customer_key = BASE64_STANDARD.encode(REPL17_SSEC_KEY);
+    let customer_key_md5 = sse_customer_key_md5_base64(REPL17_SSEC_KEY);
+
+    // Target outage: the SSE-C write cannot replicate.
+    target_env.stop_server();
+
+    source_client
+        .put_object()
+        .bucket(&source_bucket)
+        .key(key)
+        .body(ByteStream::from(body.clone()))
+        .sse_customer_algorithm("AES256")
+        .sse_customer_key(&customer_key)
+        .sse_customer_key_md5(&customer_key_md5)
+        .send()
+        .await?;
+
+    // The failure is observable on the source (SSE-C HEAD needs the key).
+    let deadline = tokio::time::Instant::now() + Duration::from_secs(30);
+    loop {
+        let head = source_client
+            .head_object()
+            .bucket(&source_bucket)
+            .key(key)
+            .sse_customer_algorithm("AES256")
+            .sse_customer_key(&customer_key)
+            .sse_customer_key_md5(&customer_key_md5)
+            .send()
+            .await?;
+        match head.replication_status().map(|status| status.as_str()) {
+            Some("PENDING") | Some("FAILED") => break,
+            other => {
+                if tokio::time::Instant::now() >= deadline {
+                    return Err(format!("source SSE-C object never reported PENDING/FAILED; last status={other:?}").into());
+                }
+                sleep(Duration::from_millis(200)).await;
+            }
+        }
+    }
+
+    // Recover the target in place; the source scanner re-drives the failure.
+    target_env
+        .restart_server_preserving_data(vec![], &[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")])
+        .await?;
+
+    wait_for_source_replication_status(&source_client, &source_bucket, key, "COMPLETED", true).await?;
+
+    // The healed replica is a REPLICA (status surfaces on HEAD) readable with
+    // the customer key.
+    let target_client = target_env.create_s3_client();
+    let replica_head = target_client
+        .head_object()
+        .bucket(&target_bucket)
+        .key(key)
+        .sse_customer_algorithm("AES256")
+        .sse_customer_key(&customer_key)
+        .sse_customer_key_md5(&customer_key_md5)
+        .send()
+        .await?;
+    assert_eq!(
+        replica_head.replication_status().map(|status| status.as_str()),
+        Some("REPLICA"),
+        "the healed copy must carry REPLICA status"
+    );
+    let replica = target_client
+        .get_object()
+        .bucket(&target_bucket)
+        .key(key)
+        .sse_customer_algorithm("AES256")
+        .sse_customer_key(&customer_key)
+        .sse_customer_key_md5(&customer_key_md5)
+        .send()
+        .await?;
+    assert_eq!(replica.sse_customer_algorithm(), Some("AES256"));
+    assert_eq!(replica.body.collect().await?.into_bytes().as_ref(), body.as_slice());
+
+    Ok(())
+}
+
+/// C1 (backlog#1675 P1-22): existing-object resync for SSE-C. An SSE-C object
+/// written BEFORE any replication config must reach the RustFS target through
+/// the existing-object resync (`replicate_all` transport, N2-audited), land as
+/// a REPLICA, and read back with the customer key.
+#[tokio::test]
+#[serial]
+async fn test_bucket_replication_sse_c_existing_object_resync() -> TestResult {
+    init_logging();
+
+    let mut source_env = RustFSTestEnvironment::new().await?;
+    let mut source_process_env = replication_fast_env();
+    source_process_env.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
+    source_process_env.extend_from_slice(FAST_SCANNER_ENV);
+    source_process_env.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
+    source_env.start_rustfs_server_with_env(vec![], &source_process_env).await?;
+
+    let mut target_env = RustFSTestEnvironment::new().await?;
+    target_env
+        .start_rustfs_server_without_cleanup_with_env(&[
+            ("NO_PROXY", "127.0.0.1,localhost"),
+            ("HTTP_PROXY", ""),
+            ("HTTPS_PROXY", ""),
+        ])
+        .await?;
+
+    let source_bucket = "ssec-existing-src";
+    let target_bucket = "ssec-existing-dst";
+    let source_client = source_env.create_s3_client();
+    let target_client = target_env.create_s3_client();
+    source_client.create_bucket().bucket(source_bucket).send().await?;
+    target_client.create_bucket().bucket(target_bucket).send().await?;
+    enable_bucket_versioning(&source_env, source_bucket).await?;
+    enable_bucket_versioning(&target_env, target_bucket).await?;
+
+    // The SSE-C object exists before any replication wiring.
+    let key = "ssec-existing-contract.txt";
+    let body = b"repl-22 ssec existing-object payload".to_vec();
+    let customer_key = BASE64_STANDARD.encode(REPL17_SSEC_KEY);
+    let customer_key_md5 = sse_customer_key_md5_base64(REPL17_SSEC_KEY);
+    source_client
+        .put_object()
+        .bucket(source_bucket)
+        .key(key)
+        .body(ByteStream::from(body.clone()))
+        .sse_customer_algorithm("AES256")
+        .sse_customer_key(&customer_key)
+        .sse_customer_key_md5(&customer_key_md5)
+        .send()
+        .await?;
+
+    // Wire replication (existing-object enabled) and drive a resync.
+    let target_arn = set_replication_target(&source_env, source_bucket, &target_env, target_bucket).await?;
+    put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
+    let (reset_arn, reset_id) = start_bucket_replication_reset(&source_env, source_bucket).await?;
+    assert_eq!(reset_arn, target_arn);
+    let terminal = wait_for_replication_reset_target(&source_env, source_bucket, &target_arn, |status| {
+        status.reset_id == reset_id && matches!(status.status.as_str(), "Completed" | "Failed")
+    })
+    .await?;
+    assert_eq!(terminal.status, "Completed", "SSE-C existing-object resync must complete");
+    assert!(terminal.replicated_count >= 1, "the existing SSE-C object must have been resynced");
+
+    // The replica is a REPLICA (status surfaces on HEAD) readable with the
+    // customer key.
+    let replica_head = target_client
+        .head_object()
+        .bucket(target_bucket)
+        .key(key)
+        .sse_customer_algorithm("AES256")
+        .sse_customer_key(&customer_key)
+        .sse_customer_key_md5(&customer_key_md5)
+        .send()
+        .await?;
+    assert_eq!(
+        replica_head.replication_status().map(|status| status.as_str()),
+        Some("REPLICA"),
+        "the resynced copy must carry REPLICA status"
+    );
+    let replica = target_client
+        .get_object()
+        .bucket(target_bucket)
+        .key(key)
+        .sse_customer_algorithm("AES256")
+        .sse_customer_key(&customer_key)
+        .sse_customer_key_md5(&customer_key_md5)
+        .send()
+        .await?;
+    assert_eq!(replica.sse_customer_algorithm(), Some("AES256"));
+    assert_eq!(replica.body.collect().await?.into_bytes().as_ref(), body.as_slice());
+
+    // No plaintext leak: the replica stays unreadable without the key.
+    assert!(
+        target_client
+            .get_object()
+            .bucket(target_bucket)
+            .key(key)
+            .send()
+            .await
+            .is_err(),
+        "SSE-C replica must not be readable without the customer key"
+    );
+
+    Ok(())
+}
+
 /// backlog#1147 repl-17 / backlog#1783: SSE-S3 objects replicate by decrypting
 /// at the source and re-encrypting on the target with the target's own KMS.
 /// The property backlog#1291 pinned — never a silent plaintext replica — still