feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets (#6172)

* feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets

Implements the MinIO active-active read-proxy protocol (P1-5 of the
replication compatibility review): when a GET/HEAD/GetObjectTagging/
PutObjectTagging/DeleteObjectTagging request fails locally with
not-found and the bucket has replication targets, the request is proxied
to the targets in rule order, mirroring bucket-replication.go
proxyGetToReplicationTarget/proxyHeadToRepTarget/proxyTaggingToRepTarget.

Protocol surface:
- Anti-loop: inbound {x-rustfs-,x-minio-}source-proxy-request is parsed
  into ObjectOptions (proxy_request + proxy_header_set, matching MinIO
  ProxyRequest/ProxyHeaderSet); a request carrying the marker with ANY
  value is never re-proxied. Outbound client proxy calls send the marker
  as "true"; replication worker convergence HEADs send it as "false" so
  a peer's proxy layer cannot answer a convergence check by proxying
  back to the source (which would fake Completed without a PUT).
- Target selection: new replication_proxy.rs get_proxy_targets — empty
  when the marker is set, versioning is suspended, or no replication
  config; otherwise filter_target_arns -> TargetClient lookup, skipping
  targets with proxying disabled.
- TargetClient gains head_object_for_proxy/get_object (streaming) and
  the three tagging calls. Proxy calls never send the replication-check
  SSE-C exemption header; customer SSE-C keys are forwarded verbatim so
  the target performs real decryption. Conditional (If-*) headers are
  not forwarded (MinIO parity); Range and part_number are, with
  parts_count/tag_count/storage_class/expiration passed through.
- Metrics: proxy counters now count only real client proxy traffic,
  MinIO-aligned (one total per proxied request, one failed when no
  target served it). The previous misattributed counters — replication
  worker HEAD/PUT (#2672) and local tagging operations (#2682) — are
  removed; ReplProxyMetric now maps the tagging counters instead of
  dropping them.

e2e (fake_s3_target extended with tagging + header journaling): proxied
GET body + outbound header contract (marker present, no
replication-check, SSE-C passthrough), HEAD, anti-loop 404 with zero
outbound requests, GetObjectTagging, and metric mapping unit tests.

Rolling note: proxying only activates for buckets with replication
targets; requests carrying the marker keep pre-upgrade behavior.

Refs rustfs/backlog#1675 (P1-5)

* fix(replication): fail SSE-C passthrough closed on targets that drop transport headers (#6178)

SSE-C ciphertext passthrough replicates via X-Rustfs-Replication-* transport
headers. A MinIO/generic-S3 target silently discards them, storing bare
ciphertext with no decryption material — yet the PUT succeeded, so the object
reported COMPLETED with a silently unreadable replica (backlog#1675 N2).

Fail-closed design:
- SsecPassthroughCapability {Unknown, Supported, Unsupported} cached in
  BucketTargetSys per target ARN with a recording timestamp. Entries reset
  whenever the target is rebuilt, edited, or removed (arn_remotes_map
  lifecycle) and expire after SSEC_PASSTHROUGH_CAPABILITY_TTL (10 minutes):
  an expired verdict in either direction is re-earned through the audit, so
  an Unsupported target recovers automatically after an upgrade (at most one
  wasted PUT+HEAD audit per bad target per TTL window) and a Supported
  verdict cannot outlive a backend swapped behind the same endpoint.
- Replication worker (replicate_object and replicate_all): fresh Unsupported
  targets never receive the PUT — the attempt fails immediately into the
  normal MRF retry channel with a "run ?replication-check to re-probe" hint.
  Unknown or expired verdicts are audited: after the PUT the worker HEADs
  the replica back through the replication-check channel (source version id
  mapped through resolve_read_api_version_id, so null-version objects audit
  correctly) and requires SSE-C evidence (the echoed customer-algorithm
  header); missing evidence records Unsupported and fails the attempt.
  Convergence HEADs are audited the same way, so a broken ciphertext replica
  from an earlier attempt can never launder itself into COMPLETED via an
  ETag match. The gate/evidence policy is pure (replication_target_boundary,
  staleness folded in as an input) for the M2 worker migration.
- replication-check grows an SsecPassthrough probe phase: a probe PUT
  carrying the live transport-header shape, HEAD-back for evidence, and a
  machine-readable Code BucketRemoteSsecPassthroughUnsupported on failure.
  The probe verdict is synced into the runtime capability cache. Unlike
  VersionFidelity, a failed SsecPassthrough phase does NOT fail the target
  overall — it is a capability limit, not a broken replication contract,
  and a plaintext-only deployment against such a target must not turn red.
- fake_s3_target: default mode now models a RustFS target (stores the
  transport headers, echoes SSE-C evidence); the new
  drop_unlisted_replication_headers mode models MinIO. The journal records
  whether a request carried transport headers.

Receiver-echo verification: the replication-check HEAD exemption only skips
SSE-C key validation; the response has always built sse-customer-algorithm
from stored metadata (rustfs/src/app/object_usecase.rs), so no receiver
change was needed — pinned end to end by the replication-check e2e against
a real RustFS target.

Rolling-upgrade constraint: RustFS targets older than the replication-check
HEAD exemption (#5898) answer the audit HEAD without SSE-C evidence (or fail
it outright), so SSE-C replication to such targets reports FAILED. This is
deliberate — FAILED-and-retryable beats a silently undecryptable replica —
and self-heals: once the target is upgraded, the next TTL expiry (or a
manual ?replication-check re-probe) re-audits and records Supported.
Plaintext and managed-SSE replication are unaffected. The capability cache
is per-node; each node audits independently.

Known limitations:
- The audit judges evidence from the echoed customer-algorithm header only.
  A hypothetical target that preserves that one header while dropping other
  transport headers (partial-drop) would pass the audit; no known target
  behaves this way — observed targets drop the whole unknown-header family.
- A mixed-version target cluster can flap the verdict between audits routed
  to different target nodes until the rollout completes; the TTL bounds how
  long each stale verdict persists.

New e2e (backlog#1675 C1 + N2, red-first): fail-closed against a
header-dropping fake (FAILED + no second PUT via the capability cache,
journal-asserted; red run showed the old COMPLETED), replication-check
reports the SsecPassthrough phase Code while the target stays OK overall,
SSE-C heal convergence after a real target outage, and SSE-C
existing-object resync landing a REPLICA readable with the customer key.
TTL expiry in both directions is pinned at the cache and gate seams.
This commit is contained in:
唐小鸭
2026-08-18 08:47:22 +08:00
committed by GitHub
parent 21c2fb42bb
commit daecb93139
22 changed files with 2769 additions and 189 deletions
@@ -2610,17 +2610,20 @@ async fn test_replication_check_succeeds_with_remote_target() -> Result<(), Box<
assert_eq!(response.status(), StatusCode::OK);
let payload: serde_json::Value = response.json().await?;
assert_eq!(payload["Status"], "OK");
assert_eq!(payload["Status"], "OK", "{payload}");
assert_eq!(payload["ActiveMutation"], true);
assert_eq!(payload["Targets"].as_array().map(Vec::len), Some(1));
assert_eq!(payload["Targets"][0]["Status"], "OK");
assert_eq!(payload["Targets"][0]["Phases"]["Put"]["Status"], "OK");
assert_eq!(payload["Targets"][0]["Status"], "OK", "{payload}");
assert_eq!(payload["Targets"][0]["Phases"]["Put"]["Status"], "OK", "{payload}");
// A RustFS target adopts the source version id, so the P1-19
// version-identity probe passes.
assert_eq!(payload["Targets"][0]["Phases"]["VersionFidelity"]["Status"], "OK");
assert_eq!(payload["Targets"][0]["Phases"]["DeleteMarker"]["Status"], "OK");
assert_eq!(payload["Targets"][0]["Phases"]["VersionDelete"]["Status"], "OK");
assert_eq!(payload["Targets"][0]["Phases"]["Cleanup"]["Status"], "OK");
assert_eq!(payload["Targets"][0]["Phases"]["VersionFidelity"]["Status"], "OK", "{payload}");
// A RustFS target preserves the SSE-C passthrough transport headers and
// echoes the customer algorithm on the replication-check HEAD (N2).
assert_eq!(payload["Targets"][0]["Phases"]["SsecPassthrough"]["Status"], "OK", "{payload}");
assert_eq!(payload["Targets"][0]["Phases"]["DeleteMarker"]["Status"], "OK", "{payload}");
assert_eq!(payload["Targets"][0]["Phases"]["VersionDelete"]["Status"], "OK", "{payload}");
assert_eq!(payload["Targets"][0]["Phases"]["Cleanup"]["Status"], "OK", "{payload}");
let target_client = target_env.create_s3_client();
let versions = target_client
@@ -4649,6 +4652,410 @@ async fn test_bucket_replication_sse_c_multipart_passthrough() -> TestResult {
Ok(())
}
/// N2 (backlog#1675 P1-22): SSE-C passthrough replication to a target that
/// silently drops the `X-Rustfs-Replication-*` transport headers (MinIO-like
/// behavior, modeled by the fake target's drop mode) used to report COMPLETED
/// while the replica had irrecoverably lost its decryption material — the red
/// light this test was born failing on. Fail-closed contract now under test:
/// the first attempt PUTs, HEAD-backs the replica, finds no SSE-C evidence,
/// records the target Unsupported and reports FAILED; a second SSE-C object
/// fails without any PUT reaching the target (capability cache, proven from
/// the target journal); plaintext objects still replicate COMPLETED.
#[tokio::test]
#[serial]
async fn test_ssec_replication_fails_closed_when_target_drops_passthrough_headers() -> TestResult {
init_logging();
let target = FakeS3Target::start().await?;
let target_bucket = "ssec-drop-dst";
target.create_bucket(target_bucket);
target.drop_unlisted_replication_headers(true);
let mut source_env = RustFSTestEnvironment::new().await?;
let mut env_vars = replication_fast_env();
env_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
env_vars.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
source_env.start_rustfs_server_with_env(vec![], &env_vars).await?;
let source_bucket = "ssec-drop-src";
let source_client = source_env.create_s3_client();
source_client.create_bucket().bucket(source_bucket).send().await?;
enable_bucket_versioning(&source_env, source_bucket).await?;
let target_arn = set_replication_target_with_options(
&source_env,
source_bucket,
ReplicationTargetOptions {
endpoint: &target.address(),
access_key: FAKE_ACCESS_KEY,
secret_key: FAKE_SECRET_KEY,
target_bucket,
secure: false,
skip_tls_verify: false,
ca_cert_pem: None,
},
)
.await?;
put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
let customer_key = BASE64_STANDARD.encode(REPL17_SSEC_KEY);
let customer_key_md5 = sse_customer_key_md5_base64(REPL17_SSEC_KEY);
let put_ssec = |key: &'static str| {
source_client
.put_object()
.bucket(source_bucket)
.key(key)
.body(ByteStream::from_static(b"ssec fail-closed payload"))
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
};
// First SSE-C object: the audit must catch the dropped material.
put_ssec("ssec-first.txt").await?;
wait_for_source_replication_status(&source_client, source_bucket, "ssec-first.txt", "FAILED", true).await?;
let requests = target.take_requests();
let first_put = requests
.iter()
.find(|record| record.operation == FakeTargetOperation::PutObject && record.key.as_deref() == Some("ssec-first.txt"))
.ok_or("the first SSE-C object must have been PUT (capability was Unknown)")?;
assert!(
first_put.proxy_headers.ssec_transport_present,
"the replication PUT must have shipped the SSE-C transport headers the target then dropped"
);
assert!(
requests.iter().any(|record| {
record.operation == FakeTargetOperation::HeadObject
&& record.key.as_deref() == Some("ssec-first.txt")
&& record.sequence > first_put.sequence
&& record.proxy_headers.replication_check.as_deref() == Some("true")
}),
"the post-PUT HEAD-back audit must have run through the replication-check channel; journal: {requests:?}"
);
// Second SSE-C object: the cached Unsupported verdict fails it closed
// before any PUT — including MRF retries of the first object.
put_ssec("ssec-second.txt").await?;
wait_for_source_replication_status(&source_client, source_bucket, "ssec-second.txt", "FAILED", true).await?;
assert!(
!target.requests().iter().any(|record| {
record.operation == FakeTargetOperation::PutObject
&& record.key.as_deref() != Some("plain-control.txt")
&& record.proxy_headers.ssec_transport_present
}),
"no further SSE-C ciphertext may reach a target recorded Unsupported; journal: {:?}",
target.requests()
);
// The gate is scoped to SSE-C: plaintext replication keeps working.
source_client
.put_object()
.bucket(source_bucket)
.key("plain-control.txt")
.body(ByteStream::from_static(b"plaintext control payload"))
.send()
.await?;
wait_for_source_replication_status(&source_client, source_bucket, "plain-control.txt", "COMPLETED", false).await?;
assert!(target.has_object(target_bucket, "plain-control.txt"));
target.shutdown().await;
Ok(())
}
/// N2 (backlog#1675 P1-22): the admin replication-check must expose the same
/// verdict operators would otherwise only learn from failing SSE-C objects —
/// an SsecPassthrough probe phase that fails with the machine-readable
/// `BucketRemoteSsecPassthroughUnsupported` code against a header-dropping
/// target, with no probe residue left behind. The target's overall status
/// stays OK: unlike version-identity drift, dropped passthrough headers are
/// a capability limit, and a plaintext-only deployment against a MinIO-like
/// target must not turn red.
#[tokio::test]
#[serial]
async fn test_replication_check_flags_ssec_passthrough_dropping_target() -> TestResult {
init_logging();
let target = FakeS3Target::start().await?;
let target_bucket = "ssec-check-dst";
target.create_bucket(target_bucket);
target.drop_unlisted_replication_headers(true);
let mut source_env = RustFSTestEnvironment::new().await?;
let mut env_vars = replication_fast_env();
env_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
env_vars.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
source_env.start_rustfs_server_with_env(vec![], &env_vars).await?;
let source_bucket = "ssec-check-src";
let source_client = source_env.create_s3_client();
source_client.create_bucket().bucket(source_bucket).send().await?;
enable_bucket_versioning(&source_env, source_bucket).await?;
let target_arn = set_replication_target_with_options(
&source_env,
source_bucket,
ReplicationTargetOptions {
endpoint: &target.address(),
access_key: FAKE_ACCESS_KEY,
secret_key: FAKE_SECRET_KEY,
target_bucket,
secure: false,
skip_tls_verify: false,
ca_cert_pem: None,
},
)
.await?;
put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
let response = run_replication_check(&source_env, source_bucket).await?;
assert_eq!(response.status(), StatusCode::OK);
let payload: serde_json::Value = response.json().await?;
assert_eq!(
payload["Status"], "OK",
"a capability-only SSE-C failure must not fail the check overall: {payload}"
);
let target_report = &payload["Targets"][0];
assert_eq!(target_report["Status"], "OK", "{payload}");
let ssec = &target_report["Phases"]["SsecPassthrough"];
assert_eq!(ssec["Status"], "FAILED", "SsecPassthrough phase must fail: {payload}");
assert_eq!(
ssec["Code"], "BucketRemoteSsecPassthroughUnsupported",
"the failure must carry the machine-readable code: {payload}"
);
// Basic replication of plaintext objects works on this target: every other
// phase passes, so the code is the discriminator operators branch on.
assert_eq!(target_report["Phases"]["Put"]["Status"], "OK", "{payload}");
assert_eq!(target_report["Phases"]["VersionFidelity"]["Status"], "OK", "{payload}");
assert_eq!(target_report["Phases"]["DeleteMarker"]["Status"], "OK", "{payload}");
assert_eq!(target_report["Phases"]["VersionDelete"]["Status"], "OK", "{payload}");
assert_eq!(target_report["Phases"]["Cleanup"]["Status"], "OK", "{payload}");
// The SSE-C probe PUT must have shipped the real transport header names —
// a mangled or missing header set would fail the phase for the wrong
// reason and mask a working target.
let requests = target.requests();
assert!(
requests
.iter()
.any(|record| record.operation == FakeTargetOperation::PutObject && record.proxy_headers.ssec_transport_present),
"the SSE-C probe PUT must carry the X-Rustfs-Replication-* transport headers; journal: {requests:?}"
);
// No probe residue, including the SSE-C probe version.
let probe_put = requests
.into_iter()
.find(|record| record.operation == FakeTargetOperation::PutObject)
.ok_or("the probe PUT never reached the fake target")?;
let probe_key = probe_put.key.ok_or("probe PUT journal record has no key")?;
assert!(
target.stored_versions(target_bucket, &probe_key).is_empty(),
"all probe versions must be cleaned up"
);
target.shutdown().await;
Ok(())
}
/// C1 (backlog#1675 P1-22): heal-path convergence for SSE-C. An SSE-C object
/// whose live replication failed during a target outage must converge through
/// the scanner/heal compensation once the target returns — passing the N2
/// HEAD-back audit against the recovered RustFS target — and the replica must
/// be readable with the customer key.
#[tokio::test]
#[serial]
async fn test_bucket_replication_sse_c_heals_after_target_outage() -> TestResult {
init_logging();
let (source_env, mut target_env, source_bucket, target_bucket) =
build_sse_replication_pair("ssec-heal", false, false).await?;
let source_client = source_env.create_s3_client();
let key = "ssec-heal-contract.txt";
let body = b"repl-22 ssec heal payload".to_vec();
let customer_key = BASE64_STANDARD.encode(REPL17_SSEC_KEY);
let customer_key_md5 = sse_customer_key_md5_base64(REPL17_SSEC_KEY);
// Target outage: the SSE-C write cannot replicate.
target_env.stop_server();
source_client
.put_object()
.bucket(&source_bucket)
.key(key)
.body(ByteStream::from(body.clone()))
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
.await?;
// The failure is observable on the source (SSE-C HEAD needs the key).
let deadline = tokio::time::Instant::now() + Duration::from_secs(30);
loop {
let head = source_client
.head_object()
.bucket(&source_bucket)
.key(key)
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
.await?;
match head.replication_status().map(|status| status.as_str()) {
Some("PENDING") | Some("FAILED") => break,
other => {
if tokio::time::Instant::now() >= deadline {
return Err(format!("source SSE-C object never reported PENDING/FAILED; last status={other:?}").into());
}
sleep(Duration::from_millis(200)).await;
}
}
}
// Recover the target in place; the source scanner re-drives the failure.
target_env
.restart_server_preserving_data(vec![], &[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")])
.await?;
wait_for_source_replication_status(&source_client, &source_bucket, key, "COMPLETED", true).await?;
// The healed replica is a REPLICA (status surfaces on HEAD) readable with
// the customer key.
let target_client = target_env.create_s3_client();
let replica_head = target_client
.head_object()
.bucket(&target_bucket)
.key(key)
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
.await?;
assert_eq!(
replica_head.replication_status().map(|status| status.as_str()),
Some("REPLICA"),
"the healed copy must carry REPLICA status"
);
let replica = target_client
.get_object()
.bucket(&target_bucket)
.key(key)
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
.await?;
assert_eq!(replica.sse_customer_algorithm(), Some("AES256"));
assert_eq!(replica.body.collect().await?.into_bytes().as_ref(), body.as_slice());
Ok(())
}
/// C1 (backlog#1675 P1-22): existing-object resync for SSE-C. An SSE-C object
/// written BEFORE any replication config must reach the RustFS target through
/// the existing-object resync (`replicate_all` transport, N2-audited), land as
/// a REPLICA, and read back with the customer key.
#[tokio::test]
#[serial]
async fn test_bucket_replication_sse_c_existing_object_resync() -> TestResult {
init_logging();
let mut source_env = RustFSTestEnvironment::new().await?;
let mut source_process_env = replication_fast_env();
source_process_env.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
source_process_env.extend_from_slice(FAST_SCANNER_ENV);
source_process_env.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
source_env.start_rustfs_server_with_env(vec![], &source_process_env).await?;
let mut target_env = RustFSTestEnvironment::new().await?;
target_env
.start_rustfs_server_without_cleanup_with_env(&[
("NO_PROXY", "127.0.0.1,localhost"),
("HTTP_PROXY", ""),
("HTTPS_PROXY", ""),
])
.await?;
let source_bucket = "ssec-existing-src";
let target_bucket = "ssec-existing-dst";
let source_client = source_env.create_s3_client();
let target_client = target_env.create_s3_client();
source_client.create_bucket().bucket(source_bucket).send().await?;
target_client.create_bucket().bucket(target_bucket).send().await?;
enable_bucket_versioning(&source_env, source_bucket).await?;
enable_bucket_versioning(&target_env, target_bucket).await?;
// The SSE-C object exists before any replication wiring.
let key = "ssec-existing-contract.txt";
let body = b"repl-22 ssec existing-object payload".to_vec();
let customer_key = BASE64_STANDARD.encode(REPL17_SSEC_KEY);
let customer_key_md5 = sse_customer_key_md5_base64(REPL17_SSEC_KEY);
source_client
.put_object()
.bucket(source_bucket)
.key(key)
.body(ByteStream::from(body.clone()))
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
.await?;
// Wire replication (existing-object enabled) and drive a resync.
let target_arn = set_replication_target(&source_env, source_bucket, &target_env, target_bucket).await?;
put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
let (reset_arn, reset_id) = start_bucket_replication_reset(&source_env, source_bucket).await?;
assert_eq!(reset_arn, target_arn);
let terminal = wait_for_replication_reset_target(&source_env, source_bucket, &target_arn, |status| {
status.reset_id == reset_id && matches!(status.status.as_str(), "Completed" | "Failed")
})
.await?;
assert_eq!(terminal.status, "Completed", "SSE-C existing-object resync must complete");
assert!(terminal.replicated_count >= 1, "the existing SSE-C object must have been resynced");
// The replica is a REPLICA (status surfaces on HEAD) readable with the
// customer key.
let replica_head = target_client
.head_object()
.bucket(target_bucket)
.key(key)
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
.await?;
assert_eq!(
replica_head.replication_status().map(|status| status.as_str()),
Some("REPLICA"),
"the resynced copy must carry REPLICA status"
);
let replica = target_client
.get_object()
.bucket(target_bucket)
.key(key)
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
.await?;
assert_eq!(replica.sse_customer_algorithm(), Some("AES256"));
assert_eq!(replica.body.collect().await?.into_bytes().as_ref(), body.as_slice());
// No plaintext leak: the replica stays unreadable without the key.
assert!(
target_client
.get_object()
.bucket(target_bucket)
.key(key)
.send()
.await
.is_err(),
"SSE-C replica must not be readable without the customer key"
);
Ok(())
}
/// backlog#1147 repl-17 / backlog#1783: SSE-S3 objects replicate by decrypting
/// at the source and re-encrypting on the target with the target's own KMS.
/// The property backlog#1291 pinned — never a silent plaintext replica — still
@@ -8417,3 +8824,304 @@ async fn test_scanner_never_compensates_when_existing_object_replication_disable
Ok(())
}
/// Shared setup for the P1-5 read-proxy scenarios (backlog#1675): a RustFS
/// source with an enabled replication rule pointing at the fake target, and
/// an object seeded DIRECTLY on the target — it exists remotely but not
/// locally, exactly the active-active replication-lag window the read proxy
/// serves.
async fn start_read_proxy_lab(
source_bucket: &str,
target_bucket: &str,
) -> Result<(FakeS3Target, RustFSTestEnvironment, Client, Client), Box<dyn Error + Send + Sync>> {
let target = FakeS3Target::start().await?;
target.create_bucket(target_bucket);
target.assign_own_version_ids(true);
let mut source_env = RustFSTestEnvironment::new().await?;
let mut process_env = replication_fast_env();
process_env.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
process_env.extend_from_slice(&[
("NO_PROXY", "127.0.0.1,localhost"),
("HTTP_PROXY", ""),
("HTTPS_PROXY", ""),
("RUST_LOG", "error"),
]);
source_env.start_rustfs_server_with_env(vec![], &process_env).await?;
let source_client = source_env.create_s3_client();
source_client.create_bucket().bucket(source_bucket).send().await?;
enable_bucket_versioning(&source_env, source_bucket).await?;
let target_arn = set_replication_target_with_options(
&source_env,
source_bucket,
ReplicationTargetOptions {
endpoint: &target.address(),
access_key: FAKE_ACCESS_KEY,
secret_key: FAKE_SECRET_KEY,
target_bucket,
secure: false,
skip_tls_verify: false,
ca_cert_pem: None,
},
)
.await?;
put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
let target_client = Client::from_conf(crate::common::build_test_s3_config(
target.endpoint(),
FAKE_ACCESS_KEY,
FAKE_SECRET_KEY,
None,
"read-proxy-e2e",
));
Ok((target, source_env, source_client, target_client))
}
/// P1-5 (backlog#1675): during the active-active replication lag window a
/// GET/HEAD for an object the local site does not have yet is proxied to the
/// replication target. Pins the wire contract: the anti-loop
/// `source-proxy-request` marker is sent, the replication worker's
/// `source-replication-check` SSE-C exemption is NEVER sent, client SSE-C
/// headers are forwarded verbatim, and an inbound request that was itself
/// proxied is answered locally (404) without touching the target.
#[tokio::test]
#[serial]
async fn test_get_and_head_proxy_unreplicated_object_to_replication_target() -> TestResult {
init_logging();
let source_bucket = "proxy-read-src";
let target_bucket = "proxy-read-dst";
let (target, source_env, source_client, target_client) = start_read_proxy_lab(source_bucket, target_bucket).await?;
let payload = b"proxy payload".to_vec();
target_client
.put_object()
.bucket(target_bucket)
.key("proxy-only")
.body(ByteStream::from(payload.clone()))
.send()
.await?;
target.take_requests();
// a. GET of the locally-missing object is served through the proxy.
let got = source_client
.get_object()
.bucket(source_bucket)
.key("proxy-only")
.send()
.await
.map_err(|err| format!("proxied GET failed: {}", err.into_service_error()))?;
assert_eq!(got.content_length, Some(payload.len() as i64));
let body = got.body.collect().await?.into_bytes();
assert_eq!(body.as_ref(), payload.as_slice(), "proxied GET must stream the target's body");
let get_record = target
.requests()
.into_iter()
.find(|record| record.operation == FakeTargetOperation::GetObject && record.key.as_deref() == Some("proxy-only"))
.ok_or("fake target never received the proxied GET")?;
assert_eq!(
get_record.proxy_headers.source_proxy_request.as_deref(),
Some("true"),
"proxied GET must carry the anti-loop source-proxy-request marker"
);
assert!(
get_record.proxy_headers.replication_check.is_none(),
"proxied GET must never carry the replication worker's source-replication-check exemption"
);
assert!(
get_record.proxy_headers.ssec_algorithm.is_none() && !get_record.proxy_headers.ssec_key_present,
"no client SSE-C headers were sent, so none may be forwarded"
);
// a2. Client SSE-C headers travel verbatim to the target (the target owns
// the real SSE-C decryption; the plaintext fake simply ignores them).
target.take_requests();
let ssec_key = "01234567890123456789012345678901";
let ssec_key_b64 = BASE64_STANDARD.encode(ssec_key);
let ssec_key_md5 = sse_customer_key_md5_base64(ssec_key);
let _ = source_client
.get_object()
.bucket(source_bucket)
.key("proxy-only")
.sse_customer_algorithm("AES256")
.sse_customer_key(&ssec_key_b64)
.sse_customer_key_md5(&ssec_key_md5)
.send()
.await
.map_err(|err| format!("proxied SSE-C GET failed: {}", err.into_service_error()))?;
let ssec_record = target
.requests()
.into_iter()
.find(|record| record.operation == FakeTargetOperation::GetObject && record.key.as_deref() == Some("proxy-only"))
.ok_or("fake target never received the proxied SSE-C GET")?;
assert_eq!(ssec_record.proxy_headers.ssec_algorithm.as_deref(), Some("AES256"));
assert!(ssec_record.proxy_headers.ssec_key_present, "SSE-C key header must be forwarded verbatim");
assert_eq!(ssec_record.proxy_headers.ssec_key_md5.as_deref(), Some(ssec_key_md5.as_str()));
assert!(ssec_record.proxy_headers.replication_check.is_none());
// b. HEAD of the locally-missing object is served through the proxy.
target.take_requests();
let head = source_client
.head_object()
.bucket(source_bucket)
.key("proxy-only")
.send()
.await
.map_err(|err| format!("proxied HEAD failed: {}", err.into_service_error()))?;
assert_eq!(head.content_length, Some(payload.len() as i64));
let head_record = target
.requests()
.into_iter()
.find(|record| record.operation == FakeTargetOperation::HeadObject && record.key.as_deref() == Some("proxy-only"))
.ok_or("fake target never received the proxied HEAD")?;
assert_eq!(head_record.proxy_headers.source_proxy_request.as_deref(), Some("true"));
assert!(head_record.proxy_headers.replication_check.is_none());
// c. Anti-loop: an inbound request that already carries the proxy marker
// is answered locally with 404 and never forwarded to the target.
target.take_requests();
let err = source_client
.get_object()
.bucket(source_bucket)
.key("proxy-only")
.customize()
.mutate_request(|req| {
req.headers_mut().insert("x-minio-source-proxy-request", "true");
})
.send()
.await
.expect_err("anti-loop GET must fail locally instead of proxying");
let service_err = err.into_service_error();
assert!(service_err.is_no_such_key(), "anti-loop GET must 404, got: {service_err}");
assert!(
!target
.requests()
.iter()
.any(|record| record.operation == FakeTargetOperation::GetObject),
"anti-loop GET must not reach the replication target; journal: {:?}",
target.requests()
);
// c2. MinIO ProxyHeaderSet parity: the header's mere PRESENCE disables
// proxying — "false" is exactly what a peer's replication worker sends on
// its convergence HEADs, and proxying that miss back would fake
// convergence.
target.take_requests();
let err = source_client
.get_object()
.bucket(source_bucket)
.key("proxy-only")
.customize()
.mutate_request(|req| {
req.headers_mut().insert("x-minio-source-proxy-request", "false");
})
.send()
.await
.expect_err("proxy-header-set GET must fail locally instead of proxying");
let service_err = err.into_service_error();
assert!(service_err.is_no_such_key(), "proxy-header-set GET must 404, got: {service_err}");
assert!(
!target
.requests()
.iter()
.any(|record| record.operation == FakeTargetOperation::GetObject),
"proxy-header-set GET must not reach the replication target; journal: {:?}",
target.requests()
);
// d. The replication worker's own convergence HEAD against the target
// must carry `source-proxy-request: false` (never proxied back) and the
// replication-check exemption. Trigger real replication and inspect the
// fake journal.
target.take_requests();
source_client
.put_object()
.bucket(source_bucket)
.key("worker-replicated")
.body(ByteStream::from_static(b"worker payload"))
.send()
.await?;
wait_for_target_request_version_id(&target, FakeTargetOperation::PutObject, "worker-replicated").await?;
let worker_head = target
.requests()
.into_iter()
.find(|record| record.operation == FakeTargetOperation::HeadObject && record.key.as_deref() == Some("worker-replicated"))
.ok_or_else(|| format!("replication worker never HEAD-ed the target; journal: {:?}", target.requests()))?;
assert_eq!(
worker_head.proxy_headers.source_proxy_request.as_deref(),
Some("false"),
"worker convergence HEAD must send source-proxy-request: false so the target answers locally"
);
assert_eq!(
worker_head.proxy_headers.replication_check.as_deref(),
Some("true"),
"worker convergence HEAD keeps the replication-check exemption"
);
drop(source_env);
target.shutdown().await;
Ok(())
}
/// P1-5 (backlog#1675): GetObjectTagging for an object missing locally is
/// proxied to the replication target with the anti-loop marker, mirroring
/// MinIO `proxyGetTaggingToRepTarget`.
#[tokio::test]
#[serial]
async fn test_get_object_tagging_proxies_unreplicated_object_to_replication_target() -> TestResult {
init_logging();
let source_bucket = "proxy-tag-src";
let target_bucket = "proxy-tag-dst";
let (target, source_env, source_client, target_client) = start_read_proxy_lab(source_bucket, target_bucket).await?;
target_client
.put_object()
.bucket(target_bucket)
.key("proxy-tagged")
.body(ByteStream::from_static(b"tagged payload"))
.send()
.await?;
target_client
.put_object_tagging()
.bucket(target_bucket)
.key("proxy-tagged")
.tagging(
aws_sdk_s3::types::Tagging::builder()
.tag_set(aws_sdk_s3::types::Tag::builder().key("team").value("storage").build()?)
.build()?,
)
.send()
.await?;
target.take_requests();
let tags = source_client
.get_object_tagging()
.bucket(source_bucket)
.key("proxy-tagged")
.send()
.await
.map_err(|err| format!("proxied GetObjectTagging failed: {}", err.into_service_error()))?;
assert_eq!(tags.tag_set.len(), 1, "proxied tagging read must return the target's tags");
assert_eq!(tags.tag_set[0].key.as_str(), "team");
assert_eq!(tags.tag_set[0].value.as_str(), "storage");
let record = target
.requests()
.into_iter()
.find(|record| record.operation == FakeTargetOperation::GetObjectTagging && record.key.as_deref() == Some("proxy-tagged"))
.ok_or("fake target never received the proxied GetObjectTagging")?;
assert_eq!(
record.proxy_headers.source_proxy_request.as_deref(),
Some("true"),
"proxied tagging read must carry the anti-loop marker"
);
assert!(record.proxy_headers.replication_check.is_none());
drop(source_env);
target.shutdown().await;
Ok(())
}