mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-20 11:32:19 +00:00
feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets (#6172)
* feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets
Implements the MinIO active-active read-proxy protocol (P1-5 of the
replication compatibility review): when a GET/HEAD/GetObjectTagging/
PutObjectTagging/DeleteObjectTagging request fails locally with
not-found and the bucket has replication targets, the request is proxied
to the targets in rule order, mirroring bucket-replication.go
proxyGetToReplicationTarget/proxyHeadToRepTarget/proxyTaggingToRepTarget.
Protocol surface:
- Anti-loop: inbound {x-rustfs-,x-minio-}source-proxy-request is parsed
into ObjectOptions (proxy_request + proxy_header_set, matching MinIO
ProxyRequest/ProxyHeaderSet); a request carrying the marker with ANY
value is never re-proxied. Outbound client proxy calls send the marker
as "true"; replication worker convergence HEADs send it as "false" so
a peer's proxy layer cannot answer a convergence check by proxying
back to the source (which would fake Completed without a PUT).
- Target selection: new replication_proxy.rs get_proxy_targets — empty
when the marker is set, versioning is suspended, or no replication
config; otherwise filter_target_arns -> TargetClient lookup, skipping
targets with proxying disabled.
- TargetClient gains head_object_for_proxy/get_object (streaming) and
the three tagging calls. Proxy calls never send the replication-check
SSE-C exemption header; customer SSE-C keys are forwarded verbatim so
the target performs real decryption. Conditional (If-*) headers are
not forwarded (MinIO parity); Range and part_number are, with
parts_count/tag_count/storage_class/expiration passed through.
- Metrics: proxy counters now count only real client proxy traffic,
MinIO-aligned (one total per proxied request, one failed when no
target served it). The previous misattributed counters — replication
worker HEAD/PUT (#2672) and local tagging operations (#2682) — are
removed; ReplProxyMetric now maps the tagging counters instead of
dropping them.
e2e (fake_s3_target extended with tagging + header journaling): proxied
GET body + outbound header contract (marker present, no
replication-check, SSE-C passthrough), HEAD, anti-loop 404 with zero
outbound requests, GetObjectTagging, and metric mapping unit tests.
Rolling note: proxying only activates for buckets with replication
targets; requests carrying the marker keep pre-upgrade behavior.
Refs rustfs/backlog#1675 (P1-5)
* fix(replication): fail SSE-C passthrough closed on targets that drop transport headers (#6178)
SSE-C ciphertext passthrough replicates via X-Rustfs-Replication-* transport
headers. A MinIO/generic-S3 target silently discards them, storing bare
ciphertext with no decryption material — yet the PUT succeeded, so the object
reported COMPLETED with a silently unreadable replica (backlog#1675 N2).
Fail-closed design:
- SsecPassthroughCapability {Unknown, Supported, Unsupported} cached in
BucketTargetSys per target ARN with a recording timestamp. Entries reset
whenever the target is rebuilt, edited, or removed (arn_remotes_map
lifecycle) and expire after SSEC_PASSTHROUGH_CAPABILITY_TTL (10 minutes):
an expired verdict in either direction is re-earned through the audit, so
an Unsupported target recovers automatically after an upgrade (at most one
wasted PUT+HEAD audit per bad target per TTL window) and a Supported
verdict cannot outlive a backend swapped behind the same endpoint.
- Replication worker (replicate_object and replicate_all): fresh Unsupported
targets never receive the PUT — the attempt fails immediately into the
normal MRF retry channel with a "run ?replication-check to re-probe" hint.
Unknown or expired verdicts are audited: after the PUT the worker HEADs
the replica back through the replication-check channel (source version id
mapped through resolve_read_api_version_id, so null-version objects audit
correctly) and requires SSE-C evidence (the echoed customer-algorithm
header); missing evidence records Unsupported and fails the attempt.
Convergence HEADs are audited the same way, so a broken ciphertext replica
from an earlier attempt can never launder itself into COMPLETED via an
ETag match. The gate/evidence policy is pure (replication_target_boundary,
staleness folded in as an input) for the M2 worker migration.
- replication-check grows an SsecPassthrough probe phase: a probe PUT
carrying the live transport-header shape, HEAD-back for evidence, and a
machine-readable Code BucketRemoteSsecPassthroughUnsupported on failure.
The probe verdict is synced into the runtime capability cache. Unlike
VersionFidelity, a failed SsecPassthrough phase does NOT fail the target
overall — it is a capability limit, not a broken replication contract,
and a plaintext-only deployment against such a target must not turn red.
- fake_s3_target: default mode now models a RustFS target (stores the
transport headers, echoes SSE-C evidence); the new
drop_unlisted_replication_headers mode models MinIO. The journal records
whether a request carried transport headers.
Receiver-echo verification: the replication-check HEAD exemption only skips
SSE-C key validation; the response has always built sse-customer-algorithm
from stored metadata (rustfs/src/app/object_usecase.rs), so no receiver
change was needed — pinned end to end by the replication-check e2e against
a real RustFS target.
Rolling-upgrade constraint: RustFS targets older than the replication-check
HEAD exemption (#5898) answer the audit HEAD without SSE-C evidence (or fail
it outright), so SSE-C replication to such targets reports FAILED. This is
deliberate — FAILED-and-retryable beats a silently undecryptable replica —
and self-heals: once the target is upgraded, the next TTL expiry (or a
manual ?replication-check re-probe) re-audits and records Supported.
Plaintext and managed-SSE replication are unaffected. The capability cache
is per-node; each node audits independently.
Known limitations:
- The audit judges evidence from the echoed customer-algorithm header only.
A hypothetical target that preserves that one header while dropping other
transport headers (partial-drop) would pass the audit; no known target
behaves this way — observed targets drop the whole unknown-header family.
- A mixed-version target cluster can flap the verdict between audits routed
to different target nodes until the rollout completes; the TTL bounds how
long each stale verdict persists.
New e2e (backlog#1675 C1 + N2, red-first): fail-closed against a
header-dropping fake (FAILED + no second PUT via the capability cache,
journal-asserted; red run showed the old COMPLETED), replication-check
reports the SsecPassthrough phase Code while the target stays OK overall,
SSE-C heal convergence after a real target outage, and SSE-C
existing-object resync landing a REPLICA readable with the customer key.
TTL expiry in both directions is pinned at the cache and gate seams.
This commit is contained in:
@@ -2610,17 +2610,20 @@ async fn test_replication_check_succeeds_with_remote_target() -> Result<(), Box<
|
||||
|
||||
assert_eq!(response.status(), StatusCode::OK);
|
||||
let payload: serde_json::Value = response.json().await?;
|
||||
assert_eq!(payload["Status"], "OK");
|
||||
assert_eq!(payload["Status"], "OK", "{payload}");
|
||||
assert_eq!(payload["ActiveMutation"], true);
|
||||
assert_eq!(payload["Targets"].as_array().map(Vec::len), Some(1));
|
||||
assert_eq!(payload["Targets"][0]["Status"], "OK");
|
||||
assert_eq!(payload["Targets"][0]["Phases"]["Put"]["Status"], "OK");
|
||||
assert_eq!(payload["Targets"][0]["Status"], "OK", "{payload}");
|
||||
assert_eq!(payload["Targets"][0]["Phases"]["Put"]["Status"], "OK", "{payload}");
|
||||
// A RustFS target adopts the source version id, so the P1-19
|
||||
// version-identity probe passes.
|
||||
assert_eq!(payload["Targets"][0]["Phases"]["VersionFidelity"]["Status"], "OK");
|
||||
assert_eq!(payload["Targets"][0]["Phases"]["DeleteMarker"]["Status"], "OK");
|
||||
assert_eq!(payload["Targets"][0]["Phases"]["VersionDelete"]["Status"], "OK");
|
||||
assert_eq!(payload["Targets"][0]["Phases"]["Cleanup"]["Status"], "OK");
|
||||
assert_eq!(payload["Targets"][0]["Phases"]["VersionFidelity"]["Status"], "OK", "{payload}");
|
||||
// A RustFS target preserves the SSE-C passthrough transport headers and
|
||||
// echoes the customer algorithm on the replication-check HEAD (N2).
|
||||
assert_eq!(payload["Targets"][0]["Phases"]["SsecPassthrough"]["Status"], "OK", "{payload}");
|
||||
assert_eq!(payload["Targets"][0]["Phases"]["DeleteMarker"]["Status"], "OK", "{payload}");
|
||||
assert_eq!(payload["Targets"][0]["Phases"]["VersionDelete"]["Status"], "OK", "{payload}");
|
||||
assert_eq!(payload["Targets"][0]["Phases"]["Cleanup"]["Status"], "OK", "{payload}");
|
||||
|
||||
let target_client = target_env.create_s3_client();
|
||||
let versions = target_client
|
||||
@@ -4649,6 +4652,410 @@ async fn test_bucket_replication_sse_c_multipart_passthrough() -> TestResult {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// N2 (backlog#1675 P1-22): SSE-C passthrough replication to a target that
|
||||
/// silently drops the `X-Rustfs-Replication-*` transport headers (MinIO-like
|
||||
/// behavior, modeled by the fake target's drop mode) used to report COMPLETED
|
||||
/// while the replica had irrecoverably lost its decryption material — the red
|
||||
/// light this test was born failing on. Fail-closed contract now under test:
|
||||
/// the first attempt PUTs, HEAD-backs the replica, finds no SSE-C evidence,
|
||||
/// records the target Unsupported and reports FAILED; a second SSE-C object
|
||||
/// fails without any PUT reaching the target (capability cache, proven from
|
||||
/// the target journal); plaintext objects still replicate COMPLETED.
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_ssec_replication_fails_closed_when_target_drops_passthrough_headers() -> TestResult {
|
||||
init_logging();
|
||||
|
||||
let target = FakeS3Target::start().await?;
|
||||
let target_bucket = "ssec-drop-dst";
|
||||
target.create_bucket(target_bucket);
|
||||
target.drop_unlisted_replication_headers(true);
|
||||
|
||||
let mut source_env = RustFSTestEnvironment::new().await?;
|
||||
let mut env_vars = replication_fast_env();
|
||||
env_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
|
||||
env_vars.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
|
||||
source_env.start_rustfs_server_with_env(vec![], &env_vars).await?;
|
||||
|
||||
let source_bucket = "ssec-drop-src";
|
||||
let source_client = source_env.create_s3_client();
|
||||
source_client.create_bucket().bucket(source_bucket).send().await?;
|
||||
enable_bucket_versioning(&source_env, source_bucket).await?;
|
||||
let target_arn = set_replication_target_with_options(
|
||||
&source_env,
|
||||
source_bucket,
|
||||
ReplicationTargetOptions {
|
||||
endpoint: &target.address(),
|
||||
access_key: FAKE_ACCESS_KEY,
|
||||
secret_key: FAKE_SECRET_KEY,
|
||||
target_bucket,
|
||||
secure: false,
|
||||
skip_tls_verify: false,
|
||||
ca_cert_pem: None,
|
||||
},
|
||||
)
|
||||
.await?;
|
||||
put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
|
||||
|
||||
let customer_key = BASE64_STANDARD.encode(REPL17_SSEC_KEY);
|
||||
let customer_key_md5 = sse_customer_key_md5_base64(REPL17_SSEC_KEY);
|
||||
let put_ssec = |key: &'static str| {
|
||||
source_client
|
||||
.put_object()
|
||||
.bucket(source_bucket)
|
||||
.key(key)
|
||||
.body(ByteStream::from_static(b"ssec fail-closed payload"))
|
||||
.sse_customer_algorithm("AES256")
|
||||
.sse_customer_key(&customer_key)
|
||||
.sse_customer_key_md5(&customer_key_md5)
|
||||
.send()
|
||||
};
|
||||
|
||||
// First SSE-C object: the audit must catch the dropped material.
|
||||
put_ssec("ssec-first.txt").await?;
|
||||
wait_for_source_replication_status(&source_client, source_bucket, "ssec-first.txt", "FAILED", true).await?;
|
||||
|
||||
let requests = target.take_requests();
|
||||
let first_put = requests
|
||||
.iter()
|
||||
.find(|record| record.operation == FakeTargetOperation::PutObject && record.key.as_deref() == Some("ssec-first.txt"))
|
||||
.ok_or("the first SSE-C object must have been PUT (capability was Unknown)")?;
|
||||
assert!(
|
||||
first_put.proxy_headers.ssec_transport_present,
|
||||
"the replication PUT must have shipped the SSE-C transport headers the target then dropped"
|
||||
);
|
||||
assert!(
|
||||
requests.iter().any(|record| {
|
||||
record.operation == FakeTargetOperation::HeadObject
|
||||
&& record.key.as_deref() == Some("ssec-first.txt")
|
||||
&& record.sequence > first_put.sequence
|
||||
&& record.proxy_headers.replication_check.as_deref() == Some("true")
|
||||
}),
|
||||
"the post-PUT HEAD-back audit must have run through the replication-check channel; journal: {requests:?}"
|
||||
);
|
||||
|
||||
// Second SSE-C object: the cached Unsupported verdict fails it closed
|
||||
// before any PUT — including MRF retries of the first object.
|
||||
put_ssec("ssec-second.txt").await?;
|
||||
wait_for_source_replication_status(&source_client, source_bucket, "ssec-second.txt", "FAILED", true).await?;
|
||||
assert!(
|
||||
!target.requests().iter().any(|record| {
|
||||
record.operation == FakeTargetOperation::PutObject
|
||||
&& record.key.as_deref() != Some("plain-control.txt")
|
||||
&& record.proxy_headers.ssec_transport_present
|
||||
}),
|
||||
"no further SSE-C ciphertext may reach a target recorded Unsupported; journal: {:?}",
|
||||
target.requests()
|
||||
);
|
||||
|
||||
// The gate is scoped to SSE-C: plaintext replication keeps working.
|
||||
source_client
|
||||
.put_object()
|
||||
.bucket(source_bucket)
|
||||
.key("plain-control.txt")
|
||||
.body(ByteStream::from_static(b"plaintext control payload"))
|
||||
.send()
|
||||
.await?;
|
||||
wait_for_source_replication_status(&source_client, source_bucket, "plain-control.txt", "COMPLETED", false).await?;
|
||||
assert!(target.has_object(target_bucket, "plain-control.txt"));
|
||||
|
||||
target.shutdown().await;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// N2 (backlog#1675 P1-22): the admin replication-check must expose the same
|
||||
/// verdict operators would otherwise only learn from failing SSE-C objects —
|
||||
/// an SsecPassthrough probe phase that fails with the machine-readable
|
||||
/// `BucketRemoteSsecPassthroughUnsupported` code against a header-dropping
|
||||
/// target, with no probe residue left behind. The target's overall status
|
||||
/// stays OK: unlike version-identity drift, dropped passthrough headers are
|
||||
/// a capability limit, and a plaintext-only deployment against a MinIO-like
|
||||
/// target must not turn red.
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_replication_check_flags_ssec_passthrough_dropping_target() -> TestResult {
|
||||
init_logging();
|
||||
|
||||
let target = FakeS3Target::start().await?;
|
||||
let target_bucket = "ssec-check-dst";
|
||||
target.create_bucket(target_bucket);
|
||||
target.drop_unlisted_replication_headers(true);
|
||||
|
||||
let mut source_env = RustFSTestEnvironment::new().await?;
|
||||
let mut env_vars = replication_fast_env();
|
||||
env_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
|
||||
env_vars.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
|
||||
source_env.start_rustfs_server_with_env(vec![], &env_vars).await?;
|
||||
|
||||
let source_bucket = "ssec-check-src";
|
||||
let source_client = source_env.create_s3_client();
|
||||
source_client.create_bucket().bucket(source_bucket).send().await?;
|
||||
enable_bucket_versioning(&source_env, source_bucket).await?;
|
||||
let target_arn = set_replication_target_with_options(
|
||||
&source_env,
|
||||
source_bucket,
|
||||
ReplicationTargetOptions {
|
||||
endpoint: &target.address(),
|
||||
access_key: FAKE_ACCESS_KEY,
|
||||
secret_key: FAKE_SECRET_KEY,
|
||||
target_bucket,
|
||||
secure: false,
|
||||
skip_tls_verify: false,
|
||||
ca_cert_pem: None,
|
||||
},
|
||||
)
|
||||
.await?;
|
||||
put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
|
||||
|
||||
let response = run_replication_check(&source_env, source_bucket).await?;
|
||||
assert_eq!(response.status(), StatusCode::OK);
|
||||
let payload: serde_json::Value = response.json().await?;
|
||||
|
||||
assert_eq!(
|
||||
payload["Status"], "OK",
|
||||
"a capability-only SSE-C failure must not fail the check overall: {payload}"
|
||||
);
|
||||
let target_report = &payload["Targets"][0];
|
||||
assert_eq!(target_report["Status"], "OK", "{payload}");
|
||||
let ssec = &target_report["Phases"]["SsecPassthrough"];
|
||||
assert_eq!(ssec["Status"], "FAILED", "SsecPassthrough phase must fail: {payload}");
|
||||
assert_eq!(
|
||||
ssec["Code"], "BucketRemoteSsecPassthroughUnsupported",
|
||||
"the failure must carry the machine-readable code: {payload}"
|
||||
);
|
||||
// Basic replication of plaintext objects works on this target: every other
|
||||
// phase passes, so the code is the discriminator operators branch on.
|
||||
assert_eq!(target_report["Phases"]["Put"]["Status"], "OK", "{payload}");
|
||||
assert_eq!(target_report["Phases"]["VersionFidelity"]["Status"], "OK", "{payload}");
|
||||
assert_eq!(target_report["Phases"]["DeleteMarker"]["Status"], "OK", "{payload}");
|
||||
assert_eq!(target_report["Phases"]["VersionDelete"]["Status"], "OK", "{payload}");
|
||||
assert_eq!(target_report["Phases"]["Cleanup"]["Status"], "OK", "{payload}");
|
||||
|
||||
// The SSE-C probe PUT must have shipped the real transport header names —
|
||||
// a mangled or missing header set would fail the phase for the wrong
|
||||
// reason and mask a working target.
|
||||
let requests = target.requests();
|
||||
assert!(
|
||||
requests
|
||||
.iter()
|
||||
.any(|record| record.operation == FakeTargetOperation::PutObject && record.proxy_headers.ssec_transport_present),
|
||||
"the SSE-C probe PUT must carry the X-Rustfs-Replication-* transport headers; journal: {requests:?}"
|
||||
);
|
||||
|
||||
// No probe residue, including the SSE-C probe version.
|
||||
let probe_put = requests
|
||||
.into_iter()
|
||||
.find(|record| record.operation == FakeTargetOperation::PutObject)
|
||||
.ok_or("the probe PUT never reached the fake target")?;
|
||||
let probe_key = probe_put.key.ok_or("probe PUT journal record has no key")?;
|
||||
assert!(
|
||||
target.stored_versions(target_bucket, &probe_key).is_empty(),
|
||||
"all probe versions must be cleaned up"
|
||||
);
|
||||
|
||||
target.shutdown().await;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// C1 (backlog#1675 P1-22): heal-path convergence for SSE-C. An SSE-C object
|
||||
/// whose live replication failed during a target outage must converge through
|
||||
/// the scanner/heal compensation once the target returns — passing the N2
|
||||
/// HEAD-back audit against the recovered RustFS target — and the replica must
|
||||
/// be readable with the customer key.
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_bucket_replication_sse_c_heals_after_target_outage() -> TestResult {
|
||||
init_logging();
|
||||
|
||||
let (source_env, mut target_env, source_bucket, target_bucket) =
|
||||
build_sse_replication_pair("ssec-heal", false, false).await?;
|
||||
let source_client = source_env.create_s3_client();
|
||||
let key = "ssec-heal-contract.txt";
|
||||
let body = b"repl-22 ssec heal payload".to_vec();
|
||||
let customer_key = BASE64_STANDARD.encode(REPL17_SSEC_KEY);
|
||||
let customer_key_md5 = sse_customer_key_md5_base64(REPL17_SSEC_KEY);
|
||||
|
||||
// Target outage: the SSE-C write cannot replicate.
|
||||
target_env.stop_server();
|
||||
|
||||
source_client
|
||||
.put_object()
|
||||
.bucket(&source_bucket)
|
||||
.key(key)
|
||||
.body(ByteStream::from(body.clone()))
|
||||
.sse_customer_algorithm("AES256")
|
||||
.sse_customer_key(&customer_key)
|
||||
.sse_customer_key_md5(&customer_key_md5)
|
||||
.send()
|
||||
.await?;
|
||||
|
||||
// The failure is observable on the source (SSE-C HEAD needs the key).
|
||||
let deadline = tokio::time::Instant::now() + Duration::from_secs(30);
|
||||
loop {
|
||||
let head = source_client
|
||||
.head_object()
|
||||
.bucket(&source_bucket)
|
||||
.key(key)
|
||||
.sse_customer_algorithm("AES256")
|
||||
.sse_customer_key(&customer_key)
|
||||
.sse_customer_key_md5(&customer_key_md5)
|
||||
.send()
|
||||
.await?;
|
||||
match head.replication_status().map(|status| status.as_str()) {
|
||||
Some("PENDING") | Some("FAILED") => break,
|
||||
other => {
|
||||
if tokio::time::Instant::now() >= deadline {
|
||||
return Err(format!("source SSE-C object never reported PENDING/FAILED; last status={other:?}").into());
|
||||
}
|
||||
sleep(Duration::from_millis(200)).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Recover the target in place; the source scanner re-drives the failure.
|
||||
target_env
|
||||
.restart_server_preserving_data(vec![], &[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")])
|
||||
.await?;
|
||||
|
||||
wait_for_source_replication_status(&source_client, &source_bucket, key, "COMPLETED", true).await?;
|
||||
|
||||
// The healed replica is a REPLICA (status surfaces on HEAD) readable with
|
||||
// the customer key.
|
||||
let target_client = target_env.create_s3_client();
|
||||
let replica_head = target_client
|
||||
.head_object()
|
||||
.bucket(&target_bucket)
|
||||
.key(key)
|
||||
.sse_customer_algorithm("AES256")
|
||||
.sse_customer_key(&customer_key)
|
||||
.sse_customer_key_md5(&customer_key_md5)
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(
|
||||
replica_head.replication_status().map(|status| status.as_str()),
|
||||
Some("REPLICA"),
|
||||
"the healed copy must carry REPLICA status"
|
||||
);
|
||||
let replica = target_client
|
||||
.get_object()
|
||||
.bucket(&target_bucket)
|
||||
.key(key)
|
||||
.sse_customer_algorithm("AES256")
|
||||
.sse_customer_key(&customer_key)
|
||||
.sse_customer_key_md5(&customer_key_md5)
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(replica.sse_customer_algorithm(), Some("AES256"));
|
||||
assert_eq!(replica.body.collect().await?.into_bytes().as_ref(), body.as_slice());
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// C1 (backlog#1675 P1-22): existing-object resync for SSE-C. An SSE-C object
|
||||
/// written BEFORE any replication config must reach the RustFS target through
|
||||
/// the existing-object resync (`replicate_all` transport, N2-audited), land as
|
||||
/// a REPLICA, and read back with the customer key.
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_bucket_replication_sse_c_existing_object_resync() -> TestResult {
|
||||
init_logging();
|
||||
|
||||
let mut source_env = RustFSTestEnvironment::new().await?;
|
||||
let mut source_process_env = replication_fast_env();
|
||||
source_process_env.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
|
||||
source_process_env.extend_from_slice(FAST_SCANNER_ENV);
|
||||
source_process_env.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
|
||||
source_env.start_rustfs_server_with_env(vec![], &source_process_env).await?;
|
||||
|
||||
let mut target_env = RustFSTestEnvironment::new().await?;
|
||||
target_env
|
||||
.start_rustfs_server_without_cleanup_with_env(&[
|
||||
("NO_PROXY", "127.0.0.1,localhost"),
|
||||
("HTTP_PROXY", ""),
|
||||
("HTTPS_PROXY", ""),
|
||||
])
|
||||
.await?;
|
||||
|
||||
let source_bucket = "ssec-existing-src";
|
||||
let target_bucket = "ssec-existing-dst";
|
||||
let source_client = source_env.create_s3_client();
|
||||
let target_client = target_env.create_s3_client();
|
||||
source_client.create_bucket().bucket(source_bucket).send().await?;
|
||||
target_client.create_bucket().bucket(target_bucket).send().await?;
|
||||
enable_bucket_versioning(&source_env, source_bucket).await?;
|
||||
enable_bucket_versioning(&target_env, target_bucket).await?;
|
||||
|
||||
// The SSE-C object exists before any replication wiring.
|
||||
let key = "ssec-existing-contract.txt";
|
||||
let body = b"repl-22 ssec existing-object payload".to_vec();
|
||||
let customer_key = BASE64_STANDARD.encode(REPL17_SSEC_KEY);
|
||||
let customer_key_md5 = sse_customer_key_md5_base64(REPL17_SSEC_KEY);
|
||||
source_client
|
||||
.put_object()
|
||||
.bucket(source_bucket)
|
||||
.key(key)
|
||||
.body(ByteStream::from(body.clone()))
|
||||
.sse_customer_algorithm("AES256")
|
||||
.sse_customer_key(&customer_key)
|
||||
.sse_customer_key_md5(&customer_key_md5)
|
||||
.send()
|
||||
.await?;
|
||||
|
||||
// Wire replication (existing-object enabled) and drive a resync.
|
||||
let target_arn = set_replication_target(&source_env, source_bucket, &target_env, target_bucket).await?;
|
||||
put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
|
||||
let (reset_arn, reset_id) = start_bucket_replication_reset(&source_env, source_bucket).await?;
|
||||
assert_eq!(reset_arn, target_arn);
|
||||
let terminal = wait_for_replication_reset_target(&source_env, source_bucket, &target_arn, |status| {
|
||||
status.reset_id == reset_id && matches!(status.status.as_str(), "Completed" | "Failed")
|
||||
})
|
||||
.await?;
|
||||
assert_eq!(terminal.status, "Completed", "SSE-C existing-object resync must complete");
|
||||
assert!(terminal.replicated_count >= 1, "the existing SSE-C object must have been resynced");
|
||||
|
||||
// The replica is a REPLICA (status surfaces on HEAD) readable with the
|
||||
// customer key.
|
||||
let replica_head = target_client
|
||||
.head_object()
|
||||
.bucket(target_bucket)
|
||||
.key(key)
|
||||
.sse_customer_algorithm("AES256")
|
||||
.sse_customer_key(&customer_key)
|
||||
.sse_customer_key_md5(&customer_key_md5)
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(
|
||||
replica_head.replication_status().map(|status| status.as_str()),
|
||||
Some("REPLICA"),
|
||||
"the resynced copy must carry REPLICA status"
|
||||
);
|
||||
let replica = target_client
|
||||
.get_object()
|
||||
.bucket(target_bucket)
|
||||
.key(key)
|
||||
.sse_customer_algorithm("AES256")
|
||||
.sse_customer_key(&customer_key)
|
||||
.sse_customer_key_md5(&customer_key_md5)
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(replica.sse_customer_algorithm(), Some("AES256"));
|
||||
assert_eq!(replica.body.collect().await?.into_bytes().as_ref(), body.as_slice());
|
||||
|
||||
// No plaintext leak: the replica stays unreadable without the key.
|
||||
assert!(
|
||||
target_client
|
||||
.get_object()
|
||||
.bucket(target_bucket)
|
||||
.key(key)
|
||||
.send()
|
||||
.await
|
||||
.is_err(),
|
||||
"SSE-C replica must not be readable without the customer key"
|
||||
);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// backlog#1147 repl-17 / backlog#1783: SSE-S3 objects replicate by decrypting
|
||||
/// at the source and re-encrypting on the target with the target's own KMS.
|
||||
/// The property backlog#1291 pinned — never a silent plaintext replica — still
|
||||
@@ -8417,3 +8824,304 @@ async fn test_scanner_never_compensates_when_existing_object_replication_disable
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Shared setup for the P1-5 read-proxy scenarios (backlog#1675): a RustFS
|
||||
/// source with an enabled replication rule pointing at the fake target, and
|
||||
/// an object seeded DIRECTLY on the target — it exists remotely but not
|
||||
/// locally, exactly the active-active replication-lag window the read proxy
|
||||
/// serves.
|
||||
async fn start_read_proxy_lab(
|
||||
source_bucket: &str,
|
||||
target_bucket: &str,
|
||||
) -> Result<(FakeS3Target, RustFSTestEnvironment, Client, Client), Box<dyn Error + Send + Sync>> {
|
||||
let target = FakeS3Target::start().await?;
|
||||
target.create_bucket(target_bucket);
|
||||
target.assign_own_version_ids(true);
|
||||
|
||||
let mut source_env = RustFSTestEnvironment::new().await?;
|
||||
let mut process_env = replication_fast_env();
|
||||
process_env.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
|
||||
process_env.extend_from_slice(&[
|
||||
("NO_PROXY", "127.0.0.1,localhost"),
|
||||
("HTTP_PROXY", ""),
|
||||
("HTTPS_PROXY", ""),
|
||||
("RUST_LOG", "error"),
|
||||
]);
|
||||
source_env.start_rustfs_server_with_env(vec![], &process_env).await?;
|
||||
|
||||
let source_client = source_env.create_s3_client();
|
||||
source_client.create_bucket().bucket(source_bucket).send().await?;
|
||||
enable_bucket_versioning(&source_env, source_bucket).await?;
|
||||
let target_arn = set_replication_target_with_options(
|
||||
&source_env,
|
||||
source_bucket,
|
||||
ReplicationTargetOptions {
|
||||
endpoint: &target.address(),
|
||||
access_key: FAKE_ACCESS_KEY,
|
||||
secret_key: FAKE_SECRET_KEY,
|
||||
target_bucket,
|
||||
secure: false,
|
||||
skip_tls_verify: false,
|
||||
ca_cert_pem: None,
|
||||
},
|
||||
)
|
||||
.await?;
|
||||
put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
|
||||
|
||||
let target_client = Client::from_conf(crate::common::build_test_s3_config(
|
||||
target.endpoint(),
|
||||
FAKE_ACCESS_KEY,
|
||||
FAKE_SECRET_KEY,
|
||||
None,
|
||||
"read-proxy-e2e",
|
||||
));
|
||||
|
||||
Ok((target, source_env, source_client, target_client))
|
||||
}
|
||||
|
||||
/// P1-5 (backlog#1675): during the active-active replication lag window a
|
||||
/// GET/HEAD for an object the local site does not have yet is proxied to the
|
||||
/// replication target. Pins the wire contract: the anti-loop
|
||||
/// `source-proxy-request` marker is sent, the replication worker's
|
||||
/// `source-replication-check` SSE-C exemption is NEVER sent, client SSE-C
|
||||
/// headers are forwarded verbatim, and an inbound request that was itself
|
||||
/// proxied is answered locally (404) without touching the target.
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_get_and_head_proxy_unreplicated_object_to_replication_target() -> TestResult {
|
||||
init_logging();
|
||||
|
||||
let source_bucket = "proxy-read-src";
|
||||
let target_bucket = "proxy-read-dst";
|
||||
let (target, source_env, source_client, target_client) = start_read_proxy_lab(source_bucket, target_bucket).await?;
|
||||
|
||||
let payload = b"proxy payload".to_vec();
|
||||
target_client
|
||||
.put_object()
|
||||
.bucket(target_bucket)
|
||||
.key("proxy-only")
|
||||
.body(ByteStream::from(payload.clone()))
|
||||
.send()
|
||||
.await?;
|
||||
target.take_requests();
|
||||
|
||||
// a. GET of the locally-missing object is served through the proxy.
|
||||
let got = source_client
|
||||
.get_object()
|
||||
.bucket(source_bucket)
|
||||
.key("proxy-only")
|
||||
.send()
|
||||
.await
|
||||
.map_err(|err| format!("proxied GET failed: {}", err.into_service_error()))?;
|
||||
assert_eq!(got.content_length, Some(payload.len() as i64));
|
||||
let body = got.body.collect().await?.into_bytes();
|
||||
assert_eq!(body.as_ref(), payload.as_slice(), "proxied GET must stream the target's body");
|
||||
|
||||
let get_record = target
|
||||
.requests()
|
||||
.into_iter()
|
||||
.find(|record| record.operation == FakeTargetOperation::GetObject && record.key.as_deref() == Some("proxy-only"))
|
||||
.ok_or("fake target never received the proxied GET")?;
|
||||
assert_eq!(
|
||||
get_record.proxy_headers.source_proxy_request.as_deref(),
|
||||
Some("true"),
|
||||
"proxied GET must carry the anti-loop source-proxy-request marker"
|
||||
);
|
||||
assert!(
|
||||
get_record.proxy_headers.replication_check.is_none(),
|
||||
"proxied GET must never carry the replication worker's source-replication-check exemption"
|
||||
);
|
||||
assert!(
|
||||
get_record.proxy_headers.ssec_algorithm.is_none() && !get_record.proxy_headers.ssec_key_present,
|
||||
"no client SSE-C headers were sent, so none may be forwarded"
|
||||
);
|
||||
|
||||
// a2. Client SSE-C headers travel verbatim to the target (the target owns
|
||||
// the real SSE-C decryption; the plaintext fake simply ignores them).
|
||||
target.take_requests();
|
||||
let ssec_key = "01234567890123456789012345678901";
|
||||
let ssec_key_b64 = BASE64_STANDARD.encode(ssec_key);
|
||||
let ssec_key_md5 = sse_customer_key_md5_base64(ssec_key);
|
||||
let _ = source_client
|
||||
.get_object()
|
||||
.bucket(source_bucket)
|
||||
.key("proxy-only")
|
||||
.sse_customer_algorithm("AES256")
|
||||
.sse_customer_key(&ssec_key_b64)
|
||||
.sse_customer_key_md5(&ssec_key_md5)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|err| format!("proxied SSE-C GET failed: {}", err.into_service_error()))?;
|
||||
let ssec_record = target
|
||||
.requests()
|
||||
.into_iter()
|
||||
.find(|record| record.operation == FakeTargetOperation::GetObject && record.key.as_deref() == Some("proxy-only"))
|
||||
.ok_or("fake target never received the proxied SSE-C GET")?;
|
||||
assert_eq!(ssec_record.proxy_headers.ssec_algorithm.as_deref(), Some("AES256"));
|
||||
assert!(ssec_record.proxy_headers.ssec_key_present, "SSE-C key header must be forwarded verbatim");
|
||||
assert_eq!(ssec_record.proxy_headers.ssec_key_md5.as_deref(), Some(ssec_key_md5.as_str()));
|
||||
assert!(ssec_record.proxy_headers.replication_check.is_none());
|
||||
|
||||
// b. HEAD of the locally-missing object is served through the proxy.
|
||||
target.take_requests();
|
||||
let head = source_client
|
||||
.head_object()
|
||||
.bucket(source_bucket)
|
||||
.key("proxy-only")
|
||||
.send()
|
||||
.await
|
||||
.map_err(|err| format!("proxied HEAD failed: {}", err.into_service_error()))?;
|
||||
assert_eq!(head.content_length, Some(payload.len() as i64));
|
||||
let head_record = target
|
||||
.requests()
|
||||
.into_iter()
|
||||
.find(|record| record.operation == FakeTargetOperation::HeadObject && record.key.as_deref() == Some("proxy-only"))
|
||||
.ok_or("fake target never received the proxied HEAD")?;
|
||||
assert_eq!(head_record.proxy_headers.source_proxy_request.as_deref(), Some("true"));
|
||||
assert!(head_record.proxy_headers.replication_check.is_none());
|
||||
|
||||
// c. Anti-loop: an inbound request that already carries the proxy marker
|
||||
// is answered locally with 404 and never forwarded to the target.
|
||||
target.take_requests();
|
||||
let err = source_client
|
||||
.get_object()
|
||||
.bucket(source_bucket)
|
||||
.key("proxy-only")
|
||||
.customize()
|
||||
.mutate_request(|req| {
|
||||
req.headers_mut().insert("x-minio-source-proxy-request", "true");
|
||||
})
|
||||
.send()
|
||||
.await
|
||||
.expect_err("anti-loop GET must fail locally instead of proxying");
|
||||
let service_err = err.into_service_error();
|
||||
assert!(service_err.is_no_such_key(), "anti-loop GET must 404, got: {service_err}");
|
||||
assert!(
|
||||
!target
|
||||
.requests()
|
||||
.iter()
|
||||
.any(|record| record.operation == FakeTargetOperation::GetObject),
|
||||
"anti-loop GET must not reach the replication target; journal: {:?}",
|
||||
target.requests()
|
||||
);
|
||||
|
||||
// c2. MinIO ProxyHeaderSet parity: the header's mere PRESENCE disables
|
||||
// proxying — "false" is exactly what a peer's replication worker sends on
|
||||
// its convergence HEADs, and proxying that miss back would fake
|
||||
// convergence.
|
||||
target.take_requests();
|
||||
let err = source_client
|
||||
.get_object()
|
||||
.bucket(source_bucket)
|
||||
.key("proxy-only")
|
||||
.customize()
|
||||
.mutate_request(|req| {
|
||||
req.headers_mut().insert("x-minio-source-proxy-request", "false");
|
||||
})
|
||||
.send()
|
||||
.await
|
||||
.expect_err("proxy-header-set GET must fail locally instead of proxying");
|
||||
let service_err = err.into_service_error();
|
||||
assert!(service_err.is_no_such_key(), "proxy-header-set GET must 404, got: {service_err}");
|
||||
assert!(
|
||||
!target
|
||||
.requests()
|
||||
.iter()
|
||||
.any(|record| record.operation == FakeTargetOperation::GetObject),
|
||||
"proxy-header-set GET must not reach the replication target; journal: {:?}",
|
||||
target.requests()
|
||||
);
|
||||
|
||||
// d. The replication worker's own convergence HEAD against the target
|
||||
// must carry `source-proxy-request: false` (never proxied back) and the
|
||||
// replication-check exemption. Trigger real replication and inspect the
|
||||
// fake journal.
|
||||
target.take_requests();
|
||||
source_client
|
||||
.put_object()
|
||||
.bucket(source_bucket)
|
||||
.key("worker-replicated")
|
||||
.body(ByteStream::from_static(b"worker payload"))
|
||||
.send()
|
||||
.await?;
|
||||
wait_for_target_request_version_id(&target, FakeTargetOperation::PutObject, "worker-replicated").await?;
|
||||
let worker_head = target
|
||||
.requests()
|
||||
.into_iter()
|
||||
.find(|record| record.operation == FakeTargetOperation::HeadObject && record.key.as_deref() == Some("worker-replicated"))
|
||||
.ok_or_else(|| format!("replication worker never HEAD-ed the target; journal: {:?}", target.requests()))?;
|
||||
assert_eq!(
|
||||
worker_head.proxy_headers.source_proxy_request.as_deref(),
|
||||
Some("false"),
|
||||
"worker convergence HEAD must send source-proxy-request: false so the target answers locally"
|
||||
);
|
||||
assert_eq!(
|
||||
worker_head.proxy_headers.replication_check.as_deref(),
|
||||
Some("true"),
|
||||
"worker convergence HEAD keeps the replication-check exemption"
|
||||
);
|
||||
|
||||
drop(source_env);
|
||||
target.shutdown().await;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// P1-5 (backlog#1675): GetObjectTagging for an object missing locally is
|
||||
/// proxied to the replication target with the anti-loop marker, mirroring
|
||||
/// MinIO `proxyGetTaggingToRepTarget`.
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_get_object_tagging_proxies_unreplicated_object_to_replication_target() -> TestResult {
|
||||
init_logging();
|
||||
|
||||
let source_bucket = "proxy-tag-src";
|
||||
let target_bucket = "proxy-tag-dst";
|
||||
let (target, source_env, source_client, target_client) = start_read_proxy_lab(source_bucket, target_bucket).await?;
|
||||
|
||||
target_client
|
||||
.put_object()
|
||||
.bucket(target_bucket)
|
||||
.key("proxy-tagged")
|
||||
.body(ByteStream::from_static(b"tagged payload"))
|
||||
.send()
|
||||
.await?;
|
||||
target_client
|
||||
.put_object_tagging()
|
||||
.bucket(target_bucket)
|
||||
.key("proxy-tagged")
|
||||
.tagging(
|
||||
aws_sdk_s3::types::Tagging::builder()
|
||||
.tag_set(aws_sdk_s3::types::Tag::builder().key("team").value("storage").build()?)
|
||||
.build()?,
|
||||
)
|
||||
.send()
|
||||
.await?;
|
||||
target.take_requests();
|
||||
|
||||
let tags = source_client
|
||||
.get_object_tagging()
|
||||
.bucket(source_bucket)
|
||||
.key("proxy-tagged")
|
||||
.send()
|
||||
.await
|
||||
.map_err(|err| format!("proxied GetObjectTagging failed: {}", err.into_service_error()))?;
|
||||
assert_eq!(tags.tag_set.len(), 1, "proxied tagging read must return the target's tags");
|
||||
assert_eq!(tags.tag_set[0].key.as_str(), "team");
|
||||
assert_eq!(tags.tag_set[0].value.as_str(), "storage");
|
||||
|
||||
let record = target
|
||||
.requests()
|
||||
.into_iter()
|
||||
.find(|record| record.operation == FakeTargetOperation::GetObjectTagging && record.key.as_deref() == Some("proxy-tagged"))
|
||||
.ok_or("fake target never received the proxied GetObjectTagging")?;
|
||||
assert_eq!(
|
||||
record.proxy_headers.source_proxy_request.as_deref(),
|
||||
Some("true"),
|
||||
"proxied tagging read must carry the anti-loop marker"
|
||||
);
|
||||
assert!(record.proxy_headers.replication_check.is_none());
|
||||
|
||||
drop(source_env);
|
||||
target.shutdown().await;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user