feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets (#6172)

* feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets

Implements the MinIO active-active read-proxy protocol (P1-5 of the
replication compatibility review): when a GET/HEAD/GetObjectTagging/
PutObjectTagging/DeleteObjectTagging request fails locally with
not-found and the bucket has replication targets, the request is proxied
to the targets in rule order, mirroring bucket-replication.go
proxyGetToReplicationTarget/proxyHeadToRepTarget/proxyTaggingToRepTarget.

Protocol surface:
- Anti-loop: inbound {x-rustfs-,x-minio-}source-proxy-request is parsed
  into ObjectOptions (proxy_request + proxy_header_set, matching MinIO
  ProxyRequest/ProxyHeaderSet); a request carrying the marker with ANY
  value is never re-proxied. Outbound client proxy calls send the marker
  as "true"; replication worker convergence HEADs send it as "false" so
  a peer's proxy layer cannot answer a convergence check by proxying
  back to the source (which would fake Completed without a PUT).
- Target selection: new replication_proxy.rs get_proxy_targets — empty
  when the marker is set, versioning is suspended, or no replication
  config; otherwise filter_target_arns -> TargetClient lookup, skipping
  targets with proxying disabled.
- TargetClient gains head_object_for_proxy/get_object (streaming) and
  the three tagging calls. Proxy calls never send the replication-check
  SSE-C exemption header; customer SSE-C keys are forwarded verbatim so
  the target performs real decryption. Conditional (If-*) headers are
  not forwarded (MinIO parity); Range and part_number are, with
  parts_count/tag_count/storage_class/expiration passed through.
- Metrics: proxy counters now count only real client proxy traffic,
  MinIO-aligned (one total per proxied request, one failed when no
  target served it). The previous misattributed counters — replication
  worker HEAD/PUT (#2672) and local tagging operations (#2682) — are
  removed; ReplProxyMetric now maps the tagging counters instead of
  dropping them.

e2e (fake_s3_target extended with tagging + header journaling): proxied
GET body + outbound header contract (marker present, no
replication-check, SSE-C passthrough), HEAD, anti-loop 404 with zero
outbound requests, GetObjectTagging, and metric mapping unit tests.

Rolling note: proxying only activates for buckets with replication
targets; requests carrying the marker keep pre-upgrade behavior.

Refs rustfs/backlog#1675 (P1-5)

* fix(replication): fail SSE-C passthrough closed on targets that drop transport headers (#6178)

SSE-C ciphertext passthrough replicates via X-Rustfs-Replication-* transport
headers. A MinIO/generic-S3 target silently discards them, storing bare
ciphertext with no decryption material — yet the PUT succeeded, so the object
reported COMPLETED with a silently unreadable replica (backlog#1675 N2).

Fail-closed design:
- SsecPassthroughCapability {Unknown, Supported, Unsupported} cached in
  BucketTargetSys per target ARN with a recording timestamp. Entries reset
  whenever the target is rebuilt, edited, or removed (arn_remotes_map
  lifecycle) and expire after SSEC_PASSTHROUGH_CAPABILITY_TTL (10 minutes):
  an expired verdict in either direction is re-earned through the audit, so
  an Unsupported target recovers automatically after an upgrade (at most one
  wasted PUT+HEAD audit per bad target per TTL window) and a Supported
  verdict cannot outlive a backend swapped behind the same endpoint.
- Replication worker (replicate_object and replicate_all): fresh Unsupported
  targets never receive the PUT — the attempt fails immediately into the
  normal MRF retry channel with a "run ?replication-check to re-probe" hint.
  Unknown or expired verdicts are audited: after the PUT the worker HEADs
  the replica back through the replication-check channel (source version id
  mapped through resolve_read_api_version_id, so null-version objects audit
  correctly) and requires SSE-C evidence (the echoed customer-algorithm
  header); missing evidence records Unsupported and fails the attempt.
  Convergence HEADs are audited the same way, so a broken ciphertext replica
  from an earlier attempt can never launder itself into COMPLETED via an
  ETag match. The gate/evidence policy is pure (replication_target_boundary,
  staleness folded in as an input) for the M2 worker migration.
- replication-check grows an SsecPassthrough probe phase: a probe PUT
  carrying the live transport-header shape, HEAD-back for evidence, and a
  machine-readable Code BucketRemoteSsecPassthroughUnsupported on failure.
  The probe verdict is synced into the runtime capability cache. Unlike
  VersionFidelity, a failed SsecPassthrough phase does NOT fail the target
  overall — it is a capability limit, not a broken replication contract,
  and a plaintext-only deployment against such a target must not turn red.
- fake_s3_target: default mode now models a RustFS target (stores the
  transport headers, echoes SSE-C evidence); the new
  drop_unlisted_replication_headers mode models MinIO. The journal records
  whether a request carried transport headers.

Receiver-echo verification: the replication-check HEAD exemption only skips
SSE-C key validation; the response has always built sse-customer-algorithm
from stored metadata (rustfs/src/app/object_usecase.rs), so no receiver
change was needed — pinned end to end by the replication-check e2e against
a real RustFS target.

Rolling-upgrade constraint: RustFS targets older than the replication-check
HEAD exemption (#5898) answer the audit HEAD without SSE-C evidence (or fail
it outright), so SSE-C replication to such targets reports FAILED. This is
deliberate — FAILED-and-retryable beats a silently undecryptable replica —
and self-heals: once the target is upgraded, the next TTL expiry (or a
manual ?replication-check re-probe) re-audits and records Supported.
Plaintext and managed-SSE replication are unaffected. The capability cache
is per-node; each node audits independently.

Known limitations:
- The audit judges evidence from the echoed customer-algorithm header only.
  A hypothetical target that preserves that one header while dropping other
  transport headers (partial-drop) would pass the audit; no known target
  behaves this way — observed targets drop the whole unknown-header family.
- A mixed-version target cluster can flap the verdict between audits routed
  to different target nodes until the rollout completes; the TTL bounds how
  long each stale verdict persists.

New e2e (backlog#1675 C1 + N2, red-first): fail-closed against a
header-dropping fake (FAILED + no second PUT via the capability cache,
journal-asserted; red run showed the old COMPLETED), replication-check
reports the SsecPassthrough phase Code while the target stays OK overall,
SSE-C heal convergence after a real target outage, and SSE-C
existing-object resync landing a REPLICA readable with the customer key.
TTL expiry in both directions is pinned at the cache and gate seams.
This commit is contained in:
唐小鸭
2026-08-18 08:47:22 +08:00
committed by GitHub
parent 21c2fb42bb
commit daecb93139
22 changed files with 2769 additions and 189 deletions
+2 -2
View File
@@ -4,8 +4,8 @@ This module is the shared failure-injection boundary for replication end-to-end
`FakeS3Target::start()` creates the listener. Add target buckets with `create_bucket`, point a RustFS remote target at `address()`, use `FAKE_ACCESS_KEY` / `FAKE_SECRET_KEY`, then enqueue per-operation faults with `inject`. Faults for one operation are consumed in FIFO order and do not consume faults queued for another operation. A fault is consumed only after `s3s` verifies the full request signature, so anonymous, other-access-key, and bad-signature traffic cannot disturb a script.
Supported data operations are HeadBucket, GetBucketVersioning, PUT/GET/HEAD/DELETE Object, and create/upload/complete/abort multipart upload. `create_bucket` models general-purpose buckets in S3's shared global namespace; account-regional namespace buckets and their `-an` names are intentionally out of scope. Buckets are versioned: PUT creates a version, DELETE without `versionId` creates a delete marker, and DELETE with `versionId` removes exactly that version. Internal source version IDs must be UUIDs and are stored canonically. Source mtime is honored only for source-replication PUT/DELETE requests; absent or invalid values use receipt time, matching RustFS, while multipart completion always uses receipt time. Replicated versions are ordered newest-first by source mtime so late older versions and delete markers do not become current. Equal mtimes prefer objects over delete markers, then canonical UUID order; RustFS's internal FileMeta signature tie-break is intentionally out of scope because it is not part of the target S3 protocol. Multipart part numbers follow S3's `1..=10000` range, and every completed part except the final part must be at least 5 MiB.
Supported data operations are HeadBucket, GetBucketVersioning, PUT/GET/HEAD/DELETE Object, Get/Put/Delete ObjectTagging (tags live per version; Put replaces the whole set, Delete clears it), and create/upload/complete/abort multipart upload. `create_bucket` models general-purpose buckets in S3's shared global namespace; account-regional namespace buckets and their `-an` names are intentionally out of scope. Buckets are versioned: PUT creates a version, DELETE without `versionId` creates a delete marker, and DELETE with `versionId` removes exactly that version. Internal source version IDs must be UUIDs and are stored canonically. Source mtime is honored only for source-replication PUT/DELETE requests; absent or invalid values use receipt time, matching RustFS, while multipart completion always uses receipt time. Replicated versions are ordered newest-first by source mtime so late older versions and delete markers do not become current. Equal mtimes prefer objects over delete markers, then canonical UUID order; RustFS's internal FileMeta signature tie-break is intentionally out of scope because it is not part of the target S3 protocol. Multipart part numbers follow S3's `1..=10000` range, and every completed part except the final part must be at least 5 MiB.
Fault actions cover HTTP 401/403/503 responses, pre-dispatch delay, connection abort when a logical request-body threshold is reached, streaming slow drain, and a deliberately wrong response ETag (including multipart-complete XML). `requests()` returns the ordered, credential-free request journal for assertions.
Fault actions cover HTTP 401/403/503 responses, pre-dispatch delay, connection abort when a logical request-body threshold is reached, streaming slow drain, and a deliberately wrong response ETag (including multipart-complete XML). `requests()` returns the ordered, credential-free request journal for assertions. Each record also journals a `ProxyHeaderSnapshot` — the read-proxy anti-loop marker (`x-{rustfs,minio}-source-proxy-request`), the replication-check exemption header, and the client SSE-C header family (algorithm and key-MD5 values; for the key itself only its presence) — so proxy tests can pin the exact wire contract.
The listener is loopback-only. It admits at most 64 active connections and two concurrently buffered request bodies; authenticated multipart-complete XML collection and assembly take both body permits. Keep-alive is disabled, request-header reads are bounded to 30 seconds, a parsed request is bounded to 65 seconds, and the complete connection lifetime is bounded to 100 seconds. It retains at most 256 buckets, 4,096 journal entries, 4,096 scripted faults, 4,096 object versions, 256 multipart uploads, and 10,000 multipart parts. Retained identifiers are capped at 1 KiB, user metadata at 2 KiB, and content type at 1 KiB. A PUT or uploaded part is capped at 64 MiB; a completed multipart object and all stored object/part data are capped at 128 MiB. Body drain, body-permit waits, delay, and slow-drain execution are bounded to 30 seconds; each slow-drain slice delay must be below that bound.
+284 -5
View File
@@ -30,10 +30,12 @@ use s3s::access::{S3Access, S3AccessContext};
use s3s::auth::SimpleAuth;
use s3s::dto::{
AbortMultipartUploadInput, AbortMultipartUploadOutput, CompleteMultipartUploadInput, CompleteMultipartUploadOutput,
CreateMultipartUploadInput, CreateMultipartUploadOutput, DeleteMarkerEntry, DeleteObjectInput, DeleteObjectOutput, ETag,
GetBucketVersioningInput, GetBucketVersioningOutput, GetObjectInput, GetObjectOutput, HeadBucketInput, HeadBucketOutput,
CreateMultipartUploadInput, CreateMultipartUploadOutput, DeleteMarkerEntry, DeleteObjectInput, DeleteObjectOutput,
DeleteObjectTaggingInput, DeleteObjectTaggingOutput, ETag, GetBucketVersioningInput, GetBucketVersioningOutput,
GetObjectInput, GetObjectOutput, GetObjectTaggingInput, GetObjectTaggingOutput, HeadBucketInput, HeadBucketOutput,
HeadObjectInput, HeadObjectOutput, ListObjectVersionsInput, ListObjectVersionsOutput, ObjectVersionId, PutObjectInput,
PutObjectOutput, StreamingBlob, Timestamp, TimestampFormat, UploadPartInput, UploadPartOutput,
PutObjectOutput, PutObjectTaggingInput, PutObjectTaggingOutput, StreamingBlob, Tag, TagSet, Timestamp, TimestampFormat,
UploadPartInput, UploadPartOutput,
};
use s3s::service::{S3Service, S3ServiceBuilder};
use s3s::validation::{AwsNameValidation, NameValidation};
@@ -88,6 +90,13 @@ const SOURCE_LEGALHOLD_TIMESTAMP_HEADERS: [&str; 2] = [
"x-rustfs-source-replication-legalhold-timestamp",
"x-minio-source-replication-legalhold-timestamp",
];
/// Wire prefix of the SSE-C passthrough replication transport headers
/// (`X-Rustfs-Replication-*`). In the default mode the fake stores them like a
/// RustFS target and echoes SSE-C evidence back on HEAD/GET; with
/// [`FakeS3Target::drop_unlisted_replication_headers`] it models MinIO /
/// generic S3, which silently discard unknown x-* headers.
const REPLICATION_SSE_TRANSPORT_PREFIX: &str = "x-rustfs-replication-";
const REPLICATION_SSEC_ALGORITHM_TRANSPORT_HEADER: &str = "x-rustfs-replication-ssec-algorithm";
const RESERVED_BUCKET_PREFIXES: [&str; 3] = ["xn--", "sthree-", "amzn-s3-demo-"];
const RESERVED_BUCKET_SUFFIXES: [&str; 6] = ["-s3alias", "--ol-s3", ".mrap", "--x-s3", "--table-s3", "-an"];
@@ -103,6 +112,9 @@ pub enum Operation {
GetObject,
HeadObject,
DeleteObject,
GetObjectTagging,
PutObjectTagging,
DeleteObjectTagging,
ListObjectVersions,
CreateMultipartUpload,
UploadPart,
@@ -149,6 +161,42 @@ impl ReplicationTimestampHeaders {
}
}
/// Read-proxy related headers observed on a request, journaled so proxy
/// tests can assert the exact wire contract: the anti-loop marker present,
/// the replication-check exemption absent, and the client SSE-C key family
/// forwarded verbatim. The SSE-C key value itself is never retained — only
/// its presence.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct ProxyHeaderSnapshot {
pub source_proxy_request: Option<String>,
pub replication_check: Option<String>,
pub ssec_algorithm: Option<String>,
pub ssec_key_present: bool,
pub ssec_key_md5: Option<String>,
/// Whether the request carried any `X-Rustfs-Replication-*` SSE-C
/// passthrough transport header, so fail-closed tests can assert the
/// sender really shipped the material a dropping target discarded.
pub ssec_transport_present: bool,
}
impl ProxyHeaderSnapshot {
fn from_headers(headers: &HeaderMap) -> Self {
Self {
source_proxy_request: header_value(headers, &["x-rustfs-source-proxy-request", "x-minio-source-proxy-request"])
.map(bounded_journal_value),
replication_check: header_value(headers, &["x-rustfs-source-replication-check", "x-minio-source-replication-check"])
.map(bounded_journal_value),
ssec_algorithm: header_value(headers, &["x-amz-server-side-encryption-customer-algorithm"])
.map(bounded_journal_value),
ssec_key_present: headers.contains_key("x-amz-server-side-encryption-customer-key"),
ssec_key_md5: header_value(headers, &["x-amz-server-side-encryption-customer-key-md5"]).map(bounded_journal_value),
ssec_transport_present: headers
.keys()
.any(|name| name.as_str().starts_with(REPLICATION_SSE_TRANSPORT_PREFIX)),
}
}
}
/// Credential-free request metadata retained for deterministic assertions.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct RequestRecord {
@@ -163,6 +211,7 @@ pub struct RequestRecord {
pub content_length: Option<u64>,
pub consumed_bytes: Option<usize>,
pub replication_timestamps: ReplicationTimestampHeaders,
pub proxy_headers: ProxyHeaderSnapshot,
pub fault: Option<FaultAction>,
}
@@ -178,6 +227,10 @@ struct ControlState {
struct StoreState {
assign_own_version_ids: bool,
assign_own_multipart_version_ids: bool,
/// MinIO-like mode: silently discard non-whitelisted replication
/// transport headers instead of storing them (see
/// [`REPLICATION_SSE_TRANSPORT_PREFIX`]).
drop_unlisted_replication_headers: bool,
buckets: HashMap<String, BucketState>,
uploads: HashMap<String, MultipartState>,
total_bytes: usize,
@@ -199,6 +252,12 @@ struct ObjectVersion {
delete_marker: bool,
content_type: Option<String>,
metadata: Option<HashMap<String, String>>,
/// Object tags as ordered key/value pairs (PutObjectTagging replaces the
/// whole set, DeleteObjectTagging clears it).
tags: Vec<(String, String)>,
/// SSE-C passthrough transport headers stored with the version (RustFS
/// target behavior); empty when the drop mode discarded them.
replication_sse_headers: Vec<(String, String)>,
}
#[derive(Clone)]
@@ -208,6 +267,7 @@ struct MultipartState {
version_id: String,
content_type: Option<String>,
metadata: Option<HashMap<String, String>>,
replication_sse_headers: Vec<(String, String)>,
parts: BTreeMap<i32, MultipartPart>,
}
@@ -428,6 +488,15 @@ impl FakeS3Target {
/// Mint own version ids for the multipart path only — models a target
/// that adopts PutObject version ids but not CreateMultipartUpload ones.
/// MinIO-like mode: silently drop every `X-Rustfs-Replication-*` SSE-C
/// passthrough transport header instead of storing it. The default (off)
/// models a RustFS target, which preserves the headers and echoes SSE-C
/// evidence (`x-amz-server-side-encryption-customer-algorithm`) on
/// HEAD/GET of the replica.
pub fn drop_unlisted_replication_headers(&self, enabled: bool) {
lock(&self.backend.store).drop_unlisted_replication_headers = enabled;
}
pub fn assign_own_multipart_version_ids(&self, enabled: bool) {
lock(&self.backend.store).assign_own_multipart_version_ids = enabled;
}
@@ -569,6 +638,7 @@ impl S3Access for FaultAccess {
.and_then(|value| value.to_str().ok())
.and_then(|value| value.parse().ok());
let replication_timestamps = ReplicationTimestampHeaders::from_headers(context.headers());
let proxy_headers = ProxyHeaderSnapshot::from_headers(context.headers());
let fault = record_request(
&self.control,
operation,
@@ -576,6 +646,7 @@ impl S3Access for FaultAccess {
parsed,
content_length,
replication_timestamps,
proxy_headers,
);
if let Some(RequestFault {
action: FaultAction::Status(status),
@@ -615,6 +686,9 @@ fn operation_from_s3_name(name: &str) -> Operation {
"GetObject" => Operation::GetObject,
"HeadObject" => Operation::HeadObject,
"DeleteObject" => Operation::DeleteObject,
"GetObjectTagging" => Operation::GetObjectTagging,
"PutObjectTagging" => Operation::PutObjectTagging,
"DeleteObjectTagging" => Operation::DeleteObjectTagging,
"CreateMultipartUpload" => Operation::CreateMultipartUpload,
"UploadPart" => Operation::UploadPart,
"CompleteMultipartUpload" => Operation::CompleteMultipartUpload,
@@ -630,6 +704,7 @@ fn record_request(
parsed: ParsedRequest,
content_length: Option<u64>,
replication_timestamps: ReplicationTimestampHeaders,
proxy_headers: ProxyHeaderSnapshot,
) -> Option<RequestFault> {
let mut state = lock(control);
let action = parsed
@@ -655,6 +730,7 @@ fn record_request(
content_length,
consumed_bytes: None,
replication_timestamps,
proxy_headers,
fault: action.clone(),
});
action.map(|action| RequestFault { sequence, action })
@@ -721,6 +797,15 @@ fn parse_request(method: &Method, uri: &Uri) -> ParsedRequest {
(&Method::POST, true) if query.contains_key("uploads") => Operation::CreateMultipartUpload,
(&Method::POST, true) if upload_id.is_some() => Operation::CompleteMultipartUpload,
(&Method::DELETE, true) if upload_id.is_some() => Operation::AbortMultipartUpload,
(&Method::GET, true) if query.contains_key("tagging") && only_query_keys(&["tagging", "versionId"]) => {
Operation::GetObjectTagging
}
(&Method::PUT, true) if query.contains_key("tagging") && only_query_keys(&["tagging", "versionId"]) => {
Operation::PutObjectTagging
}
(&Method::DELETE, true) if query.contains_key("tagging") && only_query_keys(&["tagging", "versionId"]) => {
Operation::DeleteObjectTagging
}
// A replication PUT addresses the source version via `?versionId=`.
(&Method::PUT, true) if only_query_keys(&["versionId"]) => Operation::PutObject,
(&Method::GET, true) if only_query_keys(&["versionId"]) => Operation::GetObject,
@@ -788,6 +873,29 @@ fn new_version_id(headers: &HeaderMap, assign_own: bool) -> S3Result<String> {
Ok(version_id.to_string())
}
/// Capture the SSE-C passthrough transport headers a replication PUT carried.
/// Returns an empty set in the MinIO-like drop mode.
fn captured_replication_sse_headers(headers: &HeaderMap, drop_unlisted: bool) -> Vec<(String, String)> {
if drop_unlisted {
return Vec::new();
}
headers
.iter()
.filter(|(name, _)| name.as_str().starts_with(REPLICATION_SSE_TRANSPORT_PREFIX))
.filter_map(|(name, value)| Some((name.as_str().to_string(), value.to_str().ok()?.to_string())))
.collect()
}
/// SSE-C evidence a RustFS-like target echoes for a stored passthrough
/// replica: the customer algorithm restored from the transport headers.
fn stored_sse_customer_algorithm(version: &ObjectVersion) -> Option<String> {
version
.replication_sse_headers
.iter()
.find(|(name, _)| name == REPLICATION_SSEC_ALGORITHM_TRANSPORT_HEADER)
.map(|(_, value)| value.clone())
}
fn source_etag(headers: &HeaderMap) -> S3Result<Option<String>> {
header_value(headers, &SOURCE_ETAG_HEADERS)
.map(|value| validate_retained_identifier(value, "source ETag").map(|value| normalize_etag(&value)))
@@ -1135,6 +1243,33 @@ fn find_version(state: &StoreState, bucket: &str, key: &str, version_id: Option<
Ok(version.clone())
}
/// Replace (or clear, with an empty vec) the tag set of the addressed
/// version, returning its version id. Mirrors `find_version` addressing:
/// explicit version id or the latest version, delete markers rejected.
fn set_version_tags(
state: &mut StoreState,
bucket: &str,
key: &str,
version_id: Option<&str>,
tags: Vec<(String, String)>,
) -> S3Result<String> {
// Resolve first (immutable) so the error paths match find_version.
let resolved = find_version(state, bucket, key, version_id)?.version_id;
let versions = state
.buckets
.get_mut(bucket)
.expect("bucket existence checked by find_version")
.objects
.get_mut(key)
.expect("key existence checked by find_version");
let version = versions
.iter_mut()
.find(|version| version.version_id == resolved)
.expect("version existence checked by find_version");
version.tags = tags;
Ok(resolved)
}
#[async_trait]
impl S3 for FakeBackend {
async fn head_bucket(&self, req: S3Request<HeadBucketInput>) -> S3Result<S3Response<HeadBucketOutput>> {
@@ -1231,7 +1366,10 @@ impl S3 for FakeBackend {
let input = req.input;
let body = collect_stream(input.body, input.content_length, fault.as_ref(), &self.control).await?;
validate_stored_metadata(&input.content_type, &input.metadata)?;
let assign_own = lock(&self.store).assign_own_version_ids;
let (assign_own, drop_unlisted) = {
let state = lock(&self.store);
(state.assign_own_version_ids, state.drop_unlisted_replication_headers)
};
let version_id = new_version_id(&headers, assign_own)?;
let e_tag = match source_etag(&headers)? {
Some(value) => value,
@@ -1248,6 +1386,8 @@ impl S3 for FakeBackend {
delete_marker: false,
content_type: input.content_type,
metadata: input.metadata,
tags: Vec::new(),
replication_sse_headers: captured_replication_sse_headers(&headers, drop_unlisted),
};
upsert_version(&mut lock(&self.store), &input.bucket, input.key, version)?;
Ok(apply_response_fault(
@@ -1268,6 +1408,7 @@ impl S3 for FakeBackend {
let state = lock(&self.store);
find_version(&state, &input.bucket, &input.key, input.version_id.as_deref())?
};
let sse_customer_algorithm = stored_sse_customer_algorithm(&version);
Ok(apply_response_fault(
S3Response::new(GetObjectOutput {
body: Some(StreamingBlob::new(Body::from(version.body.clone()))),
@@ -1277,6 +1418,7 @@ impl S3 for FakeBackend {
e_tag: Some(ETag::Strong(version.e_tag)),
last_modified: Some(version.last_modified.clone()),
version_id: Some(version.version_id),
sse_customer_algorithm,
..Default::default()
}),
fault.as_ref(),
@@ -1291,6 +1433,7 @@ impl S3 for FakeBackend {
let state = lock(&self.store);
find_version(&state, &input.bucket, &input.key, input.version_id.as_deref())?
};
let sse_customer_algorithm = stored_sse_customer_algorithm(&version);
Ok(apply_response_fault(
S3Response::new(HeadObjectOutput {
content_length: Some(version.body.len() as i64),
@@ -1299,12 +1442,79 @@ impl S3 for FakeBackend {
e_tag: Some(ETag::Strong(version.e_tag)),
last_modified: Some(version.last_modified.clone()),
version_id: Some(version.version_id),
sse_customer_algorithm,
..Default::default()
}),
fault.as_ref(),
))
}
async fn get_object_tagging(&self, req: S3Request<GetObjectTaggingInput>) -> S3Result<S3Response<GetObjectTaggingOutput>> {
let fault = request_fault(&req);
apply_non_body_fault(fault.as_ref(), &self.control).await?;
let input = req.input;
let version = {
let state = lock(&self.store);
find_version(&state, &input.bucket, &input.key, input.version_id.as_deref())?
};
let tag_set: TagSet = version
.tags
.into_iter()
.map(|(key, value)| Tag {
key: Some(key),
value: Some(value),
})
.collect();
Ok(apply_response_fault(
S3Response::new(GetObjectTaggingOutput {
tag_set,
version_id: Some(ObjectVersionId::from(version.version_id)),
}),
fault.as_ref(),
))
}
async fn put_object_tagging(&self, req: S3Request<PutObjectTaggingInput>) -> S3Result<S3Response<PutObjectTaggingOutput>> {
let fault = request_fault(&req);
apply_non_body_fault(fault.as_ref(), &self.control).await?;
let input = req.input;
let tags = input
.tagging
.tag_set
.into_iter()
.map(|tag| (tag.key.unwrap_or_default(), tag.value.unwrap_or_default()))
.collect();
let version_id = {
let mut state = lock(&self.store);
set_version_tags(&mut state, &input.bucket, &input.key, input.version_id.as_deref(), tags)?
};
Ok(apply_response_fault(
S3Response::new(PutObjectTaggingOutput {
version_id: Some(ObjectVersionId::from(version_id)),
}),
fault.as_ref(),
))
}
async fn delete_object_tagging(
&self,
req: S3Request<DeleteObjectTaggingInput>,
) -> S3Result<S3Response<DeleteObjectTaggingOutput>> {
let fault = request_fault(&req);
apply_non_body_fault(fault.as_ref(), &self.control).await?;
let input = req.input;
let version_id = {
let mut state = lock(&self.store);
set_version_tags(&mut state, &input.bucket, &input.key, input.version_id.as_deref(), Vec::new())?
};
Ok(apply_response_fault(
S3Response::new(DeleteObjectTaggingOutput {
version_id: Some(ObjectVersionId::from(version_id)),
}),
fault.as_ref(),
))
}
async fn delete_object(&self, req: S3Request<DeleteObjectInput>) -> S3Result<S3Response<DeleteObjectOutput>> {
let fault = request_fault(&req);
apply_non_body_fault(fault.as_ref(), &self.control).await?;
@@ -1381,6 +1591,8 @@ impl S3 for FakeBackend {
delete_marker: true,
content_type: None,
metadata: None,
tags: Vec::new(),
replication_sse_headers: Vec::new(),
},
)?;
Ok(apply_response_fault(
@@ -1408,9 +1620,10 @@ impl S3 for FakeBackend {
ensure_upload_budget(&state)?;
validate_stored_metadata(&input.content_type, &input.metadata)?;
let upload_id = Uuid::new_v4().to_string();
// Read the flag before the mutable borrow of `state.uploads` below
// Read the flags before the mutable borrow of `state.uploads` below
// (and never re-lock the store: the mutex is not reentrant).
let mint_own = state.assign_own_version_ids || state.assign_own_multipart_version_ids;
let drop_unlisted = state.drop_unlisted_replication_headers;
let version_id = new_version_id(&headers, mint_own)?;
state.uploads.insert(
upload_id.clone(),
@@ -1420,6 +1633,7 @@ impl S3 for FakeBackend {
version_id,
content_type: input.content_type,
metadata: input.metadata,
replication_sse_headers: captured_replication_sse_headers(&headers, drop_unlisted),
parts: BTreeMap::new(),
},
);
@@ -1557,6 +1771,7 @@ impl S3 for FakeBackend {
version_id: upload.version_id.clone(),
content_type: upload.content_type.clone(),
metadata: upload.metadata.clone(),
replication_sse_headers: upload.replication_sse_headers.clone(),
parts: BTreeMap::new(),
},
selected,
@@ -1583,6 +1798,8 @@ impl S3 for FakeBackend {
delete_marker: false,
content_type: upload.content_type,
metadata: upload.metadata,
tags: Vec::new(),
replication_sse_headers: upload.replication_sse_headers,
};
let mut state = lock(&self.store);
let current = state
@@ -1787,6 +2004,65 @@ mod tests {
Ok(())
}
/// Default mode is RustFS-like: SSE-C passthrough transport headers are
/// stored and the customer algorithm is echoed on HEAD/GET. Drop mode is
/// MinIO-like: the headers are silently discarded, so no evidence comes
/// back — the exact difference the N2 fail-closed audit keys on. Both
/// modes journal that the sender shipped the transport headers.
#[tokio::test]
async fn ssec_passthrough_headers_echo_and_drop_modes() -> Result<(), BoxError> {
let target = FakeS3Target::start().await?;
target.create_bucket("target-bucket");
let client = client(&target);
let put_with_transport_headers = |key: &'static str| {
client
.put_object()
.bucket("target-bucket")
.key(key)
.body(ByteStream::from_static(b"ciphertext"))
.customize()
.map_request(move |mut request| {
let headers = request.headers_mut();
headers.insert("x-rustfs-replication-ssec-algorithm", "AES256");
headers.insert("x-rustfs-replication-ssec-key-md5", "AAAAAAAAAAAAAAAAAAAAAA==");
Ok::<_, std::convert::Infallible>(request)
})
.send()
};
put_with_transport_headers("kept").await?;
let head = client.head_object().bucket("target-bucket").key("kept").send().await?;
assert_eq!(head.sse_customer_algorithm(), Some("AES256"));
let get = client.get_object().bucket("target-bucket").key("kept").send().await?;
assert_eq!(get.sse_customer_algorithm(), Some("AES256"));
target.drop_unlisted_replication_headers(true);
put_with_transport_headers("dropped").await?;
let head = client.head_object().bucket("target-bucket").key("dropped").send().await?;
assert_eq!(head.sse_customer_algorithm(), None, "drop mode must discard SSE-C evidence");
let requests = target.requests();
for key in ["kept", "dropped"] {
let record = requests
.iter()
.find(|record| record.operation == Operation::PutObject && record.key.as_deref() == Some(key))
.expect("PUT must be journaled");
assert!(
record.proxy_headers.ssec_transport_present,
"the journal must prove the sender shipped the transport headers for {key}"
);
}
let plain_head = requests
.iter()
.find(|record| record.operation == Operation::HeadObject)
.expect("HEAD must be journaled");
assert!(!plain_head.proxy_headers.ssec_transport_present);
target.shutdown().await;
Ok(())
}
macro_rules! assert_sdk_error {
($error:expr, $status:expr, $code:expr) => {{
let error = &$error;
@@ -3052,6 +3328,7 @@ mod tests {
version_id: index.to_string(),
content_type: None,
metadata: None,
replication_sse_headers: Vec::new(),
parts: BTreeMap::new(),
},
);
@@ -3074,6 +3351,7 @@ mod tests {
},
Some(0),
ReplicationTimestampHeaders::default(),
ProxyHeaderSnapshot::default(),
);
}
let records = lock(&control).requests.clone();
@@ -3096,6 +3374,7 @@ mod tests {
},
None,
ReplicationTimestampHeaders::default(),
ProxyHeaderSnapshot::default(),
);
{
let bounded_records = lock(&bounded_control);
@@ -2610,17 +2610,20 @@ async fn test_replication_check_succeeds_with_remote_target() -> Result<(), Box<
assert_eq!(response.status(), StatusCode::OK);
let payload: serde_json::Value = response.json().await?;
assert_eq!(payload["Status"], "OK");
assert_eq!(payload["Status"], "OK", "{payload}");
assert_eq!(payload["ActiveMutation"], true);
assert_eq!(payload["Targets"].as_array().map(Vec::len), Some(1));
assert_eq!(payload["Targets"][0]["Status"], "OK");
assert_eq!(payload["Targets"][0]["Phases"]["Put"]["Status"], "OK");
assert_eq!(payload["Targets"][0]["Status"], "OK", "{payload}");
assert_eq!(payload["Targets"][0]["Phases"]["Put"]["Status"], "OK", "{payload}");
// A RustFS target adopts the source version id, so the P1-19
// version-identity probe passes.
assert_eq!(payload["Targets"][0]["Phases"]["VersionFidelity"]["Status"], "OK");
assert_eq!(payload["Targets"][0]["Phases"]["DeleteMarker"]["Status"], "OK");
assert_eq!(payload["Targets"][0]["Phases"]["VersionDelete"]["Status"], "OK");
assert_eq!(payload["Targets"][0]["Phases"]["Cleanup"]["Status"], "OK");
assert_eq!(payload["Targets"][0]["Phases"]["VersionFidelity"]["Status"], "OK", "{payload}");
// A RustFS target preserves the SSE-C passthrough transport headers and
// echoes the customer algorithm on the replication-check HEAD (N2).
assert_eq!(payload["Targets"][0]["Phases"]["SsecPassthrough"]["Status"], "OK", "{payload}");
assert_eq!(payload["Targets"][0]["Phases"]["DeleteMarker"]["Status"], "OK", "{payload}");
assert_eq!(payload["Targets"][0]["Phases"]["VersionDelete"]["Status"], "OK", "{payload}");
assert_eq!(payload["Targets"][0]["Phases"]["Cleanup"]["Status"], "OK", "{payload}");
let target_client = target_env.create_s3_client();
let versions = target_client
@@ -4649,6 +4652,410 @@ async fn test_bucket_replication_sse_c_multipart_passthrough() -> TestResult {
Ok(())
}
/// N2 (backlog#1675 P1-22): SSE-C passthrough replication to a target that
/// silently drops the `X-Rustfs-Replication-*` transport headers (MinIO-like
/// behavior, modeled by the fake target's drop mode) used to report COMPLETED
/// while the replica had irrecoverably lost its decryption material — the red
/// light this test was born failing on. Fail-closed contract now under test:
/// the first attempt PUTs, HEAD-backs the replica, finds no SSE-C evidence,
/// records the target Unsupported and reports FAILED; a second SSE-C object
/// fails without any PUT reaching the target (capability cache, proven from
/// the target journal); plaintext objects still replicate COMPLETED.
#[tokio::test]
#[serial]
async fn test_ssec_replication_fails_closed_when_target_drops_passthrough_headers() -> TestResult {
init_logging();
let target = FakeS3Target::start().await?;
let target_bucket = "ssec-drop-dst";
target.create_bucket(target_bucket);
target.drop_unlisted_replication_headers(true);
let mut source_env = RustFSTestEnvironment::new().await?;
let mut env_vars = replication_fast_env();
env_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
env_vars.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
source_env.start_rustfs_server_with_env(vec![], &env_vars).await?;
let source_bucket = "ssec-drop-src";
let source_client = source_env.create_s3_client();
source_client.create_bucket().bucket(source_bucket).send().await?;
enable_bucket_versioning(&source_env, source_bucket).await?;
let target_arn = set_replication_target_with_options(
&source_env,
source_bucket,
ReplicationTargetOptions {
endpoint: &target.address(),
access_key: FAKE_ACCESS_KEY,
secret_key: FAKE_SECRET_KEY,
target_bucket,
secure: false,
skip_tls_verify: false,
ca_cert_pem: None,
},
)
.await?;
put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
let customer_key = BASE64_STANDARD.encode(REPL17_SSEC_KEY);
let customer_key_md5 = sse_customer_key_md5_base64(REPL17_SSEC_KEY);
let put_ssec = |key: &'static str| {
source_client
.put_object()
.bucket(source_bucket)
.key(key)
.body(ByteStream::from_static(b"ssec fail-closed payload"))
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
};
// First SSE-C object: the audit must catch the dropped material.
put_ssec("ssec-first.txt").await?;
wait_for_source_replication_status(&source_client, source_bucket, "ssec-first.txt", "FAILED", true).await?;
let requests = target.take_requests();
let first_put = requests
.iter()
.find(|record| record.operation == FakeTargetOperation::PutObject && record.key.as_deref() == Some("ssec-first.txt"))
.ok_or("the first SSE-C object must have been PUT (capability was Unknown)")?;
assert!(
first_put.proxy_headers.ssec_transport_present,
"the replication PUT must have shipped the SSE-C transport headers the target then dropped"
);
assert!(
requests.iter().any(|record| {
record.operation == FakeTargetOperation::HeadObject
&& record.key.as_deref() == Some("ssec-first.txt")
&& record.sequence > first_put.sequence
&& record.proxy_headers.replication_check.as_deref() == Some("true")
}),
"the post-PUT HEAD-back audit must have run through the replication-check channel; journal: {requests:?}"
);
// Second SSE-C object: the cached Unsupported verdict fails it closed
// before any PUT — including MRF retries of the first object.
put_ssec("ssec-second.txt").await?;
wait_for_source_replication_status(&source_client, source_bucket, "ssec-second.txt", "FAILED", true).await?;
assert!(
!target.requests().iter().any(|record| {
record.operation == FakeTargetOperation::PutObject
&& record.key.as_deref() != Some("plain-control.txt")
&& record.proxy_headers.ssec_transport_present
}),
"no further SSE-C ciphertext may reach a target recorded Unsupported; journal: {:?}",
target.requests()
);
// The gate is scoped to SSE-C: plaintext replication keeps working.
source_client
.put_object()
.bucket(source_bucket)
.key("plain-control.txt")
.body(ByteStream::from_static(b"plaintext control payload"))
.send()
.await?;
wait_for_source_replication_status(&source_client, source_bucket, "plain-control.txt", "COMPLETED", false).await?;
assert!(target.has_object(target_bucket, "plain-control.txt"));
target.shutdown().await;
Ok(())
}
/// N2 (backlog#1675 P1-22): the admin replication-check must expose the same
/// verdict operators would otherwise only learn from failing SSE-C objects —
/// an SsecPassthrough probe phase that fails with the machine-readable
/// `BucketRemoteSsecPassthroughUnsupported` code against a header-dropping
/// target, with no probe residue left behind. The target's overall status
/// stays OK: unlike version-identity drift, dropped passthrough headers are
/// a capability limit, and a plaintext-only deployment against a MinIO-like
/// target must not turn red.
#[tokio::test]
#[serial]
async fn test_replication_check_flags_ssec_passthrough_dropping_target() -> TestResult {
init_logging();
let target = FakeS3Target::start().await?;
let target_bucket = "ssec-check-dst";
target.create_bucket(target_bucket);
target.drop_unlisted_replication_headers(true);
let mut source_env = RustFSTestEnvironment::new().await?;
let mut env_vars = replication_fast_env();
env_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
env_vars.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
source_env.start_rustfs_server_with_env(vec![], &env_vars).await?;
let source_bucket = "ssec-check-src";
let source_client = source_env.create_s3_client();
source_client.create_bucket().bucket(source_bucket).send().await?;
enable_bucket_versioning(&source_env, source_bucket).await?;
let target_arn = set_replication_target_with_options(
&source_env,
source_bucket,
ReplicationTargetOptions {
endpoint: &target.address(),
access_key: FAKE_ACCESS_KEY,
secret_key: FAKE_SECRET_KEY,
target_bucket,
secure: false,
skip_tls_verify: false,
ca_cert_pem: None,
},
)
.await?;
put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
let response = run_replication_check(&source_env, source_bucket).await?;
assert_eq!(response.status(), StatusCode::OK);
let payload: serde_json::Value = response.json().await?;
assert_eq!(
payload["Status"], "OK",
"a capability-only SSE-C failure must not fail the check overall: {payload}"
);
let target_report = &payload["Targets"][0];
assert_eq!(target_report["Status"], "OK", "{payload}");
let ssec = &target_report["Phases"]["SsecPassthrough"];
assert_eq!(ssec["Status"], "FAILED", "SsecPassthrough phase must fail: {payload}");
assert_eq!(
ssec["Code"], "BucketRemoteSsecPassthroughUnsupported",
"the failure must carry the machine-readable code: {payload}"
);
// Basic replication of plaintext objects works on this target: every other
// phase passes, so the code is the discriminator operators branch on.
assert_eq!(target_report["Phases"]["Put"]["Status"], "OK", "{payload}");
assert_eq!(target_report["Phases"]["VersionFidelity"]["Status"], "OK", "{payload}");
assert_eq!(target_report["Phases"]["DeleteMarker"]["Status"], "OK", "{payload}");
assert_eq!(target_report["Phases"]["VersionDelete"]["Status"], "OK", "{payload}");
assert_eq!(target_report["Phases"]["Cleanup"]["Status"], "OK", "{payload}");
// The SSE-C probe PUT must have shipped the real transport header names —
// a mangled or missing header set would fail the phase for the wrong
// reason and mask a working target.
let requests = target.requests();
assert!(
requests
.iter()
.any(|record| record.operation == FakeTargetOperation::PutObject && record.proxy_headers.ssec_transport_present),
"the SSE-C probe PUT must carry the X-Rustfs-Replication-* transport headers; journal: {requests:?}"
);
// No probe residue, including the SSE-C probe version.
let probe_put = requests
.into_iter()
.find(|record| record.operation == FakeTargetOperation::PutObject)
.ok_or("the probe PUT never reached the fake target")?;
let probe_key = probe_put.key.ok_or("probe PUT journal record has no key")?;
assert!(
target.stored_versions(target_bucket, &probe_key).is_empty(),
"all probe versions must be cleaned up"
);
target.shutdown().await;
Ok(())
}
/// C1 (backlog#1675 P1-22): heal-path convergence for SSE-C. An SSE-C object
/// whose live replication failed during a target outage must converge through
/// the scanner/heal compensation once the target returns — passing the N2
/// HEAD-back audit against the recovered RustFS target — and the replica must
/// be readable with the customer key.
#[tokio::test]
#[serial]
async fn test_bucket_replication_sse_c_heals_after_target_outage() -> TestResult {
init_logging();
let (source_env, mut target_env, source_bucket, target_bucket) =
build_sse_replication_pair("ssec-heal", false, false).await?;
let source_client = source_env.create_s3_client();
let key = "ssec-heal-contract.txt";
let body = b"repl-22 ssec heal payload".to_vec();
let customer_key = BASE64_STANDARD.encode(REPL17_SSEC_KEY);
let customer_key_md5 = sse_customer_key_md5_base64(REPL17_SSEC_KEY);
// Target outage: the SSE-C write cannot replicate.
target_env.stop_server();
source_client
.put_object()
.bucket(&source_bucket)
.key(key)
.body(ByteStream::from(body.clone()))
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
.await?;
// The failure is observable on the source (SSE-C HEAD needs the key).
let deadline = tokio::time::Instant::now() + Duration::from_secs(30);
loop {
let head = source_client
.head_object()
.bucket(&source_bucket)
.key(key)
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
.await?;
match head.replication_status().map(|status| status.as_str()) {
Some("PENDING") | Some("FAILED") => break,
other => {
if tokio::time::Instant::now() >= deadline {
return Err(format!("source SSE-C object never reported PENDING/FAILED; last status={other:?}").into());
}
sleep(Duration::from_millis(200)).await;
}
}
}
// Recover the target in place; the source scanner re-drives the failure.
target_env
.restart_server_preserving_data(vec![], &[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")])
.await?;
wait_for_source_replication_status(&source_client, &source_bucket, key, "COMPLETED", true).await?;
// The healed replica is a REPLICA (status surfaces on HEAD) readable with
// the customer key.
let target_client = target_env.create_s3_client();
let replica_head = target_client
.head_object()
.bucket(&target_bucket)
.key(key)
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
.await?;
assert_eq!(
replica_head.replication_status().map(|status| status.as_str()),
Some("REPLICA"),
"the healed copy must carry REPLICA status"
);
let replica = target_client
.get_object()
.bucket(&target_bucket)
.key(key)
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
.await?;
assert_eq!(replica.sse_customer_algorithm(), Some("AES256"));
assert_eq!(replica.body.collect().await?.into_bytes().as_ref(), body.as_slice());
Ok(())
}
/// C1 (backlog#1675 P1-22): existing-object resync for SSE-C. An SSE-C object
/// written BEFORE any replication config must reach the RustFS target through
/// the existing-object resync (`replicate_all` transport, N2-audited), land as
/// a REPLICA, and read back with the customer key.
#[tokio::test]
#[serial]
async fn test_bucket_replication_sse_c_existing_object_resync() -> TestResult {
init_logging();
let mut source_env = RustFSTestEnvironment::new().await?;
let mut source_process_env = replication_fast_env();
source_process_env.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
source_process_env.extend_from_slice(FAST_SCANNER_ENV);
source_process_env.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
source_env.start_rustfs_server_with_env(vec![], &source_process_env).await?;
let mut target_env = RustFSTestEnvironment::new().await?;
target_env
.start_rustfs_server_without_cleanup_with_env(&[
("NO_PROXY", "127.0.0.1,localhost"),
("HTTP_PROXY", ""),
("HTTPS_PROXY", ""),
])
.await?;
let source_bucket = "ssec-existing-src";
let target_bucket = "ssec-existing-dst";
let source_client = source_env.create_s3_client();
let target_client = target_env.create_s3_client();
source_client.create_bucket().bucket(source_bucket).send().await?;
target_client.create_bucket().bucket(target_bucket).send().await?;
enable_bucket_versioning(&source_env, source_bucket).await?;
enable_bucket_versioning(&target_env, target_bucket).await?;
// The SSE-C object exists before any replication wiring.
let key = "ssec-existing-contract.txt";
let body = b"repl-22 ssec existing-object payload".to_vec();
let customer_key = BASE64_STANDARD.encode(REPL17_SSEC_KEY);
let customer_key_md5 = sse_customer_key_md5_base64(REPL17_SSEC_KEY);
source_client
.put_object()
.bucket(source_bucket)
.key(key)
.body(ByteStream::from(body.clone()))
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
.await?;
// Wire replication (existing-object enabled) and drive a resync.
let target_arn = set_replication_target(&source_env, source_bucket, &target_env, target_bucket).await?;
put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
let (reset_arn, reset_id) = start_bucket_replication_reset(&source_env, source_bucket).await?;
assert_eq!(reset_arn, target_arn);
let terminal = wait_for_replication_reset_target(&source_env, source_bucket, &target_arn, |status| {
status.reset_id == reset_id && matches!(status.status.as_str(), "Completed" | "Failed")
})
.await?;
assert_eq!(terminal.status, "Completed", "SSE-C existing-object resync must complete");
assert!(terminal.replicated_count >= 1, "the existing SSE-C object must have been resynced");
// The replica is a REPLICA (status surfaces on HEAD) readable with the
// customer key.
let replica_head = target_client
.head_object()
.bucket(target_bucket)
.key(key)
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
.await?;
assert_eq!(
replica_head.replication_status().map(|status| status.as_str()),
Some("REPLICA"),
"the resynced copy must carry REPLICA status"
);
let replica = target_client
.get_object()
.bucket(target_bucket)
.key(key)
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
.await?;
assert_eq!(replica.sse_customer_algorithm(), Some("AES256"));
assert_eq!(replica.body.collect().await?.into_bytes().as_ref(), body.as_slice());
// No plaintext leak: the replica stays unreadable without the key.
assert!(
target_client
.get_object()
.bucket(target_bucket)
.key(key)
.send()
.await
.is_err(),
"SSE-C replica must not be readable without the customer key"
);
Ok(())
}
/// backlog#1147 repl-17 / backlog#1783: SSE-S3 objects replicate by decrypting
/// at the source and re-encrypting on the target with the target's own KMS.
/// The property backlog#1291 pinned — never a silent plaintext replica — still
@@ -8417,3 +8824,304 @@ async fn test_scanner_never_compensates_when_existing_object_replication_disable
Ok(())
}
/// Shared setup for the P1-5 read-proxy scenarios (backlog#1675): a RustFS
/// source with an enabled replication rule pointing at the fake target, and
/// an object seeded DIRECTLY on the target — it exists remotely but not
/// locally, exactly the active-active replication-lag window the read proxy
/// serves.
async fn start_read_proxy_lab(
source_bucket: &str,
target_bucket: &str,
) -> Result<(FakeS3Target, RustFSTestEnvironment, Client, Client), Box<dyn Error + Send + Sync>> {
let target = FakeS3Target::start().await?;
target.create_bucket(target_bucket);
target.assign_own_version_ids(true);
let mut source_env = RustFSTestEnvironment::new().await?;
let mut process_env = replication_fast_env();
process_env.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
process_env.extend_from_slice(&[
("NO_PROXY", "127.0.0.1,localhost"),
("HTTP_PROXY", ""),
("HTTPS_PROXY", ""),
("RUST_LOG", "error"),
]);
source_env.start_rustfs_server_with_env(vec![], &process_env).await?;
let source_client = source_env.create_s3_client();
source_client.create_bucket().bucket(source_bucket).send().await?;
enable_bucket_versioning(&source_env, source_bucket).await?;
let target_arn = set_replication_target_with_options(
&source_env,
source_bucket,
ReplicationTargetOptions {
endpoint: &target.address(),
access_key: FAKE_ACCESS_KEY,
secret_key: FAKE_SECRET_KEY,
target_bucket,
secure: false,
skip_tls_verify: false,
ca_cert_pem: None,
},
)
.await?;
put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
let target_client = Client::from_conf(crate::common::build_test_s3_config(
target.endpoint(),
FAKE_ACCESS_KEY,
FAKE_SECRET_KEY,
None,
"read-proxy-e2e",
));
Ok((target, source_env, source_client, target_client))
}
/// P1-5 (backlog#1675): during the active-active replication lag window a
/// GET/HEAD for an object the local site does not have yet is proxied to the
/// replication target. Pins the wire contract: the anti-loop
/// `source-proxy-request` marker is sent, the replication worker's
/// `source-replication-check` SSE-C exemption is NEVER sent, client SSE-C
/// headers are forwarded verbatim, and an inbound request that was itself
/// proxied is answered locally (404) without touching the target.
#[tokio::test]
#[serial]
async fn test_get_and_head_proxy_unreplicated_object_to_replication_target() -> TestResult {
init_logging();
let source_bucket = "proxy-read-src";
let target_bucket = "proxy-read-dst";
let (target, source_env, source_client, target_client) = start_read_proxy_lab(source_bucket, target_bucket).await?;
let payload = b"proxy payload".to_vec();
target_client
.put_object()
.bucket(target_bucket)
.key("proxy-only")
.body(ByteStream::from(payload.clone()))
.send()
.await?;
target.take_requests();
// a. GET of the locally-missing object is served through the proxy.
let got = source_client
.get_object()
.bucket(source_bucket)
.key("proxy-only")
.send()
.await
.map_err(|err| format!("proxied GET failed: {}", err.into_service_error()))?;
assert_eq!(got.content_length, Some(payload.len() as i64));
let body = got.body.collect().await?.into_bytes();
assert_eq!(body.as_ref(), payload.as_slice(), "proxied GET must stream the target's body");
let get_record = target
.requests()
.into_iter()
.find(|record| record.operation == FakeTargetOperation::GetObject && record.key.as_deref() == Some("proxy-only"))
.ok_or("fake target never received the proxied GET")?;
assert_eq!(
get_record.proxy_headers.source_proxy_request.as_deref(),
Some("true"),
"proxied GET must carry the anti-loop source-proxy-request marker"
);
assert!(
get_record.proxy_headers.replication_check.is_none(),
"proxied GET must never carry the replication worker's source-replication-check exemption"
);
assert!(
get_record.proxy_headers.ssec_algorithm.is_none() && !get_record.proxy_headers.ssec_key_present,
"no client SSE-C headers were sent, so none may be forwarded"
);
// a2. Client SSE-C headers travel verbatim to the target (the target owns
// the real SSE-C decryption; the plaintext fake simply ignores them).
target.take_requests();
let ssec_key = "01234567890123456789012345678901";
let ssec_key_b64 = BASE64_STANDARD.encode(ssec_key);
let ssec_key_md5 = sse_customer_key_md5_base64(ssec_key);
let _ = source_client
.get_object()
.bucket(source_bucket)
.key("proxy-only")
.sse_customer_algorithm("AES256")
.sse_customer_key(&ssec_key_b64)
.sse_customer_key_md5(&ssec_key_md5)
.send()
.await
.map_err(|err| format!("proxied SSE-C GET failed: {}", err.into_service_error()))?;
let ssec_record = target
.requests()
.into_iter()
.find(|record| record.operation == FakeTargetOperation::GetObject && record.key.as_deref() == Some("proxy-only"))
.ok_or("fake target never received the proxied SSE-C GET")?;
assert_eq!(ssec_record.proxy_headers.ssec_algorithm.as_deref(), Some("AES256"));
assert!(ssec_record.proxy_headers.ssec_key_present, "SSE-C key header must be forwarded verbatim");
assert_eq!(ssec_record.proxy_headers.ssec_key_md5.as_deref(), Some(ssec_key_md5.as_str()));
assert!(ssec_record.proxy_headers.replication_check.is_none());
// b. HEAD of the locally-missing object is served through the proxy.
target.take_requests();
let head = source_client
.head_object()
.bucket(source_bucket)
.key("proxy-only")
.send()
.await
.map_err(|err| format!("proxied HEAD failed: {}", err.into_service_error()))?;
assert_eq!(head.content_length, Some(payload.len() as i64));
let head_record = target
.requests()
.into_iter()
.find(|record| record.operation == FakeTargetOperation::HeadObject && record.key.as_deref() == Some("proxy-only"))
.ok_or("fake target never received the proxied HEAD")?;
assert_eq!(head_record.proxy_headers.source_proxy_request.as_deref(), Some("true"));
assert!(head_record.proxy_headers.replication_check.is_none());
// c. Anti-loop: an inbound request that already carries the proxy marker
// is answered locally with 404 and never forwarded to the target.
target.take_requests();
let err = source_client
.get_object()
.bucket(source_bucket)
.key("proxy-only")
.customize()
.mutate_request(|req| {
req.headers_mut().insert("x-minio-source-proxy-request", "true");
})
.send()
.await
.expect_err("anti-loop GET must fail locally instead of proxying");
let service_err = err.into_service_error();
assert!(service_err.is_no_such_key(), "anti-loop GET must 404, got: {service_err}");
assert!(
!target
.requests()
.iter()
.any(|record| record.operation == FakeTargetOperation::GetObject),
"anti-loop GET must not reach the replication target; journal: {:?}",
target.requests()
);
// c2. MinIO ProxyHeaderSet parity: the header's mere PRESENCE disables
// proxying — "false" is exactly what a peer's replication worker sends on
// its convergence HEADs, and proxying that miss back would fake
// convergence.
target.take_requests();
let err = source_client
.get_object()
.bucket(source_bucket)
.key("proxy-only")
.customize()
.mutate_request(|req| {
req.headers_mut().insert("x-minio-source-proxy-request", "false");
})
.send()
.await
.expect_err("proxy-header-set GET must fail locally instead of proxying");
let service_err = err.into_service_error();
assert!(service_err.is_no_such_key(), "proxy-header-set GET must 404, got: {service_err}");
assert!(
!target
.requests()
.iter()
.any(|record| record.operation == FakeTargetOperation::GetObject),
"proxy-header-set GET must not reach the replication target; journal: {:?}",
target.requests()
);
// d. The replication worker's own convergence HEAD against the target
// must carry `source-proxy-request: false` (never proxied back) and the
// replication-check exemption. Trigger real replication and inspect the
// fake journal.
target.take_requests();
source_client
.put_object()
.bucket(source_bucket)
.key("worker-replicated")
.body(ByteStream::from_static(b"worker payload"))
.send()
.await?;
wait_for_target_request_version_id(&target, FakeTargetOperation::PutObject, "worker-replicated").await?;
let worker_head = target
.requests()
.into_iter()
.find(|record| record.operation == FakeTargetOperation::HeadObject && record.key.as_deref() == Some("worker-replicated"))
.ok_or_else(|| format!("replication worker never HEAD-ed the target; journal: {:?}", target.requests()))?;
assert_eq!(
worker_head.proxy_headers.source_proxy_request.as_deref(),
Some("false"),
"worker convergence HEAD must send source-proxy-request: false so the target answers locally"
);
assert_eq!(
worker_head.proxy_headers.replication_check.as_deref(),
Some("true"),
"worker convergence HEAD keeps the replication-check exemption"
);
drop(source_env);
target.shutdown().await;
Ok(())
}
/// P1-5 (backlog#1675): GetObjectTagging for an object missing locally is
/// proxied to the replication target with the anti-loop marker, mirroring
/// MinIO `proxyGetTaggingToRepTarget`.
#[tokio::test]
#[serial]
async fn test_get_object_tagging_proxies_unreplicated_object_to_replication_target() -> TestResult {
init_logging();
let source_bucket = "proxy-tag-src";
let target_bucket = "proxy-tag-dst";
let (target, source_env, source_client, target_client) = start_read_proxy_lab(source_bucket, target_bucket).await?;
target_client
.put_object()
.bucket(target_bucket)
.key("proxy-tagged")
.body(ByteStream::from_static(b"tagged payload"))
.send()
.await?;
target_client
.put_object_tagging()
.bucket(target_bucket)
.key("proxy-tagged")
.tagging(
aws_sdk_s3::types::Tagging::builder()
.tag_set(aws_sdk_s3::types::Tag::builder().key("team").value("storage").build()?)
.build()?,
)
.send()
.await?;
target.take_requests();
let tags = source_client
.get_object_tagging()
.bucket(source_bucket)
.key("proxy-tagged")
.send()
.await
.map_err(|err| format!("proxied GetObjectTagging failed: {}", err.into_service_error()))?;
assert_eq!(tags.tag_set.len(), 1, "proxied tagging read must return the target's tags");
assert_eq!(tags.tag_set[0].key.as_str(), "team");
assert_eq!(tags.tag_set[0].value.as_str(), "storage");
let record = target
.requests()
.into_iter()
.find(|record| record.operation == FakeTargetOperation::GetObjectTagging && record.key.as_deref() == Some("proxy-tagged"))
.ok_or("fake target never received the proxied GetObjectTagging")?;
assert_eq!(
record.proxy_headers.source_proxy_request.as_deref(),
Some("true"),
"proxied tagging read must carry the anti-loop marker"
);
assert!(record.proxy_headers.replication_check.is_none());
drop(source_env);
target.shutdown().await;
Ok(())
}