fix(replication): fail SSE-C passthrough closed on targets that drop transport headers

SSE-C ciphertext passthrough replicates via X-Rustfs-Replication-* transport
headers. A MinIO/generic-S3 target silently discards them, storing bare
ciphertext with no decryption material — yet the PUT succeeded, so the object
reported COMPLETED with a silently unreadable replica (backlog#1675 N2).

Fail-closed design:
- SsecPassthroughCapability {Unknown, Supported, Unsupported} cached in
  BucketTargetSys per target ARN with a recording timestamp. Entries reset
  whenever the target is rebuilt, edited, or removed (arn_remotes_map
  lifecycle) and expire after SSEC_PASSTHROUGH_CAPABILITY_TTL (10 minutes):
  an expired verdict in either direction is re-earned through the audit, so
  an Unsupported target recovers automatically after an upgrade (at most one
  wasted PUT+HEAD audit per bad target per TTL window) and a Supported
  verdict cannot outlive a backend swapped behind the same endpoint.
- Replication worker (replicate_object and replicate_all): fresh Unsupported
  targets never receive the PUT — the attempt fails immediately into the
  normal MRF retry channel with a "run ?replication-check to re-probe" hint.
  Unknown or expired verdicts are audited: after the PUT the worker HEADs
  the replica back through the replication-check channel (source version id
  mapped through resolve_read_api_version_id, so null-version objects audit
  correctly) and requires SSE-C evidence (the echoed customer-algorithm
  header); missing evidence records Unsupported and fails the attempt.
  Convergence HEADs are audited the same way, so a broken ciphertext replica
  from an earlier attempt can never launder itself into COMPLETED via an
  ETag match. The gate/evidence policy is pure (replication_target_boundary,
  staleness folded in as an input) for the M2 worker migration.
- replication-check grows an SsecPassthrough probe phase: a probe PUT
  carrying the live transport-header shape, HEAD-back for evidence, and a
  machine-readable Code BucketRemoteSsecPassthroughUnsupported on failure.
  The probe verdict is synced into the runtime capability cache. Unlike
  VersionFidelity, a failed SsecPassthrough phase does NOT fail the target
  overall — it is a capability limit, not a broken replication contract,
  and a plaintext-only deployment against such a target must not turn red.
- fake_s3_target: default mode now models a RustFS target (stores the
  transport headers, echoes SSE-C evidence); the new
  drop_unlisted_replication_headers mode models MinIO. The journal records
  whether a request carried transport headers.

Receiver-echo verification: the replication-check HEAD exemption only skips
SSE-C key validation; the response has always built sse-customer-algorithm
from stored metadata (rustfs/src/app/object_usecase.rs), so no receiver
change was needed — pinned end to end by the replication-check e2e against
a real RustFS target.

Rolling-upgrade constraint: RustFS targets older than the replication-check
HEAD exemption (#5898) answer the audit HEAD without SSE-C evidence (or fail
it outright), so SSE-C replication to such targets reports FAILED. This is
deliberate — FAILED-and-retryable beats a silently undecryptable replica —
and self-heals: once the target is upgraded, the next TTL expiry (or a
manual ?replication-check re-probe) re-audits and records Supported.
Plaintext and managed-SSE replication are unaffected. The capability cache
is per-node; each node audits independently.

Known limitations:
- The audit judges evidence from the echoed customer-algorithm header only.
  A hypothetical target that preserves that one header while dropping other
  transport headers (partial-drop) would pass the audit; no known target
  behaves this way — observed targets drop the whole unknown-header family.
- A mixed-version target cluster can flap the verdict between audits routed
  to different target nodes until the rollout completes; the TTL bounds how
  long each stale verdict persists.

New e2e (backlog#1675 C1 + N2, red-first): fail-closed against a
header-dropping fake (FAILED + no second PUT via the capability cache,
journal-asserted; red run showed the old COMPLETED), replication-check
reports the SsecPassthrough phase Code while the target stays OK overall,
SSE-C heal convergence after a real target outage, and SSE-C
existing-object resync landing a REPLICA readable with the customer key.
TTL expiry in both directions is pinned at the cache and gate seams.
This commit is contained in:
唐小鸭
2026-08-18 00:54:26 +08:00
parent 9baa92563a
commit d20476a66c
8 changed files with 1300 additions and 31 deletions
+129 -2
View File
@@ -90,6 +90,13 @@ const SOURCE_LEGALHOLD_TIMESTAMP_HEADERS: [&str; 2] = [
"x-rustfs-source-replication-legalhold-timestamp", "x-rustfs-source-replication-legalhold-timestamp",
"x-minio-source-replication-legalhold-timestamp", "x-minio-source-replication-legalhold-timestamp",
]; ];
/// Wire prefix of the SSE-C passthrough replication transport headers
/// (`X-Rustfs-Replication-*`). In the default mode the fake stores them like a
/// RustFS target and echoes SSE-C evidence back on HEAD/GET; with
/// [`FakeS3Target::drop_unlisted_replication_headers`] it models MinIO /
/// generic S3, which silently discard unknown x-* headers.
const REPLICATION_SSE_TRANSPORT_PREFIX: &str = "x-rustfs-replication-";
const REPLICATION_SSEC_ALGORITHM_TRANSPORT_HEADER: &str = "x-rustfs-replication-ssec-algorithm";
const RESERVED_BUCKET_PREFIXES: [&str; 3] = ["xn--", "sthree-", "amzn-s3-demo-"]; const RESERVED_BUCKET_PREFIXES: [&str; 3] = ["xn--", "sthree-", "amzn-s3-demo-"];
const RESERVED_BUCKET_SUFFIXES: [&str; 6] = ["-s3alias", "--ol-s3", ".mrap", "--x-s3", "--table-s3", "-an"]; const RESERVED_BUCKET_SUFFIXES: [&str; 6] = ["-s3alias", "--ol-s3", ".mrap", "--x-s3", "--table-s3", "-an"];
@@ -166,6 +173,10 @@ pub struct ProxyHeaderSnapshot {
pub ssec_algorithm: Option<String>, pub ssec_algorithm: Option<String>,
pub ssec_key_present: bool, pub ssec_key_present: bool,
pub ssec_key_md5: Option<String>, pub ssec_key_md5: Option<String>,
/// Whether the request carried any `X-Rustfs-Replication-*` SSE-C
/// passthrough transport header, so fail-closed tests can assert the
/// sender really shipped the material a dropping target discarded.
pub ssec_transport_present: bool,
} }
impl ProxyHeaderSnapshot { impl ProxyHeaderSnapshot {
@@ -179,6 +190,9 @@ impl ProxyHeaderSnapshot {
.map(bounded_journal_value), .map(bounded_journal_value),
ssec_key_present: headers.contains_key("x-amz-server-side-encryption-customer-key"), ssec_key_present: headers.contains_key("x-amz-server-side-encryption-customer-key"),
ssec_key_md5: header_value(headers, &["x-amz-server-side-encryption-customer-key-md5"]).map(bounded_journal_value), ssec_key_md5: header_value(headers, &["x-amz-server-side-encryption-customer-key-md5"]).map(bounded_journal_value),
ssec_transport_present: headers
.keys()
.any(|name| name.as_str().starts_with(REPLICATION_SSE_TRANSPORT_PREFIX)),
} }
} }
} }
@@ -213,6 +227,10 @@ struct ControlState {
struct StoreState { struct StoreState {
assign_own_version_ids: bool, assign_own_version_ids: bool,
assign_own_multipart_version_ids: bool, assign_own_multipart_version_ids: bool,
/// MinIO-like mode: silently discard non-whitelisted replication
/// transport headers instead of storing them (see
/// [`REPLICATION_SSE_TRANSPORT_PREFIX`]).
drop_unlisted_replication_headers: bool,
buckets: HashMap<String, BucketState>, buckets: HashMap<String, BucketState>,
uploads: HashMap<String, MultipartState>, uploads: HashMap<String, MultipartState>,
total_bytes: usize, total_bytes: usize,
@@ -237,6 +255,9 @@ struct ObjectVersion {
/// Object tags as ordered key/value pairs (PutObjectTagging replaces the /// Object tags as ordered key/value pairs (PutObjectTagging replaces the
/// whole set, DeleteObjectTagging clears it). /// whole set, DeleteObjectTagging clears it).
tags: Vec<(String, String)>, tags: Vec<(String, String)>,
/// SSE-C passthrough transport headers stored with the version (RustFS
/// target behavior); empty when the drop mode discarded them.
replication_sse_headers: Vec<(String, String)>,
} }
#[derive(Clone)] #[derive(Clone)]
@@ -246,6 +267,7 @@ struct MultipartState {
version_id: String, version_id: String,
content_type: Option<String>, content_type: Option<String>,
metadata: Option<HashMap<String, String>>, metadata: Option<HashMap<String, String>>,
replication_sse_headers: Vec<(String, String)>,
parts: BTreeMap<i32, MultipartPart>, parts: BTreeMap<i32, MultipartPart>,
} }
@@ -466,6 +488,15 @@ impl FakeS3Target {
/// Mint own version ids for the multipart path only — models a target /// Mint own version ids for the multipart path only — models a target
/// that adopts PutObject version ids but not CreateMultipartUpload ones. /// that adopts PutObject version ids but not CreateMultipartUpload ones.
/// MinIO-like mode: silently drop every `X-Rustfs-Replication-*` SSE-C
/// passthrough transport header instead of storing it. The default (off)
/// models a RustFS target, which preserves the headers and echoes SSE-C
/// evidence (`x-amz-server-side-encryption-customer-algorithm`) on
/// HEAD/GET of the replica.
pub fn drop_unlisted_replication_headers(&self, enabled: bool) {
lock(&self.backend.store).drop_unlisted_replication_headers = enabled;
}
pub fn assign_own_multipart_version_ids(&self, enabled: bool) { pub fn assign_own_multipart_version_ids(&self, enabled: bool) {
lock(&self.backend.store).assign_own_multipart_version_ids = enabled; lock(&self.backend.store).assign_own_multipart_version_ids = enabled;
} }
@@ -842,6 +873,29 @@ fn new_version_id(headers: &HeaderMap, assign_own: bool) -> S3Result<String> {
Ok(version_id.to_string()) Ok(version_id.to_string())
} }
/// Capture the SSE-C passthrough transport headers a replication PUT carried.
/// Returns an empty set in the MinIO-like drop mode.
fn captured_replication_sse_headers(headers: &HeaderMap, drop_unlisted: bool) -> Vec<(String, String)> {
if drop_unlisted {
return Vec::new();
}
headers
.iter()
.filter(|(name, _)| name.as_str().starts_with(REPLICATION_SSE_TRANSPORT_PREFIX))
.filter_map(|(name, value)| Some((name.as_str().to_string(), value.to_str().ok()?.to_string())))
.collect()
}
/// SSE-C evidence a RustFS-like target echoes for a stored passthrough
/// replica: the customer algorithm restored from the transport headers.
fn stored_sse_customer_algorithm(version: &ObjectVersion) -> Option<String> {
version
.replication_sse_headers
.iter()
.find(|(name, _)| name == REPLICATION_SSEC_ALGORITHM_TRANSPORT_HEADER)
.map(|(_, value)| value.clone())
}
fn source_etag(headers: &HeaderMap) -> S3Result<Option<String>> { fn source_etag(headers: &HeaderMap) -> S3Result<Option<String>> {
header_value(headers, &SOURCE_ETAG_HEADERS) header_value(headers, &SOURCE_ETAG_HEADERS)
.map(|value| validate_retained_identifier(value, "source ETag").map(|value| normalize_etag(&value))) .map(|value| validate_retained_identifier(value, "source ETag").map(|value| normalize_etag(&value)))
@@ -1312,7 +1366,10 @@ impl S3 for FakeBackend {
let input = req.input; let input = req.input;
let body = collect_stream(input.body, input.content_length, fault.as_ref(), &self.control).await?; let body = collect_stream(input.body, input.content_length, fault.as_ref(), &self.control).await?;
validate_stored_metadata(&input.content_type, &input.metadata)?; validate_stored_metadata(&input.content_type, &input.metadata)?;
let assign_own = lock(&self.store).assign_own_version_ids; let (assign_own, drop_unlisted) = {
let state = lock(&self.store);
(state.assign_own_version_ids, state.drop_unlisted_replication_headers)
};
let version_id = new_version_id(&headers, assign_own)?; let version_id = new_version_id(&headers, assign_own)?;
let e_tag = match source_etag(&headers)? { let e_tag = match source_etag(&headers)? {
Some(value) => value, Some(value) => value,
@@ -1330,6 +1387,7 @@ impl S3 for FakeBackend {
content_type: input.content_type, content_type: input.content_type,
metadata: input.metadata, metadata: input.metadata,
tags: Vec::new(), tags: Vec::new(),
replication_sse_headers: captured_replication_sse_headers(&headers, drop_unlisted),
}; };
upsert_version(&mut lock(&self.store), &input.bucket, input.key, version)?; upsert_version(&mut lock(&self.store), &input.bucket, input.key, version)?;
Ok(apply_response_fault( Ok(apply_response_fault(
@@ -1350,6 +1408,7 @@ impl S3 for FakeBackend {
let state = lock(&self.store); let state = lock(&self.store);
find_version(&state, &input.bucket, &input.key, input.version_id.as_deref())? find_version(&state, &input.bucket, &input.key, input.version_id.as_deref())?
}; };
let sse_customer_algorithm = stored_sse_customer_algorithm(&version);
Ok(apply_response_fault( Ok(apply_response_fault(
S3Response::new(GetObjectOutput { S3Response::new(GetObjectOutput {
body: Some(StreamingBlob::new(Body::from(version.body.clone()))), body: Some(StreamingBlob::new(Body::from(version.body.clone()))),
@@ -1359,6 +1418,7 @@ impl S3 for FakeBackend {
e_tag: Some(ETag::Strong(version.e_tag)), e_tag: Some(ETag::Strong(version.e_tag)),
last_modified: Some(version.last_modified.clone()), last_modified: Some(version.last_modified.clone()),
version_id: Some(version.version_id), version_id: Some(version.version_id),
sse_customer_algorithm,
..Default::default() ..Default::default()
}), }),
fault.as_ref(), fault.as_ref(),
@@ -1373,6 +1433,7 @@ impl S3 for FakeBackend {
let state = lock(&self.store); let state = lock(&self.store);
find_version(&state, &input.bucket, &input.key, input.version_id.as_deref())? find_version(&state, &input.bucket, &input.key, input.version_id.as_deref())?
}; };
let sse_customer_algorithm = stored_sse_customer_algorithm(&version);
Ok(apply_response_fault( Ok(apply_response_fault(
S3Response::new(HeadObjectOutput { S3Response::new(HeadObjectOutput {
content_length: Some(version.body.len() as i64), content_length: Some(version.body.len() as i64),
@@ -1381,6 +1442,7 @@ impl S3 for FakeBackend {
e_tag: Some(ETag::Strong(version.e_tag)), e_tag: Some(ETag::Strong(version.e_tag)),
last_modified: Some(version.last_modified.clone()), last_modified: Some(version.last_modified.clone()),
version_id: Some(version.version_id), version_id: Some(version.version_id),
sse_customer_algorithm,
..Default::default() ..Default::default()
}), }),
fault.as_ref(), fault.as_ref(),
@@ -1530,6 +1592,7 @@ impl S3 for FakeBackend {
content_type: None, content_type: None,
metadata: None, metadata: None,
tags: Vec::new(), tags: Vec::new(),
replication_sse_headers: Vec::new(),
}, },
)?; )?;
Ok(apply_response_fault( Ok(apply_response_fault(
@@ -1557,9 +1620,10 @@ impl S3 for FakeBackend {
ensure_upload_budget(&state)?; ensure_upload_budget(&state)?;
validate_stored_metadata(&input.content_type, &input.metadata)?; validate_stored_metadata(&input.content_type, &input.metadata)?;
let upload_id = Uuid::new_v4().to_string(); let upload_id = Uuid::new_v4().to_string();
// Read the flag before the mutable borrow of `state.uploads` below // Read the flags before the mutable borrow of `state.uploads` below
// (and never re-lock the store: the mutex is not reentrant). // (and never re-lock the store: the mutex is not reentrant).
let mint_own = state.assign_own_version_ids || state.assign_own_multipart_version_ids; let mint_own = state.assign_own_version_ids || state.assign_own_multipart_version_ids;
let drop_unlisted = state.drop_unlisted_replication_headers;
let version_id = new_version_id(&headers, mint_own)?; let version_id = new_version_id(&headers, mint_own)?;
state.uploads.insert( state.uploads.insert(
upload_id.clone(), upload_id.clone(),
@@ -1569,6 +1633,7 @@ impl S3 for FakeBackend {
version_id, version_id,
content_type: input.content_type, content_type: input.content_type,
metadata: input.metadata, metadata: input.metadata,
replication_sse_headers: captured_replication_sse_headers(&headers, drop_unlisted),
parts: BTreeMap::new(), parts: BTreeMap::new(),
}, },
); );
@@ -1706,6 +1771,7 @@ impl S3 for FakeBackend {
version_id: upload.version_id.clone(), version_id: upload.version_id.clone(),
content_type: upload.content_type.clone(), content_type: upload.content_type.clone(),
metadata: upload.metadata.clone(), metadata: upload.metadata.clone(),
replication_sse_headers: upload.replication_sse_headers.clone(),
parts: BTreeMap::new(), parts: BTreeMap::new(),
}, },
selected, selected,
@@ -1733,6 +1799,7 @@ impl S3 for FakeBackend {
content_type: upload.content_type, content_type: upload.content_type,
metadata: upload.metadata, metadata: upload.metadata,
tags: Vec::new(), tags: Vec::new(),
replication_sse_headers: upload.replication_sse_headers,
}; };
let mut state = lock(&self.store); let mut state = lock(&self.store);
let current = state let current = state
@@ -1937,6 +2004,65 @@ mod tests {
Ok(()) Ok(())
} }
/// Default mode is RustFS-like: SSE-C passthrough transport headers are
/// stored and the customer algorithm is echoed on HEAD/GET. Drop mode is
/// MinIO-like: the headers are silently discarded, so no evidence comes
/// back — the exact difference the N2 fail-closed audit keys on. Both
/// modes journal that the sender shipped the transport headers.
#[tokio::test]
async fn ssec_passthrough_headers_echo_and_drop_modes() -> Result<(), BoxError> {
let target = FakeS3Target::start().await?;
target.create_bucket("target-bucket");
let client = client(&target);
let put_with_transport_headers = |key: &'static str| {
client
.put_object()
.bucket("target-bucket")
.key(key)
.body(ByteStream::from_static(b"ciphertext"))
.customize()
.map_request(move |mut request| {
let headers = request.headers_mut();
headers.insert("x-rustfs-replication-ssec-algorithm", "AES256");
headers.insert("x-rustfs-replication-ssec-key-md5", "AAAAAAAAAAAAAAAAAAAAAA==");
Ok::<_, std::convert::Infallible>(request)
})
.send()
};
put_with_transport_headers("kept").await?;
let head = client.head_object().bucket("target-bucket").key("kept").send().await?;
assert_eq!(head.sse_customer_algorithm(), Some("AES256"));
let get = client.get_object().bucket("target-bucket").key("kept").send().await?;
assert_eq!(get.sse_customer_algorithm(), Some("AES256"));
target.drop_unlisted_replication_headers(true);
put_with_transport_headers("dropped").await?;
let head = client.head_object().bucket("target-bucket").key("dropped").send().await?;
assert_eq!(head.sse_customer_algorithm(), None, "drop mode must discard SSE-C evidence");
let requests = target.requests();
for key in ["kept", "dropped"] {
let record = requests
.iter()
.find(|record| record.operation == Operation::PutObject && record.key.as_deref() == Some(key))
.expect("PUT must be journaled");
assert!(
record.proxy_headers.ssec_transport_present,
"the journal must prove the sender shipped the transport headers for {key}"
);
}
let plain_head = requests
.iter()
.find(|record| record.operation == Operation::HeadObject)
.expect("HEAD must be journaled");
assert!(!plain_head.proxy_headers.ssec_transport_present);
target.shutdown().await;
Ok(())
}
macro_rules! assert_sdk_error { macro_rules! assert_sdk_error {
($error:expr, $status:expr, $code:expr) => {{ ($error:expr, $status:expr, $code:expr) => {{
let error = &$error; let error = &$error;
@@ -3202,6 +3328,7 @@ mod tests {
version_id: index.to_string(), version_id: index.to_string(),
content_type: None, content_type: None,
metadata: None, metadata: None,
replication_sse_headers: Vec::new(),
parts: BTreeMap::new(), parts: BTreeMap::new(),
}, },
); );
@@ -2610,17 +2610,20 @@ async fn test_replication_check_succeeds_with_remote_target() -> Result<(), Box<
assert_eq!(response.status(), StatusCode::OK); assert_eq!(response.status(), StatusCode::OK);
let payload: serde_json::Value = response.json().await?; let payload: serde_json::Value = response.json().await?;
assert_eq!(payload["Status"], "OK"); assert_eq!(payload["Status"], "OK", "{payload}");
assert_eq!(payload["ActiveMutation"], true); assert_eq!(payload["ActiveMutation"], true);
assert_eq!(payload["Targets"].as_array().map(Vec::len), Some(1)); assert_eq!(payload["Targets"].as_array().map(Vec::len), Some(1));
assert_eq!(payload["Targets"][0]["Status"], "OK"); assert_eq!(payload["Targets"][0]["Status"], "OK", "{payload}");
assert_eq!(payload["Targets"][0]["Phases"]["Put"]["Status"], "OK"); assert_eq!(payload["Targets"][0]["Phases"]["Put"]["Status"], "OK", "{payload}");
// A RustFS target adopts the source version id, so the P1-19 // A RustFS target adopts the source version id, so the P1-19
// version-identity probe passes. // version-identity probe passes.
assert_eq!(payload["Targets"][0]["Phases"]["VersionFidelity"]["Status"], "OK"); assert_eq!(payload["Targets"][0]["Phases"]["VersionFidelity"]["Status"], "OK", "{payload}");
assert_eq!(payload["Targets"][0]["Phases"]["DeleteMarker"]["Status"], "OK"); // A RustFS target preserves the SSE-C passthrough transport headers and
assert_eq!(payload["Targets"][0]["Phases"]["VersionDelete"]["Status"], "OK"); // echoes the customer algorithm on the replication-check HEAD (N2).
assert_eq!(payload["Targets"][0]["Phases"]["Cleanup"]["Status"], "OK"); assert_eq!(payload["Targets"][0]["Phases"]["SsecPassthrough"]["Status"], "OK", "{payload}");
assert_eq!(payload["Targets"][0]["Phases"]["DeleteMarker"]["Status"], "OK", "{payload}");
assert_eq!(payload["Targets"][0]["Phases"]["VersionDelete"]["Status"], "OK", "{payload}");
assert_eq!(payload["Targets"][0]["Phases"]["Cleanup"]["Status"], "OK", "{payload}");
let target_client = target_env.create_s3_client(); let target_client = target_env.create_s3_client();
let versions = target_client let versions = target_client
@@ -4649,6 +4652,410 @@ async fn test_bucket_replication_sse_c_multipart_passthrough() -> TestResult {
Ok(()) Ok(())
} }
/// N2 (backlog#1675 P1-22): SSE-C passthrough replication to a target that
/// silently drops the `X-Rustfs-Replication-*` transport headers (MinIO-like
/// behavior, modeled by the fake target's drop mode) used to report COMPLETED
/// while the replica had irrecoverably lost its decryption material — the red
/// light this test was born failing on. Fail-closed contract now under test:
/// the first attempt PUTs, HEAD-backs the replica, finds no SSE-C evidence,
/// records the target Unsupported and reports FAILED; a second SSE-C object
/// fails without any PUT reaching the target (capability cache, proven from
/// the target journal); plaintext objects still replicate COMPLETED.
#[tokio::test]
#[serial]
async fn test_ssec_replication_fails_closed_when_target_drops_passthrough_headers() -> TestResult {
init_logging();
let target = FakeS3Target::start().await?;
let target_bucket = "ssec-drop-dst";
target.create_bucket(target_bucket);
target.drop_unlisted_replication_headers(true);
let mut source_env = RustFSTestEnvironment::new().await?;
let mut env_vars = replication_fast_env();
env_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
env_vars.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
source_env.start_rustfs_server_with_env(vec![], &env_vars).await?;
let source_bucket = "ssec-drop-src";
let source_client = source_env.create_s3_client();
source_client.create_bucket().bucket(source_bucket).send().await?;
enable_bucket_versioning(&source_env, source_bucket).await?;
let target_arn = set_replication_target_with_options(
&source_env,
source_bucket,
ReplicationTargetOptions {
endpoint: &target.address(),
access_key: FAKE_ACCESS_KEY,
secret_key: FAKE_SECRET_KEY,
target_bucket,
secure: false,
skip_tls_verify: false,
ca_cert_pem: None,
},
)
.await?;
put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
let customer_key = BASE64_STANDARD.encode(REPL17_SSEC_KEY);
let customer_key_md5 = sse_customer_key_md5_base64(REPL17_SSEC_KEY);
let put_ssec = |key: &'static str| {
source_client
.put_object()
.bucket(source_bucket)
.key(key)
.body(ByteStream::from_static(b"ssec fail-closed payload"))
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
};
// First SSE-C object: the audit must catch the dropped material.
put_ssec("ssec-first.txt").await?;
wait_for_source_replication_status(&source_client, source_bucket, "ssec-first.txt", "FAILED", true).await?;
let requests = target.take_requests();
let first_put = requests
.iter()
.find(|record| record.operation == FakeTargetOperation::PutObject && record.key.as_deref() == Some("ssec-first.txt"))
.ok_or("the first SSE-C object must have been PUT (capability was Unknown)")?;
assert!(
first_put.proxy_headers.ssec_transport_present,
"the replication PUT must have shipped the SSE-C transport headers the target then dropped"
);
assert!(
requests.iter().any(|record| {
record.operation == FakeTargetOperation::HeadObject
&& record.key.as_deref() == Some("ssec-first.txt")
&& record.sequence > first_put.sequence
&& record.proxy_headers.replication_check.as_deref() == Some("true")
}),
"the post-PUT HEAD-back audit must have run through the replication-check channel; journal: {requests:?}"
);
// Second SSE-C object: the cached Unsupported verdict fails it closed
// before any PUT — including MRF retries of the first object.
put_ssec("ssec-second.txt").await?;
wait_for_source_replication_status(&source_client, source_bucket, "ssec-second.txt", "FAILED", true).await?;
assert!(
!target.requests().iter().any(|record| {
record.operation == FakeTargetOperation::PutObject
&& record.key.as_deref() != Some("plain-control.txt")
&& record.proxy_headers.ssec_transport_present
}),
"no further SSE-C ciphertext may reach a target recorded Unsupported; journal: {:?}",
target.requests()
);
// The gate is scoped to SSE-C: plaintext replication keeps working.
source_client
.put_object()
.bucket(source_bucket)
.key("plain-control.txt")
.body(ByteStream::from_static(b"plaintext control payload"))
.send()
.await?;
wait_for_source_replication_status(&source_client, source_bucket, "plain-control.txt", "COMPLETED", false).await?;
assert!(target.has_object(target_bucket, "plain-control.txt"));
target.shutdown().await;
Ok(())
}
/// N2 (backlog#1675 P1-22): the admin replication-check must expose the same
/// verdict operators would otherwise only learn from failing SSE-C objects —
/// an SsecPassthrough probe phase that fails with the machine-readable
/// `BucketRemoteSsecPassthroughUnsupported` code against a header-dropping
/// target, with no probe residue left behind. The target's overall status
/// stays OK: unlike version-identity drift, dropped passthrough headers are
/// a capability limit, and a plaintext-only deployment against a MinIO-like
/// target must not turn red.
#[tokio::test]
#[serial]
async fn test_replication_check_flags_ssec_passthrough_dropping_target() -> TestResult {
init_logging();
let target = FakeS3Target::start().await?;
let target_bucket = "ssec-check-dst";
target.create_bucket(target_bucket);
target.drop_unlisted_replication_headers(true);
let mut source_env = RustFSTestEnvironment::new().await?;
let mut env_vars = replication_fast_env();
env_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
env_vars.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
source_env.start_rustfs_server_with_env(vec![], &env_vars).await?;
let source_bucket = "ssec-check-src";
let source_client = source_env.create_s3_client();
source_client.create_bucket().bucket(source_bucket).send().await?;
enable_bucket_versioning(&source_env, source_bucket).await?;
let target_arn = set_replication_target_with_options(
&source_env,
source_bucket,
ReplicationTargetOptions {
endpoint: &target.address(),
access_key: FAKE_ACCESS_KEY,
secret_key: FAKE_SECRET_KEY,
target_bucket,
secure: false,
skip_tls_verify: false,
ca_cert_pem: None,
},
)
.await?;
put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
let response = run_replication_check(&source_env, source_bucket).await?;
assert_eq!(response.status(), StatusCode::OK);
let payload: serde_json::Value = response.json().await?;
assert_eq!(
payload["Status"], "OK",
"a capability-only SSE-C failure must not fail the check overall: {payload}"
);
let target_report = &payload["Targets"][0];
assert_eq!(target_report["Status"], "OK", "{payload}");
let ssec = &target_report["Phases"]["SsecPassthrough"];
assert_eq!(ssec["Status"], "FAILED", "SsecPassthrough phase must fail: {payload}");
assert_eq!(
ssec["Code"], "BucketRemoteSsecPassthroughUnsupported",
"the failure must carry the machine-readable code: {payload}"
);
// Basic replication of plaintext objects works on this target: every other
// phase passes, so the code is the discriminator operators branch on.
assert_eq!(target_report["Phases"]["Put"]["Status"], "OK", "{payload}");
assert_eq!(target_report["Phases"]["VersionFidelity"]["Status"], "OK", "{payload}");
assert_eq!(target_report["Phases"]["DeleteMarker"]["Status"], "OK", "{payload}");
assert_eq!(target_report["Phases"]["VersionDelete"]["Status"], "OK", "{payload}");
assert_eq!(target_report["Phases"]["Cleanup"]["Status"], "OK", "{payload}");
// The SSE-C probe PUT must have shipped the real transport header names —
// a mangled or missing header set would fail the phase for the wrong
// reason and mask a working target.
let requests = target.requests();
assert!(
requests
.iter()
.any(|record| record.operation == FakeTargetOperation::PutObject && record.proxy_headers.ssec_transport_present),
"the SSE-C probe PUT must carry the X-Rustfs-Replication-* transport headers; journal: {requests:?}"
);
// No probe residue, including the SSE-C probe version.
let probe_put = requests
.into_iter()
.find(|record| record.operation == FakeTargetOperation::PutObject)
.ok_or("the probe PUT never reached the fake target")?;
let probe_key = probe_put.key.ok_or("probe PUT journal record has no key")?;
assert!(
target.stored_versions(target_bucket, &probe_key).is_empty(),
"all probe versions must be cleaned up"
);
target.shutdown().await;
Ok(())
}
/// C1 (backlog#1675 P1-22): heal-path convergence for SSE-C. An SSE-C object
/// whose live replication failed during a target outage must converge through
/// the scanner/heal compensation once the target returns — passing the N2
/// HEAD-back audit against the recovered RustFS target — and the replica must
/// be readable with the customer key.
#[tokio::test]
#[serial]
async fn test_bucket_replication_sse_c_heals_after_target_outage() -> TestResult {
init_logging();
let (source_env, mut target_env, source_bucket, target_bucket) =
build_sse_replication_pair("ssec-heal", false, false).await?;
let source_client = source_env.create_s3_client();
let key = "ssec-heal-contract.txt";
let body = b"repl-22 ssec heal payload".to_vec();
let customer_key = BASE64_STANDARD.encode(REPL17_SSEC_KEY);
let customer_key_md5 = sse_customer_key_md5_base64(REPL17_SSEC_KEY);
// Target outage: the SSE-C write cannot replicate.
target_env.stop_server();
source_client
.put_object()
.bucket(&source_bucket)
.key(key)
.body(ByteStream::from(body.clone()))
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
.await?;
// The failure is observable on the source (SSE-C HEAD needs the key).
let deadline = tokio::time::Instant::now() + Duration::from_secs(30);
loop {
let head = source_client
.head_object()
.bucket(&source_bucket)
.key(key)
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
.await?;
match head.replication_status().map(|status| status.as_str()) {
Some("PENDING") | Some("FAILED") => break,
other => {
if tokio::time::Instant::now() >= deadline {
return Err(format!("source SSE-C object never reported PENDING/FAILED; last status={other:?}").into());
}
sleep(Duration::from_millis(200)).await;
}
}
}
// Recover the target in place; the source scanner re-drives the failure.
target_env
.restart_server_preserving_data(vec![], &[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")])
.await?;
wait_for_source_replication_status(&source_client, &source_bucket, key, "COMPLETED", true).await?;
// The healed replica is a REPLICA (status surfaces on HEAD) readable with
// the customer key.
let target_client = target_env.create_s3_client();
let replica_head = target_client
.head_object()
.bucket(&target_bucket)
.key(key)
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
.await?;
assert_eq!(
replica_head.replication_status().map(|status| status.as_str()),
Some("REPLICA"),
"the healed copy must carry REPLICA status"
);
let replica = target_client
.get_object()
.bucket(&target_bucket)
.key(key)
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
.await?;
assert_eq!(replica.sse_customer_algorithm(), Some("AES256"));
assert_eq!(replica.body.collect().await?.into_bytes().as_ref(), body.as_slice());
Ok(())
}
/// C1 (backlog#1675 P1-22): existing-object resync for SSE-C. An SSE-C object
/// written BEFORE any replication config must reach the RustFS target through
/// the existing-object resync (`replicate_all` transport, N2-audited), land as
/// a REPLICA, and read back with the customer key.
#[tokio::test]
#[serial]
async fn test_bucket_replication_sse_c_existing_object_resync() -> TestResult {
init_logging();
let mut source_env = RustFSTestEnvironment::new().await?;
let mut source_process_env = replication_fast_env();
source_process_env.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
source_process_env.extend_from_slice(FAST_SCANNER_ENV);
source_process_env.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
source_env.start_rustfs_server_with_env(vec![], &source_process_env).await?;
let mut target_env = RustFSTestEnvironment::new().await?;
target_env
.start_rustfs_server_without_cleanup_with_env(&[
("NO_PROXY", "127.0.0.1,localhost"),
("HTTP_PROXY", ""),
("HTTPS_PROXY", ""),
])
.await?;
let source_bucket = "ssec-existing-src";
let target_bucket = "ssec-existing-dst";
let source_client = source_env.create_s3_client();
let target_client = target_env.create_s3_client();
source_client.create_bucket().bucket(source_bucket).send().await?;
target_client.create_bucket().bucket(target_bucket).send().await?;
enable_bucket_versioning(&source_env, source_bucket).await?;
enable_bucket_versioning(&target_env, target_bucket).await?;
// The SSE-C object exists before any replication wiring.
let key = "ssec-existing-contract.txt";
let body = b"repl-22 ssec existing-object payload".to_vec();
let customer_key = BASE64_STANDARD.encode(REPL17_SSEC_KEY);
let customer_key_md5 = sse_customer_key_md5_base64(REPL17_SSEC_KEY);
source_client
.put_object()
.bucket(source_bucket)
.key(key)
.body(ByteStream::from(body.clone()))
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
.await?;
// Wire replication (existing-object enabled) and drive a resync.
let target_arn = set_replication_target(&source_env, source_bucket, &target_env, target_bucket).await?;
put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
let (reset_arn, reset_id) = start_bucket_replication_reset(&source_env, source_bucket).await?;
assert_eq!(reset_arn, target_arn);
let terminal = wait_for_replication_reset_target(&source_env, source_bucket, &target_arn, |status| {
status.reset_id == reset_id && matches!(status.status.as_str(), "Completed" | "Failed")
})
.await?;
assert_eq!(terminal.status, "Completed", "SSE-C existing-object resync must complete");
assert!(terminal.replicated_count >= 1, "the existing SSE-C object must have been resynced");
// The replica is a REPLICA (status surfaces on HEAD) readable with the
// customer key.
let replica_head = target_client
.head_object()
.bucket(target_bucket)
.key(key)
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
.await?;
assert_eq!(
replica_head.replication_status().map(|status| status.as_str()),
Some("REPLICA"),
"the resynced copy must carry REPLICA status"
);
let replica = target_client
.get_object()
.bucket(target_bucket)
.key(key)
.sse_customer_algorithm("AES256")
.sse_customer_key(&customer_key)
.sse_customer_key_md5(&customer_key_md5)
.send()
.await?;
assert_eq!(replica.sse_customer_algorithm(), Some("AES256"));
assert_eq!(replica.body.collect().await?.into_bytes().as_ref(), body.as_slice());
// No plaintext leak: the replica stays unreadable without the key.
assert!(
target_client
.get_object()
.bucket(target_bucket)
.key(key)
.send()
.await
.is_err(),
"SSE-C replica must not be readable without the customer key"
);
Ok(())
}
/// backlog#1147 repl-17 / backlog#1783: SSE-S3 objects replicate by decrypting /// backlog#1147 repl-17 / backlog#1783: SSE-S3 objects replicate by decrypting
/// at the source and re-encrypting on the target with the target's own KMS. /// at the source and re-encrypting on the target with the target's own KMS.
/// The property backlog#1291 pinned — never a silent plaintext replica — still /// The property backlog#1291 pinned — never a silent plaintext replica — still
+1 -1
View File
@@ -32,7 +32,7 @@ pub mod bucket {
pub mod bucket_target_sys { pub mod bucket_target_sys {
pub use crate::bucket::bucket_target_sys::{ pub use crate::bucket::bucket_target_sys::{
AdvancedPutOptions, BucketTargetError, BucketTargetSys, PutObjectOptions, RemoveObjectOptions, S3ClientError, AdvancedPutOptions, BucketTargetError, BucketTargetSys, PutObjectOptions, RemoveObjectOptions, S3ClientError,
TargetClient, append_version_id_query, SsecPassthroughCapability, TargetClient, append_version_id_query,
}; };
} }
+143 -3
View File
@@ -299,9 +299,51 @@ struct TargetClientBuildProbe {
release: Arc<tokio::sync::Semaphore>, release: Arc<tokio::sync::Semaphore>,
} }
/// Whether a replication target preserves the SSE-C passthrough transport
/// headers (`X-Rustfs-Replication-*`) end to end.
///
/// A target that silently drops those headers (MinIO, generic S3) stores the
/// forwarded ciphertext without its decryption material — an unreadable
/// replica that used to report COMPLETED. The replication worker audits the
/// first passthrough PUT per target (HEAD-back for SSE-C evidence) and caches
/// the verdict here; a fresh `Unsupported` fails SSE-C replication closed
/// before any PUT is sent. Entries follow the `arn_remotes_map` lifecycle
/// (rebuilding or removing a target resets its capability to `Unknown`) and
/// additionally expire after [`SSEC_PASSTHROUGH_CAPABILITY_TTL`], after which
/// the next attempt re-audits.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub enum SsecPassthroughCapability {
#[default]
Unknown,
Supported,
Unsupported,
}
/// How long an audited SSE-C passthrough verdict stays authoritative.
///
/// Trade-off: without a TTL a verdict is sticky for the process lifetime —
/// an `Unsupported` target that gets upgraded (or re-probed only via
/// replication-check) would keep failing SSE-C replication forever, and the
/// fail-open twin: a `Supported` verdict would outlive a backend swapped
/// behind the same endpoint/ARN. With the TTL, a bad target costs at most
/// one wasted PUT+HEAD audit per TTL window, and a changed backend is
/// re-discovered within the same window.
pub const SSEC_PASSTHROUGH_CAPABILITY_TTL: Duration = Duration::from_secs(10 * 60);
/// A recorded SSE-C passthrough verdict plus when it was recorded, so reads
/// can report staleness against [`SSEC_PASSTHROUGH_CAPABILITY_TTL`].
#[derive(Debug, Clone, Copy)]
struct SsecPassthroughRecord {
capability: SsecPassthroughCapability,
recorded_at: Instant,
}
#[derive(Debug, Default)] #[derive(Debug, Default)]
pub struct BucketTargetSys { pub struct BucketTargetSys {
pub arn_remotes_map: Arc<RwLock<HashMap<String, ArnTarget>>>, pub arn_remotes_map: Arc<RwLock<HashMap<String, ArnTarget>>>,
/// SSE-C passthrough capability verdicts keyed by target ARN. See
/// [`SsecPassthroughCapability`]; reset alongside `arn_remotes_map`.
ssec_passthrough_map: Arc<RwLock<HashMap<String, SsecPassthroughRecord>>>,
pub targets_map: Arc<RwLock<HashMap<String, Vec<BucketTarget>>>>, pub targets_map: Arc<RwLock<HashMap<String, Vec<BucketTarget>>>>,
pub h_mutex: Arc<RwLock<HashMap<String, EpHealth>>>, pub h_mutex: Arc<RwLock<HashMap<String, EpHealth>>>,
target_h_mutex: Arc<RwLock<HashMap<String, EpHealth>>>, target_h_mutex: Arc<RwLock<HashMap<String, EpHealth>>>,
@@ -322,6 +364,7 @@ impl BucketTargetSys {
fn new() -> Self { fn new() -> Self {
Self { Self {
arn_remotes_map: Arc::new(RwLock::new(HashMap::new())), arn_remotes_map: Arc::new(RwLock::new(HashMap::new())),
ssec_passthrough_map: Arc::new(RwLock::new(HashMap::new())),
targets_map: Arc::new(RwLock::new(HashMap::new())), targets_map: Arc::new(RwLock::new(HashMap::new())),
h_mutex: Arc::new(RwLock::new(HashMap::new())), h_mutex: Arc::new(RwLock::new(HashMap::new())),
target_h_mutex: Arc::new(RwLock::new(HashMap::new())), target_h_mutex: Arc::new(RwLock::new(HashMap::new())),
@@ -585,19 +628,59 @@ impl BucketTargetSys {
let update_mutex = self.target_update_mutex(bucket).await; let update_mutex = self.target_update_mutex(bucket).await;
let _update_guard = update_mutex.lock().await; let _update_guard = update_mutex.lock().await;
// Lock order: targets_map, then arn_remotes_map, then target_h_mutex. // Lock order: targets_map, then arn_remotes_map, then target_h_mutex,
// then ssec_passthrough_map (always last; also taken standalone by the
// capability accessors).
let mut targets_map = self.targets_map.write().await; let mut targets_map = self.targets_map.write().await;
let mut arn_remotes_map = self.arn_remotes_map.write().await; let mut arn_remotes_map = self.arn_remotes_map.write().await;
let mut health_map = self.target_h_mutex.write().await; let mut health_map = self.target_h_mutex.write().await;
if let Some(targets) = targets_map.remove(bucket) { if let Some(targets) = targets_map.remove(bucket) {
let mut ssec_map = self.ssec_passthrough_map.write().await;
for target in targets { for target in targets {
arn_remotes_map.remove(&target.arn); arn_remotes_map.remove(&target.arn);
health_map.remove(&target.arn); health_map.remove(&target.arn);
ssec_map.remove(&target.arn);
} }
} }
} }
/// Cached SSE-C passthrough capability for a target ARN, plus whether the
/// verdict is older than [`SSEC_PASSTHROUGH_CAPABILITY_TTL`]. `(Unknown,
/// false)` when no verdict has been recorded since the target was built.
/// Staleness is computed here so the gate policy stays a pure function.
pub async fn ssec_passthrough_capability(&self, arn: &str) -> (SsecPassthroughCapability, bool) {
match self.ssec_passthrough_map.read().await.get(arn) {
Some(record) => (record.capability, record.recorded_at.elapsed() >= SSEC_PASSTHROUGH_CAPABILITY_TTL),
None => (SsecPassthroughCapability::Unknown, false),
}
}
/// Record an audited SSE-C passthrough verdict for a target ARN. Written by
/// the replication worker's HEAD-back audit and by the replication-check
/// SsecPassthrough probe phase.
pub async fn record_ssec_passthrough_capability(&self, arn: &str, capability: SsecPassthroughCapability) {
self.ssec_passthrough_map.write().await.insert(
arn.to_string(),
SsecPassthroughRecord {
capability,
recorded_at: Instant::now(),
},
);
}
/// Test hook: age an existing verdict so TTL expiry is observable without
/// waiting out the real window.
#[cfg(test)]
pub(crate) async fn backdate_ssec_passthrough_capability(&self, arn: &str, age: Duration) {
let backdated = Instant::now()
.checked_sub(age)
.expect("system uptime must exceed the backdate age");
if let Some(record) = self.ssec_passthrough_map.write().await.get_mut(arn) {
record.recorded_at = backdated;
}
}
pub async fn set_target( pub async fn set_target(
&self, &self,
bucket: &str, bucket: &str,
@@ -953,15 +1036,21 @@ impl BucketTargetSys {
} }
} }
// Lock order: targets_map, then arn_remotes_map, then target_h_mutex. // Lock order: targets_map, then arn_remotes_map, then target_h_mutex,
// then ssec_passthrough_map (always last; also taken standalone by the
// capability accessors).
let mut targets_map = self.targets_map.write().await; let mut targets_map = self.targets_map.write().await;
let mut arn_remotes_map = self.arn_remotes_map.write().await; let mut arn_remotes_map = self.arn_remotes_map.write().await;
let mut health_map = self.target_h_mutex.write().await; let mut health_map = self.target_h_mutex.write().await;
// Remove existing targets // Remove existing targets
if let Some(existing_targets) = targets_map.remove(bucket) { if let Some(existing_targets) = targets_map.remove(bucket) {
let mut ssec_map = self.ssec_passthrough_map.write().await;
for target in existing_targets { for target in existing_targets {
arn_remotes_map.remove(&target.arn); arn_remotes_map.remove(&target.arn);
health_map.remove(&target.arn); health_map.remove(&target.arn);
// A rebuilt/edited target may point at a different service:
// the SSE-C passthrough verdict must be re-audited from Unknown.
ssec_map.remove(&target.arn);
self.update_bandwidth_limit(bucket, &target.arn, 0); self.update_bandwidth_limit(bucket, &target.arn, 0);
} }
} }
@@ -1455,7 +1544,7 @@ fn resolve_put_api_version_id(source_version_id: &str) -> Option<&str> {
/// RustFS represents the null version internally as the nil UUID while the S3 /// RustFS represents the null version internally as the nil UUID while the S3
/// API addresses it as the literal "null" (same mapping as /// API addresses it as the literal "null" (same mapping as
/// [`resolve_put_api_version_id`]); empty means "no version requested". /// [`resolve_put_api_version_id`]); empty means "no version requested".
fn resolve_read_api_version_id(version_id: Option<String>) -> Option<String> { pub(crate) fn resolve_read_api_version_id(version_id: Option<String>) -> Option<String> {
let version_id = version_id?; let version_id = version_id?;
let trimmed = version_id.trim(); let trimmed = version_id.trim();
if trimmed.is_empty() { if trimmed.is_empty() {
@@ -2676,6 +2765,57 @@ mod tests {
assert_eq!(health.last_online, Some(now)); assert_eq!(health.last_online, Some(now));
} }
/// N2 TTL contract, both flip directions: a recorded verdict is fresh
/// until [`SSEC_PASSTHROUGH_CAPABILITY_TTL`], then reads as expired; a
/// re-audit that records the OPPOSITE verdict replaces it as fresh. The
/// worker gate maps expired verdicts to ProceedWithAudit (pinned in
/// `replication_target_boundary`), so together this proves an Unsupported
/// target recovers to Supported through the audit once its verdict ages
/// out — and a stale Supported one is re-proven rather than trusted.
#[tokio::test]
async fn ssec_passthrough_capability_ttl_expires_and_reaudit_flips_verdict() {
let sys = BucketTargetSys::default();
let arn = "arn:rustfs:replication:us-east-1:bucket:ssec-ttl";
let expired_age = SSEC_PASSTHROUGH_CAPABILITY_TTL + Duration::from_secs(1);
assert_eq!(
sys.ssec_passthrough_capability(arn).await,
(SsecPassthroughCapability::Unknown, false),
"an unrecorded target must read Unknown and never expired"
);
sys.record_ssec_passthrough_capability(arn, SsecPassthroughCapability::Unsupported)
.await;
assert_eq!(
sys.ssec_passthrough_capability(arn).await,
(SsecPassthroughCapability::Unsupported, false)
);
sys.backdate_ssec_passthrough_capability(arn, expired_age).await;
assert_eq!(
sys.ssec_passthrough_capability(arn).await,
(SsecPassthroughCapability::Unsupported, true),
"an aged-out Unsupported verdict must read expired so the gate re-audits"
);
// The re-audit against an upgraded target records Supported afresh.
sys.record_ssec_passthrough_capability(arn, SsecPassthroughCapability::Supported)
.await;
assert_eq!(
sys.ssec_passthrough_capability(arn).await,
(SsecPassthroughCapability::Supported, false),
"a fresh Supported verdict replaces the expired Unsupported one"
);
// And the fail-open twin: Supported also ages out.
sys.backdate_ssec_passthrough_capability(arn, expired_age).await;
assert_eq!(
sys.ssec_passthrough_capability(arn).await,
(SsecPassthroughCapability::Supported, true),
"an aged-out Supported verdict must read expired so the gate re-proves it"
);
}
#[tokio::test] #[tokio::test]
async fn list_targets_applies_health_stats_by_arn_and_preserves_endpoint_port() { async fn list_targets_applies_health_stats_by_arn_and_preserves_endpoint_port() {
let sys = BucketTargetSys::default(); let sys = BucketTargetSys::default();
@@ -49,10 +49,12 @@ use super::replication_storage_boundary::{
ReplicationDeletedObject, ReplicationObjectIO, ReplicationStorage, StatObjectOptions, StorageObjectInfoOrErr, WalkOptions, ReplicationDeletedObject, ReplicationObjectIO, ReplicationStorage, StatObjectOptions, StorageObjectInfoOrErr, WalkOptions,
}; };
use super::replication_target_boundary::{ use super::replication_target_boundary::{
PutObjectOptions, PutObjectPartOptions, ReplicationTargetStore, TargetClient, replication_action_for_target_head, ERR_REPLICATION_SSEC_PASSTHROUGH_UNSUPPORTED, PutObjectOptions, PutObjectPartOptions, ReplicationTargetStore,
SsecPassthroughCapability, SsecPassthroughGate, TargetClient, replication_action_for_target_head,
replication_complete_multipart_options, replication_delete_marker_purge_remove_options, replication_delete_remove_options, replication_complete_multipart_options, replication_delete_marker_purge_remove_options, replication_delete_remove_options,
replication_force_delete_remove_options, replication_object_is_ssec_encrypted, replication_put_object_header_size, replication_force_delete_remove_options, replication_object_is_ssec_encrypted, replication_put_object_header_size,
replication_put_object_options, replication_target_head_is_newer_null_version, replication_put_object_options, replication_target_head_is_newer_null_version, resolve_read_api_version_id,
ssec_passthrough_evidence_present, ssec_passthrough_gate,
}; };
use super::replication_versioning_boundary::ReplicationVersioningStore; use super::replication_versioning_boundary::ReplicationVersioningStore;
use super::runtime_boundary as runtime_sources; use super::runtime_boundary as runtime_sources;
@@ -279,6 +281,114 @@ async fn head_object_fallback(
} }
} }
/// Resolve the N2 fail-closed gate for an SSE-C passthrough attempt against
/// this target. Returns `Some(audit_required)` when replication may proceed;
/// on a freshly-flagged header-dropping target it settles `rinfo` as FAILED
/// (no PUT is ever sent — the object stays on the normal MRF retry channel
/// and re-audits once the verdict's TTL expires or replication-check
/// re-probes the target) and returns `None`.
async fn resolve_ssec_passthrough_gate(
ssec: bool,
tgt_client: &TargetClient,
bucket: &str,
object: &str,
rinfo: &mut ReplicatedTargetInfo,
) -> Option<bool> {
let (capability, expired) = ReplicationTargetStore::ssec_passthrough_capability(&tgt_client.arn).await;
match ssec_passthrough_gate(ssec, capability, expired) {
SsecPassthroughGate::Proceed => Some(false),
SsecPassthroughGate::ProceedWithAudit => Some(true),
SsecPassthroughGate::FailClosed => {
rinfo.replication_status = ReplicationStatusType::Failed;
rinfo.error = Some(ERR_REPLICATION_SSEC_PASSTHROUGH_UNSUPPORTED.to_string());
warn!(
event = EVENT_RESYNC_TARGET_OPERATION_FAILED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
bucket = %bucket,
object = %object,
arn = %tgt_client.arn,
operation = "ssec_passthrough_gate",
error = ERR_REPLICATION_SSEC_PASSTHROUGH_UNSUPPORTED,
"Replication target operation failed"
);
None
}
}
}
/// Judge SSE-C passthrough evidence on a HEAD of the replica and record the
/// capability verdict for the target. Returns true when the SSE-C material
/// provably survived; otherwise records `Unsupported` and settles `rinfo` as
/// FAILED so the attempt never reports a silently unreadable COMPLETED.
async fn settle_ssec_passthrough_evidence(
head: &HeadObjectOutput,
tgt_client: &TargetClient,
bucket: &str,
object: &str,
rinfo: &mut ReplicatedTargetInfo,
) -> bool {
if ssec_passthrough_evidence_present(head) {
ReplicationTargetStore::record_ssec_passthrough_capability(&tgt_client.arn, SsecPassthroughCapability::Supported).await;
return true;
}
ReplicationTargetStore::record_ssec_passthrough_capability(&tgt_client.arn, SsecPassthroughCapability::Unsupported).await;
rinfo.replication_status = ReplicationStatusType::Failed;
rinfo.error = Some(ERR_REPLICATION_SSEC_PASSTHROUGH_UNSUPPORTED.to_string());
warn!(
event = EVENT_RESYNC_TARGET_OPERATION_FAILED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
bucket = %bucket,
object = %object,
arn = %tgt_client.arn,
endpoint = %tgt_client.endpoint,
operation = "ssec_passthrough_audit",
error = ERR_REPLICATION_SSEC_PASSTHROUGH_UNSUPPORTED,
"Replication target operation failed"
);
false
}
/// Post-PUT HEAD-back audit for an SSE-C passthrough replica, over the worker
/// HEAD channel (replication-check exemption plus the `source-proxy-request:
/// false` suppression header, so the target answers locally without a
/// customer key). A HEAD transport failure leaves the capability `Unknown`
/// but still fails this attempt: an unverifiable SSE-C replica must not
/// report COMPLETED.
async fn audit_ssec_passthrough_replica(
tgt_client: &Arc<TargetClient>,
bucket: &str,
object: &str,
version_id: Option<String>,
rinfo: &mut ReplicatedTargetInfo,
) -> bool {
// Address the replica the way the PUT named it: a nil source version id
// (versioning-suspended / null-version objects) maps to the "null"
// version, so the audit HEAD does not 4xx-loop on those objects.
let version_id = resolve_read_api_version_id(version_id);
match head_object_for_worker(tgt_client.as_ref(), &tgt_client.bucket, object, version_id).await {
Ok(head) => settle_ssec_passthrough_evidence(&head, tgt_client, bucket, object, rinfo).await,
Err(e) => {
rinfo.replication_status = ReplicationStatusType::Failed;
rinfo.error = Some(format!("SSE-C passthrough audit HEAD failed: {e}"));
warn!(
event = EVENT_RESYNC_TARGET_OPERATION_FAILED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
bucket = %bucket,
object = %object,
arn = %tgt_client.arn,
operation = "ssec_passthrough_audit_head",
error = %e,
"Replication target operation failed"
);
mark_replication_target_offline_if_needed(tgt_client, &e).await;
false
}
}
}
static RESYNC_WORKER_COUNT: usize = 10; static RESYNC_WORKER_COUNT: usize = 10;
fn resync_status_duration( fn resync_status_duration(
@@ -2872,6 +2982,21 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
return rinfo; return rinfo;
} }
// N2 fail-closed: never PUT SSE-C ciphertext at a target known to drop
// the passthrough transport headers, and never trust a convergence HEAD
// against such a target — a previous broken replica matches by ETag.
let Some(ssec_audit_required) = resolve_ssec_passthrough_gate(self.ssec, &tgt_client, &bucket, &object, &mut rinfo).await
else {
send_local_event(EventArgs {
event_name: EventName::ObjectReplicationNotTracked.to_string(),
bucket_name: bucket.clone(),
object: self.to_object_info(),
user_agent: "Internal: [Replication]".to_string(),
..Default::default()
});
return rinfo;
};
let versioned = ReplicationVersioningStore::prefix_enabled(&bucket, &object).await; let versioned = ReplicationVersioningStore::prefix_enabled(&bucket, &object).await;
let version_suspended = ReplicationVersioningStore::prefix_suspended(&bucket, &object).await; let version_suspended = ReplicationVersioningStore::prefix_suspended(&bucket, &object).await;
@@ -2975,6 +3100,14 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
Ok(oi) => { Ok(oi) => {
replication_action = replication_action_for_target_head(&object_info, &oi, self.op_type); replication_action = replication_action_for_target_head(&object_info, &oi, self.op_type);
if replication_action == ReplicationAction::None { if replication_action == ReplicationAction::None {
// An SSE-C replica only counts as converged when the same
// HEAD proves its decryption material survived; a broken
// ciphertext copy from an earlier attempt matches by ETag.
if ssec_audit_required
&& !settle_ssec_passthrough_evidence(&oi, &tgt_client, &bucket, &object, &mut rinfo).await
{
return rinfo;
}
rinfo.replication_status = ReplicationStatusType::Completed; rinfo.replication_status = ReplicationStatusType::Completed;
rinfo.replication_resynced = true; rinfo.replication_resynced = true;
rinfo.replication_action = ReplicationAction::None; rinfo.replication_action = ReplicationAction::None;
@@ -2989,6 +3122,11 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
// Version-ID format mismatch: retry without versionId and compare ETags. // Version-ID format mismatch: retry without versionId and compare ETags.
match head_object_fallback(&tgt_client, &object).await { match head_object_fallback(&tgt_client, &object).await {
Ok(Some(oi)) if replication_etags_match(object_info.etag.as_deref(), oi.e_tag.as_deref()) => { Ok(Some(oi)) if replication_etags_match(object_info.etag.as_deref(), oi.e_tag.as_deref()) => {
if ssec_audit_required
&& !settle_ssec_passthrough_evidence(&oi, &tgt_client, &bucket, &object, &mut rinfo).await
{
return rinfo;
}
rinfo.replication_status = ReplicationStatusType::Completed; rinfo.replication_status = ReplicationStatusType::Completed;
rinfo.replication_resynced = true; rinfo.replication_resynced = true;
rinfo.replication_action = ReplicationAction::None; rinfo.replication_action = ReplicationAction::None;
@@ -3113,6 +3251,15 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
return rinfo; return rinfo;
} }
// First SSE-C passthrough PUT against this target: verify the replica
// kept its decryption material before reporting COMPLETED.
if ssec_audit_required
&& !audit_ssec_passthrough_replica(&tgt_client, &bucket, &object, self.version_id.map(|v| v.to_string()), &mut rinfo)
.await
{
return rinfo;
}
rinfo.replication_status = ReplicationStatusType::Completed; rinfo.replication_status = ReplicationStatusType::Completed;
rinfo rinfo
@@ -3135,6 +3282,21 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
return rinfo; return rinfo;
} }
// N2 fail-closed: see the gate in `replicate_object` — the same policy
// applies to the metadata/existing-object transport.
let Some(ssec_audit_required) = resolve_ssec_passthrough_gate(self.ssec, &tgt_client, &bucket, &object, &mut rinfo).await
else {
send_local_event(EventArgs {
event_name: EventName::ObjectReplicationNotTracked.to_string(),
bucket_name: bucket.clone(),
object: self.to_object_info(),
user_agent: "Internal: [Replication]".to_string(),
..Default::default()
});
rinfo.duration = (OffsetDateTime::now_utc() - start_time).unsigned_abs();
return rinfo;
};
let versioned = ReplicationVersioningStore::prefix_enabled(&bucket, &object).await; let versioned = ReplicationVersioningStore::prefix_enabled(&bucket, &object).await;
let version_suspended = ReplicationVersioningStore::prefix_suspended(&bucket, &object).await; let version_suspended = ReplicationVersioningStore::prefix_suspended(&bucket, &object).await;
@@ -3173,8 +3335,19 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
let _sopts = replicate_all_stat_options(&object_info, &bucket, &tgt_client); let _sopts = replicate_all_stat_options(&object_info, &bucket, &tgt_client);
let Some((replication_action, object_info)) = let Some((replication_action, object_info)) = resolve_replicate_all_action(
resolve_replicate_all_action(self, &tgt_client, &bucket, &object, object_info, start_time, &mut rinfo).await ReplicateAllActionContext {
roi: self,
tgt_client: &tgt_client,
bucket: &bucket,
object: &object,
start_time,
ssec_audit_required,
},
object_info,
&mut rinfo,
)
.await
else { else {
return rinfo; return rinfo;
}; };
@@ -3229,6 +3402,16 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
return rinfo; return rinfo;
} }
// First SSE-C passthrough PUT against this target: verify the replica
// kept its decryption material before reporting COMPLETED.
if ssec_audit_required
&& !audit_ssec_passthrough_replica(&tgt_client, &bucket, &object, self.version_id.map(|v| v.to_string()), &mut rinfo)
.await
{
rinfo.duration = (OffsetDateTime::now_utc() - start_time).unsigned_abs();
return rinfo;
}
rinfo rinfo
} }
@@ -3441,25 +3624,49 @@ fn apply_replication_resync_timestamp(rinfo: &mut ReplicatedTargetInfo, reset_id
rinfo.replication_resynced = true; rinfo.replication_resynced = true;
} }
/// Borrowed inputs for [`resolve_replicate_all_action`].
struct ReplicateAllActionContext<'a> {
roi: &'a ReplicateObjectInfo,
tgt_client: &'a Arc<TargetClient>,
bucket: &'a str,
object: &'a str,
start_time: OffsetDateTime,
/// N2: the target's SSE-C passthrough capability is still `Unknown`, so a
/// converged-looking replica must additionally prove its SSE-C material
/// survived before the comparison may settle COMPLETED.
ssec_audit_required: bool,
}
/// Compare the source object against the target via HEAD and decide which /// Compare the source object against the target via HEAD and decide which
/// replication action is still required. Returns `None` after fully settling /// replication action is still required. Returns `None` after fully settling
/// `rinfo` when replication must stop here — either because the target already /// `rinfo` when replication must stop here — either because the target already
/// matches or because the comparison failed. /// matches or because the comparison failed.
async fn resolve_replicate_all_action( async fn resolve_replicate_all_action(
roi: &ReplicateObjectInfo, ctx: ReplicateAllActionContext<'_>,
tgt_client: &Arc<TargetClient>,
bucket: &str,
object: &str,
object_info: ObjectInfo, object_info: ObjectInfo,
start_time: OffsetDateTime,
rinfo: &mut ReplicatedTargetInfo, rinfo: &mut ReplicatedTargetInfo,
) -> Option<(ReplicationAction, ObjectInfo)> { ) -> Option<(ReplicationAction, ObjectInfo)> {
let ReplicateAllActionContext {
roi,
tgt_client,
bucket,
object,
start_time,
ssec_audit_required,
} = ctx;
let replication_action; let replication_action;
match head_object_for_worker(tgt_client.as_ref(), &tgt_client.bucket, object, roi.version_id.map(|v| v.to_string())).await { match head_object_for_worker(tgt_client.as_ref(), &tgt_client.bucket, object, roi.version_id.map(|v| v.to_string())).await {
Ok(oi) => { Ok(oi) => {
replication_action = replication_action_for_target_head(&object_info, &oi, roi.op_type); replication_action = replication_action_for_target_head(&object_info, &oi, roi.op_type);
rinfo.replication_status = ReplicationStatusType::Completed; rinfo.replication_status = ReplicationStatusType::Completed;
if replication_action == ReplicationAction::None { if replication_action == ReplicationAction::None {
// An SSE-C replica only counts as converged when the same HEAD
// proves its decryption material survived; a broken ciphertext
// copy from an earlier attempt matches by ETag.
if ssec_audit_required && !settle_ssec_passthrough_evidence(&oi, tgt_client, bucket, object, rinfo).await {
rinfo.duration = (OffsetDateTime::now_utc() - start_time).unsigned_abs();
return None;
}
if roi.op_type == ReplicationType::ExistingObject if roi.op_type == ReplicationType::ExistingObject
&& replication_target_head_is_newer_null_version(&object_info, &oi) && replication_target_head_is_newer_null_version(&object_info, &oi)
{ {
@@ -3509,6 +3716,12 @@ async fn resolve_replicate_all_action(
match head_object_fallback(tgt_client, object).await { match head_object_fallback(tgt_client, object).await {
Ok(Some(oi)) => { Ok(Some(oi)) => {
replication_action = if replication_etags_match(object_info.etag.as_deref(), oi.e_tag.as_deref()) { replication_action = if replication_etags_match(object_info.etag.as_deref(), oi.e_tag.as_deref()) {
if ssec_audit_required
&& !settle_ssec_passthrough_evidence(&oi, tgt_client, bucket, object, rinfo).await
{
rinfo.duration = (OffsetDateTime::now_utc() - start_time).unsigned_abs();
return None;
}
ReplicationAction::None ReplicationAction::None
} else { } else {
ReplicationAction::All ReplicationAction::All
@@ -36,7 +36,8 @@ use time::OffsetDateTime;
use time::format_description::well_known::Rfc3339; use time::format_description::well_known::Rfc3339;
pub(crate) use crate::bucket::bucket_target_sys::{ pub(crate) use crate::bucket::bucket_target_sys::{
AdvancedPutOptions, PutObjectOptions, PutObjectPartOptions, RemoveObjectOptions, TargetClient, AdvancedPutOptions, PutObjectOptions, PutObjectPartOptions, RemoveObjectOptions, SsecPassthroughCapability, TargetClient,
resolve_read_api_version_id,
}; };
#[cfg(test)] #[cfg(test)]
pub(crate) use crate::bucket::target::BucketTarget; pub(crate) use crate::bucket::target::BucketTarget;
@@ -65,6 +66,8 @@ static STANDARD_HEADERS: &[&str] = &[
]; ];
const ERR_REPLICATION_ENCRYPTION_METADATA_UNSUPPORTED: &str = "replication source contains unsupported encryption metadata"; const ERR_REPLICATION_ENCRYPTION_METADATA_UNSUPPORTED: &str = "replication source contains unsupported encryption metadata";
pub(crate) const ERR_REPLICATION_SSEC_PASSTHROUGH_UNSUPPORTED: &str = "replication target does not support SSE-C passthrough: the replica would lose its decryption material \
(run ?replication-check to re-probe)";
#[derive(Debug, Clone, Copy, PartialEq, Eq)] #[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum ReplicationSourceEncryption { enum ReplicationSourceEncryption {
@@ -146,6 +149,54 @@ pub(crate) fn replication_object_is_ssec_encrypted(user_defined: &HashMap<String
rustfs_replication::is_ssec_encrypted(user_defined) rustfs_replication::is_ssec_encrypted(user_defined)
} }
/// Fail-closed decision for an SSE-C passthrough replication attempt, derived
/// from the target's cached [`SsecPassthroughCapability`]. Pure so the policy
/// can migrate with the worker (M2) without dragging the cache along; the
/// caller computes `expired` from the cache record's age (see
/// `SSEC_PASSTHROUGH_CAPABILITY_TTL`).
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum SsecPassthroughGate {
/// Not an SSE-C object, or the target has a fresh proof that it preserves
/// the passthrough transport headers: replicate without a HEAD-back audit.
Proceed,
/// No usable verdict — first SSE-C attempt since the target was (re)built,
/// or the recorded verdict (in either direction) aged out: PUT, then HEAD
/// the replica back and require SSE-C evidence before reporting COMPLETED.
ProceedWithAudit,
/// The target was recently proven to drop the passthrough headers: do not
/// send the PUT, report FAILED (the object stays on the normal MRF retry
/// channel and re-audits once the verdict expires).
FailClosed,
}
pub(crate) fn ssec_passthrough_gate(ssec: bool, capability: SsecPassthroughCapability, expired: bool) -> SsecPassthroughGate {
if !ssec {
return SsecPassthroughGate::Proceed;
}
// An expired verdict — Supported or Unsupported — must be re-earned: a
// stale Unsupported would otherwise stick forever after a target upgrade,
// and a stale Supported would fail open after a backend swap behind the
// same endpoint.
if expired {
return SsecPassthroughGate::ProceedWithAudit;
}
match capability {
SsecPassthroughCapability::Supported => SsecPassthroughGate::Proceed,
SsecPassthroughCapability::Unknown => SsecPassthroughGate::ProceedWithAudit,
SsecPassthroughCapability::Unsupported => SsecPassthroughGate::FailClosed,
}
}
/// True when a replication-check HEAD of the replica proves the SSE-C
/// material survived passthrough: a RustFS target restores the transport
/// headers into the stored SSE-C keys and its HEAD echoes
/// `x-amz-server-side-encryption-customer-algorithm` (the replication-check
/// exemption skips key validation but not the metadata echo). A target that
/// dropped the headers stored a plain object and echoes nothing.
pub(crate) fn ssec_passthrough_evidence_present(head: &HeadObjectOutput) -> bool {
head.sse_customer_algorithm.as_deref().is_some_and(|algo| !algo.is_empty())
}
pub(crate) struct ReplicationTargetStore; pub(crate) struct ReplicationTargetStore;
impl ReplicationTargetStore { impl ReplicationTargetStore {
@@ -165,6 +216,17 @@ impl ReplicationTargetStore {
BucketTargetSys::get().mark_target_offline(target_client).await BucketTargetSys::get().mark_target_offline(target_client).await
} }
/// Returns the cached verdict and whether it has outlived its TTL.
pub(crate) async fn ssec_passthrough_capability(arn: &str) -> (SsecPassthroughCapability, bool) {
BucketTargetSys::get().ssec_passthrough_capability(arn).await
}
pub(crate) async fn record_ssec_passthrough_capability(arn: &str, capability: SsecPassthroughCapability) {
BucketTargetSys::get()
.record_ssec_passthrough_capability(arn, capability)
.await
}
#[cfg(test)] #[cfg(test)]
pub(crate) async fn register_test_target(target_client: &Arc<TargetClient>) { pub(crate) async fn register_test_target(target_client: &Arc<TargetClient>) {
BucketTargetSys::get().arn_remotes_map.write().await.insert( BucketTargetSys::get().arn_remotes_map.write().await.insert(
@@ -898,6 +960,71 @@ mod tests {
} }
} }
/// N2 fail-closed policy: SSE-C replication may only proceed silently
/// against a target with a FRESH proof that it preserves the passthrough
/// transport headers. Unknown targets must be audited; freshly-flagged
/// dropping targets must never receive the PUT; an expired verdict in
/// EITHER direction must be re-earned through the audit — a sticky
/// Unsupported would outlive a target upgrade, and a sticky Supported
/// would fail open after a backend swap behind the same endpoint.
#[test]
fn ssec_passthrough_gate_is_fail_closed_and_ttl_bounded() {
for capability in [
SsecPassthroughCapability::Unknown,
SsecPassthroughCapability::Supported,
SsecPassthroughCapability::Unsupported,
] {
for expired in [false, true] {
assert_eq!(
ssec_passthrough_gate(false, capability, expired),
SsecPassthroughGate::Proceed,
"non-SSE-C objects must never be gated on the passthrough capability"
);
}
}
assert_eq!(
ssec_passthrough_gate(true, SsecPassthroughCapability::Supported, false),
SsecPassthroughGate::Proceed
);
assert_eq!(
ssec_passthrough_gate(true, SsecPassthroughCapability::Unknown, false),
SsecPassthroughGate::ProceedWithAudit
);
assert_eq!(
ssec_passthrough_gate(true, SsecPassthroughCapability::Unsupported, false),
SsecPassthroughGate::FailClosed
);
// Expiry flips both directions back to the audit.
assert_eq!(
ssec_passthrough_gate(true, SsecPassthroughCapability::Unsupported, true),
SsecPassthroughGate::ProceedWithAudit,
"an expired Unsupported verdict must allow a re-audit (upgraded target recovers without operator action)"
);
assert_eq!(
ssec_passthrough_gate(true, SsecPassthroughCapability::Supported, true),
SsecPassthroughGate::ProceedWithAudit,
"an expired Supported verdict must be re-proven (backend swap behind the same endpoint must not fail open)"
);
}
#[test]
fn ssec_passthrough_evidence_requires_customer_algorithm_echo() {
let with_evidence = HeadObjectOutput::builder().sse_customer_algorithm("AES256").build();
assert!(ssec_passthrough_evidence_present(&with_evidence));
let empty_algorithm = HeadObjectOutput::builder().sse_customer_algorithm("").build();
assert!(
!ssec_passthrough_evidence_present(&empty_algorithm),
"an empty echo is not evidence of preserved SSE-C material"
);
let without_evidence = HeadObjectOutput::builder().e_tag("\"abc\"").content_length(8).build();
assert!(
!ssec_passthrough_evidence_present(&without_evidence),
"a plain HEAD response must classify the target as having dropped the material"
);
}
#[test] #[test]
fn replication_put_options_adds_ssec_checksum_metadata() { fn replication_put_options_adds_ssec_checksum_metadata() {
let metadata = HashMap::from([(SSEC_ALGORITHM_HEADER.to_string(), "AES256".to_string())]); let metadata = HashMap::from([(SSEC_ALGORITHM_HEADER.to_string(), "AES256".to_string())]);
+262 -8
View File
@@ -17,7 +17,8 @@ use super::storage_api::bucket::metadata_sys;
use super::storage_api::bucket::replication::{self, BucketReplicationResyncStatus, BucketStats, ReplicationStatusType}; use super::storage_api::bucket::replication::{self, BucketReplicationResyncStatus, BucketStats, ReplicationStatusType};
use super::storage_api::bucket::target::{BucketTarget, BucketTargetType, BucketTargets}; use super::storage_api::bucket::target::{BucketTarget, BucketTargetType, BucketTargets};
use super::storage_api::bucket::target_sys::{ use super::storage_api::bucket::target_sys::{
BucketTargetSys, PutObjectOptions, RemoveObjectOptions, S3ClientError, TargetClient, append_version_id_query, BucketTargetSys, PutObjectOptions, RemoveObjectOptions, S3ClientError, SsecPassthroughCapability, TargetClient,
append_version_id_query,
}; };
use super::storage_api::bucket::versioning_sys::BucketVersioningSys; use super::storage_api::bucket::versioning_sys::BucketVersioningSys;
use super::storage_api::bucket::{AdminReplicationConfigExt as _, AdminVersioningConfigExt as _}; use super::storage_api::bucket::{AdminReplicationConfigExt as _, AdminVersioningConfigExt as _};
@@ -67,6 +68,9 @@ use rustfs_policy::policy::action::{Action, S3Action};
use rustfs_s3_types::EventName; use rustfs_s3_types::EventName;
use rustfs_signer::pre_sign_v4; use rustfs_signer::pre_sign_v4;
use rustfs_utils::egress::{OutboundDnsResolver, OutboundPolicy}; use rustfs_utils::egress::{OutboundDnsResolver, OutboundPolicy};
use rustfs_utils::http::object_encryption_keys::{
REPLICATION_SSEC_ALGORITHM_HEADER, REPLICATION_SSEC_KEY_MD5_HEADER, REPLICATION_SSEC_ORIGINAL_SIZE_HEADER,
};
use rustfs_utils::http::{ use rustfs_utils::http::{
SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_CHECK, SUFFIX_SOURCE_REPLICATION_REQUEST, SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_CHECK, SUFFIX_SOURCE_REPLICATION_REQUEST,
SUFFIX_SOURCE_VERSION_ID, get_source_scheme, insert_header, SUFFIX_SOURCE_VERSION_ID, get_source_scheme, insert_header,
@@ -210,6 +214,13 @@ const REPLICATION_CHECK_ERROR_MAX_BYTES: usize = 512;
/// RustFS extension code (no madmin analogue): the target does not adopt the /// RustFS extension code (no madmin analogue): the target does not adopt the
/// source version id, breaking the version-identity replication contract. /// source version id, breaking the version-identity replication contract.
const REPLICATION_CHECK_CODE_VERSION_MISMATCH: &str = "BucketRemoteTargetVersionMismatch"; const REPLICATION_CHECK_CODE_VERSION_MISMATCH: &str = "BucketRemoteTargetVersionMismatch";
/// RustFS extension code (no madmin analogue): the target drops the
/// `X-Rustfs-Replication-*` SSE-C passthrough headers, so an SSE-C replica
/// would lose its decryption material (N2 fail-closed).
const REPLICATION_CHECK_CODE_SSEC_PASSTHROUGH: &str = "BucketRemoteSsecPassthroughUnsupported";
/// Syntactically valid stand-in SSE-C key MD5 for the passthrough probe (the
/// probe object is never decrypted; it only has to round-trip the metadata).
const REPLICATION_CHECK_SSEC_PROBE_KEY_MD5: &str = "AAAAAAAAAAAAAAAAAAAAAA==";
#[derive(Debug, Clone, serde::Serialize)] #[derive(Debug, Clone, serde::Serialize)]
struct ReplicationCheckResponse { struct ReplicationCheckResponse {
@@ -251,6 +262,8 @@ struct ReplicationCheckPhases {
put: ReplicationCheckPhaseStatus, put: ReplicationCheckPhaseStatus,
#[serde(rename = "VersionFidelity")] #[serde(rename = "VersionFidelity")]
version_fidelity: ReplicationCheckPhaseStatus, version_fidelity: ReplicationCheckPhaseStatus,
#[serde(rename = "SsecPassthrough")]
ssec_passthrough: ReplicationCheckPhaseStatus,
#[serde(rename = "DeleteMarker")] #[serde(rename = "DeleteMarker")]
delete_marker: ReplicationCheckPhaseStatus, delete_marker: ReplicationCheckPhaseStatus,
#[serde(rename = "VersionDelete")] #[serde(rename = "VersionDelete")]
@@ -1853,7 +1866,7 @@ fn build_replication_check_response(mut targets: Vec<ReplicationCheckTargetStatu
let data = serde_json::to_vec(&ReplicationCheckResponse { let data = serde_json::to_vec(&ReplicationCheckResponse {
status: status.to_string(), status: status.to_string(),
active_mutation: true, active_mutation: true,
mutation_description: "Writes a probe object, creates a delete marker, deletes the probe version, and cleans up all probe artifacts on each target.", mutation_description: "Writes probe objects (including an SSE-C passthrough probe), creates a delete marker, deletes the probe versions, and cleans up all probe artifacts on each target.",
probe_namespace: REPLICATION_CHECK_PROBE_PREFIX, probe_namespace: REPLICATION_CHECK_PROBE_PREFIX,
targets, targets,
}) })
@@ -2070,6 +2083,25 @@ async fn check_replication_target(
time: OffsetDateTime::now_utc(), time: OffsetDateTime::now_utc(),
}; };
execute_replication_probe(&mut result, &mut operations).await; execute_replication_probe(&mut result, &mut operations).await;
// Sync the probe verdict into the runtime capability cache: the
// replication worker then fails SSE-C replication closed on a flagged
// target (or skips its own HEAD-back audit on a proven one) without
// re-learning what the probe just established.
match (result.phases.ssec_passthrough.status, result.phases.ssec_passthrough.code) {
("OK", _) => {
BucketTargetSys::get()
.record_ssec_passthrough_capability(&target.arn, SsecPassthroughCapability::Supported)
.await;
}
("FAILED", Some(REPLICATION_CHECK_CODE_SSEC_PASSTHROUGH)) => {
BucketTargetSys::get()
.record_ssec_passthrough_capability(&target.arn, SsecPassthroughCapability::Unsupported)
.await;
}
_ => {}
}
result result
} }
@@ -2088,6 +2120,15 @@ struct ReplicationProbePutOutcome {
response_version_id: Option<String>, response_version_id: Option<String>,
} }
/// Outcome of the SSE-C passthrough probe: whether the HEAD-back of the probe
/// replica echoed SSE-C evidence (the customer-algorithm header a RustFS
/// target restores from the passthrough transport headers), plus the version
/// the target assigned so cleanup can address it.
struct ReplicationSsecProbeOutcome {
evidence_present: bool,
version_id: Option<String>,
}
struct ReplicationProbeMultipartError { struct ReplicationProbeMultipartError {
primary: S3ClientError, primary: S3ClientError,
cleanup_error: Option<String>, cleanup_error: Option<String>,
@@ -2110,9 +2151,14 @@ trait ReplicationProbeOperations {
/// there: a target can adopt PutObject version ids and still mint its own /// there: a target can adopt PutObject version ids and still mint its own
/// for CreateMultipartUpload. /// for CreateMultipartUpload.
async fn multipart_put(&mut self) -> Result<ReplicationProbePutOutcome, ReplicationProbeMultipartError>; async fn multipart_put(&mut self) -> Result<ReplicationProbePutOutcome, ReplicationProbeMultipartError>;
/// PUT a probe version carrying the SSE-C passthrough transport headers,
/// HEAD it back through the replication-check channel, and report whether
/// the SSE-C evidence survived. Cleanup of the created version is the
/// caller's job (the outcome carries its version id).
async fn ssec_passthrough_probe(&mut self) -> Result<ReplicationSsecProbeOutcome, S3ClientError>;
async fn create_delete_marker(&mut self, version_id: Option<&str>) -> Result<Option<String>, S3ClientError>; async fn create_delete_marker(&mut self, version_id: Option<&str>) -> Result<Option<String>, S3ClientError>;
async fn delete_version(&mut self, version_id: Option<&str>) -> Result<(), S3ClientError>; async fn delete_version(&mut self, version_id: Option<&str>) -> Result<(), S3ClientError>;
async fn cleanup(&mut self, known_version_ids: [Option<&str>; 3]) -> Result<(), String>; async fn cleanup(&mut self, known_version_ids: [Option<&str>; 4]) -> Result<(), String>;
} }
struct RemoteReplicationProbeOperations<'a> { struct RemoteReplicationProbeOperations<'a> {
@@ -2132,6 +2178,10 @@ impl ReplicationProbeOperations for RemoteReplicationProbeOperations<'_> {
multipart_put_replication_probe_object(self.client, self.bucket, self.key, self.time).await multipart_put_replication_probe_object(self.client, self.bucket, self.key, self.time).await
} }
async fn ssec_passthrough_probe(&mut self) -> Result<ReplicationSsecProbeOutcome, S3ClientError> {
ssec_passthrough_probe_object(self.client, self.bucket, self.key, self.time).await
}
async fn create_delete_marker(&mut self, version_id: Option<&str>) -> Result<Option<String>, S3ClientError> { async fn create_delete_marker(&mut self, version_id: Option<&str>) -> Result<Option<String>, S3ClientError> {
delete_replication_probe_object( delete_replication_probe_object(
self.client, self.client,
@@ -2155,7 +2205,7 @@ impl ReplicationProbeOperations for RemoteReplicationProbeOperations<'_> {
.map(|_| ()) .map(|_| ())
} }
async fn cleanup(&mut self, known_version_ids: [Option<&str>; 3]) -> Result<(), String> { async fn cleanup(&mut self, known_version_ids: [Option<&str>; 4]) -> Result<(), String> {
cleanup_replication_probe(self.client, self.bucket, self.key, known_version_ids).await cleanup_replication_probe(self.client, self.bucket, self.key, known_version_ids).await
} }
} }
@@ -2176,6 +2226,7 @@ fn version_fidelity_error(api: &str, outcome: &ReplicationProbePutOutcome) -> Op
async fn execute_replication_probe(result: &mut ReplicationCheckTargetStatus, operations: &mut impl ReplicationProbeOperations) { async fn execute_replication_probe(result: &mut ReplicationCheckTargetStatus, operations: &mut impl ReplicationProbeOperations) {
let mut probe_version_id = None; let mut probe_version_id = None;
let mut multipart_probe_version_id = None; let mut multipart_probe_version_id = None;
let mut ssec_probe_version_id = None;
let mut delete_marker_version_id = None; let mut delete_marker_version_id = None;
let mut cleanup_required = true; let mut cleanup_required = true;
let mut multipart_cleanup_error = None; let mut multipart_cleanup_error = None;
@@ -2231,6 +2282,38 @@ async fn execute_replication_probe(result: &mut ReplicationCheckTargetStatus, op
} }
} }
// N2: probe SSE-C passthrough with the same transport headers live
// replication sends. A target that drops them (MinIO, generic S3) stores
// the probe as a plain object and echoes no SSE-C evidence on the
// HEAD-back; SSE-C replicas there would silently lose their decryption
// material, so the target must be flagged with a machine-readable code.
// Deliberately unlike VersionFidelity, a failed SsecPassthrough phase
// does NOT fail the target overall: version-identity drift breaks the
// replication contract for every object, while dropped SSE-C passthrough
// headers only limit a capability — a plaintext-only deployment against a
// MinIO target is perfectly healthy and must not turn red. The phase's
// own FAILED + machine-readable Code remains for madmin consumers (and
// the verdict still reaches the runtime capability cache).
if result.phases.put.status == "OK" && result.phases.version_fidelity.status == "OK" {
match operations.ssec_passthrough_probe().await {
Ok(outcome) => {
ssec_probe_version_id = outcome.version_id;
if outcome.evidence_present {
result.phases.ssec_passthrough = ReplicationCheckPhaseStatus::passed();
} else {
let error = "target drops SSE-C passthrough replication headers; \
SSE-C replicas would lose their decryption material on this target";
result.phases.ssec_passthrough =
ReplicationCheckPhaseStatus::failed_with_code(error, REPLICATION_CHECK_CODE_SSEC_PASSTHROUGH);
}
}
Err(err) => {
let error = format_replication_check_client_error(&err, ReplicationCheckFailureContext::ReplicateObject);
result.phases.ssec_passthrough = ReplicationCheckPhaseStatus::failed(&error);
}
}
}
if result.phases.put.status == "OK" && result.phases.version_fidelity.status == "OK" { if result.phases.put.status == "OK" && result.phases.version_fidelity.status == "OK" {
match operations.create_delete_marker(probe_version_id.as_deref()).await { match operations.create_delete_marker(probe_version_id.as_deref()).await {
Ok(version_id) => { Ok(version_id) => {
@@ -2259,6 +2342,7 @@ async fn execute_replication_probe(result: &mut ReplicationCheckTargetStatus, op
.cleanup([ .cleanup([
probe_version_id.as_deref(), probe_version_id.as_deref(),
multipart_probe_version_id.as_deref(), multipart_probe_version_id.as_deref(),
ssec_probe_version_id.as_deref(),
delete_marker_version_id.as_deref(), delete_marker_version_id.as_deref(),
]) ])
.await .await
@@ -2553,6 +2637,72 @@ async fn put_replication_probe_object(
}) })
} }
/// PUT a fresh probe version carrying the SSE-C passthrough transport headers
/// (the wire shape live SSE-C replication uses), then HEAD it back through the
/// worker channel (replication-check exemption + proxy suppression). A RustFS
/// target restores the transport headers into stored SSE-C metadata and its
/// HEAD echoes `x-amz-server-side-encryption-customer-algorithm`; a target
/// that dropped the headers echoes nothing. The probe body is never SSE-C
/// encrypted — only the metadata round-trip matters — and the version is
/// deleted by the shared probe cleanup.
async fn ssec_passthrough_probe_object(
target_client: &TargetClient,
target_bucket: &str,
probe_key: &str,
now: OffsetDateTime,
) -> Result<ReplicationSsecProbeOutcome, S3ClientError> {
let options = build_replication_probe_put_options(now);
let sent_version_id = options.internal.source_version_id.clone();
let mut headers = build_replication_probe_headers(&options);
// These are full wire names (not x-rustfs/x-minio suffixes), so they must
// be inserted verbatim — `insert_header` would mangle them.
for (name, value) in [
(REPLICATION_SSEC_ALGORITHM_HEADER, "AES256"),
(REPLICATION_SSEC_KEY_MD5_HEADER, REPLICATION_CHECK_SSEC_PROBE_KEY_MD5),
(REPLICATION_SSEC_ORIGINAL_SIZE_HEADER, "8"),
] {
let name = name
.parse::<HeaderName>()
.map_err(|err| S3ClientError::new(format!("invalid ssec probe header name: {err}")))?;
let value =
HeaderValue::from_str(value).map_err(|err| S3ClientError::new(format!("invalid ssec probe header value: {err}")))?;
headers.insert(name, value);
}
let query_version_id = sent_version_id.clone();
let response = target_client
.client
.put_object()
.bucket(target_bucket)
.key(probe_key)
.content_length(8)
.body(AwsByteStream::from_static(b"aaaaaaaa"))
.customize()
.map_request(move |mut req| {
for (key, value) in headers.clone() {
req.headers_mut().insert(key.expect("operation should succeed"), value);
}
let uri = append_version_id_query(req.uri(), &query_version_id);
req.set_uri(uri).map_err(std::io::Error::other)?;
Result::<_, std::io::Error>::Ok(req)
})
.send()
.await
.map_err(S3ClientError::from)?;
let version_id = response.version_id().map(ToOwned::to_owned);
let head_version = version_id.clone().or_else(|| Some(sent_version_id.clone()));
let head = target_client
.head_object(target_bucket, probe_key, head_version)
.await
.map_err(S3ClientError::from)?;
Ok(ReplicationSsecProbeOutcome {
evidence_present: head.sse_customer_algorithm().is_some_and(|algorithm| !algorithm.is_empty()),
version_id,
})
}
async fn delete_replication_probe_object( async fn delete_replication_probe_object(
target_client: &TargetClient, target_client: &TargetClient,
target_bucket: &str, target_bucket: &str,
@@ -3723,6 +3873,12 @@ mod tests {
/// Same, for the multipart leg: a target may mirror PutObject ids and /// Same, for the multipart leg: a target may mirror PutObject ids and
/// still mint its own at CreateMultipartUpload. /// still mint its own at CreateMultipartUpload.
minted_multipart_version_id: Option<&'static str>, minted_multipart_version_id: Option<&'static str>,
/// Transport failure of the SSE-C passthrough probe itself.
ssec_probe_error: Option<&'static str>,
/// Models a MinIO-like target that drops the SSE-C passthrough
/// headers: the probe HEAD-back echoes no SSE-C evidence. The default
/// (false) models a RustFS target that preserves them.
ssec_evidence_missing: bool,
delete_marker_error: Option<&'static str>, delete_marker_error: Option<&'static str>,
version_delete_error: Option<&'static str>, version_delete_error: Option<&'static str>,
cleanup_error: Option<&'static str>, cleanup_error: Option<&'static str>,
@@ -3760,6 +3916,17 @@ mod tests {
}) })
} }
async fn ssec_passthrough_probe(&mut self) -> Result<ReplicationSsecProbeOutcome, S3ClientError> {
self.calls.push("ssec-probe");
match self.ssec_probe_error {
Some(code) => Err(scripted_probe_error(code)),
None => Ok(ReplicationSsecProbeOutcome {
evidence_present: !self.ssec_evidence_missing,
version_id: Some("ssec-version".to_string()),
}),
}
}
async fn create_delete_marker(&mut self, _version_id: Option<&str>) -> Result<Option<String>, S3ClientError> { async fn create_delete_marker(&mut self, _version_id: Option<&str>) -> Result<Option<String>, S3ClientError> {
self.calls.push("delete-marker"); self.calls.push("delete-marker");
match self.delete_marker_error { match self.delete_marker_error {
@@ -3776,7 +3943,7 @@ mod tests {
} }
} }
async fn cleanup(&mut self, known_version_ids: [Option<&str>; 3]) -> Result<(), String> { async fn cleanup(&mut self, known_version_ids: [Option<&str>; 4]) -> Result<(), String> {
self.calls.push("cleanup"); self.calls.push("cleanup");
self.cleanup_ids = known_version_ids self.cleanup_ids = known_version_ids
.into_iter() .into_iter()
@@ -3811,8 +3978,9 @@ mod tests {
assert_eq!(result.phases.version_fidelity.code, Some(REPLICATION_CHECK_CODE_VERSION_MISMATCH)); assert_eq!(result.phases.version_fidelity.code, Some(REPLICATION_CHECK_CODE_VERSION_MISMATCH));
assert_eq!(result.phases.delete_marker.status, "SKIPPED"); assert_eq!(result.phases.delete_marker.status, "SKIPPED");
assert_eq!(result.phases.version_delete.status, "SKIPPED"); assert_eq!(result.phases.version_delete.status, "SKIPPED");
assert_eq!(result.phases.ssec_passthrough.status, "SKIPPED");
assert_eq!(result.phases.cleanup.status, "OK"); assert_eq!(result.phases.cleanup.status, "OK");
assert_eq!(operations.cleanup_ids, [Some("target-minted-version".to_string()), None, None]); assert_eq!(operations.cleanup_ids, [Some("target-minted-version".to_string()), None, None, None]);
} }
#[tokio::test] #[tokio::test]
@@ -3825,6 +3993,70 @@ mod tests {
assert_eq!(result.status, "OK"); assert_eq!(result.status, "OK");
assert_eq!(result.phases.version_fidelity.status, "OK"); assert_eq!(result.phases.version_fidelity.status, "OK");
assert_eq!(result.phases.version_fidelity.code, None); assert_eq!(result.phases.version_fidelity.code, None);
assert_eq!(result.phases.ssec_passthrough.status, "OK");
assert_eq!(result.phases.ssec_passthrough.code, None);
}
/// N2: a target that drops the SSE-C passthrough transport headers must
/// fail the SsecPassthrough phase with the machine-readable code while the
/// target overall stays OK — deliberately unlike VersionFidelity: this is
/// a capability limit, not a broken replication contract, and a
/// plaintext-only deployment against such a target must not turn red. The
/// other mutation phases keep running and the probe version is cleaned up.
#[tokio::test]
async fn replication_probe_flags_ssec_passthrough_dropping_target_without_failing_target() {
let mut result = replication_check_target("arn:a", "OK", None);
let mut operations = ScriptedReplicationProbe {
ssec_evidence_missing: true,
..Default::default()
};
execute_replication_probe(&mut result, &mut operations).await;
assert_eq!(
operations.calls,
[
"put",
"multipart-put",
"ssec-probe",
"delete-marker",
"version-delete",
"cleanup"
]
);
assert_eq!(result.status, "OK", "a capability-only failure must not fail the target overall");
assert_eq!(result.error, None);
assert_eq!(result.phases.ssec_passthrough.status, "FAILED");
assert_eq!(result.phases.ssec_passthrough.code, Some(REPLICATION_CHECK_CODE_SSEC_PASSTHROUGH));
assert_eq!(
operations.cleanup_ids,
[
Some("object-version".to_string()),
Some("multipart-version".to_string()),
Some("ssec-version".to_string()),
Some("marker-version".to_string())
]
);
}
/// A transport failure of the SSE-C probe is not evidence of a dropping
/// target: the phase fails without the capability code (the runtime cache
/// stays Unknown and the worker keeps auditing), and the target overall
/// stays OK.
#[tokio::test]
async fn replication_probe_ssec_transport_failure_carries_no_capability_code() {
let mut result = replication_check_target("arn:a", "OK", None);
let mut operations = ScriptedReplicationProbe {
ssec_probe_error: Some("InternalError"),
..Default::default()
};
execute_replication_probe(&mut result, &mut operations).await;
assert_eq!(result.status, "OK");
assert_eq!(result.phases.ssec_passthrough.status, "FAILED");
assert_eq!(result.phases.ssec_passthrough.code, None);
assert_eq!(operations.cleanup_ids[2], None, "a failed ssec probe leaves no version to clean");
} }
#[tokio::test] #[tokio::test]
@@ -3855,12 +4087,23 @@ mod tests {
execute_replication_probe(&mut result, &mut operations).await; execute_replication_probe(&mut result, &mut operations).await;
assert_eq!(operations.calls, ["put", "multipart-put", "delete-marker", "version-delete", "cleanup"]); assert_eq!(
operations.calls,
[
"put",
"multipart-put",
"ssec-probe",
"delete-marker",
"version-delete",
"cleanup"
]
);
assert_eq!( assert_eq!(
operations.cleanup_ids, operations.cleanup_ids,
[ [
Some("object-version".to_string()), Some("object-version".to_string()),
Some("multipart-version".to_string()), Some("multipart-version".to_string()),
Some("ssec-version".to_string()),
None None
] ]
); );
@@ -3880,12 +4123,23 @@ mod tests {
execute_replication_probe(&mut result, &mut operations).await; execute_replication_probe(&mut result, &mut operations).await;
assert_eq!(operations.calls, ["put", "multipart-put", "delete-marker", "version-delete", "cleanup"]); assert_eq!(
operations.calls,
[
"put",
"multipart-put",
"ssec-probe",
"delete-marker",
"version-delete",
"cleanup"
]
);
assert_eq!( assert_eq!(
operations.cleanup_ids, operations.cleanup_ids,
[ [
Some("object-version".to_string()), Some("object-version".to_string()),
Some("multipart-version".to_string()), Some("multipart-version".to_string()),
Some("ssec-version".to_string()),
Some("marker-version".to_string()) Some("marker-version".to_string())
] ]
); );
+1
View File
@@ -196,6 +196,7 @@ pub(crate) mod bucket_target_sys {
pub(crate) type PutObjectOptions = super::ecstore_bucket::bucket_target_sys::PutObjectOptions; pub(crate) type PutObjectOptions = super::ecstore_bucket::bucket_target_sys::PutObjectOptions;
pub(crate) type RemoveObjectOptions = super::ecstore_bucket::bucket_target_sys::RemoveObjectOptions; pub(crate) type RemoveObjectOptions = super::ecstore_bucket::bucket_target_sys::RemoveObjectOptions;
pub(crate) type S3ClientError = super::ecstore_bucket::bucket_target_sys::S3ClientError; pub(crate) type S3ClientError = super::ecstore_bucket::bucket_target_sys::S3ClientError;
pub(crate) type SsecPassthroughCapability = super::ecstore_bucket::bucket_target_sys::SsecPassthroughCapability;
pub(crate) type TargetClient = super::ecstore_bucket::bucket_target_sys::TargetClient; pub(crate) type TargetClient = super::ecstore_bucket::bucket_target_sys::TargetClient;
} }