Compare commits

...

13 Commits

Author SHA1 Message Date
唐小鸭 194d4aeef2 fix(sse): drop the SSE-S3 header from SSE-C CopyObject responses
CopyObject to an SSE-C destination answered with
`x-amz-server-side-encryption: AES256` next to the customer-key
headers, for the same reason the multipart responses did: the SSE-C
material records `AES256` as its server-side-encryption value and the
response forwarded it. It now returns only the customer-algorithm and
customer-key-MD5 headers, as PutObject does. Copies to SSE-S3 and
SSE-KMS destinations are unchanged.
2026-11-22 22:38:22 +08:00
唐小鸭 4e680145bd fix(sse): drop the SSE-S3 header from SSE-C multipart responses
CreateMultipartUpload, UploadPart, UploadPartCopy and
CompleteMultipartUpload on an SSE-C upload answered with
`x-amz-server-side-encryption: AES256` next to the customer-key
headers. That header names a managed algorithm, so the response
described an SSE-S3 object.

SSE-C metadata and encryption material record `AES256` as their
server-side-encryption value. PutObject, GetObject and HeadObject
already leave it out of an SSE-C response; the four multipart
projections forwarded it unchanged. They now return only the
customer-algorithm and customer-key-MD5 headers for SSE-C. SSE-S3 and
SSE-KMS multipart responses are unchanged.
2026-11-22 22:33:01 +08:00
overtrue 818a6b6a14 test(multipart): own the retry copy envelope
The copy after the UploadPart permit is released carries no request headers,
so it uses the owned request and still checks the copied bytes.
2026-11-22 22:31:32 +08:00
overtrue 67d306b0de test(multipart): own the unconditional create envelope
The second create in the If-None-Match test carries no request headers, so
it uses the owned request and still requires an upload id.
2026-11-22 21:55:59 +08:00
overtrue a6f881b42d test(multipart): own the KMS upload-part envelope 2026-11-22 21:25:04 +08:00
overtrue dba23c4f55 test(multipart): own the KMS create envelope
The versioned KMS upload is created on the owned request, and the upload id
is still required before the encrypted part is written.
2026-11-22 20:54:59 +08:00
overtrue 0d17f5f713 test(multipart): own the KMS completion and replay envelope
The versioned KMS completion uses the same owned mapping on both attempts,
and the listing assertions stay in place.
2026-11-22 20:18:13 +08:00
overtrue e09df1f7cd test(multipart): own the other concurrent completion envelope
The second concurrent completion uses the same owned mapping, and exactly
one of the two commits still succeeds.
2026-11-22 19:45:16 +08:00
overtrue c5c65e112e test(multipart): own one concurrent completion envelope
The first concurrent completion still shares the durable quota, so exactly
one of the two commits.
2026-11-22 19:14:04 +08:00
overtrue dfa3d7a2cd test(multipart): own the If-Match completion refusal
A weak or nonmatching If-Match still fails as PreconditionFailed before the
existing object is overwritten.
2026-11-22 18:43:06 +08:00
overtrue 578025149b test(multipart): own the weak If-None-Match completion refusal
A weak If-None-Match ETag on complete is still rejected as unsupported.
2026-11-22 18:10:07 +08:00
overtrue bfc51cf3da test(multipart): own the DTO-only If-None-Match completion refusal
A DTO-only If-None-Match star still fails before the existing object is overwritten.
2026-11-22 17:38:36 +08:00
overtrue 78b35c3ea8 test(multipart): own the SSE-C completion checksum envelope
Keyless completion still fails before the plaintext checksum is disclosed, and a verified key still completes and replays.
2026-11-22 17:07:38 +08:00
8 changed files with 822 additions and 72 deletions
+2
View File
@@ -46,6 +46,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- **SSE-C multipart and copy responses no longer carry `x-amz-server-side-encryption`**: `CreateMultipartUpload`, `UploadPart`, `UploadPartCopy` and `CompleteMultipartUpload` on an SSE-C upload, and `CopyObject` to an SSE-C destination, answered with `x-amz-server-side-encryption: AES256` next to the customer-key headers, so a client reading the response saw an SSE-S3 object. They now return only `x-amz-server-side-encryption-customer-algorithm` and `x-amz-server-side-encryption-customer-key-MD5`, as `PutObject`, `GetObject` and `HeadObject` already did. SSE-S3 and SSE-KMS responses are unchanged.
- **Vault KMS backends no longer fail when the server answers reads from a replica**: a Vault or OpenBao node that serves reads locally (an OpenBao standby, a Vault Enterprise performance standby) can return the state from before a write for a short while after acknowledging it. Against such a node the KV2 backend failed every key creation with `404 Key not found` while leaving the key stored without its description and tags, failed operations issued right after a create or update with `Key not found` or `Concurrent modification`, and the Transit backend failed a disable followed quickly by an enable. Key creation on the KV2 backend is now a single create-only write that carries the description and tags. Both backends now re-read a key record when the answer is older than a write the same RustFS node made in the previous five seconds, pause before re-reading after a lost check-and-set write and allow five attempts instead of three, and retry a Transit key delete that is refused because the deletion switch is not visible yet. A different RustFS node reading the same replica can still observe the earlier state until the replica catches up; see [OpenBao as the Vault KMS backend](docs/operations/kms-openbao-compatibility.md).
- **KMS keys could not be described by ARN through the path route**: `GET /rustfs/admin/v3/kms/keys/{key_id}` read the path segment verbatim, so a key id that has to be percent-encoded to fit in one segment — an AWS key ARN, with its `:` and `/` — was looked up, authorized and audited under its encoded spelling and answered `404`. The segment is now percent-decoded first; a segment that is not valid UTF-8 after decoding is refused with `400 InvalidArgument`. The `describe-key?keyId=` form was unaffected.
+311
View File
@@ -789,4 +789,315 @@ mod tests {
assert_eq!(authorizer.calls().len(), 4);
});
}
/// The SSE fields one multipart response carries, each of which becomes a response header.
#[derive(Debug, Default, PartialEq, Eq)]
struct SseResponseFields {
server_side_encryption: Option<String>,
ssekms_key_id: Option<String>,
sse_customer_algorithm: Option<String>,
sse_customer_key_md5: Option<String>,
}
const SSE_HEADER_PREFIX: &str = "x-amz-server-side-encryption";
/// Runs one upload through CreateMultipartUpload, UploadPart, UploadPartCopy and
/// CompleteMultipartUpload and returns the SSE fields of each response, followed by the
/// SSE header names of the encoded completion response.
async fn multipart_sse_responses(
bucket: &str,
key: &str,
requested_sse: Option<&str>,
requested_kms_key_id: Option<&str>,
customer_key: Option<(&str, &str)>,
) -> (Vec<(&'static str, SseResponseFields)>, Vec<String>) {
use crate::app::storage_api::s3::dto::{CompletedPart, CopySource, StreamingBlob, UploadPartInput};
use crate::app::storage_api::test::contract::object::ObjectIO as _;
use rustfs_gateway::{Limits, MetaView, OperationCodec, TargetKind, WireRequest};
use rustfs_gateway_types::compat::s3s_5761ddfe::generated::ops::complete_multipart_upload;
let store = crate::app::gating_test_env::shared_gating_ecstore().await;
let usecase = DefaultMultipartUsecase::from_global();
let source_key = format!("{key}-source");
store
.put_object(
bucket,
&source_key,
&mut crate::storage::PutObjReader::from_vec(b"copied".to_vec()),
&ObjectOptions::default(),
)
.await
.expect("write plaintext copy source");
let customer_algorithm = customer_key.map(|_| "AES256".to_string());
let customer_key_md5 = customer_key.map(|(_, md5)| md5.to_string());
let customer_key = customer_key.map(|(key, _)| key.to_string());
let create = usecase
.execute_create_multipart_upload(
owned_envelope(Method::POST),
CreateMultipartUploadFields {
bucket: bucket.to_string(),
key: key.to_string(),
metadata: None,
tagging: None,
storage_class: None,
server_side_encryption: requested_sse.map(ToOwned::to_owned),
sse_customer_algorithm: customer_algorithm.clone(),
sse_customer_key: customer_key.clone(),
sse_customer_key_md5: customer_key_md5.clone(),
ssekms_key_id: requested_kms_key_id.map(ToOwned::to_owned),
object_lock_legal_hold_status: None,
object_lock_mode: None,
object_lock_retain_until_date: None,
},
None,
)
.await
.expect("create multipart upload");
let upload_id = create.upload_id;
let part = usecase
.execute_upload_part_with_audit(
owned_envelope(Method::PUT),
UploadPartWriteInput::from_legacy(
UploadPartInput {
bucket: bucket.to_string(),
key: key.to_string(),
upload_id: upload_id.clone(),
part_number: 1,
content_length: Some(3),
body: Some(StreamingBlob::from(bytes::Bytes::from_static(b"abc"))),
sse_customer_algorithm: customer_algorithm.clone(),
sse_customer_key: customer_key.clone(),
sse_customer_key_md5: customer_key_md5.clone(),
..Default::default()
},
None,
),
)
.await
.expect("upload part");
let copy = Box::pin(usecase.execute_upload_part_copy_with_audit(
owned_envelope(Method::PUT),
UploadPartCopyWriteInput {
target: UploadPartTarget::Legacy {
bucket: bucket.to_string(),
key: key.to_string(),
upload_id: upload_id.clone(),
},
part_number: 2,
copy_source: rustfs_gateway_types::compat::s3s_5761ddfe::copy_source::from_s3s(CopySource::Bucket {
bucket: bucket.into(),
key: source_key.into(),
version_id: None,
}),
copy_source_range: None,
copy_source_if_match: None,
copy_source_if_none_match: None,
sse_customer_algorithm: customer_algorithm.clone(),
sse_customer_key: customer_key.clone(),
sse_customer_key_md5: customer_key_md5.clone(),
copy_source_sse_customer_algorithm: None,
copy_source_sse_customer_key: None,
copy_source_sse_customer_key_md5: None,
},
))
.await
.expect("copy part");
let mut complete_input = complete_multipart_upload::input_from_s3s(CompleteMultipartUploadInput {
bucket: bucket.to_string(),
key: key.to_string(),
upload_id,
multipart_upload: Some(CompletedMultipartUpload {
parts: Some(vec![CompletedPart {
part_number: Some(1),
..Default::default()
}]),
}),
sse_customer_algorithm: customer_algorithm,
sse_customer_key: customer_key,
sse_customer_key_md5: customer_key_md5,
..Default::default()
})
.map_err(crate::app::dto_values::seam_error)
.expect("owned completion input");
complete_input.multipart_upload.parts[0].e_tag = part
.e_tag
.as_deref()
.map(|tag| rustfs_gateway_types::ETag::from_stored(to_s3s_etag(tag).value().to_owned()));
let complete = Box::pin(usecase.execute_complete_multipart_upload(owned_envelope(Method::POST), complete_input))
.await
.expect("complete multipart upload");
let request = http::Request::builder()
.method("POST")
.uri("http://localhost/bucket/object")
.header("host", "localhost")
.body(bytes::Bytes::new())
.expect("wire request");
let wire = WireRequest::accept(request, &Limits::default()).expect("accepted wire request");
let view = MetaView::of(&wire, TargetKind::Object).expect("object view");
let encoded = rustfs_gateway::dto::CompleteMultipartUpload::encode(complete.clone(), &view, 200)
.expect("completion response encodes");
let mut completion_sse_headers: Vec<String> = encoded
.headers
.keys()
.map(|name| name.as_str().to_owned())
.filter(|name| name.starts_with(SSE_HEADER_PREFIX))
.collect();
completion_sse_headers.sort();
let responses = vec![
(
"CreateMultipartUpload",
SseResponseFields {
server_side_encryption: create.server_side_encryption,
ssekms_key_id: create.ssekms_key_id,
sse_customer_algorithm: create.sse_customer_algorithm,
sse_customer_key_md5: create.sse_customer_key_md5,
},
),
(
"UploadPart",
SseResponseFields {
server_side_encryption: part.server_side_encryption,
ssekms_key_id: part.ssekms_key_id,
sse_customer_algorithm: part.sse_customer_algorithm,
sse_customer_key_md5: part.sse_customer_key_md5,
},
),
(
"UploadPartCopy",
SseResponseFields {
server_side_encryption: copy.server_side_encryption,
ssekms_key_id: copy.ssekms_key_id,
sse_customer_algorithm: copy.sse_customer_algorithm,
sse_customer_key_md5: copy.sse_customer_key_md5,
},
),
(
"CompleteMultipartUpload",
SseResponseFields {
server_side_encryption: complete.server_side_encryption.map(|value| value.as_str().to_owned()),
ssekms_key_id: complete.ssekms_key_id,
sse_customer_algorithm: complete.sse_customer_algorithm,
sse_customer_key_md5: complete.sse_customer_key_md5,
},
),
];
(responses, completion_sse_headers)
}
fn every_multipart_response(expected: impl Fn() -> SseResponseFields) -> Vec<(&'static str, SseResponseFields)> {
[
"CreateMultipartUpload",
"UploadPart",
"UploadPartCopy",
"CompleteMultipartUpload",
]
.into_iter()
.map(|operation| (operation, expected()))
.collect()
}
// `x-amz-server-side-encryption` names the SSE-S3 or SSE-KMS algorithm. An SSE-C upload
// answers with the customer-key headers only, as PutObject, GetObject and HeadObject do.
#[test]
#[serial_test::serial]
fn ssec_multipart_responses_carry_only_the_customer_key_headers() {
crate::app::gating_test_env::run_large_stack_test("mpu-ssec-response-headers", || async {
let (_store, bucket) =
crate::app::gating_test_env::durable_quota_test_bucket("mpu-ssec-response-headers", 16_384).await;
let customer_key = "KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKio=";
let customer_key_md5 = "SaaDheEjzuynJH8eW6AEpQ==";
let (responses, completion_sse_headers) =
multipart_sse_responses(&bucket, "customer-encrypted", None, None, Some((customer_key, customer_key_md5))).await;
assert_eq!(
responses,
every_multipart_response(|| SseResponseFields {
sse_customer_algorithm: Some("AES256".to_string()),
sse_customer_key_md5: Some(customer_key_md5.to_string()),
..Default::default()
})
);
assert_eq!(
completion_sse_headers,
[
"x-amz-server-side-encryption-customer-algorithm",
"x-amz-server-side-encryption-customer-key-md5",
]
);
});
}
#[test]
#[serial_test::serial]
fn managed_sse_multipart_responses_name_the_managed_algorithm() {
crate::app::gating_test_env::run_large_stack_test("mpu-managed-sse-response-headers", || async {
use rustfs_kms::types::{CreateKeyRequest, KeyUsage};
let (_store, bucket) =
crate::app::gating_test_env::durable_quota_test_bucket("mpu-managed-sse-response-headers", 16_384).await;
let key_dir = tempfile::TempDir::new().expect("local KMS key directory");
let manager = crate::app::gating_test_env::shared_gating_ambient().await.kms().handle();
let local_kms = || rustfs_kms::KmsConfig::local(key_dir.path().to_path_buf()).with_insecure_development_defaults();
manager.reconfigure(local_kms()).await.expect("configure local KMS");
let kms_key = manager
.get_encryption_service()
.await
.expect("local KMS encryption service")
.create_key(CreateKeyRequest {
key_name: Some("multipart-response-key".to_string()),
key_usage: KeyUsage::EncryptDecrypt,
description: None,
policy: None,
tags: HashMap::new(),
origin: None,
})
.await
.expect("create multipart KMS key");
// SSE-S3 names no key, so it wraps its data key under the service default.
manager
.reconfigure(local_kms().with_default_key(kms_key.key_id.clone()))
.await
.expect("select the default KMS key");
let (sse_s3, sse_s3_completion_headers) =
multipart_sse_responses(&bucket, "sse-s3-encrypted", Some(ServerSideEncryption::AES256), None, None).await;
assert_eq!(
sse_s3,
every_multipart_response(|| SseResponseFields {
server_side_encryption: Some("AES256".to_string()),
..Default::default()
})
);
assert_eq!(sse_s3_completion_headers, ["x-amz-server-side-encryption"]);
let (sse_kms, sse_kms_completion_headers) = multipart_sse_responses(
&bucket,
"sse-kms-encrypted",
Some(ServerSideEncryption::AWS_KMS),
Some(&kms_key.key_id),
None,
)
.await;
assert_eq!(
sse_kms,
every_multipart_response(|| SseResponseFields {
server_side_encryption: Some("aws:kms".to_string()),
ssekms_key_id: Some(kms_key.key_id.clone()),
..Default::default()
})
);
assert_eq!(
sse_kms_completion_headers,
["x-amz-server-side-encryption", "x-amz-server-side-encryption-aws-kms-key-id"]
);
manager.stop().await.expect("stop local KMS");
});
}
}
+401 -54
View File
@@ -588,8 +588,15 @@ impl DefaultMultipartUsecase {
// Storage has authorized the actual source or committed replay candidate
// before comparing checksums/parts, recovery publication or replay effects.
let encryption_metadata = upload_user_defined.as_ref().unwrap_or(obj_info.user_defined.as_ref());
let ssec_algorithm = encryption_metadata
.get("x-amz-server-side-encryption-customer-algorithm")
.cloned();
// SSE-C metadata also records `x-amz-server-side-encryption: AES256`, but
// that header names a managed algorithm; an SSE-C upload answers with its
// customer-key headers only.
let server_side_encryption = encryption_metadata
.get("x-amz-server-side-encryption")
.filter(|_| ssec_algorithm.is_none())
.map(|s| ServerSideEncryption::custom(s.clone()));
let ssekms_key_id = match server_side_encryption.as_ref() {
Some(sse) if sse.as_str() == ServerSideEncryption::AWS_KMS.as_str() => encryption_metadata
@@ -597,9 +604,6 @@ impl DefaultMultipartUsecase {
.cloned(),
_ => None,
};
let ssec_algorithm = encryption_metadata
.get("x-amz-server-side-encryption-customer-algorithm")
.cloned();
let ssec_key_md5 = encryption_metadata
.get("x-amz-server-side-encryption-customer-key-md5")
.cloned();
@@ -1357,7 +1361,58 @@ mod tests {
part.e_tag = Some(ETag::Strong("guessed-etag".to_string()));
part.checksum_sha256 = Some("AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=".to_string());
}
let result = usecase.complete_for_s3s_request(build_request(input, Method::POST)).await;
use rustfs_gateway_types::compat::s3s_5761ddfe::generated::ops::complete_multipart_upload;
let mut s3s_input = input;
let has_upload = s3s_input.multipart_upload.is_some();
s3s_input.multipart_upload.get_or_insert_with(Default::default);
let tags: Vec<Option<String>> = s3s_input
.multipart_upload
.iter_mut()
.flat_map(|upload| upload.parts.iter_mut().flatten())
.map(|part| {
part.part_number.get_or_insert(0);
part.e_tag.take().map(|tag| tag.value().to_owned())
})
.collect();
for checksum in [
&mut s3s_input.checksum_crc32,
&mut s3s_input.checksum_crc32c,
&mut s3s_input.checksum_crc64nvme,
&mut s3s_input.checksum_md5,
&mut s3s_input.checksum_sha1,
&mut s3s_input.checksum_sha256,
&mut s3s_input.checksum_sha512,
&mut s3s_input.checksum_xxhash128,
&mut s3s_input.checksum_xxhash3,
&mut s3s_input.checksum_xxhash64,
] {
*checksum = None;
}
let result = match complete_multipart_upload::input_from_s3s(s3s_input) {
Err(error) => Err(crate::app::dto_values::seam_error(error)),
Ok(mut owned) => {
for (part, tag) in owned.multipart_upload.parts.iter_mut().zip(tags) {
part.e_tag = tag.map(rustfs_gateway_types::ETag::from_stored);
}
let taken = std::mem::take(&mut owned.multipart_upload);
let parts = has_upload.then_some(taken);
match parts {
Some(parts) => {
owned.multipart_upload = parts;
Box::pin(usecase.execute_complete_multipart_upload(owned_envelope(Method::POST), owned)).await
}
None => {
Box::pin(usecase.execute_complete_multipart_upload_parts(
owned_envelope(Method::POST),
owned,
None,
))
.await
}
}
}
};
if keyed {
let response = result.expect("verified key completes and replays");
assert!(response.checksum_sha256.is_some(), "verified key retains checksum response");
@@ -1563,36 +1618,36 @@ mod tests {
.await
.expect("configure bucket-default SSE-KMS");
let usecase = DefaultMultipartUsecase::from_global();
let upload_id = usecase
.execute_create_multipart_upload_from_s3s(build_request(
CreateMultipartUploadInput {
let created = usecase
.execute_create_multipart_upload(
owned_envelope(Method::POST),
super::super::CreateMultipartUploadFields::from_legacy(CreateMultipartUploadInput {
bucket: bucket.clone(),
key: key.to_string(),
..Default::default()
},
Method::POST,
))
}),
None,
)
.await
.expect("create KMS multipart upload")
.output
.upload_id
.expect("KMS upload id");
let part = usecase
.execute_upload_part(build_request(
UploadPartInput {
bucket: bucket.clone(),
key: key.to_string(),
upload_id: upload_id.clone(),
part_number: 1,
content_length: Some(3),
body: Some(StreamingBlob::from(Bytes::from_static(b"abc"))),
..Default::default()
},
Method::PUT,
))
.expect("create KMS multipart upload");
let upload_id = created.upload_id;
assert!(!upload_id.is_empty(), "KMS upload id");
let input = UploadPartInput {
bucket: bucket.clone(),
key: key.to_string(),
upload_id: upload_id.clone(),
part_number: 1,
content_length: Some(3),
body: Some(StreamingBlob::from(Bytes::from_static(b"abc"))),
..Default::default()
};
let output = usecase
.execute_upload_part_with_audit(
owned_envelope(Method::PUT),
super::super::UploadPartWriteInput::from_legacy(input, None),
)
.await
.expect("upload encrypted KMS part")
.output;
.expect("upload encrypted KMS part");
let complete = CompleteMultipartUploadInput {
bucket: bucket.clone(),
key: key.to_string(),
@@ -1600,17 +1655,68 @@ mod tests {
multipart_upload: Some(CompletedMultipartUpload {
parts: Some(vec![CompletedPart {
part_number: Some(1),
e_tag: part.e_tag,
e_tag: output
.e_tag
.map(|tag| crate::app::storage_api::test::object_utils::to_s3s_etag(&tag)),
..Default::default()
}]),
}),
..Default::default()
};
for _ in 0..2 {
let response = usecase
.complete_for_s3s_request(build_request(complete.clone(), Method::POST))
.await
.expect("complete and replay KMS multipart upload");
use rustfs_gateway_types::compat::s3s_5761ddfe::generated::ops::complete_multipart_upload;
let mut s3s_input = complete.clone();
let has_upload = s3s_input.multipart_upload.is_some();
s3s_input.multipart_upload.get_or_insert_with(Default::default);
let tags: Vec<Option<String>> = s3s_input
.multipart_upload
.iter_mut()
.flat_map(|upload| upload.parts.iter_mut().flatten())
.map(|part| {
part.part_number.get_or_insert(0);
part.e_tag.take().map(|tag| tag.value().to_owned())
})
.collect();
for checksum in [
&mut s3s_input.checksum_crc32,
&mut s3s_input.checksum_crc32c,
&mut s3s_input.checksum_crc64nvme,
&mut s3s_input.checksum_md5,
&mut s3s_input.checksum_sha1,
&mut s3s_input.checksum_sha256,
&mut s3s_input.checksum_sha512,
&mut s3s_input.checksum_xxhash128,
&mut s3s_input.checksum_xxhash3,
&mut s3s_input.checksum_xxhash64,
] {
*checksum = None;
}
let response = match complete_multipart_upload::input_from_s3s(s3s_input) {
Err(error) => Err(crate::app::dto_values::seam_error(error)),
Ok(mut owned) => {
for (part, tag) in owned.multipart_upload.parts.iter_mut().zip(tags) {
part.e_tag = tag.map(rustfs_gateway_types::ETag::from_stored);
}
let taken = std::mem::take(&mut owned.multipart_upload);
let parts = has_upload.then_some(taken);
match parts {
Some(parts) => {
owned.multipart_upload = parts;
Box::pin(usecase.execute_complete_multipart_upload(owned_envelope(Method::POST), owned)).await
}
None => {
Box::pin(usecase.execute_complete_multipart_upload_parts(
owned_envelope(Method::POST),
owned,
None,
))
.await
}
}
}
}
.expect("complete and replay KMS multipart upload");
let etag = etag_to_s3s(&response.e_tag.expect("KMS completed ETag"));
let version_id = response.version_id.expect("KMS completed version");
assert_ne!(version_id, "null");
@@ -1720,13 +1826,16 @@ mod tests {
.expect_err("create multipart upload with If-None-Match: * on existing key must fail");
assert_eq!(create_err.code(), &crate::app::storage_api::s3::S3ErrorCode::PreconditionFailed);
let second_upload_id = usecase
.execute_create_multipart_upload_from_s3s(build_request(create_input, Method::POST))
let second_created = usecase
.execute_create_multipart_upload(
owned_envelope(Method::POST),
super::super::CreateMultipartUploadFields::from_legacy(create_input),
None,
)
.await
.expect("unconditional create multipart upload should succeed")
.output
.upload_id
.expect("second create response should contain upload id");
.expect("unconditional create multipart upload should succeed");
let second_upload_id = second_created.upload_id;
assert!(!second_upload_id.is_empty(), "second create response should contain upload id");
let mut second_reader = PutObjReader::from_vec(b"replacement multipart body".to_vec());
let second_part = store
.put_object_part(&bucket, object, &second_upload_id, 1, &mut second_reader, &ObjectOptions::default())
@@ -1749,10 +1858,54 @@ mod tests {
.multipart_upload(Some(completed_parts.clone()))
.build()
.expect("DTO-only complete multipart input should build");
let dto_complete_err = usecase
.complete_for_s3s_request(build_request(dto_only_complete_input, Method::POST))
.await
.expect_err("DTO-only If-None-Match: * on existing key must fail");
use rustfs_gateway_types::compat::s3s_5761ddfe::generated::ops::complete_multipart_upload;
let mut s3s_input = dto_only_complete_input;
let has_upload = s3s_input.multipart_upload.is_some();
s3s_input.multipart_upload.get_or_insert_with(Default::default);
let tags: Vec<Option<String>> = s3s_input
.multipart_upload
.iter_mut()
.flat_map(|upload| upload.parts.iter_mut().flatten())
.map(|part| {
part.part_number.get_or_insert(0);
part.e_tag.take().map(|tag| tag.value().to_owned())
})
.collect();
for checksum in [
&mut s3s_input.checksum_crc32,
&mut s3s_input.checksum_crc32c,
&mut s3s_input.checksum_crc64nvme,
&mut s3s_input.checksum_md5,
&mut s3s_input.checksum_sha1,
&mut s3s_input.checksum_sha256,
&mut s3s_input.checksum_sha512,
&mut s3s_input.checksum_xxhash128,
&mut s3s_input.checksum_xxhash3,
&mut s3s_input.checksum_xxhash64,
] {
*checksum = None;
}
let rejected = match complete_multipart_upload::input_from_s3s(s3s_input) {
Err(error) => Err(crate::app::dto_values::seam_error(error)),
Ok(mut owned) => {
for (part, tag) in owned.multipart_upload.parts.iter_mut().zip(tags) {
part.e_tag = tag.map(rustfs_gateway_types::ETag::from_stored);
}
let taken = std::mem::take(&mut owned.multipart_upload);
let parts = has_upload.then_some(taken);
match parts {
Some(parts) => {
owned.multipart_upload = parts;
Box::pin(usecase.execute_complete_multipart_upload(owned_envelope(Method::POST), owned)).await
}
None => {
Box::pin(usecase.execute_complete_multipart_upload_parts(owned_envelope(Method::POST), owned, None)).await
}
}
}
};
let dto_complete_err = rejected.expect_err("DTO-only If-None-Match: * on existing key must fail");
assert_eq!(dto_complete_err.code(), &S3ErrorCode::PreconditionFailed);
for condition in [ETag::Weak(committed_etag.clone()), ETag::Strong("different-etag".to_string())] {
@@ -1764,10 +1917,53 @@ mod tests {
.multipart_upload(Some(completed_parts.clone()))
.build()
.expect("If-Match complete input should build");
let err = usecase
.complete_for_s3s_request(build_request(input, Method::POST))
.await
.expect_err("weak or nonmatching If-Match must reject the overwrite");
let mut s3s_input = input;
let has_upload = s3s_input.multipart_upload.is_some();
s3s_input.multipart_upload.get_or_insert_with(Default::default);
let tags: Vec<Option<String>> = s3s_input
.multipart_upload
.iter_mut()
.flat_map(|upload| upload.parts.iter_mut().flatten())
.map(|part| {
part.part_number.get_or_insert(0);
part.e_tag.take().map(|tag| tag.value().to_owned())
})
.collect();
for checksum in [
&mut s3s_input.checksum_crc32,
&mut s3s_input.checksum_crc32c,
&mut s3s_input.checksum_crc64nvme,
&mut s3s_input.checksum_md5,
&mut s3s_input.checksum_sha1,
&mut s3s_input.checksum_sha256,
&mut s3s_input.checksum_sha512,
&mut s3s_input.checksum_xxhash128,
&mut s3s_input.checksum_xxhash3,
&mut s3s_input.checksum_xxhash64,
] {
*checksum = None;
}
let rejected = match complete_multipart_upload::input_from_s3s(s3s_input) {
Err(error) => Err(crate::app::dto_values::seam_error(error)),
Ok(mut owned) => {
for (part, tag) in owned.multipart_upload.parts.iter_mut().zip(tags) {
part.e_tag = tag.map(rustfs_gateway_types::ETag::from_stored);
}
let taken = std::mem::take(&mut owned.multipart_upload);
let parts = has_upload.then_some(taken);
match parts {
Some(parts) => {
owned.multipart_upload = parts;
Box::pin(usecase.execute_complete_multipart_upload(owned_envelope(Method::POST), owned)).await
}
None => {
Box::pin(usecase.execute_complete_multipart_upload_parts(owned_envelope(Method::POST), owned, None))
.await
}
}
}
};
let err = rejected.expect_err("weak or nonmatching If-Match must reject the overwrite");
assert_eq!(err.code(), &S3ErrorCode::PreconditionFailed);
}
@@ -1782,10 +1978,52 @@ mod tests {
.multipart_upload(Some(completed_parts.clone()))
.build()
.expect("weak If-None-Match complete multipart input should build");
let weak_complete_err = usecase
.complete_for_s3s_request(build_request(weak_if_none_match_input, Method::POST))
.await
.expect_err("weak If-None-Match ETag must be rejected as unsupported");
let mut s3s_input = weak_if_none_match_input;
let has_upload = s3s_input.multipart_upload.is_some();
s3s_input.multipart_upload.get_or_insert_with(Default::default);
let tags: Vec<Option<String>> = s3s_input
.multipart_upload
.iter_mut()
.flat_map(|upload| upload.parts.iter_mut().flatten())
.map(|part| {
part.part_number.get_or_insert(0);
part.e_tag.take().map(|tag| tag.value().to_owned())
})
.collect();
for checksum in [
&mut s3s_input.checksum_crc32,
&mut s3s_input.checksum_crc32c,
&mut s3s_input.checksum_crc64nvme,
&mut s3s_input.checksum_md5,
&mut s3s_input.checksum_sha1,
&mut s3s_input.checksum_sha256,
&mut s3s_input.checksum_sha512,
&mut s3s_input.checksum_xxhash128,
&mut s3s_input.checksum_xxhash3,
&mut s3s_input.checksum_xxhash64,
] {
*checksum = None;
}
let rejected = match complete_multipart_upload::input_from_s3s(s3s_input) {
Err(error) => Err(crate::app::dto_values::seam_error(error)),
Ok(mut owned) => {
for (part, tag) in owned.multipart_upload.parts.iter_mut().zip(tags) {
part.e_tag = tag.map(rustfs_gateway_types::ETag::from_stored);
}
let taken = std::mem::take(&mut owned.multipart_upload);
let parts = has_upload.then_some(taken);
match parts {
Some(parts) => {
owned.multipart_upload = parts;
Box::pin(usecase.execute_complete_multipart_upload(owned_envelope(Method::POST), owned)).await
}
None => {
Box::pin(usecase.execute_complete_multipart_upload_parts(owned_envelope(Method::POST), owned, None)).await
}
}
}
};
let weak_complete_err = rejected.expect_err("weak If-None-Match ETag must be rejected as unsupported");
assert_eq!(weak_complete_err.code(), &S3ErrorCode::NotImplemented);
let second_complete_input = CompleteMultipartUploadInput::builder()
@@ -2022,8 +2260,117 @@ mod tests {
}
let first_usecase = usecase.clone();
let first = first_usecase.complete_for_s3s_request(build_request(inputs.remove(0), Method::POST));
let second = usecase.complete_for_s3s_request(build_request(inputs.remove(0), Method::POST));
let first_input = inputs.remove(0);
let first = async move {
use rustfs_gateway_types::compat::s3s_5761ddfe::generated::ops::complete_multipart_upload;
let mut s3s_input = first_input;
let has_upload = s3s_input.multipart_upload.is_some();
s3s_input.multipart_upload.get_or_insert_with(Default::default);
let tags: Vec<Option<String>> = s3s_input
.multipart_upload
.iter_mut()
.flat_map(|upload| upload.parts.iter_mut().flatten())
.map(|part| {
part.part_number.get_or_insert(0);
part.e_tag.take().map(|tag| tag.value().to_owned())
})
.collect();
for checksum in [
&mut s3s_input.checksum_crc32,
&mut s3s_input.checksum_crc32c,
&mut s3s_input.checksum_crc64nvme,
&mut s3s_input.checksum_md5,
&mut s3s_input.checksum_sha1,
&mut s3s_input.checksum_sha256,
&mut s3s_input.checksum_sha512,
&mut s3s_input.checksum_xxhash128,
&mut s3s_input.checksum_xxhash3,
&mut s3s_input.checksum_xxhash64,
] {
*checksum = None;
}
match complete_multipart_upload::input_from_s3s(s3s_input) {
Err(error) => Err(crate::app::dto_values::seam_error(error)),
Ok(mut owned) => {
for (part, tag) in owned.multipart_upload.parts.iter_mut().zip(tags) {
part.e_tag = tag.map(rustfs_gateway_types::ETag::from_stored);
}
let taken = std::mem::take(&mut owned.multipart_upload);
let parts = has_upload.then_some(taken);
match parts {
Some(parts) => {
owned.multipart_upload = parts;
Box::pin(first_usecase.execute_complete_multipart_upload(owned_envelope(Method::POST), owned)).await
}
None => {
Box::pin(first_usecase.execute_complete_multipart_upload_parts(
owned_envelope(Method::POST),
owned,
None,
))
.await
}
}
}
}
};
let second_usecase = usecase;
let second_input = inputs.remove(0);
let second = async move {
use rustfs_gateway_types::compat::s3s_5761ddfe::generated::ops::complete_multipart_upload;
let mut s3s_input = second_input;
let has_upload = s3s_input.multipart_upload.is_some();
s3s_input.multipart_upload.get_or_insert_with(Default::default);
let tags: Vec<Option<String>> = s3s_input
.multipart_upload
.iter_mut()
.flat_map(|upload| upload.parts.iter_mut().flatten())
.map(|part| {
part.part_number.get_or_insert(0);
part.e_tag.take().map(|tag| tag.value().to_owned())
})
.collect();
for checksum in [
&mut s3s_input.checksum_crc32,
&mut s3s_input.checksum_crc32c,
&mut s3s_input.checksum_crc64nvme,
&mut s3s_input.checksum_md5,
&mut s3s_input.checksum_sha1,
&mut s3s_input.checksum_sha256,
&mut s3s_input.checksum_sha512,
&mut s3s_input.checksum_xxhash128,
&mut s3s_input.checksum_xxhash3,
&mut s3s_input.checksum_xxhash64,
] {
*checksum = None;
}
match complete_multipart_upload::input_from_s3s(s3s_input) {
Err(error) => Err(crate::app::dto_values::seam_error(error)),
Ok(mut owned) => {
for (part, tag) in owned.multipart_upload.parts.iter_mut().zip(tags) {
part.e_tag = tag.map(rustfs_gateway_types::ETag::from_stored);
}
let taken = std::mem::take(&mut owned.multipart_upload);
let parts = has_upload.then_some(taken);
match parts {
Some(parts) => {
owned.multipart_upload = parts;
Box::pin(second_usecase.execute_complete_multipart_upload(owned_envelope(Method::POST), owned)).await
}
None => {
Box::pin(second_usecase.execute_complete_multipart_upload_parts(
owned_envelope(Method::POST),
owned,
None,
))
.await
}
}
}
}
};
let (first, second) = tokio::join!(first, second);
assert_eq!(usize::from(first.is_ok()) + usize::from(second.is_ok()), 1);
+3 -1
View File
@@ -436,7 +436,9 @@ impl DefaultMultipartUsecase {
bucket,
key,
upload_id,
server_side_encryption: effective_sse,
// `x-amz-server-side-encryption` names a managed algorithm; an SSE-C
// upload answers with its customer-key headers only.
server_side_encryption: effective_sse.filter(|_| sse_customer_algorithm.is_none()),
sse_customer_algorithm,
sse_customer_key_md5,
ssekms_key_id: effective_kms_key_id,
@@ -402,7 +402,8 @@ impl DefaultMultipartUsecase {
}
};
write_plan = write_plan.with_encryption(ssec_write);
(Some(ssec_material.server_side_encryption), ssec_material.kms_key_id)
// An SSE-C part answers with its customer-key headers only.
(None, None)
} else if let Some(server_side_encryption) = server_side_encryption {
// Reuse the stored envelope after authorizing this caller against its
// pinned destination key above.
@@ -474,11 +474,8 @@ impl DefaultMultipartUsecase {
}
};
write_plan = write_plan.with_encryption(ssec_write);
(
Some(ssec_material.server_side_encryption),
ssec_material.kms_key_id,
mp_info.user_defined.clone(),
)
// An SSE-C part answers with its customer-key headers only.
(None, None, mp_info.user_defined.clone())
} else if let Some(server_side_encryption) = server_side_encryption {
// Reuse the destination envelope after authorizing this caller against
// its pinned key above. The source read is separately authorized.
@@ -116,9 +116,32 @@ fn upload_part_copy_shares_foreground_admission_with_upload_part() {
.expect("list parts");
assert!(parts.parts.is_empty(), "rejected copy must not create a destination part");
drop(held);
let copied = Box::pin(usecase.execute_upload_part_copy(build_request(input, Method::PUT)))
.await
.expect("retry copy after releasing UploadPart permit");
let copied = match Box::pin(usecase.execute_upload_part_copy_with_audit(
owned_envelope(Method::PUT),
UploadPartCopyWriteInput {
target: UploadPartTarget::Legacy {
bucket: input.bucket,
key: input.key,
upload_id: input.upload_id,
},
part_number: input.part_number,
copy_source: rustfs_gateway_types::compat::s3s_5761ddfe::copy_source::from_s3s(input.copy_source),
copy_source_range: input.copy_source_range,
copy_source_if_match: input.copy_source_if_match.map(&condition),
copy_source_if_none_match: input.copy_source_if_none_match.map(&condition),
sse_customer_algorithm: input.sse_customer_algorithm,
sse_customer_key: input.sse_customer_key,
sse_customer_key_md5: input.sse_customer_key_md5,
copy_source_sse_customer_algorithm: input.copy_source_sse_customer_algorithm,
copy_source_sse_customer_key: input.copy_source_sse_customer_key,
copy_source_sse_customer_key_md5: input.copy_source_sse_customer_key_md5,
},
))
.await
{
Ok(output) => output,
Err(error) => panic!("retry copy after releasing UploadPart permit: {error:?}"),
};
assert!(source_reader_observed());
assert_eq!(
manager.put_object_admission_snapshot().active,
@@ -126,14 +149,11 @@ fn upload_part_copy_shares_foreground_admission_with_upload_part() {
"success releases the copy permit"
);
let etag = copied
.output
.copy_part_result
.expect("part result")
.e_tag
.expect("copy ETag")
.value()
.trim_matches('"')
.to_owned();
.as_deref()
.map(crate::storage::storage_api::to_s3s_xml_etag)
.expect("copy ETag");
let etag = etag.value().trim_matches('"').to_owned();
Arc::clone(&store)
.complete_multipart_upload(
&bucket,
+71 -1
View File
@@ -904,7 +904,9 @@ impl DefaultObjectUsecase {
last_modified: oi.mod_time,
copy_source_version_id,
version_id: dest_version,
server_side_encryption: effective_sse,
// `x-amz-server-side-encryption` names a managed algorithm; an SSE-C
// destination answers with its customer-key headers only.
server_side_encryption: effective_sse.filter(|_| sse_customer_algorithm.is_none()),
ssekms_key_id: effective_kms_key_id,
sse_customer_algorithm,
sse_customer_key_md5,
@@ -1483,6 +1485,74 @@ mod tests {
.expect("directory copy test bucket should be removed");
}
// `x-amz-server-side-encryption` names a managed algorithm. A copy to an SSE-C destination
// answers with its customer-key headers only, as PutObject does.
#[tokio::test]
#[serial_test::serial]
async fn copy_object_to_ssec_destination_answers_with_the_customer_key_headers_only() {
use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions};
let store = crate::app::gating_test_env::shared_gating_ecstore().await;
if current_app_context().is_none() {
crate::app::runtime_sources::install_test_app_context(Arc::clone(&store)).await;
}
let ambient = current_app_context().expect("SSE-C copy test requires an AppContext");
let context = Arc::new(AppContext::new(Arc::clone(&store), ambient.iam(), ambient.kms()));
let bucket = format!("copy-ssec-response-{}", Uuid::new_v4());
let customer_key_md5 = "SaaDheEjzuynJH8eW6AEpQ==";
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("SSE-C copy test bucket should be created");
store
.put_object(
&bucket,
"source.bin",
&mut PutObjReader::from_vec(b"copy source".to_vec()),
&ObjectOptions::default(),
)
.await
.expect("plaintext copy source should be written");
let mut input = CopyObjectInput::builder()
.copy_source(CopySource::Bucket {
bucket: bucket.clone().into(),
key: "source.bin".into(),
version_id: None,
})
.bucket(bucket.clone())
.key("customer-encrypted.bin".to_string())
.build()
.expect("SSE-C copy input should build");
input.sse_customer_algorithm = Some("AES256".to_string());
input.sse_customer_key = Some("KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKio=".to_string());
input.sse_customer_key_md5 = Some(customer_key_md5.to_string());
let response = copy_owned(&DefaultObjectUsecase::with_context(Some(context)), build_request(input, Method::PUT))
.await
.expect("copying to an SSE-C destination should succeed");
assert_eq!(
(
response.server_side_encryption.as_deref(),
response.ssekms_key_id.as_deref(),
response.sse_customer_algorithm.as_deref(),
response.sse_customer_key_md5.as_deref(),
),
(None, None, Some("AES256"), Some(customer_key_md5))
);
store
.delete_bucket(
&bucket,
&DeleteBucketOptions {
force: true,
..Default::default()
},
)
.await
.expect("SSE-C copy test bucket should be removed");
}
#[tokio::test]
async fn execute_copy_object_allows_self_copy_of_null_version() {
// A "null" source version id is a restore of the null version, not a no-op self-copy.