Compare commits

...

2 Commits

Author SHA1 Message Date
唐小鸭 cdc5563ccc fix(rio-v2): fail multipart DARE reads that end before a listed part
A multipart DARE stream is the concatenation of each part's package
sequence. When the stream ended exactly on a part boundary, right after a
final-flagged package, DecryptReader had read no package of the next part
and took the clean EOF for an empty object, so a stream missing whole
trailing parts read back as complete. A full-object read was still caught
by the outer length limit, but a ranged read returned a short body without
an error.

The reader could not tell a missing part from an empty one, because an
empty part stores no package. The caller now supplies that distinction:
the multipart constructors document that only parts with stored bytes are
listed, a clean EOF before a listed part's first package is an
`UnexpectedEof`, and the ecstore read path drops zero-size parts from the
list it hands to the rio-v2 reader. An object whose last part is empty
keeps reading back, which the new ecstore test pins down.
2026-11-22 23:17:59 +08:00
唐小鸭 10561f6757 fix(rio-v2): resume DARE package reads after a mid-ciphertext Pending
DecryptReader reset `ciphertext_read` to zero every time it entered the
ciphertext loop. When the inner reader returned `Pending` partway through
a package, the next poll skipped the already-buffered header and refilled
the ciphertext buffer from its start, overwriting the bytes read before
the `Pending` and shifting the rest of the stream. A single-package object
then ended in "unexpected EOF while reading DARE ciphertext" and a
multi-package object in "DARE authentication failed".

The legacy GET path feeds the decrypt reader from a bounded duplex pipe,
so every rio-v2 encrypted object with a package straddling a point where
the reader outruns the producer failed to read back: stored sizes of
64 KiB + 1 to 128 KiB (64 KiB pipe, smaller than one package), 512 KiB + 1
to 1 MiB (512 KiB pipe), and larger objects whenever the next erasure
block arrived late. Stored data was never affected, so objects written
before this change read back correctly.

Keep the counter across polls: it starts at zero and is cleared only once
a package has been opened. The regression tests cover the reader behind a
pipe that reports `Pending` at each refill, and the GET reader stack
behind the pipe size production picks for inline-range, multi-megabyte,
and multipart objects.
2026-11-22 21:55:48 +08:00
3 changed files with 424 additions and 10 deletions
+23 -6
View File
@@ -974,12 +974,18 @@ impl ReadPlan {
let _ = sequence_number;
let decrypted_reader: Box<dyn AsyncRead + Unpin + Send + Sync> = if is_multipart {
match material.mode {
ReadEncryptionMode::Object => crate::io_support::rio::decrypt_multipart_reader_with_object_key(
reader,
material.key_bytes,
part_numbers,
sequence_number,
),
ReadEncryptionMode::Object => {
// The rio-v2 reader fails a stream that ends before a
// listed part, so it must not hear about empty ones.
#[cfg(feature = "rio-v2")]
let part_numbers = without_empty_parts(part_numbers, &oi.parts);
crate::io_support::rio::decrypt_multipart_reader_with_object_key(
reader,
material.key_bytes,
part_numbers,
sequence_number,
)
}
ReadEncryptionMode::Direct { base_nonce } => crate::io_support::rio::decrypt_multipart_reader(
reader,
material.key_bytes,
@@ -1434,6 +1440,17 @@ fn multipart_part_numbers(parts: &[ObjectPartInfo]) -> Vec<usize> {
parts.iter().map(|part| part.number).collect()
}
/// Drops from `part_numbers` the parts that stored no bytes: a DARE writer
/// emits no package for an empty part, so none follows on disk.
#[cfg(feature = "rio-v2")]
fn without_empty_parts(mut part_numbers: Vec<usize>, parts: &[ObjectPartInfo]) -> Vec<usize> {
let empty_parts: Vec<usize> = parts.iter().filter(|part| part.size == 0).map(|part| part.number).collect();
if !empty_parts.is_empty() {
part_numbers.retain(|number| !empty_parts.contains(number));
}
part_numbers
}
#[cfg(test)]
mod tests {
use super::*;
+218
View File
@@ -7194,6 +7194,224 @@ mod legacy_duplex_producer_reader_tests {
}
}
#[cfg(feature = "rio-v2")]
struct ObjectKeyResolver {
object_key: [u8; 32],
}
#[cfg(feature = "rio-v2")]
#[async_trait::async_trait]
impl ObjectEncryptionResolver for ObjectKeyResolver {
async fn resolve_read_material(
&self,
_request: crate::object_api::ReadEncryptionRequest<'_>,
) -> std::result::Result<Option<ReadEncryptionMaterial>, EncryptionResolutionError> {
Ok(Some(ReadEncryptionMaterial {
key_bytes: self.object_key,
mode: ReadEncryptionMode::Object,
}))
}
}
#[cfg(feature = "rio-v2")]
const RIO_V2_FIXTURE_OBJECT_KEY: [u8; 32] = [0x67; 32];
#[cfg(feature = "rio-v2")]
fn rio_v2_fixture_plaintext(len: usize, seed: usize) -> Vec<u8> {
(0..len)
.map(|index| u8::try_from((index + seed) % 251).expect("a remainder below 251 fits in u8"))
.collect()
}
/// Reads a rio-v2 encrypted object the way the legacy GET path does: a
/// producer task feeds the stored bytes into a pipe sized by
/// `adaptive_duplex_buffer_size`, so the decrypt reader sees `Pending`
/// wherever a DARE package outruns the pipe. `parts` lists each part's
/// `(plaintext, stored)` length as the object metadata records it, and
/// `stored` is what the drives actually hold, which a test may cut short.
#[cfg(feature = "rio-v2")]
async fn read_rio_v2_object_through_get_pipe(
stored: Vec<u8>,
parts: &[(usize, usize)],
range: Option<HTTPRangeSpec>,
) -> std::io::Result<Vec<u8>> {
let plaintext_size: usize = parts.iter().map(|(plaintext_len, _)| plaintext_len).sum();
let multipart = parts.len() > 1;
let object_info = ObjectInfo {
bucket: "bucket".to_string(),
name: "rio-v2-encrypted-object".to_string(),
size: i64::try_from(parts.iter().map(|(_, stored_len)| stored_len).sum::<usize>())
.expect("stored fixture length should fit in i64"),
etag: multipart.then(|| format!("{}-{}", "0".repeat(32), parts.len())),
parts: Arc::new(if multipart {
parts
.iter()
.enumerate()
.map(|(index, (plaintext_len, stored_len))| rustfs_filemeta::ObjectPartInfo {
number: index + 1,
size: *stored_len,
actual_size: i64::try_from(*plaintext_len).expect("part plaintext length should fit in i64"),
..Default::default()
})
.collect()
} else {
Vec::new()
}),
user_defined: Arc::new(HashMap::from([
("x-amz-server-side-encryption-customer-algorithm".to_string(), "AES256".to_string()),
(
"x-amz-server-side-encryption-customer-original-size".to_string(),
plaintext_size.to_string(),
),
])),
..Default::default()
};
let ctx = InstanceContext::new();
assert!(
ctx.set_object_encryption_resolver(Arc::new(ObjectKeyResolver {
object_key: RIO_V2_FIXTURE_OBJECT_KEY
}))
.is_ok(),
"fresh context should accept resolver"
);
let (mut writer, reader) = tokio::io::duplex(adaptive_duplex_buffer_size(object_info.size));
let (terminal_tx, terminal_rx) = tokio::sync::oneshot::channel();
let (mut reader, storage_offset, storage_length) = get_legacy_object_reader_with_context(
&ctx,
reader,
terminal_rx,
range,
&object_info,
&ObjectOptions::default(),
&HeaderMap::new(),
)
.await
.expect("encrypted read plan should build");
let storage_length = usize::try_from(storage_length).expect("storage length should not be negative");
let producer = tokio::spawn(async move {
let window_end = stored.len().min(storage_offset + storage_length);
let window_start = storage_offset.min(window_end);
let written = writer.write_all(&stored[window_start..window_end]).await;
drop(writer);
// The reader may already have failed and dropped the receiver.
let _ = terminal_tx.send(Ok(()));
written
});
let mut out = Vec::new();
let read = reader.read_to_end(&mut out).await;
drop(reader);
// A failed read drops the pipe early, so the producer's own result only
// matters once the read itself succeeded.
let written = producer.await.expect("producer task should not panic");
read?;
written?;
Ok(out)
}
#[cfg(feature = "rio-v2")]
#[tokio::test]
async fn rio_v2_encrypted_get_survives_pipe_refills_mid_package() {
const MIB: usize = 1024 * 1024;
// Stored sizes up to 128 KiB get a 64 KiB pipe, smaller than one DARE
// package; 3 MiB objects get a 1 MiB pipe that packages straddle.
let mut failures = Vec::new();
for plaintext_len in [65_535, 65_536, 65_537, 71_680, 3 * MIB, 3 * MIB + 1] {
let plaintext = rio_v2_fixture_plaintext(plaintext_len, 0);
let mut stored = Vec::new();
crate::io_support::rio::EncryptReader::new_with_object_key(
std::io::Cursor::new(plaintext.clone()),
RIO_V2_FIXTURE_OBJECT_KEY,
)
.read_to_end(&mut stored)
.await
.expect("encrypt singlepart fixture");
let stored_len = stored.len();
match read_rio_v2_object_through_get_pipe(stored, &[(plaintext_len, stored_len)], None).await {
Ok(out) if out == plaintext => {}
Ok(_) => failures.push(format!("{plaintext_len}-byte object: plaintext mismatch")),
Err(err) => failures.push(format!("{plaintext_len}-byte object: {err}")),
}
}
assert!(failures.is_empty(), "rio-v2 encrypted GET failed for: {failures:#?}");
}
#[cfg(feature = "rio-v2")]
#[tokio::test]
async fn rio_v2_encrypted_multipart_get_survives_pipe_refills_mid_package() {
const MIB: usize = 1024 * 1024;
let part_plaintexts = [rio_v2_fixture_plaintext(5 * MIB, 1), rio_v2_fixture_plaintext(MIB + 17, 2)];
let (stored, parts) = rio_v2_multipart_fixture(&part_plaintexts).await;
let out = read_rio_v2_object_through_get_pipe(stored, &parts, None)
.await
.expect("read multipart object through the GET pipe");
assert!(out == part_plaintexts.concat(), "multipart plaintext mismatch");
}
/// Encrypts each part with its own part key and returns the concatenated
/// stored stream with every part's `(plaintext, stored)` length.
#[cfg(feature = "rio-v2")]
async fn rio_v2_multipart_fixture(part_plaintexts: &[Vec<u8>]) -> (Vec<u8>, Vec<(usize, usize)>) {
let mut stored = Vec::new();
let mut parts = Vec::new();
for (part_number, part_plaintext) in (1u32..).zip(part_plaintexts) {
let stored_before = stored.len();
crate::io_support::rio::EncryptReader::new_multipart_with_object_key(
std::io::Cursor::new(part_plaintext.clone()),
RIO_V2_FIXTURE_OBJECT_KEY,
part_number,
)
.read_to_end(&mut stored)
.await
.expect("encrypt multipart fixture part");
parts.push((part_plaintext.len(), stored.len() - stored_before));
}
(stored, parts)
}
#[cfg(feature = "rio-v2")]
#[tokio::test]
async fn rio_v2_encrypted_multipart_range_get_rejects_a_missing_trailing_part() {
let part_plaintexts = [rio_v2_fixture_plaintext(70_000, 1), rio_v2_fixture_plaintext(140_000, 2)];
let (mut stored, parts) = rio_v2_multipart_fixture(&part_plaintexts).await;
// The drives hold part 1 only, although the metadata still lists part 2.
stored.truncate(parts[0].1);
// The range starts in part 1 and ends inside the missing part 2.
let range = HTTPRangeSpec {
is_suffix_length: false,
start: 69_900,
end: 70_099,
};
let outcome = read_rio_v2_object_through_get_pipe(stored, &parts, Some(range)).await;
match outcome {
Err(err) => assert_eq!(err.kind(), std::io::ErrorKind::UnexpectedEof, "{err}"),
Ok(out) => panic!("a 200-byte range over a missing part read {} bytes as a clean EOF", out.len()),
}
}
#[cfg(feature = "rio-v2")]
#[tokio::test]
async fn rio_v2_encrypted_multipart_get_accepts_an_empty_trailing_part() {
let part_plaintexts = [rio_v2_fixture_plaintext(70_000, 1), Vec::new()];
let (stored, parts) = rio_v2_multipart_fixture(&part_plaintexts).await;
assert_eq!(parts[1], (0, 0), "an empty part stores no DARE package");
let out = read_rio_v2_object_through_get_pipe(stored, &parts, None)
.await
.expect("an empty trailing part is not a truncated stream");
assert!(out == part_plaintexts[0], "plaintext mismatch for an object with an empty trailing part");
}
#[tokio::test]
async fn legacy_duplex_reader_fails_closed_when_terminal_channel_closes() {
let (mut writer, reader) = tokio::io::duplex(64);
+183 -4
View File
@@ -334,14 +334,23 @@ where
}
}
/// See [`Self::new_multipart_with_sequence`] for the `multipart_parts` contract.
pub fn new_multipart(inner: R, key: [u8; 32], base_nonce: [u8; 12], multipart_parts: Vec<usize>) -> Self {
Self::new_multipart_with_sequence(inner, key, base_nonce, multipart_parts, 0)
}
/// See [`Self::new_multipart_with_object_key_and_sequence`] for the `multipart_parts` contract.
pub fn new_multipart_with_object_key(inner: R, object_key: [u8; 32], multipart_parts: Vec<usize>) -> Self {
Self::new_multipart_with_object_key_and_sequence(inner, object_key, multipart_parts, 0)
}
/// Decrypts the concatenated package sequences of `multipart_parts`, in
/// order, starting at package `sequence_number` of the first one.
///
/// List only the parts that stored bytes: an empty part has no package on
/// disk, and reading fails with [`io::ErrorKind::UnexpectedEof`] when the
/// stream ends before a listed part's final package, including on the
/// boundary before a listed part.
pub fn new_multipart_with_sequence(
inner: R,
key: [u8; 32],
@@ -373,6 +382,14 @@ where
}
}
/// Decrypts the concatenated package sequences of `multipart_parts`, in
/// order, deriving each part key from `object_key` and starting at package
/// `sequence_number` of the first one.
///
/// List only the parts that stored bytes: an empty part has no package on
/// disk, and reading fails with [`io::ErrorKind::UnexpectedEof`] when the
/// stream ends before a listed part's final package, including on the
/// boundary before a listed part.
pub fn new_multipart_with_object_key_and_sequence(
inner: R,
object_key: [u8; 32],
@@ -466,15 +483,24 @@ where
// with `finalized == false` (the finalized case is consumed at the
// loop top). If at least one package of the current part has been
// decrypted (`ref_nonce.is_some()`) but we never saw a final-flagged
// package, the final package is missing => DARE truncation. Zero
// decrypted packages (`ref_nonce.is_none()`) is a legitimately empty
// object (encrypt emits no packages for empty plaintext), so accept.
// package, the final package is missing => DARE truncation.
if this.ref_nonce.is_some() {
return Poll::Ready(Err(io::Error::new(
io::ErrorKind::UnexpectedEof,
"DARE stream truncated before a finalized package",
)));
}
// Zero decrypted packages. A listed part always carries at least
// one package, so the stream ended before that part: without this
// a stream cut exactly on a part boundary reads as complete.
if let Some(missing_part) = this.multipart_parts.get(*this.current_part_index) {
return Poll::Ready(Err(io::Error::new(
io::ErrorKind::UnexpectedEof,
format!("DARE stream ended before multipart part {missing_part}"),
)));
}
// No part is expected here: a legitimately empty object (encrypt
// emits no packages for empty plaintext), so accept.
*this.finished = true;
return Poll::Ready(Ok(()));
}
@@ -513,8 +539,11 @@ where
if this.ciphertext_buf.len() < package_len {
this.ciphertext_buf.resize(package_len, 0);
}
// `ciphertext_read` is zero when a package starts and is reset only
// once that package is opened: a `Pending` below re-enters here with
// the header already buffered and must resume after the ciphertext
// already read rather than refill the buffer from its start.
*this.ciphertext_len = package_len;
*this.ciphertext_read = 0;
while *this.ciphertext_read < *this.ciphertext_len {
let mut read_buf = ReadBuf::new(&mut this.ciphertext_buf[*this.ciphertext_read..*this.ciphertext_len]);
@@ -984,4 +1013,154 @@ mod tests {
.expect_err("multipart stream truncated within a part must fail");
assert_eq!(err.kind(), io::ErrorKind::UnexpectedEof);
}
#[tokio::test]
async fn multipart_decrypt_reader_rejects_a_stream_missing_a_listed_part() {
let object_key = [0x64u8; 32];
let part_plaintexts = [vec![0xA1; DARE_PAYLOAD_SIZE + 31], vec![0xB2; DARE_PAYLOAD_SIZE * 2 + 7]];
let mut encrypted_parts = Vec::new();
for (part_number, part_plaintext) in (1u32..).zip(&part_plaintexts) {
let mut encrypted = Vec::new();
EncryptReader::new_multipart_with_object_key(Cursor::new(part_plaintext.clone()), object_key, part_number)
.read_to_end(&mut encrypted)
.await
.expect("encrypt multipart part");
encrypted_parts.push(encrypted);
}
// Every stream below ends on a part boundary, right after a
// final-flagged package, with a listed part still undelivered.
for (stream, listed_parts) in [
(encrypted_parts[0].clone(), vec![1, 2]),
(encrypted_parts.concat(), vec![1, 2, 3]),
(Vec::new(), vec![1]),
] {
let mut sink = Vec::new();
let outcome = DecryptReader::new_multipart_with_object_key(Cursor::new(stream), object_key, listed_parts.clone())
.read_to_end(&mut sink)
.await;
match outcome {
Err(err) => assert_eq!(err.kind(), io::ErrorKind::UnexpectedEof, "parts {listed_parts:?}: {err}"),
Ok(read) => panic!("parts {listed_parts:?}: a stream missing a listed part read {read} bytes as a clean EOF"),
}
}
}
#[tokio::test]
async fn multipart_decrypt_reader_accepts_an_empty_stream_without_listed_parts() {
let mut decrypted = Vec::new();
DecryptReader::new_multipart_with_object_key(Cursor::new(Vec::new()), [0x64u8; 32], Vec::new())
.read_to_end(&mut decrypted)
.await
.expect("an object whose parts all stored no bytes is empty, not truncated");
assert!(decrypted.is_empty());
}
/// A bounded pipe whose producer lags the consumer: it hands out at most
/// `capacity` bytes, then reports `Pending` once before the next refill.
struct RefillingPipe {
data: Vec<u8>,
position: usize,
capacity: usize,
available: usize,
}
impl RefillingPipe {
fn new(data: Vec<u8>, capacity: usize) -> Self {
Self {
data,
position: 0,
capacity,
available: capacity,
}
}
}
impl AsyncRead for RefillingPipe {
fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
let unread = self.data.len() - self.position;
if unread > 0 && self.available == 0 {
self.available = self.capacity;
cx.waker().wake_by_ref();
return Poll::Pending;
}
let len = min(min(unread, self.available), buf.remaining());
let start = self.position;
buf.put_slice(&self.data[start..start + len]);
self.position += len;
self.available -= len;
Poll::Ready(Ok(()))
}
}
fn patterned_plaintext(len: usize, seed: usize) -> Vec<u8> {
(0..len)
.map(|index| u8::try_from((index + seed) % 251).expect("a remainder below 251 fits in u8"))
.collect()
}
#[tokio::test]
async fn decrypt_reader_resumes_a_package_after_pending_mid_ciphertext() {
const KIB: usize = 1024;
const MIB: usize = 1024 * KIB;
let object_key = [0x3Du8; 32];
// (plaintext length, pipe capacity). The first group is a single package
// slightly larger than a 64 KiB pipe; the second is a multi-package stream
// whose packages straddle 1 MiB refills; the last drains byte by byte so
// every header and ciphertext position sees a `Pending`.
let mut failures = Vec::new();
for (plaintext_len, capacity) in [
(DARE_PAYLOAD_SIZE - 1, 64 * KIB),
(DARE_PAYLOAD_SIZE, 64 * KIB),
(DARE_PAYLOAD_SIZE + 1, 64 * KIB),
(70 * KIB, 64 * KIB),
(3 * MIB, MIB),
(3 * MIB + 1, MIB),
(DARE_PAYLOAD_SIZE + 17, 1),
] {
let plaintext = patterned_plaintext(plaintext_len, capacity);
let mut encrypted = Vec::new();
EncryptReader::new_with_object_key(Cursor::new(plaintext.clone()), object_key)
.read_to_end(&mut encrypted)
.await
.expect("encrypt singlepart stream");
let mut decrypted = Vec::new();
let outcome = DecryptReader::new_with_object_key(RefillingPipe::new(encrypted, capacity), object_key)
.read_to_end(&mut decrypted)
.await;
match outcome {
Ok(_) if decrypted == plaintext => {}
Ok(_) => failures.push(format!("{plaintext_len} bytes through a {capacity}-byte pipe: plaintext mismatch")),
Err(err) => failures.push(format!("{plaintext_len} bytes through a {capacity}-byte pipe: {err}")),
}
}
assert!(failures.is_empty(), "decrypt failed for: {failures:#?}");
}
#[tokio::test]
async fn multipart_decrypt_reader_resumes_a_package_after_pending_mid_ciphertext() {
const MIB: usize = 1024 * 1024;
let object_key = [0x5Eu8; 32];
let part_plaintexts = [patterned_plaintext(5 * MIB, 1), patterned_plaintext(MIB + 17, 2)];
let mut encrypted = Vec::new();
for (part_number, part_plaintext) in (1u32..).zip(&part_plaintexts) {
EncryptReader::new_multipart_with_object_key(Cursor::new(part_plaintext.clone()), object_key, part_number)
.read_to_end(&mut encrypted)
.await
.expect("encrypt multipart part");
}
let mut decrypted = Vec::new();
DecryptReader::new_multipart_with_object_key(RefillingPipe::new(encrypted, MIB), object_key, vec![1, 2])
.read_to_end(&mut decrypted)
.await
.expect("decrypt multipart stream through a 1 MiB pipe");
assert!(decrypted == part_plaintexts.concat(), "multipart plaintext mismatch through a 1 MiB pipe");
}
}