fix(ecstore): correct codec-streaming byte accounting and partNumber routing (#4535)

Two correctness defects on the opt-in codec-streaming GET path.

ECA-02 (#943): ErasureDecodeReader only decremented `remaining` for the
main fill buffer. Under the default DualInFlight policy each fill also
produces a queued stripe that is delivered to the client via
`prefetched_bufs.pop_front()` without touching `remaining`, so any object
larger than one erasure block finished with `remaining > 0` and the GET
terminated with LessData despite delivering all bytes. The inflated
`remaining` was also fed back into the fill worker, which used it to trim
the final stripe and to decide whether to read past EOF. Account for the
queued-stripe bytes when they enter the prefetch queue; queued buffers
come only from `Ok(true)` decodes so they are non-empty and bounded by
`remaining - main_buf.len()`, ruling out underflow.

ECA-04 (#945): the codec-streaming gate did not inspect `opts.part_number`.
A partNumber GET carries `range == None`, so it was not classified as a
Range request and reached the full-object codec-streaming reader, which
drops the storage offset/length returned by GetObjectReader::new. A
partNumber >= 2 request would then stream the whole object. Mirror the
direct-memory part_number fallback and route any partNumber request back
to the legacy duplex path, which applies the offset/length correctly.

Regression tests: DualInFlight read_to_end on a multi-block object and on
a non-block-aligned object; SingleInFlight vs DualInFlight byte-identical
output; gate fallback on partNumber requests.

Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
2026-07-09 01:35:39 +08:00
committed by GitHub
parent 5a372557e5
commit 15808254d3
4 changed files with 174 additions and 9 deletions
+17
View File
@@ -974,6 +974,7 @@ enum GetCodecStreamingFallbackReason {
HeaderCompatibilityUnconfirmed,
LockOptimizationDisabled,
Range,
PartNumber,
BelowMinSize,
Encrypted,
Compressed,
@@ -994,6 +995,7 @@ impl GetCodecStreamingFallbackReason {
Self::HeaderCompatibilityUnconfirmed => "header_compatibility_unconfirmed",
Self::LockOptimizationDisabled => "lock_optimization_disabled",
Self::Range => "range",
Self::PartNumber => "part_number",
Self::BelowMinSize => "below_min_size",
Self::Encrypted => "encrypted",
Self::Compressed => "compressed",
@@ -1298,6 +1300,7 @@ fn get_codec_streaming_reader_gate(
bucket: &str,
object: &str,
range: &Option<HTTPRangeSpec>,
part_number: Option<usize>,
object_info: &ObjectInfo,
fi: &FileInfo,
lock_optimization_enabled: bool,
@@ -1346,6 +1349,20 @@ fn get_codec_streaming_reader_gate(
prefer_data_blocks_first_reader_setup: false,
};
}
// A partNumber GET arrives with `range == None`, so it is not caught by the
// Range class above, yet it still requires a non-zero storage offset/length
// (synthesized from the part size). The codec-streaming path builds a
// full-object reader and drops the offset/length returned by
// `GetObjectReader::new`, so partNumber >= 2 would stream the whole object.
// Mirror the direct-memory part_number fallback and route these requests back
// to the legacy duplex path, which applies the offset/length correctly.
if part_number.is_some() {
return GetCodecStreamingGate {
object_class,
decision: GetCodecStreamingDecision::Fallback(GetCodecStreamingFallbackReason::PartNumber),
prefer_data_blocks_first_reader_setup: false,
};
}
if !lock_optimization_enabled {
return GetCodecStreamingGate {
object_class,