fix(ecstore): box the legacy GET producer read pipeline

Since eca95f297 (#9304) every GET through the legacy duplex pipeline
built `run_legacy_producer_segment` futures that embedded the whole
`get_object_with_fileinfo` read future inline, wrapped again in an
`async move` block and the shard-emitter scope. `get_object_reader`
builds one such producer per downstream arm and per segment, so its
debug poll frame grew to 641,888 bytes. Any test that read an object
back through a real store on a 2 MiB thread overflowed the stack and
aborted (28 workspace tests and one e2e test on main, e.g.
rustfs-heal mrf_partial_write_test, heal_integration_test, and
rustfs app::object::internal_put::tests).

The read future is now boxed inside `run_legacy_producer_segment`, so
the producer future is pointer-sized. No behaviour change.

Measured (debug, aarch64-apple-darwin, line-tables-only):
- run_legacy_producer_segment future: 110,200 -> 320 bytes
- <SetDisks as ObjectIO>::get_object_reader poll frame:
  641,888 -> 202,352 bytes (prologue stack allocation)

Regression test (negative): set_disk::ops::object::
legacy_producer_stack_tests::legacy_producer_future_keeps_the_read_pipeline_off_the_stack
asserts the producer future is at most 1024 bytes. Mutation: removing
the Box::pin makes it fail with "got 110200 bytes".

Verified locally: the 7 previously aborting rustfs-heal tests pass;
the rustfs lib tests that aborted on main (app::object::internal_put,
on_demand_migration_put, put sse_c, owned_authorization_test,
storage::access, object_zip_download, directory_index, data_usage
snapshot gating, snowball extract) pass, 94 of 94 selected.
clippy -p rustfs-ecstore --all-targets reports only the existing
unused PoolSpaceForTest items in store/rebalance.rs, not this change.
Not run locally: the full workspace, the e2e test
sts_query_compat_test::test_bucket_tags_opa_contract, and release-build
frame sizes.
This commit is contained in:
overtrue
2026-11-13 13:52:34 +08:00
parent f6d64bc1fb
commit c83f47871d
+42 -2
View File
@@ -3859,7 +3859,10 @@ async fn run_legacy_producer_segment(
// Disk read timeouts must be enforced at the actual disk I/O operations.
// A sink that can take shards out of arena memory (zero-copy emit) is
// offered to the decode loop for the whole read.
let read = crate::set_disk::with_get_object_read_policy(
// The read future is boxed so the producer future stays pointer-sized:
// `get_object_reader` builds one producer per arm (and per segment), and
// embedding the whole read pipeline inline there overflows a 2 MiB stack.
let read = Box::pin(crate::set_disk::with_get_object_read_policy(
context.read_policy,
SetDisks::get_object_with_fileinfo(
&context.bucket,
@@ -3881,7 +3884,7 @@ async fn run_legacy_producer_segment(
context.object_class,
context.size_bucket,
),
);
));
let read = async move {
match emitter {
Some(emitter) => crate::erasure::codec::shard_emit::with_shard_emitter(emitter, read).await,
@@ -4058,6 +4061,43 @@ mod sink_segment_plan_tests {
}
}
#[cfg(test)]
mod legacy_producer_stack_tests {
use super::*;
/// `get_object_reader` builds a producer future per arm and per segment;
/// each one inlined the whole read pipeline and the GET path overflowed a
/// 2 MiB debug stack. The producer must keep that pipeline on the heap.
#[test]
fn legacy_producer_future_keeps_the_read_pipeline_off_the_stack() {
let context = Arc::new(LegacyProducerContext {
bucket: "bucket".to_owned(),
object: "object".to_owned(),
erasure_cache: Arc::new(super::super::super::ErasureCache::new()),
fi: FileInfo::default(),
files: Vec::new(),
disks: Vec::new(),
set_index: 0,
pool_index: 0,
skip_verify: false,
suppress_read_repair: false,
read_policy: Default::default(),
object_class: "plain",
size_bucket: "small",
cancellation: None,
});
let writer = GetObjectSinkWriter::new(Box::new(tokio::io::sink()));
let future = run_legacy_producer_segment(context, 0, 0, writer);
let future_size = std::mem::size_of_val(&future);
assert!(
future_size <= 1024,
"legacy GET producer future must stay stack-bounded, got {future_size} bytes"
);
}
}
/// `ReplicationState::target_delete_marker_version_ids` is skipped by the
/// positional `FileInfo` wire form, so a remote disk would otherwise receive a
/// delete with an empty map and lose the exact per-target version. Copy it into