fix(ecstore): bound listing walks by drive stall, not total duration (#4647)

Foreground S3 listings wrapped the entire walk_dir stream in a single
wall-clock timeout (RUSTFS_DRIVE_WALKDIR_TIMEOUT_SECS, default 5s). That
budget measured how much data the walk had to produce rather than whether
the drive was still answering, so a healthy but large prefix on slow media
returned 500 InternalError / "Io error: timeout" to the client. The timer
also kept running while the walk was blocked writing to a slow consumer,
charging merge-side backpressure to the producer.

WalkDirOptions::stall_timeout_ms already carried the right semantics but was
only honored by the remote-disk RPC walk; LocalDisk::walk_dir ignored it
entirely. A single-drive deployment therefore had no way to distinguish a
hung drive from a big directory.

Teach LocalDisk::walk_dir to bound each individual drive read with the stall
budget, defaulting it from RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS when the
caller does not pin one, and let the foreground listing path skip the
wrapper-level total timeout. A walk that keeps making progress now runs to
completion; a drive that stops answering still fails with DiskError::Timeout.
Time spent blocked on the consumer stays outside the budget.

Heal and rebalance walks already skipped the total timeout and previously ran
unbounded on local drives. Give them an explicit, generous 60s stall budget so
this change does not tighten them from "no bound" to the 5s default.

Fixes #4644
Refs #2999, #3001

Co-authored-by: heihutu <heihutu@gmail.com>

crates/ecstore/src/set_disk/ops/heal_walk.rs

@@ -25,6 +25,12 @@
 use super::super::*;
 use std::sync::Mutex;
 use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
+use std::time::Duration;
+
+/// Background walks skip the total timeout, so the per-read stall budget is what
+/// catches a drive that stops answering. Keep it generous: a heal walk is not
+/// latency-sensitive, and one slow read is not a dead drive.
+const BACKGROUND_WALKDIR_STALL_TIMEOUT: Duration = Duration::from_secs(60);
 
 /// A single `(object, version)` unit surfaced by the disk-walk union enumerator.
 ///
@@ -199,6 +205,7 @@ impl SetDisks {
             report_not_found: false,
             per_disk_limit: 0,
             skip_walkdir_total_timeout: true,
+            walkdir_stall_timeout: Some(BACKGROUND_WALKDIR_STALL_TIMEOUT),
             agreed: Some(Box::new(move |entry: MetaCacheEntry| {
                 let collector = agreed_collector.clone();
                 Box::pin(async move {