fix(ecstore): bound remote shard writers with a progress deadline so one black-hole peer cannot pin write quorum (#4925)

A PUT that fans out erasure shards to remote peers awaited every shard writer to completion on both the per-block write and the final shutdown, and the remote HttpWriter had no progress deadline. A peer that accepts the TCP connection but never drains the request body (or never sends a response) therefore wedges the writer forever once the bounded buffers fill, pinning an otherwise-healthy write quorum indefinitely — a cluster-level write-availability hazard triggered by a single bad peer (rustfs/backlog#1319, https://github.com/rustfs/backlog/issues/1319).

MultiWriter now wraps each shard write and each shard-writer shutdown in a forward-progress deadline. The budget is re-armed on every block, so it bounds a stall rather than the total transfer time of a large object: a slow-but-honest writer that keeps completing shards is never killed, while a writer that makes no progress within the budget is failed and its disk dropped before commit. An optional absolute per-object cap (disabled by default) backstops a slow-drip peer that dribbles just enough progress to reset the per-block timer without ever converging; it is off by default so a legitimate large upload over a slow link is not killed on total time alone. Both knobs come from RUSTFS_OBJECT_DISK_WRITE_STALL_TIMEOUT (default 30s) and RUSTFS_OBJECT_DISK_WRITE_ABSOLUTE_CAP (default 0 = disabled); setting the stall timeout to 0 restores the previous wait-forever behavior for a conservative rollback.

The deadline enforcement lives in MultiWriter (writer-agnostic), so it covers local and remote writers alike and keeps the existing control-flow shape: a timed-out shard is marked failed (Error::Timeout, which is not an ignored error) and excluded from the write quorum exactly like any other shard write failure, and the unchanged nil_count/quorum check then continues on quorum or fails cleanly. This deliberately stays out of the MultiWriter lifecycle / commit-coordinator territory owned by rustfs/backlog#1312.

When a stalled writer is dropped to fail its shard, the remote HttpWriter must stop holding the connection and its buffered body. HttpWriter previously left its spawned request task running on drop; it now aborts that background task in Drop (it is no longer pin-projected, since every field is Unpin and the AsyncWrite impl already used get_mut). Bytes already handed to the transport cannot be unsent, but they land only in this upload's unique tmp path and are reclaimed by tmp GC — they never touch a committed object.

Tests, all on a paused virtual clock so they are deterministic and non-flaky:
- one black-hole writer still meets a 3/4 write quorum without hanging; two black holes fail the quorum cleanly (both for the per-block write and the shutdown paths).
- a slow-but-honest writer that keeps making progress within the stall budget is never failed across many blocks.
- the absolute cap bounds a slow-drip writer within a finite budget while the healthy writers keep quorum.
- the default policy is armed by default and honors 0 as disabled.
- HttpWriter aborts its background request task on drop against a hanging peer.

The toxiproxy/black-hole 4x4 end-to-end acceptance depends on black-box test facilities from rustfs/backlog#1325, which are not built yet; that acceptance is deferred to #1325 and intentionally not faked here.

crates/ecstore/src/disk/disk_store.rs

@@ -217,6 +217,27 @@ pub fn get_object_disk_read_timeout() -> Duration {
     )
 }
 
+/// Per-shard erasure write stall budget: a shard write (or shutdown) that makes
+/// no forward progress for this long is failed and its disk dropped before
+/// commit. Re-armed on every shard write, so it bounds a stall rather than the
+/// whole transfer. `0` disables the deadline (wait indefinitely).
+pub fn get_object_disk_write_stall_timeout() -> Duration {
+    Duration::from_secs(rustfs_utils::get_env_u64(
+        rustfs_config::ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT,
+        rustfs_config::DEFAULT_OBJECT_DISK_WRITE_STALL_TIMEOUT,
+    ))
+}
+
+/// Optional absolute per-object erasure write cap (administrator slow-drip
+/// backstop). `0` (default) disables the cap; the per-shard stall timeout is the
+/// primary guarantee.
+pub fn get_object_disk_write_absolute_cap() -> Duration {
+    Duration::from_secs(rustfs_utils::get_env_u64(
+        rustfs_config::ENV_OBJECT_DISK_WRITE_ABSOLUTE_CAP,
+        rustfs_config::DEFAULT_OBJECT_DISK_WRITE_ABSOLUTE_CAP,
+    ))
+}
+
 pub fn get_drive_active_check_interval() -> Duration {
     Duration::from_secs(rustfs_utils::get_env_u64(
         rustfs_config::ENV_DRIVE_ACTIVE_CHECK_INTERVAL_SECS,
@@ -1653,6 +1674,32 @@ mod tests {
         });
     }
 
+    #[test]
+    fn object_disk_write_stall_timeout_default_and_override() {
+        temp_env::with_var_unset(rustfs_config::ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT, || {
+            assert_eq!(
+                get_object_disk_write_stall_timeout(),
+                Duration::from_secs(rustfs_config::DEFAULT_OBJECT_DISK_WRITE_STALL_TIMEOUT)
+            );
+        });
+        temp_env::with_var(rustfs_config::ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT, Some("9"), || {
+            assert_eq!(get_object_disk_write_stall_timeout(), Duration::from_secs(9));
+        });
+        temp_env::with_var(rustfs_config::ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT, Some("0"), || {
+            assert!(get_object_disk_write_stall_timeout().is_zero(), "0 disables the stall deadline");
+        });
+    }
+
+    #[test]
+    fn object_disk_write_absolute_cap_defaults_disabled() {
+        temp_env::with_var_unset(rustfs_config::ENV_OBJECT_DISK_WRITE_ABSOLUTE_CAP, || {
+            assert!(get_object_disk_write_absolute_cap().is_zero(), "absolute cap is disabled by default");
+        });
+        temp_env::with_var(rustfs_config::ENV_OBJECT_DISK_WRITE_ABSOLUTE_CAP, Some("120"), || {
+            assert_eq!(get_object_disk_write_absolute_cap(), Duration::from_secs(120));
+        });
+    }
+
     #[test]
     fn object_disk_read_timeout_uses_default_when_unset() {
         temp_env::with_var_unset(rustfs_config::ENV_OBJECT_DISK_READ_TIMEOUT, || {