fix(ecstore): bound remote shard writers with a progress deadline so one black-hole peer cannot pin write quorum (#4925)

A PUT that fans out erasure shards to remote peers awaited every shard writer to completion on both the per-block write and the final shutdown, and the remote HttpWriter had no progress deadline. A peer that accepts the TCP connection but never drains the request body (or never sends a response) therefore wedges the writer forever once the bounded buffers fill, pinning an otherwise-healthy write quorum indefinitely — a cluster-level write-availability hazard triggered by a single bad peer (rustfs/backlog#1319, https://github.com/rustfs/backlog/issues/1319).

MultiWriter now wraps each shard write and each shard-writer shutdown in a forward-progress deadline. The budget is re-armed on every block, so it bounds a stall rather than the total transfer time of a large object: a slow-but-honest writer that keeps completing shards is never killed, while a writer that makes no progress within the budget is failed and its disk dropped before commit. An optional absolute per-object cap (disabled by default) backstops a slow-drip peer that dribbles just enough progress to reset the per-block timer without ever converging; it is off by default so a legitimate large upload over a slow link is not killed on total time alone. Both knobs come from RUSTFS_OBJECT_DISK_WRITE_STALL_TIMEOUT (default 30s) and RUSTFS_OBJECT_DISK_WRITE_ABSOLUTE_CAP (default 0 = disabled); setting the stall timeout to 0 restores the previous wait-forever behavior for a conservative rollback.

The deadline enforcement lives in MultiWriter (writer-agnostic), so it covers local and remote writers alike and keeps the existing control-flow shape: a timed-out shard is marked failed (Error::Timeout, which is not an ignored error) and excluded from the write quorum exactly like any other shard write failure, and the unchanged nil_count/quorum check then continues on quorum or fails cleanly. This deliberately stays out of the MultiWriter lifecycle / commit-coordinator territory owned by rustfs/backlog#1312.

When a stalled writer is dropped to fail its shard, the remote HttpWriter must stop holding the connection and its buffered body. HttpWriter previously left its spawned request task running on drop; it now aborts that background task in Drop (it is no longer pin-projected, since every field is Unpin and the AsyncWrite impl already used get_mut). Bytes already handed to the transport cannot be unsent, but they land only in this upload's unique tmp path and are reclaimed by tmp GC — they never touch a committed object.

Tests, all on a paused virtual clock so they are deterministic and non-flaky:
- one black-hole writer still meets a 3/4 write quorum without hanging; two black holes fail the quorum cleanly (both for the per-block write and the shutdown paths).
- a slow-but-honest writer that keeps making progress within the stall budget is never failed across many blocks.
- the absolute cap bounds a slow-drip writer within a finite budget while the healthy writers keep quorum.
- the default policy is armed by default and honors 0 as disabled.
- HttpWriter aborts its background request task on drop against a hanging peer.

The toxiproxy/black-hole 4x4 end-to-end acceptance depends on black-box test facilities from rustfs/backlog#1325, which are not built yet; that acceptance is deferred to #1325 and intentionally not faked here.

crates/config/src/constants/object.rs

@@ -128,6 +128,39 @@ pub const ENV_OBJECT_DISK_READ_TIMEOUT: &str = "RUSTFS_OBJECT_DISK_READ_TIMEOUT"
 /// Default disk read timeout in seconds.
 pub const DEFAULT_OBJECT_DISK_READ_TIMEOUT: u64 = 10;
 
+/// Environment variable for the per-shard erasure write stall timeout (seconds).
+///
+/// A single shard write (or shard-writer shutdown) that makes no forward
+/// progress for longer than this budget is failed and its disk is dropped
+/// before commit, so a black-hole peer that accepts the connection but never
+/// drains the body cannot pin an otherwise-healthy write quorum forever
+/// (see `MultiWriter` in `erasure/coding/encode.rs`). The budget is re-armed on
+/// every shard write, so it bounds a *stall* rather than the total transfer
+/// time of a large object.
+///
+/// Unit: seconds (u64). `0` disables the stall deadline (previous behavior:
+/// wait indefinitely). Default: 30 seconds.
+pub const ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT: &str = "RUSTFS_OBJECT_DISK_WRITE_STALL_TIMEOUT";
+
+/// Default per-shard erasure write stall timeout in seconds.
+pub const DEFAULT_OBJECT_DISK_WRITE_STALL_TIMEOUT: u64 = 30;
+
+/// Environment variable for the absolute per-object erasure write cap (seconds).
+///
+/// Optional administrator backstop against a "slow-drip" peer that produces
+/// just enough forward progress to reset the per-shard stall timeout on every
+/// block while never converging. When set, the shard writers for one object are
+/// engaged for at most this long in aggregate before a stalled writer is failed
+/// and dropped. It is disabled by default because a legitimate large upload
+/// over a slow-but-honest link must not be killed on total time alone; the
+/// per-shard stall timeout is the primary guarantee.
+///
+/// Unit: seconds (u64). `0` (default) disables the absolute cap.
+pub const ENV_OBJECT_DISK_WRITE_ABSOLUTE_CAP: &str = "RUSTFS_OBJECT_DISK_WRITE_ABSOLUTE_CAP";
+
+/// Default absolute per-object erasure write cap in seconds (`0` = disabled).
+pub const DEFAULT_OBJECT_DISK_WRITE_ABSOLUTE_CAP: u64 = 0;
+
 /// Environment variable for minimum GetObject timeout in seconds.
 ///
 /// When dynamic timeout calculation is enabled, this is the minimum timeout