fix(odm): declare source retry policy and time out a stalled inline read (#7111)

* fix(odm): declare the remote client retry policy per consumer

The SDK retry policy was an inherited default: one logical call could cost
three wire requests, so the migration breaker counted logical calls on top
of a threefold amplification against a source that was already failing.

Make it an explicit RemoteS3EndpointSpec field. Replication targets declare
today's standard three attempts and keep their behaviour; the on-demand
migration source and its admin probe declare a disabled policy, so one
counted failure is exactly one source request and pull.rs owns the only
retry budget.

* fix(odm): count a stalled inline source as a source timeout

The inline tee wraps its source body in the idle guard, but the tee turns a
stalled source into an ordinary body read error, so the write-back reported
it as a local write failure. Hand commit_inline the guard so the pull is
counted under source_timeout instead.

The background pump now enforces the idle budget through the same guard
rather than a second copy of the timeout loop.

* test(odm): cover a stalled source body end to end

The fake target can now deliver a GetObject body in slices with a pause
between them, so the inline abort can be driven by a stalled source instead
of a truncated one. Two fault cases drop the workarounds they carried for
the SDK's retries: the scripted fault count and the observed source request
count now have to agree.

The operations guide records the retry and idle-timeout guarantees.

docs/operations/on-demand-migration.md

@@ -165,6 +165,7 @@ Behaviour a client can observe. The "Test" column names the case that pins it: `
 | Source answers 404 | 404 to the client and the key is negative-cached for `negative_cache_ttl_secs` | `get_basic_test.rs::get_source_not_found_is_404_and_negative_cached`, `fault_test.rs::test_odm_source_not_found_is_negative_cached_for_the_ttl` |
 | Source answers 403 | 424 (or 404 under `not_found`); the breaker is **not** opened — a credential problem is not a transient one | `fault_test.rs::test_odm_source_access_denied_propagates_without_opening_the_breaker` |
 | Repeated source 5xx / timeouts | The breaker opens, GETs answer per `source_error` and HEADs answer 404; a probe closes it again after the open window | `fault_test.rs::test_odm_repeated_source_errors_open_the_breaker_and_recover`, `head.rs::odm_head_source_errors_follow_policy_and_open_the_breaker` |
+| Source stalls mid-body on an inline pull | `source_timeout.idle_ms` ends both tee ends: the client's read fails instead of receiving a silently truncated 200, the pull counts under `source_timeout`, and no object or multipart upload is left behind | `fault_test.rs::test_odm_inline_pull_aborts_when_the_source_body_stalls`, `pull.rs::commit_inline_reports_a_stalled_source_as_a_source_timeout` |
 | Bucket default SSE | The pulled object is stored encrypted and reads back in plaintext; the ETag override is dropped, and the source ETag is kept in metadata | `interaction_test.rs::test_odm_pulled_object_uses_bucket_default_encryption`, `on_demand_migration_put.rs::write_back_under_bucket_default_sse_stores_ciphertext_and_records_source_etag` |
 | Object Lock bucket | The pulled object inherits the bucket's default retention | `interaction_test.rs::test_odm_pulled_object_inherits_object_lock_retention` |
 | Bucket quota exceeded | The client is still served from the source; the write-back is rejected, nothing is stored, and the failure counts under `quota` | `interaction_test.rs::test_odm_write_back_respects_the_bucket_quota`, `on_demand_migration_put.rs::write_back_reports_a_full_bucket_quota` |
@@ -210,7 +211,8 @@ Five provenance keys are written on every pulled object under both internal pref
 | Concurrency limit | Local write amplification | `max_concurrent_pulls` permits shared by inline and background pulls |
 | Bounded queue | Unbounded memory on a burst | `pull_queue_capacity` waiting jobs; overflow is counted as `queue_full` and never fails a client response |
 | Bandwidth limit | Source and network saturation | `bandwidth_limit_bytes_per_sec` (minimum 64 KiB/s) on the source client |
-| Retry budget | Transient source blips | Background pulls retry a retryable failure up to 3 times (1 s / 4 s / 16 s plus jitter). Inline pulls never retry: the bytes are already on their way to the client |
+| Retry budget | Transient source blips | Background pulls retry a retryable failure up to 3 times (1 s / 4 s / 16 s plus jitter). Inline pulls never retry: the bytes are already on their way to the client. The SDK retry policy on the source client is disabled (`RemoteS3RetryPolicy::Disabled`), so this is the only retry budget and one logical source call is exactly one wire request — replication targets keep the SDK's three attempts, declared on their own spec |
+| Idle timeout | A source that answers and then goes quiet mid-body | `source_timeout.idle_ms` per body chunk on both paths. The budget measures the source read, upstream of the inline tee, so a slow client is never mistaken for an idle source; when it fires the client stream ends in an error and the write-back is discarded |
 | Anti-loop marker | Migration chains between RustFS/MinIO deployments | Every source request carries `x-rustfs-source-proxy-request` and `x-minio-source-proxy-request`; a request carrying it is always answered locally |
 | Outbound endpoint policy | SSRF | See [outbound-connection-policy.md](outbound-connection-policy.md) |
 
@@ -291,7 +293,7 @@ sum by (bucket, reason) (rate(rustfs_on_demand_migration_pull_failures_total[5m]
 
 **`NoSuchBucket` from the source, or a 301/307 redirect.** The addressing style is wrong: a virtual-host request against a path-style-only server looks like a missing bucket, and a path-style request against AWS gets redirected. Set `source.path_style` explicitly instead of relying on `auto`.
 
-**Large objects are served but never stored.** Check `pull_failures_total`: `queue_full` means bursts exceed `pull_queue_capacity` (raise it, or raise `max_concurrent_pulls`); `quota` means the bucket quota rejected the write-back and `local_write` a genuine local write failure; `etag_mismatch` means the source body did not match the ETag the source advertised. The client response is served in all three cases — only the local copy is missing.
+**Large objects are served but never stored.** Check `pull_failures_total`: `queue_full` means bursts exceed `pull_queue_capacity` (raise it, or raise `max_concurrent_pulls`); `quota` means the bucket quota rejected the write-back and `local_write` a genuine local write failure; `etag_mismatch` means the source body did not match the ETag the source advertised. The client response is served in all these cases — only the local copy is missing. `source_timeout` is different: the source stopped sending mid-body, so the client's own read fails too and nothing is stored.
 
 **The first read of a key is slow, later ones are fast.** Expected: the first read pays a source HEAD plus a source GET. Watch `source_latency_seconds_*` for the source's contribution and lower `inline_max_bytes` if teeing large objects is hurting first-byte latency.