fix(odm): declare source retry policy and time out a stalled inline read (#7111)

* fix(odm): declare the remote client retry policy per consumer

The SDK retry policy was an inherited default: one logical call could cost
three wire requests, so the migration breaker counted logical calls on top
of a threefold amplification against a source that was already failing.

Make it an explicit RemoteS3EndpointSpec field. Replication targets declare
today's standard three attempts and keep their behaviour; the on-demand
migration source and its admin probe declare a disabled policy, so one
counted failure is exactly one source request and pull.rs owns the only
retry budget.

* fix(odm): count a stalled inline source as a source timeout

The inline tee wraps its source body in the idle guard, but the tee turns a
stalled source into an ordinary body read error, so the write-back reported
it as a local write failure. Hand commit_inline the guard so the pull is
counted under source_timeout instead.

The background pump now enforces the idle budget through the same guard
rather than a second copy of the timeout loop.

* test(odm): cover a stalled source body end to end

The fake target can now deliver a GetObject body in slices with a pause
between them, so the inline abort can be driven by a stalled source instead
of a truncated one. Two fault cases drop the workarounds they carried for
the SDK's retries: the scripted fault count and the observed source request
count now have to agree.

The operations guide records the retry and idle-timeout guarantees.

rustfs/src/admin/handlers/on_demand_migration.rs

@@ -47,7 +47,9 @@ use crate::admin::storage_api::bucket::on_demand_migration::source_client::{
 use crate::admin::storage_api::bucket::on_demand_migration::{
     OdmBucketSnapshot, OnDemandMigrationConfig, OnDemandMigrationConfigError, OnDemandMigrationSys, PathStyle, ValidationContext,
 };
-use crate::admin::storage_api::bucket::remote_s3_client::{PathStyle as RemotePathStyle, RemoteCredentials, RemoteS3ClientError};
+use crate::admin::storage_api::bucket::remote_s3_client::{
+    PathStyle as RemotePathStyle, RemoteCredentials, RemoteS3ClientError, RemoteS3RetryPolicy,
+};
 use crate::admin::storage_api::contract::bucket::{BucketOperations as _, BucketOptions};
 use crate::admin::storage_api::error::StorageError;
 use crate::admin::storage_api::s3::{Body, S3Error, S3ErrorCode, S3Request, S3Response, S3Result, error as admin_s3_error};
@@ -615,6 +617,9 @@ pub(crate) fn source_client_spec(config: &OnDemandMigrationConfig) -> SourceClie
             connect: Duration::from_millis(timeout.connect_ms),
             read: Duration::from_millis(timeout.first_byte_ms),
         },
+        // The probe reports the source's own answer; an SDK retry would hide
+        // a flapping source behind a success and triple the probe's cost.
+        retry: RemoteS3RetryPolicy::Disabled,
         bandwidth_limit: config.policy.bandwidth_limit_bytes_per_sec.and_then(NonZeroU64::new),
     }
 }

rustfs/src/admin/storage_api.rs

@@ -315,6 +315,7 @@ pub(crate) mod remote_s3_client {
     pub(crate) type PathStyle = super::ecstore_bucket::remote_s3_client::PathStyle;
     pub(crate) type RemoteCredentials = super::ecstore_bucket::remote_s3_client::RemoteCredentials;
     pub(crate) type RemoteS3ClientError = super::ecstore_bucket::remote_s3_client::RemoteS3ClientError;
+    pub(crate) type RemoteS3RetryPolicy = super::ecstore_bucket::remote_s3_client::RemoteS3RetryPolicy;
 }
 
 pub(crate) mod metadata_sys {

rustfs/src/app/object/get.rs

@@ -4706,12 +4706,13 @@ async fn odm_get_inline<S: OdmGetSource>(
     // The inline path has no background pump, so `source_timeout.idle_ms` is
     // applied to the teed body here; without it a stalled source would hold
     // both the client stream and the write-back open until the SDK read
-    // timeout fires.
+    // timeout fires. The guard wraps the source read, upstream of the tee, so
+    // a slow client throttles the tee instead of ageing the source's budget.
     let source_body: SourceBody = Box::pin(tokio_util::io::ReaderStream::with_capacity(
         body.into_async_read(),
         ODM_SOURCE_BODY_CHUNK_BYTES,
     ));
-    let guarded = idle_guarded_body(source_body, Duration::from_millis(policy.source_timeout.idle_ms));
+    let (guarded, idle) = idle_guarded_body(source_body, Duration::from_millis(policy.source_timeout.idle_ms));
     let (primary, secondary) =
         tee_reader_with_options(Box::pin(tokio_util::io::StreamReader::new(guarded)), ODM_INLINE_TEE_BUFFER_BYTES, options);
     let output = Box::new(odm_get_output(&head, content_length, content_range, odm_inline_client_body(primary)));
@@ -4719,7 +4720,7 @@ async fn odm_get_inline<S: OdmGetSource>(
     let commit_key = key.to_string();
     spawn_background_with_context(request_context, async move {
         let body: WriteBackBody = Box::pin(secondary.into_stream());
-        let result = commit_inline(&commit_state, &commit_key, head, tags, body).await;
+        let result = commit_inline(&commit_state, &commit_key, head, tags, body, &idle).await;
         leader.complete(result.map(|outcome| PullOutcome {
             etag: outcome.etag,
             size: outcome.size,