fix(odm): declare source retry policy and time out a stalled inline read (#7111)

* fix(odm): declare the remote client retry policy per consumer

The SDK retry policy was an inherited default: one logical call could cost
three wire requests, so the migration breaker counted logical calls on top
of a threefold amplification against a source that was already failing.

Make it an explicit RemoteS3EndpointSpec field. Replication targets declare
today's standard three attempts and keep their behaviour; the on-demand
migration source and its admin probe declare a disabled policy, so one
counted failure is exactly one source request and pull.rs owns the only
retry budget.

* fix(odm): count a stalled inline source as a source timeout

The inline tee wraps its source body in the idle guard, but the tee turns a
stalled source into an ordinary body read error, so the write-back reported
it as a local write failure. Hand commit_inline the guard so the pull is
counted under source_timeout instead.

The background pump now enforces the idle budget through the same guard
rather than a second copy of the timeout loop.

* test(odm): cover a stalled source body end to end

The fake target can now deliver a GetObject body in slices with a pause
between them, so the inline abort can be driven by a stalled source instead
of a truncated one. Two fault cases drop the workarounds they carried for
the SDK's retries: the scripted fault count and the observed source request
count now have to agree.

The operations guide records the retry and idle-timeout guarantees.
This commit is contained in:
Zhengchao An
2026-09-04 02:24:53 +08:00
committed by GitHub
parent 3a914b429d
commit 3005efe845
14 changed files with 486 additions and 109 deletions
+4 -3
View File
@@ -162,8 +162,9 @@ pub mod bucket {
};
pub use crate::bucket::on_demand_migration::{
EnqueueOutcome, LocalObject, MAX_MULTIPART_PARTS, OdmWriteBack, PULL_MAX_RETRIES, PULL_RETRY_BASE_DELAYS,
PullCompletion, PullQueue, PullReason, PullSource, QueuedPullOutcome, SourceBody, WriteBackBody, WriteBackError,
WriteBackOutcome, WriteBackPart, WriteBackRequest, commit_inline, commit_inline_with, idle_guarded_body,
PullCompletion, PullQueue, PullReason, PullSource, QueuedPullOutcome, SourceBody, SourceIdleGuard, WriteBackBody,
WriteBackError, WriteBackOutcome, WriteBackPart, WriteBackRequest, commit_inline, commit_inline_with,
idle_guarded_body,
};
pub mod backfill {
pub use crate::bucket::on_demand_migration::backfill::{
@@ -241,7 +242,7 @@ pub mod bucket {
pub mod remote_s3_client {
pub use crate::bucket::remote_s3_client::{
PathStyle, RemoteCredentials, RemoteS3ClientError, RemoteS3EndpointSpec, build_remote_s3_client,
PathStyle, RemoteCredentials, RemoteS3ClientError, RemoteS3EndpointSpec, RemoteS3RetryPolicy, build_remote_s3_client,
validate_remote_endpoint,
};
}