fix(odm): declare source retry policy and time out a stalled inline read (#7111)

* fix(odm): declare the remote client retry policy per consumer The SDK retry policy was an inherited default: one logical call could cost three wire requests, so the migration breaker counted logical calls on top of a threefold amplification against a source that was already failing. Make it an explicit RemoteS3EndpointSpec field. Replication targets declare today's standard three attempts and keep their behaviour; the on-demand migration source and its admin probe declare a disabled policy, so one counted failure is exactly one source request and pull.rs owns the only retry budget. * fix(odm): count a stalled inline source as a source timeout The inline tee wraps its source body in the idle guard, but the tee turns a stalled source into an ordinary body read error, so the write-back reported it as a local write failure. Hand commit_inline the guard so the pull is counted under source_timeout instead. The background pump now enforces the idle budget through the same guard rather than a second copy of the timeout loop. * test(odm): cover a stalled source body end to end The fake target can now deliver a GetObject body in slices with a pause between them, so the inline abort can be driven by a stalled source instead of a truncated one. Two fault cases drop the workarounds they carried for the SDK's retries: the scripted fault count and the observed source request count now have to agree. The operations guide records the retry and idle-timeout guarantees.
2026-09-07 12:35:54 +00:00 · 2026-09-04 02:24:53 +08:00
parent 3a914b429d
commit 3005efe845
14 changed files with 486 additions and 109 deletions
@@ -12,6 +12,6 @@ Supported data operations are HeadBucket, GetBucketVersioning, ListObjectsV2, PU

 ListObjectsV2 lists current versions only (a key whose newest version is a delete marker is hidden) in byte order and supports `prefix`, `delimiter`, `max-keys` (clamped to 1000), `start-after`, and `continuation-token`; common prefixes count toward `max-keys`, `IsTruncated` / `NextContinuationToken` / `KeyCount` follow S3, and continuation tokens are opaque. `encoding-type` and `fetch-owner` are accepted but ignored, and ListObjects (v1) is not implemented. GET and HEAD honor `Range` in the `bytes=first-last`, `bytes=first-`, and `bytes=-suffix` forms with a 206 status, exact `Content-Range`, and `Accept-Ranges: bytes`; unsatisfiable ranges answer 416 `InvalidRange` with `Content-Range: bytes */<length>`. PUT and CreateMultipartUpload accept `Content-Type`, `Content-Encoding`, `Content-Disposition`, `Content-Language`, `Cache-Control`, `Expires`, and `x-amz-meta-*` (names stored lowercased), and HEAD/GET replay them verbatim together with `Last-Modified` and the ETag (hex MD5 for single PUTs, `<md5-of-part-md5s>-<parts>` for multipart objects). `put_seed_object` stores an object directly, bypassing the wire, the fault script, and the journal, so a source can be seeded without polluting the assertions a scenario later makes.

-Fault actions cover HTTP 401/403/503 responses (`Status`), any 4xx/5xx status paired with the matching S3 error code (`ResponseStatus`), pre-dispatch delay, holding a fully computed successful response before its first byte (`Stall`), connection abort when a logical request-body threshold is reached, GetObject bodies cut off after N bytes while `Content-Length` announces the full size (`TruncateBodyAt`), streaming slow drain, and a deliberately wrong response ETag (including multipart-complete XML). `requests()` returns the ordered, credential-free request journal for assertions and `count_requests(operation, key)` counts entries for one exact key. Each record journals the `Range` and `User-Agent` request headers, the ListObjectsV2 `prefix` and `continuation-token` query values, a `TransportSnapshot` — whether the body was announced as `aws-chunked`, the verbatim `Content-MD5`, the sorted `x-amz-checksum-*` / `x-amz-sdk-checksum-algorithm` header names, and whether any `x-amz-object-lock-*` header was present — and a `ProxyHeaderSnapshot` — the read-proxy anti-loop marker (`x-{rustfs,minio}-source-proxy-request`), the replication-check exemption header, and the client SSE-C header family (algorithm and key-MD5 values; for the key itself only its presence) — so proxy tests can pin the exact wire contract.
+Fault actions cover HTTP 401/403/503 responses (`Status`), any 4xx/5xx status paired with the matching S3 error code (`ResponseStatus`), pre-dispatch delay, holding a fully computed successful response before its first byte (`Stall`), connection abort when a logical request-body threshold is reached, GetObject bodies cut off after N bytes while `Content-Length` announces the full size (`TruncateBodyAt`), GetObject bodies delivered in fixed slices with a pause between them (`SlowSendBody`, a mid-body stall rather than a first-byte one), streaming slow drain, and a deliberately wrong response ETag (including multipart-complete XML). `requests()` returns the ordered, credential-free request journal for assertions and `count_requests(operation, key)` counts entries for one exact key. Each record journals the `Range` and `User-Agent` request headers, the ListObjectsV2 `prefix` and `continuation-token` query values, a `TransportSnapshot` — whether the body was announced as `aws-chunked`, the verbatim `Content-MD5`, the sorted `x-amz-checksum-*` / `x-amz-sdk-checksum-algorithm` header names, and whether any `x-amz-object-lock-*` header was present — and a `ProxyHeaderSnapshot` — the read-proxy anti-loop marker (`x-{rustfs,minio}-source-proxy-request`), the replication-check exemption header, and the client SSE-C header family (algorithm and key-MD5 values; for the key itself only its presence) — so proxy tests can pin the exact wire contract.

 The listener is loopback-only. It admits at most 64 active connections and two concurrently buffered request bodies; authenticated multipart-complete XML collection and assembly take both body permits. Keep-alive is disabled, request-header reads are bounded to 30 seconds, a parsed request is bounded to 65 seconds, and the complete connection lifetime is bounded to 100 seconds. It retains at most 256 buckets, 4,096 journal entries, 4,096 scripted faults, 4,096 object versions, 256 multipart uploads, and 10,000 multipart parts. Retained identifiers are capped at 1 KiB, user metadata at 2 KiB, and content type and each standard object header at 1 KiB. By default a PUT or uploaded part is capped at 64 MiB and a completed multipart object and all stored object/part data are capped at 128 MiB; `FakeS3Target::start_with_options(FakeS3TargetOptions { max_object_bytes })` raises the object cap up to 256 MiB, and the total budget then becomes twice the object cap (never below 128 MiB). Body drain, body-permit waits, delay, stall, and slow-drain execution are bounded to 30 seconds; each slow-drain slice delay must be below that bound.
@@ -266,6 +266,12 @@ pub enum FaultAction {
    /// body bytes, then abort the connection so the client observes a short
    /// read. Ignored by every other operation.
    TruncateBodyAt(usize),
+    /// GetObject only: deliver the response body in `chunk_bytes` slices,
+    /// sleeping `delay` between them. The head and the first slice leave
+    /// immediately, so this is a mid-body stall rather than a first-byte one;
+    /// a `delay` above the reader's idle budget is what a source that stops
+    /// pushing bytes looks like. Ignored by every other operation.
+    SlowSendBody { chunk_bytes: usize, delay: Duration },
    /// Apply the request normally, then hold the complete response (status
    /// line included) for the duration before the first byte is written —
    /// the first-byte-timeout scenario. Error responses are not held.
@@ -994,6 +1000,9 @@ fn validate_fault_action(action: &FaultAction) {
    if let FaultAction::SlowDrain { chunk_bytes: 0, .. } = action {
        panic!("slow-drain chunk size must be non-zero");
    }
+    if let FaultAction::SlowSendBody { chunk_bytes: 0, .. } = action {
+        panic!("slow-send chunk size must be non-zero");
+    }
    match action {
        FaultAction::Delay(duration) if *duration > MAX_FAULT_DURATION => {
            panic!("fault delay must not exceed 30 seconds");
@@ -1004,6 +1013,9 @@ fn validate_fault_action(action: &FaultAction) {
        FaultAction::SlowDrain { delay, .. } if *delay >= MAX_FAULT_DURATION => {
            panic!("slow-drain slice delay must be below 30 seconds");
        }
+        FaultAction::SlowSendBody { delay, .. } if *delay >= MAX_FAULT_DURATION => {
+            panic!("slow-send slice delay must be below 30 seconds");
+        }
        FaultAction::ResponseStatus(code) if !(400..=599).contains(code) => {
            panic!("scripted response status must be a 4xx or 5xx code");
        }
@@ -1409,6 +1421,7 @@ async fn apply_non_body_fault(fault: Option<&RequestFault>, control: &Mutex<Cont
        | Some(FaultAction::WrongEtag)
        | Some(FaultAction::DisconnectAfterResponse)
        | Some(FaultAction::TruncateBodyAt(_))
+        | Some(FaultAction::SlowSendBody { .. })
        | Some(FaultAction::Stall(_))
        | None => Ok(()),
    }
@@ -1457,6 +1470,7 @@ async fn collect_stream(
            Some(FaultAction::WrongEtag)
            | Some(FaultAction::DisconnectAfterResponse)
            | Some(FaultAction::TruncateBodyAt(_))
+            | Some(FaultAction::SlowSendBody { .. })
            | Some(FaultAction::Stall(_))
            | None => {}
        }
@@ -1524,6 +1538,22 @@ fn truncated_body(body: Bytes, truncate_at: usize) -> StreamingBlob {
    }))
 }

+/// GetObject body delivered in `chunk_bytes` slices with `delay` between
+/// them. The head and the first slice are written immediately, so the client
+/// starts reading and then observes the source going quiet mid-body.
+fn slow_sent_body(body: Bytes, chunk_bytes: usize, delay: Duration) -> StreamingBlob {
+    StreamingBlob::wrap(futures::stream::unfold((body, true), move |(mut rest, first)| async move {
+        if rest.is_empty() {
+            return None;
+        }
+        if !first {
+            sleep(delay).await;
+        }
+        let chunk = rest.split_to(chunk_bytes.min(rest.len()));
+        Some((Ok::<Bytes, io::Error>(chunk), (rest, false)))
+    }))
+}
+
 async fn assemble_multipart(
    parts: Vec<(Bytes, [u8; 16])>,
    total_len: usize,
@@ -2225,6 +2255,7 @@ impl S3 for FakeBackend {
        let body = version.body.slice(served.range.clone());
        let body = match fault.as_ref().map(|fault| &fault.action) {
            Some(FaultAction::TruncateBodyAt(truncate_at)) => truncated_body(body, *truncate_at),
+            Some(FaultAction::SlowSendBody { chunk_bytes, delay }) => slow_sent_body(body, *chunk_bytes, *delay),
            _ => StreamingBlob::from(body),
        };
        let mut response = S3Response::new(GetObjectOutput {
@@ -14,8 +14,8 @@

 //! Source-failure scenarios for on-demand migration (rustfs/backlog#2158):
 //! access denied, the circuit breaker, first-byte and mid-body stream
-//! failures, ETag integrity, the negative cache, and an unsupported
-//! (SSE-C) source object.
+//! failures (a cut body and a stalled one), ETag integrity, the negative
+//! cache, and an unsupported (SSE-C) source object.
 //!
 //! Every case asserts what the source was asked for, not only what the
 //! client received: a fault that silently turned into a second source
@@ -162,10 +162,10 @@ async fn test_odm_source_access_denied_propagates_without_opening_the_breaker()
 /// window closes it again. The open window is a compiled-in 30 s constant
 /// (`BREAKER_OPEN_DURATION`), so this case waits in real time.
 ///
-/// One ODM source call is several wire requests: the SDK retries a 503 on
-/// its own, and only the exhausted call counts as one breaker failure. The
-/// script is therefore deep enough to cover every retry, and the open state
-/// is waited for instead of being predicted from a request count.
+/// The source client disables SDK retries, so one logical source call is
+/// exactly one wire request: the script is exactly as deep as the number of
+/// breaker failures it has to produce, and the scripted fault count and the
+/// observed source request count must agree.
 #[tokio::test]
 async fn test_odm_repeated_source_errors_open_the_breaker_and_recover() -> TestResult {
    let bucket = "odm-fault-breaker";
@@ -175,9 +175,8 @@ async fn test_odm_repeated_source_errors_open_the_breaker_and_recover() -> TestR
    env.seed_source(SOURCE_BUCKET, &[SeedObject::new(key, body.clone())]);

    env.source
-        .inject_for_key(Operation::HeadObject, key, FaultAction::ResponseStatus(503), 200);
-    let mut opened = false;
-    for attempt in 1..=BREAKER_FAILURE_THRESHOLD * 2 {
+        .inject_for_key(Operation::HeadObject, key, FaultAction::ResponseStatus(503), BREAKER_FAILURE_THRESHOLD);
+    for attempt in 1..=BREAKER_FAILURE_THRESHOLD {
        let response = env.raw_get(bucket, key).await?;
        assert_eq!(
            response.status,
@@ -185,21 +184,19 @@ async fn test_odm_repeated_source_errors_open_the_breaker_and_recover() -> TestR
            "attempt {attempt}: {}",
            String::from_utf8_lossy(&response.body)
        );
-        if env
-            .status_json(bucket)
+    }
+    assert_eq!(
+        env.status_json(bucket)
            .await?
            .pointer("/breaker/state")
-            .and_then(|v| v.as_str())
-            == Some("open")
-        {
-            opened = true;
-            break;
-        }
-    }
-    assert!(opened, "consecutive source failures must open the breaker");
-    assert!(
-        env.source.count_requests(Operation::HeadObject, key) >= BREAKER_FAILURE_THRESHOLD,
-        "each counted failure is at least one source request"
+            .and_then(|v| v.as_str()),
+        Some("open"),
+        "the threshold of consecutive source failures must open the breaker"
+    );
+    assert_eq!(
+        env.source.count_requests(Operation::HeadObject, key),
+        BREAKER_FAILURE_THRESHOLD,
+        "every counted failure is exactly one source request"
    );

    // With the script cleared, the only thing that can still fail a read is
@@ -251,8 +248,8 @@ async fn test_odm_repeated_source_errors_open_the_breaker_and_recover() -> TestR
 }

 /// Case 3: a source that holds the response past `first_byte_ms` is a
-/// timeout, and the client never sees a 200 head. Every attempt the SDK
-/// makes on its own is stalled too, so the ODM call really does give up.
+/// timeout, and the client never sees a 200 head. One logical source call is
+/// one wire request, so a single scripted stall is enough to fail the read.
 #[tokio::test]
 async fn test_odm_source_stall_times_out_before_the_first_byte() -> TestResult {
    let bucket = "odm-fault-stall";
@@ -264,20 +261,19 @@ async fn test_odm_source_stall_times_out_before_the_first_byte() -> TestResult {
    env.seed_source(SOURCE_BUCKET, &[SeedObject::new(key, payload(4096))]);

    env.source
-        .inject_for_key(Operation::HeadObject, key, FaultAction::Stall(Duration::from_secs(5)), 8);
+        .inject_for_key(Operation::HeadObject, key, FaultAction::Stall(Duration::from_secs(5)), 1);
    let started = Instant::now();
    let response = env.raw_get(bucket, key).await?;
    let elapsed = started.elapsed();
    assert_eq!(response.status, SOURCE_UNAVAILABLE_STATUS, "{}", String::from_utf8_lossy(&response.body));
-    assert!(
-        elapsed < Duration::from_secs(30),
-        "the read timeout must cut every attempt short, took {elapsed:?}"
+    assert_eq!(
+        env.source.count_requests(Operation::HeadObject, key),
+        1,
+        "the stalled HEAD is the only source request"
    );
-    let attempts = env.source.count_requests(Operation::HeadObject, key);
-    assert!(attempts >= 1, "the stalled HEAD is the only source request");
    assert!(
-        elapsed < Duration::from_secs(5) * u32::try_from(attempts).unwrap_or(1),
-        "no attempt waited the stall out ({attempts} attempts in {elapsed:?})"
+        elapsed < Duration::from_secs(5),
+        "the read timeout must cut the attempt short, took {elapsed:?}"
    );
    assert_eq!(
        env.source.count_requests(Operation::GetObject, key),
@@ -336,7 +332,78 @@ async fn test_odm_inline_pull_aborts_when_the_source_body_is_cut() -> TestResult
    Ok(())
 }

-/// Case 5: the background pull of a large object hits a cut body, counts the
+/// Case 5: the source answers, sends part of the body and then goes quiet
+/// for longer than `source_timeout.idle_ms`. The inline tee must end both
+/// ends: the client gets a short read rather than a silently truncated 200,
+/// the pull is counted as a source timeout, and nothing (object or multipart
+/// upload) is left behind locally.
+#[tokio::test]
+async fn test_odm_inline_pull_aborts_when_the_source_body_stalls() -> TestResult {
+    let bucket = "odm-fault-inline-stall";
+    const IDLE_MS: u64 = 1_000;
+    let idle = Duration::from_millis(IDLE_MS);
+    let env = start_configured_env(bucket, SOURCE_BUCKET, |spec| {
+        spec.policy.source_timeout.idle_ms = IDLE_MS;
+    })
+    .await?;
+    let key = "stall/inline.bin";
+    let body = payload(256 * 1024);
+    env.seed_source(SOURCE_BUCKET, &[SeedObject::new(key, body.clone())]);
+
+    // The head and the first slice arrive at once; the source then pauses for
+    // four times the idle budget, which is what a stalled source looks like.
+    env.source.inject_for_key(
+        Operation::GetObject,
+        key,
+        FaultAction::SlowSendBody {
+            chunk_bytes: 32 * 1024,
+            delay: idle * 4,
+        },
+        1,
+    );
+    let started = Instant::now();
+    env.raw_get(bucket, key)
+        .await
+        .expect_err("a stalled source body must not read back as a complete object");
+    let elapsed = started.elapsed();
+    assert!(
+        elapsed < idle * 4,
+        "the idle budget, not the source's own pause, must end the read (took {elapsed:?})"
+    );
+
+    assert_eq!(env.source.count_requests(Operation::HeadObject, key), 1);
+    assert_eq!(
+        env.source.count_requests(Operation::GetObject, key),
+        1,
+        "an aborted inline pull is not retried on the same request"
+    );
+    env.wait_for_status_counter(bucket, "/counters/pull_failures_total/source_timeout", 1, SETTLE)
+        .await?;
+    // The leader releases its slot just after it records the failure.
+    let deadline = Instant::now() + SETTLE;
+    loop {
+        let inflight = env
+            .status_json(bucket)
+            .await?
+            .pointer("/inflight_pulls")
+            .and_then(|value| value.as_u64());
+        if inflight == Some(0) {
+            break;
+        }
+        assert!(Instant::now() < deadline, "the aborted pull never released its slot: {inflight:?}");
+        tokio::time::sleep(Duration::from_millis(200)).await;
+    }
+    env.assert_local_absent(bucket, key).await;
+    let uploads = env.client.list_multipart_uploads().bucket(bucket).send().await?;
+    assert!(
+        uploads.uploads().is_empty(),
+        "a stalled pull leaves no multipart upload: {:?}",
+        uploads.uploads()
+    );
+    Ok(())
+}
+
+/// Case 6: the background pull of a large object hits a cut body, counts the
 /// failure, and the retry stores the object.
 #[tokio::test]
 async fn test_odm_background_pull_retries_a_truncated_source_body() -> TestResult {
@@ -387,7 +454,7 @@ async fn test_odm_background_pull_retries_a_truncated_source_body() -> TestResul
    Ok(())
 }

-/// Case 6: the source advertises an ETag its bytes do not match. The client
+/// Case 7: the source advertises an ETag its bytes do not match. The client
 /// still gets every byte; the write-back is discarded as an integrity
 /// failure and nothing is stored.
 #[tokio::test]
@@ -415,7 +482,7 @@ async fn test_odm_wrong_source_etag_discards_the_write_back() -> TestResult {
    Ok(())
 }

-/// Case 7: a source miss is remembered for `negative_cache_ttl_secs`, and
+/// Case 8: a source miss is remembered for `negative_cache_ttl_secs`, and
 /// re-checked once the entry expires.
 #[tokio::test]
 async fn test_odm_source_not_found_is_negative_cached_for_the_ttl() -> TestResult {
@@ -455,7 +522,7 @@ async fn test_odm_source_not_found_is_negative_cached_for_the_ttl() -> TestResul
    Ok(())
 }

-/// Case 8: an SSE-C source object cannot be migrated (the key belongs to the
+/// Case 9: an SSE-C source object cannot be migrated (the key belongs to the
 /// source's client), so the read fails as unsupported without a body read.
 #[tokio::test]
 async fn test_odm_ssec_source_object_is_unsupported() -> TestResult {