A GET body that ends short of its committed Content-Length breaks every
downstream copier. Replication, site replication and `rclone sync` all read
locally and PUT remotely, so a truncated source read surfaces as an
`unexpected EOF` on the destination's PUT and an `Io error: error reading a
body from connection` 500 on the receiving server. Issue #4784 stalled for a
month because the source side reported none of it:
* `GetObjectReaderStream`'s short-read and read-error arms only fed a metric.
Their log lines sat behind the `tracing-chunk-debug` cargo feature, which is
not in the default feature set and therefore is not compiled into any
released binary.
* `GetObjectStreamingReader` did log mid-stream failures, but discarded the
bucket and key its constructor was already handed, leaving only a request_id
that cannot be resolved back to an object after the request is over.
* Those lines were `warn!`, while DEFAULT_LOG_LEVEL is `error`, so a default
deployment filtered them out even where they existed.
Keep the object identity on both readers and name it in every stream-body
failure; log the reader-stream short-read arm unconditionally; and raise the
two states that mean "the server cannot deliver the length it already
committed" - short_eof and read_failed - to `error!`. Stall timeouts, slow
first bytes and client-side drops stay at `warn!`.
The reader-stream read-error arm stays feature-gated on purpose: every
production body wraps a GetObjectStreamingReader, which already reports that
same error once with the object identity, so a second unconditional line per
failed GET would read as two distinct faults.
Tests: unit coverage asserts the captured event fields rather than just the
returned error, and a new e2e reproduces the fault over the S3 API against a
beyond-quorum damaged object, asserting the evidence at the default `error`
log level. `get_object_reader_stream_errors_on_short_eof` becomes serial: it
drives the same log callsite as the capture test, and tracing caches callsite
interest process-wide, so running it concurrently on a subscriber-less thread
re-cached that callsite as "never interested" and blinded the capture.
唐小鸭