mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-05 19:55:37 +00:00
唐小鸭
37344a84da
The guard added in #7021 fails a >5 GiB single-PutObject replication up front instead of streaming the body to a target that must reject it. Its message asserted a conclusion: "was not written as multipart on the source ... re-upload it with multipart". That text is only as right as the transport decision feeding it, and until #7047 that decision was wrong for multipart objects carrying a full-object checksum. On 1.0.0-rc.5 a 768-part object was misrouted to the single-PUT path, and the new default-level error line told the operator to re-upload as multipart an object whose own ETag ended in -768. State the evidence instead of the conclusion. The message now quotes the ETag the decision was read from and says what was read from it (no part-count suffix), so an operator can check the line against the object's listing. A misroute then reads as a visible contradiction -- a suffixed ETag on a single-PUT line -- and the message says that case is a transport-selection defect to report, not something to fix by re-uploading. A missing or empty ETag is printed as <none> rather than hidden. The routing itself is already fixed by #7047; this changes only what the guard says when it fires.
116 lines
5.6 KiB
Markdown
116 lines
5.6 KiB
Markdown
# Replication object size and shape limits (generic S3 targets)
|
|
|
|
**Use this when:** an object fails to replicate to an S3-compatible target with `EntityTooLarge`/`EntityTooSmall`, or you need to know whether a large or oddly-chunked object is replicable before relying on it.
|
|
**Source of truth:** `crates/ecstore/src/bucket/replication/` (transport selection and part replay), `crates/replication/` (target client), `crates/config/src/constants/` (`RUSTFS_OBS_LOGGER_LEVEL`).
|
|
|
|
What RustFS can and cannot replicate to a generic S3 target (AWS S3, Wasabi,
|
|
MinIO, or any other S3-compatible endpoint configured as a bucket replication
|
|
target), and how a rejected object shows up in the log.
|
|
|
|
## The route is chosen by the source object's shape, not its size
|
|
|
|
RustFS mirrors how the object was written on the source:
|
|
|
|
| Source object was written as | Replication transport |
|
|
| --- | --- |
|
|
| a single `PutObject` | a single `PutObject` on the target |
|
|
| a multipart upload | a multipart upload replaying **the source's own part layout** |
|
|
|
|
RustFS does not re-chunk on the replication side. A single-`PutObject` object is
|
|
never converted into a multipart upload for the target, and a multipart object's
|
|
parts are never merged or re-split. The target's part layout is the source's,
|
|
because heal and delete convergence address the replica by that identity.
|
|
|
|
This is why object size alone does not tell you whether an object is
|
|
replicable — how it was uploaded does.
|
|
|
|
The shape is read from the object's own ETag: a multipart ETag carries a
|
|
`-<part count>` suffix. Nothing else selects the transport — in particular the
|
|
checksum algorithm and checksum type (`COMPOSITE` or `FULL_OBJECT`) an object
|
|
was uploaded with have no bearing on it.
|
|
|
|
## Limits
|
|
|
|
### Single-`PutObject` objects: 5 GiB
|
|
|
|
S3 caps `PutObject` at **5 GiB**. This is an S3 API limit that every target
|
|
enforces, not a RustFS tunable.
|
|
|
|
An object larger than 5 GiB that was written to the source with a single
|
|
`PutObject` therefore **cannot be replicated to a generic S3 target**. RustFS
|
|
detects this before streaming the body and fails the object immediately, rather
|
|
than uploading gigabytes only to collect an `EntityTooLarge` from the remote.
|
|
|
|
**Remedy:** re-upload the object using multipart. Most S3 clients do this
|
|
automatically above a threshold (the AWS CLI defaults to 8 MiB); a client
|
|
configured with a very high multipart threshold, or one that streams a single
|
|
`PutObject`, is the usual way an object ends up on the wrong side of this limit.
|
|
|
|
### Multipart objects: the target's multipart limits, applied to the source's layout
|
|
|
|
Because the source's part layout is replayed verbatim, the target's own
|
|
multipart constraints apply to that layout:
|
|
|
|
| Constraint | Target rejects with |
|
|
| --- | --- |
|
|
| every part except the last must be ≥ 5 MiB | `EntityTooSmall` |
|
|
| no part may exceed 5 GiB | `EntityTooLarge` |
|
|
| at most 10,000 parts | failure at `CompleteMultipartUpload` |
|
|
|
|
A source object whose parts satisfy these is replicable up to the S3 multipart
|
|
maximum of 5 TiB.
|
|
|
|
## Reliability characteristics for large objects
|
|
|
|
Worth knowing before replicating multi-gigabyte objects:
|
|
|
|
- Parts are transferred **sequentially**.
|
|
- There is **no part-level retry**. A failure on any single part fails the whole
|
|
object; the target-side multipart upload is then aborted so no incomplete
|
|
upload is left behind.
|
|
- Retry happens at the object level (MRF replay / heal scanner), so a failure
|
|
late in a large transfer re-sends the object from the beginning.
|
|
|
|
For a 6 GiB object this means one long all-or-nothing transfer window. Part-level
|
|
retry and resumable transfer are tracked as a separate improvement.
|
|
|
|
## What a failed object looks like in the log
|
|
|
|
A replication attempt that ends in a terminal `FAILED` state emits one `error`
|
|
line per failed target. It is at `error` deliberately: the default log level
|
|
(`RUSTFS_OBS_LOGGER_LEVEL`, default `error`) must not hide an object that never
|
|
reached its target.
|
|
|
|
```
|
|
ERROR ... event=replication_object_failed bucket=photos object=backups/vm-image.qcow2
|
|
version_id=... arn=arn:replication::wasabi endpoint=s3.wasabisys.com
|
|
op_type=OBJECT size=6442450944 replication_status=FAILED
|
|
error="object of 6442450944 bytes exceeds the 5368709120 byte single-PutObject
|
|
limit of an S3 target; its ETag 767e7a8379c0f62c39e0ceeea0e13de9 has no
|
|
part-count suffix, so replication uses a single PutObject and does not
|
|
re-chunk; if the object is in fact multipart this is a transport-selection
|
|
defect, otherwise re-upload it as a multipart upload to make it replicable"
|
|
Replication failed for object
|
|
```
|
|
|
|
The line quotes the ETag the transport decision was made from. Check it against
|
|
the object's own listing: an ETag with a `-<part count>` suffix on this line
|
|
means the object is multipart and was misrouted, which is a RustFS defect to
|
|
report, not a reason to re-upload.
|
|
|
|
The `error` field carries the target's own error code and message where the
|
|
target produced one, so a remote rejection is diagnosable without lowering the
|
|
log level and reproducing. It is passed through the same redaction as the
|
|
persisted resync detail, so an error echoing a credential or signed URL is
|
|
replaced with `[redacted sensitive resync error detail]`.
|
|
|
|
Raise `RUSTFS_OBS_LOGGER_LEVEL` to `warn` to additionally see the per-attempt
|
|
failure branches (target offline, HEAD failures, per-part errors) that sit
|
|
underneath this summary.
|
|
|
|
## Related
|
|
|
|
- [Replication target check](replication-check.md) — validate a target's
|
|
configuration, versioning, and version fidelity before relying on it.
|
|
- [Presigned size limits](presigned-size-limits.md) — per-request and per-upload caps a backend can put on presigned uploads.
|