mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-05 19:55:37 +00:00
唐小鸭
53cabe9274
* fix(replication): send an integrity header on Object Lock replication PUTs AWS S3, MinIO and most compatible targets reject a PutObject that carries x-amz-object-lock-* headers unless it also carries Content-MD5 or an x-amz-checksum-* header. Since rustfs#6895 the replication client sends plain signed payloads with no SDK checksum, so every replicated object with a retention period or legal hold failed against such targets. TargetClient::put_object now decides per request through the pure rustfs_replication::object_lock_put_integrity: a plaintext single-part object whose source ETag is its MD5 gets Content-MD5 derived from the ETag (no body pass, framing unchanged); a multipart-layout ETag, managed SSE or SSE-C passthrough falls back to an SDK CRC32; a forwarded source checksum or an unlocked PUT is left alone. The outbound target matrix flips its two KnownFailing(rustfs#7082) cells to Completed and every Completed cell now asserts that a locked PutObject carried an integrity header. Fixes rustfs#7082. * test(e2e): keep the matrix expectation table clippy-clean under -D warnings The CI lint runs cargo clippy --all-targets -- -D warnings. With every cell green the single-arm match tripped match_single_binding and the unused KnownFailing variant tripped dead_code, and the target-client tests tripped field_reassign_with_default. Drive the expectation table from a KNOWN_FAILING_CELLS constant (so the variant stays live and adding a red cell is a one-line entry), build the test options as struct literals, and refresh the e2e-repl-nightly selection digest for the renamed table test.
3.9 KiB
3.9 KiB
Replication outbound transport
Use this when: a bucket-replication or site-replication target rejects, corrupts, or silently transforms uploads from RustFS, or you need to know which integrity headers RustFS sends to a remote target and how to change them.
Source of truth: crates/ecstore/src/bucket/remote_s3_client.rs (replication_request_checksum_calculation), crates/ecstore/src/bucket/bucket_target_sys.rs (TargetClient::put_object, PutObjectOptions::header), crates/ecstore/src/bucket/replication/replication_resyncer.rs (verify_single_part_replica).
What a replication PUT carries by default
- A plain signed body with an exact
Content-Length. The SDK does not add a streaming trailer checksum, so the body is never wrapped inaws-chunkedframing (rustfs#6853: a target that does not decode that framing stored the frames verbatim while RustFS recorded COMPLETED). - Any object-level checksum the source object was uploaded with, forwarded as its
x-amz-checksum-*header. - On a PUT that carries Object Lock parameters and no forwarded checksum:
Content-MD5derived from the source ETag, or an SDK CRC32 checksum when the ETag is not the MD5 of the wire bytes (rustfs#7082). - The source ETag, mtime and version id on
x-rustfs-source-*headers (withx-minio-source-*twins), and the Object Lock mode, retain-until date and legal hold of the source version when present. - After the PUT, the target's ETag is compared with the source ETag when both are plain single-part MD5s; a mismatch fails the replication instead of reporting a corrupted replica as COMPLETED.
Target classes and their known requirements
| Target behavior | Effect on RustFS replication | Detected by |
|---|---|---|
Rejects or mis-stores aws-chunked bodies (SeaweedFS 3.97) |
Handled by the plain-payload default above. | Outbound target matrix, RejectAwsChunked mode |
Requires Content-MD5 or x-amz-checksum-* on a PutObject with Object Lock parameters (AWS S3, MinIO, Impossible Cloud, most compatible stores) |
Satisfied: a locked single PUT carries Content-MD5 derived from the source ETag (plaintext objects whose ETag is the MD5 of the wire bytes) or an SDK CRC32 checksum (multipart-layout ETags, managed SSE, SSE-C passthrough — this one is an aws-chunked trailer, so a target that also rejects that framing cannot take such objects). Releases before this fix (1.0.0-rc.5) need RUSTFS_REPLICATION_STREAMING_CHECKSUMS=true as a workaround. |
Outbound target matrix, RequireChecksumWithObjectLock mode |
| Mints its own version ids (AWS S3, Wasabi, Impossible Cloud) | Data lands; version-addressed convergence does not. See rustfs/backlog#2085 and docs/operations/replication-check.md (VersionFidelity). |
replication-check, outbound target matrix, MintOwnVersionIds mode |
| Returns an ETag that is not the content MD5 without announcing SSE | Every single-part object fails ETag verification. Set RUSTFS_REPLICATION_REPLICA_ETAG_VERIFY=false. |
Replication status FAILED with replica etag mismatch |
Environment knobs
| Variable | Default | Meaning |
|---|---|---|
RUSTFS_REPLICATION_STREAMING_CHECKSUMS |
unset (plain payloads) | true or 1 restores SDK trailer checksums (RequestChecksumCalculation::WhenSupported). Every streaming upload is then aws-chunked with an x-amz-trailer; use only when every target decodes that framing. |
RUSTFS_REPLICATION_REPLICA_ETAG_VERIFY |
enabled | false or 0 disables the post-PUT ETag comparison for targets whose 32-hex ETags are legitimately not the content MD5. |
Both knobs are read by the RustFS process that owns the replication target, at client build time; restart the server after changing them.
Before changing any of this
Follow the SOP in docs/postmortems/2026-09-03-replication-checksum-default-regression.md: inventory the target-side rules the current default satisfies, run the outbound target matrix, and document any new knob here in the same PR.