fix(replication): send an integrity header on Object Lock PUTs (#7097)

* fix(replication): send an integrity header on Object Lock replication PUTs

AWS S3, MinIO and most compatible targets reject a PutObject that carries
x-amz-object-lock-* headers unless it also carries Content-MD5 or an
x-amz-checksum-* header. Since rustfs#6895 the replication client sends
plain signed payloads with no SDK checksum, so every replicated object
with a retention period or legal hold failed against such targets.

TargetClient::put_object now decides per request through the pure
rustfs_replication::object_lock_put_integrity: a plaintext single-part
object whose source ETag is its MD5 gets Content-MD5 derived from the
ETag (no body pass, framing unchanged); a multipart-layout ETag, managed
SSE or SSE-C passthrough falls back to an SDK CRC32; a forwarded source
checksum or an unlocked PUT is left alone.

The outbound target matrix flips its two KnownFailing(rustfs#7082) cells
to Completed and every Completed cell now asserts that a locked
PutObject carried an integrity header.

Fixes rustfs#7082.

* test(e2e): keep the matrix expectation table clippy-clean under -D warnings

The CI lint runs cargo clippy --all-targets -- -D warnings. With every cell
green the single-arm match tripped match_single_binding and the unused
KnownFailing variant tripped dead_code, and the target-client tests tripped
field_reassign_with_default. Drive the expectation table from a
KNOWN_FAILING_CELLS constant (so the variant stays live and adding a red
cell is a one-line entry), build the test options as struct literals, and
refresh the e2e-repl-nightly selection digest for the renamed table test.
This commit is contained in:
唐小鸭
2026-09-03 20:26:44 +08:00
committed by GitHub
parent 0f272ddb14
commit 53cabe9274
9 changed files with 321 additions and 38 deletions
@@ -7,6 +7,7 @@
- A plain signed body with an exact `Content-Length`. The SDK does not add a streaming trailer checksum, so the body is never wrapped in `aws-chunked` framing (rustfs#6853: a target that does not decode that framing stored the frames verbatim while RustFS recorded COMPLETED).
- Any object-level checksum the source object was uploaded with, forwarded as its `x-amz-checksum-*` header.
- On a PUT that carries Object Lock parameters and no forwarded checksum: `Content-MD5` derived from the source ETag, or an SDK CRC32 checksum when the ETag is not the MD5 of the wire bytes (rustfs#7082).
- The source ETag, mtime and version id on `x-rustfs-source-*` headers (with `x-minio-source-*` twins), and the Object Lock mode, retain-until date and legal hold of the source version when present.
- After the PUT, the target's ETag is compared with the source ETag when both are plain single-part MD5s; a mismatch fails the replication instead of reporting a corrupted replica as COMPLETED.
@@ -15,7 +16,7 @@
| Target behavior | Effect on RustFS replication | Detected by |
| --- | --- | --- |
| Rejects or mis-stores `aws-chunked` bodies (SeaweedFS 3.97) | Handled by the plain-payload default above. | Outbound target matrix, `RejectAwsChunked` mode |
| Requires `Content-MD5` or `x-amz-checksum-*` on a PutObject with Object Lock parameters (AWS S3, MinIO, Impossible Cloud, most compatible stores) | Objects with a retention period or legal hold fail until rustfs#7082 lands. Set `RUSTFS_REPLICATION_STREAMING_CHECKSUMS=true` as a workaround when the target also decodes `aws-chunked`. | Outbound target matrix, `RequireChecksumWithObjectLock` mode |
| Requires `Content-MD5` or `x-amz-checksum-*` on a PutObject with Object Lock parameters (AWS S3, MinIO, Impossible Cloud, most compatible stores) | Satisfied: a locked single PUT carries `Content-MD5` derived from the source ETag (plaintext objects whose ETag is the MD5 of the wire bytes) or an SDK CRC32 checksum (multipart-layout ETags, managed SSE, SSE-C passthrough — this one is an `aws-chunked` trailer, so a target that also rejects that framing cannot take such objects). Releases before this fix (`1.0.0-rc.5`) need `RUSTFS_REPLICATION_STREAMING_CHECKSUMS=true` as a workaround. | Outbound target matrix, `RequireChecksumWithObjectLock` mode |
| Mints its own version ids (AWS S3, Wasabi, Impossible Cloud) | Data lands; version-addressed convergence does not. See rustfs/backlog#2085 and `docs/operations/replication-check.md` (VersionFidelity). | `replication-check`, outbound target matrix, `MintOwnVersionIds` mode |
| Returns an ETag that is not the content MD5 without announcing SSE | Every single-part object fails ETag verification. Set `RUSTFS_REPLICATION_REPLICA_ETAG_VERIFY=false`. | Replication status FAILED with `replica etag mismatch` |
@@ -42,8 +42,8 @@ The same thing was missing at every layer: an inventory of what remote targets m
| Compatibility lens gains an outbound target section. | `.agents/skills/adversarial-validation/references/compatibility.md` |
| `AGENTS.md` names outbound client defaults as high risk and requires the matrix plus documented escape hatches. | `AGENTS.md`, Adversarial Validation |
| The two escape hatches from rustfs#6895 are documented. | `docs/operations/replication-outbound-transport.md` |
| Product fix for rustfs#7082: derive `Content-MD5` from the source ETag on locked PUTs. | tracked in rustfs#7082 |
| `replication-check` probe PUT carries retention headers when the target bucket has Object Lock. | tracked in rustfs#7082 |
| Product fix for rustfs#7082: a locked PUT carries `Content-MD5` derived from the source ETag when that ETag is the MD5 of the wire bytes, and an SDK CRC32 checksum otherwise; the two matrix cells flipped to `Completed` in the same change. | `crates/ecstore/src/bucket/bucket_target_sys.rs` (`object_lock_put_integrity_for`), `crates/replication/src/object.rs` (`object_lock_put_integrity`) |
| `replication-check` probe PUT with retention headers: deliberately not done. A retained probe object cannot be cleaned up on targets that deny `s3:BypassGovernanceRetention`, leaving locked residue; the matrix pins the contract instead. | `crates/e2e_test/src/replication_target_matrix_test.rs` |
## SOP: changing an outbound client default