mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-06 03:59:14 +00:00
唐小鸭
53cabe9274
* fix(replication): send an integrity header on Object Lock replication PUTs AWS S3, MinIO and most compatible targets reject a PutObject that carries x-amz-object-lock-* headers unless it also carries Content-MD5 or an x-amz-checksum-* header. Since rustfs#6895 the replication client sends plain signed payloads with no SDK checksum, so every replicated object with a retention period or legal hold failed against such targets. TargetClient::put_object now decides per request through the pure rustfs_replication::object_lock_put_integrity: a plaintext single-part object whose source ETag is its MD5 gets Content-MD5 derived from the ETag (no body pass, framing unchanged); a multipart-layout ETag, managed SSE or SSE-C passthrough falls back to an SDK CRC32; a forwarded source checksum or an unlocked PUT is left alone. The outbound target matrix flips its two KnownFailing(rustfs#7082) cells to Completed and every Completed cell now asserts that a locked PutObject carried an integrity header. Fixes rustfs#7082. * test(e2e): keep the matrix expectation table clippy-clean under -D warnings The CI lint runs cargo clippy --all-targets -- -D warnings. With every cell green the single-arm match tripped match_single_binding and the unused KnownFailing variant tripped dead_code, and the target-client tests tripped field_reassign_with_default. Drive the expectation table from a KNOWN_FAILING_CELLS constant (so the variant stays live and adding a red cell is a one-line entry), build the test options as struct literals, and refresh the e2e-repl-nightly selection digest for the renamed table test.
64 lines
7.2 KiB
Markdown
64 lines
7.2 KiB
Markdown
# Postmortem: replication checksum default flipped, Object Lock targets rejected every PUT (rustfs#7082)
|
|
|
|
**Use this when:** you change what the replication or migration client sends to a remote target by default, you are reviewing such a change, or you are triaging a report where a fix for one target class broke another.
|
|
**Source of truth:** `crates/ecstore/src/bucket/remote_s3_client.rs` (`replication_request_checksum_calculation`), `crates/e2e_test/src/replication_target_matrix_test.rs` (the outbound target matrix), `crates/e2e_test/src/fake_s3_target/mod.rs` (target failure modes).
|
|
|
|
## Summary
|
|
|
|
rustfs/rustfs#6895 fixed rustfs/rustfs#6853 (a target that stores `aws-chunked` framing verbatim) by switching the outbound SDK checksum policy from `WhenSupported` to `WhenRequired` for every replication target. The SDK-computed CRC32 had been doing a second job nobody had written down: it satisfied the S3 rule that a PutObject carrying `x-amz-object-lock-*` headers must include `Content-MD5` or an `x-amz-checksum-*` header. With the checksum gone, every replication PUT of an object with a retention period or legal hold was rejected by AWS-compatible Object Lock targets. The regression shipped in `1.0.0-rc.5` and was reported by a customer two days later.
|
|
|
|
## Timeline (UTC)
|
|
|
|
| When | Event |
|
|
| --- | --- |
|
|
| 2026-08-29 14:49 | rustfs#6853 filed from the round-2 real-VM lab (SeaweedFS stored `aws-chunked` frames). |
|
|
| 2026-08-30 14:24 | PR rustfs#6895 opened. |
|
|
| 2026-08-30 17:43 | PR merged. Zero GitHub reviews; adversarial validation ran in-session. |
|
|
| 2026-08-31 | `1.0.0-rc.5-preview.1` tagged. |
|
|
| 2026-09-01 17:35 | `1.0.0-rc.5` tagged. |
|
|
| 2026-09-02 20:29 | rustfs#7082 filed by a customer replicating to an Object Lock bucket on Impossible Cloud. |
|
|
|
|
Three days from an own-lab finding to a public release candidate, with no Object Lock target exercised in between.
|
|
|
|
## Root cause
|
|
|
|
`RequestChecksumCalculation::WhenRequired` makes the SDK add a checksum only to operations whose model marks one as required. PutObject is not such an operation, so the client now sends no integrity header at all. AWS S3, MinIO and most compatible stores enforce a server-side rule the SDK model does not know about: a PutObject with Object Lock parameters needs `Content-MD5` or `x-amz-checksum-*`. RustFS itself does not enforce that rule (the matching error constant in `crates/ecstore/src/bucket/object_lock/objectlock.rs` is unused), so RustFS-to-RustFS site replication was unaffected and the gap was invisible in every in-tree test.
|
|
|
|
## Why four layers of defense all missed it
|
|
|
|
The same thing was missing at every layer: an inventory of what remote targets may require of an outbound request.
|
|
|
|
- **Change design.** A default that applies to every target was changed to satisfy one target class. The PR text considered checksums only for objects that already carried one ("object-level `x-amz-checksum-*` forwarding is unaffected") and never asked the inverse question: which target-side rules depend on a checksum being present.
|
|
- **Tests.** The new unit test asserted the fix ("no `x-amz-trailer`"), not the contract ("the target accepted the PUT"). The fake target modeled neither failure mode: it did not store `aws-chunked` verbatim and it did not demand a checksum on locked PUTs. `replication-check` has an ObjectLock phase, but its probe PUT carries no retention headers, so it would have reported OK against the customer's target.
|
|
- **Review.** The compatibility lens (`.agents/skills/adversarial-validation/references/compatibility.md`) listed only inbound and on-disk concerns: `xl.meta`, proto, MinIO fixtures. It contained no line about outbound target behavior, so the lens passed while blind to the class.
|
|
- **Release.** Two new environment escape hatches shipped undocumented. The customer's second question in rustfs#7082 was whether a configuration option exists; it did, and it was unfindable.
|
|
|
|
## Corrective actions
|
|
|
|
| Action | Where |
|
|
| --- | --- |
|
|
| Outbound target matrix e2e: every target failure mode times every object shape, with an explicit expectation table. Known-red cells are pinned to an open issue and fail loudly when they start passing. | `crates/e2e_test/src/replication_target_matrix_test.rs` |
|
|
| Fake target gains the three failure modes a real fleet has shown: rejects `aws-chunked` uploads, requires a checksum on Object Lock PUTs (and verifies `Content-MD5`), mints its own version ids. | `crates/e2e_test/src/fake_s3_target/mod.rs` |
|
|
| Compatibility lens gains an outbound target section. | `.agents/skills/adversarial-validation/references/compatibility.md` |
|
|
| `AGENTS.md` names outbound client defaults as high risk and requires the matrix plus documented escape hatches. | `AGENTS.md`, Adversarial Validation |
|
|
| The two escape hatches from rustfs#6895 are documented. | `docs/operations/replication-outbound-transport.md` |
|
|
| Product fix for rustfs#7082: a locked PUT carries `Content-MD5` derived from the source ETag when that ETag is the MD5 of the wire bytes, and an SDK CRC32 checksum otherwise; the two matrix cells flipped to `Completed` in the same change. | `crates/ecstore/src/bucket/bucket_target_sys.rs` (`object_lock_put_integrity_for`), `crates/replication/src/object.rs` (`object_lock_put_integrity`) |
|
|
| `replication-check` probe PUT with retention headers: deliberately not done. A retained probe object cannot be cleaned up on targets that deny `s3:BypassGovernanceRetention`, leaving locked residue; the matrix pins the contract instead. | `crates/e2e_test/src/replication_target_matrix_test.rs` |
|
|
|
|
## SOP: changing an outbound client default
|
|
|
|
Applies to any change in what `TargetClient`, `PutObjectOptions`, the remote SDK configuration, or the outbound header set sends by default. Treat the change as high risk for every target class, even when it fixes one.
|
|
|
|
1. **Inventory the dependents.** Before editing, list every target-side rule that the current default satisfies, not only the one the change targets. Done when the PR body names each rule and which target classes enforce it.
|
|
2. **Run the matrix.** Run `replication_target_matrix_test` locally against the built binary. Done when every cell matches its expectation and no expectation was edited to make it pass.
|
|
3. **Extend the matrix for the new failure mode.** If the change is motivated by a target behavior the fake does not model, add the mode to the fake first and add a cell that is red before the fix. Done when the cell is green only with the fix applied.
|
|
4. **Document every escape hatch in the same PR.** Each new environment knob gets an entry in `docs/operations/replication-outbound-transport.md`. Done when the knob name appears there.
|
|
5. **Record the coverage.** The PR Impact section lists the target classes verified and, explicitly, the ones not verified. Done when a reader can tell which cells were never run.
|
|
6. **Soak before a release candidate.** A fix that changes outbound behavior runs once through the real-target lab, which must include an Object Lock target and an AWS or MinIO target, before it is tagged into a release candidate.
|
|
|
|
## Related
|
|
|
|
- rustfs/backlog#2085 — product boundary for generic S3 targets (version identity).
|
|
- `docs/operations/replication-object-size-limits.md` — single PUT and multipart limits on generic targets.
|
|
- `docs/operations/replication-check.md` — the probe phases and their contract.
|