gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-09-06 03:59:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

test(e2e): add the outbound target matrix and #7082 postmortem (#7092)

Browse Source 
test(e2e): add the outbound target matrix and the replication checksum postmortem

Defense work for rustfs#7082, the regression rustfs#6895 introduced while
fixing rustfs#6853: a fix for one target class changed a client default for
every target class and nothing in tree modeled the other classes.

- docs/postmortems: timeline, root cause, why four defense layers missed
  it, and the SOP for changing any outbound client default; AGENTS.md and
  the adversarial compatibility lens point at it; the two env knobs from
  rustfs#6895 are documented in docs/operations.
- fake_s3_target: reject_aws_chunked_uploads, require_checksum_for_object_lock
  (Content-MD5 always verified), create_bucket_with_object_lock with a
  GetObjectLockConfiguration handler, and a TransportSnapshot on every
  journal record.
- replication_target_matrix_test: six object shapes against four target
  modes with an explicit expectation table; the two rustfs#7082 cells are
  pinned KnownFailing and fail with an XPASS message once the fix lands.
  Wired into e2e-repl-nightly, excluded from e2e-full.
This commit is contained in:
唐小鸭
2026-09-03 14:57:05 +08:00
committed by GitHub
parent 54a7e9f307
commit 8023cf3e26
16 changed files with 1036 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/operations/replication-outbound-transport.md
+33
View File
@@ -0,0 +1,33 @@
# Replication outbound transport
**Use this when:** a bucket-replication or site-replication target rejects, corrupts, or silently transforms uploads from RustFS, or you need to know which integrity headers RustFS sends to a remote target and how to change them.
**Source of truth:** `crates/ecstore/src/bucket/remote_s3_client.rs` (`replication_request_checksum_calculation`), `crates/ecstore/src/bucket/bucket_target_sys.rs` (`TargetClient::put_object`, `PutObjectOptions::header`), `crates/ecstore/src/bucket/replication/replication_resyncer.rs` (`verify_single_part_replica`).
## What a replication PUT carries by default
- A plain signed body with an exact `Content-Length`. The SDK does not add a streaming trailer checksum, so the body is never wrapped in `aws-chunked` framing (rustfs#6853: a target that does not decode that framing stored the frames verbatim while RustFS recorded COMPLETED).
- Any object-level checksum the source object was uploaded with, forwarded as its `x-amz-checksum-*` header.
- The source ETag, mtime and version id on `x-rustfs-source-*` headers (with `x-minio-source-*` twins), and the Object Lock mode, retain-until date and legal hold of the source version when present.
- After the PUT, the target's ETag is compared with the source ETag when both are plain single-part MD5s; a mismatch fails the replication instead of reporting a corrupted replica as COMPLETED.
## Target classes and their known requirements
| Target behavior | Effect on RustFS replication | Detected by |
| --- | --- | --- |
| Rejects or mis-stores `aws-chunked` bodies (SeaweedFS 3.97) | Handled by the plain-payload default above. | Outbound target matrix, `RejectAwsChunked` mode |
| Requires `Content-MD5` or `x-amz-checksum-*` on a PutObject with Object Lock parameters (AWS S3, MinIO, Impossible Cloud, most compatible stores) | Objects with a retention period or legal hold fail until rustfs#7082 lands. Set `RUSTFS_REPLICATION_STREAMING_CHECKSUMS=true` as a workaround when the target also decodes `aws-chunked`. | Outbound target matrix, `RequireChecksumWithObjectLock` mode |
| Mints its own version ids (AWS S3, Wasabi, Impossible Cloud) | Data lands; version-addressed convergence does not. See rustfs/backlog#2085 and `docs/operations/replication-check.md` (VersionFidelity). | `replication-check`, outbound target matrix, `MintOwnVersionIds` mode |
| Returns an ETag that is not the content MD5 without announcing SSE | Every single-part object fails ETag verification. Set `RUSTFS_REPLICATION_REPLICA_ETAG_VERIFY=false`. | Replication status FAILED with `replica etag mismatch` |
## Environment knobs
| Variable | Default | Meaning |
| --- | --- | --- |
| `RUSTFS_REPLICATION_STREAMING_CHECKSUMS` | unset (plain payloads) | `true` or `1` restores SDK trailer checksums (`RequestChecksumCalculation::WhenSupported`). Every streaming upload is then `aws-chunked` with an `x-amz-trailer`; use only when every target decodes that framing. |
| `RUSTFS_REPLICATION_REPLICA_ETAG_VERIFY` | enabled | `false` or `0` disables the post-PUT ETag comparison for targets whose 32-hex ETags are legitimately not the content MD5. |
Both knobs are read by the RustFS process that owns the replication target, at client build time; restart the server after changing them.
## Before changing any of this
Follow the SOP in `docs/postmortems/2026-09-03-replication-checksum-default-regression.md`: inventory the target-side rules the current default satisfies, run the outbound target matrix, and document any new knob here in the same PR.
docs/postmortems/2026-09-03-replication-checksum-default-regression.md
+63
View File
@@ -0,0 +1,63 @@
# Postmortem: replication checksum default flipped, Object Lock targets rejected every PUT (rustfs#7082)
**Use this when:** you change what the replication or migration client sends to a remote target by default, you are reviewing such a change, or you are triaging a report where a fix for one target class broke another.
**Source of truth:** `crates/ecstore/src/bucket/remote_s3_client.rs` (`replication_request_checksum_calculation`), `crates/e2e_test/src/replication_target_matrix_test.rs` (the outbound target matrix), `crates/e2e_test/src/fake_s3_target/mod.rs` (target failure modes).
## Summary
rustfs/rustfs#6895 fixed rustfs/rustfs#6853 (a target that stores `aws-chunked` framing verbatim) by switching the outbound SDK checksum policy from `WhenSupported` to `WhenRequired` for every replication target. The SDK-computed CRC32 had been doing a second job nobody had written down: it satisfied the S3 rule that a PutObject carrying `x-amz-object-lock-*` headers must include `Content-MD5` or an `x-amz-checksum-*` header. With the checksum gone, every replication PUT of an object with a retention period or legal hold was rejected by AWS-compatible Object Lock targets. The regression shipped in `1.0.0-rc.5` and was reported by a customer two days later.
## Timeline (UTC)
| When | Event |
| --- | --- |
| 2026-08-29 14:49 | rustfs#6853 filed from the round-2 real-VM lab (SeaweedFS stored `aws-chunked` frames). |
| 2026-08-30 14:24 | PR rustfs#6895 opened. |
| 2026-08-30 17:43 | PR merged. Zero GitHub reviews; adversarial validation ran in-session. |
| 2026-08-31 | `1.0.0-rc.5-preview.1` tagged. |
| 2026-09-01 17:35 | `1.0.0-rc.5` tagged. |
| 2026-09-02 20:29 | rustfs#7082 filed by a customer replicating to an Object Lock bucket on Impossible Cloud. |
Three days from an own-lab finding to a public release candidate, with no Object Lock target exercised in between.
## Root cause
`RequestChecksumCalculation::WhenRequired` makes the SDK add a checksum only to operations whose model marks one as required. PutObject is not such an operation, so the client now sends no integrity header at all. AWS S3, MinIO and most compatible stores enforce a server-side rule the SDK model does not know about: a PutObject with Object Lock parameters needs `Content-MD5` or `x-amz-checksum-*`. RustFS itself does not enforce that rule (the matching error constant in `crates/ecstore/src/bucket/object_lock/objectlock.rs` is unused), so RustFS-to-RustFS site replication was unaffected and the gap was invisible in every in-tree test.
## Why four layers of defense all missed it
The same thing was missing at every layer: an inventory of what remote targets may require of an outbound request.
- **Change design.** A default that applies to every target was changed to satisfy one target class. The PR text considered checksums only for objects that already carried one ("object-level `x-amz-checksum-*` forwarding is unaffected") and never asked the inverse question: which target-side rules depend on a checksum being present.
- **Tests.** The new unit test asserted the fix ("no `x-amz-trailer`"), not the contract ("the target accepted the PUT"). The fake target modeled neither failure mode: it did not store `aws-chunked` verbatim and it did not demand a checksum on locked PUTs. `replication-check` has an ObjectLock phase, but its probe PUT carries no retention headers, so it would have reported OK against the customer's target.
- **Review.** The compatibility lens (`.agents/skills/adversarial-validation/references/compatibility.md`) listed only inbound and on-disk concerns: `xl.meta`, proto, MinIO fixtures. It contained no line about outbound target behavior, so the lens passed while blind to the class.
- **Release.** Two new environment escape hatches shipped undocumented. The customer's second question in rustfs#7082 was whether a configuration option exists; it did, and it was unfindable.
## Corrective actions
| Action | Where |
| --- | --- |
| Outbound target matrix e2e: every target failure mode times every object shape, with an explicit expectation table. Known-red cells are pinned to an open issue and fail loudly when they start passing. | `crates/e2e_test/src/replication_target_matrix_test.rs` |
| Fake target gains the three failure modes a real fleet has shown: rejects `aws-chunked` uploads, requires a checksum on Object Lock PUTs (and verifies `Content-MD5`), mints its own version ids. | `crates/e2e_test/src/fake_s3_target/mod.rs` |
| Compatibility lens gains an outbound target section. | `.agents/skills/adversarial-validation/references/compatibility.md` |
| `AGENTS.md` names outbound client defaults as high risk and requires the matrix plus documented escape hatches. | `AGENTS.md`, Adversarial Validation |
| The two escape hatches from rustfs#6895 are documented. | `docs/operations/replication-outbound-transport.md` |
| Product fix for rustfs#7082: derive `Content-MD5` from the source ETag on locked PUTs. | tracked in rustfs#7082 |
| `replication-check` probe PUT carries retention headers when the target bucket has Object Lock. | tracked in rustfs#7082 |
## SOP: changing an outbound client default
Applies to any change in what `TargetClient`, `PutObjectOptions`, the remote SDK configuration, or the outbound header set sends by default. Treat the change as high risk for every target class, even when it fixes one.
1. **Inventory the dependents.** Before editing, list every target-side rule that the current default satisfies, not only the one the change targets. Done when the PR body names each rule and which target classes enforce it.
2. **Run the matrix.** Run `replication_target_matrix_test` locally against the built binary. Done when every cell matches its expectation and no expectation was edited to make it pass.
3. **Extend the matrix for the new failure mode.** If the change is motivated by a target behavior the fake does not model, add the mode to the fake first and add a cell that is red before the fix. Done when the cell is green only with the fix applied.
4. **Document every escape hatch in the same PR.** Each new environment knob gets an entry in `docs/operations/replication-outbound-transport.md`. Done when the knob name appears there.
5. **Record the coverage.** The PR Impact section lists the target classes verified and, explicitly, the ones not verified. Done when a reader can tell which cells were never run.
6. **Soak before a release candidate.** A fix that changes outbound behavior runs once through the real-target lab, which must include an Object Lock target and an AWS or MinIO target, before it is tagged into a release candidate.
## Related
- rustfs/backlog#2085 — product boundary for generic S3 targets (version identity).
- `docs/operations/replication-object-size-limits.md` — single PUT and multipart limits on generic targets.
- `docs/operations/replication-check.md` — the probe phases and their contract.
docs/testing/README.md
+1
View File
@@ -1,3 +1,4 @@
| Outbound target matrix | Replication of every object shape (empty, plain, retention, legal hold, multipart) against every remote-target failure mode the fake target models; an explicit expectation table pins known-red cells to an open issue | `cargo nextest run -p e2e_test -E 'test(/^replication_target_matrix_test::/)'` (build `target/debug/rustfs` first) | With `e2e-repl-nightly`; required locally for any change to outbound client defaults (SOP: [`docs/postmortems/2026-09-03-replication-checksum-default-regression.md`](../postmortems/2026-09-03-replication-checksum-default-regression.md)) |
# RustFS Testing
**Use this when:** you need to pick a test layer for a change, name a test so a gate keeps selecting it, understand why `#[serial]` does nothing under nextest, or handle a flaky test.