mirror of https://github.com/rustfs/rustfs.git synced 2026-09-06 12:09:12 +00:00

Files

T

唐小鸭 8023cf3e26 test(e2e): add the outbound target matrix and #7082 postmortem (#7092 )

test(e2e): add the outbound target matrix and the replication checksum postmortem

Defense work for rustfs#7082, the regression rustfs#6895 introduced while
fixing rustfs#6853: a fix for one target class changed a client default for
every target class and nothing in tree modeled the other classes.

- docs/postmortems: timeline, root cause, why four defense layers missed
  it, and the SOP for changing any outbound client default; AGENTS.md and
  the adversarial compatibility lens point at it; the two env knobs from
  rustfs#6895 are documented in docs/operations.
- fake_s3_target: reject_aws_chunked_uploads, require_checksum_for_object_lock
  (Content-MD5 always verified), create_bucket_with_object_lock with a
  GetObjectLockConfiguration handler, and a TransportSnapshot on every
  journal record.
- replication_target_matrix_test: six object shapes against four target
  modes with an explicit expectation table; the two rustfs#7082 cells are
  pinned KnownFailing and fail with an XPASS message once the fix lands.
  Wired into e2e-repl-nightly, excluded from e2e-full.

2026-09-03 14:57:05 +08:00

3.5 KiB

Raw Blame History

Replication outbound transport

Use this when: a bucket-replication or site-replication target rejects, corrupts, or silently transforms uploads from RustFS, or you need to know which integrity headers RustFS sends to a remote target and how to change them. Source of truth: crates/ecstore/src/bucket/remote_s3_client.rs (replication_request_checksum_calculation), crates/ecstore/src/bucket/bucket_target_sys.rs (TargetClient::put_object, PutObjectOptions::header), crates/ecstore/src/bucket/replication/replication_resyncer.rs (verify_single_part_replica).

What a replication PUT carries by default

A plain signed body with an exact Content-Length. The SDK does not add a streaming trailer checksum, so the body is never wrapped in aws-chunked framing (rustfs#6853: a target that does not decode that framing stored the frames verbatim while RustFS recorded COMPLETED).
Any object-level checksum the source object was uploaded with, forwarded as its x-amz-checksum-* header.
The source ETag, mtime and version id on x-rustfs-source-* headers (with x-minio-source-* twins), and the Object Lock mode, retain-until date and legal hold of the source version when present.
After the PUT, the target's ETag is compared with the source ETag when both are plain single-part MD5s; a mismatch fails the replication instead of reporting a corrupted replica as COMPLETED.

Target classes and their known requirements

Target behavior	Effect on RustFS replication	Detected by
Rejects or mis-stores `aws-chunked` bodies (SeaweedFS 3.97)	Handled by the plain-payload default above.	Outbound target matrix, `RejectAwsChunked` mode
Requires `Content-MD5` or `x-amz-checksum-*` on a PutObject with Object Lock parameters (AWS S3, MinIO, Impossible Cloud, most compatible stores)	Objects with a retention period or legal hold fail until rustfs#7082 lands. Set `RUSTFS_REPLICATION_STREAMING_CHECKSUMS=true` as a workaround when the target also decodes `aws-chunked`.	Outbound target matrix, `RequireChecksumWithObjectLock` mode
Mints its own version ids (AWS S3, Wasabi, Impossible Cloud)	Data lands; version-addressed convergence does not. See rustfs/backlog#2085 and `docs/operations/replication-check.md` (VersionFidelity).	`replication-check`, outbound target matrix, `MintOwnVersionIds` mode
Returns an ETag that is not the content MD5 without announcing SSE	Every single-part object fails ETag verification. Set `RUSTFS_REPLICATION_REPLICA_ETAG_VERIFY=false`.	Replication status FAILED with `replica etag mismatch`

Environment knobs

Variable	Default	Meaning
`RUSTFS_REPLICATION_STREAMING_CHECKSUMS`	unset (plain payloads)	`true` or `1` restores SDK trailer checksums (`RequestChecksumCalculation::WhenSupported`). Every streaming upload is then `aws-chunked` with an `x-amz-trailer`; use only when every target decodes that framing.
`RUSTFS_REPLICATION_REPLICA_ETAG_VERIFY`	enabled	`false` or `0` disables the post-PUT ETag comparison for targets whose 32-hex ETags are legitimately not the content MD5.

Both knobs are read by the RustFS process that owns the replication target, at client build time; restart the server after changing them.

Before changing any of this

Follow the SOP in docs/postmortems/2026-09-03-replication-checksum-default-regression.md: inventory the target-side rules the current default satisfies, run the outbound target matrix, and document any new knob here in the same PR.

3.5 KiB Raw Blame History