mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-07 12:35:54 +00:00
唐小鸭
899f81f3ad
* fix(replication): resolve drifted replicas via a target version ledger A replication target that mints its own version ids (Wasabi, AWS S3) never answers to the source uuid, so every version-addressed mutation after the initial PUT failed forever: permanent version deletes answered NoSuchVersion every heal cycle, and tag / retention / legal-hold updates re-PUT the object, minting one more target version per update (rustfs/backlog#2340). Record the id the target assigned as a per-target ledger on the source version (replication-target-version-<arn>, written through the existing status writeback) and resolve every later mutation through it: version deletes DELETE the ledger id, metadata updates go through the metadata-only Object Lock and tagging APIs. Replicas written before the ledger existed are located by exact key and ETag, minus the candidates other generations of the key already claim through their own ledgers; an ambiguous remainder is refused with a backoff instead of guessed, since a wrong pick would destroy a live generation. A fresh write never consults content identity. NoSuchVersion on a version-addressed DELETE counts as purged. The fake target gains the Wasabi shape (404 NoSuchVersion on an unknown id, per-version Object Lock APIs) and the matrix covers the three mutation classes plus the same-bytes generation case. * fix(scanner): drop the unused Digest import Same one-line change as rustfs/rustfs#7366 (main is red with it under -D warnings); carried here so the stacked PRs' merge commits compile until that fix lands. * fix(admin): probe replication-check mutations by the assigned version id (#7373) On a target that mints its own version ids the DeleteMarker and VersionDelete phases of ?replication-check were skipped: they addressed the source id, which such a target never had. The replication worker now addresses the id the target assigned (the target-version ledger), and the probe already holds that id from its own PUT, so run both phases against it. VersionFidelity keeps failing with the mismatch code and the target stays FAILED; the phases report whether ledger-addressed purges work against this endpoint (rustfs/backlog#2340). * fix(replication): abandon purges to targets the bucket no longer names (#7377) * fix(admin): probe replication-check mutations by the assigned version id On a target that mints its own version ids the DeleteMarker and VersionDelete phases of ?replication-check were skipped: they addressed the source id, which such a target never had. The replication worker now addresses the id the target assigned (the target-version ledger), and the probe already holds that id from its own PUT, so run both phases against it. VersionFidelity keeps failing with the mismatch code and the target stays FAILED; the phases report whether ledger-addressed purges work against this endpoint (rustfs/backlog#2340). * fix(replication): abandon purges to targets the bucket no longer names A permanent version delete whose replication keeps failing stays in xl.meta as a PENDING purge, hidden from listings, until every target confirms it. Once the operator removes the replication configuration or the rule naming that target nothing ever confirms it: the heal path derived its delete decision from the configuration (the decision string is not persisted) and skipped the version forever, so DeleteBucket answered BucketNotEmpty for a residue the client could neither list nor remove (rustfs/backlog#2340). Owe a version purge to the targets its purge state names, let the heal path through without a configuration, and have the delete worker settle a target the configuration no longer names as abandoned: the purge is reported complete locally through the normal writeback, the replica on the former target is left alone, and the event replication_purge_abandoned plus a counter are the record. * fix(admin): send replication-check marker creation without a version id Running the DeleteMarker / VersionDelete phases on a target that mints its own version ids exposed two probe-shape bugs on real Wasabi: - the DeleteMarker phase put the assigned version id on its DELETE. A RustFS peer reads the source-deletemarker header and creates a marker, but a generic S3 target executes it as a permanent delete of the probe version, so VersionDelete then answered NoSuchVersion. Use the same wire shape as live delete replication: no versionId on a marker creation. - cleanup treated NoSuchVersion on the version the VersionDelete phase had already removed as a failure (RustFS/MinIO answer 204 there). Also gate the no-configuration heal pass-through for pending purges on a purge state that actually names targets, so a purge without a recorded target keeps the ordinary skip (scanner unit test), and merge origin/main (#7365 settles the pool-metadata probe test that failed in CI). --------- Co-authored-by: houseme <housemecn@gmail.com>
6.4 KiB
6.4 KiB
Replication outbound transport
Use this when: a bucket-replication or site-replication target rejects, corrupts, or silently transforms uploads from RustFS, or you need to know which integrity headers RustFS sends to a remote target and how to change them.
Source of truth: crates/ecstore/src/bucket/remote_s3_client.rs (replication_request_checksum_calculation), crates/ecstore/src/bucket/bucket_target_sys.rs (TargetClient::put_object, PutObjectOptions::header), crates/ecstore/src/bucket/replication/replication_resyncer.rs (verify_single_part_replica).
What a replication PUT carries by default
- A plain signed body with an exact
Content-Length. The SDK does not add a streaming trailer checksum, so the body is never wrapped inaws-chunkedframing (rustfs#6853: a target that does not decode that framing stored the frames verbatim while RustFS recorded COMPLETED). - For a single-part object, the checksum the source object was uploaded with, forwarded as its
x-amz-checksum-<algorithm>header (the value the source verified on upload). A multipart replica is rebuilt through CreateMultipartUpload/UploadPart and carries no object-level checksum header. Managed-SSE objects forward none. - On a PUT that carries Object Lock parameters and no forwarded checksum:
Content-MD5derived from the source ETag, or an SDK CRC32 checksum when the ETag is not the MD5 of the wire bytes (rustfs#7082). - The source ETag, mtime and version id on
x-rustfs-source-*headers (withx-minio-source-*twins), and the Object Lock mode, retain-until date and legal hold of the source version when present. - After the PUT, the target's ETag is compared with the source ETag when both are plain single-part MD5s; a mismatch fails the replication instead of reporting a corrupted replica as COMPLETED.
Target classes and their known requirements
| Target behavior | Effect on RustFS replication | Detected by |
|---|---|---|
Rejects or mis-stores aws-chunked bodies (SeaweedFS 3.97) |
Handled by the plain-payload default above. | Outbound target matrix, RejectAwsChunked mode |
Requires Content-MD5 or x-amz-checksum-* on a PutObject with Object Lock parameters (AWS S3, MinIO, Impossible Cloud, most compatible stores) |
Satisfied: a locked single PUT carries Content-MD5 derived from the source ETag (plaintext objects whose ETag is the MD5 of the wire bytes) or an SDK CRC32 checksum (multipart-layout ETags, managed SSE, SSE-C passthrough — this one is an aws-chunked trailer, so a target that also rejects that framing cannot take such objects). Releases before this fix (1.0.0-rc.5) need RUSTFS_REPLICATION_STREAMING_CHECKSUMS=true as a workaround. |
Outbound target matrix, RequireChecksumWithObjectLock mode |
Stores x-amz-checksum-* from a PutObject and returns it on HEAD ?ChecksumMode=ENABLED (AWS S3, Wasabi, RustFS) |
Satisfied for single-part objects: the replica answers with the source's checksum. Before this fix (1.0.0-rc.5) the checksum left the source as x-amz-meta-<algorithm> user metadata and no replica carried it (rustfs/backlog#2340). |
Outbound target matrix, Checksummed shape |
| Mints its own version ids (AWS S3, Wasabi, Impossible Cloud) | Data lands, and later mutations converge through the target-version ledger: the id the target assigned is recorded on the source version (internal key replication-target-version-<arn>) and version deletes, tag and Object Lock updates address it (tag/retention/legal-hold through the metadata-only APIs, never a re-PUT). A replica written before the ledger existed is located by exact key + ETag, minus the candidates other generations of the key already claim; an ambiguous remainder is refused with a one-hour backoff (replication_purge_replica_unresolved). NoSuchVersion on a version-addressed DELETE counts as purged. See rustfs/backlog#2340, rustfs/backlog#2085 and docs/operations/replication-check.md (VersionFidelity). |
replication-check, outbound target matrix, MintOwnVersionIds mode (also models Wasabi's 404 NoSuchVersion on unknown ids) |
| Returns an ETag that is not the content MD5 without announcing SSE | Every single-part object fails ETag verification. Set RUSTFS_REPLICATION_REPLICA_ETAG_VERIFY=false. |
Replication status FAILED with replica etag mismatch |
Environment knobs
| Variable | Default | Meaning |
|---|---|---|
RUSTFS_REPLICATION_STREAMING_CHECKSUMS |
unset (plain payloads) | true or 1 restores SDK trailer checksums (RequestChecksumCalculation::WhenSupported). Every streaming upload is then aws-chunked with an x-amz-trailer, except a single-part PUT that forwards the source's x-amz-checksum-* header, which is sent plain so the target does not receive a second algorithm; use only when every target decodes that framing. |
RUSTFS_REPLICATION_REPLICA_ETAG_VERIFY |
enabled | false or 0 disables the post-PUT ETag comparison for targets whose 32-hex ETags are legitimately not the content MD5. |
Both knobs are read by the RustFS process that owns the replication target, at client build time; restart the server after changing them.
Remote tier transport timeouts
Remote tier S3-compatible clients use separate transport budgets. These settings do not change bucket or site replication clients.
| Variable | Default | Meaning |
|---|---|---|
RUSTFS_TIER_REMOTE_CONNECT_TIMEOUT_SECS |
10 |
Maximum time to establish the remote tier TCP connection. |
RUSTFS_TIER_REMOTE_REQUEST_TIMEOUT_SECS |
86400 |
Maximum time for a remote tier request to reach response headers. The long default preserves large transition-upload headroom. |
RUSTFS_TIER_REMOTE_RESPONSE_BODY_IDLE_TIMEOUT_SECS |
60 |
Maximum time without a non-empty response-body chunk. Empty HTTP/2 frames do not count as progress. |
All three values must be positive integers. Zero fails tier client initialization instead of silently disabling the boundary. An invalid integer is logged and falls back to the default; very large values are accepted and provide a correspondingly long effective budget. The values are read when the tier client is built; recreate or reload the tier configuration after changing them.
Before changing any of this
Follow the SOP in docs/postmortems/2026-09-03-replication-checksum-default-regression.md: inventory the target-side rules the current default satisfies, run the outbound target matrix, and document any new knob here in the same PR.