9 Commits

Author	SHA1	Message	Date
唐小鸭	d20476a66c	fix(replication): fail SSE-C passthrough closed on targets that drop transport headers ... SSE-C ciphertext passthrough replicates via X-Rustfs-Replication-* transport headers. A MinIO/generic-S3 target silently discards them, storing bare ciphertext with no decryption material — yet the PUT succeeded, so the object reported COMPLETED with a silently unreadable replica (backlog#1675 N2). Fail-closed design: - SsecPassthroughCapability {Unknown, Supported, Unsupported} cached in BucketTargetSys per target ARN with a recording timestamp. Entries reset whenever the target is rebuilt, edited, or removed (arn_remotes_map lifecycle) and expire after SSEC_PASSTHROUGH_CAPABILITY_TTL (10 minutes): an expired verdict in either direction is re-earned through the audit, so an Unsupported target recovers automatically after an upgrade (at most one wasted PUT+HEAD audit per bad target per TTL window) and a Supported verdict cannot outlive a backend swapped behind the same endpoint. - Replication worker (replicate_object and replicate_all): fresh Unsupported targets never receive the PUT — the attempt fails immediately into the normal MRF retry channel with a "run ?replication-check to re-probe" hint. Unknown or expired verdicts are audited: after the PUT the worker HEADs the replica back through the replication-check channel (source version id mapped through resolve_read_api_version_id, so null-version objects audit correctly) and requires SSE-C evidence (the echoed customer-algorithm header); missing evidence records Unsupported and fails the attempt. Convergence HEADs are audited the same way, so a broken ciphertext replica from an earlier attempt can never launder itself into COMPLETED via an ETag match. The gate/evidence policy is pure (replication_target_boundary, staleness folded in as an input) for the M2 worker migration. - replication-check grows an SsecPassthrough probe phase: a probe PUT carrying the live transport-header shape, HEAD-back for evidence, and a machine-readable Code BucketRemoteSsecPassthroughUnsupported on failure. The probe verdict is synced into the runtime capability cache. Unlike VersionFidelity, a failed SsecPassthrough phase does NOT fail the target overall — it is a capability limit, not a broken replication contract, and a plaintext-only deployment against such a target must not turn red. - fake_s3_target: default mode now models a RustFS target (stores the transport headers, echoes SSE-C evidence); the new drop_unlisted_replication_headers mode models MinIO. The journal records whether a request carried transport headers. Receiver-echo verification: the replication-check HEAD exemption only skips SSE-C key validation; the response has always built sse-customer-algorithm from stored metadata (rustfs/src/app/object_usecase.rs), so no receiver change was needed — pinned end to end by the replication-check e2e against a real RustFS target. Rolling-upgrade constraint: RustFS targets older than the replication-check HEAD exemption (#5898) answer the audit HEAD without SSE-C evidence (or fail it outright), so SSE-C replication to such targets reports FAILED. This is deliberate — FAILED-and-retryable beats a silently undecryptable replica — and self-heals: once the target is upgraded, the next TTL expiry (or a manual ?replication-check re-probe) re-audits and records Supported. Plaintext and managed-SSE replication are unaffected. The capability cache is per-node; each node audits independently. Known limitations: - The audit judges evidence from the echoed customer-algorithm header only. A hypothetical target that preserves that one header while dropping other transport headers (partial-drop) would pass the audit; no known target behaves this way — observed targets drop the whole unknown-header family. - A mixed-version target cluster can flap the verdict between audits routed to different target nodes until the rollout completes; the TTL bounds how long each stale verdict persists. New e2e (backlog#1675 C1 + N2, red-first): fail-closed against a header-dropping fake (FAILED + no second PUT via the capability cache, journal-asserted; red run showed the old COMPLETED), replication-check reports the SsecPassthrough phase Code while the target stays OK overall, SSE-C heal convergence after a real target outage, and SSE-C existing-object resync landing a REPLICA readable with the customer key. TTL expiry in both directions is pinned at the cache and gate seams.	2026-08-18 01:59:26 +08:00
唐小鸭	9baa92563a	feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets ... Implements the MinIO active-active read-proxy protocol (P1-5 of the replication compatibility review): when a GET/HEAD/GetObjectTagging/ PutObjectTagging/DeleteObjectTagging request fails locally with not-found and the bucket has replication targets, the request is proxied to the targets in rule order, mirroring bucket-replication.go proxyGetToReplicationTarget/proxyHeadToRepTarget/proxyTaggingToRepTarget. Protocol surface: - Anti-loop: inbound {x-rustfs-,x-minio-}source-proxy-request is parsed into ObjectOptions (proxy_request + proxy_header_set, matching MinIO ProxyRequest/ProxyHeaderSet); a request carrying the marker with ANY value is never re-proxied. Outbound client proxy calls send the marker as "true"; replication worker convergence HEADs send it as "false" so a peer's proxy layer cannot answer a convergence check by proxying back to the source (which would fake Completed without a PUT). - Target selection: new replication_proxy.rs get_proxy_targets — empty when the marker is set, versioning is suspended, or no replication config; otherwise filter_target_arns -> TargetClient lookup, skipping targets with proxying disabled. - TargetClient gains head_object_for_proxy/get_object (streaming) and the three tagging calls. Proxy calls never send the replication-check SSE-C exemption header; customer SSE-C keys are forwarded verbatim so the target performs real decryption. Conditional (If-*) headers are not forwarded (MinIO parity); Range and part_number are, with parts_count/tag_count/storage_class/expiration passed through. - Metrics: proxy counters now count only real client proxy traffic, MinIO-aligned (one total per proxied request, one failed when no target served it). The previous misattributed counters — replication worker HEAD/PUT (#2672) and local tagging operations (#2682) — are removed; ReplProxyMetric now maps the tagging counters instead of dropping them. e2e (fake_s3_target extended with tagging + header journaling): proxied GET body + outbound header contract (marker present, no replication-check, SSE-C passthrough), HEAD, anti-loop 404 with zero outbound requests, GetObjectTagging, and metric mapping unit tests. Rolling note: proxying only activates for buckets with replication targets; requests carrying the marker keep pre-upgrade behavior. Refs rustfs/backlog#1675 (P1-5)	2026-08-17 22:50:15 +08:00
唐小鸭	dcf3e4b9e8	fix(replication): transport and persist LWW timestamps for tag, retention, and legal hold (#6129) ... * test(replication): pin missing LWW timestamp header transport Red-light tests for the replication timestamp three-header contract: - put_object_headers_carry_replication_timestamp_headers pins that PutObjectOptions::header() must emit the x-{rustfs,minio}-source-replication-{tagging,retention,legalhold}-timestamp headers when the internal timestamps are set (currently missing). - test_put_opts_from_headers_gates_replication_timestamp_persistence_on_authorization and test_complete_multipart_opts_persist_replication_timestamps_when_authorized pin that an authorized replication PUT / multipart complete must persist the inbound timestamps into the internal metadata keys while unauthorized requests must not (currently never persisted). - fake_s3_target journals the three timestamp headers per request (ReplicationTimestampHeaders on RequestRecord) so sender-side e2e assertions can observe what a real target receives; self-test included. * fix(replication): transport and persist LWW timestamps for tag, retention, and legal hold Active-active conflict resolution for concurrent tag/retention/legal-hold edits needs the source's per-category modification times on both sides of the wire; the three AdvancedPutOptions timestamp fields were dead and the headers were neither sent nor parsed. - Emit x-{rustfs,minio}-source-replication-{tagging,retention,legalhold}- timestamp from PutObjectOptions::header(); names and RFC3339 values interoperate with MinIO (minio-go constants.go, object-api-options.go), pinned by a header_compat wire-name test. - Default the three AdvancedPutOptions timestamps to UNIX_EPOCH and skip epoch values in header(), so "never modified" is not sent as a modification made now. - Parse the headers only on authorized replication PUTs and multipart completes, expose them as Option<OffsetDateTime> on ObjectOptions, and persist them into the dual-prefix internal metadata keys so the outbound pass (replication_target_boundary) reads the source's timestamps instead of the mod_time fallback. - Record the local tagging timestamp in the PutObjectTagging and DeleteObjectTagging eval metadata, mirroring the object-lock handlers; without it the sender only ever had the mod_time fallback to offer. Receiver-side LWW comparison (keep newer stored category metadata over a stale inbound copy) is left as a TODO at the parse site. * fix(replication): load the stored tagging timestamp independently of remaining tags Review: DeleteObjectTagging persists the tagging-timestamp internal key but leaves the object tagless, and the outbound mapper only loaded the key inside the user_tags-nonempty branch — the deletion's LWW timestamp stayed at the epoch and the header was omitted, so the deletion could never win conflict resolution on the replica. The stored key is now loaded unconditionally; the mod_time fallback still applies only while tags exist (MinIO parity), and a tagless object without the key keeps the epoch default (no header). Deletion-path regression test added. * fix(storage): reserve replication transport names at metadata ingest Second review round: a client PUT of x-amz-meta-x-rustfs-source-replication-tagging-timestamp materialized the bare transport key as stored user metadata. The outbound replication header builder forwards user metadata verbatim on a server-authorized request, so the receiver would persist the attacker-chosen value as trusted internal LWW state — and for a tagless object nothing later overwrites it. The ingest namespacing guard now reserves the whole x-rustfs-source- / x-minio-source- families (the new timestamps and their siblings: source-mtime/-etag/-version-id/-replication-request), folding forged keys back under x-amz-meta-. Forged-ingress regression covers both prefixes and a sibling. * fix(replication): harden timestamp replay * fix(app): route retention helper through facade --------- Co-authored-by: overtrue <anzhengchao@gmail.com>	2026-08-16 05:56:04 +08:00
唐小鸭	2ecf6b4575	fix(replication): probe the version-identity contract in replication-check (#5881) ... * test(replication): pin the version-fidelity probe contract (red) P1-19 (rustfs/backlog#1675 B2): the supported replication contract is targets that adopt the source version id — a target that mints its own ids silently breaks every version-addressed operation that follows (version deletes, heal re-drives never match), diverging the two sides with no signal. replication-check already captures the probe PUT's response version id but never compares it. Red evidence (current main): against a FakeS3Target with assign_own_version_ids enabled, ?replication-check returns Status "OK" — the drift is invisible. test_replication_check_flags_version_minting_target expects a VersionFidelity phase that fails with the machine-readable code BucketRemoteTargetVersionMismatch, skips the later mutation phases, and still cleans up the probe via the version id the target actually assigned. Test infra: FakeS3Target gains assign_own_version_ids (models a generic S3 service; validated-but-not-mirrored source version headers) and a prefix+max-keys ListObjectVersions implementation (the probe key allocation requires it); stored_versions accessor duplicated from the P1-21 branch (identical code, resolves clean on merge). * fix(replication): probe the version-identity contract in replication-check P1-19 (rustfs/backlog#1675 B2, plan B). Replication only converges on targets that adopt the source version id: version-addressed deletes and heal re-drives address the source id, so a target that mints its own ids silently diverges — nothing surfaced this. replication-check already captured the probe PUT's response version id but never compared it. - The probe PUT now carries the source version as `?versionId=` (the exact shape live replication uses since P0-5, and the only shape MinIO consumes; the internal source-version-id header alone would let the probe pass against targets the real data path drifts on). Reuses ecstore's append_version_id_query through the api facade. - New VersionFidelity phase: the probe PUT's response version id must equal the sent source id. On mismatch the phase fails with the machine-readable extension key `"Code": "BucketRemoteTargetVersionMismatch"` (new optional Code field on phase statuses; Go decoders ignore unknown keys), the overall target fails, the later version-addressed mutation phases are skipped, and cleanup still removes the probe via the id the target actually assigned (with the existing list-based sweep as backstop when the target returns no version id at all). - Runtime half: TargetClient::put_object now returns the assigned version id (mirroring remove_object), and the replication PUT path audits it — every drifting PUT increments rustfs_replication_version_identity_drift_total and the first drift per target ARN logs a structured warning pointing at ?replication-check. The drift judgment is a pure function with an exemption-matrix test (empty / literal "null" / nil-uuid sources carry no contract). - docs/operations/replication-check.md documents the phase and the code. Red -> green: test_replication_check_flags_version_minting_target (fake target with assign_own_version_ids; on main the check reported Status "OK"). The probe's query shape is pinned by a journal assertion (revert of the query hunk alone fails it), probe-level unit tests cover the mismatch/mirror matrix including cleanup addressing the minted id, and the existing success e2e now asserts VersionFidelity OK against a RustFS target. Adversarial review (seven roles): non-blocking; noted follow-ups are the multipart runtime audit (the probe phase already pins the contract) and per-target re-warning after reconfiguration. * fix(e2e): stop the fake target self-deadlocking on version-id minting The assign_own_version_ids flag was read with a fresh `lock(&self.store)` inside two paths that already hold that guard — delete_object's marker-creation branch and create_multipart_upload — and the store mutex is not reentrant, so both hung forever (CI: the fake target's own multipart and delete-marker tests ran >1560s until the job was cancelled). Read the flag from the live guard instead. The replication e2e paths did not catch this: a version-addressed purge DELETE never mints an id, and the probe PUT reads the flag before taking the guard. * chore(test): refresh the nextest replication count invariant The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata (authority: `cargo nextest list`); refresh it to this branch's post-rebase total.	2026-08-11 03:04:05 +00:00
唐小鸭	3c31eaf06f	fix(replication): retry, persist and replay failed delete-marker purges (#5864) ... * test(replication): pin delayed delete-marker purge failure handling (red) P1-21 (rustfs/backlog#1675 B2): two failing e2e tests that pin the missing failure handling of the delayed delete-marker purge: - test_delayed_delete_marker_purge_retries_after_transient_target_failure: four scripted 503s outlast every existing channel (version-purge replication + its in-process MRF fast retries + the watcher's single attempt = 3 target DELETEs, all faulted in the recorded run); the replicated marker is stranded on the target forever. - test_delayed_delete_marker_purge_exhaustion_persists_to_mrf_and_replays_on_restart: exhausted purge intents never reach the durable MRF journal, so a restart replays nothing (recorded run: 3 faulted attempts, zero post-restart). Red-light evidence (current main): - Test A: FAILED, journal shows 3x DeleteObject fault=Status(503), no clean attempt, target marker still present after 15s. - Test B: FAILED after 468s, same 3 faulted attempts, no purge DELETE after restart, marker still present. Test infra: FakeS3Target::stored_versions() exposes per-key version state so purge tests assert target state instead of inferring it from the journal; nextest count comments 36->38 nightly / 56->58 total. * fix(replication): retry, persist and replay failed delete-marker purges P1-21 (rustfs/backlog#1675 B2). The delayed delete-marker purge was fire-and-forget: the target DELETE discarded its result (`let _ =`), a missing target client was silently skipped, and nothing recorded the intent — one transient target error stranded the replicated marker on the target forever. Separately, `replicate_delete_with_outcome` held its outcome hostage to `!requires_delayed_purge`, pinning every delete-marker MRF entry to Missed so the durable backlog retained them permanently. Changes: - `replicate_delete_marker_purge_to_targets` now reports per-target results (warn + metrics on failure, including `target_client_missing`), supports retrying only the failed targets, and treats a target-side NoSuchKey/NoSuchVersion as purge success (strict-404 targets must not retain the intent forever). - The delayed watcher (`watch_and_purge_source_delete_marker`) retries failed targets across its 5x1s watch window; on exhaustion it persists the purge intent to the durable MRF journal via the new `ReplicationPoolTrait::persist_mrf_entry` (journal-only on purpose: live re-dispatch would loop unboundedly against a down target). Intent entries are shaped as marker-creation deletes so replay funnels into the stale- marker branch. - The stale-marker branch (source marker already gone) now purges the targets instead of silently returning success — closing a latent leak — and reports the purge result as the replay outcome. Heal callers retry for the full window (the startup MRF processor runs before target clients initialize); live callers attempt once and fall back to a fresh durable intent, so a down target cannot pin a replication worker. - The outcome formula (extracted as `replicate_delete_outcome` and pinned by a unit test) no longer includes the delayed purge, so successfully replayed delete-marker entries are acknowledged instead of retained forever. Verification: red -> green e2e pair (transient-failure retry; exhaustion -> durable MRF -> restart replay -> second-restart zero-replay ack) plus unit tests; `make pre-commit`, logging guardrails, clippy (ecstore + e2e_test) all clean; full ecstore lib suite 3729 passed (3 pre-existing local-DNS kubernetes endpoint failures reproduce without this change). Adversarial validation (7 roles): no blocking findings after adding the outcome-formula guard test. Known residuals recorded in the PR: watcher shutdown window (intent not yet persisted), rolling-downgrade replay acks without purging (equals pre-fix behavior), and replay falling back to the source version id on targets that mint their own version ids (P1-19). * chore(test): refresh the nextest replication count invariant The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata (authority: `cargo nextest list`); refresh it to this branch's post-rebase total. * fix(replication): purge the marker version the target actually assigned Review follow-up (#5864), two real defects: - The delayed purge watcher was spawned with the pre-merge `dobj`, so the per-target marker version ids this round recorded were invisible to it. Against a target that mints its own ids the purge fell back to a source-derived id, the target answered the versioned DELETE with an idempotent 204, and that "success" cleared the retry set while the real marker stayed behind. The watcher now receives the merged replication state (`drs`), which folds this round's target-assigned ids in. - A target whose recorded version metadata is inconsistent was skipped without entering `failed_arns`, so an empty result made both the watcher and the MRF replay treat a purge that issued no DELETE as successful and drop the intent. The refusal is now a per-target failure (own metric label): the leak stays visible and the intent is retained instead of being acknowledged. The version decision also moved ahead of the client lookup, so the refusal is decided from metadata alone. Tests: a new e2e drives a fake target with `assign_own_version_ids`, which ignores the forwarded source-version header for both objects and delete markers, and asserts the replicated marker is really gone; a unit test pins the corrupt-metadata refusal as a failed outcome without any target client registered. The detached-watcher shutdown window is documented at the watcher as a known non-durable window with the write-ahead follow-up spelled out.	2026-08-10 22:16:21 +08:00
cxymds	1be636b914	fix(replication): make resync recovery resilient (#5883) ... Co-authored-by: houseme <housemecn@gmail.com>	2026-08-09 19:42:06 +08:00
唐小鸭	ead419451a	fix(replication): send source versionId as query param to remote targets (#5752) ... * test(replication): assert remote PUT and multipart initiate carry versionId query * fix(replication): send source versionId as query param to remote targets	2026-08-06 08:27:23 +08:00
houseme	f7c1b13c0f	refactor(deps): replace md5 crate with md-5 (#5432) ... Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-29 11:29:37 +00:00
Zhengchao An	7cc92ac93c	test(replication): add programmable fake S3 target (#4929)	2026-07-17 01:20:09 +08:00