mirror of https://github.com/rustfs/rustfs.git synced 2026-08-16 18:08:21 +00:00

Files

T

唐小鸭 2ecf6b4575 fix(replication): probe the version-identity contract in replication-check (#5881 )

* test(replication): pin the version-fidelity probe contract (red)

P1-19 (rustfs/backlog#1675 B2): the supported replication contract is
targets that adopt the source version id — a target that mints its own ids
silently breaks every version-addressed operation that follows (version
deletes, heal re-drives never match), diverging the two sides with no
signal. replication-check already captures the probe PUT's response version
id but never compares it.

Red evidence (current main): against a FakeS3Target with
assign_own_version_ids enabled, ?replication-check returns Status "OK" —
the drift is invisible.

test_replication_check_flags_version_minting_target expects a
VersionFidelity phase that fails with the machine-readable code
BucketRemoteTargetVersionMismatch, skips the later mutation phases, and
still cleans up the probe via the version id the target actually assigned.

Test infra: FakeS3Target gains assign_own_version_ids (models a generic S3
service; validated-but-not-mirrored source version headers) and a
prefix+max-keys ListObjectVersions implementation (the probe key allocation
requires it); stored_versions accessor duplicated from the P1-21 branch
(identical code, resolves clean on merge).

* fix(replication): probe the version-identity contract in replication-check

P1-19 (rustfs/backlog#1675 B2, plan B). Replication only converges on
targets that adopt the source version id: version-addressed deletes and
heal re-drives address the source id, so a target that mints its own ids
silently diverges — nothing surfaced this. replication-check already
captured the probe PUT's response version id but never compared it.

- The probe PUT now carries the source version as `?versionId=` (the exact
  shape live replication uses since P0-5, and the only shape MinIO
  consumes; the internal source-version-id header alone would let the
  probe pass against targets the real data path drifts on). Reuses
  ecstore's append_version_id_query through the api facade.
- New VersionFidelity phase: the probe PUT's response version id must
  equal the sent source id. On mismatch the phase fails with the
  machine-readable extension key `"Code": "BucketRemoteTargetVersionMismatch"`
  (new optional Code field on phase statuses; Go decoders ignore unknown
  keys), the overall target fails, the later version-addressed mutation
  phases are skipped, and cleanup still removes the probe via the id the
  target actually assigned (with the existing list-based sweep as backstop
  when the target returns no version id at all).
- Runtime half: TargetClient::put_object now returns the assigned version
  id (mirroring remove_object), and the replication PUT path audits it —
  every drifting PUT increments
  rustfs_replication_version_identity_drift_total and the first drift per
  target ARN logs a structured warning pointing at ?replication-check.
  The drift judgment is a pure function with an exemption-matrix test
  (empty / literal "null" / nil-uuid sources carry no contract).
- docs/operations/replication-check.md documents the phase and the code.

Red -> green: test_replication_check_flags_version_minting_target (fake
target with assign_own_version_ids; on main the check reported Status
"OK"). The probe's query shape is pinned by a journal assertion (revert
of the query hunk alone fails it), probe-level unit tests cover the
mismatch/mirror matrix including cleanup addressing the minted id, and
the existing success e2e now asserts VersionFidelity OK against a RustFS
target. Adversarial review (seven roles): non-blocking; noted follow-ups
are the multipart runtime audit (the probe phase already pins the
contract) and per-target re-warning after reconfiguration.

* fix(e2e): stop the fake target self-deadlocking on version-id minting

The assign_own_version_ids flag was read with a fresh `lock(&self.store)`
inside two paths that already hold that guard — delete_object's
marker-creation branch and create_multipart_upload — and the store mutex
is not reentrant, so both hung forever (CI: the fake target's own
multipart and delete-marker tests ran >1560s until the job was
cancelled). Read the flag from the live guard instead.

The replication e2e paths did not catch this: a version-addressed purge
DELETE never mints an id, and the probe PUT reads the flag before taking
the guard.

* chore(test): refresh the nextest replication count invariant

The e2e-smoke/e2e-repl-nightly split comment is descriptive metadata
(authority: `cargo nextest list`); refresh it to this branch's
post-rebase total.

2026-08-11 03:04:05 +00:00

datatypes.rs

refactor(replication): isolate ecstore boundary imports (#4247 )

2026-07-03 23:18:27 +08:00

mod.rs

fix(replication): madmin reset/diff wire compat and config validation (#5799 )

2026-08-07 22:30:12 +08:00

README.md

fix(replication): madmin reset/diff wire compat and config validation (#5799 )

2026-08-07 22:30:12 +08:00

replication_bandwidth_boundary.rs

refactor(ecstore): hide replication runtime types (#4131 )

2026-07-01 15:39:02 +08:00

replication_config_boundary.rs

fix(replication): madmin reset/diff wire compat and config validation (#5799 )

2026-08-07 22:30:12 +08:00

replication_config_store.rs

fix(replication): harden resync and MRF recovery (#5694 )

2026-08-04 13:40:50 +00:00

replication_error_boundary.rs

refactor(ecstore): route replication errors (#4127 )

2026-07-01 08:13:57 +08:00

replication_event_sink.rs

refactor(ecstore): route replication event host (#4126 )

2026-07-01 06:00:50 +08:00

replication_filemeta_boundary.rs

fix(replication): madmin reset/diff wire compat and config validation (#5799 )

2026-08-07 22:30:12 +08:00

replication_lifecycle_bridge.rs

refactor(replication): own delete DTO contracts (#4253 )

2026-07-04 04:00:27 +08:00

replication_lock_boundary.rs

refactor(ecstore): name replication boundary contracts (#4134 )

2026-07-01 17:55:12 +08:00

replication_logging.rs

refactor(replication): move object config bridge (#4188 )

2026-07-02 16:32:02 +08:00

replication_metadata_boundary.rs

fix(replication): harden resync and MRF recovery (#5694 )

2026-08-04 13:40:50 +00:00

replication_migration_bridge.rs

refactor(replication): isolate resync boundary adapters (#4215 )

2026-07-03 08:59:42 +08:00

replication_msgp_boundary.rs

refactor(ecstore): name replication boundary contracts (#4134 )

2026-07-01 17:55:12 +08:00

replication_object_bridge.rs

fix(replication): persist force-delete handoff state (#5641 )

2026-08-03 01:44:28 +00:00

replication_object_config.rs

fix(replication): restrict metadata replication targets (#5696 )

2026-08-04 22:46:42 +08:00

replication_object_decision_boundary.rs

refactor(replication): route app storage through ecstore (#4251 )

2026-07-04 02:00:28 +08:00

replication_pool.rs

fix(replication): retry, persist and replay failed delete-marker purges (#5864 )

2026-08-10 22:16:21 +08:00

replication_queue_boundary.rs

feat(replication): bound DeleteObjects queue admission (#5637 )

2026-08-02 21:06:54 +08:00

replication_resync_boundary.rs

fix(replication): harden resync and MRF recovery (#5694 )

2026-08-04 13:40:50 +00:00

replication_resyncer.rs

fix(replication): probe the version-identity contract in replication-check (#5881 )

2026-08-11 03:04:05 +00:00

replication_scanner_bridge.rs

refactor(replication): move object config bridge (#4188 )

2026-07-02 16:32:02 +08:00

replication_state.rs

Expose target-scoped durable MRF backlog metrics (#5584 )

2026-08-01 17:31:50 +00:00

replication_stats_boundary.rs

feat(obs): add bounded metrics dimensions (#5645 )

2026-08-03 09:03:34 +08:00

replication_storage_boundary.rs

fix(replication): fence force-delete journal updates (#5661 )

2026-08-03 22:00:49 +08:00

replication_tagging_boundary.rs

refactor(replication): move config contracts into crate (#4162 )

2026-07-02 10:17:48 +08:00

replication_target_boundary.rs

feat(replication): replicate managed-SSE objects via target re-encryption (#5885 )

2026-08-09 10:21:35 +00:00

replication_target_config_bridge.rs

refactor(replication): isolate ecstore boundary imports (#4247 )

2026-07-03 23:18:27 +08:00

replication_timing.rs

feat(ecstore): make replication timing intervals env-overridable for tests (#4680 )

2026-07-10 21:57:43 +08:00

replication_versioning_boundary.rs

fix(replication): harden live delete admission (#5599 )

2026-08-02 12:52:11 +08:00

runtime_boundary.rs

refactor(ecstore): hide replication runtime types (#4131 )

2026-07-01 15:39:02 +08:00

README.md

ECStore Replication Split Inventory

This directory still owns the ECStore replication workers. The resync status contracts and wire format now live in rustfs-replication, while worker runtime code still depends on ECStore object IO, bucket target clients, bucket metadata systems, runtime state, notification events, and lifecycle/heal scheduling paths.

Current Modules

Module	Current role	Split blocker
`config.rs`	Replication config helpers, rule matching, and tag filtering.	Uses replication-local filemeta/tagging boundaries and S3 DTOs directly.
`datatypes.rs`	ECStore compatibility re-export for resync status enums.	Re-exports `rustfs-replication` contracts while downstream facade consumers migrate.
`replication_object_decision_boundary.rs`	Object replication option DTOs, resync target projection, delete replication decisions, and multipart planning helpers.	Keeps ECStore runtime modules from importing object decision contracts directly from `rustfs-replication`.
`replication_pool.rs`	Replication queue, worker pool, MRF persistence, bucket stats, and delete/object scheduling.	Depends on bucket target sys, bucket metadata sys, metadata paths, queue contracts through the queue boundary, file metadata replication contracts through local boundaries, config storage, storage contracts through the replication storage boundary, runtime sources, and notification state.
`replication_queue_boundary.rs`	Queue/admission DTOs, heal queue DTOs, worker sizing, and backpressure helpers.	Keeps ECStore runtime modules from importing queue/backpressure contracts directly from `rustfs-replication`.
`replication_resync_boundary.rs`	Resync DTOs, status classifiers, persisted resync/MRF codec wrappers, and ECStore error mapping.	Keeps ECStore runtime modules from importing resync contract helpers directly from `rustfs-replication`.
`replication_resyncer.rs`	Object replication, delete replication, resync execution, target calls, and multipart target upload paths.	Depends on target calls and target config types through the replication target boundary, metadata paths and metadata systems through the replication metadata boundary, file metadata replication contracts through the filemeta boundary, object decisions and multipart planning through the object decision boundary, resync contracts through the resync boundary, queue DTOs through the queue boundary, error contracts through the error boundary, versioning systems, storage contracts through the replication storage boundary, config-derived storage class labels through the config store, runtime sources, notification events and local event host selection through the event sink, bandwidth reader wrapping, and SetDisks lock timing.
`replication_state.rs`	Replication queue/stat state and worker accounting.	Reads stats DTOs through the stats boundary, runtime sources, file metadata replication contracts, error contracts, and bucket monitor handles through local boundaries, and owns shared replication pool/stat state.
`replication_stats_boundary.rs`	Bucket replication stats DTOs, queue/proxy metric caches, and worker metric snapshots.	Keeps ECStore runtime modules from importing stats contracts directly from `rustfs-replication`.
`replication_lifecycle_bridge.rs`	Lifecycle-originated delete replication admission and version-purge state construction.	Depends on replication config/rule matching, delete-replication decisions, and replication delete scheduling through a local contract type.
`replication_migration_bridge.rs`	Bucket migration access to persisted replication resync codec helpers.	Keeps migration normalization behind a bridge instead of re-exporting resyncer codec helpers.
`replication_object_bridge.rs`	App and SetDisks object replication decisions plus object/delete scheduling.	Keeps object write/delete replication call sites behind a bridge instead of exporting low-level resyncer and pool helpers.
`replication_scanner_bridge.rs`	Scanner-originated replication heal admission.	Keeps scanner-facing heal queueing behind a local contract type instead of exporting the internal queue function directly.
`replication_target_config_bridge.rs`	Bucket target removal checks against replication target rules.	Keeps bucket target sys from importing replication config helper types directly.
`rule.rs`	Rule evaluation helpers for object replication options.	Depends on ECStore replication object option types.
`mod.rs`	Explicit compatibility re-export facade for the current ECStore owner.	Wildcard re-exports are guarded so internal helpers do not leak back into the public facade.

Required Contracts

Contract	Responsibility	Current dependency to remove
`ReplicationObjectIO`	Object read/write primitives used by config, MRF, resync status, and multipart replication paths.	ECStore object API reader/writer types and storage-api object IO contracts are concentrated in `replication_storage_boundary.rs`.
`ReplicationStorage`	Object read/write/delete, object walk, metadata update, and target object IO.	ECStore object API, storage-api contracts, and read option types are concentrated in `replication_storage_boundary.rs`.
`ReplicationMetadataStore`	Replication config, MRF/resync state, target reset headers, and status persistence.	Metadata sys access and replication metadata path constants are exposed through the contract type in `replication_metadata_boundary.rs`; versioning sys and config storage imports remain separate contracts.
`EcstoreReplicationBoundaryImports`	ECStore-side imports from `rustfs-replication`.	Direct `rustfs-replication` imports under `crates/ecstore/src/bucket/replication` stay in `*_boundary.rs` modules, including config and resync facade re-exports.
`RuntimeReplicationFacadeConsumers`	Runtime owner consumers of replication DTOs and status types.	Scanner, admin, and storage owner facades import replication DTOs/status types through `rustfs-ecstore`; app storage keeps the remaining direct object/delete helper calls behind its local storage API boundary.
`ReplicationResyncContracts`	Resync options, target status, bucket status, status classifiers, and persisted resync/MRF status wire format.	Owned by `crates/replication`; ECStore imports them through `replication_resync_boundary.rs`, which maps crate errors to ECStore errors.
`ReplicationCrateFileMetaIndependence`	Replication status, decision, MRF, resync, and target-reset wire contracts owned by `rustfs-replication`.	`crates/replication/src/filemeta.rs` owns these contracts; `rustfs-replication` must not import or depend on `rustfs-filemeta`.
`ReplicationConfigStore`	Replication config persistence and config-derived labels used by target options.	Config read/save helpers and storage class labels are exposed through the contract type in `replication_config_store.rs`.
`ReplicationFileMeta`	ECStore compatibility conversions for filemeta replication state/status.	`rustfs_filemeta` to `rustfs_replication` conversions are concentrated in `replication_filemeta_boundary.rs`; `FileInfo` remains in the storage boundary for storage trait bindings and walk options.
`StorageApiReplicationContracts`	Storage-api delete DTO replication state/status helpers.	Storage-api owner DTOs keep their local replication boundary; ECStore converts them in `replication_storage_boundary.rs` before queueing replication work.
`ReplicationCrateStorageApiIndependence`	Delete work DTOs consumed by `rustfs-replication`.	`crates/replication/src/storage_api.rs` owns these DTOs; `rustfs-replication` must not import or depend on `rustfs-storage-api`.
`ReplicationObjectDecisionContracts`	Object replication options, delete replication decisions, resync target projection, multipart planning, and delete-marker retry classifiers.	Owned by `crates/replication`; ECStore imports them through `replication_object_decision_boundary.rs`.
`ReplicationQueueContracts`	Queue admission, heal queue results/actions, worker operations, worker sizing, and backpressure decisions.	Owned by `crates/replication`; ECStore imports them through `replication_queue_boundary.rs`.
`ReplicationStatsContracts`	Bucket stats, replication target stats, queue/proxy metrics, and worker metric snapshots.	Owned by `crates/replication`; ECStore imports them through `replication_stats_boundary.rs`.
`ReplicationErrorBoundary`	ECStore error/result contracts and replication-specific error classifiers.	`crate::error` imports are concentrated in `replication_error_boundary.rs`.
`ReplicationTargetStore`	Bucket target listing, target client lookup, target offline checks, target config types, target operation option types, and target HeadObject comparison adapters.	Bucket target sys access, `BucketTargets`, target operation types, and HeadObject-to-replication DTO adapters are exposed through the contract type in `replication_target_boundary.rs`.
`ReplicationRuntime`	Worker pool, queue sizing, stats, bucket monitor, local node identity, cancellation, and admission state.	Direct runtime source/global access and shared replication pool/stat state; ECStore object store and bucket monitor implementation types stay behind local storage/bandwidth boundaries.
`ReplicationBandwidthLimiter`	Target reader wrapping for replication bandwidth accounting and throttling.	Direct bucket bandwidth reader imports from resyncer paths.
`ReplicationEventSink`	Notification and audit events for skipped, failed, pending, and completed replication operations.	Event notification service calls and local event host selection are concentrated in `replication_event_sink.rs`.
`ReplicationVersioningStore`	Versioning state checks for object and delete replication decisions.	Bucket versioning sys access is exposed through the contract type in `replication_versioning_boundary.rs`.
`ReplicationLockTiming`	Namespace lock timing for replication resync, object replication, and delete replication locks.	SetDisks lock timeout access is exposed through the contract type in `replication_lock_boundary.rs`.
`ReplicationMsgpCodec`	MessagePack time encode/decode and unknown value skipping for persisted resync/MRF state.	Bucket MessagePack helpers are exposed through the contract type in `replication_msgp_boundary.rs`.
`ReplicationTagFilter`	Decode object tag strings for rule and metadata replication decisions.	Bucket tagging helper access is exposed through the contract type in `replication_tagging_boundary.rs`.
`ReplicationLifecycleBridge`	Lifecycle-originated delete and version-purge scheduling.	Lifecycle delete paths call the bridge contract in `replication_lifecycle_bridge.rs` instead of constructing replication delete work directly.
`ReplicationMigrationBridge`	Persisted resync status decode/encode access for bucket metadata migration.	Bucket migration calls the bridge contract in `replication_migration_bridge.rs` instead of importing internal resyncer codec helpers.
`ReplicationObjectBridge`	Object write/delete replication decision and scheduling entry point for app storage and SetDisks paths.	App and SetDisks object paths call the bridge contract in `replication_object_bridge.rs` instead of importing internal resyncer/pool helpers.
`ReplicationScannerBridge`	Scanner-originated replication heal scheduling.	Scanner heal paths call the bridge contract in `replication_scanner_bridge.rs` instead of importing the internal queue function directly.
`ReplicationTargetConfigBridge`	Bucket target removal checks against replication target rules.	Bucket target sys calls the bridge contract in `replication_target_config_bridge.rs` instead of importing replication config helper types directly.

Migration Rules

Do not move bucket/replication into a new crate while workers import bucket target sys, metadata sys, runtime sources, bandwidth reader, notification services, or SetDisks lock timing directly.
Keep existing queue behavior, MRF persistence, resync state, target client semantics, notification/audit events, and scanner/heal classifications unchanged during inventory and contract PRs.
Keep the current ReplicationStorage trait as the starting point. Split it only after call sites prove a narrower object read/write/delete/walk shape.
Preserve rustfs_ecstore::api::bucket::replication compatibility until lifecycle, scanner, OBS, heal, and tests compile through replacement paths.
Keep imports between modules in this directory relative to the local replication module, not crate::bucket::replication::* self paths.
Keep runtime source access from importing ECStore object store or bucket monitor implementation types directly; use local boundary-owned aliases.
Move at most one owner boundary per code-bearing PR and verify it with focused replication tests before broad gates. Non-behavioral contract-shape cleanup may batch already-established boundary wrappers when the owner and call semantics do not change.
Keep the compatibility facade in mod.rs as an explicit symbol list. Do not reintroduce wildcard re-exports for replication implementation modules.
Keep object write/delete replication helpers behind ReplicationObjectBridge; do not export internal resyncer or pool scheduling helpers through the compatibility facade.
Keep ECStore owner modules outside bucket/replication behind bridge contracts when they need replication codec or config helper behavior.
Keep storage-api replication status/state helpers behind crates/storage-api/src/replication.rs; ECStore converts owner DTOs at the replication storage boundary.
Keep rustfs-replication independent from rustfs-filemeta; ECStore compatibility conversions live in replication_filemeta_boundary.rs.
Keep rustfs-replication independent from rustfs-storage-api; ECStore compatibility conversions live in replication_storage_boundary.rs.
Keep direct rustfs-replication imports inside ECStore replication concentrated in *_boundary.rs modules.
Keep scanner, admin, and storage-owner replication status/DTO consumers behind the ECStore replication facade; only rustfs/src/app/storage_api.rs may retain direct object/delete replication helper calls.

Completion Criteria

The split is complete when the "Current dependency to remove" column in the Required Contracts table above is empty: every row is either deleted because the dependency is gone, or reduced to "none". No other signal — file count, boundary count, line count — measures completion.

Target end state:

replication_pool.rs, replication_resyncer.rs, and replication_state.rs move into crates/replication behind the contracts above;
the *_boundary.rs and *_bridge.rs micro-files dissolve naturally as the code they fence moves across the crate boundary. They are the mechanical seams of the migration ratchet — the architecture guard scripts anchor on their file names — so batch-merging them beforehand is explicitly rejected: it forces synchronized guard-script/mod/import churn with zero functional gain;
the only module that can retire early is datatypes.rs: delete it once its facade consumers import the resync status enums through rustfs-replication directly.

Milestones

Milestone	Scope	Status
M0	Record the completion criteria and end state (this section).	Done
M1	Contract extraction: resync/queue/stats/object-decision/filemeta/storage wire contracts owned by `crates/replication`; ECStore imports concentrated in `*_boundary.rs`; event sink and runtime access behind local contracts.	Done — see Required Contracts
M2	Move resyncer pure decision logic (no IO) into `crates/replication`.	Pending; sequence after splitting the oversized resyncer/pool functions (`resync_bucket`, `replicate_all`, `start_mrf_processor`) so moves stay mechanical
M3	Move the worker runtime (`replication_pool.rs`, the IO paths of `replication_resyncer.rs`, `replication_state.rs`) once the contract traits are stable. Highest-risk step of the whole plan; do it last.	Pending
M4	Retire the boundary modules together with their guard-script entries; delete `datatypes.rs`.	Pending

The original first code-bearing step (narrow ReplicationEventSink / ReplicationRuntime contracts) has landed — replication_event_sink.rs exists and runtime access goes through local boundary aliases — so new work starts from M2.

Current compatibility guard: crates/ecstore/tests/replication_facade_compat_test.rs keeps the ECStore replication facade types covered while architecture rules keep direct imports behind local storage_api boundaries.