mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-29 08:27:06 +00:00
abdullahnah92
c3aac2279f
* fix(site-replication): keep reverse direction after config broadcast `site-repl-*` rules encode the sender's outbound direction: their destination ARN names the receiver. `apply_bucket_meta_item` wrote an incoming rule set verbatim over the receiver's, leaving the receiver with a rule whose ARN is its own deployment ID. `reconcile_site_replication_bucket_targets` skips the local peer, so no bucket target can back that ARN and every object was dropped; the follow-up call reconciled targets only, so nothing rebuilt the lost reverse rule. Replication went one-directional after any PutBucketReplication broadcast — the console's Save button, `mc replicate import`, a metadata import, or `/site-replication/repair`. Only operator-authored rules now travel between sites; each site owns its `site-repl-*` rules and rebuilds them from the current peer set. Four defects kept that invisible or unrecoverable: - `update_all_targets` discarded target-client build errors silently, and `replicate_object` logged the resulting missing-target drop at debug while every other failure there logs at error. Both now report. - `site_replication_rule_complete` never checked that a rule's destination named a remote site, so two sites holding identical configs — the post-clobber state — passed as in sync. - `update_service_account` cannot rewrite `parent_user`, and IAM records encrypted with a previous root secret decode as "no such account". Startup now reconciles the account, reseeding from the secret every site-replication bucket target already stores, and refuses the delete-then-create sequence when the parent cannot back an account. Bucket rules are reconciled too, so an already-broken site heals on upgrade. - A joined site never verified it could reach the initiator, whose endpoint is derived from the Host header of the admin request that created the topology. The join now probes each peer and reports through `initial_sync_error_message`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(site-replication): report unreachable targets and reconcile on a timer Rule-shape checking cannot see an unreachable peer. A `site-repl-*` rule can be perfectly formed while the endpoint recorded for its peer is one this site cannot reach: `update_all_targets` then builds no client and `replicate_object` drops every object against that ARN, yet the rule set still reads as correct and the bucket reports in sync. Each site now reports whether all of its `site-repl-*` rules resolve to a live target (`SRBucketInfo.replicationTargetsOnline`, read from the already-resolved client map so the status path stays cheap), and the status aggregation treats an offline report as a mismatch. The field is additive and optional: peers that omit it are "unknown", never a fault, so a mixed-version topology does not flip every bucket to out of sync. The reconcilers also run on a 10-minute timer instead of at startup only, so drift is repaired without waiting for a restart. Both are no-ops when the wiring already matches — the bucket pass compares serialized targets and the rule set before writing. The tick takes the site-replication lifecycle lock with a non-blocking try_acquire and skips the round when an add/remove/endpoint-refresh holds it: those run in phases, and rebuilding rules between two of them would resurrect what the operation just tore down. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(site-replication): invert reconcile dependency to satisfy layers `startup_services.rs` sits in the infra layer and was calling the reconcilers in `admin::handlers::site_replication`, which is interface — a reverse dependency that check_layer_dependencies.sh rejects. Moving the reconcilers down is not viable in this change: they rest on the site-replication state core (`SiteReplicationState` alone has 107 in-file uses, `load_site_replication_state` 38, the state lock 33), so relocating it would move ~2000 lines and ~200 call sites through a bug-fix PR. Invert the direction instead. A new infra module owns the contract and the schedule; the admin layer registers its reconciler from `register_site_replication_route`, which runs while the admin router is built — `init_startup_http_servers` awaits that before `init_startup_runtime_services` reconciles, so the hook is always installed in time. No logic moves and no baseline entry is added: the dependency genuinely reverses. The lifecycle guard now wraps both reconcilers in one round rather than each separately, closing the window between them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(site-replication): harden reconcile per review feedback Addresses the automated review on #5292. Security: secret recovery from bucket targets accepted any target carrying the `site-replicator-0` access key. Bucket targets are writable by anyone holding `admin:SetBucketTarget`, so such a principal could plant a secret and have reconciliation recreate the broadly privileged replication account with it. A target must now name a peer in the persisted state and point at that peer's recorded endpoint, disagreeing targets abort the recovery, and only missing/unreadable-account errors may trigger it at all — a transient store failure no longer rewrites a live account. Durability: the repair no longer deletes before creating. A readable account is rebound in place through a new `parent_user` field on `UpdateServiceAccountOpts`, gated to `site-replicator-0` under `allow_site_replicator_account` exactly as the account itself is. The parent also lives in the session-token claims, and `prepare_service_account_auth` denies the account when the two disagree, so both move together. Availability: the reconcile scheduler no longer requires an inline IAM bootstrap. Deferred IAM recovers in the background with no callback into the scheduler, which left a recovered node with self-pointing rules until the next restart. It now starts unconditionally and returns early while IAM or the object store are unavailable. Its first pass runs inside the task, so walking every bucket no longer delays startup. Correctness: an endpoint refresh commits bucket targets and peer state in separate steps without holding the lifecycle lock, so a tick landing between them rewrote targets from the stale endpoint; the reconciler now also skips while any pending marker is set. Rule repair preserves an operator-authored `role` and clears only sender-owned site-replication ARNs, matching the merge path. Hot path: the per-object missing-target message returns to debug. The condition is reported once per bucket per reconcile pass instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: houseme <housemecn@gmail.com>
ECStore Replication Split Inventory
This directory still owns the ECStore replication workers. The resync status
contracts and wire format now live in rustfs-replication, while worker runtime
code still depends on ECStore object IO, bucket target clients, bucket metadata
systems, runtime state, notification events, and lifecycle/heal scheduling
paths.
Current Modules
| Module | Current role | Split blocker |
|---|---|---|
config.rs |
Replication config helpers, rule matching, and tag filtering. | Uses replication-local filemeta/tagging boundaries and S3 DTOs directly. |
datatypes.rs |
ECStore compatibility re-export for resync status enums. | Re-exports rustfs-replication contracts while downstream facade consumers migrate. |
replication_object_decision_boundary.rs |
Object replication option DTOs, resync target projection, delete replication decisions, and multipart planning helpers. | Keeps ECStore runtime modules from importing object decision contracts directly from rustfs-replication. |
replication_pool.rs |
Replication queue, worker pool, MRF persistence, bucket stats, and delete/object scheduling. | Depends on bucket target sys, bucket metadata sys, metadata paths, queue contracts through the queue boundary, file metadata replication contracts through local boundaries, config storage, storage contracts through the replication storage boundary, runtime sources, and notification state. |
replication_queue_boundary.rs |
Queue/admission DTOs, heal queue DTOs, worker sizing, and backpressure helpers. | Keeps ECStore runtime modules from importing queue/backpressure contracts directly from rustfs-replication. |
replication_resync_boundary.rs |
Resync DTOs, status classifiers, persisted resync/MRF codec wrappers, and ECStore error mapping. | Keeps ECStore runtime modules from importing resync contract helpers directly from rustfs-replication. |
replication_resyncer.rs |
Object replication, delete replication, resync execution, target calls, and multipart target upload paths. | Depends on target calls and target config types through the replication target boundary, metadata paths and metadata systems through the replication metadata boundary, file metadata replication contracts through the filemeta boundary, object decisions and multipart planning through the object decision boundary, resync contracts through the resync boundary, queue DTOs through the queue boundary, error contracts through the error boundary, versioning systems, storage contracts through the replication storage boundary, config-derived storage class labels through the config store, runtime sources, notification events and local event host selection through the event sink, bandwidth reader wrapping, and SetDisks lock timing. |
replication_state.rs |
Replication queue/stat state and worker accounting. | Reads stats DTOs through the stats boundary, runtime sources, file metadata replication contracts, error contracts, and bucket monitor handles through local boundaries, and owns shared replication pool/stat state. |
replication_stats_boundary.rs |
Bucket replication stats DTOs, queue/proxy metric caches, and worker metric snapshots. | Keeps ECStore runtime modules from importing stats contracts directly from rustfs-replication. |
replication_lifecycle_bridge.rs |
Lifecycle-originated delete replication admission and version-purge state construction. | Depends on replication config/rule matching, delete-replication decisions, and replication delete scheduling through a local contract type. |
replication_migration_bridge.rs |
Bucket migration access to persisted replication resync codec helpers. | Keeps migration normalization behind a bridge instead of re-exporting resyncer codec helpers. |
replication_object_bridge.rs |
App and SetDisks object replication decisions plus object/delete scheduling. | Keeps object write/delete replication call sites behind a bridge instead of exporting low-level resyncer and pool helpers. |
replication_scanner_bridge.rs |
Scanner-originated replication heal admission. | Keeps scanner-facing heal queueing behind a local contract type instead of exporting the internal queue function directly. |
replication_target_config_bridge.rs |
Bucket target removal checks against replication target rules. | Keeps bucket target sys from importing replication config helper types directly. |
rule.rs |
Rule evaluation helpers for object replication options. | Depends on ECStore replication object option types. |
mod.rs |
Explicit compatibility re-export facade for the current ECStore owner. | Wildcard re-exports are guarded so internal helpers do not leak back into the public facade. |
Required Contracts
| Contract | Responsibility | Current dependency to remove |
|---|---|---|
ReplicationObjectIO |
Object read/write primitives used by config, MRF, resync status, and multipart replication paths. | ECStore object API reader/writer types and storage-api object IO contracts are concentrated in replication_storage_boundary.rs. |
ReplicationStorage |
Object read/write/delete, object walk, metadata update, and target object IO. | ECStore object API, storage-api contracts, and read option types are concentrated in replication_storage_boundary.rs. |
ReplicationMetadataStore |
Replication config, MRF/resync state, target reset headers, and status persistence. | Metadata sys access and replication metadata path constants are exposed through the contract type in replication_metadata_boundary.rs; versioning sys and config storage imports remain separate contracts. |
EcstoreReplicationBoundaryImports |
ECStore-side imports from rustfs-replication. |
Direct rustfs-replication imports under crates/ecstore/src/bucket/replication stay in *_boundary.rs modules, including config and resync facade re-exports. |
RuntimeReplicationFacadeConsumers |
Runtime owner consumers of replication DTOs and status types. | Scanner, admin, and storage owner facades import replication DTOs/status types through rustfs-ecstore; app storage keeps the remaining direct object/delete helper calls behind its local storage API boundary. |
ReplicationResyncContracts |
Resync options, target status, bucket status, status classifiers, and persisted resync/MRF status wire format. | Owned by crates/replication; ECStore imports them through replication_resync_boundary.rs, which maps crate errors to ECStore errors. |
ReplicationCrateFileMetaIndependence |
Replication status, decision, MRF, resync, and target-reset wire contracts owned by rustfs-replication. |
crates/replication/src/filemeta.rs owns these contracts; rustfs-replication must not import or depend on rustfs-filemeta. |
ReplicationConfigStore |
Replication config persistence and config-derived labels used by target options. | Config read/save helpers and storage class labels are exposed through the contract type in replication_config_store.rs. |
ReplicationFileMeta |
ECStore compatibility conversions for filemeta replication state/status. | rustfs_filemeta to rustfs_replication conversions are concentrated in replication_filemeta_boundary.rs; FileInfo remains in the storage boundary for storage trait bindings and walk options. |
StorageApiReplicationContracts |
Storage-api delete DTO replication state/status helpers. | Storage-api owner DTOs keep their local replication boundary; ECStore converts them in replication_storage_boundary.rs before queueing replication work. |
ReplicationCrateStorageApiIndependence |
Delete work DTOs consumed by rustfs-replication. |
crates/replication/src/storage_api.rs owns these DTOs; rustfs-replication must not import or depend on rustfs-storage-api. |
ReplicationObjectDecisionContracts |
Object replication options, delete replication decisions, resync target projection, multipart planning, and delete-marker retry classifiers. | Owned by crates/replication; ECStore imports them through replication_object_decision_boundary.rs. |
ReplicationQueueContracts |
Queue admission, heal queue results/actions, worker operations, worker sizing, and backpressure decisions. | Owned by crates/replication; ECStore imports them through replication_queue_boundary.rs. |
ReplicationStatsContracts |
Bucket stats, replication target stats, queue/proxy metrics, and worker metric snapshots. | Owned by crates/replication; ECStore imports them through replication_stats_boundary.rs. |
ReplicationErrorBoundary |
ECStore error/result contracts and replication-specific error classifiers. | crate::error imports are concentrated in replication_error_boundary.rs. |
ReplicationTargetStore |
Bucket target listing, target client lookup, target offline checks, target config types, target operation option types, and target HeadObject comparison adapters. | Bucket target sys access, BucketTargets, target operation types, and HeadObject-to-replication DTO adapters are exposed through the contract type in replication_target_boundary.rs. |
ReplicationRuntime |
Worker pool, queue sizing, stats, bucket monitor, local node identity, cancellation, and admission state. | Direct runtime source/global access and shared replication pool/stat state; ECStore object store and bucket monitor implementation types stay behind local storage/bandwidth boundaries. |
ReplicationBandwidthLimiter |
Target reader wrapping for replication bandwidth accounting and throttling. | Direct bucket bandwidth reader imports from resyncer paths. |
ReplicationEventSink |
Notification and audit events for skipped, failed, pending, and completed replication operations. | Event notification service calls and local event host selection are concentrated in replication_event_sink.rs. |
ReplicationVersioningStore |
Versioning state checks for object and delete replication decisions. | Bucket versioning sys access is exposed through the contract type in replication_versioning_boundary.rs. |
ReplicationLockTiming |
Namespace lock timing for replication resync, object replication, and delete replication locks. | SetDisks lock timeout access is exposed through the contract type in replication_lock_boundary.rs. |
ReplicationMsgpCodec |
MessagePack time encode/decode and unknown value skipping for persisted resync/MRF state. | Bucket MessagePack helpers are exposed through the contract type in replication_msgp_boundary.rs. |
ReplicationTagFilter |
Decode object tag strings for rule and metadata replication decisions. | Bucket tagging helper access is exposed through the contract type in replication_tagging_boundary.rs. |
ReplicationLifecycleBridge |
Lifecycle-originated delete and version-purge scheduling. | Lifecycle delete paths call the bridge contract in replication_lifecycle_bridge.rs instead of constructing replication delete work directly. |
ReplicationMigrationBridge |
Persisted resync status decode/encode access for bucket metadata migration. | Bucket migration calls the bridge contract in replication_migration_bridge.rs instead of importing internal resyncer codec helpers. |
ReplicationObjectBridge |
Object write/delete replication decision and scheduling entry point for app storage and SetDisks paths. | App and SetDisks object paths call the bridge contract in replication_object_bridge.rs instead of importing internal resyncer/pool helpers. |
ReplicationScannerBridge |
Scanner-originated replication heal scheduling. | Scanner heal paths call the bridge contract in replication_scanner_bridge.rs instead of importing the internal queue function directly. |
ReplicationTargetConfigBridge |
Bucket target removal checks against replication target rules. | Bucket target sys calls the bridge contract in replication_target_config_bridge.rs instead of importing replication config helper types directly. |
Migration Rules
- Do not move
bucket/replicationinto a new crate while workers import bucket target sys, metadata sys, runtime sources, bandwidth reader, notification services, or SetDisks lock timing directly. - Keep existing queue behavior, MRF persistence, resync state, target client semantics, notification/audit events, and scanner/heal classifications unchanged during inventory and contract PRs.
- Keep the current
ReplicationStoragetrait as the starting point. Split it only after call sites prove a narrower object read/write/delete/walk shape. - Preserve
rustfs_ecstore::api::bucket::replicationcompatibility until lifecycle, scanner, OBS, heal, and tests compile through replacement paths. - Keep imports between modules in this directory relative to the local
replication module, not
crate::bucket::replication::*self paths. - Keep runtime source access from importing ECStore object store or bucket monitor implementation types directly; use local boundary-owned aliases.
- Move at most one owner boundary per code-bearing PR and verify it with focused replication tests before broad gates. Non-behavioral contract-shape cleanup may batch already-established boundary wrappers when the owner and call semantics do not change.
- Keep the compatibility facade in
mod.rsas an explicit symbol list. Do not reintroduce wildcard re-exports for replication implementation modules. - Keep object write/delete replication helpers behind
ReplicationObjectBridge; do not export internal resyncer or pool scheduling helpers through the compatibility facade. - Keep ECStore owner modules outside
bucket/replicationbehind bridge contracts when they need replication codec or config helper behavior. - Keep storage-api replication status/state helpers behind
crates/storage-api/src/replication.rs; ECStore converts owner DTOs at the replication storage boundary. - Keep
rustfs-replicationindependent fromrustfs-filemeta; ECStore compatibility conversions live inreplication_filemeta_boundary.rs. - Keep
rustfs-replicationindependent fromrustfs-storage-api; ECStore compatibility conversions live inreplication_storage_boundary.rs. - Keep direct
rustfs-replicationimports inside ECStore replication concentrated in*_boundary.rsmodules. - Keep scanner, admin, and storage-owner replication status/DTO consumers
behind the ECStore replication facade; only
rustfs/src/app/storage_api.rsmay retain direct object/delete replication helper calls.
First Code-Bearing Step
Start with ReplicationRuntime or ReplicationEventSink. Both can be added as
narrow internal contracts while keeping current queue, MRF, resync, and target
behavior unchanged. Do not start with a crate move.
Current compatibility guard: crates/ecstore/tests/replication_facade_compat_test.rs
keeps the ECStore replication facade types covered while architecture rules
keep direct imports behind local storage_api boundaries.