gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-26 16:28:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor(ecstore): route replication filemeta contracts (#4122)

Browse Source
This commit is contained in:
Zhengchao An
2026-07-01 01:01:01 +08:00
committed by GitHub
parent 942a8b505d
commit e917f1f4be
9 changed files with 62 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/ecstore/src/bucket/replication/README.md
+6 -5
View File
@@ -9,11 +9,11 @@ and lifecycle/heal scheduling paths.
| Module | Current role | Split blocker |
|---|---|---|
| `config.rs` | Replication config helpers, rule matching, and tag filtering. | Uses replication-local tagging boundary and S3 DTOs directly. |
| `config.rs` | Replication config helpers, rule matching, and tag filtering. | Uses replication-local filemeta/tagging boundaries and S3 DTOs directly. |
| `datatypes.rs` | Replication status and operation DTOs. | Publicly re-exported through the ECStore replication facade. |
| `replication_pool.rs` | Replication queue, worker pool, MRF persistence, bucket stats, and delete/object scheduling. | Depends on bucket target sys, bucket metadata sys and metadata paths through local boundaries, config storage, storage contracts through the replication storage boundary, runtime sources, and notification state. |
| `replication_resyncer.rs` | Object replication, delete replication, resync, MRF encode/decode, target calls, and multipart target upload paths. | Depends on target calls and target config types through the replication target boundary, metadata paths and metadata systems through the replication metadata boundary, versioning systems, storage contracts through the replication storage boundary, runtime sources, notification events, bandwidth reader wrapping, and SetDisks lock timing. |
| `replication_state.rs` | Replication queue/stat state and worker accounting. | Reads runtime sources and owns shared replication pool/stat state. |
| `replication_pool.rs` | Replication queue, worker pool, MRF persistence, bucket stats, and delete/object scheduling. | Depends on bucket target sys, bucket metadata sys, metadata paths, and file metadata replication contracts through local boundaries, config storage, storage contracts through the replication storage boundary, runtime sources, and notification state. |
| `replication_resyncer.rs` | Object replication, delete replication, resync, MRF encode/decode, target calls, and multipart target upload paths. | Depends on target calls and target config types through the replication target boundary, metadata paths and metadata systems through the replication metadata boundary, file metadata replication contracts through the filemeta boundary, versioning systems, storage contracts through the replication storage boundary, runtime sources, notification events, bandwidth reader wrapping, and SetDisks lock timing. |
| `replication_state.rs` | Replication queue/stat state and worker accounting. | Reads runtime sources and file metadata replication contracts through local boundaries, and owns shared replication pool/stat state. |
| `rule.rs` | Rule evaluation helpers for object replication options. | Depends on ECStore replication object option types. |
| `mod.rs` | Compatibility re-export facade for the current ECStore owner. | Must stay stable until downstream scanner, lifecycle, heal, and metrics paths compile through replacement contracts. |
@@ -23,7 +23,8 @@ and lifecycle/heal scheduling paths.
|---|---|---|
| `ReplicationObjectIO` | Object read/write primitives used by config, MRF, resync status, and multipart replication paths. | ECStore object API reader/writer types and storage-api object IO contracts are concentrated in `replication_storage_boundary.rs`. |
| `ReplicationStorage` | Object read/write/delete, object walk, metadata update, and target object IO. | ECStore object API, storage-api contracts, and read option types are concentrated in `replication_storage_boundary.rs`. |
| `ReplicationMetadataStore` | Replication config, MRF/resync state, target reset headers, and status persistence. | Metadata sys access and replication metadata path constants are concentrated in `replication_metadata_boundary.rs`; versioning sys, config storage, and file metadata imports remain. |
| `ReplicationMetadataStore` | Replication config, MRF/resync state, target reset headers, and status persistence. | Metadata sys access and replication metadata path constants are concentrated in `replication_metadata_boundary.rs`; versioning sys and config storage imports remain. |
| `ReplicationFileMeta` | Replication status, decisions, MRF entries, resync decisions, and target reset helpers. | `rustfs_filemeta` replication contracts are concentrated in `replication_filemeta_boundary.rs`; `FileInfo` remains in the storage boundary for storage trait bindings and walk options. |
| `ReplicationTargetStore` | Bucket target listing, target client lookup, target offline checks, target config types, and target operation option types. | Bucket target sys access, `BucketTargets`, and target operation types are concentrated in `replication_target_boundary.rs`. |
| `ReplicationRuntime` | Worker pool, queue sizing, stats, bucket monitor, local node identity, cancellation, and admission state. | Direct runtime source/global access and shared replication pool/stat state. |
| `ReplicationBandwidthLimiter` | Target reader wrapping for replication bandwidth accounting and throttling. | Direct bucket bandwidth reader imports from resyncer paths. |
crates/ecstore/src/bucket/replication/config.rs
+1 -1
View File
@@ -12,9 +12,9 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use super::replication_filemeta_boundary::ReplicationType;
use super::replication_tagging_boundary as tagging_boundary;
use crate::bucket::replication::ReplicationRuleExt as _;
use rustfs_filemeta::ReplicationType;
use s3s::dto::DeleteMarkerReplicationStatus;
use s3s::dto::DeleteReplicationStatus;
use s3s::dto::Destination;
crates/ecstore/src/bucket/replication/mod.rs
+1
View File
@@ -17,6 +17,7 @@ pub mod datatypes;
mod replication_bandwidth_boundary;
mod replication_config_store;
mod replication_event_sink;
mod replication_filemeta_boundary;
mod replication_lock_boundary;
mod replication_metadata_boundary;
mod replication_msgp_boundary;
crates/ecstore/src/bucket/replication/replication_filemeta_boundary.rs
+21
View File
@@ -0,0 +1,21 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
pub(crate) use rustfs_filemeta::{
MrfOpKind, MrfReplicateEntry, REPLICATE_EXISTING, REPLICATE_EXISTING_DELETE, REPLICATE_HEAL, REPLICATE_HEAL_DELETE,
ReplicateDecision, ReplicateObjectInfo, ReplicateTargetDecision, ReplicatedInfos, ReplicatedTargetInfo, ReplicationAction,
ReplicationState, ReplicationStatusType, ReplicationType, ReplicationWorkerOperation, ResyncDecision, ResyncTargetDecision,
VersionPurgeStatusType, get_replication_state, parse_replicate_decision, replication_statuses_map, target_reset_header,
version_purge_statuses_map,
};
crates/ecstore/src/bucket/replication/replication_pool.rs
+5 -13
View File
@@ -13,6 +13,11 @@
// limitations under the License.
use super::replication_config_store as config_store;
use super::replication_filemeta_boundary::{
MrfOpKind, MrfReplicateEntry, REPLICATE_EXISTING, REPLICATE_HEAL, REPLICATE_HEAL_DELETE, ReplicateDecision,
ReplicateObjectInfo, ReplicatedTargetInfo, ReplicationStatusType, ReplicationType, ReplicationWorkerOperation,
ResyncDecision, VersionPurgeStatusType, replication_statuses_map, version_purge_statuses_map,
};
use super::replication_metadata_boundary as metadata_boundary;
use super::replication_storage_boundary::{DeletedObject, ObjectInfo, ObjectOptions, ReplicationObjectIO, ReplicationStorage};
use super::replication_target_boundary as target_boundary;
@@ -29,19 +34,6 @@ use crate::bucket::replication::replication_resyncer::{
use crate::bucket::replication::replication_state::ReplicationStats;
use crate::error::Error as EcstoreError;
use lazy_static::lazy_static;
use rustfs_filemeta::MrfOpKind;
use rustfs_filemeta::MrfReplicateEntry;
use rustfs_filemeta::ReplicateDecision;
use rustfs_filemeta::ReplicateObjectInfo;
use rustfs_filemeta::ReplicatedTargetInfo;
use rustfs_filemeta::ReplicationStatusType;
use rustfs_filemeta::ReplicationType;
use rustfs_filemeta::ReplicationWorkerOperation;
use rustfs_filemeta::ResyncDecision;
use rustfs_filemeta::VersionPurgeStatusType;
use rustfs_filemeta::replication_statuses_map;
use rustfs_filemeta::version_purge_statuses_map;
use rustfs_filemeta::{REPLICATE_EXISTING, REPLICATE_HEAL, REPLICATE_HEAL_DELETE};
use rustfs_utils::http::{SUFFIX_REPLICATION_TIMESTAMP, get_str};
use std::any::Any;
use std::sync::Arc;
crates/ecstore/src/bucket/replication/replication_resyncer.rs
+7 -7
View File
@@ -15,6 +15,12 @@
use super::replication_bandwidth_boundary;
use super::replication_config_store as config_store;
use super::replication_event_sink::{EventArgs, send_event};
use super::replication_filemeta_boundary::{
MrfOpKind, MrfReplicateEntry, REPLICATE_EXISTING, REPLICATE_EXISTING_DELETE, ReplicateDecision, ReplicateObjectInfo,
ReplicateTargetDecision, ReplicatedInfos, ReplicatedTargetInfo, ReplicationAction, ReplicationState, ReplicationStatusType,
ReplicationType, ReplicationWorkerOperation, ResyncDecision, ResyncTargetDecision, VersionPurgeStatusType,
get_replication_state, parse_replicate_decision, replication_statuses_map, target_reset_header, version_purge_statuses_map,
};
use super::replication_lock_boundary as lock_boundary;
use super::replication_metadata_boundary as metadata_boundary;
use super::replication_msgp_boundary::{read_msgp_ext8_time, skip_msgp_value, write_msgp_time};
@@ -49,12 +55,6 @@ use http_body::Frame;
use http_body_util::StreamBody;
use regex::Regex;
use rmp_serde;
use rustfs_filemeta::{
MrfReplicateEntry, REPLICATE_EXISTING, REPLICATE_EXISTING_DELETE, ReplicateDecision, ReplicateObjectInfo,
ReplicateTargetDecision, ReplicatedInfos, ReplicatedTargetInfo, ReplicationAction, ReplicationState, ReplicationStatusType,
ReplicationType, ReplicationWorkerOperation, ResyncDecision, ResyncTargetDecision, VersionPurgeStatusType,
get_replication_state, parse_replicate_decision, replication_statuses_map, target_reset_header, version_purge_statuses_map,
};
use rustfs_s3_types::EventName;
use rustfs_utils::http::{
AMZ_BUCKET_REPLICATION_STATUS, AMZ_OBJECT_TAGGING, AMZ_TAGGING_DIRECTIVE, CONTENT_ENCODING, HeaderExt as _,
@@ -1262,7 +1262,7 @@ impl ReplicationWorkerOperation for DeletedObjectReplicationInfo {
version_id: self.delete_object.version_id,
retry_count: 0,
size: 0,
op: rustfs_filemeta::MrfOpKind::Delete,
op: MrfOpKind::Delete,
delete_marker_version_id: self.delete_object.delete_marker_version_id,
delete_marker: self.delete_object.delete_marker,
}
crates/ecstore/src/bucket/replication/replication_state.rs
+1 -1
View File
@@ -12,9 +12,9 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use super::replication_filemeta_boundary::{ReplicatedTargetInfo, ReplicationStatusType, ReplicationType};
use super::runtime_boundary as runtime_sources;
use crate::error::Error;
use rustfs_filemeta::{ReplicatedTargetInfo, ReplicationStatusType, ReplicationType};
use serde::{Deserialize, Serialize};
use std::collections::{HashMap, VecDeque};
use std::sync::Arc;
docs/architecture/ecstore-module-split-plan.md
+8 -1
View File
@@ -103,7 +103,8 @@ Current coupling:
- replication workers depend on `ReplicationStorage`, ECStore object APIs and
storage-api contracts through the replication storage boundary, bucket target
clients, bucket metadata, file metadata replication state, scanner repair
clients, bucket metadata, file metadata replication state through the
filemeta boundary, scanner repair
classification, runtime replication pool/stat handles, bucket monitor and
bandwidth reader access through local boundaries, local node names, and
notification events;
@@ -130,6 +131,12 @@ Required contracts before crate movement:
MRF/resync state, and status persistence. Metadata sys access and replication
metadata path constants are concentrated in
`crates/ecstore/src/bucket/replication/replication_metadata_boundary.rs`.
- `ReplicationFileMeta`: replication status, decisions, MRF entries, resync
decisions, and target reset helpers. `rustfs_filemeta` replication contracts
are concentrated in
`crates/ecstore/src/bucket/replication/replication_filemeta_boundary.rs`,
while `FileInfo` remains in the storage boundary for storage trait bindings
and walk options.
- `ReplicationTargetStore`: bucket target listing, target client lookup,
target offline checks, target config types, and target operation option
types. Bucket target sys access, `BucketTargets`, and target operation types
scripts/check_architecture_migration_rules.sh
+12
View File
@@ -201,6 +201,7 @@ REPLICATION_FACADE_BYPASS_HITS_FILE="${TMP_DIR}/replication_facade_bypass_hits.t
REPLICATION_BANDWIDTH_BOUNDARY_BYPASS_HITS_FILE="${TMP_DIR}/replication_bandwidth_boundary_bypass_hits.txt"
REPLICATION_CONFIG_STORE_BYPASS_HITS_FILE="${TMP_DIR}/replication_config_store_bypass_hits.txt"
REPLICATION_EVENT_SINK_BYPASS_HITS_FILE="${TMP_DIR}/replication_event_sink_bypass_hits.txt"
REPLICATION_FILEMETA_BOUNDARY_BYPASS_HITS_FILE="${TMP_DIR}/replication_filemeta_boundary_bypass_hits.txt"
REPLICATION_LOCK_BOUNDARY_BYPASS_HITS_FILE="${TMP_DIR}/replication_lock_boundary_bypass_hits.txt"
REPLICATION_METADATA_BOUNDARY_BYPASS_HITS_FILE="${TMP_DIR}/replication_metadata_boundary_bypass_hits.txt"
REPLICATION_MSGP_BOUNDARY_BYPASS_HITS_FILE="${TMP_DIR}/replication_msgp_boundary_bypass_hits.txt"
@@ -2608,6 +2609,17 @@ if [[ -s "$REPLICATION_TAGGING_BOUNDARY_BYPASS_HITS_FILE" ]]; then
report_failure "replication tag decoding must stay behind replication tagging boundary: $(paste -sd '; ' "$REPLICATION_TAGGING_BOUNDARY_BYPASS_HITS_FILE")"
fi
(
cd "$ROOT_DIR"
rg -n --with-filename 'rustfs_filemeta::' crates/ecstore/src/bucket/replication --glob '*.rs' |
rg -v '^crates/ecstore/src/bucket/replication/replication_filemeta_boundary\.rs:' |
rg -v '^crates/ecstore/src/bucket/replication/replication_storage_boundary\.rs:' || true
) >"$REPLICATION_FILEMETA_BOUNDARY_BYPASS_HITS_FILE"
if [[ -s "$REPLICATION_FILEMETA_BOUNDARY_BYPASS_HITS_FILE" ]]; then
report_failure "replication filemeta contracts must stay behind replication filemeta boundary: $(paste -sd '; ' "$REPLICATION_FILEMETA_BOUNDARY_BYPASS_HITS_FILE")"
fi
(
cd "$ROOT_DIR"
find crates/ecstore/src/bucket/replication -type f -name '*.rs' -print0 |