mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-27 15:37:02 +00:00
refactor(admin): move site-replication service core out of handlers (#6699)
Mechanical move-only extraction for backlog#1840 PR1+PR4: the site-replication state (load/parse/persist/RMW transaction), repair state machine, peer transport (client cache, DNS resolver, send_peer_* family), retry queue, and the four storage-side hooks move from rustfs/src/admin/handlers/site_replication.rs into the new infra-layer module rustfs/src/site_replication/ ({mod,state,state_lock,identity,transport,retry,repair,hooks}.rs). The admin handler file keeps route registration, all Operation impls, request/response glue, and the in-file test module, and re-exports the moved items so existing paths keep resolving. admin/site_replication_identity.rs and admin/site_replication_state.rs relocate wholesale as identity.rs/state_lock.rs.
Storage access from the moved code goes through a new site_replication consumer module in the root facade (rustfs/src/storage_api.rs), including an s3 shim so the module stays off the direct s3s surface (file count stays at the 215 baseline). The three admin runtime-source wrappers the moved code needs (outbound TLS generation incl. the test atomic, outbound TLS state, runtime port) are reproduced locally; the TLS-generation trio moves out of admin/runtime_sources.rs since site replication was its only consumer. The one non-verbatim rewrite: site_replication_peer_payload inlines encrypt_stream_io in its encrypted branch, which is provably the branch encode_compatible_admin_payload always took for the /minio/admin peer-join wire path.
app/bucket_usecase.rs now imports the three bucket hooks from crate::site_replication, deleting the three app->interface entries from the layer baseline (shrink-only). The peer-client cache test moves with the owner-local SITE_REPLICATION_PEER_CLIENT static into transport.rs (228+1 = 229 tests conserved). New module files are added to the logging-guardrail checked list; the s3_error! line baseline tightens 1620 -> 1619; global-state/config-consumer inventories and ARCHITECTURE.md pointers updated.
Verified: cargo check -p rustfs --all-targets clean; cargo clippy --workspace --all-targets clean; cargo nextest run -p rustfs --lib 3852/3852 passed; make pre-commit green; scripts/check_layer_dependencies.sh green with baseline-only deletions; line-multiset conservation audit over the moved code accounts for every non-verbatim line (visibility bumps, import rewrites, fmt reflow).
Refs rustfs/backlog#1840
This commit is contained in:
Zhengchao An
@@ -332,6 +332,8 @@ The binary (`main.rs`) boots in this order:
|
||||
|
||||
- **"Where is replication configured?"**
|
||||
`admin/handlers/replication.rs` and `admin/handlers/site_replication.rs` for API,
|
||||
`rustfs/src/site_replication/` for the site-replication service subsystem
|
||||
(state, peer transport, retry queue, repair, hooks),
|
||||
`ecstore/src/bucket/replication/` for engine
|
||||
|
||||
- **"Where do I add a new admin endpoint?"**
|
||||
|
||||
@@ -123,7 +123,7 @@ behind narrower contracts.
|
||||
| Files | Current usage |
|
||||
|---|---|
|
||||
| `rustfs/src/admin/handlers/kms_dynamic.rs` | Uses generic `read_config` and `save_config` for dynamic KMS config objects. |
|
||||
| `rustfs/src/admin/handlers/site_replication.rs` | Uses generic `read_config`, `save_config`, and `delete_config` for site-replication state objects. |
|
||||
| `rustfs/src/site_replication/state.rs` | Uses generic `read_config`, `save_config`, and `delete_config` (via the root storage facade) for site-replication state objects. |
|
||||
| `rustfs/src/admin/service/site_replication.rs` | Uses generic `read_config` and `save_config` for site-replication state normalization. |
|
||||
| `rustfs/src/server/module_switch.rs` | Uses generic `read_config` and `save_config` for module-switch config objects. |
|
||||
| `crates/iam/src/store/object.rs` | Uses generic `read_config_no_lock`, `read_config_with_metadata`, `save_config`, `save_config_with_opts`, and `delete_config` helper variants for IAM object-store persistence paths. |
|
||||
|
||||
@@ -110,11 +110,11 @@ inventory. Generic function-local names such as `CACHE`, `LOCK`, `INIT`, and
|
||||
| `GET_OBJECT_BUFFER_THRESHOLD_WARNED`, `GET_READER_STREAM_BUFFER_SIZE_OVERRIDE`, function-local `ENABLED`, `OBJECT_SEEK_SUPPORT_THRESHOLD`, `OBJECT_SEEK_SUPPORT_CONCURRENCY_THRESHOLDS` | `rustfs/src/app/object/get.rs` | Cache or constant / owner-local cache | Object GET/seek tuning caches and warning guards stay private to object usecase helpers. |
|
||||
| `SUPPORTED_HEADERS` | `rustfs/src/storage/options.rs` | Cache or constant / owner-local constant | Supported-header lookup state stays private to storage option parsing. |
|
||||
| `AUDIT_TARGET_SPECS`, `NOTIFICATION_TARGET_SPECS` | `rustfs/src/admin/handlers/audit.rs`, `rustfs/src/admin/handlers/event.rs`, `rustfs/src/admin/handlers/plugins_instances.rs` | Cache or constant / owner-local constant | Admin target descriptor tables stay private to their handler owners. |
|
||||
| `SITE_REPLICATION_PEER_CLIENT` | `rustfs/src/admin/handlers/site_replication.rs` | Process-global owner-local cache | Site-replication peer client cache stays private to site-replication handlers. The state RMW transaction holds no process-local mutex — see `rustfs/src/admin/site_replication_state.rs`. |
|
||||
| `SITE_REPLICATION_PEER_CLIENT` | `rustfs/src/site_replication/transport.rs` | Process-global owner-local cache | Site-replication peer client cache stays private to the site-replication transport module. The state RMW transaction holds no process-local mutex — see `rustfs/src/site_replication/state_lock.rs`. |
|
||||
| `AUDIT_MODULE_ENABLED`, `NOTIFY_MODULE_ENABLED`, `PERSISTED_NOTIFY_MODULE_ENABLED`, `PERSISTED_AUDIT_MODULE_ENABLED`, `PERSISTED_MODULE_SWITCH_CONFIGURED` | `rustfs/src/server/audit.rs`, `rustfs/src/server/event.rs`, `rustfs/src/server/module_switch.rs` | Process-global owner-local toggles | Audit/notify module snapshots stay private to the server module switch owners. |
|
||||
| `DELETE_TAIL_TOTAL`, `DELETE_CLEANUP_TOTAL`, `DELETE_REPLICATION_TOTAL`, `DELETE_NOTIFY_TOTAL` | `rustfs/src/delete_tail_activity.rs` | Process-global owner-local counters | Delete-tail activity counters stay private behind delete-tail activity helpers. |
|
||||
| `EMBEDDED_SERVER_STARTED` | `rustfs/src/startup_lifecycle.rs` | Process-global owner-local guard | Embedded startup single-start protection stays private to startup lifecycle. |
|
||||
| `TEST_OUTBOUND_TLS_GENERATION` | `rustfs/src/admin/runtime_sources.rs` | Test or fixture state | Outbound TLS generation test hook state stays private to admin runtime-source tests. |
|
||||
| `TEST_OUTBOUND_TLS_GENERATION` | `rustfs/src/site_replication/mod.rs` | Test or fixture state | Outbound TLS generation test hook state stays private to site-replication transport tests. |
|
||||
| `TEST_REMAINING_FAILURES` | `rustfs/src/startup_iam.rs` | Test or fixture state | IAM startup retry injection state stays private to debug/test startup code. |
|
||||
| `CAPACITY_DIRTY_SCOPE_ENV`, `CAPACITY_DIRTY_SCOPE_INIT`, `GLOBAL_ENV`, function-local `INIT` | `rustfs/src/app/*_test.rs` | Test or fixture state | App integration test fixture state stays private to the owning test modules. |
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -23,8 +23,6 @@ pub(crate) mod route_policy;
|
||||
pub mod router;
|
||||
pub(crate) mod runtime_sources;
|
||||
pub mod service;
|
||||
pub mod site_replication_identity;
|
||||
pub(crate) mod site_replication_state;
|
||||
pub(crate) mod storage_api;
|
||||
pub mod utils;
|
||||
|
||||
|
||||
@@ -36,10 +36,8 @@ pub(crate) use crate::runtime_sources::{
|
||||
};
|
||||
use rustfs_config::server_config::Config;
|
||||
use rustfs_kms::KmsServiceManager;
|
||||
use rustfs_tls_runtime::{GlobalPublishedOutboundTlsState, TlsGeneration};
|
||||
use rustfs_tls_runtime::GlobalPublishedOutboundTlsState;
|
||||
use std::sync::Arc;
|
||||
#[cfg(test)]
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use tokio::sync::RwLock;
|
||||
|
||||
pub(crate) fn default_admin_usecase() -> DefaultAdminUsecase {
|
||||
@@ -116,29 +114,6 @@ pub(crate) fn current_or_init_kms_runtime_service_manager() -> Arc<KmsServiceMan
|
||||
.unwrap_or_else(rustfs_kms::init_global_kms_service_manager)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
static TEST_OUTBOUND_TLS_GENERATION: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) fn set_test_outbound_tls_generation(generation: u64) {
|
||||
root_runtime_sources::set_test_outbound_tls_generation(generation);
|
||||
TEST_OUTBOUND_TLS_GENERATION.store(generation, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
pub(crate) fn current_outbound_tls_generation() -> TlsGeneration {
|
||||
root_runtime_sources::current_outbound_tls_generation().unwrap_or_else(empty_outbound_tls_generation)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
fn empty_outbound_tls_generation() -> TlsGeneration {
|
||||
TlsGeneration(TEST_OUTBOUND_TLS_GENERATION.load(Ordering::Relaxed))
|
||||
}
|
||||
|
||||
#[cfg(not(test))]
|
||||
fn empty_outbound_tls_generation() -> TlsGeneration {
|
||||
TlsGeneration(0)
|
||||
}
|
||||
|
||||
pub(crate) async fn current_outbound_tls_state() -> GlobalPublishedOutboundTlsState {
|
||||
if let Some(state) = root_runtime_sources::current_outbound_tls_state().await {
|
||||
return state;
|
||||
|
||||
@@ -13,11 +13,11 @@
|
||||
// limitations under the License.
|
||||
|
||||
use crate::admin::runtime_sources::{AppContext, current_app_context, current_object_store_handle_for_context};
|
||||
use crate::admin::site_replication_identity::{
|
||||
use crate::admin::storage_api::error::Error as StorageError;
|
||||
use crate::site_replication::identity::{
|
||||
deployment_id_for_endpoint, mark_unknown_peer_sync_enabled, normalize_peer_map_by_identity_with,
|
||||
};
|
||||
use crate::admin::site_replication_state::{SITE_REPLICATION_STATE_PATH, with_site_replication_state_lock_on};
|
||||
use crate::admin::storage_api::error::Error as StorageError;
|
||||
use crate::site_replication::state_lock::{SITE_REPLICATION_STATE_PATH, with_site_replication_state_lock_on};
|
||||
use crate::storage::storage_api::{read_config_no_lock, save_config_no_lock};
|
||||
use rustfs_madmin::PeerInfo;
|
||||
use s3s::{S3Error, S3ErrorCode, S3Result};
|
||||
|
||||
@@ -445,8 +445,8 @@ pub(crate) mod replication {
|
||||
pub(crate) use super::ecstore_bucket::replication::{
|
||||
OperatorRuleContract, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_UNSUPPORTED_FIELDS,
|
||||
REMOTE_TARGET_WRITABLE_FIELDS, REPLICATION_CAPABILITY_CONTRACT_VERSION, REPLICATION_READ_ONLY_HISTORICAL_FIELDS,
|
||||
REPLICATION_WRITABLE_FIELDS, assign_site_replication_rule_priorities, is_site_replication_role,
|
||||
merge_incoming_replication_config, replication_target_arn_deployment_id, site_replication_rule_deployment_id,
|
||||
REPLICATION_WRITABLE_FIELDS, assign_site_replication_rule_priorities, merge_incoming_replication_config,
|
||||
replication_target_arn_deployment_id,
|
||||
};
|
||||
pub(crate) type BucketReplicationResyncStatus = super::ecstore_bucket::replication::BucketReplicationResyncStatus;
|
||||
pub(crate) type BucketStats = super::ecstore_bucket::replication::BucketStats;
|
||||
@@ -653,8 +653,6 @@ pub(crate) mod replication {
|
||||
|
||||
pub(crate) mod target {
|
||||
pub(crate) use super::ecstore_bucket::target::duration_from_secs_or_nanos;
|
||||
#[allow(clippy::upper_case_acronyms)]
|
||||
pub(crate) type ARN = super::ecstore_bucket::target::ARN;
|
||||
pub(crate) type BucketTarget = super::ecstore_bucket::target::BucketTarget;
|
||||
pub(crate) type BucketTargetType = super::ecstore_bucket::target::BucketTargetType;
|
||||
pub(crate) type BucketTargets = super::ecstore_bucket::target::BucketTargets;
|
||||
|
||||
@@ -64,9 +64,6 @@ use super::storage_api::bucket_usecase::{
|
||||
get_validated_store, process_lambda_configurations, process_queue_configurations, process_topic_configurations,
|
||||
request_context, validate_list_object_unordered_with_delimiter,
|
||||
};
|
||||
use crate::admin::handlers::site_replication::{
|
||||
site_replication_bucket_meta_hook, site_replication_delete_bucket_hook, site_replication_make_bucket_hook,
|
||||
};
|
||||
use crate::app::object_data_cache::invalidate_object_data_cache_bucket_after_delete;
|
||||
use crate::app::runtime_sources::{
|
||||
AppContext, current_app_context, current_encryption_service, current_notification_system,
|
||||
@@ -75,6 +72,9 @@ use crate::app::runtime_sources::{
|
||||
use crate::auth::get_condition_values_with_client_info;
|
||||
use crate::error::ApiError;
|
||||
use crate::shared_types::RemoteAddr;
|
||||
use crate::site_replication::{
|
||||
site_replication_bucket_meta_hook, site_replication_delete_bucket_hook, site_replication_make_bucket_hook,
|
||||
};
|
||||
use crate::storage::storage_api::lock_bucket_targets_metadata;
|
||||
use http::StatusCode;
|
||||
use metrics::counter;
|
||||
|
||||
@@ -100,6 +100,7 @@ pub mod runtime_capabilities;
|
||||
pub(crate) mod runtime_sources;
|
||||
pub mod server;
|
||||
pub mod shared_types;
|
||||
pub(crate) mod site_replication;
|
||||
pub(crate) mod site_replication_reconcile;
|
||||
pub(crate) mod startup_audit;
|
||||
pub(crate) mod startup_auth;
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
+1
-1
@@ -86,7 +86,7 @@ pub(crate) fn mark_unknown_peer_sync_enabled(peers: &mut BTreeMap<String, PeerIn
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn is_https_endpoint(endpoint: &str) -> bool {
|
||||
pub(crate) fn is_https_endpoint(endpoint: &str) -> bool {
|
||||
canonical_endpoint(endpoint).starts_with("https://")
|
||||
}
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Site-replication service subsystem (backlog#1840).
|
||||
//!
|
||||
//! The parts of site replication that storage-side flows call into — the
|
||||
//! persisted cluster state and its RMW transaction, the peer HTTP transport,
|
||||
//! the retry queue, the repair state machine, and the bucket/IAM broadcast
|
||||
//! hooks — live here in the infra layer. The admin HTTP handlers stay in
|
||||
//! `crate::admin::handlers::site_replication` and call down into this module;
|
||||
//! that file re-exports these items so existing paths keep resolving.
|
||||
//!
|
||||
//! Storage access goes through the root facade (`crate::storage_api`) and
|
||||
//! never through the admin or storage interface layers — this module sits
|
||||
//! below the interface layer and must not import upward.
|
||||
|
||||
pub(crate) mod identity;
|
||||
pub(crate) mod state_lock;
|
||||
|
||||
pub(crate) mod hooks;
|
||||
pub(crate) mod repair;
|
||||
pub(crate) mod retry;
|
||||
pub(crate) mod state;
|
||||
pub(crate) mod transport;
|
||||
|
||||
pub(crate) use self::hooks::*;
|
||||
pub(crate) use self::repair::*;
|
||||
pub(crate) use self::retry::*;
|
||||
pub(crate) use self::state::*;
|
||||
pub(crate) use self::transport::*;
|
||||
|
||||
use self::identity::{
|
||||
canonical_endpoint, deployment_id_for_endpoint, mark_unknown_peer_sync_enabled, normalize_peer_map_by_identity_with,
|
||||
same_identity_endpoint,
|
||||
};
|
||||
use self::state_lock::{SITE_REPLICATION_STATE_PATH, with_site_replication_state_lock};
|
||||
use crate::auth::constant_time_eq;
|
||||
use crate::config::get_config_snapshot;
|
||||
use crate::error::ApiError;
|
||||
use crate::runtime_sources::{
|
||||
current_deployment_id, current_endpoints_handle, current_iam_handle, current_object_store_handle, current_region,
|
||||
};
|
||||
use crate::storage_api::site_replication::s3::{
|
||||
Body, BucketLifecycleConfiguration, BucketVersioningStatus, DeleteMarkerReplication, DeleteMarkerReplicationStatus,
|
||||
DeleteReplication, DeleteReplicationStatus, Destination, ExistingObjectReplication, ExistingObjectReplicationStatus,
|
||||
LifecycleRule, ReplicaModifications, ReplicaModificationsStatus, ReplicationConfiguration, ReplicationRule,
|
||||
ReplicationRuleStatus, S3Error, S3ErrorCode, S3Response, S3Result, SourceSelectionCriteria, VersioningConfiguration,
|
||||
s3_error,
|
||||
};
|
||||
#[cfg(test)]
|
||||
use crate::storage_api::site_replication::save_config as save_admin_config;
|
||||
use crate::storage_api::site_replication::{
|
||||
ARN, BUCKET_REPLICATION_CONFIG, BUCKET_TARGETS_FILE, BUCKET_VERSIONING_CONFIG, BucketOperations, BucketOptions, BucketTarget,
|
||||
BucketTargetSys, BucketTargetType, BucketTargets, Credentials, ECStore, OperatorRuleContract, StorageError,
|
||||
VersioningApi as _, assign_site_replication_rule_priorities, delete_config_no_lock, deserialize, is_site_replication_role,
|
||||
lock_bucket_targets_metadata, metadata_sys, read_config as read_admin_config, read_config_no_lock,
|
||||
replication_target_arn_deployment_id, save_config_no_lock, serialize, site_replication_rule_deployment_id,
|
||||
with_config_object_read_lock, with_config_object_write_lock,
|
||||
};
|
||||
use base64_simd::STANDARD as BASE64_STANDARD;
|
||||
use base64_simd::URL_SAFE_NO_PAD;
|
||||
use hmac::{Hmac, Mac};
|
||||
use http::header::{CONTENT_TYPE, HOST};
|
||||
use http::{HeaderMap, HeaderValue, Uri};
|
||||
use hyper::{Method, StatusCode};
|
||||
use rustfs_config::{DEFAULT_CONSOLE_ADDRESS, DEFAULT_RUSTFS_TLS_PATH, ENV_RUSTFS_CONSOLE_ADDRESS, ENV_RUSTFS_TLS_PATH};
|
||||
use rustfs_iam::store::{MappedPolicy, UserType, sr_wire_user_type};
|
||||
use rustfs_iam::sys::SITE_REPLICATOR_SERVICE_ACCOUNT;
|
||||
use rustfs_madmin::{
|
||||
AddOrUpdateUserReq, GroupAddRemove, GroupStatus, PeerInfo, PeerSite, ReplicateEditStatus, SITE_REPL_API_VERSION,
|
||||
SRBucketInfo, SRBucketMeta, SRGroupInfo, SRIAMItem, SRIAMPolicy, SRInfo, SRPolicyMapping, SRRemoveReq, SRResyncOpStatus,
|
||||
SRRetryStats, SRStateInfo, SyncStatus,
|
||||
};
|
||||
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
|
||||
use rustfs_signer::sign_v4;
|
||||
use rustfs_tls_runtime::{GlobalPublishedOutboundTlsState, TlsGeneration};
|
||||
use rustfs_utils::egress::{OutboundUrlError, validate_outbound_url};
|
||||
use rustfs_utils::http::get_source_scheme;
|
||||
use rustls_pki_types::pem::PemObject;
|
||||
use serde::Deserialize;
|
||||
use serde::Serialize;
|
||||
use serde::de::IgnoredAny;
|
||||
use serde_json::Value;
|
||||
use sha2::{Digest, Sha256};
|
||||
use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet};
|
||||
use std::net::{IpAddr, SocketAddr};
|
||||
use std::sync::{Arc, LazyLock};
|
||||
use std::time::Duration;
|
||||
use time::OffsetDateTime;
|
||||
use tokio::sync::{Mutex, RwLock};
|
||||
use tracing::{info, warn};
|
||||
use url::{Url, form_urlencoded};
|
||||
use uuid::Uuid;
|
||||
|
||||
pub(crate) const LOG_COMPONENT_ADMIN: &str = "admin";
|
||||
|
||||
pub(crate) const LOG_SUBSYSTEM_SITE_REPLICATION: &str = "site_replication";
|
||||
|
||||
pub(crate) const EVENT_ADMIN_SITE_REPLICATION_STATE: &str = "admin_site_replication_state";
|
||||
|
||||
/// Layer-local mirror of `crate::admin::utils::json_response` (the repair
|
||||
/// executor answers the admin HTTP surface but must not import upward from
|
||||
/// the infra layer).
|
||||
fn json_response<T: Serialize>(status: StatusCode, value: &T) -> S3Result<S3Response<(StatusCode, Body)>> {
|
||||
let data = serde_json::to_vec(value)
|
||||
.map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, format!("failed to serialize response: {e}")))?;
|
||||
let mut headers = HeaderMap::new();
|
||||
headers.insert(CONTENT_TYPE, HeaderValue::from_static("application/json"));
|
||||
Ok(S3Response::with_headers((status, Body::from(data)), headers))
|
||||
}
|
||||
|
||||
// The admin layer's runtime-source wrappers apply fallbacks on top of
|
||||
// `crate::runtime_sources`; this module reproduces the same fallbacks locally
|
||||
// (verbatim from `crate::admin::runtime_sources`) so it never imports upward
|
||||
// into the interface layer.
|
||||
|
||||
#[cfg(test)]
|
||||
static TEST_OUTBOUND_TLS_GENERATION: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) fn set_test_outbound_tls_generation(generation: u64) {
|
||||
crate::runtime_sources::set_test_outbound_tls_generation(generation);
|
||||
TEST_OUTBOUND_TLS_GENERATION.store(generation, std::sync::atomic::Ordering::Relaxed);
|
||||
}
|
||||
|
||||
fn current_outbound_tls_generation() -> TlsGeneration {
|
||||
crate::runtime_sources::current_outbound_tls_generation().unwrap_or_else(empty_outbound_tls_generation)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
fn empty_outbound_tls_generation() -> TlsGeneration {
|
||||
TlsGeneration(TEST_OUTBOUND_TLS_GENERATION.load(std::sync::atomic::Ordering::Relaxed))
|
||||
}
|
||||
|
||||
#[cfg(not(test))]
|
||||
fn empty_outbound_tls_generation() -> TlsGeneration {
|
||||
TlsGeneration(0)
|
||||
}
|
||||
|
||||
async fn current_outbound_tls_state() -> GlobalPublishedOutboundTlsState {
|
||||
if let Some(state) = crate::runtime_sources::current_outbound_tls_state().await {
|
||||
return state;
|
||||
}
|
||||
|
||||
crate::runtime_sources::fallback_outbound_tls_runtime_interface()
|
||||
.state()
|
||||
.await
|
||||
}
|
||||
|
||||
fn current_runtime_port() -> u16 {
|
||||
crate::runtime_sources::current_runtime_port().unwrap_or(rustfs_config::DEFAULT_PORT)
|
||||
}
|
||||
@@ -0,0 +1,845 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use super::*;
|
||||
|
||||
pub(crate) const SITE_REPLICATION_REPAIR_STATE_PATH: &str = "config/site-replication/repair-state.json";
|
||||
|
||||
pub(crate) const SITE_REPLICATION_REPAIR_EXECUTION_LOCK_PATH: &str = "config/site-replication/repair-execution.lock";
|
||||
|
||||
pub(crate) const SITE_REPLICATION_REPAIR_OPERATION_LIMIT: usize = 32;
|
||||
|
||||
pub(crate) const SITE_REPLICATION_REPAIR_IAM_FAMILY: &str = "iam";
|
||||
|
||||
pub(crate) const SITE_REPLICATION_REPAIR_BUCKET_FAMILY: &str = "bucket";
|
||||
|
||||
pub(crate) const SITE_REPLICATION_REPAIR_BUCKET_METADATA_FAMILY: &str = "bucket-metadata";
|
||||
|
||||
pub(crate) const SITE_REPLICATION_REPAIR_REPLICATION_FAMILY: &str = "replication";
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub(crate) struct SiteReplicationRepairState {
|
||||
#[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
|
||||
pub(crate) operations: BTreeMap<String, SiteReplicationRepairOperation>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub(crate) struct SiteReplicationRepairOperation {
|
||||
pub(crate) operation_id: String,
|
||||
pub(crate) preflight_token: String,
|
||||
pub(crate) plan_token: String,
|
||||
pub(crate) status: String,
|
||||
#[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
|
||||
pub(crate) sites: BTreeMap<String, SiteReplicationRepairSiteStatus>,
|
||||
#[serde(default, with = "time::serde::rfc3339::option", skip_serializing_if = "Option::is_none")]
|
||||
pub(crate) created_at: Option<OffsetDateTime>,
|
||||
#[serde(default, with = "time::serde::rfc3339::option", skip_serializing_if = "Option::is_none")]
|
||||
pub(crate) updated_at: Option<OffsetDateTime>,
|
||||
#[serde(default, with = "time::serde::rfc3339::option", skip_serializing_if = "Option::is_none")]
|
||||
pub(crate) completed_at: Option<OffsetDateTime>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub(crate) struct SiteReplicationRepairSiteStatus {
|
||||
pub(crate) deployment_id: String,
|
||||
pub(crate) name: String,
|
||||
#[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
|
||||
pub(crate) families: BTreeMap<String, SiteReplicationRepairFamilyStatus>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub(crate) struct SiteReplicationRepairFamilyStatus {
|
||||
pub(crate) planned: usize,
|
||||
pub(crate) succeeded: usize,
|
||||
pub(crate) failed: usize,
|
||||
#[serde(default)]
|
||||
pub(crate) retry_events: usize,
|
||||
#[serde(default, skip_serializing_if = "Vec::is_empty")]
|
||||
pub(crate) tasks: Vec<SiteReplicationRepairTaskStatus>,
|
||||
#[serde(default, skip_serializing_if = "Vec::is_empty")]
|
||||
pub(crate) errors: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub(crate) struct SiteReplicationRepairTaskStatus {
|
||||
pub(crate) task_id: String,
|
||||
pub(crate) status: String,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub(crate) error: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
#[serde(rename_all = "camelCase", deny_unknown_fields)]
|
||||
pub(crate) struct SiteReplicationRepairRequest {
|
||||
pub(crate) mode: SiteReplicationRepairMode,
|
||||
#[serde(default)]
|
||||
pub(crate) preflight_token: Option<String>,
|
||||
#[serde(default)]
|
||||
pub(crate) operation_id: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize, PartialEq, Eq)]
|
||||
#[serde(rename_all = "kebab-case")]
|
||||
pub(crate) enum SiteReplicationRepairMode {
|
||||
DryRun,
|
||||
Execute,
|
||||
}
|
||||
|
||||
pub(crate) struct SiteReplicationRepairExecutionRequest {
|
||||
pub(crate) local_peer: PeerInfo,
|
||||
pub(crate) preflight_token: String,
|
||||
pub(crate) operation_id: String,
|
||||
pub(crate) signing_key: String,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub(crate) struct SiteReplicationRepairPreflight {
|
||||
pub(crate) mode: &'static str,
|
||||
pub(crate) status: &'static str,
|
||||
pub(crate) preflight_token: String,
|
||||
pub(crate) retry_events: usize,
|
||||
pub(crate) sites: BTreeMap<String, SiteReplicationRepairSiteStatus>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub(crate) struct SiteReplicationRepairOperationResponse {
|
||||
pub(crate) mode: &'static str,
|
||||
pub(crate) operation_id: String,
|
||||
pub(crate) status: String,
|
||||
pub(crate) sites: BTreeMap<String, SiteReplicationRepairSiteResponse>,
|
||||
#[serde(with = "time::serde::rfc3339::option", skip_serializing_if = "Option::is_none")]
|
||||
pub(crate) created_at: Option<OffsetDateTime>,
|
||||
#[serde(with = "time::serde::rfc3339::option", skip_serializing_if = "Option::is_none")]
|
||||
pub(crate) updated_at: Option<OffsetDateTime>,
|
||||
#[serde(with = "time::serde::rfc3339::option", skip_serializing_if = "Option::is_none")]
|
||||
pub(crate) completed_at: Option<OffsetDateTime>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub(crate) struct SiteReplicationRepairSiteResponse {
|
||||
pub(crate) deployment_id: String,
|
||||
pub(crate) name: String,
|
||||
pub(crate) families: BTreeMap<String, SiteReplicationRepairFamilyResponse>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub(crate) struct SiteReplicationRepairFamilyResponse {
|
||||
pub(crate) planned: usize,
|
||||
pub(crate) succeeded: usize,
|
||||
pub(crate) failed: usize,
|
||||
pub(crate) retry_events: usize,
|
||||
pub(crate) tasks: Vec<SiteReplicationRepairTaskStatus>,
|
||||
#[serde(skip_serializing_if = "Vec::is_empty")]
|
||||
pub(crate) errors: Vec<String>,
|
||||
}
|
||||
|
||||
pub(crate) async fn load_site_replication_repair_state_from_store(store: Arc<ECStore>) -> S3Result<SiteReplicationRepairState> {
|
||||
match read_config_no_lock(store, SITE_REPLICATION_REPAIR_STATE_PATH).await {
|
||||
Ok(data) => serde_json::from_slice(&data).map_err(|e| {
|
||||
S3Error::with_message(S3ErrorCode::InternalError, format!("invalid site replication repair state: {e}"))
|
||||
}),
|
||||
Err(StorageError::ConfigNotFound) => Ok(SiteReplicationRepairState::default()),
|
||||
Err(err) => Err(S3Error::with_message(
|
||||
S3ErrorCode::InternalError,
|
||||
format!("failed to load site replication repair state: {err}"),
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) async fn save_site_replication_repair_state_to_store(
|
||||
store: Arc<ECStore>,
|
||||
state: &SiteReplicationRepairState,
|
||||
) -> S3Result<()> {
|
||||
let data = serde_json::to_vec(state)
|
||||
.map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, format!("serialize repair state failed: {e}")))?;
|
||||
save_config_no_lock(store, SITE_REPLICATION_REPAIR_STATE_PATH, data)
|
||||
.await
|
||||
.map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, format!("save repair state failed: {e}")))
|
||||
}
|
||||
|
||||
pub(crate) async fn read_site_replication_repair_state() -> S3Result<SiteReplicationRepairState> {
|
||||
let store =
|
||||
current_object_store_handle().ok_or_else(|| S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()))?;
|
||||
let read_store = store.clone();
|
||||
with_config_object_read_lock(store, SITE_REPLICATION_REPAIR_STATE_PATH.to_string(), move || async move {
|
||||
load_site_replication_repair_state_from_store(read_store).await
|
||||
})
|
||||
.await
|
||||
.map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, format!("lock repair state failed: {e}")))?
|
||||
}
|
||||
|
||||
pub(crate) async fn update_site_replication_repair_state<T, F>(update: F) -> S3Result<T>
|
||||
where
|
||||
T: Send + 'static,
|
||||
F: FnOnce(&mut SiteReplicationRepairState) -> S3Result<T> + Send + 'static,
|
||||
{
|
||||
let store =
|
||||
current_object_store_handle().ok_or_else(|| S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()))?;
|
||||
let read_store = store.clone();
|
||||
let save_store = store.clone();
|
||||
with_config_object_write_lock(store, SITE_REPLICATION_REPAIR_STATE_PATH.to_string(), move || async move {
|
||||
let mut state = load_site_replication_repair_state_from_store(read_store).await?;
|
||||
let result = update(&mut state)?;
|
||||
save_site_replication_repair_state_to_store(save_store, &state).await?;
|
||||
Ok(result)
|
||||
})
|
||||
.await
|
||||
.map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, format!("lock repair state failed: {e}")))?
|
||||
}
|
||||
|
||||
pub(crate) enum SiteReplicationRepairTask<'a> {
|
||||
Iam(&'a SRIAMItem),
|
||||
BucketMake(&'a str),
|
||||
BucketMetadata(&'a SRBucketMeta),
|
||||
Replication(&'a str),
|
||||
}
|
||||
|
||||
impl SiteReplicationRepairTask<'_> {
|
||||
pub(crate) fn family(&self) -> &'static str {
|
||||
match self {
|
||||
Self::Iam(_) => SITE_REPLICATION_REPAIR_IAM_FAMILY,
|
||||
Self::BucketMake(_) => SITE_REPLICATION_REPAIR_BUCKET_FAMILY,
|
||||
Self::BucketMetadata(_) => SITE_REPLICATION_REPAIR_BUCKET_METADATA_FAMILY,
|
||||
Self::Replication(_) => SITE_REPLICATION_REPAIR_REPLICATION_FAMILY,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn path(&self) -> &str {
|
||||
match self {
|
||||
Self::Iam(_) => "/rustfs/admin/v3/site-replication/peer/iam-item",
|
||||
Self::BucketMake(path) | Self::Replication(path) => path,
|
||||
Self::BucketMetadata(_) => "/rustfs/admin/v3/site-replication/peer/bucket-meta",
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn id(&self) -> S3Result<String> {
|
||||
let payload = match self {
|
||||
Self::Iam(item) => serde_json::to_vec(item),
|
||||
Self::BucketMake(_) | Self::Replication(_) => serde_json::to_vec(&serde_json::json!({})),
|
||||
Self::BucketMetadata(item) => serde_json::to_vec(item),
|
||||
}
|
||||
.map_err(|err| S3Error::with_message(S3ErrorCode::InternalError, format!("serialize repair task failed: {err}")))?;
|
||||
let mut digest = Sha256::new();
|
||||
digest.update(self.family().as_bytes());
|
||||
digest.update([0]);
|
||||
digest.update(self.path().as_bytes());
|
||||
digest.update([0]);
|
||||
digest.update(payload);
|
||||
Ok(URL_SAFE_NO_PAD.encode_to_string(digest.finalize()))
|
||||
}
|
||||
|
||||
pub(crate) async fn send(&self, transport: &PeerTransport, access_key: &str, secret_key: &str) -> S3Result<Vec<u8>> {
|
||||
match self {
|
||||
Self::Iam(item) => {
|
||||
send_peer_admin_request_with_client(
|
||||
&transport.client,
|
||||
&transport.connection,
|
||||
self.path(),
|
||||
access_key,
|
||||
secret_key,
|
||||
item,
|
||||
)
|
||||
.await
|
||||
}
|
||||
Self::BucketMetadata(item) => {
|
||||
send_peer_admin_request_with_client(
|
||||
&transport.client,
|
||||
&transport.connection,
|
||||
self.path(),
|
||||
access_key,
|
||||
secret_key,
|
||||
item,
|
||||
)
|
||||
.await
|
||||
}
|
||||
Self::BucketMake(_) | Self::Replication(_) => {
|
||||
send_peer_admin_request_with_client(
|
||||
&transport.client,
|
||||
&transport.connection,
|
||||
self.path(),
|
||||
access_key,
|
||||
secret_key,
|
||||
&serde_json::json!({}),
|
||||
)
|
||||
.await
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn site_replication_repair_tasks(plan: &SiteReplicationBootstrapPlan) -> Vec<(usize, SiteReplicationRepairTask<'_>)> {
|
||||
let mut tasks = Vec::with_capacity(
|
||||
plan.iam_items.len() + plan.bucket_make_ops.len() + plan.bucket_items.len() + plan.bucket_configure_ops.len(),
|
||||
);
|
||||
tasks.extend(
|
||||
plan.iam_items
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(index, item)| (index, SiteReplicationRepairTask::Iam(item))),
|
||||
);
|
||||
tasks.extend(
|
||||
plan.bucket_make_ops
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(index, path)| (index, SiteReplicationRepairTask::BucketMake(path))),
|
||||
);
|
||||
tasks.extend(
|
||||
plan.bucket_items
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(index, item)| (index, SiteReplicationRepairTask::BucketMetadata(item))),
|
||||
);
|
||||
tasks.extend(
|
||||
plan.bucket_configure_ops
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(index, path)| (index, SiteReplicationRepairTask::Replication(path))),
|
||||
);
|
||||
tasks
|
||||
}
|
||||
|
||||
pub(crate) fn site_replication_repair_plan_token(
|
||||
state: &SiteReplicationState,
|
||||
plan: &SiteReplicationBootstrapPlan,
|
||||
) -> S3Result<String> {
|
||||
let mut digest = Sha256::new();
|
||||
let snapshot = serde_json::to_vec(&(
|
||||
&state.name,
|
||||
&state.service_account_access_key,
|
||||
&state.peers,
|
||||
state.updated_at,
|
||||
state.sync_state_initialized,
|
||||
))
|
||||
.map_err(|err| S3Error::with_message(S3ErrorCode::InternalError, format!("serialize repair snapshot failed: {err}")))?;
|
||||
digest.update(snapshot);
|
||||
for (_, task) in site_replication_repair_tasks(plan) {
|
||||
digest.update(task.id()?.as_bytes());
|
||||
}
|
||||
Ok(URL_SAFE_NO_PAD.encode_to_string(digest.finalize()))
|
||||
}
|
||||
|
||||
pub(crate) fn site_replication_repair_preflight_token(
|
||||
state: &SiteReplicationState,
|
||||
plan: &SiteReplicationBootstrapPlan,
|
||||
signing_key: &[u8],
|
||||
) -> S3Result<String> {
|
||||
if signing_key.is_empty() {
|
||||
return Err(S3Error::with_message(
|
||||
S3ErrorCode::InternalError,
|
||||
"repair signing key is empty".to_string(),
|
||||
));
|
||||
}
|
||||
let mut digest = <Hmac<Sha256> as hmac::digest::KeyInit>::new_from_slice(signing_key)
|
||||
.map_err(|_| S3Error::with_message(S3ErrorCode::InternalError, "invalid repair signing key".to_string()))?;
|
||||
digest.update(b"rustfs:site-replication:repair-preflight:v1\0");
|
||||
digest.update(site_replication_repair_plan_token(state, plan)?.as_bytes());
|
||||
for event in state
|
||||
.retry_queue
|
||||
.iter()
|
||||
.filter(|event| retry_event_replayed_by_bootstrap(event))
|
||||
{
|
||||
digest.update(event.id.as_bytes());
|
||||
digest.update(&[0]);
|
||||
digest.update(event.peer_deployment_id.as_bytes());
|
||||
digest.update(&[0]);
|
||||
digest.update(event.path.as_bytes());
|
||||
digest.update(&[0]);
|
||||
}
|
||||
Ok(URL_SAFE_NO_PAD.encode_to_string(digest.finalize().into_bytes()))
|
||||
}
|
||||
|
||||
pub(crate) fn site_replication_repair_task_checkpoint_id(
|
||||
signing_key: &[u8],
|
||||
peer_deployment_id: &str,
|
||||
task: &SiteReplicationRepairTask<'_>,
|
||||
) -> S3Result<String> {
|
||||
let mut digest = <Hmac<Sha256> as hmac::digest::KeyInit>::new_from_slice(signing_key)
|
||||
.map_err(|_| S3Error::with_message(S3ErrorCode::InternalError, "invalid repair signing key".to_string()))?;
|
||||
digest.update(b"rustfs:site-replication:repair-task:v1\0");
|
||||
digest.update(peer_deployment_id.as_bytes());
|
||||
digest.update(&[0]);
|
||||
digest.update(task.id()?.as_bytes());
|
||||
Ok(URL_SAFE_NO_PAD.encode_to_string(digest.finalize().into_bytes()))
|
||||
}
|
||||
|
||||
pub(crate) fn site_replication_repair_sites(
|
||||
state: &SiteReplicationState,
|
||||
local_peer: &PeerInfo,
|
||||
plan: &SiteReplicationBootstrapPlan,
|
||||
signing_key: &[u8],
|
||||
) -> S3Result<BTreeMap<String, SiteReplicationRepairSiteStatus>> {
|
||||
let mut planned = BTreeMap::new();
|
||||
let mut family_paths = BTreeMap::<String, BTreeSet<String>>::new();
|
||||
for (_, task) in site_replication_repair_tasks(plan) {
|
||||
let family = task.family().to_string();
|
||||
let family_status = planned
|
||||
.entry(task.family().to_string())
|
||||
.or_insert_with(SiteReplicationRepairFamilyStatus::default);
|
||||
family_status.planned += 1;
|
||||
family_paths.entry(family).or_default().insert(task.path().to_string());
|
||||
}
|
||||
|
||||
let mut sites = BTreeMap::new();
|
||||
for peer in state.peers.values().filter(|peer| {
|
||||
peer.deployment_id != local_peer.deployment_id && !same_identity_endpoint(&peer.endpoint, &local_peer.endpoint)
|
||||
}) {
|
||||
let mut families = planned.clone();
|
||||
for (_, task) in site_replication_repair_tasks(plan) {
|
||||
let family = families
|
||||
.get_mut(task.family())
|
||||
.ok_or_else(|| S3Error::with_message(S3ErrorCode::InternalError, "repair task family is missing".to_string()))?;
|
||||
family.tasks.push(SiteReplicationRepairTaskStatus {
|
||||
task_id: site_replication_repair_task_checkpoint_id(signing_key, &peer.deployment_id, &task)?,
|
||||
status: "planned".to_string(),
|
||||
error: None,
|
||||
});
|
||||
}
|
||||
for (family, status) in &mut families {
|
||||
status.retry_events = state
|
||||
.retry_queue
|
||||
.iter()
|
||||
.filter(|event| {
|
||||
event.peer_deployment_id == peer.deployment_id
|
||||
&& retry_event_replayed_by_bootstrap(event)
|
||||
&& family_paths.get(family).is_some_and(|paths| paths.contains(&event.path))
|
||||
})
|
||||
.count();
|
||||
}
|
||||
sites.insert(
|
||||
peer.deployment_id.clone(),
|
||||
SiteReplicationRepairSiteStatus {
|
||||
deployment_id: peer.deployment_id.clone(),
|
||||
name: peer.name.clone(),
|
||||
families,
|
||||
},
|
||||
);
|
||||
}
|
||||
Ok(sites)
|
||||
}
|
||||
|
||||
pub(crate) fn update_site_replication_repair_task(
|
||||
operation: &mut SiteReplicationRepairOperation,
|
||||
deployment_id: &str,
|
||||
family: &str,
|
||||
family_index: usize,
|
||||
result: Result<(), &str>,
|
||||
) -> S3Result<()> {
|
||||
let site = operation
|
||||
.sites
|
||||
.get_mut(deployment_id)
|
||||
.ok_or_else(|| S3Error::with_message(S3ErrorCode::InternalError, "repair operation site is missing".to_string()))?;
|
||||
let family_status = site
|
||||
.families
|
||||
.get_mut(family)
|
||||
.ok_or_else(|| S3Error::with_message(S3ErrorCode::InternalError, "repair operation family is missing".to_string()))?;
|
||||
if family_status.succeeded != family_index {
|
||||
return Err(S3Error::with_message(
|
||||
S3ErrorCode::InternalError,
|
||||
"repair operation task checkpoint is invalid".to_string(),
|
||||
));
|
||||
}
|
||||
let task_status = family_status.tasks.get_mut(family_index).ok_or_else(|| {
|
||||
S3Error::with_message(S3ErrorCode::InternalError, "repair operation task checkpoint is missing".to_string())
|
||||
})?;
|
||||
family_status.failed = 0;
|
||||
family_status.errors.clear();
|
||||
match result {
|
||||
Ok(()) => {
|
||||
family_status.succeeded = family_status.succeeded.saturating_add(1);
|
||||
task_status.status = "succeeded".to_string();
|
||||
task_status.error = None;
|
||||
}
|
||||
Err(error) => {
|
||||
let error = classify_site_replication_repair_error(error).to_string();
|
||||
family_status.failed = 1;
|
||||
family_status.errors.push(error.clone());
|
||||
task_status.status = "failed".to_string();
|
||||
task_status.error = Some(error);
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) fn site_replication_repair_task_pending(
|
||||
operation: &SiteReplicationRepairOperation,
|
||||
deployment_id: &str,
|
||||
family: &str,
|
||||
family_index: usize,
|
||||
) -> S3Result<bool> {
|
||||
let site = operation
|
||||
.sites
|
||||
.get(deployment_id)
|
||||
.ok_or_else(|| S3Error::with_message(S3ErrorCode::InternalError, "repair operation site is missing".to_string()))?;
|
||||
let family = site
|
||||
.families
|
||||
.get(family)
|
||||
.ok_or_else(|| S3Error::with_message(S3ErrorCode::InternalError, "repair operation family is missing".to_string()))?;
|
||||
if family.succeeded > family_index {
|
||||
return Ok(false);
|
||||
}
|
||||
if family.succeeded < family_index {
|
||||
return Ok(false);
|
||||
}
|
||||
Ok(family.failed == 0)
|
||||
}
|
||||
|
||||
pub(crate) fn prepare_site_replication_repair_retry(operation: &mut SiteReplicationRepairOperation) {
|
||||
for family in operation.sites.values_mut().flat_map(|site| site.families.values_mut()) {
|
||||
family.failed = 0;
|
||||
family.errors.clear();
|
||||
for task in &mut family.tasks {
|
||||
match task.status.as_str() {
|
||||
"succeeded" => task.status = "skipped".to_string(),
|
||||
"failed" => {
|
||||
task.status = "planned".to_string();
|
||||
task.error = None;
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn classify_site_replication_repair_error(error: &str) -> &'static str {
|
||||
let error = error.to_ascii_lowercase();
|
||||
if error.contains("accessdenied")
|
||||
|| error.contains("signaturedoesnotmatch")
|
||||
|| error.contains("unauthorized")
|
||||
|| error.contains("forbidden")
|
||||
|| error.contains("401")
|
||||
|| error.contains("403")
|
||||
{
|
||||
"authorization-failed"
|
||||
} else if error.contains("timeout") {
|
||||
"remote-timeout"
|
||||
} else if error.contains("dns") {
|
||||
"remote-dns-failed"
|
||||
} else if error.contains("tls") || error.contains("certificate") {
|
||||
"remote-tls-failed"
|
||||
} else if error.contains("connect") {
|
||||
"remote-connect-failed"
|
||||
} else {
|
||||
"remote-operation-failed"
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn summarize_site_replication_repair_operation(operation: &mut SiteReplicationRepairOperation) {
|
||||
let failed = operation
|
||||
.sites
|
||||
.values()
|
||||
.flat_map(|site| site.families.values())
|
||||
.any(|family| family.failed > 0);
|
||||
let complete = operation
|
||||
.sites
|
||||
.values()
|
||||
.all(|site| site.families.values().all(|family| family.succeeded == family.planned));
|
||||
operation.status = if complete {
|
||||
"success"
|
||||
} else if failed {
|
||||
"partial"
|
||||
} else {
|
||||
"running"
|
||||
}
|
||||
.to_string();
|
||||
operation.updated_at = Some(OffsetDateTime::now_utc());
|
||||
operation.completed_at = complete.then_some(OffsetDateTime::now_utc());
|
||||
}
|
||||
|
||||
pub(crate) fn site_replication_repair_operation_response(
|
||||
operation: &SiteReplicationRepairOperation,
|
||||
) -> SiteReplicationRepairOperationResponse {
|
||||
SiteReplicationRepairOperationResponse {
|
||||
mode: "execute",
|
||||
operation_id: operation.operation_id.clone(),
|
||||
status: operation.status.clone(),
|
||||
sites: operation
|
||||
.sites
|
||||
.iter()
|
||||
.map(|(deployment_id, site)| {
|
||||
(
|
||||
deployment_id.clone(),
|
||||
SiteReplicationRepairSiteResponse {
|
||||
deployment_id: site.deployment_id.clone(),
|
||||
name: site.name.clone(),
|
||||
families: site
|
||||
.families
|
||||
.iter()
|
||||
.map(|(family, status)| {
|
||||
(
|
||||
family.clone(),
|
||||
SiteReplicationRepairFamilyResponse {
|
||||
planned: status.planned,
|
||||
succeeded: status.succeeded,
|
||||
failed: status.failed,
|
||||
retry_events: status.retry_events,
|
||||
tasks: status.tasks.clone(),
|
||||
errors: status.errors.clone(),
|
||||
},
|
||||
)
|
||||
})
|
||||
.collect(),
|
||||
},
|
||||
)
|
||||
})
|
||||
.collect(),
|
||||
created_at: operation.created_at,
|
||||
updated_at: operation.updated_at,
|
||||
completed_at: operation.completed_at,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn prune_site_replication_repair_operations(operations: &mut BTreeMap<String, SiteReplicationRepairOperation>) {
|
||||
while operations.len() > SITE_REPLICATION_REPAIR_OPERATION_LIMIT {
|
||||
let Some(oldest) = operations
|
||||
.iter()
|
||||
.filter(|(_, operation)| operation.status == "success")
|
||||
.min_by_key(|(_, operation)| operation.created_at)
|
||||
.map(|(id, _)| id.clone())
|
||||
else {
|
||||
break;
|
||||
};
|
||||
operations.remove(&oldest);
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) async fn persist_site_replication_repair_operation(operation: &SiteReplicationRepairOperation) -> S3Result<()> {
|
||||
let operation = operation.clone();
|
||||
update_site_replication_repair_state(move |state| {
|
||||
if let Some(existing) = state.operations.get(&operation.operation_id)
|
||||
&& !constant_time_eq(&existing.preflight_token, &operation.preflight_token)
|
||||
{
|
||||
return Err(S3Error::with_message(
|
||||
S3ErrorCode::ClientTokenConflict,
|
||||
"repair operation ID is already bound to a different preflight".to_string(),
|
||||
));
|
||||
}
|
||||
state.operations.insert(operation.operation_id.clone(), operation);
|
||||
prune_site_replication_repair_operations(&mut state.operations);
|
||||
Ok(())
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
pub(crate) async fn persist_site_replication_repair_task(
|
||||
operation: &SiteReplicationRepairOperation,
|
||||
peer: &PeerInfo,
|
||||
family: &str,
|
||||
path: &str,
|
||||
) -> S3Result<()> {
|
||||
persist_site_replication_repair_operation(operation).await?;
|
||||
|
||||
let family_status = operation
|
||||
.sites
|
||||
.get(&peer.deployment_id)
|
||||
.and_then(|site| site.families.get(family))
|
||||
.ok_or_else(|| S3Error::with_message(S3ErrorCode::InternalError, "repair task status is missing".to_string()))?;
|
||||
let failure = (family_status.failed > 0).then(|| {
|
||||
family_status
|
||||
.errors
|
||||
.first()
|
||||
.cloned()
|
||||
.unwrap_or_else(|| "remote-operation-failed".to_string())
|
||||
});
|
||||
let peer = peer.clone();
|
||||
let path = path.to_string();
|
||||
update_site_replication_state(move |state| {
|
||||
match failure.as_deref() {
|
||||
Some(error) => upsert_site_replication_retry_event(&mut state.retry_queue, &peer, &path, error, None),
|
||||
None => {
|
||||
dequeue_site_replication_retry_events_including_escalated(&mut state.retry_queue, &peer, &path);
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
pub(crate) fn admit_site_replication_repair_operation(
|
||||
repair_state: &mut SiteReplicationRepairState,
|
||||
operation_id: String,
|
||||
supplied_token: &str,
|
||||
candidate: SiteReplicationRepairOperation,
|
||||
) -> S3Result<SiteReplicationRepairOperation> {
|
||||
if let Some(existing) = repair_state.operations.get(&operation_id) {
|
||||
if !constant_time_eq(&existing.preflight_token, supplied_token) {
|
||||
return Err(S3Error::with_message(
|
||||
S3ErrorCode::ClientTokenConflict,
|
||||
"repair operation ID is already bound to a different preflight".to_string(),
|
||||
));
|
||||
}
|
||||
if !constant_time_eq(&existing.plan_token, &candidate.plan_token) {
|
||||
return Err(S3Error::with_message(
|
||||
S3ErrorCode::PreconditionFailed,
|
||||
"site replication repair plan changed after partial execution".to_string(),
|
||||
));
|
||||
}
|
||||
return Ok(existing.clone());
|
||||
}
|
||||
if repair_state
|
||||
.operations
|
||||
.values()
|
||||
.any(|operation| operation.status == "running")
|
||||
{
|
||||
return Err(S3Error::with_message(
|
||||
S3ErrorCode::ClientTokenConflict,
|
||||
"another site replication repair is active".to_string(),
|
||||
));
|
||||
}
|
||||
repair_state.operations.insert(operation_id, candidate.clone());
|
||||
prune_site_replication_repair_operations(&mut repair_state.operations);
|
||||
Ok(candidate)
|
||||
}
|
||||
|
||||
pub(crate) async fn execute_site_replication_repair(
|
||||
request: SiteReplicationRepairExecutionRequest,
|
||||
) -> S3Result<S3Response<(StatusCode, Body)>> {
|
||||
let store =
|
||||
current_object_store_handle().ok_or_else(|| S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()))?;
|
||||
with_config_object_write_lock(store, SITE_REPLICATION_REPAIR_EXECUTION_LOCK_PATH.to_string(), move || async move {
|
||||
execute_site_replication_repair_locked(request).await
|
||||
})
|
||||
.await
|
||||
.map_err(|_| {
|
||||
S3Error::with_message(S3ErrorCode::ClientTokenConflict, "another site replication repair is active".to_string())
|
||||
})?
|
||||
}
|
||||
|
||||
pub(crate) async fn execute_site_replication_repair_locked(
|
||||
request: SiteReplicationRepairExecutionRequest,
|
||||
) -> S3Result<S3Response<(StatusCode, Body)>> {
|
||||
let state = load_site_replication_state().await?;
|
||||
if !state.enabled() || state.service_account_access_key.is_empty() {
|
||||
return Err(s3_error!(InvalidRequest, "site replication is not configured"));
|
||||
}
|
||||
let info = build_sr_info(&state, &request.local_peer).await?;
|
||||
let plan = site_replication_bootstrap_plan(&info)?;
|
||||
let plan_token = site_replication_repair_plan_token(&state, &plan)?;
|
||||
let preflight_token = site_replication_repair_preflight_token(&state, &plan, request.signing_key.as_bytes())?;
|
||||
let sites = site_replication_repair_sites(&state, &request.local_peer, &plan, request.signing_key.as_bytes())?;
|
||||
|
||||
let repair_state = read_site_replication_repair_state().await?;
|
||||
if let Some(existing) = repair_state.operations.get(&request.operation_id) {
|
||||
if !constant_time_eq(&existing.preflight_token, &request.preflight_token) {
|
||||
return Err(S3Error::with_message(
|
||||
S3ErrorCode::ClientTokenConflict,
|
||||
"repair operation ID is already bound to a different preflight".to_string(),
|
||||
));
|
||||
}
|
||||
if existing.status == "success" {
|
||||
return json_response(StatusCode::OK, &site_replication_repair_operation_response(existing));
|
||||
}
|
||||
if !constant_time_eq(&existing.plan_token, &plan_token) {
|
||||
return Err(S3Error::with_message(
|
||||
S3ErrorCode::PreconditionFailed,
|
||||
"site replication repair plan changed after partial execution".to_string(),
|
||||
));
|
||||
}
|
||||
} else if !constant_time_eq(&request.preflight_token, &preflight_token) {
|
||||
return Err(S3Error::with_message(
|
||||
S3ErrorCode::PreconditionFailed,
|
||||
"site replication repair preflight is stale".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
let now = OffsetDateTime::now_utc();
|
||||
let candidate = SiteReplicationRepairOperation {
|
||||
operation_id: request.operation_id.clone(),
|
||||
preflight_token,
|
||||
plan_token,
|
||||
status: "running".to_string(),
|
||||
sites,
|
||||
created_at: Some(now),
|
||||
updated_at: Some(now),
|
||||
completed_at: None,
|
||||
};
|
||||
let supplied_token = request.preflight_token;
|
||||
let operation_id = request.operation_id;
|
||||
let mut operation = update_site_replication_repair_state(move |repair_state| {
|
||||
admit_site_replication_repair_operation(repair_state, operation_id, &supplied_token, candidate)
|
||||
})
|
||||
.await?;
|
||||
if operation.status == "success" {
|
||||
return json_response(StatusCode::OK, &site_replication_repair_operation_response(&operation));
|
||||
}
|
||||
|
||||
let service_account_secret_key = site_replicator_service_account_secret(&state.service_account_access_key).await?;
|
||||
prepare_site_replication_repair_retry(&mut operation);
|
||||
operation.status = "running".to_string();
|
||||
operation.completed_at = None;
|
||||
operation.updated_at = Some(OffsetDateTime::now_utc());
|
||||
persist_site_replication_repair_operation(&operation).await?;
|
||||
|
||||
let tasks = site_replication_repair_tasks(&plan);
|
||||
for peer in state.peers.values().filter(|peer| {
|
||||
peer.deployment_id != request.local_peer.deployment_id
|
||||
&& !same_identity_endpoint(&peer.endpoint, &request.local_peer.endpoint)
|
||||
}) {
|
||||
let transport = match PeerTransport::for_runtime_peer(peer).await {
|
||||
Ok(transport) => transport,
|
||||
Err(err) => {
|
||||
let error = err.to_string();
|
||||
for (family_index, task) in &tasks {
|
||||
if !site_replication_repair_task_pending(&operation, &peer.deployment_id, task.family(), *family_index)? {
|
||||
continue;
|
||||
}
|
||||
update_site_replication_repair_task(
|
||||
&mut operation,
|
||||
&peer.deployment_id,
|
||||
task.family(),
|
||||
*family_index,
|
||||
Err(&error),
|
||||
)?;
|
||||
summarize_site_replication_repair_operation(&mut operation);
|
||||
persist_site_replication_repair_task(&operation, peer, task.family(), task.path()).await?;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
};
|
||||
|
||||
for (family_index, task) in &tasks {
|
||||
if !site_replication_repair_task_pending(&operation, &peer.deployment_id, task.family(), *family_index)? {
|
||||
continue;
|
||||
}
|
||||
let result = task
|
||||
.send(&transport, &state.service_account_access_key, &service_account_secret_key)
|
||||
.await;
|
||||
let error = result.err().map(|err| err.to_string());
|
||||
update_site_replication_repair_task(
|
||||
&mut operation,
|
||||
&peer.deployment_id,
|
||||
task.family(),
|
||||
*family_index,
|
||||
match error.as_deref() {
|
||||
Some(error) => Err(error),
|
||||
None => Ok(()),
|
||||
},
|
||||
)?;
|
||||
summarize_site_replication_repair_operation(&mut operation);
|
||||
persist_site_replication_repair_task(&operation, peer, task.family(), task.path()).await?;
|
||||
}
|
||||
}
|
||||
|
||||
summarize_site_replication_repair_operation(&mut operation);
|
||||
persist_site_replication_repair_operation(&operation).await?;
|
||||
json_response(StatusCode::OK, &site_replication_repair_operation_response(&operation))
|
||||
}
|
||||
@@ -0,0 +1,928 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use super::*;
|
||||
|
||||
pub(crate) const SITE_REPLICATION_RETRY_QUEUE_LIMIT: usize = 256;
|
||||
|
||||
pub(crate) const SITE_REPLICATION_RETRY_FAILED_AFTER: u32 = 3;
|
||||
|
||||
pub(crate) const SITE_REPLICATION_ENDPOINT_REFRESH_RETRY_PATH: &str = "internal:endpoint-target-refresh";
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
pub(crate) struct SiteReplicationRetryEvent {
|
||||
pub(crate) id: String,
|
||||
pub(crate) peer_deployment_id: String,
|
||||
pub(crate) peer_endpoint: String,
|
||||
pub(crate) path: String,
|
||||
pub(crate) retry_count: u32,
|
||||
pub(crate) failed: bool,
|
||||
pub(crate) last_error: String,
|
||||
#[serde(default, with = "time::serde::rfc3339::option", skip_serializing_if = "Option::is_none")]
|
||||
pub(crate) updated_at: Option<OffsetDateTime>,
|
||||
/// Peer-edit generation whose delivery failed, when the failing send
|
||||
/// carried one. Settling a *later* success for the same (peer, path) must
|
||||
/// not erase a failure recorded for a NEWER generation — see
|
||||
/// [`settle_site_replication_retry_events`].
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub(crate) edit_generation: Option<u64>,
|
||||
}
|
||||
|
||||
pub(crate) fn retry_event_matches(event: &SiteReplicationRetryEvent, peer: &PeerInfo, path: &str) -> bool {
|
||||
(event.peer_deployment_id == peer.deployment_id || event.peer_endpoint == peer.endpoint) && event.path == path
|
||||
}
|
||||
|
||||
pub(crate) const SITE_REPLICATION_RETRY_IAM_SNAPSHOT_PATH: &str = "internal:retry-snapshot:iam";
|
||||
|
||||
pub(crate) const SITE_REPLICATION_RETRY_BUCKET_METADATA_SNAPSHOT_PATH: &str = "internal:retry-snapshot:bucket-metadata";
|
||||
|
||||
pub(crate) fn collapsed_retry_queue_path(path: &str) -> Option<&'static str> {
|
||||
let base_path = path.split_once('?').map(|(base, _)| base).unwrap_or(path);
|
||||
match base_path {
|
||||
"/rustfs/admin/v3/site-replication/peer/iam-item" | SITE_REPLICATION_RETRY_IAM_SNAPSHOT_PATH => {
|
||||
Some(SITE_REPLICATION_RETRY_IAM_SNAPSHOT_PATH)
|
||||
}
|
||||
"/rustfs/admin/v3/site-replication/peer/bucket-meta" | SITE_REPLICATION_RETRY_BUCKET_METADATA_SNAPSHOT_PATH => {
|
||||
Some(SITE_REPLICATION_RETRY_BUCKET_METADATA_SNAPSHOT_PATH)
|
||||
}
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn normalize_collapsed_retry_queue_paths(queue: &mut Vec<SiteReplicationRetryEvent>) -> bool {
|
||||
let mut changed = false;
|
||||
let mut normalized: Vec<SiteReplicationRetryEvent> = Vec::with_capacity(queue.len());
|
||||
for mut event in queue.drain(..) {
|
||||
if let Some(path) = collapsed_retry_queue_path(&event.path)
|
||||
&& event.path != path
|
||||
{
|
||||
event.path = path.to_string();
|
||||
changed = true;
|
||||
}
|
||||
|
||||
let duplicate = normalized.iter().position(|existing| {
|
||||
existing.path == event.path
|
||||
&& (existing.peer_deployment_id == event.peer_deployment_id || existing.peer_endpoint == event.peer_endpoint)
|
||||
});
|
||||
let Some(index) = duplicate else {
|
||||
normalized.push(event);
|
||||
continue;
|
||||
};
|
||||
|
||||
changed = true;
|
||||
let existing = &mut normalized[index];
|
||||
let event_is_newer = match (event.updated_at, existing.updated_at) {
|
||||
(Some(event), Some(existing)) => event >= existing,
|
||||
(Some(_), None) => true,
|
||||
_ => false,
|
||||
};
|
||||
if event_is_newer {
|
||||
let retry_count = existing.retry_count.max(event.retry_count);
|
||||
*existing = event;
|
||||
existing.retry_count = retry_count;
|
||||
} else {
|
||||
existing.retry_count = existing.retry_count.max(event.retry_count);
|
||||
}
|
||||
existing.failed = existing.retry_count >= SITE_REPLICATION_RETRY_FAILED_AFTER;
|
||||
}
|
||||
*queue = normalized;
|
||||
changed
|
||||
}
|
||||
|
||||
pub(crate) async fn migrate_collapsed_retry_queue_paths() -> S3Result<()> {
|
||||
update_site_replication_state_when_changed(|state| {
|
||||
Ok(if normalize_collapsed_retry_queue_paths(&mut state.retry_queue) {
|
||||
StateCommit::Changed(())
|
||||
} else {
|
||||
StateCommit::Unchanged(())
|
||||
})
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) fn dequeue_site_replication_retry_events(
|
||||
queue: &mut Vec<SiteReplicationRetryEvent>,
|
||||
peer: &PeerInfo,
|
||||
path: &str,
|
||||
) -> usize {
|
||||
settle_site_replication_retry_events(queue, peer, path, None)
|
||||
}
|
||||
|
||||
/// Repair-path settlement: also clears snapshot-escalated entries. Running a
|
||||
/// repair is the operator's explicit accountability transfer for the
|
||||
/// possibly-unreplayed deletion the marker records; ordinary delivery
|
||||
/// successes must not clear it (see [`settle_site_replication_retry_events`]).
|
||||
pub(crate) fn dequeue_site_replication_retry_events_including_escalated(
|
||||
queue: &mut Vec<SiteReplicationRetryEvent>,
|
||||
peer: &PeerInfo,
|
||||
path: &str,
|
||||
) -> usize {
|
||||
let before = queue.len();
|
||||
let collapsed_path = collapsed_retry_queue_path(path);
|
||||
queue.retain(|event| {
|
||||
!retry_event_matches(event, peer, path)
|
||||
&& !collapsed_path.is_some_and(|collapsed_path| retry_event_matches(event, peer, collapsed_path))
|
||||
});
|
||||
before.saturating_sub(queue.len())
|
||||
}
|
||||
|
||||
/// Remove the retry events for (peer, path) that `generation` is entitled to
|
||||
/// settle. A successful delivery only proves the peer reached the state the
|
||||
/// delivery carried: while it was in flight another edit can commit, fail its
|
||||
/// own delivery, and enqueue for the same (peer, path). Erasing that event
|
||||
/// would leave the peer on the older edit with no retry left, so an event
|
||||
/// stamped with a NEWER generation survives. `None` settles unconditionally —
|
||||
/// the broadcast paths that carry no generation, whose retry events live under
|
||||
/// their own paths and never collide with peer-edit deliveries.
|
||||
pub(crate) fn settle_site_replication_retry_events(
|
||||
queue: &mut Vec<SiteReplicationRetryEvent>,
|
||||
peer: &PeerInfo,
|
||||
path: &str,
|
||||
generation: Option<u64>,
|
||||
) -> usize {
|
||||
let before = queue.len();
|
||||
let collapsed_path = collapsed_retry_queue_path(path);
|
||||
queue.retain(|event| {
|
||||
if !retry_event_matches(event, peer, path) {
|
||||
return true;
|
||||
}
|
||||
// A wire-path success identifies no IAM or bucket-metadata entity.
|
||||
// This also protects legacy rows until the startup migration moves
|
||||
// them under their internal snapshot path.
|
||||
if collapsed_path.is_some() {
|
||||
return true;
|
||||
}
|
||||
// A snapshot-escalated entry records a possibly-unreplayed deletion.
|
||||
// Collapsed paths are shared by every entity, so a later successful
|
||||
// delivery of a DIFFERENT item proves nothing about the deleted one —
|
||||
// only a repair settles it (dequeue_..._including_escalated).
|
||||
if event.last_error == SITE_REPLICATION_RETRY_SNAPSHOT_REPLAYED_MARKER {
|
||||
return true;
|
||||
}
|
||||
match (generation, event.edit_generation) {
|
||||
(Some(settled), Some(failed)) => failed > settled,
|
||||
_ => false,
|
||||
}
|
||||
});
|
||||
before.saturating_sub(queue.len())
|
||||
}
|
||||
|
||||
pub(crate) fn upsert_site_replication_retry_event(
|
||||
queue: &mut Vec<SiteReplicationRetryEvent>,
|
||||
peer: &PeerInfo,
|
||||
path: &str,
|
||||
error: &str,
|
||||
generation: Option<u64>,
|
||||
) {
|
||||
let path = collapsed_retry_queue_path(path).unwrap_or(path);
|
||||
let now = OffsetDateTime::now_utc();
|
||||
let detail = summarize_peer_error_detail(error);
|
||||
if let Some(event) = queue.iter_mut().find(|event| retry_event_matches(event, peer, path)) {
|
||||
event.retry_count = event.retry_count.saturating_add(1);
|
||||
event.failed = event.retry_count >= SITE_REPLICATION_RETRY_FAILED_AFTER;
|
||||
event.last_error = detail;
|
||||
event.updated_at = Some(now);
|
||||
// Keep the newest generation: an older delivery that fails afterwards
|
||||
// must not lower the fence and let its own success settle the event.
|
||||
event.edit_generation = event.edit_generation.max(generation);
|
||||
return;
|
||||
}
|
||||
|
||||
queue.push(SiteReplicationRetryEvent {
|
||||
id: Uuid::new_v4().to_string(),
|
||||
peer_deployment_id: peer.deployment_id.clone(),
|
||||
peer_endpoint: peer.endpoint.clone(),
|
||||
path: path.to_string(),
|
||||
retry_count: 1,
|
||||
failed: false,
|
||||
last_error: detail,
|
||||
updated_at: Some(now),
|
||||
edit_generation: generation,
|
||||
});
|
||||
if queue.len() > SITE_REPLICATION_RETRY_QUEUE_LIMIT {
|
||||
let overflow = queue.len() - SITE_REPLICATION_RETRY_QUEUE_LIMIT;
|
||||
queue.drain(0..overflow);
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn retry_stats_for_state(state: &SiteReplicationState) -> Option<SRRetryStats> {
|
||||
if state.retry_queue.is_empty() {
|
||||
return None;
|
||||
}
|
||||
|
||||
Some(SRRetryStats {
|
||||
pending: state.retry_queue.iter().filter(|event| !event.failed).count(),
|
||||
failed: state.retry_queue.iter().filter(|event| event.failed).count(),
|
||||
last_error: state
|
||||
.retry_queue
|
||||
.iter()
|
||||
.rev()
|
||||
.find_map(|event| (!event.last_error.is_empty()).then(|| event.last_error.clone()))
|
||||
.unwrap_or_default(),
|
||||
api_version: Some(SITE_REPL_API_VERSION.to_string()),
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) async fn enqueue_site_replication_retry_event(peer: &PeerInfo, path: &str, error: &S3Error) {
|
||||
enqueue_site_replication_retry_event_for_generation(peer, path, error, None).await
|
||||
}
|
||||
|
||||
pub(crate) async fn enqueue_site_replication_retry_event_for_generation(
|
||||
peer: &PeerInfo,
|
||||
path: &str,
|
||||
error: &S3Error,
|
||||
generation: Option<u64>,
|
||||
) {
|
||||
let peer_owned = peer.clone();
|
||||
let path_owned = path.to_string();
|
||||
let error_text = error.to_string();
|
||||
let result = update_site_replication_state(move |state| {
|
||||
// A peer that left the state can never drain its entries again
|
||||
// (remove_sites already pruned them); recording a late failure for it
|
||||
// would only pollute retry_stats until the queue cap evicts it.
|
||||
if state.peers.contains_key(&peer_owned.deployment_id) {
|
||||
upsert_site_replication_retry_event(&mut state.retry_queue, &peer_owned, &path_owned, &error_text, generation);
|
||||
}
|
||||
Ok(())
|
||||
})
|
||||
.await;
|
||||
|
||||
if let Err(err) = result {
|
||||
warn!(
|
||||
component = LOG_COMPONENT_ADMIN,
|
||||
subsystem = LOG_SUBSYSTEM_SITE_REPLICATION,
|
||||
event = EVENT_ADMIN_SITE_REPLICATION_STATE,
|
||||
peer = %peer.endpoint,
|
||||
path,
|
||||
error = ?err,
|
||||
"failed to persist site replication retry event"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn retry_bucket_operation(path: &str) -> Option<String> {
|
||||
let (base_path, query) = path.split_once('?')?;
|
||||
if base_path != SITE_REPLICATION_PEER_BUCKET_OPS_PATH {
|
||||
return None;
|
||||
}
|
||||
|
||||
form_urlencoded::parse(query.as_bytes()).find_map(|(key, value)| (key == "operation").then(|| value.into_owned()))
|
||||
}
|
||||
|
||||
pub(crate) fn retry_event_replayed_by_bootstrap(event: &SiteReplicationRetryEvent) -> bool {
|
||||
matches!(
|
||||
retry_bucket_operation(&event.path).as_deref(),
|
||||
Some(SITE_REPLICATION_BUCKET_OP_MAKE_WITH_VERSIONING | SITE_REPLICATION_BUCKET_OP_CONFIGURE_REPLICATION)
|
||||
)
|
||||
}
|
||||
|
||||
/// Exponential backoff base for the background retry drain, aligned with the
|
||||
/// reconcile cadence (`site_replication_reconcile::RECONCILE_INTERVAL`).
|
||||
pub(crate) const SITE_REPLICATION_RETRY_DRAIN_BASE_BACKOFF_SECS: i64 = 600;
|
||||
|
||||
/// Backoff ceiling: a permanently failed peer is still probed daily.
|
||||
pub(crate) const SITE_REPLICATION_RETRY_DRAIN_MAX_BACKOFF_SECS: i64 = 86_400;
|
||||
|
||||
/// What the background drain may do for one retry event. Everything not
|
||||
/// representable here is operator territory (manual repair).
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub(crate) enum RetryDrainAction {
|
||||
/// Constant-path IAM item deliveries collapse into one queue entry per
|
||||
/// peer and their bodies are not persisted; the only faithful replay is
|
||||
/// the current IAM snapshot from the bootstrap plan.
|
||||
IamSnapshot,
|
||||
/// Same collapse for bucket-meta deliveries: replay the bucket metadata
|
||||
/// snapshot from the bootstrap plan.
|
||||
BucketMetadataSnapshot,
|
||||
/// A self-contained bucket op the bootstrap plan can re-derive for its
|
||||
/// bucket (`make-with-versioning` / `configure-replication`).
|
||||
BucketOpReplay { operation: String, bucket: String },
|
||||
/// Re-send the current peer records under a fresh edit generation.
|
||||
PeerEdit,
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
pub(crate) enum RetrySnapshot {
|
||||
Iam(Vec<SRIAMItem>),
|
||||
BucketMetadata(Vec<SRBucketMeta>),
|
||||
}
|
||||
|
||||
impl RetrySnapshot {
|
||||
pub(crate) fn from_plan(action: &RetryDrainAction, plan: &SiteReplicationBootstrapPlan) -> Option<Self> {
|
||||
match action {
|
||||
RetryDrainAction::IamSnapshot => Some(Self::Iam(plan.iam_items.clone())),
|
||||
RetryDrainAction::BucketMetadataSnapshot => Some(Self::BucketMetadata(plan.bucket_items.clone())),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn fingerprint(&self) -> S3Result<Vec<Vec<u8>>> {
|
||||
let mut payloads = match self {
|
||||
Self::Iam(items) => items.iter().map(serde_json::to_vec).collect::<Result<Vec<_>, _>>(),
|
||||
Self::BucketMetadata(items) => items.iter().map(serde_json::to_vec).collect::<Result<Vec<_>, _>>(),
|
||||
}
|
||||
.map_err(|err| S3Error::with_message(S3ErrorCode::InternalError, format!("serialize retry snapshot failed: {err}")))?;
|
||||
payloads.sort_unstable();
|
||||
Ok(payloads)
|
||||
}
|
||||
|
||||
pub(crate) fn replay_after_change(previous: &Self, fresh: &Self, observed_at: OffsetDateTime) -> Self {
|
||||
match (previous, fresh) {
|
||||
(Self::Iam(previous), Self::Iam(fresh)) => {
|
||||
let fresh_keys: HashSet<IamSnapshotKey> = fresh.iter().filter_map(iam_snapshot_key).collect();
|
||||
let mut replay = fresh.clone();
|
||||
for item in previous {
|
||||
if iam_snapshot_key(item).is_some_and(|key| !fresh_keys.contains(&key)) {
|
||||
replay.extend(iam_snapshot_tombstones(item, observed_at));
|
||||
}
|
||||
}
|
||||
Self::Iam(replay)
|
||||
}
|
||||
(Self::BucketMetadata(previous), Self::BucketMetadata(fresh)) => {
|
||||
let fresh_keys: HashSet<(&str, &str)> = fresh
|
||||
.iter()
|
||||
.map(|item| (item.bucket.as_str(), item.r#type.as_str()))
|
||||
.collect();
|
||||
let mut replay = fresh.clone();
|
||||
for item in previous {
|
||||
if !fresh_keys.contains(&(item.bucket.as_str(), item.r#type.as_str())) {
|
||||
replay.push(bucket_metadata_snapshot_tombstone(item, observed_at));
|
||||
}
|
||||
}
|
||||
Self::BucketMetadata(replay)
|
||||
}
|
||||
_ => fresh.clone(),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) async fn send(&self, transport: &PeerTransport, access_key: &str, secret_key: &str) -> S3Result<()> {
|
||||
match self {
|
||||
Self::Iam(items) => {
|
||||
for item in items {
|
||||
SiteReplicationRepairTask::Iam(item)
|
||||
.send(transport, access_key, secret_key)
|
||||
.await?;
|
||||
}
|
||||
}
|
||||
Self::BucketMetadata(items) => {
|
||||
for item in items {
|
||||
SiteReplicationRepairTask::BucketMetadata(item)
|
||||
.send(transport, access_key, secret_key)
|
||||
.await?;
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Hash, PartialEq, Eq)]
|
||||
pub(crate) enum IamSnapshotKey {
|
||||
Policy(String),
|
||||
User(String),
|
||||
Group(String),
|
||||
PolicyMapping { target: String, user_type: i64, is_group: bool },
|
||||
}
|
||||
|
||||
pub(crate) fn iam_snapshot_key(item: &SRIAMItem) -> Option<IamSnapshotKey> {
|
||||
match item.r#type.as_str() {
|
||||
"policy" => Some(IamSnapshotKey::Policy(item.name.clone())),
|
||||
"iam-user" => item
|
||||
.iam_user
|
||||
.as_ref()
|
||||
.map(|user| IamSnapshotKey::User(user.access_key.clone())),
|
||||
"group-info" => item
|
||||
.group_info
|
||||
.as_ref()
|
||||
.map(|group| IamSnapshotKey::Group(group.update_req.group.clone())),
|
||||
"policy-mapping" => item.policy_mapping.as_ref().map(|mapping| IamSnapshotKey::PolicyMapping {
|
||||
target: mapping.user_or_group.clone(),
|
||||
user_type: mapping.user_type,
|
||||
is_group: mapping.is_group,
|
||||
}),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn iam_snapshot_tombstones(item: &SRIAMItem, observed_at: OffsetDateTime) -> Vec<SRIAMItem> {
|
||||
let mut tombstone = item.clone();
|
||||
tombstone.updated_at = Some(observed_at);
|
||||
match item.r#type.as_str() {
|
||||
"policy" => tombstone.policy = None,
|
||||
"iam-user" => {
|
||||
if let Some(user) = tombstone.iam_user.as_mut() {
|
||||
user.is_delete_req = true;
|
||||
user.user_req = None;
|
||||
}
|
||||
}
|
||||
"group-info" => {
|
||||
let Some(group) = tombstone.group_info.as_mut() else {
|
||||
return Vec::new();
|
||||
};
|
||||
group.update_req.is_remove = true;
|
||||
if group.update_req.members.is_empty() {
|
||||
return vec![tombstone];
|
||||
}
|
||||
let mut delete = tombstone.clone();
|
||||
if let Some(group) = delete.group_info.as_mut() {
|
||||
group.update_req.members.clear();
|
||||
}
|
||||
return vec![tombstone, delete];
|
||||
}
|
||||
"policy-mapping" => {
|
||||
if let Some(mapping) = tombstone.policy_mapping.as_mut() {
|
||||
mapping.policy.clear();
|
||||
}
|
||||
}
|
||||
_ => return Vec::new(),
|
||||
}
|
||||
vec![tombstone]
|
||||
}
|
||||
|
||||
pub(crate) fn bucket_metadata_snapshot_tombstone(item: &SRBucketMeta, observed_at: OffsetDateTime) -> SRBucketMeta {
|
||||
SRBucketMeta {
|
||||
r#type: item.r#type.clone(),
|
||||
bucket: item.bucket.clone(),
|
||||
updated_at: Some(observed_at),
|
||||
expiry_updated_at: Some(observed_at),
|
||||
api_version: item.api_version.clone(),
|
||||
derived_rule_contract: item.derived_rule_contract,
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) const SITE_REPLICATION_RETRY_SNAPSHOT_STABILITY_ATTEMPTS: usize = 3;
|
||||
|
||||
pub(crate) fn classify_site_replication_retry_event(event: &SiteReplicationRetryEvent) -> Option<RetryDrainAction> {
|
||||
let snapshot_action = match event.path.as_str() {
|
||||
SITE_REPLICATION_RETRY_IAM_SNAPSHOT_PATH => Some(RetryDrainAction::IamSnapshot),
|
||||
SITE_REPLICATION_RETRY_BUCKET_METADATA_SNAPSHOT_PATH => Some(RetryDrainAction::BucketMetadataSnapshot),
|
||||
_ => None,
|
||||
};
|
||||
if snapshot_action.is_some() && event.last_error != SITE_REPLICATION_RETRY_SNAPSHOT_REPLAYED_MARKER {
|
||||
return snapshot_action;
|
||||
}
|
||||
if event.path.starts_with("internal:") {
|
||||
// Marker records store payloads in `last_error` (legacy
|
||||
// pending-endpoint-refresh backup and snapshot liabilities); they are
|
||||
// not drainable delivery failures.
|
||||
return None;
|
||||
}
|
||||
if event.last_error == SITE_REPLICATION_RETRY_SNAPSHOT_REPLAYED_MARKER {
|
||||
// Already snapshot-replayed once for this failure episode; a possible
|
||||
// deletion cannot be replayed from a snapshot, so re-sending daily
|
||||
// proves nothing. A new hook failure overwrites the marker.
|
||||
return None;
|
||||
}
|
||||
let base_path = event.path.split_once('?').map(|(base, _)| base).unwrap_or(&event.path);
|
||||
match base_path {
|
||||
"/rustfs/admin/v3/site-replication/peer/iam-item" => Some(RetryDrainAction::IamSnapshot),
|
||||
"/rustfs/admin/v3/site-replication/peer/bucket-meta" => Some(RetryDrainAction::BucketMetadataSnapshot),
|
||||
SITE_REPLICATION_PEER_EDIT_PATH => Some(RetryDrainAction::PeerEdit),
|
||||
SITE_REPLICATION_PEER_BUCKET_OPS_PATH => {
|
||||
let operation = retry_bucket_operation(&event.path)?;
|
||||
if !matches!(
|
||||
operation.as_str(),
|
||||
SITE_REPLICATION_BUCKET_OP_MAKE_WITH_VERSIONING | SITE_REPLICATION_BUCKET_OP_CONFIGURE_REPLICATION
|
||||
) {
|
||||
// Destructive ops (delete-bucket / force-delete-bucket) are
|
||||
// operator territory: replaying them against a peer whose
|
||||
// bucket was since recreated is irreversible.
|
||||
return None;
|
||||
}
|
||||
let bucket = retry_bucket_name(&event.path)?;
|
||||
Some(RetryDrainAction::BucketOpReplay { operation, bucket })
|
||||
}
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn retry_bucket_name(path: &str) -> Option<String> {
|
||||
let (_, query) = path.split_once('?')?;
|
||||
form_urlencoded::parse(query.as_bytes())
|
||||
.find_map(|(key, value)| (key == "bucket" && !value.is_empty()).then(|| value.into_owned()))
|
||||
}
|
||||
|
||||
/// A collapsed retry event after a stable snapshot resend is escalated with
|
||||
/// this marker instead of being cleared: the snapshot contains no task for a
|
||||
/// failed deletion, so remote absence remains operator-visible. Collapsed
|
||||
/// failures use an internal queue path so ordinary successes and older nodes
|
||||
/// cannot settle an unrelated entity's liability.
|
||||
pub(crate) const SITE_REPLICATION_RETRY_SNAPSHOT_REPLAYED_MARKER: &str = "snapshot replayed; a failed deletion cannot be replayed from a snapshot — run site replication repair or re-deliver to settle";
|
||||
|
||||
/// Escalate a collapsed retry event after its snapshot resend succeeded,
|
||||
/// unless a newer failure was recorded after `snapshot_updated_at` (that
|
||||
/// failure belongs to a newer local commit the snapshot did not contain and
|
||||
/// must keep the entry drain-eligible).
|
||||
pub(crate) fn escalate_site_replication_retry_events_up_to(
|
||||
queue: &mut Vec<SiteReplicationRetryEvent>,
|
||||
peer: &PeerInfo,
|
||||
path: &str,
|
||||
snapshot_updated_at: Option<OffsetDateTime>,
|
||||
) -> usize {
|
||||
let Some(marker_path) = collapsed_retry_queue_path(path) else {
|
||||
return 0;
|
||||
};
|
||||
|
||||
if path != marker_path {
|
||||
queue.retain(|event| {
|
||||
if !retry_event_matches(event, peer, path) {
|
||||
return true;
|
||||
}
|
||||
matches!((event.updated_at, snapshot_updated_at), (Some(current), Some(seen)) if current > seen)
|
||||
|| matches!((event.updated_at, snapshot_updated_at), (Some(_), None))
|
||||
});
|
||||
}
|
||||
|
||||
let marker_index = queue.iter().position(|event| retry_event_matches(event, peer, marker_path));
|
||||
let marker_index = marker_index.unwrap_or_else(|| {
|
||||
queue.push(SiteReplicationRetryEvent {
|
||||
id: Uuid::new_v4().to_string(),
|
||||
peer_deployment_id: peer.deployment_id.clone(),
|
||||
peer_endpoint: peer.endpoint.clone(),
|
||||
path: marker_path.to_string(),
|
||||
updated_at: snapshot_updated_at,
|
||||
..Default::default()
|
||||
});
|
||||
queue.len() - 1
|
||||
});
|
||||
let event = &mut queue[marker_index];
|
||||
let newer_failure_recorded = match (event.updated_at, snapshot_updated_at) {
|
||||
(Some(current), Some(seen)) => current > seen,
|
||||
(Some(_), None) => true,
|
||||
(None, _) => false,
|
||||
};
|
||||
if newer_failure_recorded && event.last_error != SITE_REPLICATION_RETRY_SNAPSHOT_REPLAYED_MARKER {
|
||||
return 0;
|
||||
}
|
||||
event.failed = true;
|
||||
event.retry_count = event.retry_count.max(SITE_REPLICATION_RETRY_FAILED_AFTER);
|
||||
event.last_error = SITE_REPLICATION_RETRY_SNAPSHOT_REPLAYED_MARKER.to_string();
|
||||
event.updated_at = Some(OffsetDateTime::now_utc());
|
||||
1
|
||||
}
|
||||
|
||||
pub(crate) async fn escalate_site_replication_retry_event_up_to(
|
||||
peer: &PeerInfo,
|
||||
path: &str,
|
||||
snapshot_updated_at: Option<OffsetDateTime>,
|
||||
) {
|
||||
let peer_owned = peer.clone();
|
||||
let path_owned = path.to_string();
|
||||
let result = update_site_replication_state(move |state| {
|
||||
escalate_site_replication_retry_events_up_to(&mut state.retry_queue, &peer_owned, &path_owned, snapshot_updated_at);
|
||||
Ok(())
|
||||
})
|
||||
.await;
|
||||
|
||||
if let Err(err) = result {
|
||||
warn!(
|
||||
component = LOG_COMPONENT_ADMIN,
|
||||
subsystem = LOG_SUBSYSTEM_SITE_REPLICATION,
|
||||
event = EVENT_ADMIN_SITE_REPLICATION_STATE,
|
||||
peer = %peer.endpoint,
|
||||
deployment_id = %peer.deployment_id,
|
||||
path,
|
||||
error = ?err,
|
||||
"failed to escalate site replication retry event"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether the drain may attempt this event now.
|
||||
pub(crate) fn site_replication_retry_backoff_elapsed(event: &SiteReplicationRetryEvent, now: OffsetDateTime) -> bool {
|
||||
let Some(updated_at) = event.updated_at else {
|
||||
return true;
|
||||
};
|
||||
// 600 * 2^8 already exceeds the daily ceiling; capping the shift keeps
|
||||
// the arithmetic overflow-free for any persisted retry_count.
|
||||
let exponent = event.retry_count.saturating_sub(1).min(8);
|
||||
let delay = (SITE_REPLICATION_RETRY_DRAIN_BASE_BACKOFF_SECS << exponent).min(SITE_REPLICATION_RETRY_DRAIN_MAX_BACKOFF_SECS);
|
||||
now.unix_timestamp().saturating_sub(updated_at.unix_timestamp()) >= delay
|
||||
}
|
||||
|
||||
/// The subset of the retry queue the background drain is allowed to touch.
|
||||
pub(crate) fn actionable_site_replication_retry_events(
|
||||
state: &SiteReplicationState,
|
||||
now: OffsetDateTime,
|
||||
) -> Vec<SiteReplicationRetryEvent> {
|
||||
state
|
||||
.retry_queue
|
||||
.iter()
|
||||
.filter(|event| classify_site_replication_retry_event(event).is_some())
|
||||
.filter(|event| state.peers.contains_key(&event.peer_deployment_id))
|
||||
.filter(|event| site_replication_retry_backoff_elapsed(event, now))
|
||||
.cloned()
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Background consumer for the retry queue, run from the reconcile tick.
|
||||
///
|
||||
/// Scope: this settles "delivered once and failed" entries whose replay is
|
||||
/// faithful (bucket ops, peer edits). Collapsed iam-item / bucket-meta
|
||||
/// entries are snapshot-resent and then *escalated*, not cleared — a failed
|
||||
/// deletion leaves no task in the snapshot, so remote absence stays unproven
|
||||
/// until a later delivery or a manual repair. A hook that never fired (crash
|
||||
/// between the local commit and the send) leaves no entry at all, so the
|
||||
/// drain is not a full cross-site diff-heal; manual repair remains the
|
||||
/// authoritative catch-all.
|
||||
pub(crate) async fn drain_site_replication_retry_queue() {
|
||||
if let Err(err) = drain_site_replication_retry_queue_inner().await {
|
||||
warn!(
|
||||
event = EVENT_ADMIN_SITE_REPLICATION_STATE,
|
||||
component = LOG_COMPONENT_ADMIN,
|
||||
subsystem = LOG_SUBSYSTEM_SITE_REPLICATION,
|
||||
result = "retry_drain_failed",
|
||||
error = ?err,
|
||||
"admin site replication state"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) async fn drain_site_replication_retry_queue_inner() -> S3Result<()> {
|
||||
let Some(runtime) = runtime_site_replication_targets().await? else {
|
||||
return Ok(());
|
||||
};
|
||||
let actionable = actionable_site_replication_retry_events(&runtime.state, OffsetDateTime::now_utc());
|
||||
if actionable.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
let Some(store) = current_object_store_handle() else {
|
||||
return Ok(());
|
||||
};
|
||||
if runtime.state.pending_endpoint_refresh.is_some()
|
||||
|| runtime.state.pending_remove.is_some()
|
||||
|| runtime.state.pending_rotation.is_some()
|
||||
{
|
||||
// The tick-level gate ran before the reconcilers; a multi-step flow
|
||||
// (endpoint refresh commits its pending marker without the lifecycle
|
||||
// guard) may have started since. Re-check on the fresh state.
|
||||
return Ok(());
|
||||
}
|
||||
// Serialize against operator repair execution. This does NOT close the
|
||||
// dry-run -> execute window (dry-run takes no lock): a drain settling a
|
||||
// replayable bucket-op entry in that window changes the preflight token
|
||||
// and execute fails safe with "preflight is stale" — the operator
|
||||
// re-runs the dry-run. Lock order matches repair: lifecycle guard (held
|
||||
// by the reconcile tick) -> repair execution lock -> state object lock
|
||||
// inside the send bookkeeping. An operator repair holding the lock makes
|
||||
// this tick skip after the lock-acquire timeout.
|
||||
with_config_object_write_lock(store, SITE_REPLICATION_REPAIR_EXECUTION_LOCK_PATH.to_string(), move || async move {
|
||||
drain_site_replication_retry_queue_locked(runtime, actionable).await
|
||||
})
|
||||
.await
|
||||
.map_err(ApiError::from)?
|
||||
}
|
||||
|
||||
pub(crate) async fn drain_site_replication_retry_queue_locked(
|
||||
runtime: SiteReplicationRuntime,
|
||||
events: Vec<SiteReplicationRetryEvent>,
|
||||
) -> S3Result<()> {
|
||||
let needs_plan = events
|
||||
.iter()
|
||||
.any(|event| !matches!(classify_site_replication_retry_event(event), Some(RetryDrainAction::PeerEdit)));
|
||||
// The plan is a full local snapshot (buckets + IAM); build it once per
|
||||
// tick and only when a snapshot resend is actually due.
|
||||
let plan = if needs_plan {
|
||||
let info = build_sr_info(&runtime.state, &runtime.local_peer).await?;
|
||||
Some(site_replication_bootstrap_plan(&info)?)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
let mut events_by_peer: BTreeMap<String, Vec<SiteReplicationRetryEvent>> = BTreeMap::new();
|
||||
for event in events {
|
||||
events_by_peer
|
||||
.entry(event.peer_deployment_id.clone())
|
||||
.or_default()
|
||||
.push(event);
|
||||
}
|
||||
|
||||
let mut settled = 0usize;
|
||||
let mut failures = 0usize;
|
||||
for (deployment_id, peer_events) in events_by_peer {
|
||||
let Some(peer) = runtime.state.peers.get(&deployment_id) else {
|
||||
continue;
|
||||
};
|
||||
if deployment_id == runtime.local_peer.deployment_id
|
||||
|| same_identity_endpoint(&peer.endpoint, &runtime.local_peer.endpoint)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
let transport = match PeerTransport::for_runtime_peer(peer).await {
|
||||
Ok(transport) => transport,
|
||||
Err(err) => {
|
||||
// Record the attempt so backoff advances for an unreachable
|
||||
// peer instead of re-dialing it every tick.
|
||||
for event in &peer_events {
|
||||
enqueue_site_replication_retry_event(peer, &event.path, &err).await;
|
||||
}
|
||||
failures += peer_events.len();
|
||||
continue;
|
||||
}
|
||||
};
|
||||
for event in peer_events {
|
||||
let Some(action) = classify_site_replication_retry_event(&event) else {
|
||||
continue;
|
||||
};
|
||||
match drain_one_site_replication_retry_event(&runtime, peer, &transport, &event, action, plan.as_ref()).await {
|
||||
Ok(true) => settled += 1,
|
||||
Ok(false) => {}
|
||||
Err(_) => failures += 1,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if settled > 0 || failures > 0 {
|
||||
info!(
|
||||
event = EVENT_ADMIN_SITE_REPLICATION_STATE,
|
||||
component = LOG_COMPONENT_ADMIN,
|
||||
subsystem = LOG_SUBSYSTEM_SITE_REPLICATION,
|
||||
result = "retry_drain_settled",
|
||||
settled,
|
||||
failures,
|
||||
"admin site replication state"
|
||||
);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Replay one retry event against its peer. Returns `Ok(true)` when the
|
||||
/// event was settled (delivered, or provably stale), `Ok(false)` when it was
|
||||
/// skipped, and `Err` after a failed delivery (already re-queued with an
|
||||
/// incremented retry count).
|
||||
pub(crate) async fn drain_one_site_replication_retry_event(
|
||||
runtime: &SiteReplicationRuntime,
|
||||
peer: &PeerInfo,
|
||||
transport: &PeerTransport,
|
||||
event: &SiteReplicationRetryEvent,
|
||||
action: RetryDrainAction,
|
||||
plan: Option<&SiteReplicationBootstrapPlan>,
|
||||
) -> S3Result<bool> {
|
||||
let access_key = &runtime.state.service_account_access_key;
|
||||
let secret_key = &runtime.service_account_secret_key;
|
||||
match action.clone() {
|
||||
RetryDrainAction::IamSnapshot | RetryDrainAction::BucketMetadataSnapshot => {
|
||||
let Some(plan) = plan else {
|
||||
return Ok(false);
|
||||
};
|
||||
let mut current_snapshot = RetrySnapshot::from_plan(&action, plan).expect("snapshot action has a snapshot");
|
||||
let mut replay = current_snapshot.clone();
|
||||
for _ in 0..SITE_REPLICATION_RETRY_SNAPSHOT_STABILITY_ATTEMPTS {
|
||||
let current_fingerprint = current_snapshot.fingerprint()?;
|
||||
if let Err(err) = replay.send(transport, access_key, secret_key).await {
|
||||
enqueue_site_replication_retry_event(peer, &event.path, &err).await;
|
||||
return Err(err);
|
||||
}
|
||||
let fresh_info = build_sr_info(&runtime.state, &runtime.local_peer).await?;
|
||||
let fresh_plan = site_replication_bootstrap_plan(&fresh_info)?;
|
||||
let fresh_snapshot = RetrySnapshot::from_plan(&action, &fresh_plan).expect("snapshot action has a snapshot");
|
||||
if fresh_snapshot.fingerprint()? == current_fingerprint {
|
||||
escalate_site_replication_retry_event_up_to(peer, &event.path, event.updated_at).await;
|
||||
return Ok(true);
|
||||
}
|
||||
replay = RetrySnapshot::replay_after_change(¤t_snapshot, &fresh_snapshot, OffsetDateTime::now_utc());
|
||||
current_snapshot = fresh_snapshot;
|
||||
}
|
||||
Ok(false)
|
||||
}
|
||||
RetryDrainAction::BucketOpReplay { operation, bucket } => {
|
||||
let Some(plan) = plan else {
|
||||
return Ok(false);
|
||||
};
|
||||
// Replay from the CURRENT plan, never the recorded path: the
|
||||
// recorded query can carry an expired one-shot bootstrap token or
|
||||
// a stale createdAt.
|
||||
let make_op = operation == SITE_REPLICATION_BUCKET_OP_MAKE_WITH_VERSIONING;
|
||||
let paths = if make_op {
|
||||
&plan.bucket_make_ops
|
||||
} else {
|
||||
&plan.bucket_configure_ops
|
||||
};
|
||||
let tasks: Vec<SiteReplicationRepairTask<'_>> = paths
|
||||
.iter()
|
||||
.filter(|path| retry_bucket_name(path).as_deref() == Some(bucket.as_str()))
|
||||
.map(|path| {
|
||||
if make_op {
|
||||
SiteReplicationRepairTask::BucketMake(path)
|
||||
} else {
|
||||
SiteReplicationRepairTask::Replication(path)
|
||||
}
|
||||
})
|
||||
.collect();
|
||||
if tasks.is_empty() {
|
||||
// The bucket left the plan (deleted, or replication no longer
|
||||
// configured): the recorded intent is stale, settle it.
|
||||
dequeue_site_replication_retry_event(peer, &event.path).await;
|
||||
return Ok(true);
|
||||
}
|
||||
for task in &tasks {
|
||||
if let Err(err) = task.send(transport, access_key, secret_key).await {
|
||||
enqueue_site_replication_retry_event(peer, &event.path, &err).await;
|
||||
return Err(err);
|
||||
}
|
||||
}
|
||||
dequeue_site_replication_retry_event(peer, &event.path).await;
|
||||
Ok(true)
|
||||
}
|
||||
RetryDrainAction::PeerEdit => {
|
||||
// The recorded generation is stale by definition — the receiver
|
||||
// fences it. Allocate a fresh generation and re-send the current
|
||||
// peer records (a superset of the failed body; the receiver
|
||||
// upserts), all inside one state transaction so the fence and the
|
||||
// bodies agree.
|
||||
let target_id = peer.deployment_id.clone();
|
||||
let (generation, bodies) = update_site_replication_state(move |state| {
|
||||
if !state.peers.contains_key(&target_id) {
|
||||
return Ok((None, Vec::new()));
|
||||
}
|
||||
Ok((Some(next_peer_edit_generation(state)), state.peers.values().cloned().collect::<Vec<_>>()))
|
||||
})
|
||||
.await?;
|
||||
let Some(generation) = generation else {
|
||||
// Peer left between the snapshot and now; the queue entry was
|
||||
// already pruned by remove_sites.
|
||||
return Ok(false);
|
||||
};
|
||||
let local_deployment_id = Some(runtime.local_peer.deployment_id.as_str()).filter(|id| !id.is_empty());
|
||||
let edit_path = peer_edit_path_with_fence(local_deployment_id, generation);
|
||||
let delivery_fence = local_deployment_id.is_some().then_some(generation);
|
||||
for body in &bodies {
|
||||
if let Err(err) = send_peer_admin_request_with_client(
|
||||
&transport.client,
|
||||
&transport.connection,
|
||||
&edit_path,
|
||||
access_key,
|
||||
secret_key,
|
||||
body,
|
||||
)
|
||||
.await
|
||||
{
|
||||
enqueue_site_replication_retry_event_for_generation(
|
||||
peer,
|
||||
SITE_REPLICATION_PEER_EDIT_PATH,
|
||||
&err,
|
||||
delivery_fence,
|
||||
)
|
||||
.await;
|
||||
return Err(err);
|
||||
}
|
||||
}
|
||||
dequeue_site_replication_retry_event_for_generation(peer, SITE_REPLICATION_PEER_EDIT_PATH, delivery_fence).await;
|
||||
Ok(true)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Remove a retry event for (peer, path) from the queue on successful delivery.
|
||||
/// This is a no-op (load + no-op persist skipped) when no matching entry exists,
|
||||
/// avoiding unnecessary I/O on the common path.
|
||||
pub(crate) async fn dequeue_site_replication_retry_event(peer: &PeerInfo, path: &str) {
|
||||
dequeue_site_replication_retry_event_for_generation(peer, path, None).await
|
||||
}
|
||||
|
||||
pub(crate) async fn dequeue_site_replication_retry_event_for_generation(peer: &PeerInfo, path: &str, generation: Option<u64>) {
|
||||
let result = async {
|
||||
// Fast path: this sits on every successful hook broadcast, so probe
|
||||
// with a plain read first and only enter the locked RMW on a hit
|
||||
// (the transaction re-checks under the lock).
|
||||
let mut probe = load_site_replication_state().await?;
|
||||
if settle_site_replication_retry_events(&mut probe.retry_queue, peer, path, generation) == 0 {
|
||||
return Ok(());
|
||||
}
|
||||
let peer_owned = peer.clone();
|
||||
let path_owned = path.to_string();
|
||||
update_site_replication_state(move |state| {
|
||||
settle_site_replication_retry_events(&mut state.retry_queue, &peer_owned, &path_owned, generation);
|
||||
Ok(())
|
||||
})
|
||||
.await?;
|
||||
Ok::<_, S3Error>(())
|
||||
}
|
||||
.await;
|
||||
|
||||
if let Err(err) = result {
|
||||
warn!(
|
||||
component = LOG_COMPONENT_ADMIN,
|
||||
subsystem = LOG_SUBSYSTEM_SITE_REPLICATION,
|
||||
event = EVENT_ADMIN_SITE_REPLICATION_STATE,
|
||||
peer = %peer.endpoint,
|
||||
deployment_id = %peer.deployment_id,
|
||||
path,
|
||||
error = ?err,
|
||||
"failed to dequeue site replication retry event"
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,593 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use super::*;
|
||||
|
||||
pub(crate) const SITE_REPLICATION_PEER_EDIT_PATH: &str = "/rustfs/admin/v3/site-replication/peer/edit";
|
||||
|
||||
/// Peer-edit fencing token, carried as query parameters so a peer that predates
|
||||
/// the fence simply ignores them (unknown query keys are dropped) and keeps the
|
||||
/// previous last-writer-wins behaviour.
|
||||
pub(crate) const SITE_REPLICATION_EDIT_ORIGIN_QUERY: &str = "editOrigin";
|
||||
|
||||
pub(crate) const SITE_REPLICATION_EDIT_GENERATION_QUERY: &str = "editGeneration";
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
pub(crate) struct SiteReplicationState {
|
||||
pub(crate) name: String,
|
||||
pub(crate) service_account_access_key: String,
|
||||
#[serde(default, skip_serializing)]
|
||||
pub(crate) service_account_secret_key: String,
|
||||
pub(crate) service_account_parent: String,
|
||||
pub(crate) peers: BTreeMap<String, PeerInfo>,
|
||||
pub(crate) updated_at: Option<OffsetDateTime>,
|
||||
pub(crate) resync_status: BTreeMap<String, SRResyncOpStatus>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub(crate) pending_rotation: Option<PendingRotation>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub(crate) pending_remove: Option<PendingRemove>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub(crate) pending_endpoint_refresh: Option<PendingEndpointRefresh>,
|
||||
#[serde(default, skip_serializing_if = "Vec::is_empty")]
|
||||
pub(crate) retry_queue: Vec<SiteReplicationRetryEvent>,
|
||||
#[serde(default)]
|
||||
pub(crate) sync_state_initialized: bool,
|
||||
/// Fencing token for peer-edit delivery, allocated inside the state
|
||||
/// transaction (the distributed state-object lock). Two nodes of THIS
|
||||
/// site that accept admin edits concurrently therefore get strictly
|
||||
/// ordered generations, and a delivery that stalls can be recognised as
|
||||
/// stale by the receiving site.
|
||||
#[serde(default)]
|
||||
pub(crate) edit_generation: u64,
|
||||
/// Per-origin high-water mark of the peer edits already applied here,
|
||||
/// keyed by the origin site's deployment id. A delivery whose generation
|
||||
/// is not above the mark arrived out of order and must not overwrite the
|
||||
/// newer edit that already landed.
|
||||
#[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
|
||||
pub(crate) applied_edit_generations: BTreeMap<String, u64>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
pub(crate) struct PendingEndpointRefresh {
|
||||
pub(crate) id: String,
|
||||
pub(crate) peer: PeerInfo,
|
||||
#[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
|
||||
pub(crate) remote_peers: BTreeMap<String, PeerInfo>,
|
||||
#[serde(default, skip_serializing_if = "BTreeSet::is_empty")]
|
||||
pub(crate) acked_deployment_ids: BTreeSet<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
pub(crate) struct PendingRotation {
|
||||
pub(crate) id: String,
|
||||
pub(crate) access_key: String,
|
||||
pub(crate) parent: String,
|
||||
pub(crate) new_secret_key: String,
|
||||
#[serde(default, skip_serializing_if = "Vec::is_empty")]
|
||||
pub(crate) secret_candidates: Vec<String>,
|
||||
#[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
|
||||
pub(crate) peers: BTreeMap<String, PeerInfo>,
|
||||
#[serde(default, skip_serializing_if = "BTreeSet::is_empty")]
|
||||
pub(crate) acked_deployment_ids: BTreeSet<String>,
|
||||
#[serde(default, with = "time::serde::rfc3339::option", skip_serializing_if = "Option::is_none")]
|
||||
pub(crate) updated_at: Option<OffsetDateTime>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
pub(crate) struct PendingRemove {
|
||||
pub(crate) id: String,
|
||||
pub(crate) req: SRRemoveReq,
|
||||
pub(crate) service_account_access_key: String,
|
||||
#[serde(default, skip_serializing_if = "Vec::is_empty")]
|
||||
pub(crate) secret_candidates: Vec<String>,
|
||||
#[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
|
||||
pub(crate) original_peers: BTreeMap<String, PeerInfo>,
|
||||
#[serde(default, skip_serializing_if = "BTreeSet::is_empty")]
|
||||
pub(crate) acked_deployment_ids: BTreeSet<String>,
|
||||
#[serde(default, with = "time::serde::rfc3339::option", skip_serializing_if = "Option::is_none")]
|
||||
pub(crate) updated_at: Option<OffsetDateTime>,
|
||||
}
|
||||
|
||||
impl SiteReplicationState {
|
||||
pub(crate) fn enabled(&self) -> bool {
|
||||
self.peers.len() > 1
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn parse_site_replication_state(data: &[u8]) -> S3Result<SiteReplicationState> {
|
||||
let mut state: SiteReplicationState = serde_json::from_slice(data)
|
||||
.map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, format!("invalid site replication state: {e}")))?;
|
||||
state.peers = normalize_peer_map_by_identity(state.peers);
|
||||
// A peer-edit high-water mark only fences a CURRENT peer. A site that
|
||||
// leaves drops below two peers, which clears its own state object and
|
||||
// restarts its generation counter — a mark left over from the previous
|
||||
// membership must not reject the edits it sends after it rejoins. This
|
||||
// pruning covers departures THIS site observed; an origin removed
|
||||
// unilaterally elsewhere stays in this peer map with its mark, and the
|
||||
// wall-clock floor in `next_peer_edit_generation` is what lifts its
|
||||
// restarted counter over that mark. Dropping departed origins on load
|
||||
// also keeps the map bounded.
|
||||
state
|
||||
.applied_edit_generations
|
||||
.retain(|origin, _| state.peers.contains_key(origin));
|
||||
if !state.sync_state_initialized {
|
||||
if state.enabled() {
|
||||
mark_unknown_peer_sync_enabled(&mut state.peers);
|
||||
}
|
||||
state.sync_state_initialized = true;
|
||||
}
|
||||
Ok(state)
|
||||
}
|
||||
|
||||
pub(crate) async fn load_site_replication_state() -> S3Result<SiteReplicationState> {
|
||||
let Some(store) = current_object_store_handle() else {
|
||||
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
|
||||
};
|
||||
|
||||
match read_admin_config(store, SITE_REPLICATION_STATE_PATH).await {
|
||||
Ok(data) => parse_site_replication_state(&data),
|
||||
Err(StorageError::ConfigNotFound) => Ok(SiteReplicationState::default()),
|
||||
Err(err) => Err(S3Error::with_message(
|
||||
S3ErrorCode::InternalError,
|
||||
format!("failed to load site replication state: {err}"),
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether this deployment participates in site replication (two or more
|
||||
/// peers in the persisted state). Read by the S3 interface layer to gate
|
||||
/// replication-config edits (MinIO `ErrReplicationDenyEditError` semantics,
|
||||
/// issue #1948); a state-read failure propagates so the gate fails closed.
|
||||
pub(crate) async fn site_replication_enabled() -> S3Result<bool> {
|
||||
Ok(load_site_replication_state().await?.enabled())
|
||||
}
|
||||
|
||||
/// Deployment ids of the remote peers the reconciler derives a
|
||||
/// `site-repl-<id>` rule for on every bucket (the same peer filter as
|
||||
/// `build_site_replication_config`); empty when site replication is not
|
||||
/// enabled. Read by the bucket usecase so an S3 replication-config edit keeps
|
||||
/// exactly the reconciler-owned rules (issue #1948); a state-read failure
|
||||
/// propagates so the edit fails closed.
|
||||
pub(crate) async fn site_replication_edit_context() -> S3Result<(HashSet<String>, OperatorRuleContract)> {
|
||||
let Some(runtime) = runtime_site_replication_targets().await? else {
|
||||
// Enabled without a service account is a state this site cannot
|
||||
// broadcast from either; the peers are still the reconciler's.
|
||||
let state = load_site_replication_state().await?;
|
||||
if !state.enabled() {
|
||||
return Ok((HashSet::new(), OperatorRuleContract::Derived));
|
||||
}
|
||||
let peers = remote_peer_deployment_ids(&state, ¤t_local_runtime_peer(&state));
|
||||
return Ok((peers, OperatorRuleContract::Legacy));
|
||||
};
|
||||
let peers = remote_peer_deployment_ids(&runtime.state, &runtime.local_peer);
|
||||
let contract = site_replication_operator_rule_contract(&runtime).await;
|
||||
Ok((peers, contract))
|
||||
}
|
||||
|
||||
/// Whether every remote peer merges replication configs under the derived
|
||||
/// contract, probed through the peer capability endpoint. A peer that does
|
||||
/// not (or cannot be asked) pins the cluster to [`OperatorRuleContract::Legacy`]
|
||||
/// for this edit: consistency across sites wins over keeping the operator's
|
||||
/// priority values, and the legacy merge keeps their order anyway.
|
||||
pub(crate) async fn site_replication_operator_rule_contract(runtime: &SiteReplicationRuntime) -> OperatorRuleContract {
|
||||
let remote_peers: Vec<&PeerInfo> = runtime
|
||||
.state
|
||||
.peers
|
||||
.values()
|
||||
.filter(|peer| {
|
||||
peer.deployment_id != runtime.local_peer.deployment_id
|
||||
&& !same_identity_endpoint(&peer.endpoint, &runtime.local_peer.endpoint)
|
||||
})
|
||||
.collect();
|
||||
let probes = futures::future::join_all(remote_peers.iter().map(|peer| async move {
|
||||
let transport = PeerTransport::for_runtime_peer(peer).await?;
|
||||
let (status, body) = send_peer_admin_request_raw_with_client(
|
||||
&transport.client,
|
||||
&transport.connection,
|
||||
SITE_REPLICATION_PEER_DERIVED_RULE_CONTRACT_CAPABILITY_PATH,
|
||||
&runtime.state.service_account_access_key,
|
||||
&runtime.service_account_secret_key,
|
||||
&(),
|
||||
)
|
||||
.await?;
|
||||
peer_capability_response_supported(peer, status, &body)
|
||||
}))
|
||||
.await;
|
||||
operator_rule_contract_from_probes(remote_peers.into_iter().zip(probes))
|
||||
}
|
||||
|
||||
pub(crate) fn operator_rule_contract_from_probes<'a>(
|
||||
probes: impl IntoIterator<Item = (&'a PeerInfo, S3Result<bool>)>,
|
||||
) -> OperatorRuleContract {
|
||||
for (peer, probe) in probes {
|
||||
match probe {
|
||||
Ok(true) => {}
|
||||
Ok(false) => return OperatorRuleContract::Legacy,
|
||||
Err(err) => {
|
||||
warn!(
|
||||
event = EVENT_ADMIN_SITE_REPLICATION_STATE,
|
||||
component = LOG_COMPONENT_ADMIN,
|
||||
subsystem = LOG_SUBSYSTEM_SITE_REPLICATION,
|
||||
result = "derived_rule_contract_probe_failed",
|
||||
peer = %peer.endpoint,
|
||||
error = %err,
|
||||
"admin site replication state"
|
||||
);
|
||||
return OperatorRuleContract::Legacy;
|
||||
}
|
||||
}
|
||||
}
|
||||
OperatorRuleContract::Derived
|
||||
}
|
||||
|
||||
pub(crate) fn remote_peer_deployment_ids(state: &SiteReplicationState, local_peer: &PeerInfo) -> HashSet<String> {
|
||||
state
|
||||
.peers
|
||||
.values()
|
||||
.filter(|peer| {
|
||||
peer.deployment_id != local_peer.deployment_id && !same_identity_endpoint(&peer.endpoint, &local_peer.endpoint)
|
||||
})
|
||||
.map(|peer| peer.deployment_id.clone())
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Deployment ids of every site in the cluster, this one included: the set
|
||||
/// a peer's derived rules can name (its rule towards this site carries this
|
||||
/// site's id). Empty when site replication is not enabled.
|
||||
pub(crate) async fn site_replication_deployment_ids() -> S3Result<HashSet<String>> {
|
||||
let state = load_site_replication_state().await?;
|
||||
if !state.enabled() {
|
||||
return Ok(HashSet::new());
|
||||
}
|
||||
Ok(state.peers.values().map(|peer| peer.deployment_id.clone()).collect())
|
||||
}
|
||||
|
||||
pub(crate) async fn load_site_replication_state_no_lock(store: Arc<ECStore>) -> S3Result<SiteReplicationState> {
|
||||
match read_config_no_lock(store, SITE_REPLICATION_STATE_PATH).await {
|
||||
Ok(data) => parse_site_replication_state(&data),
|
||||
Err(StorageError::ConfigNotFound) => Ok(SiteReplicationState::default()),
|
||||
Err(err) => Err(S3Error::with_message(
|
||||
S3ErrorCode::InternalError,
|
||||
format!("failed to load site replication state: {err}"),
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
/// Persist-or-clear under an already-held state object lock. Normalizes the
|
||||
/// peer map exactly once (the historical persist path normalized twice with
|
||||
/// two full clones — P2-22).
|
||||
pub(crate) async fn persist_site_replication_state_no_lock(store: Arc<ECStore>, mut state: SiteReplicationState) -> S3Result<()> {
|
||||
state.peers = normalize_peer_map_by_identity(state.peers);
|
||||
if state.peers.len() <= 1 && state.pending_rotation.is_none() && state.pending_remove.is_none() {
|
||||
match delete_config_no_lock(store, SITE_REPLICATION_STATE_PATH).await {
|
||||
Ok(()) | Err(StorageError::ConfigNotFound) => Ok(()),
|
||||
Err(err) => Err(S3Error::with_message(S3ErrorCode::InternalError, format!("clear state failed: {err}"))),
|
||||
}
|
||||
} else {
|
||||
let data = serde_json::to_vec(&state)
|
||||
.map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, format!("serialize state failed: {e}")))?;
|
||||
save_config_no_lock(store, SITE_REPLICATION_STATE_PATH, data)
|
||||
.await
|
||||
.map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, format!("save state failed: {e}")))
|
||||
}
|
||||
}
|
||||
|
||||
/// What a state transaction closure decided to do with the state it was
|
||||
/// handed. `Unchanged` skips the write entirely: the ack markers and the
|
||||
/// pending-clearing paths run on every retry and mostly find their pending id
|
||||
/// already gone, and the retry queue shares this object — rewriting it byte
|
||||
/// for byte only makes those misses contend with the writers that do have
|
||||
/// something to say.
|
||||
pub(crate) enum StateCommit<T> {
|
||||
Changed(T),
|
||||
Unchanged(T),
|
||||
}
|
||||
|
||||
/// The site-replication state RMW transaction: load, mutate, persist — all
|
||||
/// under the distributed state-object write lock (see
|
||||
/// crate::site_replication::state_lock). No peer network calls and no other
|
||||
/// config locks inside `update`; anything that has to talk to a peer belongs
|
||||
/// between two transactions, with the precondition re-checked inside the
|
||||
/// second one.
|
||||
pub(crate) async fn update_site_replication_state<T, F>(update: F) -> S3Result<T>
|
||||
where
|
||||
T: Send + 'static,
|
||||
F: FnOnce(&mut SiteReplicationState) -> S3Result<T> + Send + 'static,
|
||||
{
|
||||
update_site_replication_state_when_changed(move |state| update(state).map(StateCommit::Changed)).await
|
||||
}
|
||||
|
||||
/// [`update_site_replication_state`] for closures that may find nothing to
|
||||
/// do — see [`StateCommit`].
|
||||
pub(crate) async fn update_site_replication_state_when_changed<T, F>(update: F) -> S3Result<T>
|
||||
where
|
||||
T: Send + 'static,
|
||||
F: FnOnce(&mut SiteReplicationState) -> S3Result<StateCommit<T>> + Send + 'static,
|
||||
{
|
||||
with_site_replication_state_lock(move || async move {
|
||||
let store = current_object_store_handle()
|
||||
.ok_or_else(|| S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()))?;
|
||||
let mut state = load_site_replication_state_no_lock(store.clone()).await?;
|
||||
match update(&mut state)? {
|
||||
StateCommit::Changed(result) => {
|
||||
persist_site_replication_state_no_lock(store, state).await?;
|
||||
Ok(result)
|
||||
}
|
||||
StateCommit::Unchanged(result) => Ok(result),
|
||||
}
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
/// Test-only seeding of the state object. Every production write goes through
|
||||
/// [`update_site_replication_state`] — this helper is `cfg(test)` so a new
|
||||
/// call site cannot reintroduce the pre-P1-15 shape (load through one object
|
||||
/// lock, save through another, with the mutation in between unprotected).
|
||||
#[cfg(test)]
|
||||
pub(crate) async fn save_site_replication_state(state: &SiteReplicationState) -> S3Result<()> {
|
||||
let Some(store) = current_object_store_handle() else {
|
||||
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
|
||||
};
|
||||
|
||||
let mut normalized = state.clone();
|
||||
normalized.peers = normalize_peer_map_by_identity(normalized.peers);
|
||||
|
||||
let data = serde_json::to_vec(&normalized)
|
||||
.map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, format!("serialize state failed: {e}")))?;
|
||||
save_admin_config(store, SITE_REPLICATION_STATE_PATH, data)
|
||||
.await
|
||||
.map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, format!("save state failed: {e}")))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) fn request_endpoint(uri: &Uri, headers: &HeaderMap) -> String {
|
||||
let scheme = get_source_scheme(headers)
|
||||
.and_then(|value| {
|
||||
value
|
||||
.split(',')
|
||||
.next()
|
||||
.map(str::trim)
|
||||
.filter(|value| !value.is_empty())
|
||||
.map(str::to_ascii_lowercase)
|
||||
})
|
||||
.or_else(|| uri.scheme_str().map(str::to_ascii_lowercase))
|
||||
.unwrap_or_else(|| {
|
||||
if runtime_tls_enabled() {
|
||||
"https".to_string()
|
||||
} else {
|
||||
"http".to_string()
|
||||
}
|
||||
});
|
||||
|
||||
let host = headers
|
||||
.get(http::header::HOST)
|
||||
.and_then(|value| value.to_str().ok())
|
||||
.filter(|value| !value.is_empty())
|
||||
.map(str::to_string)
|
||||
.or_else(|| uri.authority().map(|value| value.as_str().to_string()))
|
||||
.or_else(|| {
|
||||
current_endpoints_handle().and_then(|endpoints| {
|
||||
endpoints
|
||||
.as_ref()
|
||||
.iter()
|
||||
.flat_map(|pool| pool.endpoints.as_ref().iter())
|
||||
.find(|endpoint| endpoint.is_local)
|
||||
.map(|endpoint| endpoint.host_port())
|
||||
})
|
||||
})
|
||||
.unwrap_or_else(|| format!("127.0.0.1:{}", current_runtime_port()));
|
||||
|
||||
format!("{scheme}://{host}")
|
||||
}
|
||||
|
||||
pub(crate) fn runtime_console_port() -> Option<u16> {
|
||||
let console_address = get_config_snapshot()
|
||||
.map(|snapshot| snapshot.console_address.clone())
|
||||
.unwrap_or_else(|| rustfs_utils::get_env_str(ENV_RUSTFS_CONSOLE_ADDRESS, DEFAULT_CONSOLE_ADDRESS));
|
||||
|
||||
let parse_target = if console_address.starts_with(':') {
|
||||
format!("127.0.0.1{console_address}")
|
||||
} else {
|
||||
console_address
|
||||
};
|
||||
|
||||
Url::parse(&format!("http://{parse_target}"))
|
||||
.ok()
|
||||
.and_then(|parsed| parsed.port_or_known_default())
|
||||
}
|
||||
|
||||
pub(crate) fn site_replication_local_endpoint(uri: &Uri, headers: &HeaderMap) -> String {
|
||||
let endpoint = request_endpoint(uri, headers);
|
||||
match Url::parse(&endpoint) {
|
||||
Ok(mut parsed) => {
|
||||
if !matches!(parsed.scheme(), "http" | "https") || parsed.host_str().is_none() {
|
||||
return request_endpoint(&Uri::from_static("/"), &HeaderMap::new());
|
||||
}
|
||||
if parsed.port_or_known_default() == runtime_console_port() && parsed.set_port(Some(current_runtime_port())).is_ok() {
|
||||
parsed.to_string().trim_end_matches('/').to_string()
|
||||
} else {
|
||||
endpoint
|
||||
}
|
||||
}
|
||||
Err(_) => request_endpoint(&Uri::from_static("/"), &HeaderMap::new()),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn current_local_runtime_endpoint() -> String {
|
||||
site_replication_local_endpoint(&Uri::from_static("/"), &HeaderMap::new())
|
||||
}
|
||||
|
||||
pub(crate) fn infer_site_name(endpoint: &str) -> String {
|
||||
endpoint
|
||||
.trim_start_matches("http://")
|
||||
.trim_start_matches("https://")
|
||||
.split('/')
|
||||
.next()
|
||||
.unwrap_or_default()
|
||||
.split(':')
|
||||
.next()
|
||||
.unwrap_or_default()
|
||||
.to_string()
|
||||
}
|
||||
|
||||
pub(crate) fn stored_peer_tls_settings(stored_peer: Option<&PeerInfo>) -> (bool, String) {
|
||||
stored_peer
|
||||
.map(|peer| (peer.skip_tls_verify, peer.ca_cert_pem.clone()))
|
||||
.unwrap_or_default()
|
||||
}
|
||||
|
||||
/// The local peer record as the given state describes it. Split out of
|
||||
/// [`current_local_peer`] so a state transaction can rebuild it against the
|
||||
/// state it just loaded: the request the endpoint came from cannot cross into
|
||||
/// the transaction closure, but the endpoint itself can.
|
||||
pub(crate) fn local_peer_at_endpoint(endpoint: String, state: &SiteReplicationState) -> PeerInfo {
|
||||
let deployment_id = current_deployment_id().unwrap_or_else(|| deployment_id_for_endpoint(&endpoint));
|
||||
let stored_peer = state.peers.get(&deployment_id);
|
||||
let (skip_tls_verify, ca_cert_pem) = stored_peer_tls_settings(stored_peer);
|
||||
|
||||
PeerInfo {
|
||||
endpoint: endpoint.clone(),
|
||||
name: if state.name.is_empty() {
|
||||
stored_peer
|
||||
.map(|peer| peer.name.clone())
|
||||
.filter(|name| !name.is_empty())
|
||||
.unwrap_or_else(|| infer_site_name(&endpoint))
|
||||
} else {
|
||||
state.name.clone()
|
||||
},
|
||||
deployment_id,
|
||||
sync_state: stored_peer.map(|peer| peer.sync_state.clone()).unwrap_or(SyncStatus::Unknown),
|
||||
default_bandwidth: stored_peer.map(|peer| peer.default_bandwidth.clone()).unwrap_or_default(),
|
||||
replicate_ilm_expiry: stored_peer.is_some_and(|peer| peer.replicate_ilm_expiry),
|
||||
object_naming_mode: stored_peer.map(|peer| peer.object_naming_mode.clone()).unwrap_or_default(),
|
||||
skip_tls_verify,
|
||||
ca_cert_pem,
|
||||
api_version: Some(SITE_REPL_API_VERSION.to_string()),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn current_local_runtime_peer(state: &SiteReplicationState) -> PeerInfo {
|
||||
local_peer_at_endpoint(current_local_runtime_endpoint(), state)
|
||||
}
|
||||
|
||||
pub(crate) fn normalize_peer_map_by_identity(peers: BTreeMap<String, PeerInfo>) -> BTreeMap<String, PeerInfo> {
|
||||
normalize_peer_map_by_identity_with(peers, normalize_peer_info)
|
||||
}
|
||||
|
||||
pub(crate) fn normalize_peer_info(mut peer: PeerInfo) -> PeerInfo {
|
||||
if peer.deployment_id.is_empty() {
|
||||
peer.deployment_id = deployment_id_for_endpoint(&peer.endpoint);
|
||||
}
|
||||
if peer.name.is_empty() {
|
||||
peer.name = infer_site_name(&peer.endpoint);
|
||||
}
|
||||
if peer.api_version.is_none() {
|
||||
peer.api_version = Some(SITE_REPL_API_VERSION.to_string());
|
||||
}
|
||||
peer
|
||||
}
|
||||
|
||||
pub(crate) async fn site_replicator_service_account_secret(access_key: &str) -> S3Result<String> {
|
||||
let Some(iam_sys) = current_iam_handle() else {
|
||||
return Err(s3_error!(InvalidRequest, "iam not init"));
|
||||
};
|
||||
|
||||
iam_sys
|
||||
.get_site_replicator_service_account_secret(access_key)
|
||||
.await
|
||||
.map_err(ApiError::from)
|
||||
.map_err(Into::into)
|
||||
}
|
||||
|
||||
pub(crate) fn legacy_site_replicator_state_secret(state: &SiteReplicationState) -> Option<String> {
|
||||
(state.service_account_access_key == SITE_REPLICATOR_SERVICE_ACCOUNT && !state.service_account_secret_key.is_empty())
|
||||
.then(|| state.service_account_secret_key.clone())
|
||||
}
|
||||
|
||||
pub(crate) fn pending_endpoint_refresh(state: &SiteReplicationState) -> Option<PendingEndpointRefresh> {
|
||||
state.pending_endpoint_refresh.clone().or_else(|| {
|
||||
state
|
||||
.retry_queue
|
||||
.iter()
|
||||
.find(|event| event.path == SITE_REPLICATION_ENDPOINT_REFRESH_RETRY_PATH)
|
||||
.and_then(|event| serde_json::from_str(&event.last_error).ok())
|
||||
})
|
||||
}
|
||||
|
||||
/// The wall clock in unix nanoseconds, clamped into u64. A pre-1970 (or
|
||||
/// post-2554) clock yields 0, which makes the hybrid allocation below
|
||||
/// degrade to the plain `previous + 1` counter — monotone, never panicking.
|
||||
pub(crate) fn edit_generation_wall_clock() -> u64 {
|
||||
u64::try_from(OffsetDateTime::now_utc().unix_timestamp_nanos()).unwrap_or(0)
|
||||
}
|
||||
|
||||
/// Allocate the next peer-edit generation as a hybrid logical clock:
|
||||
/// `max(wall clock in unix nanoseconds, previous + 1)`. Called inside the
|
||||
/// state transaction, so the value is handed out under the distributed
|
||||
/// state-object lock and two nodes of this site can never take the same one
|
||||
/// (`previous + 1` keeps the sequence strictly increasing even when two
|
||||
/// allocations land in one clock tick, and keeps it monotone on a node
|
||||
/// whose clock stepped backwards mid-lifetime).
|
||||
///
|
||||
/// The wall-clock floor is what survives the counter's death. A site
|
||||
/// removed while unreachable — the receiver never dropped it from its peer
|
||||
/// map, so the load-time mark pruning in `parse_site_replication_state`
|
||||
/// never fired — that later rejoins recreates its state object with the
|
||||
/// counter back at zero. A plain counter would then hand out generations
|
||||
/// below the receiver's stale high-water mark and every delivery would be
|
||||
/// silently fenced until the counter caught up. Jumping to wall time clears
|
||||
/// that mark: every value the deleted lifetime handed out was capped by the
|
||||
/// wall clock at its own allocation (or by a prior lifetime's cap, applied
|
||||
/// inductively), so the recreated lifetime's first allocation exceeds them
|
||||
/// all — while a pre-removal delivery still in flight stays below the new
|
||||
/// floor and remains correctly fenced. Marks recorded by pre-hybrid
|
||||
/// receivers (small plain-counter values) sit far below any wall-clock
|
||||
/// value, so a restarted origin passes those too — the fix needs only the
|
||||
/// sender upgraded, nothing on the wire or in the receiver changed.
|
||||
///
|
||||
/// A wall clock that regresses across a delete/recreate (the recreating
|
||||
/// node's clock behind the clock that fed the previous lifetime) mints
|
||||
/// below the stale mark and the origin stays fenced — but only until real
|
||||
/// time passes the previous lifetime's last allocation, because every later
|
||||
/// allocation takes the wall-clock floor again (and never longer than
|
||||
/// [`PEER_EDIT_FENCE_STALENESS_WINDOW_NANOS`]: a regression past the window
|
||||
/// leaves the mark implausibly distant and the origin runs unfenced
|
||||
/// immediately). Bounded by the skew,
|
||||
/// self-healing, and no rollback window beyond the plain counter's: a
|
||||
/// delivery applies only at or above the receiver's mark, so the one
|
||||
/// cross-lifetime interleaving that can apply stale content — a
|
||||
/// pre-removal delivery whose generation lands above everything the
|
||||
/// regressed new lifetime has minted — required the same straggler landing
|
||||
/// above the mark under the plain counter, where the recreated counter's
|
||||
/// low restart made it strictly easier to hit.
|
||||
pub(crate) fn next_peer_edit_generation(state: &mut SiteReplicationState) -> u64 {
|
||||
state.edit_generation = edit_generation_wall_clock().max(state.edit_generation.saturating_add(1));
|
||||
state.edit_generation
|
||||
}
|
||||
|
||||
/// Build the peer-edit request path carrying the fencing token. The bare
|
||||
/// constant stays the retry-queue key: the query only fences the wire
|
||||
/// delivery, and a per-generation key would make every retry event unique.
|
||||
/// Without a local deployment id there is nothing to fence against, so the
|
||||
/// unstamped path is sent and the receiver keeps its pre-fence behaviour.
|
||||
pub(crate) fn peer_edit_path_with_fence(origin: Option<&str>, generation: u64) -> String {
|
||||
let Some(origin) = origin.filter(|origin| !origin.is_empty()) else {
|
||||
return SITE_REPLICATION_PEER_EDIT_PATH.to_string();
|
||||
};
|
||||
let query = form_urlencoded::Serializer::new(String::new())
|
||||
.append_pair(SITE_REPLICATION_EDIT_ORIGIN_QUERY, origin)
|
||||
.append_pair(SITE_REPLICATION_EDIT_GENERATION_QUERY, &generation.to_string())
|
||||
.finish();
|
||||
format!("{SITE_REPLICATION_PEER_EDIT_PATH}?{query}")
|
||||
}
|
||||
+3
-4
@@ -34,12 +34,11 @@
|
||||
//! Lock order: lifecycle -> bucket operation -> repair admission
|
||||
//! -> state object lock -> per-bucket metadata.
|
||||
|
||||
use crate::admin::storage_api::runtime::ECStore;
|
||||
use crate::admin::storage_api::s3::{S3Error, S3ErrorCode, S3Result};
|
||||
use crate::storage::storage_api::with_config_object_write_lock;
|
||||
use super::{S3Error, S3ErrorCode, S3Result};
|
||||
use crate::storage_api::site_replication::{ECStore, with_config_object_write_lock};
|
||||
use std::sync::Arc;
|
||||
|
||||
use super::runtime_sources::current_object_store_handle;
|
||||
use crate::runtime_sources::current_object_store_handle;
|
||||
|
||||
/// Config object holding the whole site-replication state, including the
|
||||
/// retry-event queue. Shared by the typed handler-side accessors and the
|
||||
File diff suppressed because it is too large
Load Diff
@@ -216,6 +216,54 @@ pub(crate) mod server {
|
||||
}
|
||||
}
|
||||
|
||||
/// Storage surface of the site-replication service module
|
||||
/// (`crate::site_replication`, backlog#1840): bucket metadata, bucket
|
||||
/// targets, replication-config primitives, and the config-object lock
|
||||
/// helpers its state transaction runs on.
|
||||
pub(crate) mod site_replication {
|
||||
pub(crate) use super::storage_contracts::{BucketOperations, BucketOptions};
|
||||
pub(crate) use crate::storage::storage_api::ecstore_bucket::bucket_target_sys::BucketTargetSys;
|
||||
pub(crate) use crate::storage::storage_api::ecstore_bucket::metadata::{
|
||||
BUCKET_REPLICATION_CONFIG, BUCKET_TARGETS_FILE, BUCKET_VERSIONING_CONFIG, BucketMetadata,
|
||||
};
|
||||
|
||||
pub(crate) use crate::storage::storage_api::ecstore_bucket::replication::{
|
||||
OperatorRuleContract, assign_site_replication_rule_priorities, is_site_replication_role,
|
||||
replication_target_arn_deployment_id, site_replication_rule_deployment_id,
|
||||
};
|
||||
pub(crate) use crate::storage::storage_api::ecstore_bucket::target::{
|
||||
ARN, BucketTarget, BucketTargetType, BucketTargets, Credentials,
|
||||
};
|
||||
pub(crate) use crate::storage::storage_api::ecstore_bucket::utils::{deserialize, serialize};
|
||||
pub(crate) use crate::storage::storage_api::ecstore_bucket::versioning::VersioningApi;
|
||||
#[cfg(test)]
|
||||
pub(crate) use crate::storage::storage_api::ecstore_config::com::save_config;
|
||||
|
||||
pub(crate) use crate::storage::storage_api::{
|
||||
ECStore, EndpointServerPools, StorageError, delete_config_no_lock, lock_bucket_targets_metadata, read_config,
|
||||
read_config_no_lock, save_config_no_lock, with_config_object_read_lock, with_config_object_write_lock,
|
||||
};
|
||||
|
||||
pub(crate) mod metadata_sys {
|
||||
pub(crate) use crate::storage::storage_api::ecstore_bucket::metadata_sys::{
|
||||
capture_bucket_metadata_incarnation, get, get_replication_config, get_versioning_config, list_bucket_targets,
|
||||
update_if_incarnation,
|
||||
};
|
||||
}
|
||||
|
||||
/// S3 wire types for the service module, funneled here so the module
|
||||
/// itself stays off the direct s3s surface (s3s footprint ratchet).
|
||||
pub(crate) mod s3 {
|
||||
pub(crate) use s3s::dto::{
|
||||
BucketLifecycleConfiguration, BucketVersioningStatus, DeleteMarkerReplication, DeleteMarkerReplicationStatus,
|
||||
DeleteReplication, DeleteReplicationStatus, Destination, ExistingObjectReplication, ExistingObjectReplicationStatus,
|
||||
LifecycleRule, ReplicaModifications, ReplicaModificationsStatus, ReplicationConfiguration, ReplicationRule,
|
||||
ReplicationRuleStatus, SourceSelectionCriteria, VersioningConfiguration,
|
||||
};
|
||||
pub(crate) use s3s::{Body, S3Error, S3ErrorCode, S3Response, S3Result, s3_error};
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod startup {
|
||||
pub(crate) mod heal_control {
|
||||
#[cfg(test)]
|
||||
|
||||
@@ -29,6 +29,14 @@ checked_files=(
|
||||
"rustfs/src/admin/handlers/kms_keys.rs"
|
||||
"rustfs/src/admin/handlers/kms_key_lifecycle.rs"
|
||||
"rustfs/src/admin/handlers/site_replication.rs"
|
||||
"rustfs/src/site_replication/mod.rs"
|
||||
"rustfs/src/site_replication/identity.rs"
|
||||
"rustfs/src/site_replication/state_lock.rs"
|
||||
"rustfs/src/site_replication/state.rs"
|
||||
"rustfs/src/site_replication/transport.rs"
|
||||
"rustfs/src/site_replication/retry.rs"
|
||||
"rustfs/src/site_replication/repair.rs"
|
||||
"rustfs/src/site_replication/hooks.rs"
|
||||
"rustfs/src/admin/handlers/group.rs"
|
||||
"rustfs/src/admin/handlers/quota.rs"
|
||||
"rustfs/src/admin/handlers/rebalance.rs"
|
||||
|
||||
@@ -38,8 +38,14 @@ cd "$(dirname "$0")/.."
|
||||
# files, zero new s3s code — the same handler-layer surface redistributed).
|
||||
# The file counter is split-sensitive; the s3_error! line counter confirms
|
||||
# no growth (unchanged at 1620).
|
||||
# 1620 → 1616 on 2026-08-27: backlog#1840 moved the site-replication service
|
||||
# subsystem to rustfs/src/site_replication/ (s3s access funneled through the
|
||||
# root storage facade's s3 shim, keeping the file count at 215). The move
|
||||
# inlined one s3_error! call in transport.rs (+1); measured 1615 on the
|
||||
# pre-move main (after #6694) and 1616 after, so the slack 1620 baseline is
|
||||
# retightened to the measured 1616.
|
||||
S3S_IMPORT_FILES_BASELINE=215
|
||||
S3_ERROR_LINES_BASELINE=1620
|
||||
S3_ERROR_LINES_BASELINE=1616
|
||||
# ecstore-scoped ratchet (rustfs/backlog#1842): the storage engine must not
|
||||
# know S3 wire/DTO types (ARCHITECTURE.md invariant 4). The S3-*consuming*
|
||||
# client was extracted to crates/s3-client, where s3s usage is legitimate;
|
||||
|
||||
@@ -18,9 +18,6 @@ cycle|app<->infra
|
||||
cycle|app<->interface
|
||||
cycle|infra<->interface
|
||||
dep|rustfs/src/app/admin_usecase.rs|app->interface|crate::server::collect_dependency_readiness_report
|
||||
dep|rustfs/src/app/bucket_usecase.rs|app->interface|crate::admin::handlers::site_replication::site_replication_bucket_meta_hook
|
||||
dep|rustfs/src/app/bucket_usecase.rs|app->interface|crate::admin::handlers::site_replication::site_replication_delete_bucket_hook
|
||||
dep|rustfs/src/app/bucket_usecase.rs|app->interface|crate::admin::handlers::site_replication::site_replication_make_bucket_hook
|
||||
dep|rustfs/src/cluster_snapshot.rs|infra->interface|crate::server::snapshot_dependency_readiness_report
|
||||
dep|rustfs/src/runtime_sources.rs|infra->app|crate::app::context
|
||||
dep|rustfs/src/storage/ecfs_extend.rs|infra->interface|crate::server::cors
|
||||
|
||||
Reference in New Issue
Block a user