gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-29 16:37:07 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(replication): tolerate orphaned resync intents at startup (#6470)

Browse Source 
* fix(replication): tolerate orphaned resync intents at startup

Since #5215 (1.0.0-beta.12) startup reconciles every pending/started
resync intent in resync.bin against the bucket's configured targets and
aborts the whole server when an intent has no matching target ARN. A
resync whose remote target was later removed leaves exactly such an
orphan on disk, so every later start fails with "accepted replication
resync target ... is not configured" regardless of the binary version.

Skip orphaned intents with a warning instead of failing startup; the
resync routine already settles them to ResyncFailed. Cancel the intent
when its remote target is removed so the orphan is not created again.

Fixes #4784

* fix(replication): cancel removed-target resync under the admission lock

Canceling through this node's cached whole-bucket status map could
persist a map that predates another node's admission, erasing that
node's durable restart intent. Reload resync.bin under the bucket
admission lock, publish the fresh map, and only then mark the removed
target's intent canceled. Two-node regression covers the clobber.

* fix(replication): persist resync status via ETag CAS merge

mark_status, the periodic saver, admission, and removed-target
cancellation all persisted their node's cached whole-bucket map, so any
one node's stale cache could resurrect states another node had already
finalized (a canceled intent flipping back to Pending, an admission
vanishing). All resync.bin writers now go through update_resync_status_cas:
load the freshest document with its ETag, apply a per-target mutation
with staleness and canceled-is-terminal guards re-checked against the
persisted entry, and save conditionally, retrying on concurrent writes.
The periodic saver merges per target, letting terminal states and newer
admissions recorded elsewhere win. Cache convergence stays per-target so
locally running resyncs keep their authoritative progress counters.

Regressions: stale_peer_status_write_cannot_resurrect_canceled_intent
(node B's pre-cancel cache marking its own run Started must not revive
node A's canceled intent) plus unit coverage for the periodic-save merge.

* test(ecstore): rename resync test helper off the guarded contract name

fn resync_target is on the architecture guard's reserved list for
crates/replication operation contracts; the merge-test helper now reads
resync_target_state.

* fix(replication): serialize resync status updates

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
This commit is contained in:
唐小鸭
2026-08-24 14:25:44 +08:00
committed by GitHub
parent e091a7e702
commit fc98dbb654
5 changed files with 731 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files
rustfs/src/admin/handlers/replication.rs
+33 -2
View File
@@ -16,8 +16,8 @@ use crate::admin::auth::authorize_admin_request;
use crate::admin::handlers::site_replication::site_replication_peer_deployment_id_for_endpoint;
use crate::admin::router::{AdminOperation, Operation, S3Router};
use crate::admin::runtime_sources::{
AppContext, app_context_from_req, current_notification_system_for_context, current_replication_stats_handle_for_context,
current_runtime_port, object_store_from_req,
AppContext, app_context_from_req, current_notification_system_for_context, current_replication_pool_handle,
current_replication_stats_handle_for_context, current_runtime_port, object_store_from_req,
};
use crate::admin::storage_api::bucket::metadata::BUCKET_TARGETS_FILE;
use crate::admin::storage_api::bucket::metadata_sys;
@@ -861,6 +861,8 @@ impl Operation for RemoveRemoteTargetHandler {
let targets = sys.remove_target(bucket, arn_str).await.map_err(map_bucket_target_error)?;
cancel_active_resync_intent(bucket, arn_str).await?;
let json_targets = serde_json::to_vec(&targets).map_err(|e| {
error!("Serialization error: {}", e);
S3Error::with_message(S3ErrorCode::InternalError, "Failed to serialize targets".to_string())
@@ -878,6 +880,35 @@ impl Operation for RemoveRemoteTargetHandler {
}
}
/// Cancel a pending/started resync intent recorded for `arn` before the target
/// is removed. Without this the intent outlives its target in `resync.bin`, and
/// every later startup reconcile finds an accepted intent with no target to
/// bind it to. The pool reloads and rewrites the status under the bucket
/// admission lock so other nodes' intents are never clobbered. Buckets with no
/// resync history are a no-op.
async fn cancel_active_resync_intent(bucket: &str, arn: &str) -> S3Result<()> {
let Some(pool) = current_replication_pool_handle() else {
return Ok(());
};
let canceled = pool
.cancel_bucket_resync_for_removed_target(bucket, arn)
.await
.map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, format!("Failed to cancel resync: {e}")))?;
if let Some(opts) = canceled {
info!(
event = "replication_resync_intent_canceled",
component = "admin",
subsystem = "replication",
result = "canceled",
bucket,
arn,
resync_id = %opts.resync_id,
"canceled active resync intent for removed remote target"
);
}
Ok(())
}
/// Upper bound on the number of object versions scanned per `POST
/// /v3/replication/diff` request. RustFS has no persisted per-object
/// replication-diff index, so the diff is computed by scanning object versions
rustfs/src/storage/storage_api.rs
+83 -9
View File
@@ -748,6 +748,14 @@ impl StorageReplicationPoolHandle {
self.inner.clone().cancel_bucket_resync(opts).await
}
pub(crate) async fn cancel_bucket_resync_for_removed_target(
&self,
bucket: &str,
arn: &str,
) -> Result<Option<ecstore_bucket::replication::ResyncOpts>> {
self.inner.clone().cancel_bucket_resync_for_removed_target(bucket, arn).await
}
pub(crate) async fn admit_bucket_resync(&self, opts: ecstore_bucket::replication::ResyncOpts) -> Result<bool> {
self.inner.clone().admit_bucket_resync(opts).await
}
@@ -872,10 +880,20 @@ pub(crate) async fn init_background_replication(store: Arc<ECStore>) {
ecstore_bucket::replication::init_background_replication(store).await;
}
/// Reconcile accepted (pending/started) resync intents into the bucket's
/// target metadata. Returns whether `targets` changed.
///
/// An intent whose target ARN is no longer configured is an orphan: the
/// remote target was removed after the resync was admitted, or the record
/// predates the atomic-admission contract. Nothing can be reconciled for it,
/// so it is skipped here and left to the resync routine, which marks it
/// `ResyncFailed` through `resolve_resync_target`. Failing startup on it
/// would keep the whole server down over one stale replication record.
fn apply_active_resync_intents(
bucket: &str,
targets: &mut ecstore_bucket::target::BucketTargets,
status: &ecstore_bucket::replication::BucketReplicationResyncStatus,
) -> Result<bool> {
) -> bool {
let mut changed = false;
for (arn, intent) in &status.targets_map {
if !matches!(
@@ -885,18 +903,26 @@ fn apply_active_resync_intents(
) {
continue;
}
let target = targets
.targets
.iter_mut()
.find(|target| target.arn == *arn)
.ok_or_else(|| Error::other(format!("accepted replication resync target {arn} is not configured")))?;
let Some(target) = targets.targets.iter_mut().find(|target| target.arn == *arn) else {
tracing::warn!(
event = "replication_resync_intent_orphaned",
component = "storage",
subsystem = "replication",
result = "skipped",
bucket,
arn = %arn,
resync_status = ?intent.resync_status,
"accepted replication resync target is no longer configured; skipping startup reconcile"
);
continue;
};
if target.reset_id != intent.resync_id || target.reset_before_date != intent.resync_before_date {
target.reset_id = intent.resync_id.clone();
target.reset_before_date = intent.resync_before_date;
changed = true;
}
}
Ok(changed)
changed
}
pub(crate) async fn reconcile_bucket_resync_target_intents(buckets: &[String]) -> Result<()> {
@@ -916,7 +942,7 @@ pub(crate) async fn reconcile_bucket_resync_target_intents(buckets: &[String]) -
} else {
serde_json::from_slice(&metadata.bucket_targets_config_json).map_err(Error::other)?
};
if !apply_active_resync_intents(&mut targets, &status)? {
if !apply_active_resync_intents(bucket, &mut targets, &status) {
continue;
}
let encoded = serde_json::to_vec(&targets).map_err(Error::other)?;
@@ -1967,8 +1993,56 @@ mod tests {
},
);
assert!(apply_active_resync_intents(&mut targets, &status).expect("accepted intent should reconcile"));
assert!(apply_active_resync_intents("bucket-a", &mut targets, &status));
assert_eq!(targets.targets[0].reset_id, "durable-id");
assert_eq!(targets.targets[1].reset_id, "concurrent-id");
}
/// A pending/started intent whose target was removed (or predates the
/// atomic-admission contract) must not abort startup; it is skipped and
/// the remaining intents still reconcile.
#[test]
fn restart_reconcile_skips_orphaned_intent_without_failing_startup() {
let mut targets = ecstore_bucket::target::BucketTargets {
targets: vec![ecstore_bucket::target::BucketTarget {
arn: "arn:minio:replication::depl-1:configured".to_string(),
..Default::default()
}],
};
let mut status = ecstore_bucket::replication::BucketReplicationResyncStatus::new();
for (arn, resync_status) in [
(
"arn:rustfs:replication::2ae1d6316a2f17d8:removed",
ecstore_bucket::replication::ResyncStatusType::ResyncStarted,
),
(
"arn:minio:replication::depl-1:configured",
ecstore_bucket::replication::ResyncStatusType::ResyncPending,
),
] {
status.targets_map.insert(
arn.to_string(),
ecstore_bucket::replication::TargetReplicationResyncStatus {
resync_id: "durable-id".to_string(),
resync_status,
..Default::default()
},
);
}
assert!(apply_active_resync_intents("bucket-a", &mut targets, &status));
assert_eq!(targets.targets.len(), 1, "orphaned intent must not materialize a target");
assert_eq!(targets.targets[0].reset_id, "durable-id");
let mut only_orphan = ecstore_bucket::replication::BucketReplicationResyncStatus::new();
only_orphan.targets_map.insert(
"arn:rustfs:replication::2ae1d6316a2f17d8:removed".to_string(),
ecstore_bucket::replication::TargetReplicationResyncStatus {
resync_id: "durable-id".to_string(),
resync_status: ecstore_bucket::replication::ResyncStatusType::ResyncStarted,
..Default::default()
},
);
assert!(!apply_active_resync_intents("bucket-a", &mut targets, &only_orphan));
}
}