rustfs

mirror of https://github.com/rustfs/rustfs.git synced 2026-08-16 18:08:21 +00:00

Author	SHA1	Message	Date
Zhengchao An	7e071dc0fa	test(site-replication): cover retry endpoint matching	2026-08-16 06:17:58 +08:00
Zhengchao An	e117a5d09f	test(site-replication): preserve collapsed retry liability	2026-08-16 06:14:35 +08:00
Zhengchao An	a26e97bbff	fix(site-replication): gate test-only retry helper	2026-08-16 05:34:49 +08:00
overtrue	96a2f9c67f	fix(site-replication): preserve retry accountability	2026-08-16 04:54:11 +08:00
唐小鸭	af802756c5	fix(site-replication): only a repair settles snapshot-escalated retry entries Second review round: every iam-item / bucket-meta delivery shares a constant path, so any later successful single-item delivery (a Bob update) dequeued the escalated marker recording a possibly-unreplayed deletion (a failed Alice delete) while the entity still existed remotely. Ordinary settlement now skips escalated entries; only the repair path — the operator's explicit accountability transfer — clears them via dequeue_..._including_escalated. A new hook failure still overwrites the marker and re-arms the drain. Regression covers survive-ordinary-dequeue and repair-clears.	2026-08-16 01:02:42 +08:00
唐小鸭	971addca6e	fix(site-replication): escalate snapshot-replayed retry entries instead of clearing them Review: the bootstrap-plan snapshot cannot replay deletions — a deleted IAM entity or absent bucket config produces no task, so clearing the collapsed iam-item / bucket-meta entry after a successful snapshot resend silently lost a failed delete and the peer kept stale state permanently. The drain now keeps those entries until remote absence is proven: after a successful snapshot resend the entry is escalated (failed=true, marker last_error) so it stays operator-visible in retry_stats, and classification skips marked entries so the once-per-failure-episode snapshot is not re-sent daily. A newer hook failure overwrites the marker and re-arms the drain; a later full delivery or a manual repair settles the entry. Escalation is conditional on the snapshot timestamp, preserving the earlier review's in-flight-failure guarantee. Bucket ops and peer edits keep auto-settle — their replays are faithful.	2026-08-15 19:14:23 +08:00
唐小鸭	2c7d1f1f9f	fix(site-replication): harden the retry drain against review findings Adversarial review of the drain surfaced one real race and three cheap hardenings: - Conditional settlement for collapsed (constant-path) iam-item / bucket-meta entries: the snapshot resend proves delivery of the state as of plan-build time, so a hook failure stamped during the delivery window (a newer local commit the snapshot did not contain) must survive the snapshot's success instead of being cleared — previously the peer would silently diverge until the next same-path failure. (The operator repair path shares this collapse but keeps its existing unconditional settle; the drain runs every tick and needed the guard first.) - Re-check the pending_* gates on the freshly loaded state: an endpoint refresh can commit its pending marker mid-tick without the lifecycle guard. - Do not upsert retry events for peers that already left the state — remove_sites pruned their entries and they can never drain again. - Correct the mutual-exclusion comment: the repair-execution lock does NOT close the dry-run -> execute window (dry-run takes no lock); that window fails safe via the preflight token, which hashes the replayable entries. New tests pin the (Some settled, None failed) generation-settle combination the peer-edit drain depends on, and the snapshot-relative conditional settlement.	2026-08-15 11:01:58 +08:00
唐小鸭	5328e8b958	feat(site-replication): drain the retry queue from the reconcile tick The retry queue recorded every failed peer delivery but had no consumer — missed IAM/bucket metadata updates drifted until an operator ran a manual repair (backlog#1675 P1-3). The 600s reconcile tick now drains the queue behind the existing lifecycle guard and pending_* gates. Replay discipline (pinned by the red-light tests): - IAM / bucket-meta entries collapse per (peer, path) and their bodies are not persisted, so the only faithful replay is the current bootstrap-plan snapshot (SiteReplicationRepairTask reuse) sent to the affected peer. - make-with-versioning / configure-replication ops are re-derived from the CURRENT plan for their bucket — never the recorded path, whose query can carry an expired one-shot bootstrap token; an op whose bucket left the plan is provably stale and is settled. - Peer edits are re-sent as the current peer records under a freshly allocated generation inside one state transaction — the recorded generation is stale by definition and the receiver would fence it. - Destructive bucket ops (delete-bucket / force-delete-bucket) and internal: marker records (pending-endpoint-refresh backup store) are never background-replayed. - Exponential backoff (600s * 2^(n-1), 24h ceiling) gates each attempt; an unreachable peer's transport failure re-queues its events so the backoff still advances. Concurrency: the drain takes the repair-execution config lock — the operator repair preflight token hashes the replayable retry events, so settling them between dry-run and execute would strand the operator on a stale preflight. Lock order matches repair (lifecycle guard -> repair execution lock -> state object lock); success/failure settlement reuses the generation-fenced upsert/settle semantics from #5882/#6097. Scope note: the drain settles 'delivered once and failed' entries. A hook that never fired (crash between local commit and send) leaves no entry; a low-frequency plan-diff catch-all remains follow-up work, and manual repair stays authoritative.	2026-08-15 10:42:36 +08:00
唐小鸭	1e16e06f8a	test(site-replication): pin the background retry-drain discipline Red-light evidence for backlog#1675 P1-3: the retry queue has no background consumer — every failed peer delivery waits for a manual repair. The new tests specify the drain rules before the drain exists: - classification: IAM / bucket-meta entries collapse per (peer, path) with no persisted body, so only a bootstrap-plan snapshot resend is a faithful replay; make-with-versioning / configure-replication are re-derivable per bucket; peer edits are re-sent under a fresh generation; destructive bucket ops and internal: marker records (pending-endpoint-refresh backup store) are never background-replayed - exponential backoff (600s * 2^(n-1), 24h ceiling) gates every attempt, otherwise a dead peer's entries hit the failed threshold within 30 minutes of reconcile ticks - the actionable subset respects classification, current peer membership and backoff All fail against the placeholder implementations (no consumer).	2026-08-15 10:19:28 +08:00

2 changed files with 1051 additions and 12 deletions

									
										crates/madmin/src/site_replication.rs
									
		+3
		-3
	
												View File
												
				@@ -258,7 +258,7 @@ pub struct SRLDAPUser {

				    pub api_version: Option<String>,

				}

				#[derive(Debug, Serialize, Deserialize, Default)]

				#[derive(Debug, Clone, Serialize, Deserialize, Default)]

				pub struct SRIAMUser {

				    #[serde(rename = "accessKey", default)]

				    pub access_key: String,

				@@ -270,7 +270,7 @@ pub struct SRIAMUser {

				    pub api_version: Option<String>,

				}

				#[derive(Debug, Serialize, Deserialize, Default)]

				#[derive(Debug, Clone, Serialize, Deserialize, Default)]

				pub struct SRGroupInfo {

				    #[serde(rename = "updateReq", default)]

				    pub update_req: GroupAddRemove,

				@@ -346,7 +346,7 @@ pub struct SRCredInfo {

				    pub api_version: Option<String>,

				}

				#[derive(Debug, Serialize, Deserialize, Default)]

				#[derive(Debug, Clone, Serialize, Deserialize, Default)]

				pub struct SRIAMItem {

				    #[serde(default)]

				    pub r#type: String,

rustfs/src/admin/handlers/site_replication.rs

+1048 -9

View File

File diff suppressed because it is too large Load Diff

Compare commits

9 Commits

fix/auto-repair-20260817-013500 ... fix/p1-3-retry-queue-drain

Compare commits

9 Commits fix/auto-repair-20260817-013500 ... fix/p1-3-retry-queue-drain

9 Commits

fix/auto-repair-20260817-013500 ... fix/p1-3-retry-queue-drain