fix(site-replication): clear pending_remove on join, surface failures

A removal that could not notify its peers left `pending_remove` set
forever. That single field gates `SRPeerBucketOpsHandler` ahead of
`enabled()`, so the site rejected every peer bucket-op with
"site replication is not enabled" while nothing surfaced on the source:
`mb` succeeded, `replicate info` still reported a healthy cluster, and no
client command exited non-zero.

An accepted peer join now clears the marker, so a re-add actually
repairs the cluster instead of restoring the topology on both sides
while replication stays dead. The peer-edit high-water marks are
deliberately untouched — those fence edit ordering, not lifecycle.

Also:

- `SRPeerJoinResponse.applied` makes a no-op join distinguishable. It is
  three-valued: `None` means the peer did not report (MinIO answers a
  successful join with an empty body), so it is never read as failure.
  The rotation fan-out consumes it too: a superseded join returns before
  `apply_iam`, so acking one finalized a rotation whose new secret the
  peer never installed.
- The reconcile tick re-drives a stuck removal instead of giving up the
  round, so a peer coming back finishes it without operator action.
- `replicate info` carries `retryStats` and `pendingOperation`. Both are
  omitted when absent, so a healthy site stays wire-identical.
- A removal with unnotified peers answers `Partial` instead of the
  success string. The fully-notified path is unchanged byte for byte.

The lab script gains a `diverge` subcommand that reproduces the report
end to end and asserts recovery.

Fixes #5963
This commit is contained in:
唐小鸭
2026-08-19 09:55:42 +08:00
parent cd9c96a03c
commit 1937a0152c
3 changed files with 732 additions and 106 deletions
+10
View File
@@ -83,6 +83,16 @@ pub struct SiteReplicationInfo {
pub service_account_access_key: String,
#[serde(rename = "apiVersion", skip_serializing_if = "Option::is_none")]
pub api_version: Option<String>,
/// Outstanding peer deliveries. Absent when the retry queue is empty, so a
/// healthy site serializes exactly as it did before this field existed.
/// Present means peer operations are failing even if `enabled` is true.
#[serde(rename = "retryStats", default, skip_serializing_if = "Option::is_none")]
pub retry_stats: Option<SRRetryStats>,
/// A multi-step lifecycle operation this site has not finished — most
/// importantly a removal that could not reach its peers, which makes the
/// site reject peer operations while `enabled` may still read true.
#[serde(rename = "pendingOperation", default, skip_serializing_if = "Option::is_none")]
pub pending_operation: Option<SRPendingOperation>,
}
#[derive(Debug, Clone, Serialize, Deserialize, Default)]