fix(ecstore): split rename_data signature from heal-convergence decision (#4926)

CompleteMultipartUpload enqueued a normal-priority heal whenever
`rename_data` returned a `Some(versions)` signature. But the per-disk
signature is produced for every object with <=10 versions, and a healthy
quorum reduces to `Some` as well, so the `Option<Vec<u8>>` return value
conflated two distinct facts — "a version signature exists" and "the
committed replicas need heal". The result: nearly every healthy MPU
completion self-enqueued a heal, while >10-version objects (signature
`None`) did not — an algorithmic heal amplification on the healthy path
(rustfs/backlog#1321).

Replace the overloaded `Option<Vec<u8>>` second element of
`SetDisks::rename_data` with an explicit `RenameConvergence` classification
computed after the write-quorum gate:

- AllSuccessIdentical — every attempted disk committed with an identical,
  known signature (no heal).
- PartialCommit — write quorum met but a disk failed/offline; a committed
  replica is missing or stale (heal).
- SignatureDivergent — all committed but signatures diverge, or mix signed
  (<=10-version) with unsigned (>10-version) disks (heal).
- Unknown — all committed, no signature produced (>10 versions); latent
  divergence is left to the scanner backstop, not self-enqueued.

`RenameConvergence::needs_heal()` is the single decision point. The version
signature is now only comparison material; it no longer doubles as a heal
flag. The old `select_rename_data_versions` / `reduce_common_versions` /
`rename_data_versions_key` machinery that carried the conflation is removed.

The heal submission in `complete_multipart_upload` moves off the ACK
critical path into a detached task: it runs after the object lock is
dropped and after the durable `rename_data` commit, survives cancellation
of the completion future, and coalesces through the existing bounded /
deduplicated / observable heal-channel admission (one submit per degraded
completion, at most). A completion cancelled in the narrow window between
the durable commit and reaching the enqueue is scanner-backstopped, as is
the Unknown (>10-version) case.

The PUT path (`object.rs`) binds the second element as `_` and is
unchanged. The change is orthogonal to and composes with the #1312 commit
fence on the same `rename_data` path (epoch rejection is a commit-gate
failure surfaced through `Result::Err`, convergence is a post-commit
signal); documented in docs/architecture/unified-object-generation.md.

Tests: `classify_rename_convergence` white-box cases cover the full
acceptance matrix (healthy 4/4 and 8/8, 3-same-1-divergent, failed/offline
disk, no-common-quorum split, >10-version all-success and with-failure,
mixed signed/unsigned) and fail on revert to the old "signature exists =>
heal" semantics. The decision function is tested directly rather than
through the process-global heal channel, whose receiver is owned
exclusively by the blackbox serial test (init_heal_channel is once per
binary).

Refs: https://github.com/rustfs/backlog/issues/1321
This commit is contained in:
Zhengchao An
2026-07-17 01:38:15 +08:00
committed by GitHub
parent 6559248f55
commit b41bbe2db4
4 changed files with 257 additions and 86 deletions
+40 -11
View File
@@ -1433,7 +1433,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
// The trailing `_` drops the rename_data old-size backfill
// (rustfs/backlog#1009): CompleteMultipartUpload keeps its pre-commit
// `get_object_info` lookup, so the backfill has no consumer here yet.
let (online_disks, versions, op_old_dir, cleanup_disks, _) = Self::rename_data(
let (online_disks, convergence, op_old_dir, cleanup_disks, _) = Self::rename_data(
&shuffle_disks,
RUSTFS_META_MULTIPART_BUCKET,
&upload_id_path,
@@ -1485,17 +1485,46 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
drop(object_lock_guard); // drop object lock guard to release the lock
if let Some(versions) = versions {
let _ =
rustfs_common::heal_channel::send_heal_request(rustfs_common::heal_channel::create_heal_request_with_options(
bucket.to_string(),
Some(object.to_string()),
false,
Some(HealChannelPriority::Normal),
Some(self.pool_index),
Some(self.set_index),
))
// backlog#1321: enqueue heal only when the committed replicas actually
// need to converge — a partial commit (some disk failed/offline) or a
// signature divergence between committed replicas. A fully healthy MPU
// (identical signatures on every disk) is `AllSuccessIdentical` and
// submits nothing, which is the fix: the old `Option::is_some()` gate
// treated the mere existence of a version signature as "needs heal", so
// every healthy <=10-version completion self-enqueued.
//
// The submit is detached (`tokio::spawn`) so it stays off the ACK
// critical path AND survives cancellation of the completion future: the
// write is already durable and ACK-worthy, so the heal admission must
// not ride the client's request lifetime. The admission itself is
// bounded / deduplicated / observable (`send_heal_request` ->
// `HealAdmissionResult`), so this emits at most one submit per
// completion and coalesces with any in-flight heal for the same object.
//
// Scanner backstop (backlog#1321 patch): a `PartialCommit` whose
// completion is cancelled in the narrow window after the durable commit
// but before this spawn runs is not lost — the divergence it would have
// healed is exactly what the background scanner reconciles. `Unknown`
// (>10 versions, no signature produced) likewise relies on the scanner
// rather than self-enqueuing.
if convergence.needs_heal() {
let bucket = bucket.to_string();
let object = object.to_string();
let pool_index = self.pool_index;
let set_index = self.set_index;
tokio::spawn(async move {
let _ = rustfs_common::heal_channel::send_heal_request(
rustfs_common::heal_channel::create_heal_request_with_options(
bucket,
Some(object),
false,
Some(HealChannelPriority::Normal),
Some(pool_index),
Some(set_index),
),
)
.await;
});
}
let upload_id_path = upload_id_path.clone();