mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-11 23:56:53 +00:00
fix(ecstore): split rename_data signature from heal-convergence decision (#4926)
CompleteMultipartUpload enqueued a normal-priority heal whenever `rename_data` returned a `Some(versions)` signature. But the per-disk signature is produced for every object with <=10 versions, and a healthy quorum reduces to `Some` as well, so the `Option<Vec<u8>>` return value conflated two distinct facts — "a version signature exists" and "the committed replicas need heal". The result: nearly every healthy MPU completion self-enqueued a heal, while >10-version objects (signature `None`) did not — an algorithmic heal amplification on the healthy path (rustfs/backlog#1321). Replace the overloaded `Option<Vec<u8>>` second element of `SetDisks::rename_data` with an explicit `RenameConvergence` classification computed after the write-quorum gate: - AllSuccessIdentical — every attempted disk committed with an identical, known signature (no heal). - PartialCommit — write quorum met but a disk failed/offline; a committed replica is missing or stale (heal). - SignatureDivergent — all committed but signatures diverge, or mix signed (<=10-version) with unsigned (>10-version) disks (heal). - Unknown — all committed, no signature produced (>10 versions); latent divergence is left to the scanner backstop, not self-enqueued. `RenameConvergence::needs_heal()` is the single decision point. The version signature is now only comparison material; it no longer doubles as a heal flag. The old `select_rename_data_versions` / `reduce_common_versions` / `rename_data_versions_key` machinery that carried the conflation is removed. The heal submission in `complete_multipart_upload` moves off the ACK critical path into a detached task: it runs after the object lock is dropped and after the durable `rename_data` commit, survives cancellation of the completion future, and coalesces through the existing bounded / deduplicated / observable heal-channel admission (one submit per degraded completion, at most). A completion cancelled in the narrow window between the durable commit and reaching the enqueue is scanner-backstopped, as is the Unknown (>10-version) case. The PUT path (`object.rs`) binds the second element as `_` and is unchanged. The change is orthogonal to and composes with the #1312 commit fence on the same `rename_data` path (epoch rejection is a commit-gate failure surfaced through `Result::Err`, convergence is a post-commit signal); documented in docs/architecture/unified-object-generation.md. Tests: `classify_rename_convergence` white-box cases cover the full acceptance matrix (healthy 4/4 and 8/8, 3-same-1-divergent, failed/offline disk, no-common-quorum split, >10-version all-success and with-failure, mixed signed/unsigned) and fail on revert to the old "signature exists => heal" semantics. The decision function is tested directly rather than through the process-global heal channel, whose receiver is owned exclusively by the blackbox serial test (init_heal_channel is once per binary). Refs: https://github.com/rustfs/backlog/issues/1321
This commit is contained in:
@@ -1433,7 +1433,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
// The trailing `_` drops the rename_data old-size backfill
|
||||
// (rustfs/backlog#1009): CompleteMultipartUpload keeps its pre-commit
|
||||
// `get_object_info` lookup, so the backfill has no consumer here yet.
|
||||
let (online_disks, versions, op_old_dir, cleanup_disks, _) = Self::rename_data(
|
||||
let (online_disks, convergence, op_old_dir, cleanup_disks, _) = Self::rename_data(
|
||||
&shuffle_disks,
|
||||
RUSTFS_META_MULTIPART_BUCKET,
|
||||
&upload_id_path,
|
||||
@@ -1485,17 +1485,46 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
|
||||
|
||||
drop(object_lock_guard); // drop object lock guard to release the lock
|
||||
|
||||
if let Some(versions) = versions {
|
||||
let _ =
|
||||
rustfs_common::heal_channel::send_heal_request(rustfs_common::heal_channel::create_heal_request_with_options(
|
||||
bucket.to_string(),
|
||||
Some(object.to_string()),
|
||||
false,
|
||||
Some(HealChannelPriority::Normal),
|
||||
Some(self.pool_index),
|
||||
Some(self.set_index),
|
||||
))
|
||||
// backlog#1321: enqueue heal only when the committed replicas actually
|
||||
// need to converge — a partial commit (some disk failed/offline) or a
|
||||
// signature divergence between committed replicas. A fully healthy MPU
|
||||
// (identical signatures on every disk) is `AllSuccessIdentical` and
|
||||
// submits nothing, which is the fix: the old `Option::is_some()` gate
|
||||
// treated the mere existence of a version signature as "needs heal", so
|
||||
// every healthy <=10-version completion self-enqueued.
|
||||
//
|
||||
// The submit is detached (`tokio::spawn`) so it stays off the ACK
|
||||
// critical path AND survives cancellation of the completion future: the
|
||||
// write is already durable and ACK-worthy, so the heal admission must
|
||||
// not ride the client's request lifetime. The admission itself is
|
||||
// bounded / deduplicated / observable (`send_heal_request` ->
|
||||
// `HealAdmissionResult`), so this emits at most one submit per
|
||||
// completion and coalesces with any in-flight heal for the same object.
|
||||
//
|
||||
// Scanner backstop (backlog#1321 patch): a `PartialCommit` whose
|
||||
// completion is cancelled in the narrow window after the durable commit
|
||||
// but before this spawn runs is not lost — the divergence it would have
|
||||
// healed is exactly what the background scanner reconciles. `Unknown`
|
||||
// (>10 versions, no signature produced) likewise relies on the scanner
|
||||
// rather than self-enqueuing.
|
||||
if convergence.needs_heal() {
|
||||
let bucket = bucket.to_string();
|
||||
let object = object.to_string();
|
||||
let pool_index = self.pool_index;
|
||||
let set_index = self.set_index;
|
||||
tokio::spawn(async move {
|
||||
let _ = rustfs_common::heal_channel::send_heal_request(
|
||||
rustfs_common::heal_channel::create_heal_request_with_options(
|
||||
bucket,
|
||||
Some(object),
|
||||
false,
|
||||
Some(HealChannelPriority::Normal),
|
||||
Some(pool_index),
|
||||
Some(set_index),
|
||||
),
|
||||
)
|
||||
.await;
|
||||
});
|
||||
}
|
||||
|
||||
let upload_id_path = upload_id_path.clone();
|
||||
|
||||
Reference in New Issue
Block a user