mirror of
https://github.com/rustfs/rustfs.git
synced 2026-10-13 16:11:56 +00:00
fix(heal): bound durable MRF rescheduling (#8351)
Reuse stable live-owner proof anchors to avoid duplicate in-flight admission. Pace unverified completions and unavailable attempts with bounded jitter while retaining checkpoints, exact proofs, and generation isolation. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com>
This commit is contained in:
@@ -406,6 +406,12 @@ Rediscovery and admission observations update the recorded result but do not pos
|
||||
|
||||
With MRF enabled, a partial commit waits up to ten seconds for its checkpoint receipt. Fully converged writes do not enter this path or add MRF journal writes. The consumer batches available submissions and uses the existing count/byte budgets; an offline member or exhausted hint retry count does not evict an admitted partial-write obligation. New responsibility for the same unversioned object invalidates the previous lease, and a task that already started cannot accept that new responsibility as a merged proof target.
|
||||
|
||||
Durable partial-write replay uses the manager's full proof anchor (including bucket incarnation and ingress lease) to observe queued, running, and retrying ownership. A due record already owned by the manager is not submitted again. Ownership is observed at the existing admission-backoff interval, which defaults to five seconds. Task-ID changes caused by retry merging do not change the responsibility identity. A notice is treated as in-flight only while its task has a live queued, running, or retrying owner. If those state locks are busy, the consumer falls back to ordinary manager admission rather than assuming that a task completed or indefinitely suppressing a retry.
|
||||
|
||||
After an admitted attempt ends without verified proof, the responsibility remains in the journal and its next attempt uses bounded exponential backoff with per-responsibility jitter. With the default five-second base, successive delay ranges are 5 seconds, 5–10, 10–20, 20–40, then 30–60 seconds. A larger explicitly supplied admission interval is not shortened. Full or already-running admission uses the existing admission interval and does not increment the unverified-completion streak. Bucket identity lookup or submission errors use the same bounded failure pacing.
|
||||
|
||||
Verified-proof and unverified-legacy events are consumed on the existing checkpoint tick even while a responsibility is backing off. A matching proof can therefore discharge responsibility without waiting for the next retry deadline. This is a retry delay bound, not a repair-completion deadline. New leases, replay after restart, and explicit operator rechecks start fresh runtime pacing; same-lease duplicate admission retains it. Rechecks still require their successor checkpoint before dispatch. Pacing state is not added to the journal format, does not consume the best-effort hint attempt limit, and never authorizes dropping a durable obligation.
|
||||
|
||||
Disabled/unavailable delivery, full queues, invalid identities, persistence failures and receipt timeouts are not durable admission. An already committed object is not rolled back: when its source bucket incarnation is known, the producer falls back to the in-memory Heal channel with that same generation fence. If the source incarnation is unavailable, it does not dispatch an unfenced heal that could mutate a recreated bucket. When MRF is enabled, a failed admission attempt also records `mrf_durable_admission_failed`. An admission timeout does not cancel a record already retained by the consumer. These failure cases therefore do not promise that every successful S3 response has a durable repair obligation. Early-ACK PUTs can also return before their rename tail settles; the tail submits responsibility when its final outcome requires repair. The MRF switch is independent of automatic disk scanning, so returning members can be repaired with scanner and auto-heal disabled.
|
||||
|
||||
Legacy notices carry only bucket/object/version, not a verified storage disposition, incarnation, scope, or durable responsibility generation. They are drained without clearing hints. Terminal callbacks release only their exact node-local ingress lease so rediscovery remains possible; lease generations are not durable successor receipts. Pending migration staging is not activated, and this change does not enable durable tombstones or garbage collection. Positive cleanup requires a storage-owner receipt with the complete responsibility identity and validated commit/fence evidence; neither task status nor the bounded diagnostic outcome window supplies it.
|
||||
|
||||
Reference in New Issue
Block a user