rustfs/docs/architecture/heal-concurrency-model.md

Heal concurrency model

Use this when: changing heal, PUT/multipart commit, delete, lifecycle expiry, or data-movement code that touches the same (bucket, object) commit surface; or evaluating whether RustFS needs a persistent per-object healing marker like MinIO's x-minio-healing. Source of truth: crates/ecstore/src/set_disk/ops/heal.rs (heal_object_with_explicit_version_regen, HealObjectLockKind, HEAL_RENAME_INCOMPLETE), crates/ecstore/src/set_disk/ops/object.rs (PUT/DELETE lock sections, reconcile_old_data_cleanup_receipts), crates/ecstore/src/set_disk/core/io_primitives.rs (commit_rename_data_dir, report_old_data_dir_cleanup, reclaim_orphan_data_dirs), crates/filemeta/src/fileinfo.rs (FileInfo::set_healing), crates/heal/src/heal/manager/queue.rs (dedup keys).

For crate ownership, read crate-boundaries.md: ECStore owns erasure-set repair primitives that share this lock and commit model, while crates/heal owns repair orchestration.

Model

Heal and every foreground or background write path serialize on the same object-level namespace write lock (a quorum lock RPC in distributed mode, the in-process lock manager on a single node; granularity is the object, the version component is always None), and heal holds its guard across the whole rename commit. This describes the intended lock scope while the guard remains valid; it does not prove rejection of an already-dispatched disk syscall after distributed lease loss. The authority, delayed-mutation, and recovery boundary is specified in unified-object-generation.md. MinIO's x-minio-healing marker is an out-of-lock defence against version-cleanup logic inside RenameData interleaving with a heal commit; RustFS's commit model has no such interleaving, so no persistent marker exists (x-minio-healing does not occur in crates/ or rustfs/) and none is needed. Three layers replace it:

Layer	Mechanism	Owner
In-lock mutual exclusion	Heal and all write-path commit points take the (bucket, object) namespace write lock.	acquire_heal_object_lock in crates/ecstore/src/set_disk/ops/heal.rs; lock sections in ops/object.rs and ops/multipart.rs
Commit-model isolation	rename_data contains no version cleanup that could interleave with heal. Physical deletion of a replaced old data_dir runs after the object lock is released (the commit tail) and only for unshared directories already superseded by the new commit.	commit_rename_data_dir in crates/ecstore/src/set_disk/core/io_primitives.rs
Transient healing flag	FileInfo::set_healing sets the internal SUFFIX_HEALING key on the in-memory FileInfo of a heal commit; rename_data reads it through is_healing to clear a stale non-empty target data_dir before the rename (in-place repair reuses the data_dir, and rename(2) cannot replace a non-empty directory). The key is never persisted (is_skip_meta_key in crates/filemeta/src/filemeta.rs). A non-heal commit that meets a non-empty target fails explicitly; tests lock both directions.	crates/filemeta/src/fileinfo.rs, crates/ecstore/src/disk/local.rs

Heal lock scope

Retired bucket delete markers

An explicit version heal may remove a subquorum delete marker from a deleted bucket generation only through heal_object_with_proof in crates/ecstore/src/store/heal.rs. The coordinator holds the bucket lifecycle read fence before entering the existing capacity and object lock scopes. The current, non-nil bucket incarnation must differ from the marker's persisted incarnation.

Normal single and batch marker creation stamps both x-rustfs-internal-bucket-incarnation-id and x-minio-internal-bucket-incarnation-id. A successful bucket deletion publishes an immutable, deployment-bound retirement record under the system bucket's bucket-retirements namespace, outside ordinary bucket metadata cleanup (crates/ecstore/src/bucket/retirement.rs). Publication follows successful physical deletion and the existing all-set rollback decision, precedes metadata cleanup and the successful DELETE response, and retains the lifecycle guards through the metadata write. Failed physical deletion does not publish retirement authority. Failed or uncertain record publication returns an error; it cannot authorize cleanup without an authoritative read of the record. Records have no age-based expiration because offline members can return arbitrarily late.

Cleanup requires remove=true, dry_run=false, an explicit non-null version, a committed retirement record, and agreement on the full surviving marker metadata. The DeleteRetiredMarker RPC carries the full marker precondition in the existing authenticated request body. LocalDisk::delete_version_inner checks it after acquiring the metadata mutation lease and reading the actual xl.meta; a replaced marker or data version is preserved. Older nodes that do not implement this RPC reject it; coordinators do not fall back to DeleteVersion.

Every selected disk must be online and confirm exact version absence after cleanup. Changed topology, lost fences, failed mutations, unreadable evidence, legacy markers without an incarnation, and conflicting identities produce no completion receipt. Unproven markers remain deferred with retired_marker_proof; a normal not-found error is not globally reclassified as success. An interrupted operation can leave partial cleanup, and replay checks the remaining actual replicas before producing an absence receipt. Deployments upgraded after an old deletion without stamps or retirement evidence retain those residual markers for separate recovery; new successful DELETE operations provide the evidence needed for automatic convergence.

ListObjectVersions independently reads all reachable replicas and resolves each version with the full erasure-set metadata read quorum (SetDisks::resolve_listed_versions in crates/ecstore/src/set_disk/metadata.rs). Sample settings in RUSTFS_LIST_OBJECTS_QUORUM continue to govern ordinary object listing; they do not lower version-listing authority. Local reader page limits cannot truncate this discovery before the merged page is resolved. This increases metadata reads relative to sampled listing, but does not issue a separate GET for every version. Heal's union walk still discovers subquorum repair candidates.

heal_object delegates to heal_object_with_explicit_version_regen, which takes the namespace write lock at entry unless opts.no_lock is set and binds the guard to the function scope. The guard covers the quorum metadata read, EC reconstruction, per-disk rename commit, tmp cleanup, the HEAL_RENAME_INCOMPLETE partial-commit return, and orphan data_dir reclamation (reclaim_orphan_data_dirs).

Read-repair heals (opts.read_repair) hold a shared lock (HealObjectLockKind::Read) during reconstruction so readers keep flowing, then acquire_revalidated_read_repair_commit_lock takes the write lock and re-reads a commit fingerprint; a changed fingerprint aborts the commit (read_repair_commit_stale).

Lock-intersection matrix

#	Concurrent path	Lock held by that path	Outcome	Where
1	PUT commit	object write lock; rename_data inside it	serialized	ops/object.rs put commit
2	PUT old data_dir tail cleanup	none (runs after the lock is dropped)	unlocked, semantically safe (commit tail)	commit_rename_data_dir in core/io_primitives.rs
3	DELETE object or version	object write lock; delete_version inside it	serialized	ops/object.rs delete_object
4	Batch DELETE	per-object write locks (batch lock RPC in distributed mode)	serialized	ops/object.rs delete_objects
5	CompleteMultipartUpload	object write lock plus upload-path lock; rename inside	serialized	ops/multipart.rs
6	CompleteMultipart tail cleanup	none (after lock drop)	unlocked, semantically safe (commit tail)	ops/multipart.rs
7	AbortMultipartUpload	upload-path lock in the multipart bucket only	disjoint resources: abort never touches the object data_dir or xl.meta	ops/multipart.rs
8	ILM expiry including DeleteAllVersions	delete_prefix_object=true keeps the object lock; FreeVersionTask locks explicitly; noncurrent batches use batch locks	serialized	crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs
9	Pure prefix delete	delete_prefix without delete_prefix_object takes no child-object lock	unlocked; no production caller (prefix delete)	ops/object.rs lock condition in delete_object
10	Orphan data_dir reclamation	none inside the function; its only production caller runs inside the heal lock	serialized within heal	reclaim_orphan_data_dirs in core/io_primitives.rs
11	Old-cleanup receipt reconciliation	none inside the function; caller runs inside the heal lock and an epoch fence rejects stale receipts	serialized	reconcile_old_data_cleanup_receipts in ops/object.rs
12	Replication	data plane writes to the remote over HTTP; local metadata write-back takes the object lock	serialized or disjoint	crates/ecstore/src/bucket/replication/replication_resyncer.rs
13	Data movement, rebalance, decommission source cleanup	explicit object lock plus version-unchanged recheck; no_lock only reuses an already-held guard	serialized	crates/ecstore/src/data_movement/mod.rs
14	CopyObject	destination object lock through the PUT chain	serialized	ops/object.rs copy_object
15	Another heal task (different HealType, or force_start)	dedup keys are per HealType and force_start skips dedup, so tasks may coexist	serialized on the namespace write lock	make_dedup_key_for_type in crates/heal/src/heal/manager/queue.rs
16	Admin heal with nolock=true	caller bypasses the lock	unlocked by operator choice (no_lock)	rustfs/src/admin/handlers/heal.rs
17	Stale multipart cleanup	upload-path lock in the multipart bucket	disjoint resources	crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

Residual windows

Commit tail cleanup

Rows 2 and 6. After a write path commits and releases the object lock, it best-effort deletes the replaced old data_dir; the code deliberately does not block the next operation on this. The deletion can race a concurrent heal reading or rebuilding that same old data_dir, but the race is semantically safe:

• The target is an unshared data_dir already replaced by the new commit. Heal's canonical metadata comes from quorum arbitration (ETag, mod time), and quorum already points at the new version, so heal cannot resurrect the replaced version as canonical.
• The worst outcome is one transient failure or no-op for the heal round on the old version; the next round converges. Cleanup residue is reported and re-queued for heal via report_old_data_dir_cleanup.
• Long heals such as drive replacement request explicit versions and read quorum metadata inside the lock, so the tail does not affect them.

Pure prefix delete

Row 9. delete_prefix && !delete_prefix_object takes no child-object locks (an object namespace lock cannot protect a recursive prefix delete), so a heal running during the prefix delete could theoretically rebuild a version from stale quorum metadata. Every production delete_prefix: true call site also sets delete_prefix_object: true (and therefore takes the object lock); the remaining delete_prefix-only call sites are in test modules. A future caller that needs a pure prefix delete must prove isolation from heal and scanner at the call site (for example a bucket-level scan fence).

no_lock and force_start

Row 16. Admin heal requests pass the client's nolock parameter through (rustfs/src/admin/handlers/heal.rs), matching the MinIO madmin option. Setting it is an explicit operator choice that accepts races with concurrent writes; it is documented, not restricted.

Heal-side invariants that hold regardless of the caller:

• Dedup keys are disjoint across HealType (object, metadata, MRF, EC decode, prefix), and admin force_start skips dedup. Several heal tasks for one object can therefore exist at once, but every production entry calls heal_object with no_lock=false, so their execution bodies serialize on the namespace write lock.
• Read-repair's local TTL reservation dedups only its own source and does not block heals from other sources; the namespace lock is the backstop.
• The healing flag is never persisted, so there is no reverse risk of a leftover marker making a later commit yield incorrectly.

Graceful root-heal restart recovery

Before a graceful shutdown cancels administrator cluster-wide heals, the manager saves unfinished requests on one coordinator disk as .rustfs.sys/root-heal-<task-id>.json. Startup replays the same task IDs and remaining execution budgets. Completion, cancellation, and replacement by force_start retire the record conditionally. An uncertain write or deletion does not create a fallback copy; an unsuccessful handoff retains the unclean-shutdown marker.

Startup and administrator admission serialize through the same gate while recovery records are inventoried. A structurally invalid, unsupported, or oversized root intent or terminal receipt is retained and fenced by a separate quarantined-root-heal-{intent|terminal}-<task-id>.json marker on its owner disk. The marker binds the task ID, record kind, bounded source-prefix length, SHA-256 digest, and oversized state. A quarantined task ID cannot be reused, but its unknown scope does not prevent an independent compensation request with a fresh ID. Unavailable disks, duplicate owners, malformed quarantine markers, and source records that no longer match their marker remain fail-closed and defer root recovery without blocking the existing replacement-recovery path.

This handoff covers the same coordinator and storage topology while the record disk remains configured and readable. It does not migrate records when a pool is retired or provide failover after loss of that disk. Older versions do not understand these records; cancellation while downgraded cannot retire a newer version's pending record. If a terminal budget checkpoint cannot be written, the previous record is retained and a warning is logged; the remaining-budget guarantee requires that write to succeed. The format is separate from object metadata and erasure-set checkpoints.

Regression tests

Both live in the test module of crates/ecstore/src/set_disk/ops/heal.rs:

Test	Invariant
heal_racing_version_delete_never_resurrects_the_deleted_version	With a doomed version's shards corrupted, a versioned DELETE and a deep heal contend on the same lock; the deleted version is not resurrected and the surviving version is intact.
heal_racing_unversioned_overwrites_preserves_the_last_commit	Unversioned overwrite commits (exercising the commit-tail old data_dir deletion) race a deep-heal loop; the final current version is exactly the last commit (ETag-level equality).

Related: the atomic-commit and best-effort-rollback invariants for the write path are in erasure-coding.md.