mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-22 10:33:25 +00:00
fix(heal): retire stale delete markers after bucket recreation (#7743)
* fix(heal): prove completed historical version cleanup * test(heal): use debug runtime stack for C06 regression * fix(heal): retire stale delete markers after bucket recreation * test(ecstore): fix Clippy in retired marker regressions
This commit is contained in:
@@ -17,6 +17,18 @@ Heal and every foreground or background write path serialize on the same object-
|
||||
|
||||
## Heal lock scope
|
||||
|
||||
### Retired bucket delete markers
|
||||
|
||||
An explicit version heal may remove a subquorum delete marker from a deleted bucket generation only through `heal_object_with_proof` in `crates/ecstore/src/store/heal.rs`. The coordinator holds the bucket lifecycle read fence before entering the existing capacity and object lock scopes. The current, non-nil bucket incarnation must differ from the marker's persisted incarnation.
|
||||
|
||||
Normal single and batch marker creation stamps both `x-rustfs-internal-bucket-incarnation-id` and `x-minio-internal-bucket-incarnation-id`. A successful bucket deletion publishes an immutable, deployment-bound retirement record under the system bucket's `bucket-retirements` namespace, outside ordinary bucket metadata cleanup (`crates/ecstore/src/bucket/retirement.rs`). Publication follows successful physical deletion and the existing all-set rollback decision, precedes metadata cleanup and the successful DELETE response, and retains the lifecycle guards through the metadata write. Failed physical deletion does not publish retirement authority. Failed or uncertain record publication returns an error; it cannot authorize cleanup without an authoritative read of the record. Records have no age-based expiration because offline members can return arbitrarily late.
|
||||
|
||||
Cleanup requires `remove=true`, `dry_run=false`, an explicit non-null version, a committed retirement record, and agreement on the full surviving marker metadata. The `DeleteRetiredMarker` RPC carries the full marker precondition in the existing authenticated request body. `LocalDisk::delete_version_inner` checks it after acquiring the metadata mutation lease and reading the actual `xl.meta`; a replaced marker or data version is preserved. Older nodes that do not implement this RPC reject it; coordinators do not fall back to `DeleteVersion`.
|
||||
|
||||
Every selected disk must be online and confirm exact version absence after cleanup. Changed topology, lost fences, failed mutations, unreadable evidence, legacy markers without an incarnation, and conflicting identities produce no completion receipt. Unproven markers remain deferred with `retired_marker_proof`; a normal not-found error is not globally reclassified as success. An interrupted operation can leave partial cleanup, and replay checks the remaining actual replicas before producing an absence receipt. Deployments upgraded after an old deletion without stamps or retirement evidence retain those residual markers for separate recovery; new successful DELETE operations provide the evidence needed for automatic convergence.
|
||||
|
||||
`ListObjectVersions` independently reads all reachable replicas and resolves each version with the full erasure-set metadata read quorum (`SetDisks::resolve_listed_versions` in `crates/ecstore/src/set_disk/metadata.rs`). Sample settings in `RUSTFS_LIST_OBJECTS_QUORUM` continue to govern ordinary object listing; they do not lower version-listing authority. Local reader page limits cannot truncate this discovery before the merged page is resolved. This increases metadata reads relative to sampled listing, but does not issue a separate GET for every version. Heal's union walk still discovers subquorum repair candidates.
|
||||
|
||||
`heal_object` delegates to `heal_object_with_explicit_version_regen`, which takes the namespace write lock at entry unless `opts.no_lock` is set and binds the guard to the function scope. The guard covers the quorum metadata read, EC reconstruction, per-disk rename commit, tmp cleanup, the `HEAL_RENAME_INCOMPLETE` partial-commit return, and orphan `data_dir` reclamation (`reclaim_orphan_data_dirs`).
|
||||
|
||||
Read-repair heals (`opts.read_repair`) hold a shared lock (`HealObjectLockKind::Read`) during reconstruction so readers keep flowing, then `acquire_revalidated_read_repair_commit_lock` takes the write lock and re-reads a commit fingerprint; a changed fingerprint aborts the commit (`read_repair_commit_stale`).
|
||||
|
||||
Reference in New Issue
Block a user