gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-10-14 08:30:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(lifecycle): coordinate tier free-version recovery scans (#8387)

Browse Source 
* fix(lifecycle): coordinate tier free-version recovery scans

* fix(lifecycle): remove obsolete recovery enqueue entry point

* test(upgrade): wait for writers after each rolling replacement
This commit is contained in:
cxymds
2026-10-08 21:24:34 +08:00
committed by GitHub
parent 5b4432620e
commit d386f7741d
5 changed files with 437 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/operations/tier-ilm-debugging.md
+7 -1
View File
@@ -36,11 +36,17 @@ An expired restored local copy can be cleaned up under Object Lock because the r
The dedicated free-version recovery loop is enabled by default and is independent of the data scanner and heal switches. Setting `RUSTFS_SCANNER_ENABLED=false` does not stop this repair loop. Set `RUSTFS_TIER_FREE_VERSION_RECOVERY_ENABLED=false` before process startup to disable only the dedicated persisted-marker walk. That setting does not disable lifecycle workers or prevent another scanner path from discovering a free version, and it can leave remote cleanup markers pending for longer, so use it as a break-glass pressure control rather than a cleanup mechanism.
Enabled nodes compete for the independent `tier-free-version-recovery/leader.lock` namespace lock in the internal metadata bucket. The leader holds the renewable lock across pages, cooldowns, retries, and idle waits; followers only attempt ownership with a one-second acquisition budget and a random 15–30 second delay between attempts. This uses the existing pool 0 locker majority, so recovery pauses when that lock domain cannot form quorum even if another pool remains available. Recovery resumes when ownership can be acquired again. It does not depend on the main scanner's leader lock or enable switch.
Loss of the leader lease or process shutdown cancels discovery and waits for active walks to stop before the coordinator exits that leadership term. A successor starts a new sweep from the beginning because continuation markers remain process-local; frequent leadership changes can therefore delay discovery near the end of a large namespace. Already queued cleanup tasks retain their existing exact-version and object-lock checks and may finish after leadership changes. During a rolling upgrade, older nodes still perform independent walks until they are upgraded or recovery is disabled on them.
Recovery discoveries enter the leader's local lifecycle worker queues. Queue pressure preserves the failed object's retry position; spare workers on other nodes do not automatically take those tasks. Recovery notifications are also process-local: a failed enqueue or cleanup on a follower is rediscovered through the leader's periodic sweeps rather than immediately waking it. The ten-minute idle cap bounds the idle gap, not the total time to discover a marker, which also includes page cooldowns and traversal time. Persisted free-version markers remain the retry source of truth.
Normal transitioned deletes pass a post-commit receipt directly to the lifecycle queue. The namespace walk is the crash, queue-pressure, mixed-version, and historical-record fallback. It cannot safely skip a bucket merely because that bucket has no current lifecycle rule or the referenced tier was removed: an older `xl.meta` free-version can still be the only owner of a required remote DELETE.
One background recovery page completes at most 10,000 logical objects across at most 100 buckets and enqueues at most 1,000 recoverable free versions. The scanner can decode one additional object to detect truncation; a continuation marker preserves the first unscanned bucket or the last completely scanned object. Every truncated page and follow-up sweep waits at least 60 seconds after the previous page completes; failed pages back off from 60 seconds to 10 minutes, and complete idle sweeps exponentially back off to 10 minutes with jitter. Individual walks have no fixed total timeout, but inherit the drive walk stall timeout so a large healthy bucket can make progress without recreating the old timeout/restart loop.
The structured `lifecycle_worker_state` recovery event reports `duration_ms`, `scanned_entries`, `buckets_scanned`, queue counts, truncation, and continuation markers. `rustfs_internal_stage_duration_ms{stage="lifecycle_free_version_recovery"}` records successful page duration; `stage="lifecycle_free_version_recovery_failed"` records failures.
The structured `lifecycle_worker_state` recovery event reports `duration_ms`, `scanned_entries`, `buckets_scanned`, queue counts, truncation, and continuation markers. Coordinator states include `free_version_recovery_leader_acquired`, `free_version_recovery_leader_lost`, and `free_version_recovery_leader_acquire_failed`; normal follower contention is quiet. `rustfs_internal_stage_duration_ms{stage="lifecycle_free_version_recovery"}` records successful page duration; `stage="lifecycle_free_version_recovery_failed"` records failures. When comparing cluster load, include lock acquisition/refresh RPCs and the leader's cleanup backlog as well as namespace traversal work.
## Metadata key conventions