fix(lifecycle): make tier cleanup recoverable (#3491)

scripts/test/tier_lifecycle_cleanup_stress.md

@@ -0,0 +1,45 @@
+# Tier Lifecycle Cleanup Stress Test
+
+## Goal
+
+Verify concurrent GET, ILM expiry, and tier free-version recovery do not produce `NoSuchVersion`, unbounded queue growth, or severe throughput regression.
+
+## Setup
+
+1. Start a RustFS cluster with a configured remote tier.
+2. Upload at least 10,000 objects and transition them to the remote tier.
+3. Configure lifecycle expiry for the transitioned prefix.
+4. Enable lifecycle logs:
+
+```bash
+export RUST_LOG="info,rustfs_ecstore::bucket::lifecycle=debug"
+```
+
+## Workload
+
+Run concurrent GET while triggering expiry:
+
+```bash
+seq 1 64 | xargs -I{} -P64 sh -c 'while true; do aws s3api get-object --bucket "$BUCKET" --key "prefix/obj-$((RANDOM % 10000))" /tmp/out.$$ >/dev/null 2>>get-errors.log; done'
+```
+
+## Required Checks
+
+- `get-errors.log` contains no `NoSuchVersion`.
+- Lifecycle queue pending count does not grow without bound.
+- Missed free-version and tier journal counters may increase during injected failures, but later recovery drains persisted records.
+- Recovery logs show cursor advancement across bucket and object markers.
+- Duplicate recovery enqueue count stays bounded while remote tier delete is failing.
+- Queue full or backpressure stops the current recovery pass instead of blocking normal lifecycle enqueue.
+- Scan duration and scanned metadata entry count stay within the configured per-pass budget.
+- p95 GET latency and lifecycle expiry throughput are recorded before and after the fix.
+
+## Fault Injection
+
+Repeat the workload while:
+
+- stopping lifecycle workers
+- forcing remote tier delete to fail
+- restarting the RustFS process after local delete but before remote cleanup
+
+After recovery, verify remote tier garbage does not remain for expired objects.