docs(ilm): document persistence recovery contracts (#7106)

docs/operations/tier-ilm-debugging.md

@@ -2,7 +2,7 @@
 
 **Use this when:** a tiered (transitioned) object returns `NoSuchVersion` or fails to restore, a transition run looks stuck, or you need to inspect `xl.meta` and trace the versionId sent to the remote tier.
 
-**Source of truth:** `crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs` (transition/expiry actions), `crates/utils/src/http/metadata_compat.rs` (dual-key helpers), `crates/filemeta/src/filemeta/version.rs` (`SUFFIX_TRANSITIONED_VERSION_ID` read pattern), `crates/filemeta/examples/dump_fileinfo.rs`.
+**Source of truth:** `crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs` (transition/expiry actions, fenced free-version cleanup, and durable-job recovery), `crates/ecstore/src/bucket/lifecycle/manual_transition_job.rs` (job/task/result records), `rustfs/src/admin/handlers/ilm_transition.rs` (manual run/status/cancel and transition-transaction reconcile routes), `crates/utils/src/http/metadata_compat.rs` (dual-key helpers), `crates/filemeta/src/filemeta/version.rs` (`SUFFIX_TRANSITIONED_VERSION_ID` read pattern), `crates/filemeta/examples/dump_fileinfo.rs`.
 
 ## Code map
 
@@ -13,7 +13,10 @@
 | `WarmBackend` trait (put/get/remove/in_use) | `crates/ecstore/src/services/tier/warm_backend.rs` |
 | Per-provider tier backends (S3, MinIO, GCS, Azure, ...) | `crates/ecstore/src/services/tier/warm_backend_*.rs` |
 | Remote-tier sweep (`delete_object_from_remote_tier`) | `crates/ecstore/src/bucket/lifecycle/tier_sweeper.rs` |
-| Persisted free-version recovery (remote cleanup after local-first expiry) | `crates/ecstore/src/bucket/lifecycle/tier_free_version_recovery.rs` |
+| Persisted free-version scan and re-enqueue after local-first expiry | `crates/ecstore/src/bucket/lifecycle/tier_free_version_recovery.rs` |
+| Fenced free-version remote delete, local-marker cleanup, and rescan | `crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs` (`cleanup_free_version_exact`) |
+| Durable manual transition job/task/result records | `crates/ecstore/src/bucket/lifecycle/manual_transition_job.rs` |
+| Manual run/status/cancel and transition-transaction reconcile admin routes | `rustfs/src/admin/handlers/ilm_transition.rs` |
 | `ObjectInfo` / `TransitionedObject` types | `crates/ecstore/src/object_api/types.rs` |
 | `FileMeta` / `FileInfo` / version metadata | `crates/filemeta/src/` |
 | Dual-key internal metadata helpers (`insert_bytes` / `get_bytes`) | `crates/utils/src/http/metadata_compat.rs` |
@@ -81,12 +84,23 @@ POST /rustfs/admin/v3/ilm/transition/run?bucket=<bucket>&prefix=<prefix>&tier=<t
 |---|---|
 | `bucket` | Required. |
 | `prefix`, `tier`, `dryRun` | Narrow the run. |
+| `mode` | `enqueue_only` by default; set `async` for a durable background job. |
+| `async` | Compatibility boolean for selecting durable async mode. It must not conflict with `mode`. |
 | `maxObjects` | Defaults to `10000`, capped at `100000`. |
 | `maxDurationSeconds` | Optional, capped at `3600`; a best-effort budget checked between listed object versions and pages. An in-flight listing call is not cancelled. |
 
-The current contract is `enqueue_only`: the response reports what this bounded scan evaluated and enqueued into the in-memory transition queue. `state=completed` means the bounded scan reached the end of its current scope without queue pressure or budget truncation; it does not mean every remote tier PUT has completed. `state=partial` means the run stopped early on `maxObjects`, `maxDurationSeconds`, or queue pressure (`skipped_queue_full`, `skipped_queue_closed`, or `skipped_queue_timeout`).
+In the default `enqueue_only` mode, the response reports what this bounded scan evaluated and enqueued into the in-memory transition queue. `state=completed` means the bounded scan reached the end of its current scope without queue pressure or budget truncation; it does not mean every remote tier PUT has completed. `state=partial` means the run stopped early on `maxObjects`, `maxDurationSeconds`, or queue pressure (`skipped_queue_full`, `skipped_queue_closed`, or `skipped_queue_timeout`). Its response omits job endpoints.
 
-`job_id` and `status_endpoint` are `null`. There is no durable background job, restart recovery cursor, cluster-wide single-flight admission, status polling endpoint, or cancel endpoint for this trigger. Re-running is safe only in the normal lifecycle sense: in-flight object versions are deduplicated in the local process, but separate nodes do not share a durable manual-run admission record.
+With `mode=async` or `async=true`, a successful request returns `202 Accepted`, `mode=durable_job`, a UUID `job_id`, and status/cancel endpoints. Durable mode provides bucket-level persisted admission, restart recovery, a lease-fenced listing checkpoint, task-before-enqueue records, worker-result records, and persisted cancellation. It still delegates each object to the normal lifecycle evaluator and transition transaction; a job never bypasses object eligibility or owns remote deletion.
+
+```text
+GET    /rustfs/admin/v3/ilm/transition/jobs/<job-id>
+DELETE /rustfs/admin/v3/ilm/transition/jobs/<job-id>
+```
+
+GET returns the durable job state and report. DELETE persists `cancel_requested`, which stops new scan work for an active executor. Startup recovery reconciles task/results first only when `scan_completed = true`. An expired job whose scan is incomplete and cancellation is requested is directly CAS-terminalized as `cancelled`, so its stored counters may be stale; scope release follows as a best-effort exact delete. Non-cancelled recovery instead CAS-takes over the lease, replaces the scope admission, replays pending tasks, and resumes object-version scanning from the persisted token. A conflicting live durable job for the same bucket and run/dry-run class returns the active job and its endpoint. The async endpoint itself does not query a fleet capability gate and a direct request proceeds to durable job creation. Before enabling durable mode across a mixed fleet, caller/operator orchestration must check the `manual_transition_jobs` runtime capability on every required node and fail closed if any response is unknown or unsupported.
+
+The persisted states, ownership and recovery rules are specified in [../architecture/ilm-tiering-persistence-contracts.md](../architecture/ilm-tiering-persistence-contracts.md). Terminal job/task/result history currently has no automatic retention bound, so status data remains available until a future protocol-level collector is defined.
 
 Recommended operator flow (external `rc` CLI):
 
@@ -132,6 +146,6 @@ Historical transition transactions in `upload_outcome_unknown` state can use an
 
 ## Invariant: local-first expiry ordering
 
-`expire_transitioned_object` deletes local metadata first (making the object unreachable) and leaves remote-tier cleanup to persisted free-version recovery (`crates/ecstore/src/bucket/lifecycle/tier_free_version_recovery.rs`). Never remove a remote tier version while live local metadata still points at it: doing so lets a concurrent GET read a stored version_id whose remote version is already gone and fail with `NoSuchVersion`.
+`expire_transitioned_object` deletes local metadata first (making the object unreachable) and leaves a persisted free-version for remote-tier cleanup. `tier_free_version_recovery.rs` scans and re-enqueues that record; the lifecycle worker's `cleanup_free_version_exact` in `bucket_lifecycle_ops.rs` performs the fenced remote delete, local-marker cleanup, and rescan. Never remove a remote tier version while live local metadata still points at it: doing so lets a concurrent GET read a stored version_id whose remote version is already gone and fail with `NoSuchVersion`.
 
 Regression test: `serial_tests::test_expire_transitioned_object_never_races_concurrent_get` in `crates/scanner/tests/lifecycle_integration_test.rs` (CI ILM Integration serial lane) pins both the local-first ordering and the "concurrent GET never sees `NoSuchVersion`" contract.