gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-09-07 12:35:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(ilm): execute legacy recovery dispositions (#7304)

Browse Source 
* feat(ilm): execute legacy recovery dispositions

* feat(ilm): retry retained transition recovery (#7308)

* fix(admin): use gateway errors for recovery retries

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
This commit is contained in:
cxymds
2026-09-07 09:46:43 +08:00
committed by GitHub
parent b0f68f5d0a
commit a70c96d520
13 changed files with 3811 additions and 216 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/operations/tier-ilm-debugging.md
+34 -1
View File
@@ -173,7 +173,7 @@ Historical transition transactions in `upload_outcome_unknown` state can use an
## Inspect and disposition retained recovery records
This section describes an **approved target that is not implemented yet**. Current servers do not expose the routes below and continue to quarantine tier-delete journal v1/v2 records. Do not remove internal metadata objects by hand: that loses ETag, all-pool, decommission, export, and audit guarantees.
Current servers expose the routes below for retained recovery controls. Do not remove internal metadata objects by hand: that loses ETag, all-pool, decommission, export, and audit guarantees.
The approved read-only inventory is bounded and paginated:
@@ -223,6 +223,39 @@ Malformed/unsupported records and journal v3-v6 cannot use abandon. Known-versio
Automatic retry state survives restart. Retryable transport/quorum failures use a 60-second exponential base capped at one hour and a deterministic 80-to-100-percent multiplier, so jitter never increases the capped delay. After 32 consecutive failures or seven days from the first persisted failure, automatic work stops at `operator_required`. Unsupported or ambiguous evidence goes directly to `retained_ambiguous`/`operator_required`; age alone never deletes it. Resolved controls, immutable exports, and completed disposition receipts have minimum 30-day, 90-day, and 365-day retention respectively, and are collected only after exact source absence, decommission, successor, and audit checks.
### Retry a retained transition transaction
For a `transition_transaction` control, inspect returns an additional `transition_retry` object when the exact transaction source and recovery-control generation are still consistent. It contains `retry_ready`, `control_revision`, `source_generation_sha256`, the current classification and counters, and a bounded refusal reason. A missing `transition_retry` with `transition_retry_not_ready_reason=source_or_control_not_ready` means the server could not reconstruct exact live evidence; do not retry from an older response.
First perform a dry-run with the exact revision and source-generation digest returned by the latest inspect:
```json
POST /rustfs/admin/v3/ilm/recovery/records/<control-id>
{
"action": "retry_transition_recovery",
"mode": "dry_run",
"expected_control_revision": 7,
"expected_source_generation_sha256": "<sha256>"
}
```
After repairing the reported storage, tier, or capability problem, repeat inspect and dry-run, then execute with the newly observed values:
```json
POST /rustfs/admin/v3/ilm/recovery/records/<control-id>
{
"action": "retry_transition_recovery",
"mode": "execute",
"expected_control_revision": 7,
"expected_source_generation_sha256": "<sha256>",
"confirm": true
}
```
Execution performs one ETag-CAS update of the exact ownerless `retained_ambiguous` or `operator_required` control to `retrying`. It preserves the lifetime attempt count and failure history, clears only the consecutive-failure backoff, and does not mutate the transaction source or issue a tier PUT, GET, probe, or DELETE. The normal recovery worker then acquires a fresh bounded owner lease and repeats every source and remote proof before any side effect.
A historical v1 `UploadStarted` record can return to `retained_ambiguous` because its bytes do not prove whether PUT reached the provider. `LocalCommitStarted` becomes terminal only when the local object still matches the recorded version ID, data directory, modification time, size, ETag, and exact transitioned remote tuple; otherwise it returns to `operator_required`. Retrying is therefore a bounded re-evaluation after an underlying repair, not an override of missing evidence.
The full schema, lease, mixed-version, retry, privacy, and metric requirements are in [../architecture/ilm-tiering-persistence-contracts.md](../architecture/ilm-tiering-persistence-contracts.md#bounded-recovery-control-and-operator-disposition).
## Reconcile legacy transition-version metadata