fix(heal): retain terminal outcomes across restart (#7805)

* fix(heal): retain terminal outcomes across restart

* test(heal): use octal mode for terminal recovery fixture

* fix(ecstore): reopen healthy hedged readers after peer loss

* test(ecstore): isolate tier overwrite recovery scheduling

* test(ecstore): observe tier cleanup under object read locks

* fix(ecstore): bound decommission entry tracing

* test(ecstore): drain incarnation heal fixture writes

* test(heal): match debug server stack in deep heal fixtures

* test(heal): include identities in C06 count failures

* test(heal): drain PUT tails before inspecting B920 fixtures

* test(scanner): isolate retained MRF retry slots

* fix(ecstore): separate PUT cleanup intent from persisted metadata

docs/architecture/README.md

@@ -35,6 +35,7 @@ Required headings and strings in these files are asserted by `scripts/check_arch
 | [erasure-coding.md](erasure-coding.md) | changing anything under `crates/ecstore/src/erasure/`, `crates/filemeta/`, `crates/ecstore/src/set_disk/`, storage-class or layout code, or any decode, quorum, or heal boundary (normative spec) |
 | [placement-repair-invariants.md](placement-repair-invariants.md) | changing anything that resolves an object to a pool, set, or disk, or that admits scanner or heal work |
 | [heal-concurrency-model.md](heal-concurrency-model.md) | changing heal, PUT/multipart commit, delete, lifecycle expiry, or data-movement code that shares the `(bucket, object)` commit surface, or asking whether RustFS needs a persistent healing marker |
+| [heal-terminal-reports.md](heal-terminal-reports.md) | changing retained admin heal outcomes, terminal report publication, cancellation refinement, or report upgrade/rollback behavior |
 | [unified-object-generation.md](unified-object-generation.md) | adding or changing anything that fences a commit, scopes a read lease, gates old-directory cleanup, binds prepared pool reads, or settles quota against the current object version |
 | [ilm-tiering-persistence-contracts.md](ilm-tiering-persistence-contracts.md) | changing an ILM transition, tier configuration mutation, manual transition job, tier-delete recovery path, pool decommission, or any code that can create, transfer, or destroy ownership of a remote-tier object |
 | [decommission-compatibility.md](decommission-compatibility.md) | changing pool decommission or rebalance behavior, its admin API shape, the persisted `PoolMeta` fields, or how tier free versions move between pools |

docs/architecture/compat-cleanup-register.md

@@ -11,6 +11,7 @@
 
 ## Open Items
 
+- `backlog-2519` retained admin heal reports: keep the schema-1 terminal as the commit and replay fence, with a bounded versioned report in a separate namespace on the same disk. Older rollback readers can still query the terminal and suppress replay, but cannot expose its outcome; newer readers mark missing outcomes as unavailable. Remove the legacy marker and missing-report adapter only after all supported direct-upgrade and rollback readers understand the report format and retained schema-1-only receipts have expired.
 - `odm-list-bare-envelope` historical ODM continuation tokens: preserve complete bare v1/v2 envelopes. Framed issuance defaults on for the deployed framed-only generation; upgrades from older bare-only readers must explicitly disable it before starting new nodes and keep it off until reader convergence. Remove the legacy classifier and framing issuance override only after every supported reader accepts framing and outstanding bare listings have drained or clients explicitly restarted them; tokens have no automatic expiry. Exact full-envelope object keys remain intrinsically ambiguous during this compatibility period.
 - `backlog-2263` legacy heal MRF inspection: retained per-record journals remain readable while committed-snapshot ownership and writer activation are staged. Remove legacy import only after all supported direct-upgrade and rollback readers understand committed snapshots and migration tooling confirms that no retained or restorable legacy journal requires it. This does not enable a new writer or change the automatic legacy consumer.
 - `backlog-1337` legacy restore orphan recovery: releases that predate the restore worker-lock marker can leave a valid operation-id and `ongoing-request="true"` after cancellation or process failure, with no durable liveness proof. New servers allow an exact, non-nil legacy generation to be superseded only when its consistently parsed request date is at least 24 hours old. Remove the clock-based legacy fallback after the minimum supported direct-upgrade release writes the v1 worker-lock marker on every restore and operators have resolved every retained pre-v1 ongoing generation.

docs/architecture/heal-terminal-reports.md

@@ -0,0 +1,29 @@
+# Retained Admin Heal Reports
+
+Admin heal tokens retain their terminal status for ten minutes from the original completion time. The persistence owner is `RootHealRecovery` in `crates/heal/src/heal/manager/root_recovery.rs`; the bounded report codec is in `crates/heal/src/heal/manager/root_recovery/report.rs`.
+
+## Publication and Recovery
+
+The original schema-1 `terminal-root-heal-<token>.json` remains the commit marker. Its separate `heal-terminal-report-<token>.json` report has its own schema version and embeds the exact terminal identity. Both files belong to the same coordinator disk as the pending intent.
+
+Publication writes the report through the storage owner's conditional file update, writes the terminal marker, and then conditionally removes the pending intent. Recovery never treats a report without its marker as committed. A committed terminal continues to suppress stale pending work even when its report is unreadable. An initial publication failure leaves the existing responsibility available for recovery.
+
+The report preserves the canonical execution, traversal coverage, cumulative counters, diagnostic object window, progress, legacy result window, truncation flags, and incremental cursors. Restoring it neither recounts objects nor derives counters from progress. Its embedded terminal must match the retained marker before the report can be returned.
+
+## Cancellation
+
+An active cancellation first publishes an `aborted/cancelled` partial snapshot before retiring active ownership. The worker can subsequently finish recording its last object. Its final report replaces the earlier report, while the terminal marker and original completion timestamp remain unchanged. Cancelling a retry preserves any retained preceding attempt's outcome; unavailable historical counters remain unavailable.
+
+If the final report update fails, the scheduler rereads the disk because an error can occur after publication. It returns that persisted report when readable. If the result cannot be resolved, the cached response exposes no canonical outcome or result window and marks the detail as truncated. It does not assume that the previous report won an uncertain write.
+
+## Bounds and Retention
+
+Report encoding and reading each enforce an 8 MiB JSON limit. The canonical diagnostic window retains at most 128 objects and 64 KiB of object storage accounting, with at most 1 KiB per detail. The legacy result window retains the existing 1 MiB memory budget. Decoding validates the report version, terminal identity, terminal execution, counters, object bounds, and cursor ordering, and reconstructs memory accounting.
+
+GC uses the original completion timestamp, including after repeated restarts or cancellation refinement. For an expired terminal, it removes stale pending work before the report and marker. Report deletions share the existing 64-deletion budget. Expired orphan reports left before marker publication or by rollback GC are removed without deleting pending responsibility. Corrupt records are retained and reported as errors.
+
+## Upgrade and Rollback
+
+Schema-1 terminals without reports remain queryable. A terminal response without a canonical outcome adds `outcomeStatus: "unavailable"`; it does not fabricate counters or complete coverage. Missing historical result windows are marked truncated. Responses with an outcome and running responses do not add this field.
+
+Older binaries retain their original terminal decoder and ignore the distinct report namespace. They continue to query status and prevent replay, but do not expose the new report's outcome. A later upgrade can read a retained report again; reports whose markers were collected by the old binary are handled as uncommitted orphans. Rollback therefore preserves the old status/replay contract, not the new outcome capability.