fix(ecstore): recover interrupted delete transactions (#8468)

* fix(ecstore): recover crash-interrupted delete retries

Retain set-quorum cleanup proof and restore metadata before dropping rollback
markers. Reclaim eligible exact-key residue during explicit delete retries,
with bucket identity, version, retention and namespace fencing guards.

Preserve prefix children and snapshot ownership through batched physical
cleanup, and cover interrupted transactions and failure boundaries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): route recovery test imports through contracts

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): preserve missing-key deletes at quorum

* feat(ecstore): add certified set-wide delete recovery

Atomically prepare deletion journals and exact targets, publish durable set
receipts, and recover through one fenced disk/RPC core used by startup and
scoped administrator healing. Preserve returning generations, ordinary
missing-key quorum semantics, and legacy rollback wire compatibility.

Retain namespace and volume ownership through late physical operations,
order related transactions causally, and keep proof until full convergence.
Remove the superseded local-only recovery path and add storage, protocol,
RPC, durability and isolated Docker regression coverage.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): reclaim completed multipart delete sidecars

* fix(ecstore): recover multipart sidecars with verified proofs

Recognize native multipart sidecars at the shared recovery boundary and
restore them without treating metadata or proof files as shard payloads.
Validate required proof generations and bind every present live/staged
shard copy to its metadata-bound Merkle proof before restoring metadata.

Cover protected native MPU rollback, split source layouts, missing or
corrupt proofs, and payload corruption with recomputed bitrot checksums.
Add real multipart Docker regressions in both integrity modes.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): settle superseded committed version deletes

Allow a validated committed decision to finish its own data cleanup after
a later decision removes current metadata. Prepared copies need an
inspected same-transaction receipt matching immutable local intent before
promotion, and retain their local journal without importing peer hashes.

Reuse the authority matcher for dry-run and settlement. Cover chained
explicit-version deletions, partial receipt publication, mismatch rejection
and no-resurrection, while retaining the no-decision absence protection.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): clarify delete recovery case label

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): restore recovery CI guard coverage

Keep production error sites visible to the ratchet by placing test modules last. Preserve native encoding errors, group preparation identity and ownership, and reduce RPC enum size without changing the wire payload.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(admin): preserve owned OIDC validation errors

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): unify recovery authority and snapshot work

Use one user-bucket legacy settlement boundary and one adoption source predicate for preview and execution. Reuse immutable snapshot digests, consume owned generation metadata, index version membership, and prepare empty witnesses through the canonical journal path.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): preserve delete recovery invariants

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>

docs/operations/delete-transaction-recovery.md

@@ -0,0 +1,137 @@
+# Delete transaction recovery
+
+An interrupted object deletion can leave metadata backups, reserved shards, or
+an unpublished target. Normal S3 listings cannot discover all of these files.
+Recovery therefore inspects raw transaction records on every configured drive
+in the owning erasure set under the bucket incarnation fence and exact-key
+write lock. It does not modify GET or PUT behavior.
+
+## Decisions and replay
+
+New deletions assemble the backup, versioned prepared journal, and exact target
+in the private system temporary area, then atomically publish the complete
+transaction directory before modifying metadata. Interrupted preparation cannot
+leave an undiscoverable target-only record or temporary file in the object key. After the metadata write quorum, the coordinator publishes durable
+commit receipts to a write quorum before returning success. A failed or lost
+publication response is indeterminate: recovery must inspect the persisted
+records instead of immediately restoring the backup.
+
+| Evidence | Action |
+| --- | --- |
+| New prepared transaction, no decision, intact original shards and matching generation | Restore and sync the original metadata before removing preparation evidence. |
+| Structured commit receipt | Publish the certified target to matching returning participants and finish deletion. |
+| Missing, unavailable, or unsupported participant | Preserve records and retry when every configured participant can be inspected. |
+| Newer generation or partially overlapping version history | Preserve records and report blocked recovery. |
+| Historical backup or disk-local marker without a structured set decision | Preserve records during automatic recovery. |
+
+Missing local metadata does not authorize user-bucket residue cleanup, even
+when the current object exists on a metadata quorum. Legacy local cleanup is
+restricted to internal system-bucket maintenance. User-bucket legacy residue
+uses only the guarded set-wide removal described below. Dry-run and execution
+share the same source-generation check; an already-absent explicit non-null
+version can converge without touching unrelated live versions.
+
+Rollback preserves native multipart metadata and integrity sidecars. Protected
+parts require their declared proof generation; recovery verifies both the proof
+records and actual shard payloads before restoring metadata or removing
+recovery evidence. Missing or corrupt required proofs leave the transaction
+blocked with its backup intact.
+
+A disk-local `delete-data.<T>` marker does not prove an erasure-set decision.
+Returning drives cannot restore stale data after a certified deletion. Commit
+receipts remain until every participant and related transaction for the key
+has converged. For consecutive certified version deletions, an older decision
+can finish its own data cleanup after later deletion removes current metadata.
+A prepared copy needs a matching same-transaction decision before promotion;
+absence alone cannot certify it. Cross-transaction null-version supersession also requires causal
+predecessor evidence; timestamps alone cannot authorize it. Cleanup respects active snapshots and removes only validated
+transaction files, with directory removal performed nonrecursively.
+
+## Entry points
+
+Unclean-shutdown erasure-set healing performs raw transaction recovery before
+ordinary object listing. Administrator bucket, set, prefix, and object heal use the
+same recovery core. Existing heal selection, lifecycle admission, namespace
+locking, and movement guards still apply. Prefix recovery settles only matching
+keys, including transactions hidden from listings. Dry-run inspects and reports without
+settling transactions.
+
+An unconditional DELETE of a missing exact key, or an administrator heal with
+`remove` enabled, can authorize historical residue removal only when the
+bucket is authoritatively never-versioned and has no Object Lock. Every
+configured drive must be inspected, the current object must be absent, and all
+residue must match the guarded legacy layout. Versioned/suspended buckets,
+retained versions, conditions, unknown entries, symlinks, and prefix children
+are not covered by that authorization. The same fresh intent can certify the
+absence of an unprotected new prepared null-version transaction when every
+configured drive has no current metadata. This is a new authorized decision;
+it does not infer an old commit from missing causal evidence. Mixed or
+unrecognized historical layouts remain blocked.
+
+A missing specific-version DELETE first recovers hidden transactions and then
+reads the current version again. It must not acknowledge an unresolved hidden
+version as an idempotent no-op.
+
+## Upgrade and failure handling
+
+Upgrade all participating nodes before relying on the new protocol. Recovery
+uses one authenticated additive RPC; unsupported peers cannot silently accept
+new flags through the legacy delete endpoint. An online older participant
+causes the modern operation to fail before decision publication. Modern nodes
+also reject legacy forward transactional user-bucket deletes. A cluster still
+controlled by an older majority retains the older behavior. Do not downgrade
+while recovery records remain; old binaries cannot interpret the new protocol.
+
+Keep unavailable drives and transaction directories intact. Bring the full
+set online, run a scoped heal in dry-run mode, inspect its recovery result,
+and then run the same scoped heal without dry-run. Use `remove` for ambiguous
+historical residue only after verifying that the exact objects are intended
+to be deleted and the bucket meets the restrictions above. Do not recursively
+remove transaction directories by hand to make a bucket appear empty.
+
+Recovery reports discovered, committed, rolled-back, reclaimed historical,
+and blocked transaction counts. Unavailable participants return a retryable
+quorum error; blocked evidence requires operator investigation. A successful
+S3 delete response does not imply immediate physical cleanup while snapshots
+or returning participants still retain necessary proof.
+
+## Settlement deadline and durability
+
+`RUSTFS_DELETE_RECOVERY_TIMEOUT_SECS` controls the settlement RPC deadline,
+including full shard verification during rollback. Key and bucket lifecycle
+ownership stay attached to late local filesystem operations after cancellation
+or timeout. The default is 900 seconds;
+valid values are 30 through 86400 seconds. Invalid values fail explicitly.
+Increase it when intact large objects need more verification time. It does not
+change normal object read or delete-apply deadlines. A timeout can conceal a
+completed mutation, so rerun inspection and recovery rather than guessing the
+outcome from the RPC error.
+
+The records and restores follow the configured durability policy. Strict mode
+syncs the required metadata and directory changes. Process SIGKILL testing
+checks restart behavior; it does not establish power-loss behavior on every
+filesystem or storage device. Mode `none` intentionally offers weaker sync
+guarantees. Do not treat a throughput comparison as a durability substitute.
+
+## Reproduce the Docker regression suite
+
+Build the candidate with the test-only barriers enabled:
+
+```sh
+cargo build --locked --release -p rustfs --bin rustfs --no-default-features --features e2e-test-hooks
+python3 scripts/test/delete_recovery_docker.py --help
+```
+
+Pass a native Linux binary with `--binary`, a Linux image containing Python and
+curl with `--image`, and a unique Docker namespace. The optional
+`--baseline-binary` enables the mixed-version case. The harness uses synthetic
+S3 objects, localhost ports, labeled isolated volumes, and a non-root server.
+It stops only its own containers and keeps JSON results and logs in the chosen
+artifact directory. Production builds must not enable `e2e-test-hooks`.
+
+Run the preactivation/prepare/apply/decision process-stop cases, interrupted restore,
+returning and untouched peers, both generation orderings, version/retention
+and prefix protection, administrator dry-run/removal, response-loss,
+permission/corruption, and mixed-version cases. Preserve the source identity
+of both binaries with the results. A passing process-stop matrix does not
+cover hardware power cuts, Windows runtime behavior, or DELETE throughput.