fix(ecstore): recover interrupted delete transactions (#8468)

* fix(ecstore): recover crash-interrupted delete retries

Retain set-quorum cleanup proof and restore metadata before dropping rollback
markers. Reclaim eligible exact-key residue during explicit delete retries,
with bucket identity, version, retention and namespace fencing guards.

Preserve prefix children and snapshot ownership through batched physical
cleanup, and cover interrupted transactions and failure boundaries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): route recovery test imports through contracts

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): preserve missing-key deletes at quorum

* feat(ecstore): add certified set-wide delete recovery

Atomically prepare deletion journals and exact targets, publish durable set
receipts, and recover through one fenced disk/RPC core used by startup and
scoped administrator healing. Preserve returning generations, ordinary
missing-key quorum semantics, and legacy rollback wire compatibility.

Retain namespace and volume ownership through late physical operations,
order related transactions causally, and keep proof until full convergence.
Remove the superseded local-only recovery path and add storage, protocol,
RPC, durability and isolated Docker regression coverage.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): reclaim completed multipart delete sidecars

* fix(ecstore): recover multipart sidecars with verified proofs

Recognize native multipart sidecars at the shared recovery boundary and
restore them without treating metadata or proof files as shard payloads.
Validate required proof generations and bind every present live/staged
shard copy to its metadata-bound Merkle proof before restoring metadata.

Cover protected native MPU rollback, split source layouts, missing or
corrupt proofs, and payload corruption with recomputed bitrot checksums.
Add real multipart Docker regressions in both integrity modes.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): settle superseded committed version deletes

Allow a validated committed decision to finish its own data cleanup after
a later decision removes current metadata. Prepared copies need an
inspected same-transaction receipt matching immutable local intent before
promotion, and retain their local journal without importing peer hashes.

Reuse the authority matcher for dry-run and settlement. Cover chained
explicit-version deletions, partial receipt publication, mismatch rejection
and no-resurrection, while retaining the no-decision absence protection.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): clarify delete recovery case label

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): restore recovery CI guard coverage

Keep production error sites visible to the ratchet by placing test modules last. Preserve native encoding errors, group preparation identity and ownership, and reduce RPC enum size without changing the wire payload.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(admin): preserve owned OIDC validation errors

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): unify recovery authority and snapshot work

Use one user-bucket legacy settlement boundary and one adoption source predicate for preview and execution. Reuse immutable snapshot digests, consume owned generation metadata, index version membership, and prepare empty witnesses through the canonical journal path.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): preserve delete recovery invariants

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>

crates/e2e_test/src/reliant/grpc_lock_server.rs

@@ -465,6 +465,13 @@ impl NodeService for MinimalLockNodeService {
         Err(Status::unimplemented("lock-only test server"))
     }
 
+    async fn recover_delete_transaction(
+        &self,
+        _request: Request<rustfs_protos::proto_gen::node_service::RecoverDeleteTransactionRequest>,
+    ) -> Result<Response<rustfs_protos::proto_gen::node_service::RecoverDeleteTransactionResponse>, Status> {
+        Err(Status::unimplemented("delete recovery is not supported by the lock fixture"))
+    }
+
     async fn settle_part_transaction(
         &self,
         _request: Request<rustfs_protos::proto_gen::node_service::SettlePartTransactionRequest>,

crates/ecstore/src/api/mod.rs

@@ -423,11 +423,12 @@ pub mod disk {
     pub use crate::disk::os::{LocalPublicationPause, LocalPublicationStage};
     pub use crate::disk::{
         BATCH_READ_VERSION_MAX_ITEMS, BUCKET_META_PREFIX, BatchReadVersionItem, BatchReadVersionReq, BatchReadVersionResp,
-        CheckPartsResp, ConditionalFileUpdate, DeleteOptions, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption,
-        DiskStore, FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize,
-        PartTransactionAction, RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp,
-        STORAGE_FORMAT_FILE, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, new_disk,
-        validate_batch_read_version_item_count,
+        CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION, DeleteOptions, DeleteRecoveryAction,
+        DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection, DeleteRecoveryPage, DeleteRecoveryRpcOperation,
+        DeleteRecoveryRpcReply, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption, DiskStore, FileInfoVersions,
+        FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize, PartTransactionAction,
+        RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, STORAGE_FORMAT_FILE,
+        SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, new_disk, validate_batch_read_version_item_count,
     };
     pub use bytes::Bytes;
     pub use endpoint::Endpoint;

crates/ecstore/src/cluster/rpc/client.rs

@@ -124,6 +124,22 @@ pub async fn node_service_time_out_client_for_class(
         .max_encoding_message_size(max_message_size))
 }
 
+pub(crate) async fn node_service_delete_recovery_client(
+    addr: &str,
+    interceptor: TonicInterceptor,
+    timeout: std::time::Duration,
+) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, DiskError> {
+    let interceptor = interceptor.with_rpc_audience(addr).map_err(DiskError::from)?;
+    let channel = rustfs_protos::create_channel_with_timeout(addr, timeout)
+        .await
+        .map_err(|error| DiskError::RemoteClientUnavailable(error.to_string()))?;
+    let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
+    let max_message_size = rustfs_protos::internode_rpc_max_message_size();
+    Ok(NodeServiceClient::with_interceptor(channel, interceptor)
+        .max_decoding_message_size(max_message_size)
+        .max_encoding_message_size(max_message_size))
+}
+
 pub async fn node_service_time_out_client_no_auth(
     addr: &String,
 ) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
@@ -528,6 +544,92 @@ mod tests {
     use tracing_opentelemetry::OpenTelemetrySpanExt;
     use tracing_subscriber::{Registry, layer::SubscriberExt};
 
+    #[tokio::test]
+    async fn delete_recovery_channel_uses_its_scoped_deadline() {
+        use bytes::Bytes;
+        use http_body_util::StreamBody;
+        use hyper::{body::Frame, service::service_fn};
+        use hyper_util::rt::{TokioExecutor, TokioIo};
+        use rustfs_protos::proto_gen::node_service::{RecoverDeleteTransactionRequest, RecoverDeleteTransactionResponse};
+        use tonic_prost::prost::Message;
+        let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
+            .await
+            .expect("bind recovery peer");
+        let addr = format!("http://{}", listener.local_addr().expect("peer address"));
+        let server = tokio::spawn(async move {
+            loop {
+                let (stream, _) = listener.accept().await.expect("accept recovery connection");
+                tokio::spawn(async move {
+                    let service = service_fn(|_request: HttpRequest<hyper::body::Incoming>| async {
+                        tokio::time::sleep(std::time::Duration::from_millis(500)).await;
+                        let payload = RecoverDeleteTransactionResponse {
+                            success: true,
+                            protocol_version: 1,
+                            result: Bytes::new(),
+                            error: None,
+                        }
+                        .encode_to_vec();
+                        let mut data = vec![0];
+                        data.extend_from_slice(&u32::try_from(payload.len()).expect("small response").to_be_bytes());
+                        data.extend_from_slice(&payload);
+                        let mut trailers = http::HeaderMap::new();
+                        trailers.insert("grpc-status", "0".parse().expect("status"));
+                        let frames: Vec<Result<Frame<Bytes>, std::convert::Infallible>> =
+                            vec![Ok(Frame::data(data.into())), Ok(Frame::trailers(trailers))];
+                        let mut response = HttpResponse::new(StreamBody::new(futures::stream::iter(frames)));
+                        response
+                            .headers_mut()
+                            .insert("content-type", "application/grpc".parse().expect("content type"));
+                        Ok::<_, std::convert::Infallible>(response)
+                    });
+                    let _ = hyper::server::conn::http2::Builder::new(TokioExecutor::new())
+                        .serve_connection(TokioIo::new(stream), service)
+                        .await;
+                });
+            }
+        });
+        let request = || RecoverDeleteTransactionRequest {
+            disk: "disk".into(),
+            volume: "bucket".into(),
+            protocol_version: 1,
+            operation: Bytes::new(),
+        };
+        // Seed an existing short-lived control channel to reproduce the inherited deadline.
+        let control = tonic::transport::Endpoint::from_shared(addr.clone())
+            .expect("test endpoint")
+            .timeout(std::time::Duration::from_millis(50))
+            .connect()
+            .await
+            .expect("control connect");
+        rustfs_common::cache_connection(addr.clone(), control).await;
+        let mut foreground = node_service_time_out_client(&addr, TonicInterceptor::NoOp(NoOpInterceptor))
+            .await
+            .expect("control client");
+        let error = foreground
+            .recover_delete_transaction(request())
+            .await
+            .expect_err("short control deadline");
+        assert!(matches!(error.code(), tonic::Code::Cancelled | tonic::Code::DeadlineExceeded));
+        // The 2-second test override bypasses the production range parser, which rejects 2.
+        let mut recovery = node_service_delete_recovery_client(
+            &addr,
+            TonicInterceptor::NoOp(NoOpInterceptor),
+            std::time::Duration::from_secs(2),
+        )
+        .await
+        .expect("scoped recovery client");
+        assert!(
+            recovery
+                .recover_delete_transaction(request())
+                .await
+                .expect("scoped deadline must survive control deadline")
+                .into_inner()
+                .success
+        );
+        server.abort();
+        rustfs_protos::evict_failed_connection(&addr).await;
+    }
+
     #[tokio::test]
     async fn rpc_client_factories_do_not_reuse_https_channels_without_tls_generation() {
         let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("reserve local address");

crates/ecstore/src/cluster/rpc/remote_disk.rs

@@ -14,7 +14,8 @@
 
 use crate::cluster::rpc::client::{
     AuthenticatedChannel, TonicInterceptor, gen_tonic_signature_interceptor, is_network_like_disk_error,
-    node_service_time_out_client, node_service_time_out_client_for_class, node_service_time_out_client_no_auth,
+    node_service_delete_recovery_client, node_service_time_out_client, node_service_time_out_client_for_class,
+    node_service_time_out_client_no_auth,
 };
 use crate::cluster::rpc::internode_data_transport::{
     InternodeDataTransport, NsScannerCapabilityRequest, NsScannerStreamRequest, ReadStreamRequest, WalkDirStreamRequest,
@@ -22,10 +23,11 @@ use crate::cluster::rpc::internode_data_transport::{
 };
 use crate::disk::error::{Error, Result};
 use crate::disk::{
-    BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DeleteOptions, DiskAPI, DiskInfo,
-    DiskInfoOptions, DiskLocation, DiskOption, FileInfoVersions, FileReader, FileWriter, PartTransactionAction, ReadMultipleReq,
-    ReadMultipleResp, ReadOptions, RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions,
-    batch_read_version_one_by_one,
+    BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION,
+    DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection,
+    DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation,
+    DiskOption, FileInfoVersions, FileReader, FileWriter, PartTransactionAction, ReadMultipleReq, ReadMultipleResp, ReadOptions,
+    RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, batch_read_version_one_by_one,
     disk_store::{
         DEFAULT_RUSTFS_DRIVE_ACTIVE_MONITORING, ENV_RUSTFS_DRIVE_ACTIVE_MONITORING, SKIP_IF_SUCCESS_BEFORE,
         get_drive_active_check_interval, get_drive_active_check_timeout, get_drive_disk_info_timeout, get_drive_list_dir_timeout,
@@ -59,10 +61,10 @@ use rustfs_protos::proto_gen::node_service::{
     CompareAndUpdateFileRequest, DeletePathsRequest, DeleteRequest, DeleteVersionRequest, DeleteVersionsRequest,
     DeleteVersionsResponse, DeleteVolumeRequest, DiskInfoRequest, ListDirRequest, ListVolumesRequest, MakeVolumeRequest,
     MakeVolumesRequest, PreparePartTransactionRequest, ReadAllRequest, ReadMetadataRequest, ReadMultipleRequest,
-    ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, RenameDataRequest, RenameFileRequest,
-    SettlePartTransactionRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest, SnapshotLeaseRequest,
-    SnapshotLeaseResponse, StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest, WriteMetadataRequest,
-    node_service_client::NodeServiceClient,
+    ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, RecoverDeleteTransactionRequest,
+    RenameDataRequest, RenameFileRequest, SettlePartTransactionRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest,
+    SnapshotLeaseRequest, SnapshotLeaseResponse, StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest,
+    WriteMetadataRequest, node_service_client::NodeServiceClient,
 };
 use serde::{Serialize, de::DeserializeOwned};
 use std::{
@@ -142,6 +144,45 @@ fn decode_delete_versions_errors(response: DeleteVersionsResponse, expected_len:
         .collect()
 }
 
+fn decode_delete_recovery_batch_errors(codes: Vec<Option<u32>>, expected_len: usize) -> Result<Vec<Option<DiskError>>> {
+    if codes.len() != expected_len {
+        return Err(Error::other("malformed recoverable batch delete response length"));
+    }
+    codes
+        .into_iter()
+        .map(|code| match code {
+            None => Ok(None),
+            Some(code) => DiskError::from_u32(code)
+                .map(Some)
+                .ok_or_else(|| Error::other("invalid recoverable batch delete error code")),
+        })
+        .collect()
+}
+
+fn validate_delete_recovery_inspection(
+    volume: &str,
+    object: &str,
+    transaction: Uuid,
+    inspection: &DeleteRecoveryInspection,
+) -> Result<()> {
+    if inspection.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
+        || inspection.id.object != object
+        || inspection.id.transaction != transaction
+        || transaction.is_nil()
+    {
+        return Err(Error::other("delete recovery inspection does not match request"));
+    }
+    if let Some(journal) = &inspection.journal
+        && (journal.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
+            || journal.volume != volume
+            || journal.object != object
+            || journal.transaction != transaction)
+    {
+        return Err(Error::other("delete recovery journal does not match request"));
+    }
+    Ok(())
+}
+
 fn snapshot_lease_token_from_response(response: SnapshotLeaseResponse) -> Result<SnapshotLeaseToken> {
     if !response.success {
         return Err(response.error.unwrap_or_default().into());
@@ -858,6 +899,75 @@ fn spawn_control_channel_prewarm(addr: String) {
 }
 
 impl RemoteDisk {
+    async fn recover_delete_transaction(
+        &self,
+        volume: &str,
+        operation: DeleteRecoveryRpcOperation,
+    ) -> Result<DeleteRecoveryRpcReply> {
+        let payload = rmp_serde::to_vec_named(&operation)?;
+        if payload.len() > operation.max_wire_payload_bytes() {
+            return Err(Error::other("delete recovery operation exceeds payload limit"));
+        }
+        let settle_timeout = if matches!(&operation, DeleteRecoveryRpcOperation::Settle { .. }) {
+            Some(crate::disk::get_delete_recovery_timeout()?)
+        } else {
+            None
+        };
+        self.execute_with_timeout(
+            || async {
+                let mut client = if let Some(timeout) = settle_timeout {
+                    if let Some(error) = self.offline_bypass_error() {
+                        return Err(error);
+                    }
+                    node_service_delete_recovery_client(
+                        &self.addr,
+                        TonicInterceptor::Signature(gen_tonic_signature_interceptor()),
+                        timeout,
+                    )
+                    .await?
+                } else {
+                    self.get_client().await?
+                };
+                let mut request = Request::new(RecoverDeleteTransactionRequest {
+                    disk: self.endpoint.to_string(),
+                    volume: volume.to_owned(),
+                    protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
+                    operation: payload.into(),
+                });
+                let canonical_body = rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref())
+                    .map_err(|_| Error::other("recover_delete_transaction request length cannot be represented"))?;
+                crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, &canonical_body).map_err(Error::other)?;
+                // UNIMPLEMENTED is deliberately propagated: older peers cannot settle transactions.
+                let response = client
+                    .recover_delete_transaction(request)
+                    .await
+                    .map_err(|status| {
+                        if status.code() == Code::Unimplemented {
+                            DiskError::UnsupportedDisk
+                        } else {
+                            status.into()
+                        }
+                    })?
+                    .into_inner();
+                if response.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION {
+                    return Err(DiskError::UnsupportedDisk);
+                }
+                if !response.success {
+                    return Err(response
+                        .error
+                        .ok_or_else(|| Error::other("missing delete recovery error"))?
+                        .into());
+                }
+                if response.error.is_some() {
+                    return Err(Error::other("malformed delete recovery success"));
+                }
+                rustfs_protos::decode_delete_recovery_payload(&response.result).map_err(Into::into)
+            },
+            settle_timeout.unwrap_or_else(get_max_timeout_duration),
+        )
+        .await
+    }
+
     async fn rename_file_with_durability(
         &self,
         src_volume: &str,
@@ -1920,6 +2030,12 @@ fn encode_msgpack<T: Serialize>(value: &T) -> Result<Vec<u8>> {
     encode_msgpack_with_capacity(value, MSGPACK_ENCODE_CAPACITY_HINT)
 }
 
+// Optional fields are omitted by serde; named maps keep rollback flags and
+// transaction UUIDs independent of field order across peer versions.
+fn encode_delete_options_msgpack(value: &DeleteOptions) -> Result<Vec<u8>> {
+    Ok(rmp_serde::to_vec_named(value)?)
+}
+
 fn encode_file_info_msgpack(value: &FileInfo) -> Result<Vec<u8>> {
     encode_msgpack_with_capacity(value, FILE_INFO_MSGPACK_ENCODE_CAPACITY_HINT)
 }
@@ -2566,7 +2682,7 @@ impl DiskAPI for RemoteDisk {
                 // `_bin` support for DeleteVersion is new (grpc-optimization P2); always dual-write
                 // JSON + msgpack until its fallback counter has read zero across a release window.
                 let file_info_bin = encode_file_info_msgpack(&fi)?;
-                let opts_bin = encode_msgpack(&opts)?;
+                let opts_bin = encode_delete_options_msgpack(&opts)?;
                 let conditional_marker = opts.expected_delete_marker.is_some();
                 let file_info = serde_json::to_string(&fi)?;
                 let opts = serde_json::to_string(&opts)?;
@@ -2638,7 +2754,7 @@ impl DiskAPI for RemoteDisk {
 
         // `_bin` support for DeleteVersions is new (grpc-optimization P2); always dual-write JSON +
         // msgpack until its fallback counter has read zero across a release window.
-        let opts_bin = match encode_msgpack(&opts) {
+        let opts_bin = match encode_delete_options_msgpack(&opts) {
             Ok(opts_bin) => opts_bin,
             Err(err) => {
                 let mut errors = Vec::with_capacity(versions.len());
@@ -3617,6 +3733,138 @@ impl DiskAPI for RemoteDisk {
     }
 
     #[tracing::instrument(level = "trace", skip_all)]
+    async fn delete_version_recoverable(
+        &self,
+        volume: &str,
+        object: &str,
+        file_info: FileInfo,
+        force_del_marker: bool,
+        opts: DeleteOptions,
+    ) -> Result<()> {
+        match self
+            .recover_delete_transaction(
+                volume,
+                DeleteRecoveryRpcOperation::ApplyVersion {
+                    object: object.to_owned(),
+                    file_info: Box::new(file_info),
+                    force_del_marker,
+                    opts,
+                },
+            )
+            .await?
+        {
+            DeleteRecoveryRpcReply::AppliedVersion => Ok(()),
+            _ => Err(Error::other("unexpected recoverable delete reply")),
+        }
+    }
+
+    async fn delete_versions_recoverable(
+        &self,
+        volume: &str,
+        versions: Vec<FileInfoVersions>,
+        opts: DeleteOptions,
+    ) -> Vec<Option<DiskError>> {
+        let expected_len = versions.len();
+        let result = match self
+            .recover_delete_transaction(volume, DeleteRecoveryRpcOperation::ApplyVersions { versions, opts })
+            .await
+        {
+            Ok(DeleteRecoveryRpcReply::AppliedVersions(codes)) => decode_delete_recovery_batch_errors(codes, expected_len),
+            Ok(_) => Err(Error::other("unexpected recoverable batch delete reply")),
+            Err(error) => Err(error),
+        };
+        match result {
+            Ok(errors) => errors,
+            Err(error) => vec![Some(error); expected_len],
+        }
+    }
+
+    async fn discover_delete_transactions(
+        &self,
+        volume: &str,
+        object: Option<&str>,
+        cursor: Option<DeleteRecoveryCursor>,
+        limit: u32,
+    ) -> Result<DeleteRecoveryPage> {
+        match self
+            .recover_delete_transaction(
+                volume,
+                DeleteRecoveryRpcOperation::Discover {
+                    object: object.map(str::to_owned),
+                    cursor,
+                    limit,
+                },
+            )
+            .await?
+        {
+            DeleteRecoveryRpcReply::Discovered(page) => {
+                if object.is_some_and(|object| page.transactions.iter().any(|id| id.object != object)) {
+                    return Err(Error::other("delete recovery discovery escaped requested object"));
+                }
+                if page.transactions.len() > usize::try_from(limit).map_err(|_| Error::other("invalid discovery limit"))?
+                    || page
+                        .transactions
+                        .iter()
+                        .any(|id| id.transaction.is_nil() || id.object.is_empty())
+                {
+                    return Err(Error::other("malformed delete discovery page"));
+                }
+                Ok(page)
+            }
+            _ => Err(Error::other("unexpected delete discovery reply")),
+        }
+    }
+    async fn inspect_delete_transaction(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: Uuid,
+    ) -> Result<DeleteRecoveryInspection> {
+        match self
+            .recover_delete_transaction(
+                volume,
+                DeleteRecoveryRpcOperation::Inspect {
+                    object: object.to_owned(),
+                    transaction,
+                },
+            )
+            .await?
+        {
+            DeleteRecoveryRpcReply::Inspected(inspection) => {
+                validate_delete_recovery_inspection(volume, object, transaction, &inspection)?;
+                Ok(inspection)
+            }
+            _ => Err(Error::other("unexpected delete inspection reply")),
+        }
+    }
+    async fn settle_delete_transaction(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: Uuid,
+        expected_fingerprint: DeleteRecoveryDigest,
+        action: DeleteRecoveryAction,
+    ) -> Result<DeleteRecoveryInspection> {
+        match self
+            .recover_delete_transaction(
+                volume,
+                DeleteRecoveryRpcOperation::Settle {
+                    object: object.to_owned(),
+                    transaction,
+                    expected_fingerprint,
+                    action,
+                },
+            )
+            .await?
+        {
+            DeleteRecoveryRpcReply::Settled(inspection) => {
+                validate_delete_recovery_inspection(volume, object, transaction, &inspection)?;
+                Ok(inspection)
+            }
+            _ => Err(Error::other("unexpected delete settlement reply")),
+        }
+    }
+
     async fn settle_part_transaction(&self, volume: &str, path: &str, action: PartTransactionAction) -> Result<()> {
         self.execute_with_timeout(
             || async {
@@ -4030,6 +4278,45 @@ mod tests {
 
     static INIT: Once = Once::new();
 
+    #[test]
+    fn recoverable_batch_delete_rejects_invalid_error_codes_and_length() {
+        let errors = decode_delete_recovery_batch_errors(vec![None, Some(DiskError::FileNotFound.to_u32())], 2)
+            .expect("valid per-item errors");
+        assert!(errors[0].is_none());
+        assert_eq!(errors[1], Some(DiskError::FileNotFound));
+        assert!(decode_delete_recovery_batch_errors(vec![None], 2).is_err());
+        assert!(decode_delete_recovery_batch_errors(vec![Some(0)], 1).is_err());
+        assert!(decode_delete_recovery_batch_errors(vec![Some(u32::MAX)], 1).is_err());
+    }
+
+    #[test]
+    fn delete_recovery_inspection_rejects_wrong_transaction_object_and_protocol() {
+        let transaction = Uuid::new_v4();
+        let inspection = DeleteRecoveryInspection {
+            protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
+            id: crate::disk::DeleteTransactionId {
+                object: "object".into(),
+                transaction,
+            },
+            state: crate::disk::DeleteRecoveryState::Absent,
+            journal: None,
+            current_digest: None,
+            current_versions: Vec::new(),
+            backup_digest: None,
+            rollback_data_complete: false,
+            commit_data_safe: false,
+            explicit_delete_safe: false,
+            fingerprint: [0; 32],
+        };
+        validate_delete_recovery_inspection("bucket", "object", transaction, &inspection).expect("matching inspection");
+        assert!(validate_delete_recovery_inspection("bucket", "other", transaction, &inspection).is_err());
+        assert!(validate_delete_recovery_inspection("bucket", "object", Uuid::new_v4(), &inspection).is_err());
+        assert!(validate_delete_recovery_inspection("bucket", "object", Uuid::nil(), &inspection).is_err());
+        let mut incompatible = inspection;
+        incompatible.protocol_version += 1;
+        assert!(validate_delete_recovery_inspection("bucket", "object", transaction, &incompatible).is_err());
+    }
+
     #[test]
     fn compare_and_update_wire_outcome_fails_closed() {
         assert_eq!(
@@ -4063,32 +4350,19 @@ mod tests {
     }
 
     #[test]
-    fn retired_marker_options_preserve_legacy_positional_wire_shape() {
+    fn delete_options_named_wire_preserves_rollback_and_legacy_marker_fields() {
         #[derive(Debug, serde::Serialize, serde::Deserialize, PartialEq)]
         struct LegacyDeleteOptions {
             recursive: bool,
             immediate: bool,
             undo_write: bool,
+            #[serde(default)]
             undo_delete: bool,
             old_data_dir: Option<Uuid>,
+            #[serde(default, skip_serializing_if = "Option::is_none")]
+            expected_delete_marker: Option<rustfs_filemeta::MetaDeleteMarker>,
         }
-        let legacy = LegacyDeleteOptions {
-            recursive: false,
-            immediate: false,
-            undo_write: false,
-            undo_delete: false,
-            old_data_dir: None,
-        };
-        let original = encode_msgpack(&legacy).unwrap();
-        let current = encode_msgpack(&DeleteOptions::default()).unwrap();
-        assert_eq!(current, original, "ordinary deletes must retain the older peer's positional payload");
-        assert_eq!(rmp_serde::from_slice::<LegacyDeleteOptions>(&current).unwrap(), legacy);
-        assert!(
-            rmp_serde::from_slice::<DeleteOptions>(&original)
-                .unwrap()
-                .expected_delete_marker
-                .is_none()
-        );
+        let transaction = Uuid::new_v4();
         let mut marker = FileInfo {
             deleted: true,
             version_id: Some(Uuid::new_v4()),
@@ -4097,12 +4371,69 @@ mod tests {
         };
         marker.set_delete_marker_incarnation(Uuid::new_v4());
         let marker = rustfs_filemeta::MetaDeleteMarker::from(marker);
+        for expected_delete_marker in [None, Some(marker)] {
+            for (undo_write, undo_delete) in [(true, false), (false, true)] {
+                let legacy = LegacyDeleteOptions {
+                    recursive: false,
+                    immediate: false,
+                    undo_write,
+                    undo_delete,
+                    old_data_dir: Some(transaction),
+                    expected_delete_marker: expected_delete_marker.clone(),
+                };
+                // Older senders use positional arrays. Their unchanged field prefix and
+                // missing optional fields must also decode on the named-map sender's peer.
+                for (encoding, payload) in [
+                    ("positional", rmp_serde::to_vec(&legacy).expect("encode legacy positional options")),
+                    ("named", rmp_serde::to_vec_named(&legacy).expect("encode legacy named options")),
+                ] {
+                    let decoded: DeleteOptions = rmp_serde::from_slice(&payload).expect("decode legacy options on current peer");
+                    assert_eq!(decoded.undo_write, undo_write, "{encoding} undo-write flag");
+                    assert_eq!(decoded.undo_delete, undo_delete, "{encoding} undo-delete flag");
+                    assert_eq!(decoded.old_data_dir, Some(transaction), "{encoding} transaction UUID");
+                    assert_eq!(decoded.expected_delete_marker, expected_delete_marker, "{encoding} marker precondition");
+                    assert!(decoded.delete_recovery_bucket_incarnation.is_none(), "{encoding} omitted incarnation");
+                    assert!(!decoded.recoverable_delete, "{encoding} omitted recovery flag");
+                }
+                for incarnation in [None, Some(Uuid::new_v4())] {
+                    let current = DeleteOptions {
+                        undo_write,
+                        undo_delete,
+                        old_data_dir: Some(transaction),
+                        expected_delete_marker: expected_delete_marker.clone(),
+                        delete_recovery_bucket_incarnation: incarnation,
+                        recoverable_delete: true,
+                        ..Default::default()
+                    };
+                    let payload = encode_delete_options_msgpack(&current).expect("encode current named options");
+                    let decoded: DeleteOptions = rmp_serde::from_slice(&payload).expect("decode current options on current peer");
+                    assert_eq!(decoded.undo_write, undo_write);
+                    assert_eq!(decoded.undo_delete, undo_delete);
+                    assert_eq!(decoded.old_data_dir, Some(transaction));
+                    assert_eq!(decoded.expected_delete_marker, expected_delete_marker);
+                    assert_eq!(decoded.delete_recovery_bucket_incarnation, incarnation);
+                    assert!(decoded.recoverable_delete);
+                    let decoded: LegacyDeleteOptions =
+                        rmp_serde::from_slice(&payload).expect("decode current options on legacy peer");
+                    assert_eq!(decoded, legacy, "unknown recovery fields cannot shift rollback or marker values");
+                }
+            }
+        }
+    }
+
+    #[test]
+    fn malformed_nonempty_delete_options_do_not_fall_back_to_json() {
         let options = DeleteOptions {
-            expected_delete_marker: Some(marker.clone()),
+            undo_write: true,
+            old_data_dir: Some(Uuid::new_v4()),
             ..Default::default()
         };
-        let decoded: DeleteOptions = rmp_serde::from_slice(&encode_msgpack(&options).unwrap()).unwrap();
-        assert_eq!(decoded.expected_delete_marker, Some(marker));
+        let json = serde_json::to_string(&options).expect("valid compatibility JSON");
+        assert!(decode_msgpack_or_json::<DeleteOptions>(&[0xc1], &json, "DeleteOptions").is_err());
+        let decoded: DeleteOptions =
+            decode_msgpack_or_json(&[], &json, "DeleteOptions").expect("absent MessagePack permits JSON compatibility");
+        assert!(decoded.undo_write);
+        assert_eq!(decoded.old_data_dir, options.old_data_dir);
     }
 
     #[test]

crates/ecstore/src/disk/disk_store.rs

@@ -466,12 +466,11 @@ impl LocalDiskWrapper {
         self.track_disk_health_mutation(
             "delete_version",
             DiskMetricMutation::Delete,
-            || async {
+            || {
                 Box::pin(
                     self.disk
                         .delete_version_with_namespace_owner(volume, path, fi, force_del_marker, opts, namespace_owner),
                 )
-                .await
             },
             get_max_timeout_duration(),
         )
@@ -1816,6 +1815,8 @@ impl LocalDiskWrapper {
                     undo_delete: false,
                     old_data_dir: None,
                     expected_delete_marker: None,
+                    recoverable_delete: false,
+                    delete_recovery_bucket_incarnation: None,
                 },
             )
             .await?;
@@ -2001,10 +2002,86 @@ impl LocalDiskWrapper {
     }
 
     /// Get the current disk ID
+    pub(crate) async fn delete_versions_with_namespace_owner(
+        &self,
+        volume: &str,
+        versions: Vec<FileInfoVersions>,
+        opts: DeleteOptions,
+        namespace_owner: Option<Arc<dyn Send + Sync>>,
+    ) -> Vec<Option<Error>> {
+        let version_count = versions.len();
+        self.metrics.record_operation_call("delete_versions");
+        // Check if disk is faulty before proceeding
+        if self.health.is_faulty() {
+            self.metrics.record_availability_error();
+            return vec![Some(DiskError::FaultyDisk); versions.len()];
+        }
+
+        // Check if disk is stale
+        if let Err(e) = self.check_disk_stale().await {
+            self.metrics.record_availability_error();
+            return vec![Some(e); versions.len()];
+        }
+
+        let _io_permit = match self.acquire_disk_io_permit().await {
+            Ok(permit) => permit,
+            Err(error) => return vec![Some(error); version_count],
+        };
+
+        // Record operation start
+        self.health.last_started.store(current_unix_nanos(), Ordering::Relaxed);
+        let waiting_guard = self.health.waiting_guard();
+        let metric_waiting_guard = self.metrics.waiting_guard();
+        let started = Instant::now();
+
+        // Execute the operation
+        let result = self
+            .disk
+            .delete_versions_with_namespace_owner(volume, versions, opts, namespace_owner)
+            .await;
+        self.metrics.record_operation_latency("delete_versions", started.elapsed());
+        self.record_batch_delete_error_metrics(&result);
+
+        drop(waiting_guard);
+        drop(metric_waiting_guard);
+        let has_err = result.iter().any(|e| e.is_some());
+        if !has_err {
+            // Log success and decrement waiting counter
+            self.health.record_operation_success(&self.endpoint(), "operation_success");
+            self.metrics.record_mutation_success(DiskMetricMutation::Delete);
+        }
+
+        result
+    }
+
     pub async fn get_current_disk_id(&self) -> Option<Uuid> {
         *self.disk_id.read().await
     }
 
+    pub(crate) async fn settle_delete_transaction_with_namespace_owner(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: uuid::Uuid,
+        expected_fingerprint: crate::disk::DeleteRecoveryDigest,
+        action: crate::disk::DeleteRecoveryAction,
+        namespace_owner: Option<Arc<dyn Send + Sync>>,
+    ) -> Result<crate::disk::DeleteRecoveryInspection> {
+        self.track_disk_health_mutation(
+            "settle_delete_transaction",
+            DiskMetricMutation::Delete,
+            || {
+                Box::pin(async {
+                    self.disk
+                        .delete_recovery_settle(volume, object, transaction, expected_fingerprint, action, namespace_owner)
+                        .await
+                })
+            },
+            crate::disk::get_delete_recovery_timeout()?,
+        )
+        .await
+    }
+
     /// Track disk health for an operation.
     /// This method should wrap disk operations to ensure health checking.
     pub async fn track_disk_health<T, F, Fut>(&self, operation: F, timeout_duration: Duration) -> Result<T>
@@ -2195,6 +2272,87 @@ impl LocalDiskWrapper {
 
 #[async_trait::async_trait]
 impl DiskAPI for LocalDiskWrapper {
+    async fn delete_version_recoverable(
+        &self,
+        volume: &str,
+        object: &str,
+        file_info: FileInfo,
+        force_del_marker: bool,
+        mut opts: DeleteOptions,
+    ) -> Result<()> {
+        crate::disk::validate_delete_recovery_options(&opts)?;
+        if object.starts_with('/') {
+            return Err(DiskError::InvalidPath);
+        }
+        opts.recoverable_delete = true;
+        self.delete_version(volume, object, file_info, force_del_marker, opts).await
+    }
+    async fn delete_versions_recoverable(
+        &self,
+        volume: &str,
+        versions: Vec<FileInfoVersions>,
+        mut opts: DeleteOptions,
+    ) -> Vec<Option<DiskError>> {
+        if let Err(err) = crate::disk::validate_delete_recovery_options(&opts) {
+            return vec![Some(err); versions.len()];
+        }
+        if versions.iter().any(|version| version.name.starts_with('/')) {
+            return vec![Some(DiskError::InvalidPath); versions.len()];
+        }
+        opts.recoverable_delete = true;
+        self.delete_versions(volume, versions, opts).await
+    }
+    async fn discover_delete_transactions(
+        &self,
+        volume: &str,
+        object: Option<&str>,
+        cursor: Option<crate::disk::DeleteRecoveryCursor>,
+        limit: u32,
+    ) -> Result<crate::disk::DeleteRecoveryPage> {
+        self.track_disk_health_with_op(
+            "discover_delete_transactions",
+            || async {
+                self.disk
+                    .discover_delete_transactions(volume, object, cursor.clone(), limit)
+                    .await
+            },
+            get_max_timeout_duration(),
+        )
+        .await
+    }
+    async fn inspect_delete_transaction(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: uuid::Uuid,
+    ) -> Result<crate::disk::DeleteRecoveryInspection> {
+        self.track_disk_health_with_op(
+            "inspect_delete_transaction",
+            || async { self.disk.inspect_delete_transaction(volume, object, transaction).await },
+            get_max_timeout_duration(),
+        )
+        .await
+    }
+    async fn settle_delete_transaction(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: uuid::Uuid,
+        expected_fingerprint: crate::disk::DeleteRecoveryDigest,
+        action: crate::disk::DeleteRecoveryAction,
+    ) -> Result<crate::disk::DeleteRecoveryInspection> {
+        self.track_disk_health_mutation(
+            "settle_delete_transaction",
+            DiskMetricMutation::Delete,
+            || async {
+                self.disk
+                    .settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
+                    .await
+            },
+            crate::disk::get_delete_recovery_timeout()?,
+        )
+        .await
+    }
     fn has_replacement_mount_lease(&self) -> bool {
         self.disk.has_replacement_mount_lease()
     }
@@ -2408,46 +2566,7 @@ impl DiskAPI for LocalDiskWrapper {
     }
 
     async fn delete_versions(&self, volume: &str, versions: Vec<FileInfoVersions>, opts: DeleteOptions) -> Vec<Option<Error>> {
-        let version_count = versions.len();
-        self.metrics.record_operation_call("delete_versions");
-        // Check if disk is faulty before proceeding
-        if self.health.is_faulty() {
-            self.metrics.record_availability_error();
-            return vec![Some(DiskError::FaultyDisk); versions.len()];
-        }
-
-        // Check if disk is stale
-        if let Err(e) = self.check_disk_stale().await {
-            self.metrics.record_availability_error();
-            return vec![Some(e); versions.len()];
-        }
-
-        let _io_permit = match self.acquire_disk_io_permit().await {
-            Ok(permit) => permit,
-            Err(error) => return vec![Some(error); version_count],
-        };
-
-        // Record operation start
-        self.health.last_started.store(current_unix_nanos(), Ordering::Relaxed);
-        let waiting_guard = self.health.waiting_guard();
-        let metric_waiting_guard = self.metrics.waiting_guard();
-        let started = Instant::now();
-
-        // Execute the operation
-        let result = self.disk.delete_versions(volume, versions, opts).await;
-        self.metrics.record_operation_latency("delete_versions", started.elapsed());
-        self.record_batch_delete_error_metrics(&result);
-
-        drop(waiting_guard);
-        drop(metric_waiting_guard);
-        let has_err = result.iter().any(|e| e.is_some());
-        if !has_err {
-            // Log success and decrement waiting counter
-            self.health.record_operation_success(&self.endpoint(), "operation_success");
-            self.metrics.record_mutation_success(DiskMetricMutation::Delete);
-        }
-
-        result
+        self.delete_versions_with_namespace_owner(volume, versions, opts, None).await
     }
 
     async fn delete_paths(&self, volume: &str, paths: &[String]) -> Result<()> {

crates/ecstore/src/disk/mod.rs

@@ -230,6 +230,105 @@ pub enum Disk {
 }
 
 impl Disk {
+    pub(crate) async fn delete_versions_with_namespace_owner(
+        &self,
+        volume: &str,
+        versions: Vec<FileInfoVersions>,
+        opts: DeleteOptions,
+        namespace_owner: Option<Arc<dyn Send + Sync>>,
+    ) -> Vec<Option<Error>> {
+        match self {
+            Self::Local(disk) => {
+                disk.delete_versions_with_namespace_owner(volume, versions, opts, namespace_owner)
+                    .await
+            }
+            Self::Remote(disk) => {
+                let _owner = namespace_owner;
+                disk.delete_versions(volume, versions, opts).await
+            }
+        }
+    }
+
+    pub(crate) async fn delete_versions_recoverable_with_namespace_owner(
+        &self,
+        volume: &str,
+        versions: Vec<FileInfoVersions>,
+        mut opts: DeleteOptions,
+        namespace_owner: Option<Arc<dyn Send + Sync>>,
+    ) -> Vec<Option<Error>> {
+        if let Err(error) = validate_delete_recovery_options(&opts) {
+            return vec![Some(error); versions.len()];
+        }
+        if versions.iter().any(|version| version.name.starts_with('/')) {
+            return vec![Some(DiskError::InvalidPath); versions.len()];
+        }
+        opts.recoverable_delete = true;
+        match self {
+            Self::Local(disk) => {
+                disk.delete_versions_with_namespace_owner(volume, versions, opts, namespace_owner)
+                    .await
+            }
+            Self::Remote(disk) => {
+                let _owner = namespace_owner;
+                disk.delete_versions_recoverable(volume, versions, opts).await
+            }
+        }
+    }
+
+    pub(crate) async fn settle_delete_transaction_with_namespace_owner(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: Uuid,
+        expected_fingerprint: DeleteRecoveryDigest,
+        action: DeleteRecoveryAction,
+        namespace_owner: Option<Arc<dyn Send + Sync>>,
+    ) -> Result<DeleteRecoveryInspection> {
+        match self {
+            Self::Local(disk) => {
+                disk.settle_delete_transaction_with_namespace_owner(
+                    volume,
+                    object,
+                    transaction,
+                    expected_fingerprint,
+                    action,
+                    namespace_owner,
+                )
+                .await
+            }
+            Self::Remote(disk) => {
+                let _owner = namespace_owner;
+                disk.settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
+                    .await
+            }
+        }
+    }
+
+    pub(crate) async fn delete_version_recoverable_with_namespace_owner(
+        &self,
+        volume: &str,
+        path: &str,
+        fi: FileInfo,
+        force_del_marker: bool,
+        mut opts: DeleteOptions,
+        namespace_owner: Option<Arc<dyn Send + Sync>>,
+    ) -> Result<()> {
+        validate_delete_recovery_options(&opts)?;
+        if path.starts_with('/') {
+            return Err(DiskError::InvalidPath);
+        }
+        opts.recoverable_delete = true;
+        match self {
+            Self::Local(disk) => {
+                disk.delete_version_with_namespace_owner(volume, path, fi, force_del_marker, opts, namespace_owner)
+                    .await
+            }
+            Self::Remote(disk) => {
+                disk.delete_version_recoverable(volume, path, fi, force_del_marker, opts)
+                    .await
+            }
+        }
+    }
     pub(crate) async fn set_disk_id_state(&self, id: Option<Uuid>) -> Result<()> {
         match self {
             Disk::Local(local_disk) => {
@@ -257,6 +356,70 @@ impl Disk {
 
 #[async_trait::async_trait]
 impl DiskAPI for Disk {
+    async fn delete_version_recoverable(
+        &self,
+        volume: &str,
+        object: &str,
+        file_info: FileInfo,
+        force_del_marker: bool,
+        opts: DeleteOptions,
+    ) -> Result<()> {
+        self.delete_version_recoverable_with_namespace_owner(volume, object, file_info, force_del_marker, opts, None)
+            .await
+    }
+    async fn delete_versions_recoverable(
+        &self,
+        volume: &str,
+        versions: Vec<FileInfoVersions>,
+        opts: DeleteOptions,
+    ) -> Vec<Option<DiskError>> {
+        match self {
+            Self::Local(disk) => disk.delete_versions_recoverable(volume, versions, opts).await,
+            Self::Remote(disk) => disk.delete_versions_recoverable(volume, versions, opts).await,
+        }
+    }
+    async fn discover_delete_transactions(
+        &self,
+        volume: &str,
+        object: Option<&str>,
+        cursor: Option<DeleteRecoveryCursor>,
+        limit: u32,
+    ) -> Result<DeleteRecoveryPage> {
+        match self {
+            Self::Local(disk) => disk.discover_delete_transactions(volume, object, cursor, limit).await,
+            Self::Remote(disk) => disk.discover_delete_transactions(volume, object, cursor, limit).await,
+        }
+    }
+    async fn inspect_delete_transaction(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: Uuid,
+    ) -> Result<DeleteRecoveryInspection> {
+        match self {
+            Self::Local(disk) => disk.inspect_delete_transaction(volume, object, transaction).await,
+            Self::Remote(disk) => disk.inspect_delete_transaction(volume, object, transaction).await,
+        }
+    }
+    async fn settle_delete_transaction(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: Uuid,
+        expected_fingerprint: DeleteRecoveryDigest,
+        action: DeleteRecoveryAction,
+    ) -> Result<DeleteRecoveryInspection> {
+        match self {
+            Self::Local(disk) => {
+                disk.settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
+                    .await
+            }
+            Self::Remote(disk) => {
+                disk.settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
+                    .await
+            }
+        }
+    }
     fn to_string(&self) -> String {
         match self {
             Disk::Local(local_disk) => local_disk.to_string(),
@@ -1246,6 +1409,51 @@ pub trait DiskAPI: Debug + Send + Sync + 'static {
         Err(DiskError::MethodNotAllowed)
     }
     async fn delete(&self, volume: &str, path: &str, opt: DeleteOptions) -> Result<()>;
+    async fn discover_delete_transactions(
+        &self,
+        _volume: &str,
+        _object: Option<&str>,
+        _cursor: Option<DeleteRecoveryCursor>,
+        _limit: u32,
+    ) -> Result<DeleteRecoveryPage> {
+        Err(DiskError::MethodNotAllowed)
+    }
+    async fn delete_version_recoverable(
+        &self,
+        _volume: &str,
+        _object: &str,
+        _file_info: FileInfo,
+        _force_del_marker: bool,
+        _opts: DeleteOptions,
+    ) -> Result<()> {
+        Err(DiskError::MethodNotAllowed)
+    }
+    async fn delete_versions_recoverable(
+        &self,
+        _volume: &str,
+        versions: Vec<FileInfoVersions>,
+        _opts: DeleteOptions,
+    ) -> Vec<Option<DiskError>> {
+        vec![Some(DiskError::MethodNotAllowed); versions.len()]
+    }
+    async fn inspect_delete_transaction(
+        &self,
+        _volume: &str,
+        _object: &str,
+        _transaction: Uuid,
+    ) -> Result<DeleteRecoveryInspection> {
+        Err(DiskError::MethodNotAllowed)
+    }
+    async fn settle_delete_transaction(
+        &self,
+        _volume: &str,
+        _object: &str,
+        _transaction: Uuid,
+        _expected_fingerprint: DeleteRecoveryDigest,
+        _action: DeleteRecoveryAction,
+    ) -> Result<DeleteRecoveryInspection> {
+        Err(DiskError::MethodNotAllowed)
+    }
     // VerifyFile
     async fn verify_file(&self, volume: &str, path: &str, fi: &FileInfo) -> Result<CheckPartsResp>;
     // CheckParts
@@ -1582,6 +1790,269 @@ pub struct DeleteOptions {
     /// Remote calls carrying it must use DeleteRetiredMarker, never DeleteVersion.
     #[serde(default, skip_serializing_if = "Option::is_none")]
     pub expected_delete_marker: Option<rustfs_filemeta::MetaDeleteMarker>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub delete_recovery_bucket_incarnation: Option<Uuid>,
+    #[serde(default)]
+    pub recoverable_delete: bool,
+}
+
+pub const DELETE_RECOVERY_PROTOCOL_VERSION: u32 = 1;
+pub(crate) fn get_delete_recovery_timeout() -> Result<std::time::Duration> {
+    let value = std::env::var("RUSTFS_DELETE_RECOVERY_TIMEOUT_SECS");
+    match value {
+        Ok(value) => parse_delete_recovery_timeout(Some(&value)),
+        Err(std::env::VarError::NotPresent) => parse_delete_recovery_timeout(None),
+        Err(std::env::VarError::NotUnicode(_)) => Err(DiskError::other("invalid delete recovery timeout configuration")),
+    }
+}
+
+fn parse_delete_recovery_timeout(value: Option<&str>) -> Result<std::time::Duration> {
+    let seconds = match value {
+        None => 900,
+        Some(value) => value
+            .parse::<u64>()
+            .ok()
+            .filter(|seconds| (30..=86400).contains(seconds))
+            .ok_or_else(|| DiskError::other("delete recovery timeout must be an integer between 30 and 86400 seconds"))?,
+    };
+    Ok(std::time::Duration::from_secs(seconds))
+}
+
+pub(crate) fn validate_delete_recovery_options(opts: &DeleteOptions) -> Result<()> {
+    if opts.old_data_dir.is_none_or(|id| id.is_nil())
+        || opts.undo_write
+        || opts.undo_delete
+        || opts.recursive
+        || opts.immediate
+        || opts.expected_delete_marker.is_some()
+        || opts.delete_recovery_bucket_incarnation.is_some_and(|id| id.is_nil())
+    {
+        return Err(DiskError::FileCorrupt);
+    }
+    Ok(())
+}
+pub type DeleteRecoveryDigest = [u8; 32];
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct DeleteTransactionId {
+    pub object: String,
+    pub transaction: Uuid,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct DeleteRecoveryCursor {
+    pub token: Uuid,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct DeleteRecoveryPage {
+    pub transactions: Vec<DeleteTransactionId>,
+    pub next: Option<DeleteRecoveryCursor>,
+}
+
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct DeleteRecoveryJournal {
+    pub protocol_version: u32,
+    pub volume: String,
+    pub object: String,
+    pub transaction: Uuid,
+    pub bucket_incarnation: Option<Uuid>,
+    pub original_digest: Option<DeleteRecoveryDigest>,
+    pub target_digest: Option<DeleteRecoveryDigest>,
+    pub data_dirs: Vec<Uuid>,
+    pub deleted_versions: Vec<Uuid>,
+    pub requests: Vec<FileInfo>,
+    pub original_versions: Vec<DeleteRecoveryVersionIdentity>,
+    #[serde(default)]
+    pub predecessors: Vec<Uuid>,
+}
+
+impl DeleteRecoveryJournal {
+    /// Checks source generations only; the coordinator must supply a certified
+    /// decision and the receiver must retain its namespace/incarnation fences.
+    pub(crate) fn matches_adoption_source(&self, current: &[DeleteRecoveryVersionIdentity]) -> bool {
+        if self.requests.is_empty() {
+            return false;
+        }
+        let mut originals = std::collections::HashMap::with_capacity(self.original_versions.len());
+        for version in &self.original_versions {
+            originals.entry(version.version_id.unwrap_or_default()).or_insert(version);
+        }
+        let mut locals = std::collections::HashMap::with_capacity(current.len());
+        for version in current {
+            locals.entry(version.version_id.unwrap_or_default()).or_insert(version);
+        }
+        self.requests.iter().all(|request| {
+            if request.deleted && request.mark_deleted {
+                return current.first() == self.original_versions.first();
+            }
+            let id = request.version_id.unwrap_or_default();
+            let Some(original) = originals.get(&id) else {
+                return false;
+            };
+            match locals.get(&id) {
+                Some(local) => local == original,
+                None => !id.is_nil(),
+            }
+        })
+    }
+
+    /// Matches immutable local preparation intent against an independently
+    /// inspected same-transaction commit receipt. This match is not authority
+    /// by itself; the coordinator must supply the certified receipt.
+    pub(crate) fn matches_prepared_authority(&self, authority: &Self) -> bool {
+        if self.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
+            || authority.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
+            || self.transaction.is_nil()
+            || self.transaction != authority.transaction
+            || self.volume != authority.volume
+            || self.object != authority.object
+            || self.bucket_incarnation != authority.bucket_incarnation
+            || self.bucket_incarnation.is_some_and(|id| id.is_nil())
+            || self.requests.is_empty()
+            || self.requests != authority.requests
+            || self.deleted_versions.iter().collect::<std::collections::HashSet<_>>()
+                != authority.deleted_versions.iter().collect::<std::collections::HashSet<_>>()
+        {
+            return false;
+        }
+        let sources: std::collections::HashMap<_, _> = self
+            .original_versions
+            .iter()
+            .map(|version| (version.version_id.unwrap_or_default(), version))
+            .collect();
+        let certified: std::collections::HashMap<_, _> = authority
+            .original_versions
+            .iter()
+            .map(|version| (version.version_id.unwrap_or_default(), version))
+            .collect();
+        if sources.len() != self.original_versions.len() || certified.len() != authority.original_versions.len() {
+            return false;
+        }
+        self.requests.iter().all(|request| {
+            let (local, remote) = if request.deleted && request.mark_deleted {
+                (self.original_versions.first(), authority.original_versions.first())
+            } else {
+                let id = request.version_id.unwrap_or_default();
+                (sources.get(&id).copied(), certified.get(&id).copied())
+            };
+            match (local, remote) {
+                (Some(local), Some(remote)) => local == remote,
+                (None, None) => self.original_digest.is_none() && authority.original_digest.is_none(),
+                _ => false,
+            }
+        })
+    }
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct DeleteRecoveryVersionIdentity {
+    pub version_id: Option<Uuid>,
+    pub mod_time: Option<OffsetDateTime>,
+    pub data_dir: Option<Uuid>,
+    pub deleted: bool,
+    pub metadata_digest: DeleteRecoveryDigest,
+}
+
+impl DeleteRecoveryVersionIdentity {
+    pub(crate) fn from_file_info(info: FileInfo) -> error::Result<Self> {
+        local::delete_recovery::generation(info)
+    }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+pub enum DeleteRecoveryState {
+    Absent,
+    NewPrepared,
+    QuorumCommitted,
+    CommitFinished,
+    LegacyAmbiguous,
+}
+
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct DeleteRecoveryInspection {
+    pub protocol_version: u32,
+    pub id: DeleteTransactionId,
+    pub state: DeleteRecoveryState,
+    pub journal: Option<DeleteRecoveryJournal>,
+    pub current_digest: Option<DeleteRecoveryDigest>,
+    pub current_versions: Vec<DeleteRecoveryVersionIdentity>,
+    pub backup_digest: Option<DeleteRecoveryDigest>,
+    pub rollback_data_complete: bool,
+    pub commit_data_safe: bool,
+    pub explicit_delete_safe: bool,
+    pub fingerprint: DeleteRecoveryDigest,
+}
+
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
+pub enum DeleteRecoveryAction {
+    PublishCommit,
+    FinishCommit,
+    FinalizeCommit,
+    Rollback,
+    DiscardLegacy,
+    AdoptCommit { authority: Box<DeleteRecoveryJournal> },
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub enum DeleteRecoveryRpcOperation {
+    Discover {
+        object: Option<String>,
+        cursor: Option<DeleteRecoveryCursor>,
+        limit: u32,
+    },
+    Inspect {
+        object: String,
+        transaction: Uuid,
+    },
+    Settle {
+        object: String,
+        transaction: Uuid,
+        expected_fingerprint: DeleteRecoveryDigest,
+        action: DeleteRecoveryAction,
+    },
+    ApplyVersion {
+        object: String,
+        file_info: Box<FileInfo>,
+        force_del_marker: bool,
+        opts: DeleteOptions,
+    },
+    ApplyVersions {
+        versions: Vec<FileInfoVersions>,
+        opts: DeleteOptions,
+    },
+}
+
+impl DeleteRecoveryRpcOperation {
+    /// Full metadata application and adoption share the existing mutation envelope.
+    pub fn max_wire_payload_bytes(&self) -> usize {
+        match self {
+            Self::ApplyVersion { .. }
+            | Self::ApplyVersions { .. }
+            | Self::Settle {
+                action: DeleteRecoveryAction::AdoptCommit { .. },
+                ..
+            } => rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES,
+            _ => rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES,
+        }
+    }
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub enum DeleteRecoveryRpcReply {
+    Discovered(DeleteRecoveryPage),
+    Inspected(DeleteRecoveryInspection),
+    Settled(DeleteRecoveryInspection),
+    AppliedVersion,
+    AppliedVersions(Vec<Option<u32>>),
 }
 
 #[derive(Debug, Clone, Serialize, Deserialize)]
@@ -1872,6 +2343,8 @@ mod tests {
             undo_delete: false,
             old_data_dir: Some(Uuid::new_v4()),
             expected_delete_marker: None,
+            delete_recovery_bucket_incarnation: None,
+            recoverable_delete: false,
         };
 
         assert!(opts.recursive);
@@ -2296,3 +2769,183 @@ mod tests {
         }
     }
 }
+
+#[cfg(test)]
+mod delete_recovery_timeout_tests {
+    use super::*;
+    #[test]
+    fn recovery_timeout_defaults_and_validates_range() {
+        assert_eq!(
+            parse_delete_recovery_timeout(None).expect("default timeout"),
+            std::time::Duration::from_secs(900)
+        );
+        for seconds in ["30", "900", "86400"] {
+            assert!(parse_delete_recovery_timeout(Some(seconds)).is_ok());
+        }
+        for value in ["", "invalid", "0", "2", "29", "86401", "-1", "18446744073709551616"] {
+            assert!(parse_delete_recovery_timeout(Some(value)).is_err());
+        }
+    }
+}
+
+#[cfg(test)]
+mod delete_recovery_wire_limit_tests {
+    use super::*;
+
+    #[test]
+    fn boxed_apply_version_preserves_existing_wire_payload() {
+        #[derive(Serialize)]
+        enum PriorOperation<'a> {
+            ApplyVersion {
+                object: &'a str,
+                file_info: &'a FileInfo,
+                force_del_marker: bool,
+                opts: &'a DeleteOptions,
+            },
+        }
+        let file_info = FileInfo {
+            name: "object".into(),
+            version_id: Some(Uuid::new_v4()),
+            metadata: std::collections::HashMap::from([("metadata".into(), "value".into())]),
+            ..Default::default()
+        };
+        let opts = DeleteOptions {
+            old_data_dir: Some(Uuid::new_v4()),
+            recoverable_delete: true,
+            delete_recovery_bucket_incarnation: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+        let prior = PriorOperation::ApplyVersion {
+            object: "object",
+            file_info: &file_info,
+            force_del_marker: true,
+            opts: &opts,
+        };
+        let encoded = rmp_serde::to_vec_named(&prior).expect("encode the previous unboxed request shape");
+        let operation = DeleteRecoveryRpcOperation::ApplyVersion {
+            object: "object".into(),
+            file_info: Box::new(file_info.clone()),
+            force_del_marker: true,
+            opts: opts.clone(),
+        };
+        assert_eq!(rmp_serde::to_vec_named(&operation).expect("encode boxed request"), encoded);
+        assert_eq!(
+            serde_json::to_value(&operation).expect("boxed request JSON"),
+            serde_json::to_value(prior).expect("previous request JSON")
+        );
+        let decoded: DeleteRecoveryRpcOperation =
+            rustfs_protos::decode_delete_recovery_payload(&encoded).expect("decode the previous payload");
+        assert_eq!(rmp_serde::to_vec_named(&decoded).expect("re-encode decoded prior request"), encoded);
+    }
+    #[test]
+    fn adoption_and_inspection_reply_support_large_version_journals() {
+        let transaction = Uuid::new_v4();
+        let request = FileInfo {
+            metadata: std::collections::HashMap::from([("metadata".into(), "x".repeat(2048))]),
+            ..Default::default()
+        };
+        let journal = DeleteRecoveryJournal {
+            protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
+            volume: "bucket".into(),
+            object: "object".into(),
+            transaction,
+            bucket_incarnation: None,
+            original_digest: None,
+            target_digest: None,
+            data_dirs: Vec::new(),
+            deleted_versions: Vec::new(),
+            requests: vec![request; 1000],
+            original_versions: Vec::new(),
+            predecessors: Vec::new(),
+        };
+        let operation = DeleteRecoveryRpcOperation::Settle {
+            object: "object".into(),
+            transaction,
+            expected_fingerprint: [0; 32],
+            action: DeleteRecoveryAction::AdoptCommit {
+                authority: Box::new(journal.clone()),
+            },
+        };
+        let encoded = rmp_serde::to_vec_named(&operation).expect("encode large adoption");
+        assert!(encoded.len() > rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES);
+        assert!(encoded.len() < operation.max_wire_payload_bytes());
+        let decoded: DeleteRecoveryRpcOperation =
+            rustfs_protos::decode_delete_recovery_payload(&encoded).expect("large adoption can decode");
+        assert_eq!(decoded.max_wire_payload_bytes(), rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES);
+        let reply = DeleteRecoveryRpcReply::Inspected(DeleteRecoveryInspection {
+            protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
+            id: DeleteTransactionId {
+                object: "object".into(),
+                transaction,
+            },
+            state: DeleteRecoveryState::QuorumCommitted,
+            journal: Some(journal),
+            current_digest: None,
+            current_versions: Vec::new(),
+            backup_digest: None,
+            rollback_data_complete: false,
+            commit_data_safe: true,
+            explicit_delete_safe: false,
+            fingerprint: [0; 32],
+        });
+        let encoded = rmp_serde::to_vec_named(&reply).expect("encode large inspection");
+        assert!(encoded.len() > rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES);
+        assert!(rustfs_protos::decode_delete_recovery_payload::<DeleteRecoveryRpcReply>(&encoded).is_ok());
+        let inspect = DeleteRecoveryRpcOperation::Inspect {
+            object: "object".into(),
+            transaction,
+        };
+        assert_eq!(inspect.max_wire_payload_bytes(), rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES);
+    }
+}
+
+#[cfg(test)]
+mod delete_recovery_adoption_tests {
+    use super::*;
+
+    #[test]
+    fn source_matching_preserves_non_null_absence_and_rejects_foreign_or_missing_null() {
+        let id = Uuid::new_v4();
+        let source = DeleteRecoveryVersionIdentity {
+            version_id: Some(id),
+            mod_time: None,
+            data_dir: None,
+            deleted: false,
+            metadata_digest: [1; 32],
+        };
+        let mut journal = DeleteRecoveryJournal {
+            protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
+            volume: "bucket".into(),
+            object: "object".into(),
+            transaction: Uuid::new_v4(),
+            bucket_incarnation: None,
+            original_digest: None,
+            target_digest: None,
+            data_dirs: Vec::new(),
+            deleted_versions: vec![id],
+            requests: vec![FileInfo {
+                version_id: Some(id),
+                ..Default::default()
+            }],
+            original_versions: vec![source.clone()],
+            predecessors: Vec::new(),
+        };
+        assert!(journal.matches_adoption_source(&[]), "certified non-null removal can already be absent");
+        assert!(journal.matches_adoption_source(std::slice::from_ref(&source)));
+        let mut foreign = source.clone();
+        foreign.metadata_digest = [2; 32];
+        assert!(!journal.matches_adoption_source(std::slice::from_ref(&foreign)));
+        journal.original_versions.push(foreign.clone());
+        assert!(
+            journal.matches_adoption_source(&[source.clone(), foreign.clone()]),
+            "retain original first-match semantics"
+        );
+        assert!(!journal.matches_adoption_source(&[foreign, source]));
+        journal.requests[0].version_id = None;
+        journal.original_versions[0].version_id = None;
+        assert!(!journal.matches_adoption_source(&[]), "null absence does not establish its generation");
+        assert!(journal.matches_adoption_source(std::slice::from_ref(&journal.original_versions[0])));
+        journal.requests.clear();
+        assert!(!journal.matches_adoption_source(&journal.original_versions));
+    }
+}

crates/ecstore/src/object_api/types.rs

@@ -938,6 +938,10 @@ pub struct ObjectOptions {
     /// A genuine target write guard; loss fences alone do not authorize a borrowed write.
     #[doc(hidden)]
     pub write_lock_context: Option<WriteLockContext>,
+    /// In-process delete ownership for late filesystem completion. Never
+    /// serialized or copied into object metadata.
+    #[doc(hidden)]
+    pub delete_mutation_owner: Option<Arc<dyn Send + Sync>>,
     /// Internal read-only inspection must not enqueue metadata or payload repairs.
     pub suppress_read_repair: bool,
     /// Control-plane writers that immediately read or CAS the same namespace
@@ -1158,6 +1162,7 @@ impl std::fmt::Debug for ObjectOptions {
             .field("http_preconditions", &self.http_preconditions.is_some())
             .field("delete_replication", &self.delete_replication.is_some())
             .field("delete_replication_config_snapshot", &self.delete_replication_config_snapshot)
+            .field("delete_mutation_owner", &self.delete_mutation_owner.is_some())
             .field("namespace_lock_fence", &self.namespace_lock_fence.is_some())
             .field("bucket_lifecycle_lock_fence", &self.bucket_lifecycle_lock_fence.is_some())
             .field("replication_request", &self.replication_request)

crates/ecstore/src/set_disk/mod.rs

@@ -874,6 +874,7 @@ mod ctx;
 mod metadata;
 mod ops;
 pub(crate) use ops::bucket::BucketInfoQuorum;
+pub(crate) use ops::delete_recovery::{DeleteRecoveryContext, DeleteRecoverySummary};
 pub(crate) use ops::heal::AbsenceProofRequest;
 #[cfg(test)]
 pub(crate) use ops::heal::DanglingDeleteFailure;

crates/ecstore/src/set_disk/ops/heal.rs

@@ -25,11 +25,14 @@ use super::super::{
 use crate::bucket::retirement::MarkerRetirementContext;
 use crate::diagnostics::object_lock::ObjectLockAttempt;
 use crate::set_disk::{ObjectLockDiagGuard, is_object_lock_diag_enabled};
+use std::sync::Arc;
 
 #[derive(Clone, Copy)]
 struct ExplicitVersionHeal<'a> {
     opts: &'a HealOpts,
     allow_regeneration: bool,
+    preserve_unrepairable: bool,
+    commit_fence: Option<&'a (dyn Fn() -> bool + Sync)>,
     retirement: Option<&'a MarkerRetirementContext<'a>>,
 }
 
@@ -770,6 +773,8 @@ impl SetDisks {
             ExplicitVersionHeal {
                 opts,
                 allow_regeneration: true,
+                preserve_unrepairable: false,
+                commit_fence: None,
                 retirement: None,
             },
             &mut None,
@@ -907,6 +912,8 @@ impl SetDisks {
         let ExplicitVersionHeal {
             opts,
             allow_regeneration: allow_explicit_version_regen,
+            preserve_unrepairable,
+            commit_fence,
             retirement,
         } = run;
         trace!(
@@ -925,6 +932,9 @@ impl SetDisks {
         if let Some(scope) = &bucket_heal_scope {
             scope.check()?;
         }
+        if commit_fence.is_some_and(|check| !check()) {
+            return Err(DiskError::DiskOngoingReq);
+        }
 
         let mut result = HealResultItem {
             heal_item_type: HealItemType::Object.to_string(),
@@ -1248,7 +1258,7 @@ impl SetDisks {
                             // return here and never reach the post-heal reclaim tail
                             // below. Sweep the strays on this path too (issues #3231,
                             // #3191). Skipped on dry-run, like every mutating step.
-                            if !opts.dry_run {
+                            if !opts.dry_run && !preserve_unrepairable {
                                 self.reclaim_orphan_data_dirs_best_effort(bucket, object).await;
                             }
                             return Ok((result, None));
@@ -1328,7 +1338,7 @@ impl SetDisks {
                                 );
                             }
 
-                            if read_repair_uses_shared_lock {
+                            if read_repair_uses_shared_lock || preserve_unrepairable {
                                 return Ok((result, Some(cannot_heal_err)));
                             }
 
@@ -1527,6 +1537,9 @@ impl SetDisks {
                             if let Some(scope) = &bucket_heal_scope {
                                 scope.check()?;
                             }
+                            if commit_fence.is_some_and(|check| !check()) {
+                                return Err(DiskError::DiskOngoingReq);
+                            }
                             let disk = outdated_disk.as_ref().expect("checked above").clone();
                             let endpoint = disk.endpoint().to_string();
                             match disk.make_volume(bucket).await {
@@ -1562,6 +1575,9 @@ impl SetDisks {
                         if let Some(scope) = &bucket_heal_scope {
                             scope.check()?;
                         }
+                        if commit_fence.is_some_and(|check| !check()) {
+                            return Err(DiskError::DiskOngoingReq);
+                        }
                         if disks_to_heal_count == 0 {
                             return Ok((result, volume_creation_error.or(Some(DiskError::ErasureWriteQuorum))));
                         }
@@ -1873,6 +1889,10 @@ impl SetDisks {
                                 if let Some(scope) = &bucket_heal_scope {
                                     scope.check()?;
                                 }
+                                if commit_fence.is_some_and(|check| !check()) {
+                                    let _ = self.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_id).await;
+                                    return Err(DiskError::DiskOngoingReq);
+                                }
                                 rename_attempts += 1;
                                 // record the index of the updated disks
                                 parts_metadata[index].erasure.index = index + 1;
@@ -2004,7 +2024,9 @@ impl SetDisks {
                         // by pre-#3510 unversioned overwrites, which the dangling paths
                         // above never touch (issues #3231, #3191). Best effort — a
                         // failure must not fail the heal.
-                        self.reclaim_orphan_data_dirs_best_effort(bucket, object).await;
+                        if !preserve_unrepairable {
+                            self.reclaim_orphan_data_dirs_best_effort(bucket, object).await;
+                        }
 
                         Ok((result, None))
                     }
@@ -2070,7 +2092,7 @@ impl SetDisks {
                     .await;
                 }
 
-                if opts.dry_run {
+                if opts.dry_run || preserve_unrepairable {
                     return Ok((
                         self.default_heal_result(FileInfo::default(), &errs, bucket, object, version_id)
                             .await,
@@ -3347,6 +3369,74 @@ impl crate::storage_api_contracts::heal::HealOperations for SetDisks {
 }
 
 impl SetDisks {
+    /// The recovery coordinator retains the key write lock and lifecycle fence.
+    /// Only current live metadata can supply this exact source; backups and
+    /// dangling cleanup cannot authorize prerequisite repair.
+    pub(in crate::set_disk) async fn heal_delete_recovery_source(
+        &self,
+        bucket: &str,
+        object: &str,
+        source: &crate::disk::DeleteRecoveryVersionIdentity,
+        can_commit: &(dyn Fn() -> bool + Sync),
+    ) -> disk::error::Result<()> {
+        if !can_commit() {
+            return Err(DiskError::DiskOngoingReq);
+        }
+        let version = source.version_id.ok_or(DiskError::OutdatedXLMeta)?.to_string();
+        let disks = self.get_disks_internal().await;
+        let (metadata, errors, _) = Self::read_metadata(&disks, "", bucket, object, &version, true, false, false)
+            .await?
+            .into_legacy();
+        let (read_quorum, write_quorum) = Self::object_quorum_from_meta(&metadata, &errors, self.default_parity_count)?;
+        let read_quorum = usize::try_from(read_quorum).map_err(|_| DiskError::ErasureReadQuorum)?;
+        let write_quorum = usize::try_from(write_quorum).map_err(|_| DiskError::ErasureWriteQuorum)?;
+        let live_sources = metadata
+            .iter()
+            .zip(&errors)
+            .filter(|(info, error)| {
+                error.is_none()
+                    && !info.deleted
+                    && !info.is_remote()
+                    && crate::disk::DeleteRecoveryVersionIdentity::from_file_info((*info).clone())
+                        .is_ok_and(|identity| identity == *source)
+            })
+            .count();
+        if live_sources < write_quorum || !can_commit() {
+            return Err(DiskError::OutdatedXLMeta);
+        }
+        let (_, mod_time, etag) = Self::list_online_disks(&disks, &metadata, &errors, read_quorum);
+        let selected = Self::pick_valid_fileinfo(&metadata, mod_time, etag, read_quorum)?;
+        if crate::disk::DeleteRecoveryVersionIdentity::from_file_info(selected)? != *source {
+            return Err(DiskError::OutdatedXLMeta);
+        }
+        let opts = HealOpts {
+            no_lock: true,
+            scan_mode: HealScanMode::Deep,
+            ..Default::default()
+        };
+        let (_, error) = Box::pin(self.heal_object_with_explicit_version_regen(
+            bucket,
+            object,
+            &version,
+            ExplicitVersionHeal {
+                opts: &opts,
+                allow_regeneration: false,
+                preserve_unrepairable: true,
+                commit_fence: Some(can_commit),
+                retirement: None,
+            },
+            &mut None,
+        ))
+        .await?;
+        if let Some(error) = error {
+            return Err(error);
+        }
+        if !can_commit() {
+            return Err(DiskError::DiskOngoingReq);
+        }
+        Ok(())
+    }
+
     pub(crate) async fn heal_object_with_absence(
         &self,
         bucket: &str,
@@ -3375,11 +3465,11 @@ impl SetDisks {
                 .new_ns_lock(bucket, object)
                 .await
                 .map_err(|e| e.narrow_to_disk().unwrap_or_else(DiskError::other))?;
-            Some(ns_lock.get_write_lock(get_lock_acquire_timeout()).await.map_err(|e| {
+            Some(Arc::new(ns_lock.get_write_lock(get_lock_acquire_timeout()).await.map_err(|e| {
                 self.map_namespace_lock_error(bucket, object, "write", e)
                     .narrow_to_disk()
                     .unwrap_or_else(DiskError::other)
-            })?)
+            })?))
         } else {
             None
         };
@@ -3389,6 +3479,75 @@ impl SetDisks {
             return Ok((result, err.map(|e| e.into())));
         }
 
+        let mut recovery_removed_version = false;
+        if !opts.read_repair
+            && _write_lock_guard.is_some()
+            && let Some(scope) = crate::store::bucket_heal_scope_for_object(bucket, object)
+            && !crate::bucket::utils::is_meta_bucketname(bucket)
+        {
+            let requested = Uuid::parse_str(version_id).ok();
+            if !opts.dry_run && (version_id.is_empty() || requested.is_some()) {
+                let disks = self.get_disks_internal().await;
+                let before = Self::read_metadata(&disks, "", bucket, object, version_id, false, false, false)
+                    .await
+                    .map_err(|e| to_object_err(e.into(), vec![bucket, object]))?;
+                recovery_removed_version = before
+                    .slots
+                    .iter()
+                    .filter_map(|slot| slot.file_info())
+                    .any(|info| version_id.is_empty() || Some(info.version_id.unwrap_or_default()) == requested);
+            }
+            let explicit_delete = opts.remove
+                && version_id.is_empty()
+                && crate::bucket::metadata_sys::has_authoritative_never_versioned_state_in(&self.ctx, bucket).await?
+                && matches!(
+                    crate::bucket::metadata_sys::get_object_lock_config_state_in(&self.ctx, bucket).await?,
+                    crate::bucket::metadata_sys::ObjectLockConfigState::ConfirmedAbsent
+                );
+            let recovery = self
+                .recover_delete_key(
+                    bucket,
+                    object,
+                    crate::set_disk::DeleteRecoveryContext {
+                        dry_run: opts.dry_run,
+                        explicit_delete,
+                        repair_marker_source: true,
+                        namespace_owner: (!opts.dry_run).then(|| {
+                            Arc::new((scope.clone(), self.ctx.begin_namespace_commit(), _write_lock_guard.clone()))
+                                as Arc<dyn Send + Sync>
+                        }),
+                        ..crate::set_disk::DeleteRecoveryContext::new(Some(scope.incarnation), &|| {
+                            scope.check().is_ok() && !_write_lock_guard.as_deref().is_some_and(|guard| guard.is_lock_lost())
+                        })
+                    },
+                )
+                .await?;
+            if opts.dry_run && recovery.discovered > 0 {
+                return Ok((
+                    HealResultItem {
+                        heal_item_type: HealItemType::Object.to_string(),
+                        bucket: bucket.to_owned(),
+                        object: object.to_owned(),
+                        version_id: version_id.to_owned(),
+                        detail: format!(
+                            "delete recovery (dry run): discovered={}, committed={}, rolled_back={}, reclaimed={}, blocked={}, pending={}",
+                            recovery.discovered,
+                            recovery.committed,
+                            recovery.rolled_back,
+                            recovery.reclaimed,
+                            recovery.blocked,
+                            recovery.pending
+                        ),
+                        ..Default::default()
+                    },
+                    None,
+                ));
+            }
+            if recovery.pending > 0 || recovery.blocked > 0 {
+                return Err(StorageError::SlowDown);
+            }
+        }
+
         // The inner heal and missing-object report read the registry again;
         // release this snapshot guard before a topology writer can queue between reads.
         let disks = self.get_disks_internal().await;
@@ -3440,12 +3599,12 @@ impl SetDisks {
                 && errs
                     .iter()
                     .all(|err| matches!(err, Some(DiskError::FileNotFound | DiskError::FileVersionNotFound)))
-                && !_write_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
+                && !_write_lock_guard.as_deref().is_some_and(|guard| guard.is_lock_lost())
             {
                 *absence = Some(HealedObjectAbsence {
                     pool_index: self.pool_index,
                     set_index: self.set_index,
-                    removed: false,
+                    removed: recovery_removed_version,
                 });
             }
             return Ok((result, Some(err)));
@@ -3463,6 +3622,8 @@ impl SetDisks {
                 ExplicitVersionHeal {
                     opts: &inner_opts,
                     allow_regeneration: true,
+                    preserve_unrepairable: false,
+                    commit_fence: None,
                     retirement: proof.retirement,
                 },
                 absence,
@@ -3483,6 +3644,8 @@ impl SetDisks {
                             ExplicitVersionHeal {
                                 opts: &inner_opts,
                                 allow_regeneration: true,
+                                preserve_unrepairable: false,
+                                commit_fence: None,
                                 retirement: proof.retirement,
                             },
                             absence,
@@ -3493,7 +3656,7 @@ impl SetDisks {
                         result,
                         err.map(Into::into),
                         absence,
-                        _write_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()),
+                        _write_lock_guard.as_deref().is_some_and(|guard| guard.is_lock_lost()),
                         bucket,
                         object,
                     ));
@@ -3505,7 +3668,7 @@ impl SetDisks {
             result,
             err.map(Into::into),
             absence,
-            _write_lock_guard.as_ref().is_some_and(|guard| guard.is_lock_lost()),
+            _write_lock_guard.as_deref().is_some_and(|guard| guard.is_lock_lost()),
             bucket,
             object,
         ))

crates/ecstore/src/set_disk/ops/mod.rs

@@ -19,6 +19,7 @@
 
 pub(crate) mod bitrot_self_verify;
 pub(crate) mod bucket;
+pub(crate) mod delete_recovery;
 pub(crate) mod heal;
 pub(crate) mod heal_walk;
 pub(crate) mod list;

crates/ecstore/src/store/heal.rs

@@ -270,6 +270,134 @@ impl ECStore {
         .map_err(|error| Error::other_with_context("bucket heal owner task failed", error))?
     }
 
+    /// Recover hidden delete journals through the lifecycle-owned heal boundary.
+    pub async fn recover_delete_transactions(self: &Arc<Self>, bucket: &str, opts: &HealOpts) -> Result<HealResultItem> {
+        Ok(self
+            .recover_delete_transactions_at_incarnation(bucket, None, None, opts)
+            .await?
+            .unwrap_or_else(|| HealResultItem {
+                heal_item_type: HealItemType::Bucket.to_string(),
+                bucket: bucket.to_owned(),
+                detail: "delete recovery: discovered=0".to_owned(),
+                ..Default::default()
+            }))
+    }
+
+    pub async fn recover_delete_transactions_at_incarnation(
+        self: &Arc<Self>,
+        bucket: &str,
+        object: Option<&str>,
+        expected: Option<Uuid>,
+        opts: &HealOpts,
+    ) -> Result<Option<HealResultItem>> {
+        self.recover_delete_transactions_in_scope(bucket, object, None, expected, opts)
+            .await
+    }
+
+    pub async fn recover_delete_prefix_at_incarnation(
+        self: &Arc<Self>,
+        bucket: &str,
+        prefix: &str,
+        expected: Option<Uuid>,
+        opts: &HealOpts,
+    ) -> Result<Option<HealResultItem>> {
+        self.recover_delete_transactions_in_scope(bucket, None, Some(prefix), expected, opts)
+            .await
+    }
+
+    async fn recover_delete_transactions_in_scope(
+        self: &Arc<Self>,
+        bucket: &str,
+        object: Option<&str>,
+        prefix: Option<&str>,
+        expected: Option<Uuid>,
+        opts: &HealOpts,
+    ) -> Result<Option<HealResultItem>> {
+        if opts.no_lock && !opts.dry_run {
+            return Err(StorageError::InvalidArgument(
+                "heal".to_string(),
+                "no_lock".to_string(),
+                "delete recovery requires namespace locking".to_string(),
+            ));
+        }
+        let expected = match expected {
+            Some(expected) => expected,
+            None => self.bucket_incarnation_id_from_disk(bucket).await?,
+        };
+        let object = object.map(str::to_owned);
+        let prefix = prefix.map(str::to_owned);
+        self.run_bucket_heal_at_incarnation(bucket, expected, opts, |store, bucket, opts| async move {
+            let scope = bucket_heal_scope(&bucket).ok_or(StorageError::PreconditionFailed)?;
+            let explicit_delete = opts.remove
+                && metadata_sys::has_authoritative_never_versioned_state_in(&store.ctx, &bucket).await?
+                && matches!(
+                    metadata_sys::get_object_lock_config_state_in(&store.ctx, &bucket).await?,
+                    metadata_sys::ObjectLockConfigState::ConfirmedAbsent
+                );
+            let movement_gate = store.ctx.data_movement_operation_gate();
+            let _movement_guard = movement_gate.read().await;
+            let can_commit = || scope.check().is_ok();
+            let recovery_context = crate::set_disk::DeleteRecoveryContext {
+                dry_run: opts.dry_run,
+                explicit_delete,
+                namespace_owner: (!opts.dry_run)
+                    .then(|| Arc::new((scope.clone(), store.ctx.begin_namespace_commit())) as Arc<dyn Send + Sync>),
+                ..crate::set_disk::DeleteRecoveryContext::new(Some(scope.incarnation), &can_commit)
+            };
+            let mut summary = crate::set_disk::DeleteRecoverySummary::default();
+            for pool in store.get_pools_for_heal_object(&opts)? {
+                if store.is_suspended(pool.pool_idx).await || store.is_pool_rebalancing(pool.pool_idx).await {
+                    return Err(StorageError::SlowDown);
+                }
+                for (set_index, set) in pool.disk_set.iter().enumerate() {
+                    if opts.set.is_some_and(|requested| requested != set_index) {
+                        continue;
+                    }
+                    if let Some(object) = &object {
+                        if !Arc::ptr_eq(set, &pool.get_disks_by_key(object)) {
+                            continue;
+                        }
+                        summary.add(set.recover_delete_object(&bucket, object, recovery_context.clone()).await?);
+                    } else if let Some(prefix) = &prefix {
+                        summary.add(set.recover_delete_prefix(&bucket, prefix, recovery_context.clone()).await?);
+                    } else {
+                        summary.add(set.recover_delete_bucket(&bucket, recovery_context.clone()).await?);
+                    }
+                }
+            }
+            if summary.pending > 0 {
+                return Err(StorageError::ErasureReadQuorum);
+            }
+            if object.is_some() && !opts.dry_run && summary.blocked > 0 {
+                return Err(StorageError::SlowDown);
+            }
+            if summary.discovered == 0 {
+                return Ok(None);
+            }
+            Ok(Some(HealResultItem {
+                heal_item_type: if object.is_some() {
+                    HealItemType::Object
+                } else {
+                    HealItemType::Bucket
+                }
+                .to_string(),
+                object: object.or(prefix).unwrap_or_default(),
+                bucket,
+                detail: format!(
+                    "delete recovery{}: discovered={}, committed={}, rolled_back={}, reclaimed={}, blocked={}",
+                    if opts.dry_run { " (dry run)" } else { "" },
+                    summary.discovered,
+                    summary.committed,
+                    summary.rolled_back,
+                    summary.reclaimed,
+                    summary.blocked
+                ),
+                ..Default::default()
+            }))
+        })
+        .await
+    }
+
     pub async fn heal_bucket_at_incarnation(
         self: &Arc<Self>,
         bucket: &str,

crates/ecstore/src/store/object.rs

@@ -4849,7 +4849,9 @@ impl ECStore {
         opts: ObjectOptions,
         guards: impl IntoIterator<Item = rustfs_lock::NamespaceLockGuard>,
     ) -> Result<ObjectInfo> {
-        let guards: Vec<_> = guards.into_iter().collect();
+        let guards = Arc::new(guards.into_iter().collect::<Vec<_>>());
+        let mut opts = opts;
+        opts.delete_mutation_owner = Some(Arc::new((opts.delete_mutation_owner.take(), guards.clone())));
         let store = Arc::clone(self);
         let bucket = bucket.to_owned();
         let object = object.to_owned();
@@ -4861,7 +4863,7 @@ impl ECStore {
                 if result.is_ok() {
                     list_objects::observe_list_objects_mutation(&store, &bucket).await;
                 }
-                // Keep: the only use of `guards` here, it moves them into the task so the locks outlive a dropped caller.
+                // Transient options also retain these guards through late disk I/O.
                 drop(guards);
                 result
             })
@@ -4930,7 +4932,7 @@ impl ECStore {
             let mut opts = opts;
             let receipt_sink = install_tier_free_version_receipt_sink(&mut opts);
             let result = self
-                .handle_delete_object_with_journal_inner(bucket, object, opts, tier_journal_api)
+                .handle_delete_object_with_journal_inner_boxed(bucket, object, opts, tier_journal_api)
                 .await;
             enqueue_recorded_tier_free_versions(self, receipt_sink).await;
             result
@@ -4938,6 +4940,16 @@ impl ECStore {
         .await
     }
 
+    fn handle_delete_object_with_journal_inner_boxed<'a>(
+        &'a self,
+        bucket: &'a str,
+        object: &'a str,
+        opts: ObjectOptions,
+        tier_journal_api: Option<Arc<ECStore>>,
+    ) -> impl std::future::Future<Output = Result<ObjectInfo>> + Send + 'a {
+        Box::pin(self.handle_delete_object_with_journal_inner(bucket, object, opts, tier_journal_api))
+    }
+
     async fn handle_delete_object_with_journal_inner(
         &self,
         bucket: &str,
@@ -4955,9 +4967,9 @@ impl ECStore {
             if is_meta_bucketname(bucket) || (opts.delete_prefix && opts.bucket_lifecycle_lock_fence.is_some()) {
                 None
             } else if opts.delete_prefix {
-                Some(self.acquire_bucket_lifecycle_write_lock(bucket).await?)
+                Some(Arc::new(self.acquire_bucket_lifecycle_write_lock(bucket).await?))
             } else {
-                Some(self.acquire_bucket_lifecycle_read_lock(bucket).await?)
+                Some(Arc::new(self.acquire_bucket_lifecycle_read_lock(bucket).await?))
             };
         let object = if opts.delete_prefix && !opts.delete_prefix_object {
             object.to_owned()
@@ -4975,6 +4987,7 @@ impl ECStore {
         opts.tier_delete_journal_api = tier_journal_api.clone();
         if let Some(guard) = _bucket_lifecycle_guard.as_ref() {
             opts.add_bucket_lifecycle_lock_guard(guard);
+            opts.delete_mutation_owner = Some(Arc::new((opts.delete_mutation_owner.take(), guard.clone())));
         }
 
         if !is_meta_bucketname(bucket) {
@@ -5227,6 +5240,40 @@ impl ECStore {
                 return Ok(obj);
             }
             Err(err) => {
+                if (is_err_object_not_found(&err) || is_err_version_not_found(&err))
+                    && (opts.version_id.is_some() || (!opts.versioned && !opts.version_suspended))
+                    && opts.http_preconditions.is_none()
+                    && !opts.replication_request
+                {
+                    let mut reclaimed = None;
+                    for pool in &self.pools {
+                        if self.is_suspended(pool.pool_idx).await || self.is_pool_rebalancing(pool.pool_idx).await {
+                            return Err(StorageError::SlowDown);
+                        }
+                        let pool_idx = pool.pool_idx;
+                        let pool = pool.clone();
+                        match self
+                            .run_external_decommission_capacity_object_delete(
+                                pool_idx,
+                                bucket,
+                                object,
+                                object,
+                                opts.clone(),
+                                |opts| async move { pool.delete_object(bucket, object, opts).await },
+                            )
+                            .await
+                        {
+                            Ok(result) => reclaimed = Some(result),
+                            Err(missing) if is_err_object_not_found(&missing) || is_err_version_not_found(&missing) => {}
+                            Err(error) => return Err(error),
+                        }
+                    }
+                    if let Some(mut result) = reclaimed {
+                        result.bucket = bucket.to_owned();
+                        result.name = decode_dir_object(object);
+                        return Ok(result);
+                    }
+                }
                 // A folder key (`prefix/`) with no object metadata may still exist on
                 // disk as an orphan empty-directory tree (issue #4189): listings show
                 // it as a common prefix, but no regular delete path can remove it.
@@ -5422,12 +5469,13 @@ impl ECStore {
             None
         } else {
             match self.acquire_bucket_lifecycle_read_lock(bucket).await {
-                Ok(guard) => Some(guard),
+                Ok(guard) => Some(Arc::new(guard)),
                 Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err),
             }
         };
         if let Some(guard) = _bucket_lifecycle_guard.as_ref() {
             opts.add_bucket_lifecycle_lock_guard(guard);
+            opts.delete_mutation_owner = Some(Arc::new((opts.delete_mutation_owner.take(), guard.clone())));
         }
         if opts.delete_replication_config_snapshot.is_none() {
             match ReplicationObjectBridge::delete_request_config_in(&self.ctx, bucket).await {
@@ -5493,10 +5541,11 @@ impl ECStore {
         if current_bucket_incarnation_id.is_some() {
             pause_delete_after_object_lock_snapshot(bucket).await;
         }
-        let _object_lock_guards = match self.acquire_delete_objects_write_locks(bucket, &objects, &mut opts).await {
+        let _object_lock_guards = Arc::new(match self.acquire_delete_objects_write_locks(bucket, &objects, &mut opts).await {
             Ok(guards) => guards,
             Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err),
-        };
+        });
+        opts.delete_mutation_owner = Some(Arc::new((opts.delete_mutation_owner.take(), _object_lock_guards.clone())));
         #[cfg(any(test, feature = "test-util"))]
         if !_object_lock_guards.is_empty() {
             notify_delete_namespace_acquired(bucket);
@@ -9241,8 +9290,31 @@ mod tests {
                 .expect("read original marker");
             assert_eq!(info.delete_marker_incarnation(), Some(old_incarnation));
         }
-        // Taking four actual members offline preserves their existing xl.meta.
-        // All deletion/recreation work below goes through the normal store API.
+        // Save each stale member's inline versions before converging the old
+        // bucket's delete receipts, then reintroduce them after recreation.
+        let mut stale_metadata = Vec::new();
+        for disk in before.iter().skip(12).flatten() {
+            for version in &historical {
+                let info = disk
+                    .read_version(
+                        "",
+                        bucket,
+                        "history.bin",
+                        &version.to_string(),
+                        &crate::disk::ReadOptions {
+                            read_data: true,
+                            ..Default::default()
+                        },
+                    )
+                    .await
+                    .expect("read stale inline data version");
+                assert!(info.inline_data() && info.data.is_some(), "fixture payload must be preserved in xl.meta");
+            }
+            stale_metadata.push((
+                disk.read_all(bucket, "history.bin/xl.meta").await.unwrap(),
+                disk.read_all(bucket, &format!("{object}/xl.meta")).await.unwrap(),
+            ));
+        }
         for slot in 12..16 {
             set.disks.write().await[slot] = None;
         }
@@ -9272,6 +9344,29 @@ mod tests {
             )
             .await
             .expect("delete exact old marker with four drives offline");
+        let pending = store
+            .handle_delete_bucket(bucket, &crate::storage_api_contracts::bucket::DeleteBucketOptions::default())
+            .await
+            .expect_err("offline participants require retained delete receipts");
+        assert!(matches!(
+            pending,
+            StorageError::BucketNotEmpty(_) | StorageError::BucketNotEmptyWithDetails { .. }
+        ));
+        assert!(
+            !crate::bucket::retirement::is_retired(store.clone(), bucket, old_incarnation)
+                .await
+                .unwrap()
+        );
+        *set.disks.write().await = before.clone();
+        store
+            .recover_delete_transactions_at_incarnation(
+                bucket,
+                None,
+                Some(old_incarnation),
+                &rustfs_heal_contracts::heal_channel::HealOpts::default(),
+            )
+            .await
+            .expect("converge old bucket deletes on all participants");
         store
             .handle_delete_bucket(bucket, &crate::storage_api_contracts::bucket::DeleteBucketOptions::default())
             .await
@@ -9281,6 +9376,9 @@ mod tests {
                 .await
                 .expect("durable retirement")
         );
+        for slot in 12..16 {
+            set.disks.write().await[slot] = None;
+        }
         store
             .handle_make_bucket(bucket, &create)
             .await
@@ -9306,6 +9404,13 @@ mod tests {
                 .expect("write new generation");
             new_versions.push((key, info.version_id.unwrap(), payload));
         }
+        for (slot, (history, marker)) in (12..16).zip(stale_metadata) {
+            for (key, bytes) in [("history.bin", history), (object, marker)] {
+                let root = dirs[slot].path().join(bucket).join(key);
+                tokio::fs::create_dir_all(&root).await.unwrap();
+                tokio::fs::write(root.join("xl.meta"), bytes).await.unwrap();
+            }
+        }
         *set.disks.write().await = before.clone();
         let heal = rustfs_heal_contracts::heal_channel::HealOpts {
             remove: true,
@@ -11477,3 +11582,160 @@ mod tests {
         .await;
     }
 }
+
+#[cfg(test)]
+mod missing_key_delete_quorum_tests {
+    use super::*;
+    use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
+    use crate::storage_api_contracts::bucket::{BucketOperations, BucketOptions, MakeBucketOptions};
+
+    async fn degraded_store(bucket: &str) -> (Vec<tempfile::TempDir>, Arc<ECStore>) {
+        let (dirs, store) = isolated_store_over_temp_disks().await;
+        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        store
+            .make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("create authoritative never-versioned bucket");
+        store.pools[0].disk_set[0].disks.write().await[0] = None;
+        let info = store
+            .get_bucket_info(bucket, &BucketOptions::default())
+            .await
+            .expect("existing bucket must remain readable at quorum");
+        assert_eq!(info.name, bucket);
+        let lookup = store
+            .get_object_info_for_delete(bucket, "never-existed", &ObjectOptions::default())
+            .await;
+        assert!(
+            lookup.as_ref().is_err_and(is_err_object_not_found),
+            "remaining disks must establish absence through production lookup: {lookup:?}"
+        );
+        (dirs, store)
+    }
+
+    #[tokio::test]
+    async fn single_delete_missing_key_preserves_quorum_noop() {
+        let (_dirs, store) = degraded_store("missing-key-quorum-single").await;
+        let result = store
+            .delete_object_with_tier_delete_journal("missing-key-quorum-single", "never-existed", ObjectOptions::default())
+            .await;
+        assert!(
+            result.as_ref().is_err_and(is_err_object_not_found),
+            "single missing-key delete must preserve its idempotent not-found outcome at quorum: {result:?}"
+        );
+    }
+
+    #[tokio::test]
+    async fn batch_delete_missing_key_preserves_quorum_noop() {
+        let (_dirs, store) = degraded_store("missing-key-quorum-batch").await;
+        let (_, errors, _) = store
+            .delete_objects_with_tier_delete_journal_and_accounting(
+                "missing-key-quorum-batch",
+                vec![ObjectToDelete {
+                    object_name: "never-existed".to_owned(),
+                    ..Default::default()
+                }],
+                ObjectOptions::default(),
+            )
+            .await;
+        assert_eq!(errors.len(), 1, "batch DELETE must return one result per key");
+        assert!(
+            errors.iter().all(Option::is_none),
+            "batch missing-key delete must preserve its idempotent success at quorum: {errors:?}"
+        );
+    }
+
+    #[cfg(not(windows))]
+    #[tokio::test]
+    async fn batch_delete_timeout_retains_physical_key_guard() {
+        use crate::disk::os::prepared_publication_test_hooks as hooks;
+        use crate::disk::{DiskAPI, ReadOptions};
+        use crate::storage_api_contracts::object::ObjectIO;
+        temp_env::async_with_vars([("RUSTFS_DELETE_RECOVERY_TIMEOUT_SECS", Some("30"))], async {
+            let (_dirs, store) = isolated_store_over_temp_disks().await;
+            init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+            let bucket = "review-batch-late-owner";
+            let key = "object";
+            store
+                .make_bucket(bucket, &MakeBucketOptions::default())
+                .await
+                .expect("fixture bucket");
+            let mut reader = PutObjReader::from_vec(vec![0x51; 1024 * 1024]);
+            store
+                .put_object(
+                    bucket,
+                    key,
+                    &mut reader,
+                    &ObjectOptions {
+                        write_completion: crate::object_api::WriteCompletion::TailDrained,
+                        ..Default::default()
+                    },
+                )
+                .await
+                .expect("seed real external object");
+            let disks = store.pools[0].disk_set[0].disk_inventory().await;
+            let disk = disks[0].as_ref().expect("online drive");
+            let info = disk
+                .read_version("", bucket, key, "", &ReadOptions::default())
+                .await
+                .expect("seed metadata");
+            assert!(!info.inline_data());
+            let crate::disk::Disk::Local(local) = disk.as_ref() else {
+                panic!("local disk");
+            };
+            let part = local
+                .get_disk()
+                .get_object_path_for_io(bucket, key)
+                .expect("leased key")
+                .join(info.data_dir.expect("external data directory").to_string())
+                .join("part.1");
+            let (entered_tx, entered_rx) = tokio::sync::oneshot::channel();
+            let (release, release_rx) = std::sync::mpsc::channel::<()>();
+            let hook = hooks::install_at(hooks::Stage::Remove, &part, move || {
+                let _ = entered_tx.send(());
+                let _ = release_rx.recv();
+            });
+            let deleting = store.clone();
+            let delete = tokio::spawn(async move {
+                deleting
+                    .delete_objects_with_tier_delete_journal_and_accounting(
+                        bucket,
+                        vec![ObjectToDelete {
+                            object_name: key.to_owned(),
+                            ..Default::default()
+                        }],
+                        ObjectOptions::default(),
+                    )
+                    .await
+            });
+            tokio::time::timeout(Duration::from_secs(10), entered_rx)
+                .await
+                .expect("physical cleanup entry")
+                .expect("entry hook");
+            assert!(part.exists(), "payload unlink has not run");
+            let (_, errors, _) = tokio::time::timeout(Duration::from_secs(40), delete)
+                .await
+                .expect("real settlement timeout must return")
+                .expect("delete task");
+            assert!(errors.iter().all(Option::is_none), "certified batch response: {errors:?}");
+            assert!(
+                store.ctx.namespace_commits_pending(),
+                "background physical operation must retain its owner"
+            );
+            let ns = store.handle_new_ns_lock(bucket, key).await.expect("same production key lock");
+            let probe = ns.get_write_lock(Duration::from_millis(300)).await;
+            let acquired_while_pending = probe.is_ok();
+            drop(probe);
+            drop(release);
+            drop(hook);
+            assert!(
+                !acquired_while_pending,
+                "batch released the exact-key guard while physical cleanup was still paused after its real recovery timeout"
+            );
+            let _released = ns
+                .get_write_lock(Duration::from_secs(10))
+                .await
+                .expect("physical cleanup must eventually release the exact-key guard");
+        })
+        .await;
+    }
+}

crates/heal/src/heal/storage.rs

@@ -574,6 +574,29 @@ pub trait HealStorageAPI: Send + Sync {
     /// Heal bucket using ecstore
     async fn heal_bucket(&self, bucket: &str, opts: &HealOpts) -> Result<HealResultItem>;
 
+    /// Recover transactions omitted by ordinary listings. Alternate providers
+    /// may explicitly expose no transaction-recovery capability.
+    async fn recover_delete_transactions(
+        &self,
+        _bucket: &str,
+        _object: Option<&str>,
+        _expected: Option<Uuid>,
+        _opts: &HealOpts,
+    ) -> Result<Option<HealResultItem>> {
+        Ok(None)
+    }
+
+    /// Recover hidden transactions only within the requested object prefix.
+    async fn recover_delete_prefix_transactions(
+        &self,
+        _bucket: &str,
+        _prefix: &str,
+        _expected: Option<Uuid>,
+        _opts: &HealOpts,
+    ) -> Result<Option<HealResultItem>> {
+        Ok(None)
+    }
+
     /// Heal format using ecstore
     async fn heal_format(&self, dry_run: bool) -> Result<(HealResultItem, Option<Error>)>;
 
@@ -1498,6 +1521,32 @@ impl HealStorageAPI for ECStoreHealStorage {
             .await)
     }
 
+    async fn recover_delete_transactions(
+        &self,
+        bucket: &str,
+        object: Option<&str>,
+        expected: Option<Uuid>,
+        opts: &HealOpts,
+    ) -> Result<Option<HealResultItem>> {
+        self.ecstore
+            .recover_delete_transactions_at_incarnation(bucket, object, expected, opts)
+            .await
+            .map_err(Error::Storage)
+    }
+
+    async fn recover_delete_prefix_transactions(
+        &self,
+        bucket: &str,
+        prefix: &str,
+        expected: Option<Uuid>,
+        opts: &HealOpts,
+    ) -> Result<Option<HealResultItem>> {
+        self.ecstore
+            .recover_delete_prefix_at_incarnation(bucket, prefix, expected, opts)
+            .await
+            .map_err(Error::Storage)
+    }
+
     async fn heal_bucket(&self, bucket: &str, opts: &HealOpts) -> Result<HealResultItem> {
         debug!(
             target: "rustfs::heal::storage",

crates/heal/src/heal/task/heal_bucket.rs

@@ -220,6 +220,22 @@ impl HealTask {
             set: self.options.set_index,
         };
 
+        let recovery_opts = HealOpts {
+            remove: self.source == HealRequestSource::Admin && self.options.remove_corrupted,
+            ..heal_opts
+        };
+        if let Some(result) = self
+            .await_with_control(self.storage.recover_delete_transactions(
+                bucket,
+                None,
+                self.bucket_incarnation_id,
+                &recovery_opts,
+            ))
+            .await?
+        {
+            self.record_result_item(result).await;
+        }
+
         let heal_result = match self.bucket_incarnation_id {
             Some(expected) => {
                 self.await_with_control(self.storage.heal_bucket_at_incarnation(bucket, expected, &heal_opts))
@@ -446,6 +462,26 @@ impl HealTask {
             "Heal prefix started"
         );
 
+        let opts = HealOpts {
+            dry_run: self.options.dry_run,
+            remove: self.source == HealRequestSource::Admin && self.options.remove_corrupted,
+            no_lock: self.options.no_lock,
+            pool: self.options.pool_index,
+            set: self.options.set_index,
+            ..Default::default()
+        };
+        if let Some(result) = self
+            .await_with_control(self.storage.recover_delete_prefix_transactions(
+                bucket,
+                prefix,
+                self.bucket_incarnation_id,
+                &opts,
+            ))
+            .await?
+        {
+            self.record_result_item(result).await;
+        }
+
         self.heal_bucket_objects(bucket, prefix).await
     }
 

crates/heal/src/heal/task/heal_erasure_set.rs

@@ -460,6 +460,21 @@ impl HealTask {
             set: self.options.set_index,
         };
 
+        let recovery_opts = HealOpts {
+            remove: self.source == HealRequestSource::Admin && self.options.remove_corrupted,
+            ..bucket_heal_opts
+        };
+        // Include metadata-less transactions for both startup and admin heals.
+        for bucket in &buckets {
+            self.check_control_flags().await?;
+            if let Some(result) = self
+                .await_with_control(self.storage.recover_delete_transactions(bucket, None, None, &recovery_opts))
+                .await?
+            {
+                self.record_result_item(result).await;
+            }
+        }
+
         for bucket in buckets.iter().filter(|_| self.source != HealRequestSource::Admin) {
             // Check control flags before starting each bucket heal
             self.check_control_flags().await?;

crates/heal/src/heal/task/heal_object.rs

@@ -99,6 +99,40 @@ impl HealTask {
             // recreated bucket after the replay anchor was captured.
             return self.heal_mrf_partial_write_object(bucket, object, version_id).await;
         }
+        let heal_opts = HealOpts {
+            recursive: self.options.recursive,
+            dry_run: self.options.dry_run,
+            remove: self.options.remove_corrupted,
+            recreate: self.options.recreate_missing,
+            scan_mode: self.options.scan_mode,
+            update_parity: self.options.update_parity,
+            no_lock: self.options.no_lock,
+            read_repair: self.source == HealRequestSource::ReadRepair,
+            pool: self.options.pool_index,
+            set: self.options.set_index,
+        };
+        let mut recovery_handled = false;
+        if matches!(self.source, HealRequestSource::Admin | HealRequestSource::Internal) {
+            let recovery_opts = HealOpts {
+                remove: heal_opts.remove && version_id.is_none_or(str::is_empty),
+                ..heal_opts
+            };
+            if let Some(result) = self
+                .await_with_control(self.storage.recover_delete_transactions(
+                    bucket,
+                    Some(object),
+                    self.bucket_incarnation_id,
+                    &recovery_opts,
+                ))
+                .await?
+            {
+                recovery_handled = true;
+                self.record_result_item(result).await;
+                if self.options.dry_run {
+                    return Ok(());
+                }
+            }
+        }
         let mut object_exists = match self.await_with_control(self.storage.object_exists(bucket, object)).await {
             Ok(exists) => exists,
             Err(err @ Error::TransientSkip { .. }) => {
@@ -129,7 +163,7 @@ impl HealTask {
             object
         };
 
-        if !object_exists {
+        if !object_exists && !recovery_handled {
             // Background loops (scanner/MRF/autoheal/read-repair) routinely
             // race object deletion, so a missing target is per-object noise
             // for them; only foreground admin/internal requests keep the warn.
@@ -197,18 +231,6 @@ impl HealTask {
             update_parity = self.options.update_parity,
             "Heal object stage entered"
         );
-        let heal_opts = HealOpts {
-            recursive: self.options.recursive,
-            dry_run: self.options.dry_run,
-            remove: self.options.remove_corrupted,
-            recreate: self.options.recreate_missing,
-            scan_mode: self.options.scan_mode,
-            update_parity: self.options.update_parity,
-            no_lock: self.options.no_lock,
-            read_repair: self.source == HealRequestSource::ReadRepair,
-            pool: self.options.pool_index,
-            set: self.options.set_index,
-        };
         let mut expected_identity =
             self.outcome_identity(bucket, object, version_id, self.options.pool_index, self.options.set_index);
         expected_identity.bucket_incarnation_id = self.outcome_bucket_incarnation_id(bucket, self.options.dry_run).await?;

crates/protos/src/generated/proto_gen/node_service.rs

@@ -527,6 +527,29 @@ pub struct DeletePathsResponse {
     #[prost(message, optional, tag = "2")]
     pub error: ::core::option::Option<Error>,
 }
+/// Protocol 1 is additive and has no fallback to generic file mutation RPCs.
+#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)]
+pub struct RecoverDeleteTransactionRequest {
+    #[prost(string, tag = "1")]
+    pub disk: ::prost::alloc::string::String,
+    #[prost(string, tag = "2")]
+    pub volume: ::prost::alloc::string::String,
+    #[prost(uint32, tag = "3")]
+    pub protocol_version: u32,
+    #[prost(bytes = "bytes", tag = "4")]
+    pub operation: ::prost::bytes::Bytes,
+}
+#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)]
+pub struct RecoverDeleteTransactionResponse {
+    #[prost(bool, tag = "1")]
+    pub success: bool,
+    #[prost(uint32, tag = "2")]
+    pub protocol_version: u32,
+    #[prost(bytes = "bytes", tag = "3")]
+    pub result: ::prost::bytes::Bytes,
+    #[prost(message, optional, tag = "4")]
+    pub error: ::core::option::Option<Error>,
+}
 #[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)]
 pub struct SnapshotLeaseRequest {
     #[prost(string, tag = "1")]
@@ -2082,6 +2105,21 @@ pub mod node_service_client {
                 .insert(GrpcMethod::new("node_service.NodeService", "DeleteAtIncarnation"));
             self.inner.unary(req, path, codec).await
         }
+        pub async fn recover_delete_transaction(
+            &mut self,
+            request: impl tonic::IntoRequest<super::RecoverDeleteTransactionRequest>,
+        ) -> std::result::Result<tonic::Response<super::RecoverDeleteTransactionResponse>, tonic::Status> {
+            self.inner
+                .ready()
+                .await
+                .map_err(|e| tonic::Status::unknown(format!("Service was not ready: {}", e.into())))?;
+            let codec = tonic_prost::ProstCodec::default();
+            let path = http::uri::PathAndQuery::from_static("/node_service.NodeService/RecoverDeleteTransaction");
+            let mut req = request.into_request();
+            req.extensions_mut()
+                .insert(GrpcMethod::new("node_service.NodeService", "RecoverDeleteTransaction"));
+            self.inner.unary(req, path, codec).await
+        }
         pub async fn acquire_snapshot_lease(
             &mut self,
             request: impl tonic::IntoRequest<super::SnapshotLeaseRequest>,
@@ -3463,6 +3501,10 @@ pub mod node_service_server {
             &self,
             request: tonic::Request<super::DeleteRequest>,
         ) -> std::result::Result<tonic::Response<super::DeleteResponse>, tonic::Status>;
+        async fn recover_delete_transaction(
+            &self,
+            request: tonic::Request<super::RecoverDeleteTransactionRequest>,
+        ) -> std::result::Result<tonic::Response<super::RecoverDeleteTransactionResponse>, tonic::Status>;
         async fn acquire_snapshot_lease(
             &self,
             request: tonic::Request<super::SnapshotLeaseRequest>,
@@ -4237,6 +4279,34 @@ pub mod node_service_server {
                     };
                     Box::pin(fut)
                 }
+                "/node_service.NodeService/RecoverDeleteTransaction" => {
+                    #[allow(non_camel_case_types)]
+                    struct RecoverDeleteTransactionSvc<T: NodeService>(pub Arc<T>);
+                    impl<T: NodeService> tonic::server::UnaryService<super::RecoverDeleteTransactionRequest> for RecoverDeleteTransactionSvc<T> {
+                        type Response = super::RecoverDeleteTransactionResponse;
+                        type Future = BoxFuture<tonic::Response<Self::Response>, tonic::Status>;
+                        fn call(&mut self, request: tonic::Request<super::RecoverDeleteTransactionRequest>) -> Self::Future {
+                            let inner = Arc::clone(&self.0);
+                            let fut = async move { <T as NodeService>::recover_delete_transaction(&inner, request).await };
+                            Box::pin(fut)
+                        }
+                    }
+                    let accept_compression_encodings = self.accept_compression_encodings;
+                    let send_compression_encodings = self.send_compression_encodings;
+                    let max_decoding_message_size = self.max_decoding_message_size;
+                    let max_encoding_message_size = self.max_encoding_message_size;
+                    let inner = self.inner.clone();
+                    let fut = async move {
+                        let method = RecoverDeleteTransactionSvc(inner);
+                        let codec = tonic_prost::ProstCodec::default();
+                        let mut grpc = tonic::server::Grpc::new(codec)
+                            .apply_compression_config(accept_compression_encodings, send_compression_encodings)
+                            .apply_max_message_size_config(max_decoding_message_size, max_encoding_message_size);
+                        let res = grpc.unary(method, req).await;
+                        Ok(res)
+                    };
+                    Box::pin(fut)
+                }
                 "/node_service.NodeService/AcquireSnapshotLease" => {
                     #[allow(non_camel_case_types)]
                     struct AcquireSnapshotLeaseSvc<T: NodeService>(pub Arc<T>);

crates/protos/src/lib.rs

@@ -1259,6 +1259,37 @@ pub fn canonical_prepare_part_transaction_request_body(
     Ok(body.finish())
 }
 
+/// Recovery messages are bounded and must contain exactly one MessagePack value.
+pub const DELETE_RECOVERY_MAX_PAYLOAD_BYTES: usize = DEFAULT_GRPC_SERVER_MESSAGE_LEN;
+pub const DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES: usize = 1024 * 1024;
+
+pub fn decode_delete_recovery_payload<T: serde::de::DeserializeOwned>(payload: &[u8]) -> std::io::Result<T> {
+    if payload.is_empty() || payload.len() > DELETE_RECOVERY_MAX_PAYLOAD_BYTES {
+        return Err(std::io::Error::new(
+            std::io::ErrorKind::InvalidData,
+            "invalid delete recovery payload length",
+        ));
+    }
+    let mut cursor = std::io::Cursor::new(payload);
+    let value = rmp_serde::from_read(&mut cursor)
+        .map_err(|_| std::io::Error::new(std::io::ErrorKind::InvalidData, "invalid delete recovery payload"))?;
+    if cursor.position() != u64::try_from(payload.len()).map_err(std::io::Error::other)? {
+        return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, "trailing delete recovery payload"));
+    }
+    Ok(value)
+}
+
+pub fn canonical_recover_delete_transaction_request_body(
+    request: &proto_gen::node_service::RecoverDeleteTransactionRequest,
+) -> Result<Vec<u8>, std::num::TryFromIntError> {
+    let mut body = CanonicalBodyBuilder::new(b"rustfs-recover-delete-transaction-request-v1\0");
+    body.push_str(&request.disk)?;
+    body.push_str(&request.volume)?;
+    body.push_u32(request.protocol_version);
+    body.push_bytes(&request.operation)?;
+    Ok(body.finish())
+}
+
 pub fn canonical_settle_part_transaction_request_body(
     request: &proto_gen::node_service::SettlePartTransactionRequest,
 ) -> Result<Vec<u8>, std::num::TryFromIntError> {
@@ -1313,6 +1344,40 @@ mod disk_mutation_canonical_tests {
     };
     use super::*;
 
+    #[test]
+    fn delete_recovery_canonical_body_binds_all_wire_fields() {
+        let request = proto_gen::node_service::RecoverDeleteTransactionRequest {
+            disk: "disk".into(),
+            volume: "volume".into(),
+            protocol_version: 1,
+            operation: vec![0x80].into(),
+        };
+        let mut bodies = vec![canonical_recover_delete_transaction_request_body(&request).expect("canonical body")];
+        for mutate in [
+            |r: &mut proto_gen::node_service::RecoverDeleteTransactionRequest| r.disk = "other".into(),
+            |r: &mut proto_gen::node_service::RecoverDeleteTransactionRequest| r.volume = "other".into(),
+            |r: &mut proto_gen::node_service::RecoverDeleteTransactionRequest| r.protocol_version = 2,
+            |r: &mut proto_gen::node_service::RecoverDeleteTransactionRequest| r.operation = vec![0x81].into(),
+        ] {
+            let mut changed = request.clone();
+            mutate(&mut changed);
+            bodies.push(canonical_recover_delete_transaction_request_body(&changed).expect("changed canonical body"));
+        }
+        assert_all_distinct(&bodies);
+    }
+
+    #[test]
+    fn delete_recovery_payload_rejects_trailing_malformed_and_oversized_values() {
+        let payload = rmp_serde::to_vec_named(&vec![1_u32, 2]).expect("encode test payload");
+        assert_eq!(decode_delete_recovery_payload::<Vec<u32>>(&payload).expect("one payload"), vec![1, 2]);
+        let mut trailing = payload;
+        trailing.push(0xc0);
+        assert!(decode_delete_recovery_payload::<Vec<u32>>(&trailing).is_err());
+        assert!(decode_delete_recovery_payload::<Vec<u32>>(&[]).is_err());
+        assert!(decode_delete_recovery_payload::<Vec<u32>>(&[0xc1]).is_err());
+        assert!(decode_delete_recovery_payload::<Vec<u32>>(&vec![0xc0; DELETE_RECOVERY_MAX_PAYLOAD_BYTES + 1]).is_err());
+    }
+
     fn assert_all_distinct(bodies: &[Vec<u8>]) {
         for (i, a) in bodies.iter().enumerate() {
             for (j, b) in bodies.iter().enumerate() {
@@ -2905,7 +2970,7 @@ pub async fn get_channel_for_class(addr: &str, class: ChannelClass) -> Result<Ch
                 debug!("Using cached control gRPC channel for: {}", addr);
                 return Ok(channel);
             }
-            return build_channel_until(addr, addr, deadline).await;
+            return build_channel_until(addr, addr, deadline, internode_rpc_timeout()).await;
         }
 
         let pool_size = bulk_channel_pool_size();
@@ -2915,7 +2980,7 @@ pub async fn get_channel_for_class(addr: &str, class: ChannelClass) -> Result<Ch
             debug!("Using cached bulk gRPC channel {} for: {}", idx, addr);
             return Ok(channel);
         }
-        build_channel_until(addr, &cache_key, deadline).await
+        build_channel_until(addr, &cache_key, deadline, internode_rpc_timeout()).await
     })
     .await
     .map_err(channel_acquisition_timeout)?
@@ -2943,13 +3008,45 @@ pub async fn create_new_channel(addr: &str) -> Result<Channel, Box<dyn Error>> {
 /// channels to the same peer be cached independently.
 async fn build_channel(dial_addr: &str, cache_key: &str) -> Result<Channel, Box<dyn Error>> {
     let deadline = tokio::time::Instant::now() + internode_connect_timeout();
-    build_channel_until(dial_addr, cache_key, deadline).await
+    build_channel_until(dial_addr, cache_key, deadline, internode_rpc_timeout()).await
+}
+
+/// A separate channel deadline for slow authenticated recovery work. The cache
+/// key keeps this deadline from changing foreground lock, read, and write RPCs.
+#[derive(Debug)]
+pub enum DeleteRecoveryChannelError {
+    Connection(Box<dyn Error>),
+}
+
+impl std::fmt::Display for DeleteRecoveryChannelError {
+    fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            Self::Connection(error) => write!(formatter, "delete recovery channel connection failed: {error}"),
+        }
+    }
+}
+
+impl Error for DeleteRecoveryChannelError {
+    fn source(&self) -> Option<&(dyn Error + 'static)> {
+        match self {
+            Self::Connection(error) => Some(error.as_ref()),
+        }
+    }
+}
+
+pub async fn create_channel_with_timeout(addr: &str, rpc_timeout: Duration) -> Result<Channel, DeleteRecoveryChannelError> {
+    let cache_key = format!("delete-recovery:{}:{addr}", rpc_timeout.as_millis());
+    let deadline = tokio::time::Instant::now() + internode_connect_timeout();
+    build_channel_until(addr, &cache_key, deadline, rpc_timeout)
+        .await
+        .map_err(DeleteRecoveryChannelError::Connection)
 }
 
 async fn build_channel_until(
     dial_addr: &str,
     cache_key: &str,
     deadline: tokio::time::Instant,
+    rpc_timeout: Duration,
 ) -> Result<Channel, Box<dyn Error>> {
     tokio::time::timeout_at(deadline, async {
         if let Some(channel) = cached_channel_for_tls_generation(dial_addr, cache_key).await {
@@ -2962,7 +3059,7 @@ async fn build_channel_until(
             if let Some(channel) = cached_channel_for_tls_generation(dial_addr, cache_key).await {
                 return Ok(channel);
             }
-            let (channel, generation) = dial_channel(dial_addr, cache_key).await?;
+            let (channel, generation) = dial_channel(dial_addr, cache_key, rpc_timeout).await?;
             if cache_dialed_channel(dial_addr, cache_key, &channel, generation).await
                 && let Some(channel) = cached_channel_for_tls_generation(dial_addr, cache_key).await
             {
@@ -2980,15 +3077,13 @@ async fn build_channel_until(
     .map_err(channel_acquisition_timeout)?
 }
 
-async fn dial_channel(dial_addr: &str, cache_key: &str) -> Result<(Channel, Option<u64>), Box<dyn Error>> {
+async fn dial_channel(dial_addr: &str, cache_key: &str, rpc_timeout: Duration) -> Result<(Channel, Option<u64>), Box<dyn Error>> {
     debug!("Creating new gRPC channel to: {} (cache key: {})", dial_addr, cache_key);
     let dial_started_at = Instant::now();
     let connect_timeout = internode_connect_timeout();
     let tcp_keepalive = internode_tcp_keepalive();
     let http2_keepalive_interval = internode_http2_keep_alive_interval();
     let http2_keepalive_timeout = internode_http2_keep_alive_timeout();
-    let rpc_timeout = internode_rpc_timeout();
-
     let mut connector = Endpoint::from_shared(dial_addr.to_string())?
         // Fast connection timeout for dead peer detection
         .connect_timeout(connect_timeout)
@@ -3291,6 +3386,31 @@ mod tests {
         }
     }
 
+    #[tokio::test]
+    async fn tls02_delete_recovery_channels_reject_unbound_and_stale_https_cache() {
+        let _test_guard = TLS02_TEST_LOCK.lock().await;
+        let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("reserve recovery peer address");
+        let addr = format!("https://{}", listener.local_addr().expect("recovery peer address"));
+        drop(listener);
+        let rpc_timeout = Duration::from_secs(20);
+        let cache_key = format!("delete-recovery:{}:{addr}", rpc_timeout.as_millis());
+        let generation = runtime_sources::outbound_tls_generation();
+        for cached_generation in [None, Some(generation.checked_add(1).unwrap_or(0))] {
+            cache_connection(cache_key.clone(), Endpoint::from_static("http://127.0.0.1:1").connect_lazy()).await;
+            if let Some(generation) = cached_generation {
+                TLS_GENERATION_CACHE.lock().await.insert(cache_key.clone(), generation);
+            } else {
+                TLS_GENERATION_CACHE.lock().await.remove(&cache_key);
+            }
+            assert!(
+                create_channel_with_timeout(&addr, rpc_timeout).await.is_err(),
+                "recovery must dial the unavailable HTTPS peer instead of returning an unbound or stale cached channel"
+            );
+            assert!(!rustfs_common::has_cached_connection(&cache_key).await);
+            assert!(!TLS_GENERATION_CACHE.lock().await.contains_key(&cache_key));
+        }
+    }
+
     #[tokio::test]
     async fn tls02_current_https_control_and_bulk_cache_entries_are_reused() {
         let _test_guard = TLS02_TEST_LOCK.lock().await;
@@ -3444,7 +3564,7 @@ mod tests {
         for _ in 0..8 {
             tasks.push(tokio::spawn(async move {
                 let deadline = tokio::time::Instant::now() + Duration::from_millis(40);
-                let result = build_channel_until(addr, addr, deadline).await;
+                let result = build_channel_until(addr, addr, deadline, internode_rpc_timeout()).await;
                 result.err().is_some_and(|error| {
                     error
                         .downcast_ref::<std::io::Error>()

crates/protos/src/node.proto

@@ -391,6 +391,21 @@ message DeletePathsResponse {
   optional Error error = 2;
 }
 
+// Protocol 1 is additive and has no fallback to generic file mutation RPCs.
+message RecoverDeleteTransactionRequest {
+  string disk = 1;
+  string volume = 2;
+  uint32 protocol_version = 3;
+  bytes operation = 4;
+}
+
+message RecoverDeleteTransactionResponse {
+  bool success = 1;
+  uint32 protocol_version = 2;
+  bytes result = 3;
+  optional Error error = 4;
+}
+
 message SnapshotLeaseRequest {
   string disk = 1;
   string volume = 2;
@@ -1218,6 +1233,7 @@ service NodeService {
   rpc CompareAndUpdateFile(CompareAndUpdateFileRequest) returns (CompareAndUpdateFileResponse) {}; // auth-policy: body-bound
   rpc Delete(DeleteRequest) returns (DeleteResponse) {}; // auth-policy: body-bound
   rpc DeleteAtIncarnation(DeleteRequest) returns (DeleteResponse) {}; // auth-policy: body-bound
+  rpc RecoverDeleteTransaction(RecoverDeleteTransactionRequest) returns (RecoverDeleteTransactionResponse) {}; // auth-policy: body-bound
   rpc AcquireSnapshotLease(SnapshotLeaseRequest) returns (SnapshotLeaseResponse) {}; // auth-policy: body-bound
   rpc RenewSnapshotLease(SnapshotLeaseRenewRequest) returns (SnapshotLeaseResponse) {}; // auth-policy: body-bound
   rpc ReleaseSnapshotLease(SnapshotLeaseReleaseRequest) returns (SnapshotLeaseMutationResponse) {}; // auth-policy: body-bound

docs/README.md

@@ -47,6 +47,9 @@ For concurrent bucket creation and deletion, see
 For persisted administrator bucket tasks and bucket recreation, see
 [Bucket heal recovery](operations/bucket-heal-recovery.md).
 
+For interrupted object deletions and returning drives, see
+[Delete transaction recovery](operations/delete-transaction-recovery.md).
+
 For disk replacement across VM restarts and schema 5/6 maintenance migration,
 see [Replacement generation recovery](operations/replacement-generation-recovery.md).
 

docs/operations/delete-transaction-recovery.md

@@ -0,0 +1,137 @@
+# Delete transaction recovery
+
+An interrupted object deletion can leave metadata backups, reserved shards, or
+an unpublished target. Normal S3 listings cannot discover all of these files.
+Recovery therefore inspects raw transaction records on every configured drive
+in the owning erasure set under the bucket incarnation fence and exact-key
+write lock. It does not modify GET or PUT behavior.
+
+## Decisions and replay
+
+New deletions assemble the backup, versioned prepared journal, and exact target
+in the private system temporary area, then atomically publish the complete
+transaction directory before modifying metadata. Interrupted preparation cannot
+leave an undiscoverable target-only record or temporary file in the object key. After the metadata write quorum, the coordinator publishes durable
+commit receipts to a write quorum before returning success. A failed or lost
+publication response is indeterminate: recovery must inspect the persisted
+records instead of immediately restoring the backup.
+
+| Evidence | Action |
+| --- | --- |
+| New prepared transaction, no decision, intact original shards and matching generation | Restore and sync the original metadata before removing preparation evidence. |
+| Structured commit receipt | Publish the certified target to matching returning participants and finish deletion. |
+| Missing, unavailable, or unsupported participant | Preserve records and retry when every configured participant can be inspected. |
+| Newer generation or partially overlapping version history | Preserve records and report blocked recovery. |
+| Historical backup or disk-local marker without a structured set decision | Preserve records during automatic recovery. |
+
+Missing local metadata does not authorize user-bucket residue cleanup, even
+when the current object exists on a metadata quorum. Legacy local cleanup is
+restricted to internal system-bucket maintenance. User-bucket legacy residue
+uses only the guarded set-wide removal described below. Dry-run and execution
+share the same source-generation check; an already-absent explicit non-null
+version can converge without touching unrelated live versions.
+
+Rollback preserves native multipart metadata and integrity sidecars. Protected
+parts require their declared proof generation; recovery verifies both the proof
+records and actual shard payloads before restoring metadata or removing
+recovery evidence. Missing or corrupt required proofs leave the transaction
+blocked with its backup intact.
+
+A disk-local `delete-data.<T>` marker does not prove an erasure-set decision.
+Returning drives cannot restore stale data after a certified deletion. Commit
+receipts remain until every participant and related transaction for the key
+has converged. For consecutive certified version deletions, an older decision
+can finish its own data cleanup after later deletion removes current metadata.
+A prepared copy needs a matching same-transaction decision before promotion;
+absence alone cannot certify it. Cross-transaction null-version supersession also requires causal
+predecessor evidence; timestamps alone cannot authorize it. Cleanup respects active snapshots and removes only validated
+transaction files, with directory removal performed nonrecursively.
+
+## Entry points
+
+Unclean-shutdown erasure-set healing performs raw transaction recovery before
+ordinary object listing. Administrator bucket, set, prefix, and object heal use the
+same recovery core. Existing heal selection, lifecycle admission, namespace
+locking, and movement guards still apply. Prefix recovery settles only matching
+keys, including transactions hidden from listings. Dry-run inspects and reports without
+settling transactions.
+
+An unconditional DELETE of a missing exact key, or an administrator heal with
+`remove` enabled, can authorize historical residue removal only when the
+bucket is authoritatively never-versioned and has no Object Lock. Every
+configured drive must be inspected, the current object must be absent, and all
+residue must match the guarded legacy layout. Versioned/suspended buckets,
+retained versions, conditions, unknown entries, symlinks, and prefix children
+are not covered by that authorization. The same fresh intent can certify the
+absence of an unprotected new prepared null-version transaction when every
+configured drive has no current metadata. This is a new authorized decision;
+it does not infer an old commit from missing causal evidence. Mixed or
+unrecognized historical layouts remain blocked.
+
+A missing specific-version DELETE first recovers hidden transactions and then
+reads the current version again. It must not acknowledge an unresolved hidden
+version as an idempotent no-op.
+
+## Upgrade and failure handling
+
+Upgrade all participating nodes before relying on the new protocol. Recovery
+uses one authenticated additive RPC; unsupported peers cannot silently accept
+new flags through the legacy delete endpoint. An online older participant
+causes the modern operation to fail before decision publication. Modern nodes
+also reject legacy forward transactional user-bucket deletes. A cluster still
+controlled by an older majority retains the older behavior. Do not downgrade
+while recovery records remain; old binaries cannot interpret the new protocol.
+
+Keep unavailable drives and transaction directories intact. Bring the full
+set online, run a scoped heal in dry-run mode, inspect its recovery result,
+and then run the same scoped heal without dry-run. Use `remove` for ambiguous
+historical residue only after verifying that the exact objects are intended
+to be deleted and the bucket meets the restrictions above. Do not recursively
+remove transaction directories by hand to make a bucket appear empty.
+
+Recovery reports discovered, committed, rolled-back, reclaimed historical,
+and blocked transaction counts. Unavailable participants return a retryable
+quorum error; blocked evidence requires operator investigation. A successful
+S3 delete response does not imply immediate physical cleanup while snapshots
+or returning participants still retain necessary proof.
+
+## Settlement deadline and durability
+
+`RUSTFS_DELETE_RECOVERY_TIMEOUT_SECS` controls the settlement RPC deadline,
+including full shard verification during rollback. Key and bucket lifecycle
+ownership stay attached to late local filesystem operations after cancellation
+or timeout. The default is 900 seconds;
+valid values are 30 through 86400 seconds. Invalid values fail explicitly.
+Increase it when intact large objects need more verification time. It does not
+change normal object read or delete-apply deadlines. A timeout can conceal a
+completed mutation, so rerun inspection and recovery rather than guessing the
+outcome from the RPC error.
+
+The records and restores follow the configured durability policy. Strict mode
+syncs the required metadata and directory changes. Process SIGKILL testing
+checks restart behavior; it does not establish power-loss behavior on every
+filesystem or storage device. Mode `none` intentionally offers weaker sync
+guarantees. Do not treat a throughput comparison as a durability substitute.
+
+## Reproduce the Docker regression suite
+
+Build the candidate with the test-only barriers enabled:
+
+```sh
+cargo build --locked --release -p rustfs --bin rustfs --no-default-features --features e2e-test-hooks
+python3 scripts/test/delete_recovery_docker.py --help
+```
+
+Pass a native Linux binary with `--binary`, a Linux image containing Python and
+curl with `--image`, and a unique Docker namespace. The optional
+`--baseline-binary` enables the mixed-version case. The harness uses synthetic
+S3 objects, localhost ports, labeled isolated volumes, and a non-root server.
+It stops only its own containers and keeps JSON results and logs in the chosen
+artifact directory. Production builds must not enable `e2e-test-hooks`.
+
+Run the preactivation/prepare/apply/decision process-stop cases, interrupted restore,
+returning and untouched peers, both generation orderings, version/retention
+and prefix protection, administrator dry-run/removal, response-loss,
+permission/corruption, and mixed-version cases. Preserve the source identity
+of both binaries with the results. A passing process-stop matrix does not
+cover hardware power cuts, Windows runtime behavior, or DELETE throughput.

rustfs/src/server/http.rs

@@ -4167,6 +4167,51 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn cold_delete_recovery_discovery_uses_production_body_auth() {
+        use storage::DiskAPI;
+        temp_env::async_with_vars([("RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT", Some("false"))], async {
+            let _ = rustfs_credentials::set_global_rpc_secret("rpc-http-test-secret".to_string());
+            let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind recovery RPC peer");
+            let addr = listener.local_addr().expect("recovery RPC address");
+            let previous_node_name = rustfs_common::get_global_local_node_name().await;
+            rustfs_common::set_global_local_node_name(&addr.to_string()).await;
+            let node = InterceptedService::new(NodeServiceServer::new(make_server()), check_auth);
+            let service = RpcRequestPathService::new(Routes::new(node).prepare());
+            let server = tokio::spawn(async move {
+                let (socket, _) = listener.accept().await.expect("accept recovery RPC connection");
+                ConnBuilder::new(TokioExecutor::new())
+                    .serve_connection(TokioIo::new(socket), TowerToHyperService::new(service))
+                    .await
+                    .expect("serve recovery RPC connection");
+            });
+            let mut endpoint =
+                Endpoint::try_from(format!("http://{addr}/cold-recovery-disk").as_str()).expect("remote recovery endpoint");
+            endpoint.is_local = false;
+            let disk = storage::new_disk(
+                &endpoint,
+                &storage::DiskOption {
+                    cleanup: false,
+                    health_check: false,
+                },
+            )
+            .await
+            .expect("production remote disk client");
+            // Recovery may race a heal-control peer-state refresh, before another
+            // authenticated response has repopulated the shared peer state.
+            let result = disk.discover_delete_transactions("cold-recovery-bucket", None, None, 1).await;
+            server.abort();
+            let _ = server.await;
+            rustfs_common::set_global_local_node_name(&previous_node_name).await;
+            assert!(
+                matches!(result, Err(crate::storage_api::error::DiskError::DiskNotFound)),
+                "the cold request must pass production authentication and reach the missing-disk handler: {result:?}"
+            );
+        })
+        .await;
+    }
+
     #[tokio::test]
     #[serial_test::serial]
     async fn peer_rest_heal_control_uses_production_auth_and_keeps_validation_errors_online() {

rustfs/src/storage/rpc/node_service.rs

@@ -1536,6 +1536,13 @@ impl Node for NodeService {
         self.handle_rename_part(request).await
     }
 
+    async fn recover_delete_transaction(
+        &self,
+        request: Request<RecoverDeleteTransactionRequest>,
+    ) -> Result<Response<RecoverDeleteTransactionResponse>, Status> {
+        self.handle_recover_delete_transaction(request).await
+    }
+
     async fn settle_part_transaction(
         &self,
         request: Request<SettlePartTransactionRequest>,
@@ -3073,13 +3080,14 @@ mod tests {
         LoadPolicyMappingRequest, LoadPolicyRequest, LoadRebalanceMetaRequest, LoadServiceAccountRequest,
         LoadTransitionTierConfigRequest, LoadUserRequest, LocalStorageInfoRequest, MakeBucketRequest, MakeVolumeRequest,
         MakeVolumesRequest, Mss, PingRequest, PreparePartTransactionRequest, ReadAllRequest, ReadAtRequest, ReadMultipleRequest,
-        ReadVersionRequest, ReadXlRequest, ReloadPoolMetaRequest, ReloadSiteReplicationConfigRequest, RenameDataRequest,
-        RenameFileRequest, RenamePartRequest, ScannerActivityRequest, ScannerDirtyUsageSnapshotRequest,
-        ScannerPublicationLeaseReleaseRequest, ScannerPublicationLeaseRequest, ServerInfoRequest, SettlePartTransactionRequest,
-        SignalServiceRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest, SnapshotLeaseRequest,
-        StartDecommissionRequest, StartProfilingRequest, StatVolumeRequest, StopRebalanceRequest, TierMutationAbortRequest,
-        TierMutationFailureClass, TierMutationPeerState, TierMutationPrepareRequest, UpdateMetacacheListingRequest,
-        UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest, WriteMetadataRequest, WriteRequest,
+        ReadVersionRequest, ReadXlRequest, RecoverDeleteTransactionRequest, ReloadPoolMetaRequest,
+        ReloadSiteReplicationConfigRequest, RenameDataRequest, RenameFileRequest, RenamePartRequest, ScannerActivityRequest,
+        ScannerDirtyUsageSnapshotRequest, ScannerPublicationLeaseReleaseRequest, ScannerPublicationLeaseRequest,
+        ServerInfoRequest, SettlePartTransactionRequest, SignalServiceRequest, SnapshotLeaseReleaseRequest,
+        SnapshotLeaseRenewRequest, SnapshotLeaseRequest, StartDecommissionRequest, StartProfilingRequest, StatVolumeRequest,
+        StopRebalanceRequest, TierMutationAbortRequest, TierMutationFailureClass, TierMutationPeerState,
+        TierMutationPrepareRequest, UpdateMetacacheListingRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest,
+        WriteMetadataRequest, WriteRequest,
         heal_control_service_client::HealControlServiceClient,
         heal_control_service_server::{HealControlService as _, HealControlServiceServer},
         node_service_client::NodeServiceClient,
@@ -3097,7 +3105,7 @@ mod tests {
     use tonic::{Request, Response, Status};
     use uuid::Uuid;
 
-    const DISK_MUTATION_RPC_METHODS: [&str; 24] = [
+    const DISK_MUTATION_RPC_METHODS: [&str; 25] = [
         "renamedata",
         "renamedataatincarnation",
         "writemetadataatincarnation",
@@ -3116,6 +3124,7 @@ mod tests {
         "renamepart",
         "prepareparttransaction",
         "settleparttransaction",
+        "recoverdeletetransaction",
         "deletevolume",
         "makevolume",
         "makevolumes",
@@ -4386,6 +4395,177 @@ mod tests {
         assert!(!signed.into_inner().success, "the unknown test disk cannot resolve");
     }
 
+    #[tokio::test]
+    async fn delete_recovery_apply_rejects_excessive_batch_cardinality_before_lookup() {
+        use crate::storage::storage_api::{DeleteOptions, DeleteRecoveryRpcOperation, FileInfoVersions};
+        let service = make_server();
+        for versions in [
+            vec![FileInfoVersions::default(); 1001],
+            vec![FileInfoVersions {
+                name: "object".into(),
+                versions: vec![Default::default(); 1001],
+                ..Default::default()
+            }],
+        ] {
+            let operation = DeleteRecoveryRpcOperation::ApplyVersions {
+                versions,
+                opts: DeleteOptions {
+                    old_data_dir: Some(Uuid::new_v4()),
+                    ..Default::default()
+                },
+            };
+            let mut request = Request::new(RecoverDeleteTransactionRequest {
+                disk: "unknown".into(),
+                volume: "bucket".into(),
+                protocol_version: 1,
+                operation: rmp_serde::to_vec_named(&operation).expect("encode batch").into(),
+            });
+            let body =
+                rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref()).expect("canonical batch");
+            set_tonic_canonical_body_digest(&mut request, &body).expect("batch digest");
+            mark_v2_authenticated(&mut request);
+            let error = service
+                .recover_delete_transaction(request)
+                .await
+                .expect_err("oversized batch must reject before disk lookup");
+            assert_eq!(error.code(), tonic::Code::InvalidArgument);
+        }
+    }
+
+    #[tokio::test]
+    async fn delete_recovery_apply_rejects_legacy_cleanup_options_and_absolute_keys() {
+        use crate::storage::storage_api::{DeleteOptions, DeleteRecoveryRpcOperation};
+        let service = make_server();
+        let valid = DeleteOptions {
+            old_data_dir: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+        let mut invalid = vec![DeleteOptions::default()];
+        for mutate in [
+            |opts: &mut DeleteOptions| opts.old_data_dir = Some(Uuid::nil()),
+            |opts: &mut DeleteOptions| opts.undo_delete = true,
+            |opts: &mut DeleteOptions| opts.undo_write = true,
+            |opts: &mut DeleteOptions| opts.recursive = true,
+            |opts: &mut DeleteOptions| opts.immediate = true,
+        ] {
+            let mut opts = valid.clone();
+            mutate(&mut opts);
+            invalid.push(opts);
+        }
+        let mut operations = Vec::new();
+        for opts in invalid {
+            operations.push(DeleteRecoveryRpcOperation::ApplyVersion {
+                object: "object".into(),
+                file_info: Default::default(),
+                force_del_marker: false,
+                opts: opts.clone(),
+            });
+            operations.push(DeleteRecoveryRpcOperation::ApplyVersions {
+                versions: Vec::new(),
+                opts,
+            });
+        }
+        operations.push(DeleteRecoveryRpcOperation::ApplyVersion {
+            object: "/object".into(),
+            file_info: Default::default(),
+            force_del_marker: false,
+            opts: valid.clone(),
+        });
+        operations.push(DeleteRecoveryRpcOperation::ApplyVersions {
+            versions: vec![crate::storage::storage_api::FileInfoVersions {
+                name: "/object".into(),
+                ..Default::default()
+            }],
+            opts: valid,
+        });
+        for operation in operations {
+            let mut request = Request::new(RecoverDeleteTransactionRequest {
+                disk: "unknown".into(),
+                volume: "bucket".into(),
+                protocol_version: 1,
+                operation: rmp_serde::to_vec_named(&operation).expect("encode apply operation").into(),
+            });
+            let body =
+                rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref()).expect("canonical apply");
+            set_tonic_canonical_body_digest(&mut request, &body).expect("apply digest");
+            mark_v2_authenticated(&mut request);
+            let error = service
+                .recover_delete_transaction(request)
+                .await
+                .expect_err("invalid apply must reject before disk lookup");
+            assert_eq!(error.code(), tonic::Code::InvalidArgument);
+        }
+    }
+
+    #[tokio::test]
+    async fn delete_recovery_requires_digest_and_valid_protocol_payload() {
+        let service = make_server();
+        let operation = crate::storage::storage_api::DeleteRecoveryRpcOperation::Discover {
+            object: None,
+            cursor: None,
+            limit: 1,
+        };
+        let message = RecoverDeleteTransactionRequest {
+            disk: "unknown".into(),
+            volume: "bucket".into(),
+            protocol_version: 1,
+            operation: rmp_serde::to_vec_named(&operation).expect("encode discovery").into(),
+        };
+        let unsigned = service
+            .recover_delete_transaction(Request::new(message.clone()))
+            .await
+            .expect_err("new RPC must reject unsigned requests");
+        assert_eq!(unsigned.code(), tonic::Code::PermissionDenied);
+        for digest in ["", "UNSIGNED-PAYLOAD"] {
+            let mut request = Request::new(message.clone());
+            request
+                .metadata_mut()
+                .insert("x-rustfs-content-sha256", digest.parse().expect("metadata"));
+            mark_v2_authenticated(&mut request);
+            let error = service
+                .recover_delete_transaction(request)
+                .await
+                .expect_err("missing or unsigned digest");
+            assert_eq!(error.code(), tonic::Code::PermissionDenied);
+        }
+        let mut invalid = message.clone();
+        invalid.protocol_version = 2;
+        let mut malformed = message.clone();
+        malformed.operation = vec![0xc1].into();
+        let mut trailing = message.clone();
+        let mut payload = trailing.operation.to_vec();
+        payload.push(0xc0);
+        trailing.operation = payload.into();
+        for (message, expected) in [
+            (invalid, tonic::Code::Unimplemented),
+            (malformed, tonic::Code::InvalidArgument),
+            (trailing, tonic::Code::InvalidArgument),
+        ] {
+            let mut request = Request::new(message);
+            let body =
+                rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref()).expect("canonical recovery");
+            set_tonic_canonical_body_digest(&mut request, &body).expect("body digest");
+            mark_v2_authenticated(&mut request);
+            let error = service
+                .recover_delete_transaction(request)
+                .await
+                .expect_err("invalid recovery input");
+            assert_eq!(error.code(), expected);
+        }
+        let mut request = Request::new(message);
+        let body =
+            rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref()).expect("canonical recovery");
+        set_tonic_canonical_body_digest(&mut request, &body).expect("body digest");
+        mark_v2_authenticated(&mut request);
+        let response = service
+            .recover_delete_transaction(request)
+            .await
+            .expect("valid request reaches disk lookup")
+            .into_inner();
+        assert!(!response.success);
+        assert!(response.error.is_some());
+    }
+
     /// Per-handler wiring check for every mutating disk RPC. A mismatched digest must be rejected
     /// (catches a handler that omits its `verify_disk_mutation_digest` gate) and a correctly
     /// body-bound digest must pass the gate (catches a handler wired to the wrong
@@ -4659,6 +4839,16 @@ mod tests {
             },
             rustfs_protos::canonical_prepare_part_transaction_request_body
         );
+        assert_gated!(
+            recover_delete_transaction,
+            RecoverDeleteTransactionRequest {
+                disk: disk.clone(),
+                volume: "dst".into(),
+                protocol_version: 1,
+                operation: vec![0x80].into()
+            },
+            rustfs_protos::canonical_recover_delete_transaction_request_body
+        );
         assert_gated!(
             settle_part_transaction,
             SettlePartTransactionRequest {
@@ -6597,7 +6787,7 @@ mod tests {
         };
         let rollback_dir = Uuid::new_v4();
         remote
-            .delete_version(
+            .delete_version_recoverable(
                 bucket,
                 object,
                 marker.clone(),

rustfs/src/storage/rpc/node_service/disk.rs

@@ -19,7 +19,10 @@ use crate::storage::storage_api::rpc_consumer::node_service::{
     validate_batch_read_version_item_count,
 };
 use crate::storage::storage_api::runtime_sources_consumer::runtime_sources;
-use crate::storage::storage_api::{PartTransactionAction, RenameDataResp, SnapshotLeaseToken, verify_tonic_mutation_body_digest};
+use crate::storage::storage_api::{
+    DELETE_RECOVERY_PROTOCOL_VERSION, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, PartTransactionAction, RenameDataResp,
+    SnapshotLeaseToken, verify_tonic_canonical_body_digest, verify_tonic_mutation_body_digest,
+};
 use bytes::Bytes;
 use rustfs_filemeta::FileInfo;
 use rustfs_io_metrics::internode_metrics::{
@@ -118,6 +121,16 @@ const MSGPACK_ENCODE_CAPACITY_HINT: usize = 512;
 const FILE_INFO_MSGPACK_ENCODE_CAPACITY_HINT: usize = 1024;
 const SNAPSHOT_LEASE_PROTOCOL_VERSION: u32 = 1;
 
+fn rejects_legacy_delete_transaction(volume: &str, opts: &DeleteOptions) -> bool {
+    !(volume == ".rustfs.sys"
+        || volume.starts_with(".rustfs.sys/")
+        || volume == ".minio.sys"
+        || volume.starts_with(".minio.sys/"))
+        && opts.old_data_dir.is_some()
+        && !opts.recoverable_delete
+        && !opts.undo_write
+}
+
 fn snapshot_lease_response(result: Result<SnapshotLeaseToken, DiskError>) -> Response<SnapshotLeaseResponse> {
     match result {
         Ok(token) => Response::new(SnapshotLeaseResponse {
@@ -715,6 +728,14 @@ impl NodeService {
                 }
             };
 
+            if rejects_legacy_delete_transaction(&request.volume, &opts) {
+                return Ok(Response::new(DeleteVersionsResponse {
+                    success: false,
+                    errors: Vec::new(),
+                    error: Some(DiskError::UnsupportedDisk.into()),
+                    item_errors: Vec::new(),
+                }));
+            }
             let (errors, item_errors) =
                 encode_delete_versions_errors(disk.delete_versions(&request.volume, versions, opts).await);
 
@@ -782,6 +803,13 @@ impl NodeService {
                 }));
             }
         };
+        if rejects_legacy_delete_transaction(&request.volume, &opts) {
+            return Ok(Response::new(DeleteVersionResponse {
+                success: false,
+                raw_file_info: String::new(),
+                error: Some(DiskError::UnsupportedDisk.into()),
+            }));
+        }
         if require_marker_condition && opts.expected_delete_marker.is_none() {
             return Err(Status::invalid_argument("retired marker deletion requires a marker precondition"));
         }
@@ -1578,6 +1606,131 @@ impl NodeService {
         }
     }
 
+    pub(super) async fn handle_recover_delete_transaction(
+        &self,
+        request: Request<RecoverDeleteTransactionRequest>,
+    ) -> Result<Response<RecoverDeleteTransactionResponse>, Status> {
+        if request.get_ref().operation.len() > rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES {
+            return Err(Status::resource_exhausted("delete recovery operation exceeds payload limit"));
+        }
+        let body = rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref())
+            .map_err(|_| Status::invalid_argument("delete recovery request length cannot be represented"))?;
+        // This additive protocol has no digestless rolling-upgrade exception.
+        verify_tonic_canonical_body_digest(&request, &body)
+            .map_err(|_| Status::permission_denied("delete recovery requires body-bound v2 authentication"))?;
+        let request = request.into_inner();
+        if request.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION {
+            return Err(Status::unimplemented("incompatible delete recovery protocol"));
+        }
+        let operation: DeleteRecoveryRpcOperation = rustfs_protos::decode_delete_recovery_payload(&request.operation)
+            .map_err(|_| Status::invalid_argument("invalid delete recovery operation"))?;
+        if request.operation.len() > operation.max_wire_payload_bytes() {
+            return Err(Status::resource_exhausted("delete recovery operation exceeds payload limit"));
+        }
+        match &operation {
+            DeleteRecoveryRpcOperation::ApplyVersions { versions, .. }
+                if versions.len() > 1000
+                    || versions
+                        .iter()
+                        .try_fold(0_usize, |count, versions| count.checked_add(versions.versions.len()))
+                        .is_none_or(|count| count > 1000) =>
+            {
+                return Err(Status::invalid_argument("recoverable batch delete exceeds 1000 object/version requests"));
+            }
+            DeleteRecoveryRpcOperation::Discover { limit, .. } if *limit == 0 || *limit > 256 => {
+                return Err(Status::invalid_argument("delete recovery page limit must be between 1 and 256"));
+            }
+            DeleteRecoveryRpcOperation::Inspect { transaction, .. } | DeleteRecoveryRpcOperation::Settle { transaction, .. }
+                if transaction.is_nil() =>
+            {
+                return Err(Status::invalid_argument("delete transaction must be a non-nil UUID"));
+            }
+            DeleteRecoveryRpcOperation::ApplyVersion { opts, .. } | DeleteRecoveryRpcOperation::ApplyVersions { opts, .. }
+                if opts.old_data_dir.is_none_or(|transaction| transaction.is_nil())
+                    || opts.undo_delete
+                    || opts.undo_write
+                    || opts.recursive
+                    || opts.immediate
+                    || opts.expected_delete_marker.is_some()
+                    || opts
+                        .delete_recovery_bucket_incarnation
+                        .is_some_and(|incarnation| incarnation.is_nil()) =>
+            {
+                return Err(Status::invalid_argument(
+                    "recoverable delete requires a non-nil transaction and valid apply options",
+                ));
+            }
+            DeleteRecoveryRpcOperation::ApplyVersion { object, .. } if object.starts_with('/') => {
+                return Err(Status::invalid_argument("recoverable delete requires a relative object key"));
+            }
+            DeleteRecoveryRpcOperation::ApplyVersions { versions, .. }
+                if versions.iter().any(|version| version.name.starts_with('/')) =>
+            {
+                return Err(Status::invalid_argument("recoverable delete requires relative object keys"));
+            }
+            _ => {}
+        }
+        let result = if let Some(disk) = self.find_disk(&request.disk).await {
+            match operation {
+                DeleteRecoveryRpcOperation::ApplyVersion {
+                    object,
+                    file_info,
+                    force_del_marker,
+                    opts,
+                } => disk
+                    .delete_version_recoverable(&request.volume, &object, *file_info, force_del_marker, opts)
+                    .await
+                    .map(|()| DeleteRecoveryRpcReply::AppliedVersion),
+                DeleteRecoveryRpcOperation::ApplyVersions { versions, opts } => {
+                    let errors = disk.delete_versions_recoverable(&request.volume, versions, opts).await;
+                    Ok(DeleteRecoveryRpcReply::AppliedVersions(
+                        errors.into_iter().map(|error| error.map(|error| error.to_u32())).collect(),
+                    ))
+                }
+                DeleteRecoveryRpcOperation::Discover { object, cursor, limit } => disk
+                    .discover_delete_transactions(&request.volume, object.as_deref(), cursor, limit)
+                    .await
+                    .map(DeleteRecoveryRpcReply::Discovered),
+                DeleteRecoveryRpcOperation::Inspect { object, transaction } => disk
+                    .inspect_delete_transaction(&request.volume, &object, transaction)
+                    .await
+                    .map(DeleteRecoveryRpcReply::Inspected),
+                DeleteRecoveryRpcOperation::Settle {
+                    object,
+                    transaction,
+                    expected_fingerprint,
+                    action,
+                } => disk
+                    .settle_delete_transaction(&request.volume, &object, transaction, expected_fingerprint, action)
+                    .await
+                    .map(DeleteRecoveryRpcReply::Settled),
+            }
+        } else {
+            Err(DiskError::DiskNotFound)
+        };
+        match result {
+            Ok(reply) => {
+                let result = encode_msgpack_named(&reply, "delete recovery reply")
+                    .map_err(|_| Status::internal("cannot encode delete recovery reply"))?;
+                if result.len() > rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES {
+                    return Err(Status::resource_exhausted("delete recovery reply exceeds payload limit"));
+                }
+                Ok(Response::new(RecoverDeleteTransactionResponse {
+                    success: true,
+                    protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
+                    result: result.into(),
+                    error: None,
+                }))
+            }
+            Err(error) => Ok(Response::new(RecoverDeleteTransactionResponse {
+                success: false,
+                protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
+                result: Bytes::new(),
+                error: Some(error.into()),
+            })),
+        }
+    }
+
     pub(super) async fn handle_settle_part_transaction(
         &self,
         request: Request<SettlePartTransactionRequest>,
@@ -2007,9 +2160,10 @@ impl NodeService {
 #[cfg(test)]
 mod tests {
     use super::{
-        compat_response_json, decode_msgpack_or_json, decode_rename_data_request_file_info,
+        DeleteOptions, compat_response_json, decode_msgpack_or_json, decode_rename_data_request_file_info,
         encode_batch_read_version_response_payloads, encode_delete_versions_errors, encode_file_info_msgpack, encode_msgpack,
         encode_msgpack_named, encode_read_multiple_response_payloads, encode_rename_data_response_payloads,
+        rejects_legacy_delete_transaction,
     };
     use crate::storage::DiskError;
     use crate::storage::rpc::node_service::make_server;
@@ -2022,6 +2176,7 @@ mod tests {
     use serde::{Deserialize, Serialize};
     use serial_test::serial;
     use tonic::Request;
+    use uuid::Uuid;
 
     #[derive(Debug, PartialEq, Eq, Serialize, Deserialize)]
     struct SamplePayload {
@@ -2029,6 +2184,26 @@ mod tests {
         count: u32,
     }
 
+    #[test]
+    fn legacy_delete_transaction_policy_preserves_rollback_and_internal_maintenance() {
+        let mut opts = DeleteOptions {
+            old_data_dir: Some(Uuid::new_v4()),
+            ..Default::default()
+        };
+        assert!(rejects_legacy_delete_transaction("bucket", &opts));
+        assert!(!rejects_legacy_delete_transaction(".rustfs.sys", &opts));
+        assert!(!rejects_legacy_delete_transaction(".rustfs.sys/tmp", &opts));
+        assert!(!rejects_legacy_delete_transaction(".minio.sys", &opts));
+        opts.undo_write = true;
+        assert!(!rejects_legacy_delete_transaction("bucket", &opts));
+        opts.undo_write = false;
+        opts.recoverable_delete = true;
+        assert!(!rejects_legacy_delete_transaction("bucket", &opts));
+        opts.recoverable_delete = false;
+        opts.old_data_dir = None;
+        assert!(!rejects_legacy_delete_transaction("bucket", &opts));
+    }
+
     #[test]
     fn response_compat_send_sites_keep_manifest_json_encoders() {
         // Rolling-upgrade contract (rustfs-protos compat manifest): every

rustfs/src/storage/storage_api.rs

@@ -478,11 +478,12 @@ pub(crate) mod ecstore_data_usage {
 #[allow(unused_imports)]
 pub(crate) mod ecstore_disk {
     pub(crate) use rustfs_ecstore::api::disk::{
-        BUCKET_META_PREFIX, BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DeleteOptions,
-        DiskAPI, DiskInfo, DiskInfoOptions, DiskStore, FileInfoVersions, FileReader, FileWriter, OldCurrentSize,
-        PartTransactionAction, RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp,
-        SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, get_object_disk_read_timeout,
-        validate_batch_read_version_item_count,
+        BUCKET_META_PREFIX, BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate,
+        DELETE_RECOVERY_PROTOCOL_VERSION, DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest,
+        DeleteRecoveryInspection, DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DiskAPI, DiskInfo,
+        DiskInfoOptions, DiskStore, FileInfoVersions, FileReader, FileWriter, OldCurrentSize, PartTransactionAction,
+        RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, SnapshotLeaseToken,
+        UpdateMetadataOpts, VolumeInfo, WalkDirOptions, get_object_disk_read_timeout, validate_batch_read_version_item_count,
     };
     #[cfg(test)]
     pub(crate) use rustfs_ecstore::api::disk::{DiskOption, new_disk};
@@ -734,6 +735,14 @@ pub(crate) type ReadMultipleReq = ecstore_disk::ReadMultipleReq;
 pub(crate) type ReadMultipleResp = ecstore_disk::ReadMultipleResp;
 pub(crate) type ReadOptions = ecstore_disk::ReadOptions;
 pub(crate) type OldCurrentSize = ecstore_disk::OldCurrentSize;
+pub(crate) type DeleteRecoveryCursor = ecstore_disk::DeleteRecoveryCursor;
+pub(crate) type DeleteRecoveryPage = ecstore_disk::DeleteRecoveryPage;
+pub(crate) type DeleteRecoveryInspection = ecstore_disk::DeleteRecoveryInspection;
+pub(crate) type DeleteRecoveryDigest = ecstore_disk::DeleteRecoveryDigest;
+pub(crate) type DeleteRecoveryAction = ecstore_disk::DeleteRecoveryAction;
+pub(crate) type DeleteRecoveryRpcOperation = ecstore_disk::DeleteRecoveryRpcOperation;
+pub(crate) type DeleteRecoveryRpcReply = ecstore_disk::DeleteRecoveryRpcReply;
+pub(crate) const DELETE_RECOVERY_PROTOCOL_VERSION: u32 = ecstore_disk::DELETE_RECOVERY_PROTOCOL_VERSION;
 pub(crate) type PartTransactionAction = ecstore_disk::PartTransactionAction;
 pub(crate) type RenameDataResp = ecstore_disk::RenameDataResp;
 pub(crate) type ReplicationStatusType = ecstore_bucket::replication::ReplicationStatusType;
@@ -1444,6 +1453,41 @@ pub(crate) trait StorageDiskRpcExt {
         dst_path: &str,
         meta: bytes::Bytes,
     ) -> DiskResult<()>;
+    async fn delete_version_recoverable(
+        &self,
+        volume: &str,
+        object: &str,
+        file_info: rustfs_filemeta::FileInfo,
+        force_del_marker: bool,
+        opts: DeleteOptions,
+    ) -> DiskResult<()>;
+    async fn delete_versions_recoverable(
+        &self,
+        volume: &str,
+        versions: Vec<FileInfoVersions>,
+        opts: DeleteOptions,
+    ) -> Vec<Option<DiskError>>;
+    async fn discover_delete_transactions(
+        &self,
+        volume: &str,
+        object: Option<&str>,
+        cursor: Option<DeleteRecoveryCursor>,
+        limit: u32,
+    ) -> DiskResult<DeleteRecoveryPage>;
+    async fn inspect_delete_transaction(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: uuid::Uuid,
+    ) -> DiskResult<DeleteRecoveryInspection>;
+    async fn settle_delete_transaction(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: uuid::Uuid,
+        expected_fingerprint: DeleteRecoveryDigest,
+        action: DeleteRecoveryAction,
+    ) -> DiskResult<DeleteRecoveryInspection>;
     async fn settle_part_transaction(&self, volume: &str, path: &str, action: PartTransactionAction) -> DiskResult<()>;
     async fn delete(&self, volume: &str, path: &str, options: DeleteOptions) -> DiskResult<()>;
     async fn verify_file(&self, volume: &str, path: &str, file_info: &rustfs_filemeta::FileInfo) -> DiskResult<CheckPartsResp>;
@@ -1616,6 +1660,53 @@ where
         ecstore_disk::DiskAPI::prepare_part_transaction(self, src_volume, src_path, dst_volume, dst_path, meta).await
     }
 
+    async fn delete_version_recoverable(
+        &self,
+        volume: &str,
+        object: &str,
+        file_info: rustfs_filemeta::FileInfo,
+        force_del_marker: bool,
+        opts: DeleteOptions,
+    ) -> DiskResult<()> {
+        ecstore_disk::DiskAPI::delete_version_recoverable(self, volume, object, file_info, force_del_marker, opts).await
+    }
+    async fn delete_versions_recoverable(
+        &self,
+        volume: &str,
+        versions: Vec<FileInfoVersions>,
+        opts: DeleteOptions,
+    ) -> Vec<Option<DiskError>> {
+        ecstore_disk::DiskAPI::delete_versions_recoverable(self, volume, versions, opts).await
+    }
+
+    async fn discover_delete_transactions(
+        &self,
+        volume: &str,
+        object: Option<&str>,
+        cursor: Option<DeleteRecoveryCursor>,
+        limit: u32,
+    ) -> DiskResult<DeleteRecoveryPage> {
+        ecstore_disk::DiskAPI::discover_delete_transactions(self, volume, object, cursor, limit).await
+    }
+    async fn inspect_delete_transaction(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: uuid::Uuid,
+    ) -> DiskResult<DeleteRecoveryInspection> {
+        ecstore_disk::DiskAPI::inspect_delete_transaction(self, volume, object, transaction).await
+    }
+    async fn settle_delete_transaction(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: uuid::Uuid,
+        expected_fingerprint: DeleteRecoveryDigest,
+        action: DeleteRecoveryAction,
+    ) -> DiskResult<DeleteRecoveryInspection> {
+        ecstore_disk::DiskAPI::settle_delete_transaction(self, volume, object, transaction, expected_fingerprint, action).await
+    }
+
     async fn settle_part_transaction(&self, volume: &str, path: &str, action: PartTransactionAction) -> DiskResult<()> {
         ecstore_disk::DiskAPI::settle_part_transaction(self, volume, path, action).await
     }

rustfs/src/storage_api.rs

@@ -180,6 +180,8 @@ pub(crate) mod server {
     }
 
     pub(crate) mod http {
+        #[cfg(test)]
+        pub(crate) use crate::storage::storage_api::ecstore_disk::{DiskAPI, DiskOption, new_disk};
         pub(crate) use crate::storage::storage_api::{
             ServerContextSlot, TONIC_RPC_PREFIX, normalize_tonic_rpc_audience, tonic_boot_epoch_challenge,
             tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason, verify_tonic_rpc_signature_with_bootstrap,