fix(ci): accept binary target IDs in workspace shards (#9075)

* perf(ecstore): fuse local PUT/DELETE commit I/O and eliminate redundant metadata disk reads (#9038)

- Fuse authoritative bucket metadata existence and parse reads via tokio::join!, add a LOCAL_META_MUTATION_GEN-validated local authority cache, and piggyback one-shot quota snapshot reads onto single-PUT Object Lock snapshot loads.
- Reuse prepared ObjectOptions in post-write lifecycle transition checks instead of re-reading bucket versioning and Object Lock config after commit.
- Defer temporary shard file open/mkdir/write into a single spawn_blocking call via LazyLocalFileWriter, pre-sync single-block shards before page-cache reclaim, and fuse rename_data preparation + commit into a single Fsync-lane dispatch on Linux.
- Enable rename_data old_current_source reuse by default across all-local sets, guard early-ack commit on quorum source stability, and skip redundant post-commit tmp_delete when all online local disks succeeded.
- Enable data-first batch_shard_pread when deferred parity handles are available and replace zero-filled allocation with Vec::with_capacity + read_to_end for offset-0 shard reads.
- Defer pre-delete get_object_info_for_delete to PrefixAccessDenied error handling, skip redundant single-pool unversioned delete pre-stats, and add a fused local delete preparation and 2-barrier settlement fast path.

Fixes #9038

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a

* fix(ecstore): preserve delete identity and durability policy

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): cover fused delete recovery stages

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): preserve inline versions after commit sync failure

* perf(ecstore,s3): accelerate PutObjectLegalHold, CopyObject, and large PUT fdatasync (#9038)

- Fuse LocalDisk::update_metadata (read xl.meta + FileMeta::load + update_object_version_with_opts + marshal_msg + staged tmp write + rename_all_with_precondition_and_parent_sync_std) into a single Fsync-lane blocking operation, and skip redundant get_bucket_info / get_object_info calls in put_object_legal_hold when object lock is enabled and replication is not configured (PutObjectLegalHold: 314.8 -> 988.3 ops/s, 3.14x faster, 0.95x of MinIO).
- Pass destination bucket as orig_volume through create_bitrot_writer_with_orig_volume into LocalDisk::create_file so single-shot non-inline part.1 writes under syncs_data_shards() durability run fdatasync before fadvise(POSIX_FADV_DONTNEED) and skip the duplicate sync_dir_files_std(src_data_path) in FusedCommitAttempt without fsyncing the temporary parent directory (warp PUT 1MiB: 658.3 vs MinIO 656.8 obj/s, 1.00x parity, p50 26.10ms vs MinIO 46.60ms).
- Add single_pool() fast paths to ECStore::handle_copy_object and ECStore::handle_new_multipart_upload, skip no-op destination prelookup get_object_info when put_like_write_creates_new_version is true, and backfill previous_current_size from OldCurrentSize in copy_object (CopyObject 1KB: 398.4 -> 1,086.1 ops/s, 2.73x faster, 1.01x faster than MinIO).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a

* fix(ecstore): provide non-Linux has_rename_data_test_failpoints stub for Windows CI

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a

* fix(ecstore): preserve presynced shard directory durability

Sync the shard directory before publishing presynced payloads and keep the file sync test probe available on non-Linux targets.

* fix(ecstore,s3): fix non-consuming crash_inject check, test hooks, and rubber-duck findings (#9038)

- Add non-consuming crash_inject::is_armed(point, key) so FusedCommitAttempt and LocalDisk::update_metadata do not consume one-shot crash arms before the fallback path runs (fixes crash_consistency::* and meta_write_crash_consistency::* tests).
- Check INLINE_PREPARATION_BEFORE_BACKUP and RENAME_DATA_AFTER_FIRST_PUBLICATION in has_rename_data_test_failpoints so inline preparation admission and observed-owner panic tests run on their expected path.
- Resolve staged_dir via get_object_path_for_io in test_rename_data_presynced_shard_keeps_directory_sync and avoid premature orig_volume access check during tmp staging in put_object (fixes put_object_failure_cleans_tmp_workspace_inline).
- Address rubber-duck review findings: fall back from FusedCommitAttempt on any existing xl.meta when sync_commit_metadata is true, fsync object dir after activating delete rollback dir before unlinking xl.meta, classify data-dir entries before removal in fused delete finalization, include self_copy_lock in copy_prelookup_required, and avoid .expect() on non-test mutexes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a

* fix(ecstore): defer shard sync until destination is known

* test(rustfs): cover legal hold replication fast path

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Chris An <anzhengchao@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a

.config/e2e-distributed-selection.txt

@@ -1,2 +1,2 @@
-sha256-linux=686dcc075e41a75dde3d37e5bd0393d016469f02f83aa3508f92bfc17f26a4d4
-sha256-darwin=686dcc075e41a75dde3d37e5bd0393d016469f02f83aa3508f92bfc17f26a4d4
+sha256-linux=eb33067d90ee66c0a97e8f3658e46c152e47f897d2fbfdc6cb5ad93cfa55414b
+sha256-darwin=eb33067d90ee66c0a97e8f3658e46c152e47f897d2fbfdc6cb5ad93cfa55414b

crates/e2e_test/Cargo.toml

@@ -25,7 +25,6 @@ workspace = true
 
 [features]
 default = []
-scanner-peer-mixed-version = []
 hotpath = [
     "hotpath/hotpath",
     "hotpath/tokio",

crates/e2e_test/README.md

@@ -64,25 +64,6 @@ Each build/run holds an exclusive `rustfs.e2e.lock` marker beside the binary; co
 
 The protocols suite has its own fixed-port and single-worker contract in [`src/protocols/README.md`](src/protocols/README.md). Use its command under [Troubleshooting](#troubleshooting).
 
-### Scanner mixed-version gate
-
-The Linux N/O scanner gate needs a separately qualified `33482bb` server, which supports scoped ACKs. Set `RUSTFS_SCANNER_COMPAT_OLD_BINARY` to that executable and `RUSTFS_SCANNER_COMPAT_OLD_SHA256` to its independently verified digest. The release binary used by ordinary upgrade CI is a different baseline. Missing or mismatched inputs fail the test.
-
-Enable the harness-only `scanner-peer-mixed-version` feature explicitly; keep the server wrapper features identical between build and run:
-
-```bash
-python3 scripts/e2e_binary.py build --features e2e-test-hooks
-python3 scripts/e2e_binary.py run --features e2e-test-hooks -- \
-  cargo nextest run --profile e2e-distributed -p e2e_test \
-  --features scanner-peer-mixed-version --ignore-default-filter \
-  -E 'test(=distributed::scanner_peer_bucket_test::scanner_peer_bucket_new_coordinator_certifies_old_peers_and_scopes_next_cycle)' \
-  --test-threads 1 --retries 0 --no-tests=fail
-```
-
-The separate `scanner_peer_bucket_new_coordinator_certifies_old_worker_peers_and_scopes_next_cycle` case uses the same feature and qualified old binary inputs. Select its exact name in the command above. It keeps the original N process enabled and starts the three official O replacements with `RUSTFS_SCANNER_ENABLED=false`: their autonomous scanner loops are disabled, while real S3 dirty tracking, authenticated snapshots, publication leases, scoped ACKs, and N-dispatched RemoteScanner workers remain active. It records each node's actual scanner status and retains the same identity, durability, root, ACK, data-conservation, and physical-walk requirements. This fixed-role case does not validate autonomous O scanning or all-enabled election behavior. The original all-enabled N/O test remains separate and has not passed acceptance; a fixed-role result cannot satisfy that gate.
-
-For this fixed-role case, the `e2e-test-hooks` server also records the N client's complete bounded worker inventory. Requests are observed before opening the normal stream and paired only after authenticated source/plan validation and terminal EOF. Closed manifests and final health receipts must bind both cycles to their actual candidate/root; the next cycle needs exactly four sets of hot-only execution and at least one positive-progress worker mapped to a measured official O process. A zero-progress Current response, missing/extra request, unhealthy recorder, or missing physical coverage fails the gate. The recorder does not add O hooks or change any protocol or timeout. Final health reads check immutable closed-cycle evidence, not current N liveness.
-
 ### `#[ignore]` semantics
 
 Ignored tests are excluded from the default `cargo nextest run` pass because

crates/e2e_test/src/distributed/mod.rs

@@ -39,8 +39,6 @@ mod s3_during_data_movement_test;
 mod scanner_heal_topology_fixture;
 mod scanner_heal_topology_test;
 mod scanner_heal_topology_tier_test;
-mod scanner_peer_bucket_fixture;
-mod scanner_peer_bucket_test;
 mod site_replication_test;
 mod table_catalog_test;
 mod upgrade_test;

crates/e2e_test/src/rpc_fault_proxy.rs

@@ -28,8 +28,7 @@ use hyper_util::{
 use prost::Message as _;
 use rustfs_protos::proto_gen::node_service::{
     BatchGenerallyLockRequest, DeleteRequest, GenerallyLockRequest, PreparePartTransactionResponse, ReadVersionRequest,
-    RecoverMultipartSourceRequest, RenameDataResponse, ScannerScopedDirtyUsageAckRequest, SettlePartTransactionRequest,
-    SettlePartTransactionResponse,
+    RecoverMultipartSourceRequest, RenameDataResponse, SettlePartTransactionRequest, SettlePartTransactionResponse,
 };
 use std::{
     collections::HashSet,
@@ -50,9 +49,6 @@ use tokio::{
 type ProxyBody = BoxBody<Bytes, io::Error>;
 type ProxyClient = Client<HttpConnector, ProxyBody>;
 
-mod scoped_ack;
-pub(crate) use scoped_ack::{ScopedAckFault, ScopedAckQueue, ScopedAckScope};
-
 #[derive(Clone, Copy, PartialEq, Eq)]
 enum Disposition {
     Hold,
@@ -125,8 +121,7 @@ impl RpcPause {
         self.state.blocked.load(Ordering::SeqCst)
     }
 
-    /// Selected RPCs with successful HTTP/gRPC response headers, excluding
-    /// discarded packets. This does not prove a successful gRPC trailer or ACK apply.
+    /// Selected RPCs with successful HTTP/gRPC headers, excluding discarded packets.
     pub(crate) fn forwarded(&self) -> usize {
         self.state.forwarded.load(Ordering::SeqCst)
     }
@@ -238,7 +233,6 @@ pub(crate) struct RpcFaultProxy {
     address: SocketAddr,
     slot: PauseSlot,
     lock_slot: PauseSlot,
-    scoped_ack: scoped_ack::QueueSlot,
     task: JoinHandle<()>,
     late_requests: LateRequests,
 }
@@ -251,8 +245,6 @@ impl RpcFaultProxy {
         let active = slot.clone();
         let lock_slot: PauseSlot = Arc::new(Mutex::new(None));
         let active_lock = lock_slot.clone();
-        let scoped_ack = scoped_ack::empty_slot();
-        let active_ack = scoped_ack.clone();
         let late_requests: LateRequests = Arc::new(Mutex::new(Some(Vec::new())));
         let late = late_requests.clone();
         let task = tokio::spawn(async move {
@@ -267,7 +259,6 @@ impl RpcFaultProxy {
                         let h2 = h2.clone();
                         let active = active.clone();
                         let active_lock = active_lock.clone();
-                        let active_ack = active_ack.clone();
                         let late = late.clone();
                         connections.spawn(async move {
                             let service = service_fn(move |request: Request<Incoming>| {
@@ -281,7 +272,6 @@ impl RpcFaultProxy {
                                     None
                                 };
                                 let pause = owner_pause.or_else(|| active.lock().expect("RPC pause slot").clone());
-                                let ack = active_ack.lock().expect("scoped ACK queue slot").clone();
                                 let retain = pause.as_ref().is_some_and(|pause| {
                                     pause.retain_after_disconnect
                                         && request.uri().path().strip_prefix("/node_service.NodeService/")
@@ -290,11 +280,11 @@ impl RpcFaultProxy {
                                 let late = late.clone();
                                 async move {
                                     if !retain {
-                                        return forward_request(request, client, target, pause, ack).await;
+                                        return forward_request(request, client, target, pause).await;
                                     }
                                     // This opt-in fault owns the packet after client cancellation.
                                     // Proxy teardown still aborts every retained request.
-                                    let task = tokio::spawn(forward_request(request, client, target, pause, ack));
+                                    let task = tokio::spawn(forward_request(request, client, target, pause));
                                     {
                                         let mut requests = late.lock().expect("late RPC ownership");
                                         if let Some(requests) = requests.as_mut() {
@@ -319,7 +309,6 @@ impl RpcFaultProxy {
             address,
             slot,
             lock_slot,
-            scoped_ack,
             task,
             late_requests,
         })
@@ -329,11 +318,6 @@ impl RpcFaultProxy {
         self.address
     }
 
-    /// Retain only genuine scoped ACKs for one receiver/session and bucket set.
-    pub(crate) fn scoped_ack_queue(&self, scope: ScopedAckScope, capacity: usize) -> io::Result<ScopedAckQueue> {
-        ScopedAckQueue::arm(self.scoped_ack.clone(), scope, capacity)
-    }
-
     /// Pause at most limit requests to the named unary methods; other streams
     /// on the same HTTP/2 connection continue forwarding.
     pub(crate) fn pause(&self, methods: &[&'static str], limit: usize, scope: Option<&str>) -> io::Result<RpcPause> {
@@ -485,9 +469,6 @@ fn unary_payload(body: &[u8]) -> io::Result<&[u8]> {
 }
 
 fn scope_matches(method: &str, body: &[u8], scope: &[u8]) -> io::Result<bool> {
-    if method == "ScannerScopedDirtyUsageAck" {
-        return scoped_ack_matches(body, Some(scope));
-    }
     if method == "RecoverMultipartSource" {
         let request = RecoverMultipartSourceRequest::decode(unary_payload(body)?).map_err(io::Error::other)?;
         if request.protocol_version != u32::from(MULTIPART_SOURCE_PROTOCOL_VERSION) || request.path.as_bytes() != scope {
@@ -544,9 +525,7 @@ fn scoped_lock_owners(method: &str, body: &[u8], scope: &[u8]) -> io::Result<Vec
     let args = match method {
         "Lock" => vec![GenerallyLockRequest::decode(payload).map_err(io::Error::other)?.args],
         "LockBatch" => BatchGenerallyLockRequest::decode(payload).map_err(io::Error::other)?.args,
-        _ => {
-            return Err(io::Error::new(io::ErrorKind::InvalidInput, "lock observer requires a lock method"));
-        }
+        _ => return Err(io::Error::new(io::ErrorKind::InvalidInput, "lock observer requires a lock method")),
     };
     args.into_iter()
         .filter(|args| args.as_bytes().windows(scope.len()).any(|window| window == scope))
@@ -561,39 +540,18 @@ fn scoped_lock_owners(method: &str, body: &[u8], scope: &[u8]) -> io::Result<Vec
         .collect()
 }
 
-fn scoped_ack_matches(body: &[u8], scope: Option<&[u8]>) -> io::Result<bool> {
-    let request = ScannerScopedDirtyUsageAckRequest::decode(unary_payload(body)?).map_err(io::Error::other)?;
-    Ok(!request.probe_only && scope.is_none_or(|scope| request.entries.iter().any(|entry| entry.bucket.as_bytes() == scope)))
-}
-
 async fn forward_request(
     request: Request<Incoming>,
     client: ProxyClient,
     target: SocketAddr,
     pause: Option<Arc<PauseState>>,
-    ack: Option<Arc<scoped_ack::QueueState>>,
 ) -> io::Result<hyper::Response<ProxyBody>> {
-    let request = request.map(|body| body.map_err(io::Error::other).boxed());
-    let request = if let Some(ack) = ack
-        && request.uri().path() == "/node_service.ScannerControlService/ScannerScopedDirtyUsageAck"
-    {
-        match scoped_ack::intercept(request, client.clone(), target, ack).await? {
-            scoped_ack::Interception::Pass(request) => request,
-            scoped_ack::Interception::Handled(response) => return Ok(response),
-        }
-    } else {
-        request
-    };
-    let (parts, body) = request.into_parts();
-    let mut body = body;
+    let (mut parts, body) = request.into_parts();
+    let mut body = body.map_err(io::Error::other).boxed();
     let mut observed = None;
     let mut inspection = None;
     if let Some(pause) = pause {
-        let method = parts
-            .uri
-            .path()
-            .strip_prefix("/node_service.NodeService/")
-            .or_else(|| parts.uri.path().strip_prefix("/node_service.ScannerControlService/"));
+        let method = parts.uri.path().strip_prefix("/node_service.NodeService/");
         if let Some(method) = method.filter(|name| pause.methods.contains(name)) {
             let mut route = None;
             let mut lock_owners = Vec::new();
@@ -641,16 +599,6 @@ async fn forward_request(
                     .with_trailers(async move { trailers.map(Ok::<_, io::Error>) })
                     .boxed();
                 matches
-            } else if method == "ScannerScopedDirtyUsageAck" {
-                let collected = Limited::new(body, 1024 * 1024).collect().await.map_err(io::Error::other)?;
-                let trailers = collected.trailers().cloned();
-                let bytes = collected.to_bytes();
-                let matches = scoped_ack_matches(&bytes, None)?;
-                body = Full::new(bytes)
-                    .map_err(|never| match never {})
-                    .with_trailers(async move { trailers.map(Ok::<_, io::Error>) })
-                    .boxed();
-                matches
             } else {
                 true
             };
@@ -694,7 +642,24 @@ async fn forward_request(
             }
         }
     }
-    let response = forward_to_backend(Request::from_parts(parts, body), client, target).await?;
+    if !parts.headers.contains_key(http::header::HOST)
+        && let Some(authority) = parts.uri.authority()
+    {
+        parts.headers.insert(
+            http::header::HOST,
+            http::HeaderValue::from_str(authority.as_str()).map_err(io::Error::other)?,
+        );
+    }
+    parts.uri = http::Uri::builder()
+        .scheme("http")
+        .authority(target.to_string())
+        .path_and_query(parts.uri.path_and_query().map_or("/", |value| value.as_str()))
+        .build()
+        .map_err(io::Error::other)?;
+    let response = client
+        .request(Request::from_parts(parts, body))
+        .await
+        .map_err(io::Error::other)?;
     if let Some((pause, disk)) = inspection
         && response.status().is_success()
         && response.headers().get("grpc-status").is_some_and(|status| status == "12")
@@ -747,32 +712,6 @@ async fn forward_request(
     Ok(response.map(|body| body.map_err(io::Error::other).boxed()))
 }
 
-async fn forward_to_backend(
-    request: Request<ProxyBody>,
-    client: ProxyClient,
-    target: SocketAddr,
-) -> io::Result<hyper::Response<Incoming>> {
-    let (mut parts, body) = request.into_parts();
-    if !parts.headers.contains_key(http::header::HOST)
-        && let Some(authority) = parts.uri.authority()
-    {
-        parts.headers.insert(
-            http::header::HOST,
-            http::HeaderValue::from_str(authority.as_str()).map_err(io::Error::other)?,
-        );
-    }
-    parts.uri = http::Uri::builder()
-        .scheme("http")
-        .authority(target.to_string())
-        .path_and_query(parts.uri.path_and_query().map_or("/", |value| value.as_str()))
-        .build()
-        .map_err(io::Error::other)?;
-    client
-        .request(Request::from_parts(parts, body))
-        .await
-        .map_err(io::Error::other)
-}
-
 impl Drop for RpcFaultProxy {
     fn drop(&mut self) {
         if let Some(requests) = self.late_requests.lock().expect("late RPC ownership").take() {
@@ -792,7 +731,6 @@ mod tests {
 
     struct Backend {
         received: Arc<AtomicUsize>,
-        auth_headers: Arc<Mutex<Vec<(String, String)>>>,
         address: SocketAddr,
         task: JoinHandle<()>,
     }
@@ -816,8 +754,6 @@ mod tests {
         let address = listener.local_addr()?;
         let received = Arc::new(AtomicUsize::new(0));
         let received_task = received.clone();
-        let auth_headers = Arc::new(Mutex::new(Vec::new()));
-        let backend_auth = auth_headers.clone();
         let task = tokio::spawn(async move {
             let mut children = JoinSet::new();
             loop {
@@ -825,12 +761,10 @@ mod tests {
                     accepted = listener.accept() => {
                         let Ok((socket, _)) = accepted else { break };
                         let received = received_task.clone();
-                        let backend_auth = backend_auth.clone();
                         let reply = reply.clone();
                         children.spawn(async move {
                             let service = service_fn(move |request: Request<Incoming>| {
                                 let received = received.clone();
-                                let backend_auth = backend_auth.clone();
                                 let reply = reply.clone();
                                 async move {
                                 received.fetch_add(1, Ordering::SeqCst);
@@ -839,10 +773,6 @@ mod tests {
                                         http::HeaderValue::from_str(authority.as_str()).expect("valid HTTP authority")
                                     })
                                 });
-                                if let (Some(nonce), Some(signature)) = (request.headers().get("x-rustfs-rpc-nonce"), request.headers().get("x-rustfs-rpc-signature-v2")) {
-                                    backend_auth.lock().expect("backend auth header observations").push((
-                                        nonce.to_str().expect("test nonce").to_string(), signature.to_str().expect("test signature").to_string()));
-                                }
                                 let body = request.into_body().collect().await?.to_bytes();
                                 let body = reply.unwrap_or(body);
                                 let mut trailers = http::HeaderMap::new();
@@ -865,12 +795,7 @@ mod tests {
                 }
             }
         });
-        Ok(Backend {
-            received,
-            auth_headers,
-            address,
-            task,
-        })
+        Ok(Backend { received, address, task })
     }
 
     async fn call(client: &ProxyClient, address: SocketAddr, method: &str) -> io::Result<hyper::Response<Incoming>> {
@@ -884,14 +809,9 @@ mod tests {
         bytes: Bytes,
     ) -> io::Result<hyper::Response<Incoming>> {
         let body: ProxyBody = Full::new(bytes).map_err(|never| match never {}).boxed();
-        let service = if method == "ScannerScopedDirtyUsageAck" {
-            "ScannerControlService"
-        } else {
-            "NodeService"
-        };
         let request = Request::builder()
             .method("POST")
-            .uri(format!("http://{address}/node_service.{service}/{method}"))
+            .uri(format!("http://{address}/node_service.NodeService/{method}"))
             .header(http::header::TE, "trailers")
             .body(body)
             .map_err(io::Error::other)?;
@@ -906,353 +826,6 @@ mod tests {
         frame.into()
     }
 
-    const ACK_OWNER: &str = "11111111-1111-1111-1111-111111111111";
-    const ACK_INSTANCE: &str = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
-
-    fn ack_scope() -> ScopedAckScope {
-        ScopedAckScope {
-            nonce: uuid::Uuid::new_v4().to_string(),
-            owner_id: ACK_OWNER.to_string(),
-            instance_id: ACK_INSTANCE.to_string(),
-            buckets: std::collections::BTreeSet::from(["cold-bucket".to_string()]),
-        }
-    }
-
-    fn ack_request(bucket: &str, generation: u64) -> ScannerScopedDirtyUsageAckRequest {
-        ScannerScopedDirtyUsageAckRequest {
-            challenge: vec![7; 16].into(),
-            protocol_version: 1,
-            owner_id: ACK_OWNER.to_string(),
-            instance_id: ACK_INSTANCE.to_string(),
-            scope: 1,
-            probe_only: false,
-            entries: vec![rustfs_protos::proto_gen::node_service::ScannerScopedDirtyUsageEntry {
-                bucket: bucket.to_string(),
-                bucket_incarnation: vec![4; 16].into(),
-                generation,
-            }],
-        }
-    }
-
-    fn ack_frame(request: &ScannerScopedDirtyUsageAckRequest) -> Bytes {
-        let payload = request.encode_to_vec();
-        let mut frame = vec![0];
-        frame.extend_from_slice(&u32::try_from(payload.len()).expect("bounded test frame").to_be_bytes());
-        frame.extend_from_slice(&payload);
-        Bytes::from(frame)
-    }
-
-    fn ack_hash(bytes: &[u8]) -> String {
-        use sha2::Digest as _;
-        hex_simd::encode_to_string(&sha2::Sha256::digest(bytes)[..], hex_simd::AsciiCase::Lower)
-    }
-
-    // Transport-only fixtures: the distributed tests supply production signatures.
-    async fn call_ack_frame(
-        client: &ProxyClient,
-        address: SocketAddr,
-        frame: Bytes,
-        nonce: &str,
-    ) -> io::Result<hyper::Response<Incoming>> {
-        let request = Request::builder()
-            .method("POST")
-            .uri(format!("http://{address}/node_service.ScannerControlService/ScannerScopedDirtyUsageAck"))
-            .header(http::header::TE, "trailers")
-            .header("x-rustfs-rpc-nonce", nonce)
-            .header("x-rustfs-rpc-signature-v2", format!("opaque-signature-{nonce}"))
-            .body(Full::new(frame).map_err(|never| match never {}).boxed())
-            .map_err(io::Error::other)?;
-        client.request(request).await.map_err(io::Error::other)
-    }
-
-    #[tokio::test]
-    async fn scoped_ack_queue_loss_preserves_backend_boundary() {
-        tokio::time::timeout(Duration::from_secs(10), async {
-            for fault in [ScopedAckFault::RequestLoss, ScopedAckFault::ResponseLoss] {
-                let backend = backend().await.expect("backend");
-                let proxy = RpcFaultProxy::start_on("127.0.0.1:0".parse().expect("loopback"), backend.address)
-                    .await
-                    .expect("proxy");
-                let queue = proxy.scoped_ack_queue(ack_scope(), 1).expect("ACK queue");
-                let client: ProxyClient = Client::builder(TokioExecutor::new()).http2_only(true).build_http();
-                let frame = ack_frame(&ack_request("cold-bucket", 7));
-                let address = proxy.local_addr();
-                let body = frame.clone();
-                let caller = tokio::spawn(async move { call_ack_frame(&client, address, body, "loss").await });
-                queue.wait_for_packets(1).await.expect("captured ACK");
-                assert_eq!(backend.received.load(Ordering::SeqCst), 0, "retention must precede backend dispatch");
-                queue.settle(0, fault).expect("settle actual retained packet");
-                assert!(
-                    caller.await.expect("caller task").is_err(),
-                    "the caller must observe the injected transport failure"
-                );
-                let receipt = queue.wait_for_completion(0).await.expect("complete fault receipt");
-                let response_loss = fault == ScopedAckFault::ResponseLoss;
-                assert_eq!(receipt["forwarded"], response_loss);
-                assert_eq!(receipt["backend_complete"], response_loss);
-                assert_eq!(backend.received.load(Ordering::SeqCst), usize::from(response_loss));
-                if response_loss {
-                    assert_eq!(receipt["outcome"], "response_lost");
-                    assert_eq!(receipt["response"]["grpc_status"], "0", "status exists only in real backend trailers");
-                    assert_eq!(receipt["response"]["bytes"], frame.len());
-                    assert_eq!(receipt["response"]["sha256"], ack_hash(&frame));
-                    assert!(
-                        receipt["response"]["trailers_sha256"]
-                            .as_str()
-                            .is_some_and(|value| value.len() == 64)
-                    );
-                } else {
-                    assert_eq!(receipt["outcome"], "request_lost");
-                }
-            }
-        })
-        .await
-        .expect("bounded ACK loss checks");
-    }
-
-    #[tokio::test]
-    async fn scoped_ack_queue_keeps_cancelled_packets_and_releases_out_of_order() {
-        tokio::time::timeout(Duration::from_secs(10), async {
-            let backend = backend().await.expect("backend");
-            let proxy = RpcFaultProxy::start_on("127.0.0.1:0".parse().expect("loopback"), backend.address)
-                .await
-                .expect("proxy");
-            let queue = proxy.scoped_ack_queue(ack_scope(), 2).expect("two-packet ACK queue");
-            let client: ProxyClient = Client::builder(TokioExecutor::new()).http2_only(true).build_http();
-            let first = ack_frame(&ack_request("cold-bucket", 7));
-            let second = ack_frame(&ack_request("cold-bucket", 9));
-            let address = proxy.local_addr();
-            let old_client = client.clone();
-            let old_body = first.clone();
-            let old_caller = tokio::spawn(async move { call_ack_frame(&old_client, address, old_body, "first").await });
-            queue.wait_for_packets(1).await.expect("first retained packet");
-            old_caller.abort();
-            assert!(old_caller.await.expect_err("cancel original caller").is_cancelled());
-            let new_body = second.clone();
-            let new_caller = tokio::spawn(async move { call_ack_frame(&client, address, new_body, "second").await });
-            queue.wait_for_packets(2).await.expect("second retained packet");
-            assert_eq!(backend.received.load(Ordering::SeqCst), 0);
-            queue.settle(1, ScopedAckFault::Forward).expect("release newer packet first");
-            let response = new_caller
-                .await
-                .expect("new caller task")
-                .expect("new response")
-                .into_body()
-                .collect()
-                .await
-                .expect("complete body");
-            assert_eq!(response.trailers().expect("response trailers")["x-proof"], "unchanged-proof");
-            assert_eq!(response.to_bytes(), second);
-            queue.wait_for_completion(1).await.expect("new packet complete");
-            queue
-                .settle(0, ScopedAckFault::Forward)
-                .expect("release old packet after its caller disappeared");
-            let old = queue.wait_for_completion(0).await.expect("detached old packet complete");
-            assert_eq!(old["backend_complete"], true);
-            assert_eq!(old["request_sha256"], ack_hash(&first));
-            assert_eq!(old["response"]["sha256"], ack_hash(&first));
-            assert_eq!(backend.received.load(Ordering::SeqCst), 2);
-            assert_eq!(
-                *backend.auth_headers.lock().expect("backend header observations"),
-                vec![
-                    ("second".to_string(), "opaque-signature-second".to_string()),
-                    ("first".to_string(), "opaque-signature-first".to_string())
-                ]
-            );
-            assert!(
-                queue.settle(0, ScopedAckFault::Forward).is_err(),
-                "one original packet may only be forwarded once"
-            );
-        })
-        .await
-        .expect("bounded detached retention check");
-    }
-
-    #[tokio::test]
-    async fn scoped_ack_queue_bounds_capacity_and_excludes_other_scopes() {
-        tokio::time::timeout(Duration::from_secs(10), async {
-            let backend = backend().await.expect("backend");
-            let proxy = RpcFaultProxy::start_on("127.0.0.1:0".parse().expect("loopback"), backend.address)
-                .await
-                .expect("proxy");
-            assert!(proxy.scoped_ack_queue(ack_scope(), 0).is_err());
-            assert!(proxy.scoped_ack_queue(ack_scope(), 5).is_err());
-            let mut invalid = ack_scope();
-            invalid.nonce.clear();
-            assert!(proxy.scoped_ack_queue(invalid, 1).is_err());
-            let mut scope = ack_scope();
-            scope.buckets.insert("cold-bucket-z".to_string());
-            let queue = proxy.scoped_ack_queue(scope, 1).expect("bounded ACK queue");
-            let client: ProxyClient = Client::builder(TokioExecutor::new()).http2_only(true).build_http();
-            let mut probe = ack_request("cold-bucket", 7);
-            probe.probe_only = true;
-            let mut other_peer = ack_request("cold-bucket", 7);
-            other_peer.owner_id = uuid::Uuid::new_v4().to_string();
-            let mut other_instance = ack_request("cold-bucket", 7);
-            other_instance.instance_id = "b".repeat(32);
-            let mut invalid_challenge = ack_request("cold-bucket", 7);
-            invalid_challenge.challenge = vec![7; 15].into();
-            let mut noncanonical_entries = ack_request("cold-bucket-z", 7);
-            noncanonical_entries.entries.extend(ack_request("cold-bucket", 7).entries);
-            let unselected = [
-                probe,
-                other_peer,
-                other_instance,
-                ack_request("cold-bucket-other", 7),
-                ack_request("cold-bucket", u64::MAX),
-                invalid_challenge,
-                noncanonical_entries,
-            ];
-            let unselected_count = unselected.len();
-            for (index, request) in unselected.into_iter().enumerate() {
-                let body = ack_frame(&request);
-                let response = call_ack_frame(&client, proxy.local_addr(), body.clone(), "unselected")
-                    .await
-                    .expect("unselected ACK progresses");
-                assert_eq!(response.into_body().collect().await.expect("unselected body").to_bytes(), body);
-                assert!(queue.receipts().is_empty(), "unselected or protocol-invalid ACK must not consume a slot");
-                assert_eq!(backend.received.load(Ordering::SeqCst), index + 1);
-            }
-            assert!(queue.receipts().is_empty());
-            let address = proxy.local_addr();
-            let held_client = client.clone();
-            let valid_frame = ack_frame(&ack_request("cold-bucket", 7));
-            let held_body = valid_frame.clone();
-            let held = tokio::spawn(async move { call_ack_frame(&held_client, address, held_body, "held").await });
-            queue.wait_for_packets(1).await.expect("one matching packet");
-            assert!(!held.is_finished(), "the subsequent valid ACK must be retained");
-            assert_eq!(queue.receipts()[0]["request_sha256"], ack_hash(&valid_frame));
-            assert_eq!(backend.received.load(Ordering::SeqCst), unselected_count);
-            let extra = call_ack_frame(&client, address, ack_frame(&ack_request("cold-bucket", 8)), "beyond-capacity")
-                .await
-                .expect("capacity overflow passes without retention");
-            extra.into_body().collect().await.expect("overflow response");
-            assert_eq!(queue.receipts().len(), 1);
-            assert_eq!(backend.received.load(Ordering::SeqCst), unselected_count + 1);
-            drop(queue);
-            assert!(
-                held.await.expect("held caller").is_err(),
-                "dropping the session must cancel its retained packet"
-            );
-            assert_eq!(
-                backend.received.load(Ordering::SeqCst),
-                unselected_count + 1,
-                "session drop must never forward the held packet"
-            );
-        })
-        .await
-        .expect("bounded scope and lifecycle checks");
-    }
-
-    #[tokio::test]
-    async fn scoped_ack_queue_expiry_never_forwards_retained_packet() {
-        let (backend, _proxy, queue, caller) = tokio::time::timeout(Duration::from_secs(10), async {
-            let backend = backend().await.expect("backend");
-            let proxy = RpcFaultProxy::start_on("127.0.0.1:0".parse().expect("loopback"), backend.address)
-                .await
-                .expect("proxy");
-            let queue = proxy.scoped_ack_queue(ack_scope(), 1).expect("ACK queue");
-            let client: ProxyClient = Client::builder(TokioExecutor::new()).http2_only(true).build_http();
-            let address = proxy.local_addr();
-            let caller = tokio::spawn(async move {
-                call_ack_frame(&client, address, ack_frame(&ack_request("cold-bucket", 7)), "expires").await
-            });
-            queue.wait_for_packets(1).await.expect("retained packet");
-            (backend, proxy, queue, caller)
-        })
-        .await
-        .expect("bounded queue setup");
-        tokio::time::pause();
-        tokio::time::advance(Duration::from_secs(120)).await;
-        tokio::time::resume();
-        assert!(
-            tokio::time::timeout(Duration::from_secs(10), caller)
-                .await
-                .expect("expiry completes caller")
-                .expect("caller task")
-                .is_err()
-        );
-        let receipt = queue.receipts().into_iter().next().expect("expired receipt");
-        assert_eq!(receipt["completed"], true);
-        assert_eq!(receipt["forwarded"], false);
-        assert_eq!(receipt["backend_complete"], false);
-        assert_eq!(backend.received.load(Ordering::SeqCst), 0);
-        assert_eq!(
-            queue
-                .settle(0, ScopedAckFault::Forward)
-                .expect_err("expired packet cannot be released")
-                .kind(),
-            io::ErrorKind::TimedOut
-        );
-    }
-
-    #[tokio::test]
-    async fn scanner_ack_fault_matches_exact_bucket_and_excludes_probes() {
-        use rustfs_protos::proto_gen::node_service::ScannerScopedDirtyUsageEntry;
-
-        tokio::time::timeout(Duration::from_secs(10), async {
-            let backend = backend().await.expect("backend listener");
-            let proxy = RpcFaultProxy::start_on("127.0.0.1:0".parse().expect("loopback"), backend.address)
-                .await
-                .expect("proxy listener");
-            let client: ProxyClient = Client::builder(TokioExecutor::new()).http2_only(true).build_http();
-            let pause = proxy
-                .pause(&["ScannerScopedDirtyUsageAck"], 1, Some("cold-bucket"))
-                .expect("scoped ACK fault");
-            let frame = |bucket: &str, probe_only: bool| {
-                let payload = ScannerScopedDirtyUsageAckRequest {
-                    challenge: vec![3; 16].into(),
-                    protocol_version: 1,
-                    owner_id: "11111111-1111-1111-1111-111111111111".to_string(),
-                    instance_id: "a".repeat(32),
-                    scope: 1,
-                    probe_only,
-                    entries: vec![ScannerScopedDirtyUsageEntry {
-                        bucket: bucket.to_string(),
-                        bucket_incarnation: vec![4; 16].into(),
-                        generation: 17,
-                    }],
-                }
-                .encode_to_vec();
-                let mut frame = vec![0];
-                frame.extend_from_slice(&u32::try_from(payload.len()).expect("small ACK frame").to_be_bytes());
-                frame.extend_from_slice(&payload);
-                Bytes::from(frame)
-            };
-            for bytes in [frame("cold-bucket", true), frame("cold-bucket-other", false)] {
-                let response = call_body(&client, proxy.local_addr(), "ScannerScopedDirtyUsageAck", bytes.clone())
-                    .await
-                    .expect("probe and other bucket must progress");
-                assert_eq!(response.into_body().collect().await.expect("unselected response").to_bytes(), bytes);
-            }
-            assert_eq!(pause.hits(), 0);
-            let signed_bytes = frame("cold-bucket", false);
-            let task_bytes = signed_bytes.clone();
-            let address = proxy.local_addr();
-            let blocked =
-                tokio::spawn(async move { call_body(&client, address, "ScannerScopedDirtyUsageAck", task_bytes).await });
-            pause
-                .wait_for_hits(1)
-                .await
-                .expect("real ScannerControlService method must reach the fault");
-            assert!(!blocked.is_finished());
-            assert_eq!(backend.received.load(Ordering::SeqCst), 2, "held ACK must not apply at the backend");
-            pause.release();
-            let collected = blocked
-                .await
-                .expect("ACK task")
-                .expect("released ACK")
-                .into_body()
-                .collect()
-                .await
-                .expect("ACK body and trailers");
-            assert_eq!(collected.trailers().expect("ACK trailers")["x-proof"], "unchanged-proof");
-            assert_eq!(collected.to_bytes(), signed_bytes, "proxy must preserve the complete signed ACK frame");
-        })
-        .await
-        .expect("scanner ACK fault deadline");
-    }
-
     fn native_abort_request(path: &str) -> RecoverMultipartSourceRequest {
         use crate::storage_api::multipart_source_rpc::{
             MultipartAbortCertificate, MultipartAbortProof, MultipartAbortProofKind, MultipartAbortRound, MultipartSourcePhase,

crates/e2e_test/src/rpc_fault_proxy/scoped_ack.rs

@@ -1,404 +0,0 @@
-// Copyright 2026 RustFS Team
-// SPDX-License-Identifier: Apache-2.0
-
-//! Bounded retention of original scoped ACK frames, independent of caller cancellation.
-
-use super::{ProxyBody, ProxyClient, forward_to_backend, unary_payload};
-use bytes::Bytes;
-use http_body_util::{BodyExt as _, Full, Limited};
-use hyper::{Request, Response};
-use prost::Message as _;
-use rustfs_protos::proto_gen::node_service::{ScannerScopedDirtyUsageAckRequest, ScannerScopedDirtyUsageAckResponse};
-use rustfs_protos::scoped_dirty_usage::validate_scoped_dirty_usage_request;
-use serde_json::{Value, json};
-use sha2::{Digest as _, Sha256};
-use std::{
-    collections::BTreeSet,
-    io,
-    net::SocketAddr,
-    sync::{
-        Arc, Mutex,
-        atomic::{AtomicBool, Ordering},
-    },
-    time::Duration,
-};
-use tokio::{
-    sync::{Notify, oneshot, watch},
-    task::AbortHandle,
-    time::Instant,
-};
-
-const MAX_RETAINED: usize = 4;
-const MAX_BYTES: usize = 1024 * 1024;
-const RETENTION: Duration = Duration::from_secs(120);
-
-pub(crate) struct ScopedAckScope {
-    pub nonce: String,
-    pub owner_id: String,
-    pub instance_id: String,
-    pub buckets: BTreeSet<String>,
-}
-
-impl ScopedAckScope {
-    fn validate(&self) -> io::Result<()> {
-        let valid_uuid = |value: &str| uuid::Uuid::parse_str(value).is_ok_and(|value| !value.is_nil());
-        if !valid_uuid(&self.nonce)
-            || !valid_uuid(&self.owner_id)
-            || self.instance_id.is_empty()
-            || self.instance_id.len() > 128
-            || self.buckets.is_empty()
-            || self.buckets.len() > 16
-            || self.buckets.iter().any(|bucket| bucket.is_empty() || bucket.len() > 63)
-        {
-            return Err(io::Error::new(
-                io::ErrorKind::InvalidInput,
-                "scoped ACK queue needs a bounded session, receiver and bucket scope",
-            ));
-        }
-        Ok(())
-    }
-
-    fn matches(&self, request: &ScannerScopedDirtyUsageAckRequest) -> bool {
-        !request.probe_only
-            && validate_scoped_dirty_usage_request(request).is_ok()
-            && request.owner_id == self.owner_id
-            && request.instance_id == self.instance_id
-            && request.entries.iter().all(|entry| self.buckets.contains(&entry.bucket))
-    }
-}
-
-#[derive(Clone, Copy, Debug, PartialEq, Eq)]
-pub(crate) enum ScopedAckFault {
-    Forward,
-    RequestLoss,
-    ResponseLoss,
-}
-
-struct Packet {
-    action: watch::Sender<Option<ScopedAckFault>>,
-    receipt: Mutex<Value>,
-    task: Mutex<Option<AbortHandle>>,
-}
-
-pub(super) struct QueueState {
-    scope: ScopedAckScope,
-    capacity: usize,
-    started: Instant,
-    deadline: Instant,
-    closed: AtomicBool,
-    // The packet inventory lock is released before any packet lock or await.
-    packets: Mutex<Vec<Arc<Packet>>>,
-    changed: Notify,
-}
-
-pub(super) type QueueSlot = Arc<Mutex<Option<Arc<QueueState>>>>;
-
-pub(super) fn empty_slot() -> QueueSlot {
-    Arc::new(Mutex::new(None))
-}
-
-pub(crate) struct ScopedAckQueue {
-    state: Arc<QueueState>,
-    slot: QueueSlot,
-}
-
-impl ScopedAckQueue {
-    pub(super) fn arm(slot: QueueSlot, scope: ScopedAckScope, capacity: usize) -> io::Result<Self> {
-        scope.validate()?;
-        if !(1..=MAX_RETAINED).contains(&capacity) {
-            return Err(io::Error::new(io::ErrorKind::InvalidInput, "scoped ACK capacity must be 1..=4"));
-        }
-        let mut active = slot.lock().expect("scoped ACK queue slot");
-        if active.is_some() {
-            return Err(io::Error::new(io::ErrorKind::AlreadyExists, "scoped ACK queue is already armed"));
-        }
-        let started = Instant::now();
-        let state = Arc::new(QueueState {
-            scope,
-            capacity,
-            started,
-            deadline: started + RETENTION,
-            closed: AtomicBool::new(false),
-            packets: Mutex::new(Vec::new()),
-            changed: Notify::new(),
-        });
-        *active = Some(state.clone());
-        drop(active);
-        Ok(Self { state, slot })
-    }
-
-    pub(crate) fn receipts(&self) -> Vec<Value> {
-        let packets = self.state.packets.lock().expect("scoped ACK packets").clone();
-        packets
-            .iter()
-            .map(|packet| packet.receipt.lock().expect("scoped ACK receipt").clone())
-            .collect()
-    }
-
-    pub(crate) async fn wait_for_packets(&self, count: usize) -> io::Result<()> {
-        self.wait_until(|| self.receipts().len() >= count).await
-    }
-
-    pub(crate) async fn wait_for_completion(&self, id: usize) -> io::Result<Value> {
-        self.wait_until(|| {
-            self.receipts()
-                .get(id)
-                .is_some_and(|receipt| receipt["completed"].as_bool() == Some(true))
-        })
-        .await?;
-        self.receipts()
-            .get(id)
-            .cloned()
-            .ok_or_else(|| io::Error::other("scoped ACK receipt disappeared"))
-    }
-
-    async fn wait_until(&self, ready: impl Fn() -> bool) -> io::Result<()> {
-        tokio::time::timeout_at(self.state.deadline, async {
-            loop {
-                let changed = self.state.changed.notified();
-                if ready() {
-                    return;
-                }
-                changed.await;
-            }
-        })
-        .await
-        .map_err(|_| io::Error::new(io::ErrorKind::TimedOut, "scoped ACK queue lifetime elapsed"))
-    }
-
-    pub(crate) fn settle(&self, id: usize, fault: ScopedAckFault) -> io::Result<()> {
-        if Instant::now() >= self.state.deadline {
-            return Err(io::Error::new(io::ErrorKind::TimedOut, "scoped ACK retention expired"));
-        }
-        let packet = self
-            .state
-            .packets
-            .lock()
-            .expect("scoped ACK packets")
-            .get(id)
-            .cloned()
-            .ok_or_else(|| io::Error::new(io::ErrorKind::NotFound, "scoped ACK packet is not retained"))?;
-        if !packet.action.send_if_modified(|current| {
-            if current.is_none() {
-                *current = Some(fault);
-                true
-            } else {
-                false
-            }
-        }) {
-            return Err(io::Error::new(io::ErrorKind::AlreadyExists, "scoped ACK packet was already settled"));
-        }
-        Ok(())
-    }
-}
-
-impl Drop for ScopedAckQueue {
-    fn drop(&mut self) {
-        self.state.closed.store(true, Ordering::SeqCst);
-        let mut slot = self.slot.lock().expect("scoped ACK queue slot");
-        if slot.as_ref().is_some_and(|current| Arc::ptr_eq(current, &self.state)) {
-            *slot = None;
-        }
-        drop(slot);
-        let packets = self.state.packets.lock().expect("scoped ACK packets").clone();
-        for packet in packets {
-            if let Some(task) = packet.task.lock().expect("scoped ACK task").take() {
-                task.abort();
-            }
-        }
-    }
-}
-
-pub(super) enum Interception {
-    Pass(Request<ProxyBody>),
-    Handled(Response<ProxyBody>),
-}
-
-fn restored_body(bytes: Bytes, trailers: Option<http::HeaderMap>) -> ProxyBody {
-    Full::new(bytes)
-        .map_err(|never| match never {})
-        .with_trailers(async move { trailers.map(Ok::<_, io::Error>) })
-        .boxed()
-}
-
-fn hash(bytes: &[u8]) -> String {
-    hex(Sha256::digest(bytes).as_ref())
-}
-
-fn hex(bytes: &[u8]) -> String {
-    hex_simd::encode_to_string(bytes, hex_simd::AsciiCase::Lower)
-}
-
-fn trailer_receipt(trailers: Option<&http::HeaderMap>) -> Value {
-    trailers.map_or(Value::Null, |headers| {
-        json!(
-            headers
-                .iter()
-                .map(|(name, value)| { json!({"name": name.as_str(), "value_hex": hex(value.as_bytes())}) })
-                .collect::<Vec<_>>()
-        )
-    })
-}
-
-fn header_hash(headers: &http::HeaderMap) -> String {
-    let mut hasher = Sha256::new();
-    for (name, value) in headers {
-        hasher.update(name.as_str().as_bytes());
-        hasher.update([0]);
-        hasher.update(value.as_bytes());
-        hasher.update([0]);
-    }
-    hex(hasher.finalize().as_ref())
-}
-
-fn millis(state: &QueueState) -> u64 {
-    u64::try_from(state.started.elapsed().as_millis()).unwrap_or(u64::MAX)
-}
-
-pub(super) async fn intercept(
-    request: Request<ProxyBody>,
-    client: ProxyClient,
-    target: SocketAddr,
-    state: Arc<QueueState>,
-) -> io::Result<Interception> {
-    if Instant::now() >= state.deadline || state.closed.load(Ordering::SeqCst) {
-        return Ok(Interception::Pass(request));
-    }
-    let (parts, body) = request.into_parts();
-    let collected = tokio::time::timeout_at(state.deadline, Limited::new(body, MAX_BYTES).collect())
-        .await
-        .map_err(|_| io::Error::new(io::ErrorKind::TimedOut, "scoped ACK frame collection expired"))?
-        .map_err(io::Error::other)?;
-    let trailers = collected.trailers().cloned();
-    let bytes = collected.to_bytes();
-    let decoded = ScannerScopedDirtyUsageAckRequest::decode(unary_payload(&bytes)?).map_err(io::Error::other)?;
-    let request = Request::from_parts(parts, restored_body(bytes.clone(), trailers.clone()));
-    if Instant::now() >= state.deadline || !state.scope.matches(&decoded) {
-        return Ok(Interception::Pass(request));
-    }
-    let header = |name| request.headers().get(name).and_then(|value| value.to_str().ok());
-    let (action, mut selected) = watch::channel(None);
-    let packet = {
-        let mut packets = state.packets.lock().expect("scoped ACK packets");
-        if packets.len() >= state.capacity || state.closed.load(Ordering::SeqCst) {
-            return Ok(Interception::Pass(request));
-        }
-        let id = packets.len();
-        let packet = Arc::new(Packet {
-            action,
-            task: Mutex::new(None),
-            receipt: Mutex::new(json!({
-                "id": id, "session_nonce": state.scope.nonce, "receiver": target.to_string(),
-                "authority": request.uri().authority().map(|value| value.as_str()),
-                "captured_at_ms": millis(&state), "completed": false, "forwarded": false,
-                "backend_complete": false, "outcome": "held", "request_bytes": bytes.len(),
-            "request_sha256": hash(&bytes), "request_frame_hex": hex(&bytes), "request_headers_sha256": header_hash(request.headers()),
-                "request_trailers_sha256": trailers.as_ref().map(header_hash),
-            "request_trailers": trailer_receipt(trailers.as_ref()),
-                "auth_nonce": header("x-rustfs-rpc-nonce"), "replay_nonce": header("x-rustfs-rpc-replay-nonce"),
-                "signature_sha256": header("x-rustfs-rpc-signature-v2").map(|value| hash(value.as_bytes())),
-                "replay_signature_sha256": header("x-rustfs-rpc-signature-v3").map(|value| hash(value.as_bytes())),
-                "request": {"owner_id": decoded.owner_id, "instance_id": decoded.instance_id,
-                    "probe_only": decoded.probe_only, "entries": decoded.entries.iter().map(|entry| json!({
-                        "bucket": entry.bucket, "bucket_incarnation": uuid::Uuid::from_slice(&entry.bucket_incarnation).ok().map(|value| value.to_string()),
-                        "generation": entry.generation })).collect::<Vec<_>>() }
-            })),
-        });
-        packets.push(packet.clone());
-        packet
-    };
-    let (reply, response) = oneshot::channel();
-    let worker_packet = packet.clone();
-    let worker_state = state.clone();
-    // This owner survives an HTTP/2 stream reset from the original 5 s caller.
-    // Every request is settled once and is discarded at the session deadline.
-    let task = tokio::spawn(async move {
-        let result = tokio::time::timeout_at(worker_state.deadline, async {
-            let fault = loop {
-                if let Some(fault) = *selected.borrow_and_update() {
-                    break fault;
-                }
-                selected
-                    .changed()
-                    .await
-                    .map_err(|_| io::Error::other("scoped ACK queue closed"))?;
-            };
-            if fault == ScopedAckFault::RequestLoss {
-                worker_packet.receipt.lock().expect("scoped ACK receipt")["outcome"] = json!("request_lost");
-                return Err(io::Error::new(io::ErrorKind::ConnectionAborted, "injected scoped ACK request loss"));
-            }
-            if worker_state.closed.load(Ordering::SeqCst) || Instant::now() >= worker_state.deadline {
-                return Err(io::Error::new(
-                    io::ErrorKind::ConnectionAborted,
-                    "scoped ACK session ended or expired before forwarding",
-                ));
-            }
-            {
-                let mut receipt = worker_packet.receipt.lock().expect("scoped ACK receipt");
-                receipt["forwarded"] = json!(true);
-                receipt["forwarded_at_ms"] = json!(millis(&worker_state));
-            }
-            let backend = forward_to_backend(request, client, target).await?;
-            let (parts, body) = backend.into_parts();
-            let collected = Limited::new(body, MAX_BYTES).collect().await.map_err(io::Error::other)?;
-            let trailers = collected.trailers().cloned();
-            let bytes = collected.to_bytes();
-            let grpc_status = trailers
-                .as_ref()
-                .and_then(|values| values.get("grpc-status"))
-                .or_else(|| parts.headers.get("grpc-status"))
-                .and_then(|value| value.to_str().ok());
-            let response = unary_payload(&bytes)
-                .ok()
-                .and_then(|payload| ScannerScopedDirtyUsageAckResponse::decode(payload).ok())
-                .map(|response| {
-                    json!({"owner_id": response.owner_id, "instance_id": response.instance_id,
-                    "supported": response.supported, "cleared": response.cleared,
-                    "response_proof_bytes": response.response_proof.len()})
-                });
-            {
-                let mut receipt = worker_packet.receipt.lock().expect("scoped ACK receipt");
-                receipt["backend_complete"] = json!(true);
-                receipt["backend_completed_at_ms"] = json!(millis(&worker_state));
-                receipt["response"] = json!({"status": parts.status.as_u16(), "grpc_status": grpc_status,
-                    "bytes": bytes.len(), "sha256": hash(&bytes), "frame_hex": hex(&bytes), "headers_sha256": header_hash(&parts.headers),
-                    "trailers": trailer_receipt(trailers.as_ref()),
-                    "trailers_sha256": trailers.as_ref().map(header_hash), "ack": response});
-                receipt["outcome"] = json!(if fault == ScopedAckFault::ResponseLoss {
-                    "response_lost"
-                } else {
-                    "forwarded"
-                });
-            }
-            if fault == ScopedAckFault::ResponseLoss {
-                return Err(io::Error::new(
-                    io::ErrorKind::ConnectionAborted,
-                    "injected scoped ACK response loss after backend trailers",
-                ));
-            }
-            Ok(Response::from_parts(parts, restored_body(bytes, trailers)))
-        })
-        .await
-        .unwrap_or_else(|_| Err(io::Error::new(io::ErrorKind::TimedOut, "scoped ACK retention expired")));
-        {
-            let mut receipt = worker_packet.receipt.lock().expect("scoped ACK receipt");
-            if result.is_err() && receipt["outcome"] == "held" {
-                receipt["outcome"] = json!("transport_failed_or_expired");
-            }
-            receipt["completed"] = json!(true);
-            receipt["completed_at_ms"] = json!(millis(&worker_state));
-        }
-        let _ = reply.send(result);
-        worker_state.changed.notify_one();
-    });
-    *packet.task.lock().expect("scoped ACK task") = Some(task.abort_handle());
-    if state.closed.load(Ordering::SeqCst) {
-        task.abort();
-    }
-    drop(task);
-    state.changed.notify_one();
-    response
-        .await
-        .map_err(|_| io::Error::new(io::ErrorKind::ConnectionAborted, "scoped ACK queue ended"))?
-        .map(Interception::Handled)
-}

crates/e2e_test/src/storage_api.rs

@@ -80,14 +80,3 @@ pub(crate) mod multipart_source_rpc {
         PublicationIdentity, SourceGeneration, SourceGeometry, SourceLineage,
     };
 }
-
-#[cfg(test)]
-pub(crate) mod scanner_peer_bucket {
-    use rustfs_ecstore::api::disk as ecstore_disk;
-    use rustfs_ecstore::api::erasure as ecstore_erasure;
-    use rustfs_ecstore::api::rpc as ecstore_rpc;
-
-    pub(crate) use ecstore_disk::{BUCKET_META_PREFIX, RUSTFS_META_BUCKET};
-    pub(crate) use ecstore_erasure::{ReedSolomonEncoder, calc_shard_size};
-    pub(crate) use ecstore_rpc::PeerRestClient;
-}

crates/ecstore/src/bucket/metadata.rs

@@ -1503,7 +1503,11 @@ pub(crate) async fn load_bucket_metadata_parse_with_presence(
     bucket: &str,
     parse: bool,
 ) -> Result<(BucketMetadata, bool)> {
-    let (mut bm, persisted) = match read_bucket_metadata(api.clone(), bucket).await {
+    let (metadata_res, incarnation_res) = tokio::join!(
+        Box::pin(read_bucket_metadata(api.clone(), bucket)),
+        Box::pin(load_bucket_incarnation(api.clone(), bucket)),
+    );
+    let (mut bm, persisted) = match metadata_res {
         Ok(res) => (res, true),
         Err(err) => {
             if err != Error::ConfigNotFound {
@@ -1514,7 +1518,7 @@ pub(crate) async fn load_bucket_metadata_parse_with_presence(
         }
     };
 
-    let incarnation = load_bucket_incarnation(api.clone(), bucket).await?;
+    let incarnation = incarnation_res?;
     if persisted {
         if bm.name.is_empty() {
             bm.name = bucket.to_string();

crates/ecstore/src/bucket/metadata_sys.rs

@@ -1435,6 +1435,7 @@ pub(crate) async fn get_object_lock_config_state_in(
 
 /// Re-read the Object Lock state and bucket incarnation from the same
 /// authoritative metadata blob while the caller holds the transaction lock.
+#[cfg(all(test, feature = "test-util"))]
 pub(crate) async fn get_object_lock_config_and_incarnation_from_disk_in(
     ctx: &crate::runtime::instance::InstanceContext,
     bucket: &str,
@@ -1453,6 +1454,27 @@ pub(crate) async fn get_object_lock_config_and_incarnation_from_disk_with_guard_
     get_delete_config_from_disk_in(ctx, bucket, guard, None).await
 }
 
+#[allow(clippy::type_complexity)]
+pub(crate) async fn get_object_lock_and_quota_config_from_disk_with_guard_in(
+    ctx: &crate::runtime::instance::InstanceContext,
+    bucket: &str,
+    guard: Option<&rustfs_lock::NamespaceLockGuard>,
+) -> Result<(
+    ObjectLockConfigState,
+    Uuid,
+    OffsetDateTime,
+    std::result::Result<(Option<BucketQuota>, Uuid, OffsetDateTime), String>,
+)> {
+    let (state, incarnation_id, updated_at, quota_res) =
+        get_delete_and_quota_config_from_disk_in(ctx, bucket, guard, None, true).await?;
+    Ok((
+        state,
+        incarnation_id,
+        updated_at,
+        quota_res.unwrap_or(Ok((None, incarnation_id, updated_at))),
+    ))
+}
+
 /// Validate queued expiration against the same fresh metadata used for Object
 /// Lock admission. The caller holds the transaction read lock through commit,
 /// so a successful lifecycle update cannot be followed by an old-policy delete.
@@ -1462,6 +1484,24 @@ pub(crate) async fn get_delete_config_from_disk_in(
     guard: Option<&rustfs_lock::NamespaceLockGuard>,
     expiry_policy: Option<&rustfs_lifecycle::ExpiryPolicySnapshot>,
 ) -> Result<(ObjectLockConfigState, Uuid, OffsetDateTime)> {
+    let (state, incarnation_id, updated_at, _) =
+        get_delete_and_quota_config_from_disk_in(ctx, bucket, guard, expiry_policy, false).await?;
+    Ok((state, incarnation_id, updated_at))
+}
+
+#[allow(clippy::type_complexity)]
+async fn get_delete_and_quota_config_from_disk_in(
+    ctx: &crate::runtime::instance::InstanceContext,
+    bucket: &str,
+    guard: Option<&rustfs_lock::NamespaceLockGuard>,
+    expiry_policy: Option<&rustfs_lifecycle::ExpiryPolicySnapshot>,
+    include_quota: bool,
+) -> Result<(
+    ObjectLockConfigState,
+    Uuid,
+    OffsetDateTime,
+    Option<std::result::Result<(Option<BucketQuota>, Uuid, OffsetDateTime), String>>,
+)> {
     let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?;
     let bucket_meta_sys = bucket_meta_sys_lock.read().await.clone();
 
@@ -1477,7 +1517,7 @@ pub(crate) async fn get_delete_config_from_disk_in(
                 .await?
         }
     };
-    match authority {
+    match &authority {
         BucketMetadataAuthority::Authoritative(metadata)
             if metadata.bucket_incarnation_sidecar && !metadata.bucket_incarnation_id.is_nil() =>
         {
@@ -1489,11 +1529,11 @@ pub(crate) async fn get_delete_config_from_disk_in(
                 }
             }
             let object_lock_config_updated_at = metadata.object_lock_config_updated_at;
-            Ok((
-                object_lock_config_state_from_authoritative_metadata(&metadata)?,
-                metadata.bucket_incarnation_id,
-                object_lock_config_updated_at,
-            ))
+            let state = object_lock_config_state_from_authoritative_metadata(metadata)?;
+            let incarnation_id = metadata.bucket_incarnation_id;
+            let quota_res = include_quota
+                .then(|| quota_config_and_incarnation_from_authority(bucket, authority).map_err(|err| err.to_string()));
+            Ok((state, incarnation_id, object_lock_config_updated_at, quota_res))
         }
         BucketMetadataAuthority::Authoritative(_) => {
             Err(Error::other(format!("bucket incarnation metadata is not authoritative: {bucket}")))
@@ -1564,6 +1604,23 @@ pub(crate) async fn get_quota_config_and_incarnation_from_disk_for_options_in(
         return Err(Error::other("quota snapshot metadata transaction fence was released"));
     }
 
+    #[cfg(test)]
+    let has_injected_error = sys.object_lock_disk_read_errors.read().await.contains(bucket);
+    #[cfg(not(test))]
+    let has_injected_error = false;
+    if !ctx.lock_manager().is_disabled()
+        && !has_injected_error
+        && let Some(fresh_quota) = snapshot.take_fresh_put_quota()
+    {
+        if snapshot_guard.is_lock_lost()
+            || snapshot_guard.is_released()
+            || !snapshot.is_valid_for_destructive_put(store.id, bucket, opts.expected_bucket_incarnation_id.unwrap_or_default())
+        {
+            return Err(Error::other("quota snapshot fence or bucket incarnation changed during the disk read"));
+        }
+        return fresh_quota.map_err(Error::other);
+    }
+
     // The snapshot carries the caller's lifecycle fence, excluding deletion.
     // Its transaction guard excludes config writes, so a same-name object
     // owner need not recursively acquire the bucket namespace here.
@@ -1750,9 +1807,22 @@ struct MetadataPublishGuard {
     _guard: tokio::sync::OwnedMutexGuard<MetadataPublishLockState>,
 }
 
+static LOCAL_META_MUTATION_GEN: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(1);
+
+pub(crate) fn bump_local_meta_mutation_gen() {
+    LOCAL_META_MUTATION_GEN.fetch_add(1, std::sync::atomic::Ordering::Release);
+}
+
+pub(crate) fn current_local_meta_mutation_gen() -> u64 {
+    LOCAL_META_MUTATION_GEN.load(std::sync::atomic::Ordering::Acquire)
+}
+
+type LocalAuthorityCache = Arc<StdMutex<HashMap<String, (u64, Arc<BucketMetadata>)>>>;
+
 #[derive(Debug, Clone)]
 pub struct BucketMetadataSys {
     metadata_map: Arc<RwLock<HashMap<String, Arc<BucketMetadata>>>>,
+    local_authority_cache: LocalAuthorityCache,
     /// Serializes metadata-map commits and their derived cache updates for one
     /// bucket. Namespace locks, when present, are acquired before this lock.
     metadata_publish_locks: Arc<MetadataPublishLockRegistry>,
@@ -1796,6 +1866,7 @@ impl BucketMetadataSys {
     pub fn new(api: Arc<ECStore>) -> Self {
         Self {
             metadata_map: Arc::new(RwLock::new(HashMap::new())),
+            local_authority_cache: Arc::new(StdMutex::new(HashMap::new())),
             metadata_publish_locks: Arc::new(MetadataPublishLockRegistry {
                 locks: StdMutex::new(HashMap::new()),
             }),
@@ -2160,6 +2231,7 @@ impl BucketMetadataSys {
     }
 
     pub async fn set(&self, bucket: String, bm: Arc<BucketMetadata>) {
+        bump_local_meta_mutation_gen();
         if !is_meta_bucketname(&bucket) {
             let publish_lock = self.metadata_publish_lock(&bucket);
             let _publish_guard = publish_lock.lock().await;
@@ -2178,6 +2250,7 @@ impl BucketMetadataSys {
     ///
     /// Returns `true` if an entry was present. Reserved meta buckets are ignored.
     pub async fn remove(&self, bucket: &str) -> bool {
+        bump_local_meta_mutation_gen();
         if is_meta_bucketname(bucket) {
             return false;
         }
@@ -2198,6 +2271,7 @@ impl BucketMetadataSys {
     }
 
     async fn _reset(&mut self) {
+        bump_local_meta_mutation_gen();
         let mut map = self.metadata_map.write().await;
         map.clear();
         drop(map);
@@ -3001,30 +3075,77 @@ impl BucketMetadataSys {
             return Err(Error::other(format!("injected Object Lock metadata disk read failure: {bucket}")));
         }
 
-        match await_bucket_namespace_operation(Some(guard), bucket, "bucket metadata snapshot existence check", async {
-            self.object_store()
-                .get_bucket_info_from_sets_at_read_quorum(bucket, &crate::storage_api_contracts::bucket::BucketOptions::default())
-                .await
-        })
-        .await
-        {
+        let store = self.object_store();
+        let can_use_local_authority_cache = !store.ctx.lock_manager().is_disabled() && store.all_disks_local_and_online();
+        let gen_before = current_local_meta_mutation_gen();
+        let cached_authority = can_use_local_authority_cache
+            .then(|| {
+                self.local_authority_cache
+                    .lock()
+                    .unwrap_or_else(std::sync::PoisonError::into_inner)
+                    .get(bucket)
+                    .filter(|(cached_gen, _)| *cached_gen == gen_before)
+                    .map(|(_, bm)| Arc::clone(bm))
+            })
+            .flatten();
+        if let Some(cached) = cached_authority {
+            await_bucket_namespace_operation(
+                Some(guard),
+                bucket,
+                "bucket metadata authoritative snapshot",
+                std::future::ready(Ok::<_, Error>(())),
+            )
+            .await?;
+            if current_local_meta_mutation_gen() == gen_before && store.all_disks_local_and_online() {
+                return Ok(BucketMetadataAuthority::Authoritative(cached));
+            }
+        }
+
+        let (bucket_info_res, metadata_res) = tokio::join!(
+            Box::pin(await_bucket_namespace_operation(
+                Some(guard),
+                bucket,
+                "bucket metadata snapshot existence check",
+                async {
+                    store
+                        .get_bucket_info_from_sets_at_read_quorum(
+                            bucket,
+                            &crate::storage_api_contracts::bucket::BucketOptions::default(),
+                        )
+                        .await
+                },
+            )),
+            Box::pin(await_bucket_namespace_operation(
+                Some(guard),
+                bucket,
+                "bucket metadata authoritative snapshot",
+                load_bucket_metadata_parse_with_presence(Arc::clone(&store), bucket, true),
+            )),
+        );
+        match bucket_info_res {
             Ok(_) => {}
             Err(Error::VolumeNotFound) => return Ok(BucketMetadataAuthority::MissingBucket),
             Err(err) => return Err(err),
         }
 
-        let (metadata, persisted) = await_bucket_namespace_operation(
-            Some(guard),
-            bucket,
-            "bucket metadata authoritative snapshot",
-            load_bucket_metadata_parse_with_presence(self.object_store(), bucket, true),
-        )
-        .await?;
+        let (metadata, persisted) = metadata_res?;
         if persisted && metadata.lock_enabled && !metadata.bucket_creation_committed {
             return Err(Error::ErasureWriteQuorum);
         }
         if persisted {
-            Ok(BucketMetadataAuthority::Authoritative(Arc::new(metadata)))
+            let metadata = Arc::new(metadata);
+            if can_use_local_authority_cache
+                && metadata.bucket_incarnation_sidecar
+                && !metadata.bucket_incarnation_id.is_nil()
+                && current_local_meta_mutation_gen() == gen_before
+                && store.all_disks_local_and_online()
+            {
+                self.local_authority_cache
+                    .lock()
+                    .unwrap_or_else(std::sync::PoisonError::into_inner)
+                    .insert(bucket.to_string(), (gen_before, Arc::clone(&metadata)));
+            }
+            Ok(BucketMetadataAuthority::Authoritative(metadata))
         } else {
             Ok(BucketMetadataAuthority::Fabricated)
         }

crates/ecstore/src/crash_inject.rs

@@ -84,7 +84,7 @@ pub(crate) enum CrashPoint {
 }
 
 #[cfg(test)]
-pub(crate) use armed::{arm, disarm, should_crash_at};
+pub(crate) use armed::{arm, disarm, is_armed, should_crash_at};
 
 /// Production no-op: the compiler inlines this to `false`, so armed crash points
 /// exist only in `#[cfg(test)]` builds and never add a branch to real commits.
@@ -94,6 +94,12 @@ pub(crate) fn should_crash_at(_point: CrashPoint, _key: &str) -> bool {
     false
 }
 
+#[cfg(not(test))]
+#[inline(always)]
+pub(crate) fn is_armed(_point: CrashPoint, _key: &str) -> bool {
+    false
+}
+
 #[cfg(test)]
 mod armed {
     use super::CrashPoint;
@@ -132,6 +138,10 @@ mod armed {
         guard().retain(|(p, k)| !(*p == point && k == key));
     }
 
+    pub(crate) fn is_armed(point: CrashPoint, key: &str) -> bool {
+        guard().iter().any(|(p, k)| *p == point && k == key)
+    }
+
     /// Returns `true` (consuming the arm) iff a crash is armed for exactly this
     /// `point` and `key`.
     pub(crate) fn should_crash_at(point: CrashPoint, key: &str) -> bool {

crates/ecstore/src/disk/disk_store.rs

@@ -2327,6 +2327,51 @@ impl LocalDiskWrapper {
         .await
     }
 
+    pub(crate) async fn try_publish_prepared_delete_local(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: uuid::Uuid,
+        namespace_owner: Option<Arc<dyn Send + Sync>>,
+    ) -> Result<Option<(crate::disk::DeleteRecoveryJournal, bool, bool)>> {
+        self.track_disk_health_mutation(
+            "settle_delete_transaction",
+            DiskMetricMutation::Delete,
+            || {
+                Box::pin(async {
+                    self.disk
+                        .try_publish_prepared_delete_local(volume, object, transaction, namespace_owner)
+                        .await
+                })
+            },
+            crate::disk::get_delete_recovery_timeout()?,
+        )
+        .await
+    }
+
+    pub(crate) async fn try_finish_and_finalize_prepared_delete_local(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: uuid::Uuid,
+        journal: &crate::disk::DeleteRecoveryJournal,
+        namespace_owner: Option<Arc<dyn Send + Sync>>,
+    ) -> Result<bool> {
+        self.track_disk_health_mutation(
+            "settle_delete_transaction",
+            DiskMetricMutation::Delete,
+            || {
+                Box::pin(async {
+                    self.disk
+                        .try_finish_and_finalize_prepared_delete_local(volume, object, transaction, journal, namespace_owner)
+                        .await
+                })
+            },
+            crate::disk::get_delete_recovery_timeout()?,
+        )
+        .await
+    }
+
     /// Track disk health for an operation.
     /// This method should wrap disk operations to ensure health checking.
     pub async fn track_disk_health<T, F, Fut>(&self, operation: F, timeout_duration: Duration) -> Result<T>

crates/ecstore/src/disk/fs.rs

@@ -304,6 +304,7 @@ pub async fn cached_access(path: impl AsRef<Path>) -> io::Result<()> {
 }
 
 pub fn invalidate_bucket_cache(path: impl AsRef<Path>) {
+    crate::bucket::metadata_sys::bump_local_meta_mutation_gen();
     BUCKET_EXISTENCE_CACHE.invalidate(path.as_ref());
 }
 

crates/ecstore/src/disk/local.rs

@@ -74,8 +74,8 @@ use crate::disk::{
     error_conv::{to_access_error, to_file_error, to_unformatted_disk_error, to_volume_error},
     format::FormatV3,
     fs::{
-        O_APPEND, O_CREATE, O_RDONLY, O_TRUNC, O_WRONLY, access, cached_access, invalidate_bucket_cache, lstat, lstat_std,
-        remove, remove_all_std, remove_std, rename,
+        O_APPEND, O_CREATE, O_RDONLY, O_WRONLY, access, cached_access, invalidate_bucket_cache, lstat, lstat_std, remove,
+        remove_all_std, remove_std, rename,
     },
     is_quota_mutation_fence_path, os,
     os::{check_path_length, is_dir_not_empty_error, is_empty_dir, is_root_disk, rename_all, rename_all_ignore_missing_source},
@@ -2985,6 +2985,51 @@ fn has_inline_commit_test_failpoints(dst_path: &str) -> bool {
             == Some(dst_path)
 }
 
+#[cfg(all(test, target_os = "linux"))]
+pub(super) fn has_rename_data_test_failpoints(dst_path: &str) -> bool {
+    has_inline_commit_test_failpoints(dst_path)
+        || INLINE_PREPARATION_BEFORE_BACKUP
+            .lock()
+            .expect("test preparation hook lock should not be poisoned")
+            .contains_key(dst_path)
+        || !RENAME_DATA_AFTER_FIRST_PUBLICATION
+            .lock()
+            .expect("test publication hook lock should not be poisoned")
+            .is_empty()
+        || RENAME_DATA_FAIL_BEFORE_OLD_METADATA_BACKUP
+            .lock()
+            .expect("test failpoint lock should not be poisoned")
+            .as_deref()
+            == Some(dst_path)
+        || RENAME_DATA_FAIL_AFTER_METADATA_COMMIT
+            .lock()
+            .unwrap_or_else(std::sync::PoisonError::into_inner)
+            .iter()
+            .any(|target| target == dst_path)
+        || RENAME_DATA_FAIL_COMMIT_RENAME
+            .lock()
+            .expect("test failpoint lock should not be poisoned")
+            .as_deref()
+            == Some(dst_path)
+        || os::has_dir_fsync_test_failures()
+        || os::file_sync_probe::is_active()
+}
+
+#[cfg(all(not(test), feature = "test-util", target_os = "linux"))]
+pub(super) fn has_rename_data_test_failpoints(dst_path: &str) -> bool {
+    RENAME_DATA_FAIL_AFTER_METADATA_COMMIT
+        .lock()
+        .unwrap_or_else(std::sync::PoisonError::into_inner)
+        .iter()
+        .any(|target| target == dst_path)
+        || os::has_dir_fsync_test_failures()
+}
+
+#[cfg(any(not(any(test, feature = "test-util")), not(target_os = "linux")))]
+pub(super) fn has_rename_data_test_failpoints(_dst_path: &str) -> bool {
+    false
+}
+
 #[cfg(test)]
 fn run_inline_preparation_before_backup(dst_path: &str) {
     let hook = INLINE_PREPARATION_BEFORE_BACKUP
@@ -3056,6 +3101,43 @@ fn should_fail_after_delete_data_staged(path: &str) -> bool {
     }
 }
 
+#[cfg(test)]
+pub(crate) fn has_delete_version_test_failpoints() -> bool {
+    #[cfg(not(windows))]
+    if os::prepared_publication_test_hooks::has_any_hooks() {
+        return true;
+    }
+    #[cfg(feature = "e2e-test-hooks")]
+    if std::env::var_os("RUSTFS_DELETE_RECOVERY_BARRIER_DIR").is_some() {
+        return true;
+    }
+    !OWNED_FILE_WRITE_BEFORE_OPEN
+        .lock()
+        .expect("test file write hook lock should not be poisoned")
+        .is_empty()
+        || !DELETE_VERSION_FAIL_AFTER_DATA_STAGED
+            .lock()
+            .expect("test failpoint lock should not be poisoned")
+            .is_empty()
+        || !DELETE_VERSION_FAIL_AFTER_COMMIT
+            .lock()
+            .expect("test failpoint lock should not be poisoned")
+            .is_empty()
+}
+
+#[cfg(not(test))]
+pub(crate) fn has_delete_version_test_failpoints() -> bool {
+    #[cfg(all(feature = "test-util", not(windows)))]
+    if os::prepared_publication_test_hooks::has_any_hooks() {
+        return true;
+    }
+    #[cfg(feature = "e2e-test-hooks")]
+    if std::env::var_os("RUSTFS_DELETE_RECOVERY_BARRIER_DIR").is_some() {
+        return true;
+    }
+    false
+}
+
 #[cfg(test)]
 fn should_fail_after_delete_commit(root: &Path, path: &str) -> bool {
     let mut targets = DELETE_VERSION_FAIL_AFTER_COMMIT
@@ -3590,6 +3672,283 @@ impl AsyncRead for FileCacheReclaimReader {
     }
 }
 
+#[cfg(target_os = "linux")]
+fn reclaim_std_file_cache(file: &std::fs::File, reclaim_len: usize, reclaim_on_shutdown: bool) {
+    use core::num::NonZeroU64;
+    use rustix::fs::{Advice, fadvise};
+
+    if !reclaim_on_shutdown || reclaim_len == 0 {
+        return;
+    }
+    let started = std::time::Instant::now();
+    if let Some(len) = u64::try_from(reclaim_len).ok().and_then(NonZeroU64::new) {
+        if let Err(err) = fadvise(file, 0, Some(len), Advice::DontNeed) {
+            record_file_cache_reclaim_error("write");
+            debug!(error = ?err, reclaim_len, "failed to reclaim file cache after write");
+        } else {
+            record_file_cache_reclaim_success("write", reclaim_len, started);
+        }
+    }
+}
+
+#[cfg(not(target_os = "linux"))]
+fn reclaim_std_file_cache(_file: &std::fs::File, _reclaim_len: usize, _reclaim_on_shutdown: bool) {}
+
+static PRESYNCED_TMP_DATA_DIRS: std::sync::LazyLock<ParkingLotMutex<HashSet<PathBuf>>> =
+    std::sync::LazyLock::new(|| ParkingLotMutex::new(HashSet::new()));
+
+fn mark_presynced_tmp_data_dir(dir: PathBuf) {
+    let mut set = PRESYNCED_TMP_DATA_DIRS.lock();
+    if set.len() >= 4096 {
+        set.clear();
+    }
+    set.insert(dir);
+}
+
+fn unmark_presynced_tmp_data_dir(dir: &Path) {
+    PRESYNCED_TMP_DATA_DIRS.lock().remove(dir);
+}
+
+#[cfg(target_os = "linux")]
+pub(in crate::disk) fn take_presynced_tmp_data_dir(dir: &Path) -> bool {
+    PRESYNCED_TMP_DATA_DIRS.lock().remove(dir)
+}
+
+fn open_lazy_local_file_std(file_path: &Path, volume_dir: &Path, _reclaim_on_shutdown: bool) -> std::io::Result<std::fs::File> {
+    if let Some(parent) = file_path.parent()
+        && (volume_dir.as_os_str().is_empty() || !volume_dir.starts_with(parent))
+        && let Err(err) = super::fs::create_dir_all_std(parent)
+        && err.kind() != std::io::ErrorKind::AlreadyExists
+    {
+        return Err(to_file_error(err));
+    }
+    let file = std::fs::OpenOptions::new()
+        .write(true)
+        .create(true)
+        .truncate(true)
+        .open(file_path)
+        .map_err(to_file_error)?;
+    #[cfg(target_os = "macos")]
+    if _reclaim_on_shutdown {
+        let _ = set_std_fd_nocache(&file);
+    }
+    Ok(file)
+}
+
+struct LazyLocalFileWriter {
+    file_path: PathBuf,
+    volume_dir: PathBuf,
+    reclaim_len: usize,
+    reclaim_on_shutdown: bool,
+    sync_on_close: bool,
+    bytes_written: usize,
+    state: LazyLocalFileWriterState,
+}
+
+enum LazyLocalFileWriterState {
+    Unopened,
+    Writing(tokio::task::JoinHandle<std::io::Result<(Option<std::fs::File>, usize)>>),
+    Open(std::fs::File),
+    Closing(tokio::task::JoinHandle<std::io::Result<()>>),
+    Closed,
+}
+
+impl LazyLocalFileWriter {
+    fn new(file_path: PathBuf, volume_dir: PathBuf, reclaim_len: usize, reclaim_on_shutdown: bool, sync_on_close: bool) -> Self {
+        Self {
+            file_path,
+            volume_dir,
+            reclaim_len,
+            reclaim_on_shutdown,
+            sync_on_close,
+            bytes_written: 0,
+            state: LazyLocalFileWriterState::Unopened,
+        }
+    }
+
+    fn start_write_payload(&mut self, payload: Vec<u8>) {
+        let len = payload.len();
+        let close_after = self.reclaim_len > 0 && self.bytes_written.saturating_add(len) >= self.reclaim_len;
+        let reclaim_len = self.bytes_written.saturating_add(len).max(self.reclaim_len);
+        let reclaim_on_shutdown = self.reclaim_on_shutdown;
+        let sync_on_close = self.sync_on_close;
+        match std::mem::replace(&mut self.state, LazyLocalFileWriterState::Closed) {
+            LazyLocalFileWriterState::Unopened => {
+                let file_path = self.file_path.clone();
+                let volume_dir = self.volume_dir.clone();
+                self.state = LazyLocalFileWriterState::Writing(tokio::task::spawn_blocking(move || {
+                    let mut file = open_lazy_local_file_std(&file_path, &volume_dir, reclaim_on_shutdown)?;
+                    std::io::Write::write_all(&mut file, &payload).map_err(to_file_error)?;
+                    if close_after {
+                        if sync_on_close {
+                            file.sync_data().map_err(to_file_error)?;
+                        }
+                        reclaim_std_file_cache(&file, reclaim_len, reclaim_on_shutdown);
+                        drop(file);
+                        if sync_on_close && let Some(parent) = file_path.parent() {
+                            mark_presynced_tmp_data_dir(parent.to_path_buf());
+                        }
+                        Ok((None, len))
+                    } else {
+                        Ok((Some(file), len))
+                    }
+                }));
+            }
+            LazyLocalFileWriterState::Open(mut file) => {
+                let file_path = self.file_path.clone();
+                self.state = LazyLocalFileWriterState::Writing(tokio::task::spawn_blocking(move || {
+                    std::io::Write::write_all(&mut file, &payload).map_err(to_file_error)?;
+                    if close_after {
+                        if sync_on_close {
+                            file.sync_data().map_err(to_file_error)?;
+                        }
+                        reclaim_std_file_cache(&file, reclaim_len, reclaim_on_shutdown);
+                        drop(file);
+                        if sync_on_close && let Some(parent) = file_path.parent() {
+                            mark_presynced_tmp_data_dir(parent.to_path_buf());
+                        }
+                        Ok((None, len))
+                    } else {
+                        Ok((Some(file), len))
+                    }
+                }));
+            }
+            other => self.state = other,
+        }
+    }
+
+    fn poll_in_flight_write(&mut self, cx: &mut std::task::Context<'_>) -> std::task::Poll<std::io::Result<usize>> {
+        let LazyLocalFileWriterState::Writing(fut) = &mut self.state else {
+            return std::task::Poll::Ready(Ok(0));
+        };
+        match std::pin::Pin::new(fut).poll(cx) {
+            std::task::Poll::Pending => std::task::Poll::Pending,
+            std::task::Poll::Ready(Err(join_err)) => {
+                self.state = LazyLocalFileWriterState::Closed;
+                std::task::Poll::Ready(Err(std::io::Error::other(join_err)))
+            }
+            std::task::Poll::Ready(Ok(Err(err))) => {
+                self.state = LazyLocalFileWriterState::Closed;
+                std::task::Poll::Ready(Err(err))
+            }
+            std::task::Poll::Ready(Ok(Ok((maybe_file, written)))) => {
+                self.bytes_written = self.bytes_written.saturating_add(written);
+                self.state = maybe_file.map_or(LazyLocalFileWriterState::Closed, LazyLocalFileWriterState::Open);
+                std::task::Poll::Ready(Ok(written))
+            }
+        }
+    }
+}
+
+impl AsyncWrite for LazyLocalFileWriter {
+    fn poll_write(
+        mut self: std::pin::Pin<&mut Self>,
+        cx: &mut std::task::Context<'_>,
+        buf: &[u8],
+    ) -> std::task::Poll<std::io::Result<usize>> {
+        if matches!(self.state, LazyLocalFileWriterState::Writing(_)) {
+            return self.poll_in_flight_write(cx);
+        }
+        if buf.is_empty() {
+            return std::task::Poll::Ready(Ok(0));
+        }
+        if matches!(self.state, LazyLocalFileWriterState::Closed | LazyLocalFileWriterState::Closing(_)) {
+            return std::task::Poll::Ready(Err(std::io::Error::new(
+                std::io::ErrorKind::BrokenPipe,
+                "local file writer is already closed",
+            )));
+        }
+        self.start_write_payload(buf.to_vec());
+        self.poll_in_flight_write(cx)
+    }
+
+    fn poll_write_vectored(
+        mut self: std::pin::Pin<&mut Self>,
+        cx: &mut std::task::Context<'_>,
+        bufs: &[std::io::IoSlice<'_>],
+    ) -> std::task::Poll<std::io::Result<usize>> {
+        if matches!(self.state, LazyLocalFileWriterState::Writing(_)) {
+            return self.poll_in_flight_write(cx);
+        }
+        let total_len: usize = bufs.iter().map(|b| b.len()).sum();
+        if total_len == 0 {
+            return std::task::Poll::Ready(Ok(0));
+        }
+        if matches!(self.state, LazyLocalFileWriterState::Closed | LazyLocalFileWriterState::Closing(_)) {
+            return std::task::Poll::Ready(Err(std::io::Error::new(
+                std::io::ErrorKind::BrokenPipe,
+                "local file writer is already closed",
+            )));
+        }
+        let mut payload = Vec::with_capacity(total_len);
+        for slice in bufs {
+            payload.extend_from_slice(slice);
+        }
+        self.start_write_payload(payload);
+        self.poll_in_flight_write(cx)
+    }
+
+    fn is_write_vectored(&self) -> bool {
+        true
+    }
+
+    fn poll_flush(mut self: std::pin::Pin<&mut Self>, cx: &mut std::task::Context<'_>) -> std::task::Poll<std::io::Result<()>> {
+        if matches!(self.state, LazyLocalFileWriterState::Writing(_)) {
+            std::task::ready!(self.poll_in_flight_write(cx))?;
+        }
+        std::task::Poll::Ready(Ok(()))
+    }
+
+    fn poll_shutdown(
+        mut self: std::pin::Pin<&mut Self>,
+        cx: &mut std::task::Context<'_>,
+    ) -> std::task::Poll<std::io::Result<()>> {
+        if matches!(self.state, LazyLocalFileWriterState::Writing(_)) {
+            std::task::ready!(self.poll_in_flight_write(cx))?;
+        }
+        if !matches!(self.state, LazyLocalFileWriterState::Closing(_)) {
+            match std::mem::replace(&mut self.state, LazyLocalFileWriterState::Closed) {
+                LazyLocalFileWriterState::Closed => return std::task::Poll::Ready(Ok(())),
+                LazyLocalFileWriterState::Unopened => {
+                    let file_path = self.file_path.clone();
+                    let volume_dir = self.volume_dir.clone();
+                    let reclaim_on_shutdown = self.reclaim_on_shutdown;
+                    self.state = LazyLocalFileWriterState::Closing(tokio::task::spawn_blocking(move || {
+                        open_lazy_local_file_std(&file_path, &volume_dir, reclaim_on_shutdown).map(drop)
+                    }));
+                }
+                LazyLocalFileWriterState::Open(file) => {
+                    let file_path = self.file_path.clone();
+                    let reclaim_len = self.bytes_written.max(self.reclaim_len);
+                    let reclaim_on_shutdown = self.reclaim_on_shutdown;
+                    let sync_on_close = self.sync_on_close;
+                    self.state = LazyLocalFileWriterState::Closing(tokio::task::spawn_blocking(move || {
+                        if sync_on_close {
+                            file.sync_data().map_err(to_file_error)?;
+                        }
+                        reclaim_std_file_cache(&file, reclaim_len, reclaim_on_shutdown);
+                        drop(file);
+                        if sync_on_close && let Some(parent) = file_path.parent() {
+                            mark_presynced_tmp_data_dir(parent.to_path_buf());
+                        }
+                        Ok(())
+                    }));
+                }
+                other => {
+                    self.state = other;
+                }
+            }
+        }
+        let LazyLocalFileWriterState::Closing(fut) = &mut self.state else {
+            return std::task::Poll::Ready(Ok(()));
+        };
+        let res = std::task::ready!(std::pin::Pin::new(fut).poll(cx));
+        self.state = LazyLocalFileWriterState::Closed;
+        std::task::Poll::Ready(res.unwrap_or_else(|join_err| Err(std::io::Error::other(join_err))))
+    }
+}
+
+#[cfg(test)]
 impl FileCacheReclaimWriter {
     fn new(inner: File, reclaim_len: usize, reclaim_on_shutdown: bool) -> Self {
         #[cfg(target_os = "macos")]
@@ -3630,6 +3989,7 @@ impl FileCacheReclaimWriter {
     }
 }
 
+#[cfg(test)]
 impl AsyncWrite for FileCacheReclaimWriter {
     fn poll_write(
         mut self: std::pin::Pin<&mut Self>,
@@ -3727,6 +4087,7 @@ fn should_reclaim_file_cache_after_read(length: usize) -> bool {
 #[derive(Clone, Copy, Debug)]
 pub(crate) enum WriteMode {
     Truncate { size_hint: i64 },
+    TruncateSync { size_hint: i64 },
     Append,
 }
 
@@ -4494,13 +4855,16 @@ impl LocalIoBackend for StdBackend {
     async fn open_write(&self, volume: &str, path: &str, mode: WriteMode) -> Result<FileWriter> {
         self.invalidate_cached_fd(volume, path).await;
         match mode {
-            WriteMode::Truncate { size_hint } => {
+            WriteMode::Truncate { size_hint } | WriteMode::TruncateSync { size_hint } => {
+                let sync_on_close = matches!(mode, WriteMode::TruncateSync { .. });
                 let volume_dir = local_disk_bucket_path(self.io_root(), volume)?;
                 let file_path = local_disk_object_path(self.io_root(), volume, path)?;
                 check_path_length(file_path.to_string_lossy().as_ref())?;
-
                 if let Some(parent) = file_path.parent() {
-                    os::make_dir_all(parent, &volume_dir).await?;
+                    check_path_length(parent.to_string_lossy().as_ref())?;
+                    if volume == RUSTFS_META_TMP_BUCKET {
+                        unmark_presynced_tmp_data_dir(parent);
+                    }
                 }
 
                 // O_DIRECT streaming write (Linux, opt-in): shard bytes stream
@@ -4511,6 +4875,9 @@ impl LocalIoBackend for StdBackend {
                 // EINVAL (which would masquerade as a missing shard).
                 #[cfg(target_os = "linux")]
                 if is_direct_io_write_enabled() && self.direct_io_write.supported.load(Ordering::Relaxed) {
+                    if let Some(parent) = file_path.parent() {
+                        os::make_dir_all(parent, &volume_dir).await?;
+                    }
                     let write_state = self.direct_io_write.clone();
                     let direct_path = file_path.clone();
                     let direct = tokio::task::spawn_blocking(move || open_direct_writer(&direct_path, &write_state))
@@ -4521,14 +4888,14 @@ impl LocalIoBackend for StdBackend {
                     }
                 }
 
-                // O_TRUNC: if a file already exists at this path, stale trailing bytes past
-                // the new content would otherwise survive and mismatch the metadata size.
-                let f = super::fs::open_file(&file_path, O_CREATE | O_WRONLY | O_TRUNC)
-                    .await
-                    .map_err(to_file_error)?;
                 let reclaim_on_shutdown = should_reclaim_file_cache_after_write(size_hint);
-
-                Ok(Box::new(FileCacheReclaimWriter::new(f, size_hint.max(0) as usize, reclaim_on_shutdown)))
+                Ok(Box::new(LazyLocalFileWriter::new(
+                    file_path,
+                    volume_dir,
+                    size_hint.max(0) as usize,
+                    reclaim_on_shutdown,
+                    sync_on_close,
+                )))
             }
             WriteMode::Append => {
                 let volume_dir = local_disk_bucket_path(self.io_root(), volume)?;
@@ -7077,6 +7444,9 @@ impl LocalDisk {
         fi: FileInfo,
         namespace_owner: Option<Arc<dyn Send + Sync>>,
     ) -> Result<()> {
+        if volume == RUSTFS_META_BUCKET {
+            crate::bucket::metadata_sys::bump_local_meta_mutation_gen();
+        }
         if volume == RUSTFS_META_MULTIPART_BUCKET {
             crate::disk::multipart_source::native_source_protocol(&fi)?;
         }
@@ -7323,6 +7693,9 @@ impl LocalDisk {
         mut fi: FileInfo,
         mutation: DeleteVersionMutation,
     ) -> Result<()> {
+        if volume == RUSTFS_META_BUCKET {
+            crate::bucket::metadata_sys::bump_local_meta_mutation_gen();
+        }
         pool_meta_native::reject_noncanonical_object(&self.root, volume, path)?;
         let DeleteVersionMutation {
             force_del_marker,
@@ -7394,6 +7767,13 @@ impl LocalDisk {
         }
 
         let rollback_dir = opts.old_data_dir;
+        if let Some(rollback_dir) = rollback_dir
+            && let Some(()) = self
+                .try_fused_recoverable_delete_version(volume, path, &fi, &opts, rollback_dir, namespace_owner.clone())
+                .await?
+        {
+            return Ok(());
+        }
         let buf = match self.read_all_data(volume, &volume_dir, &xl_path).await {
             Ok(res) => res,
             Err(err) => {
@@ -7919,6 +8299,9 @@ impl LocalDisk {
         opt: DeleteOptions,
         namespace_owner: Option<Arc<dyn Send + Sync>>,
     ) -> Result<()> {
+        if volume == RUSTFS_META_BUCKET {
+            crate::bucket::metadata_sys::bump_local_meta_mutation_gen();
+        }
         multipart_staging::reject_native_control_write(&self.root, self.io_root(), volume, path)?;
         let mut source_owner = self.acquire_multipart_delete_owner(volume, path, namespace_owner).await?;
         let namespace_owner = source_owner.namespace_owner();
@@ -9536,6 +9919,9 @@ impl LocalDisk {
     // write_all_private with check_path_length
     #[tracing::instrument(level = "trace", skip_all)]
     async fn write_all_private(&self, volume: &str, path: &str, buf: Bytes, sync: SyncMode, skip_parent: &Path) -> Result<()> {
+        if volume == RUSTFS_META_BUCKET {
+            crate::bucket::metadata_sys::bump_local_meta_mutation_gen();
+        }
         let file_path = self.io_get_object_path(volume, path)?;
         check_path_length(file_path.to_string_lossy().as_ref())?;
 
@@ -11476,20 +11862,32 @@ pub(crate) async fn batch_shard_pread(requests: Vec<(std::path::PathBuf, usize,
                     return Err(DiskError::FileCorrupt);
                 }
 
-                let mut buf = diagnostic_stage!(Allocation, _index, || Ok::<_, DiskError>(vec![0u8; length]))?;
+                let mut buf = diagnostic_stage!(Allocation, _index, || Ok::<_, DiskError>(Vec::with_capacity(length)))?;
                 if length > 0 {
                     diagnostic_stage!(Read, _index, || measure_local_batch_stage(
                         metrics.as_deref(),
                         LocalBatchReadStage::Read,
                         || {
-                            while total < length {
-                                let nbytes = file
-                                    .read_at(&mut buf[total..], u64::try_from(offset + total).unwrap_or(u64::MAX))
+                            if offset == 0 {
+                                use std::io::Read as _;
+                                total = (&file)
+                                    .take(u64::try_from(length).unwrap_or(u64::MAX))
+                                    .read_to_end(&mut buf)
                                     .map_err(DiskError::from)?;
-                                if nbytes == 0 {
+                                if total < length {
                                     return Err(DiskError::FileCorrupt);
                                 }
-                                total += nbytes;
+                            } else {
+                                buf.resize(length, 0);
+                                while total < length {
+                                    let nbytes = file
+                                        .read_at(&mut buf[total..], u64::try_from(offset + total).unwrap_or(u64::MAX))
+                                        .map_err(DiskError::from)?;
+                                    if nbytes == 0 {
+                                        return Err(DiskError::FileCorrupt);
+                                    }
+                                    total += nbytes;
+                                }
                             }
                             Ok::<_, DiskError>(())
                         }
@@ -12741,9 +13139,23 @@ impl DiskAPI for LocalDisk {
             }
         }
 
-        self.io_backend
-            .open_write(volume, path, WriteMode::Truncate { size_hint: _file_size })
-            .await
+        #[cfg(test)]
+        let probe_active = os::file_sync_probe::is_active();
+        #[cfg(not(test))]
+        let probe_active = false;
+        let sync_on_close = cfg!(target_os = "linux")
+            && volume == RUSTFS_META_TMP_BUCKET
+            && !origvolume.is_empty()
+            && path.ends_with("/part.1")
+            && effective_durability(origvolume).syncs_data_shards()
+            && !os::file_fdatasync_group_commit_enabled()
+            && !probe_active;
+        let mode = if sync_on_close {
+            WriteMode::TruncateSync { size_hint: _file_size }
+        } else {
+            WriteMode::Truncate { size_hint: _file_size }
+        };
+        self.io_backend.open_write(volume, path, mode).await
     }
 
     async fn rename_file_durable(&self, src_volume: &str, src_path: &str, dst_volume: &str, dst_path: &str) -> Result<()> {
@@ -13308,16 +13720,70 @@ impl DiskAPI for LocalDisk {
 
             check_path_length(file_path.to_string_lossy().as_ref())?;
 
-            let buf = self
-                .read_all(volume, format!("{}/{}", path, STORAGE_FORMAT_FILE).as_str())
-                .await
-                .map_err(|e| {
-                    if e == DiskError::FileNotFound && fi.version_id.is_some() {
-                        DiskError::FileVersionNotFound
-                    } else {
-                        e
+            let xl_rel_path = format!("{path}/{STORAGE_FORMAT_FILE}");
+            if self.multipart_source_admission(volume, &xl_rel_path, true, false)?.is_none()
+                && !crash_inject::is_armed(CrashPoint::MetaWriteAfterTmpBeforeRename, &xl_rel_path)
+                && !has_delete_version_test_failpoints()
+                && !has_rename_data_test_failpoints(&xl_rel_path)
+            {
+                multipart_staging::reject_reserved_anchor(&xl_rel_path)?;
+                let volume_dir = self.io_get_bucket_path(volume)?;
+                let xl_file_path = self.io_get_object_path(volume, &xl_rel_path)?;
+                check_path_length(xl_file_path.to_string_lossy().as_ref())?;
+                let tmp_file_path = self.io_get_object_path(RUSTFS_META_TMP_BUCKET, Uuid::new_v4().to_string().as_str())?;
+                let sync = !opts.no_persistence && effective_durability(volume).syncs_commit_metadata();
+                let replace_user_metadata = opts.replace_user_metadata;
+                let has_version_id = fi.version_id.is_some();
+                let publication_root = self.publication_root.clone();
+                let lease = os::acquire_namespace_mutation_lease_with_owner(&xl_file_path, namespace_owner).await;
+                return os::run_blocking_namespace_operation_with_sync(lease, sync, move || -> std::io::Result<Result<()>> {
+                    let buf = match std::fs::read(&xl_file_path) {
+                        Ok(buf) => buf,
+                        Err(err) if err.kind() == ErrorKind::NotFound && has_version_id => {
+                            return Ok(Err(DiskError::FileVersionNotFound));
+                        }
+                        Err(err) => return Err(err),
+                    };
+                    if !FileMeta::is_xl2_v1_format(&buf) {
+                        return Ok(Err(DiskError::FileVersionNotFound));
                     }
-                })?;
+                    let wbuf = match FileMeta::load(&buf).and_then(|mut xl_meta| {
+                        xl_meta.update_object_version_with_opts(fi, replace_user_metadata)?;
+                        xl_meta.marshal_msg()
+                    }) {
+                        Ok(wbuf) => wbuf,
+                        Err(err) => return Ok(Err(err.into())),
+                    };
+                    if let Some(parent) = tmp_file_path.parent() {
+                        std::fs::create_dir_all(parent)?;
+                    }
+                    let mut tmp_file = std::fs::File::create(&tmp_file_path)?;
+                    std::io::Write::write_all(&mut tmp_file, &wbuf)?;
+                    if sync && !wbuf.is_empty() {
+                        tmp_file.sync_data()?;
+                    }
+                    drop(tmp_file);
+                    os::rename_all_with_precondition_and_parent_sync_std(
+                        &tmp_file_path,
+                        &xl_file_path,
+                        &volume_dir,
+                        &publication_root,
+                        None,
+                        sync,
+                    )?;
+                    Ok(Ok(()))
+                })
+                .await
+                .map_err(to_file_error)?;
+            }
+
+            let buf = self.read_all(volume, xl_rel_path.as_str()).await.map_err(|e| {
+                if e == DiskError::FileNotFound && fi.version_id.is_some() {
+                    DiskError::FileVersionNotFound
+                } else {
+                    e
+                }
+            })?;
 
             if !FileMeta::is_xl2_v1_format(buf.as_ref()) {
                 return Err(DiskError::FileVersionNotFound);
@@ -13330,13 +13796,7 @@ impl DiskAPI for LocalDisk {
             let wbuf = xl_meta.marshal_msg()?;
 
             return self
-                .write_all_meta_with_namespace_owner(
-                    volume,
-                    format!("{path}/{STORAGE_FORMAT_FILE}").as_str(),
-                    &wbuf,
-                    !opts.no_persistence,
-                    namespace_owner,
-                )
+                .write_all_meta_with_namespace_owner(volume, xl_rel_path.as_str(), &wbuf, !opts.no_persistence, namespace_owner)
                 .await;
         }
 
@@ -17417,6 +17877,54 @@ mod test {
         );
     }
 
+    #[cfg(target_os = "linux")]
+    #[tokio::test]
+    #[serial_test::serial(dst_dir_fsync_group_commit, file_fdatasync_group_commit)]
+    #[allow(clippy::await_holding_lock)]
+    async fn rename_data_inline_new_version_fsync_failure_preserves_previous_metadata() {
+        let _group_commit = os::set_dst_dir_fsync_group_commit_for_test(false);
+        let _file_group_commit = os::set_file_fdatasync_group_commit_for_test(false);
+        let _mode = durability_mode_override::set(DurabilityMode::Strict);
+        let dir = tempfile::tempdir().expect("temp dir should be created");
+        let endpoint = Endpoint::try_from(dir.path().to_str().expect("temp dir should be utf8")).expect("endpoint should parse");
+        let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
+        let bucket = "inline-version-fsync-failure";
+        let object = "dir/object";
+        ensure_test_volume(&disk, bucket).await;
+        ensure_test_volume(&disk, RUSTFS_META_TMP_BUCKET).await;
+
+        let old_version = Uuid::new_v4();
+        let old_fi = test_file_info(object, old_version, None, Some(Bytes::from_static(b"previous-inline-version")));
+        disk.rename_data(RUSTFS_META_TMP_BUCKET, "old-inline-stage", old_fi, bucket, object)
+            .await
+            .expect("previous inline version should commit");
+        let dst_meta = disk
+            .get_object_path(bucket, &format!("{object}/{STORAGE_FORMAT_FILE}"))
+            .expect("metadata path should resolve");
+        let previous = fs::read(&dst_meta).await.expect("previous metadata should be readable");
+        let destination_dir = dst_meta.parent().expect("metadata should have a parent").to_path_buf();
+        // Arm the failure after the fused-path selection so the test exercises
+        // its normal admission path instead of the failpoint fallback.
+        set_inline_before_file_sync_admission(object, move || {
+            os::fsync_dir_recorder::set_failure(&destination_dir, io::ErrorKind::PermissionDenied);
+        });
+
+        let new_version = Uuid::new_v4();
+        let new_fi = test_file_info(object, new_version, None, Some(Bytes::from_static(b"incoming-inline-version")));
+        let err = disk
+            .rename_data(RUSTFS_META_TMP_BUCKET, "new-inline-stage", new_fi, bucket, object)
+            .await
+            .expect_err("directory fsync failure must reject the new version");
+        assert!(matches!(err, DiskError::Io(ref err) if err.kind() == io::ErrorKind::PermissionDenied));
+        let restored = fs::read(&dst_meta)
+            .await
+            .expect("previous metadata must survive the failed commit");
+        assert_eq!(restored, previous, "rollback must restore all previously committed inline versions");
+        let metadata = FileMeta::load(&restored).expect("restored metadata must be valid");
+        assert!(metadata.find_version(Some(old_version)).is_ok());
+        assert!(metadata.find_version(Some(new_version)).is_err());
+    }
+
     // Seed a first PUT of `object` (no prior version) through the non-inline
     // rename_data path and return (disk, tempdir). The object dir and any prefix
     // dirs are created during the commit.
@@ -19643,6 +20151,116 @@ mod test {
         );
     }
 
+    #[cfg(target_os = "linux")]
+    #[tokio::test]
+    #[serial_test::serial(dst_dir_fsync_group_commit, file_fdatasync_group_commit, file_sync_probe)]
+    #[allow(clippy::await_holding_lock)]
+    async fn test_rename_data_unknown_destination_defers_shard_sync() {
+        let _group_commit = os::set_dst_dir_fsync_group_commit_for_test(false);
+        let _file_group_commit = os::set_file_fdatasync_group_commit_for_test(false);
+        let _mode = durability_mode_override::set(DurabilityMode::Strict);
+        for destination_mode in [DurabilityMode::None, DurabilityMode::Strict] {
+            let bucket = "unknown-destination-shard-bucket";
+            let _bucket_mode = BucketOverrideGuard::set(bucket, destination_mode);
+            let dir = tempfile::tempdir().expect("temp dir should be created");
+            let endpoint =
+                Endpoint::try_from(dir.path().to_str().expect("temp dir should be utf8")).expect("endpoint should parse");
+            let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
+            let object = "dir/object";
+            let tmp_object = "tmp-unknown-destination";
+            let data_dir = Uuid::new_v4();
+            ensure_test_volume(&disk, bucket).await;
+            ensure_test_volume(&disk, RUSTFS_META_TMP_BUCKET).await;
+
+            let shard_path = format!("{tmp_object}/{data_dir}/part.1");
+            let mut writer = disk
+                .create_file("", RUSTFS_META_TMP_BUCKET, &shard_path, 8)
+                .await
+                .expect("open staged shard writer without a destination");
+            writer.write_all(b"new-data").await.expect("write staged shard");
+            writer.shutdown().await.expect("close staged shard");
+            let staged_dir = disk
+                .get_object_path_for_io(RUSTFS_META_TMP_BUCKET, &format!("{tmp_object}/{data_dir}"))
+                .expect("staged shard directory should resolve");
+            assert!(
+                !PRESYNCED_TMP_DATA_DIRS.lock().contains(&staged_dir),
+                "an unknown destination must defer payload sync until commit"
+            );
+
+            let mut fi = test_file_info(object, Uuid::new_v4(), Some(data_dir), None);
+            fi.size = 8;
+            fi.parts[0].size = 8;
+            fi.parts[0].actual_size = 8;
+            disk.rename_data(RUSTFS_META_TMP_BUCKET, tmp_object, fi, bucket, object)
+                .await
+                .expect("rename_data should apply the destination policy");
+            assert_eq!(
+                os::fsync_dir_recorder::was_fsynced(&staged_dir),
+                destination_mode.syncs_data_shards(),
+                "the commit must apply the destination durability override"
+            );
+            let shard = disk
+                .get_object_path(bucket, &format!("{object}/{data_dir}/part.1"))
+                .expect("committed shard should resolve");
+            assert_eq!(fs::read(shard).await.expect("committed shard should be readable"), b"new-data");
+        }
+    }
+
+    #[cfg(target_os = "linux")]
+    #[tokio::test]
+    #[serial_test::serial(dst_dir_fsync_group_commit, file_fdatasync_group_commit, file_sync_probe)]
+    #[allow(clippy::await_holding_lock)]
+    async fn test_rename_data_presynced_shard_keeps_directory_sync() {
+        let _group_commit = os::set_dst_dir_fsync_group_commit_for_test(false);
+        let _file_group_commit = os::set_file_fdatasync_group_commit_for_test(false);
+        for mode in [DurabilityMode::Strict, DurabilityMode::Relaxed] {
+            let _mode = durability_mode_override::set(mode);
+            let dir = tempfile::tempdir().expect("temp dir should be created");
+            let endpoint =
+                Endpoint::try_from(dir.path().to_str().expect("temp dir should be utf8")).expect("endpoint should parse");
+            let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
+            let bucket = "presynced-shard-bucket";
+            let object = "dir/object";
+            let tmp_object = "tmp-presynced-write";
+            let data_dir = Uuid::new_v4();
+            ensure_test_volume(&disk, bucket).await;
+            ensure_test_volume(&disk, RUSTFS_META_TMP_BUCKET).await;
+
+            let shard_path = format!("{tmp_object}/{data_dir}/part.1");
+            let mut writer = disk
+                .create_file(bucket, RUSTFS_META_TMP_BUCKET, &shard_path, 8)
+                .await
+                .expect("open staged shard writer");
+            writer.write_all(b"new-data").await.expect("write staged shard");
+            writer.shutdown().await.expect("close staged shard");
+            let staged_dir = disk
+                .get_object_path_for_io(RUSTFS_META_TMP_BUCKET, &format!("{tmp_object}/{data_dir}"))
+                .expect("staged shard directory should resolve");
+            assert!(PRESYNCED_TMP_DATA_DIRS.lock().contains(&staged_dir), "the writer must pre-sync the shard");
+            assert!(!os::fsync_dir_recorder::was_fsynced(&staged_dir));
+
+            let mut fi = test_file_info(object, Uuid::new_v4(), Some(data_dir), None);
+            fi.size = 8;
+            fi.parts[0].size = 8;
+            fi.parts[0].actual_size = 8;
+            disk.rename_data(RUSTFS_META_TMP_BUCKET, tmp_object, fi, bucket, object)
+                .await
+                .expect("rename_data should commit");
+            assert!(
+                !PRESYNCED_TMP_DATA_DIRS.lock().contains(&staged_dir),
+                "the fused commit must consume the pre-sync marker"
+            );
+            assert!(
+                os::fsync_dir_recorder::was_fsynced(&staged_dir),
+                "pre-syncing the payload must not skip the shard directory barrier"
+            );
+            let shard = disk
+                .get_object_path(bucket, &format!("{object}/{data_dir}/part.1"))
+                .expect("committed shard should resolve");
+            assert_eq!(fs::read(shard).await.expect("committed shard should be readable"), b"new-data");
+        }
+    }
+
     #[tokio::test]
     #[allow(clippy::await_holding_lock)]
     async fn test_rename_data_legacy_off_skips_all_fsyncs() {
@@ -19663,17 +20281,20 @@ mod test {
         ensure_test_volume(&disk, bucket).await;
         ensure_test_volume(&disk, RUSTFS_META_TMP_BUCKET).await;
 
-        let tmp_data_dir = dir
-            .path()
-            .join(RUSTFS_META_TMP_BUCKET)
-            .join(tmp_object)
-            .join(new_data_dir.to_string());
-        fs::create_dir_all(&tmp_data_dir)
+        let shard_path = format!("{tmp_object}/{new_data_dir}/part.1");
+        let mut writer = disk
+            .create_file("", RUSTFS_META_TMP_BUCKET, &shard_path, 8)
             .await
-            .expect("new tmp data dir should be created");
-        fs::write(tmp_data_dir.join("part.1"), b"new-data")
-            .await
-            .expect("new tmp data should be written");
+            .expect("open staged shard writer");
+        writer.write_all(b"new-data").await.expect("write staged shard");
+        writer.shutdown().await.expect("close staged shard");
+        let resolved_tmp_data_dir = disk
+            .get_object_path(RUSTFS_META_TMP_BUCKET, &format!("{tmp_object}/{new_data_dir}"))
+            .expect("tmp data dir path should resolve");
+        assert!(
+            !os::fsync_dir_recorder::was_fsynced(&resolved_tmp_data_dir),
+            "the staged writer must not sync before the destination policy is known"
+        );
 
         let new_fi = test_file_info(object, version_id, Some(new_data_dir), None);
         disk.rename_data(RUSTFS_META_TMP_BUCKET, tmp_object, new_fi, bucket, object)

crates/ecstore/src/disk/local/commit.rs

@@ -329,6 +329,38 @@ pub(super) async fn lock_rename_commit_directories(
     result.map_err(to_file_error).map_err(DiskError::from)
 }
 
+#[cfg(target_os = "linux")]
+fn prepare_rename_commit_and_destination_metadata_std(
+    create_source_parent: bool,
+    source_parent: &Path,
+    destination_parent: &Path,
+    destination_metadata: &Path,
+    base_dir: &Path,
+    publication_root: &os::PublicationRoot,
+) -> std::io::Result<(os::RenameCommitGuard, Option<Bytes>)> {
+    if create_source_parent {
+        std::fs::create_dir_all(source_parent)?;
+    }
+    let guard = match os::prepare_rename_commit_guard(source_parent, destination_parent, base_dir, publication_root) {
+        Ok(guard) => guard,
+        Err(err) => {
+            return Err(match std::fs::symlink_metadata(base_dir) {
+                Err(base_err) if base_err.kind() == ErrorKind::NotFound => base_err,
+                _ => err,
+            });
+        }
+    };
+    let data = os::read_destination_file_with_commit_guard(destination_metadata, &guard)?;
+    Ok((guard, data.map(Bytes::from)))
+}
+
+#[cfg(target_os = "linux")]
+enum FusedCommitAttempt {
+    Committed(RenameDataResp),
+    CommitErr(DiskError),
+    Fallback(os::RenameCommitGuard, Option<Bytes>, FileInfo),
+}
+
 async fn prepare_rename_commit_and_destination_metadata(
     create_source_parent: bool,
     source_parent: &Path,
@@ -571,6 +603,9 @@ impl LocalDisk {
                 return Err(err);
             }
         }
+        if dst_volume == crate::disk::RUSTFS_META_BUCKET {
+            crate::bucket::metadata_sys::bump_local_meta_mutation_gen();
+        }
         if fi.is_legacy_indexed_delete_marker() {
             fi.erasure.index = 0;
         }
@@ -698,6 +733,216 @@ impl LocalDisk {
         let dst_file_parent = dst_file_path
             .parent()
             .ok_or_else(|| DiskError::other("missing object metadata parent"))?;
+
+        #[cfg(target_os = "linux")]
+        let (rename_commit_guard, has_dst_buf) = if !preserve_upload_metadata
+            && multipart_publication.is_none()
+            && pool_meta_condition.is_none()
+            && !fi_healing
+            && !os::dst_dir_fsync_group_commit_enabled()
+            && !os::file_fdatasync_group_commit_enabled()
+            && !super::has_rename_data_test_failpoints(dst_path)
+            && !crash_inject::is_armed(CrashPoint::RenameAfterDataRename, dst_path)
+            && !crash_inject::is_armed(CrashPoint::RenameAfterBackupBeforeMetaCommit, dst_path)
+            && !crash_inject::is_armed(CrashPoint::RenameAfterMetaCommit, dst_path)
+            && !super::has_delete_version_test_failpoints()
+        {
+            #[cfg(test)]
+            if !no_inline {
+                run_inline_before_file_sync_admission(dst_path);
+            }
+            let sync_data_shards = no_inline && durability.syncs_data_shards();
+            let sync_commit_metadata = durability.syncs_commit_metadata();
+            let file_sync_admission = if sync_data_shards || sync_commit_metadata {
+                Some(
+                    os::acquire_file_sync_admission(self.file_sync_permits.clone())
+                        .await
+                        .map_err(to_file_error)
+                        .map_err(DiskError::from)?,
+                )
+            } else {
+                None
+            };
+            let src_parent_owned = src_file_parent.to_path_buf();
+            let dst_parent_owned = dst_file_parent.to_path_buf();
+            let src_file_path_owned = src_file_path.clone();
+            let dst_file_path_owned = dst_file_path.clone();
+            let dst_volume_dir_owned = dst_volume_dir.clone();
+            let publication_root_owned = self.publication_root.clone();
+            let has_data_dir_path_owned = has_data_dir_path.clone();
+            let dst_volume_str = dst_volume.to_string();
+            let dst_path_str = dst_path.to_string();
+            let fi_moved = fi;
+            #[cfg(test)]
+            let root_owned = self.root.clone();
+
+            let fused_attempt = move |is_cancelled: &dyn Fn() -> bool| -> std::io::Result<FusedCommitAttempt> {
+                let (guard, has_dst_buf) = prepare_rename_commit_and_destination_metadata_std(
+                    !no_inline,
+                    &src_parent_owned,
+                    &dst_parent_owned,
+                    &dst_file_path_owned,
+                    &dst_volume_dir_owned,
+                    &publication_root_owned,
+                )?;
+                // An existing xl.meta can share versions or require rollback
+                // restoration when a post-rename directory sync fails.
+                if is_cancelled() || (sync_commit_metadata && has_dst_buf.is_some()) {
+                    return Ok(FusedCommitAttempt::Fallback(guard, has_dst_buf, fi_moved));
+                }
+                let mut xlmeta = FileMeta::new();
+                let mut dst_meta_unparsable = false;
+                if let Some(ref dst_buf) = has_dst_buf {
+                    if FileMeta::is_xl2_v1_format(dst_buf)
+                        && let Ok(nmeta) = FileMeta::load(dst_buf)
+                    {
+                        xlmeta = nmeta;
+                    } else {
+                        dst_meta_unparsable = true;
+                    }
+                }
+                let version_id = fi_moved.version_id.unwrap_or_default();
+                let has_old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id));
+                let rollback_data_dir = has_old_data_dir.or_else(|| {
+                    if has_dst_buf.is_some() && xlmeta.find_version(Some(version_id)).is_ok() {
+                        Some(inline_metadata_rollback_dir(version_id, &xlmeta))
+                    } else {
+                        None
+                    }
+                });
+                if rollback_data_dir.is_some() {
+                    return Ok(FusedCommitAttempt::Fallback(guard, has_dst_buf, fi_moved));
+                }
+                let (old_current_size, old_current_source_checked, old_current_source) = observe_old_current(
+                    has_dst_buf.is_some(),
+                    dst_meta_unparsable,
+                    &xlmeta,
+                    &dst_volume_str,
+                    &dst_path_str,
+                    version_id,
+                );
+                xlmeta.add_version(fi_moved)?;
+                let version_signature = rename_data_versions_signature(&xlmeta);
+                let new_dst_buf = xlmeta.marshal_msg().map_err(std::io::Error::other)?;
+                let destination_was_absent = has_dst_buf.is_none();
+                let mut skip_parent = dst_volume_dir_owned.clone();
+                if has_dst_buf.is_some() {
+                    skip_parent = dst_parent_owned.clone();
+                }
+
+                let mut prepared_metadata_source =
+                    os::create_prepared_rename_source_with_commit_guard(&src_file_path_owned, &dst_file_path_owned, &guard)?;
+                prepared_metadata_source.write_all(&new_dst_buf, sync_commit_metadata)?;
+
+                if no_inline {
+                    if sync_data_shards && let Some((src_data_path, _)) = has_data_dir_path_owned.as_ref() {
+                        let sync_result = if super::take_presynced_tmp_data_dir(src_data_path) {
+                            // The payload was synced, but its part.1 directory entry
+                            // must also be persisted before moving the data directory.
+                            os::fsync_dir_std(src_data_path)
+                        } else {
+                            os::sync_dir_files_std(src_data_path)
+                        };
+                        if let Err(err) = sync_result
+                            && err.kind() != ErrorKind::NotFound
+                        {
+                            return Err(err);
+                        }
+                    }
+                    if let Some((src_data_path, dst_data_path)) = has_data_dir_path_owned.as_ref()
+                        && let Err(err) = os::rename_all_with_commit_guard_on_thread(src_data_path, dst_data_path, &guard)
+                    {
+                        return Ok(FusedCommitAttempt::CommitErr(err));
+                    }
+                    #[cfg(test)]
+                    if has_data_dir_path_owned.is_some() {
+                        run_rename_data_after_first_publication(&root_owned, &dst_volume_str, &dst_path_str);
+                    }
+                }
+
+                if let Err(err) = os::rename_prepared_source_on_thread(
+                    prepared_metadata_source,
+                    &src_file_path_owned,
+                    &dst_file_path_owned,
+                    &skip_parent,
+                    &publication_root_owned,
+                    &guard,
+                ) {
+                    if no_inline && let Some((src_data_path, dst_data_path)) = has_data_dir_path_owned.as_ref() {
+                        let _ = std::fs::rename(dst_data_path, src_data_path);
+                    }
+                    return Ok(FusedCommitAttempt::CommitErr(err));
+                }
+
+                if sync_commit_metadata
+                    && let Err(err) =
+                        fsync_inline_commit_std(&dst_file_path_owned, &dst_volume_dir_owned, destination_was_absent, None, &guard)
+                {
+                    if no_inline && let Some((src_data_path, dst_data_path)) = has_data_dir_path_owned.as_ref() {
+                        let _ = std::fs::rename(dst_data_path, src_data_path);
+                    }
+                    return Err(err);
+                }
+
+                if let Some(parent) = src_file_path_owned.parent() {
+                    let _ = std::fs::remove_dir(parent);
+                }
+
+                Ok(FusedCommitAttempt::Committed(RenameDataResp {
+                    multipart_source: None,
+                    old_data_dir: None,
+                    rollback_data_dir: None,
+                    cleanup_data_dir: None,
+                    sign: version_signature,
+                    old_current_size,
+                    old_current_source_checked,
+                    old_current_source,
+                }))
+            };
+
+            let attempt_res = if let Some(admission) = file_sync_admission.as_ref() {
+                os::run_cancellable_blocking_namespace_file_sync_operation(mutation_lease.clone(), admission, fused_attempt).await
+            } else {
+                os::run_cancellable_blocking_namespace_operation(mutation_lease.clone(), fused_attempt).await
+            }
+            .map_err(to_file_error)
+            .map_err(DiskError::from);
+            drop(file_sync_admission);
+            match attempt_res {
+                Ok(FusedCommitAttempt::Committed(resp)) => {
+                    for part_path in &invalidate_part_paths {
+                        self.io_backend.invalidate_cached_fd(dst_volume, part_path).await;
+                    }
+                    #[cfg(unix)]
+                    if let Some(receipt_mutation) = receipt_mutation.as_mut() {
+                        receipt_mutation.complete();
+                    }
+                    return Ok(resp);
+                }
+                Ok(FusedCommitAttempt::CommitErr(err)) | Err(err) => {
+                    for part_path in &invalidate_part_paths {
+                        self.io_backend.invalidate_cached_fd(dst_volume, part_path).await;
+                    }
+                    return Err(err);
+                }
+                Ok(FusedCommitAttempt::Fallback(guard, dst_buf, restored_fi)) => {
+                    fi = restored_fi;
+                    (guard, dst_buf)
+                }
+            }
+        } else {
+            prepare_rename_commit_and_destination_metadata(
+                !no_inline && !preserve_upload_metadata,
+                src_file_parent,
+                dst_file_parent,
+                &dst_file_path,
+                &dst_volume_dir,
+                &self.publication_root,
+                mutation_lease.clone(),
+            )
+            .await?
+        };
+        #[cfg(not(target_os = "linux"))]
         // Acquire the common trees before reading destination metadata. On
         // Windows this pins the object directory identity across publication.
         // Multipart authority already created its source tree. Recreating it

crates/ecstore/src/disk/local/delete_recovery.rs

@@ -2090,6 +2090,474 @@ impl LocalDisk {
     pub(super) async fn delete_recovery_barrier(&self, volume: &str, object: &str, transaction: Uuid, stage: &str) -> Result<()> {
         barrier(volume, object, transaction, stage, &self.root).await
     }
+
+    pub(super) async fn try_fused_recoverable_delete_version(
+        &self,
+        volume: &str,
+        object: &str,
+        fi: &FileInfo,
+        opts: &DeleteOptions,
+        transaction: Uuid,
+        owner: Option<Arc<dyn Send + Sync>>,
+    ) -> Result<Option<()>> {
+        if !opts.recoverable_delete
+            || opts.undo_write
+            || opts.expected_delete_marker.is_some()
+            || pool_meta_native::is_object(&self.root, volume, object)
+            || has_delete_version_test_failpoints()
+        {
+            return Ok(None);
+        }
+        self.reject_multipart_delete_recovery_mutation(volume, object)?;
+        self.wait_for_startup_cleanup().await;
+
+        let (_, file_path) = self.io_get_bucket_and_object_path(volume, object)?;
+        check_path_length(file_path.to_string_lossy().as_ref())?;
+        let xl_path = path_join(&[file_path.as_path(), Path::new(STORAGE_FORMAT_FILE)]);
+        let destination = file_path.join(transaction.to_string());
+        let staging_root = self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?;
+        let sync = effective_durability(volume).syncs_commit_metadata();
+        let incarnation = opts.delete_recovery_bucket_incarnation;
+        let lease = os::acquire_namespace_mutation_lease_with_owner(&destination, owner).await;
+        let fi_clone = fi.clone();
+        let volume_str = volume.to_owned();
+        let object_str = object.to_owned();
+
+        let res = os::run_blocking_namespace_operation_with_sync(
+            lease,
+            sync,
+            move || -> std::io::Result<Result<Option<Option<Uuid>>>> {
+                let buf = match std::fs::read(&xl_path) {
+                    Ok(buf) => buf,
+                    Err(err) if err.kind() == ErrorKind::NotFound => return Ok(Ok(None)),
+                    Err(err) => return Err(err),
+                };
+                let (old_dir, journal_raw) = match (|| -> Result<Option<(Option<Uuid>, Vec<u8>)>> {
+                    let mut meta = FileMeta::load(&buf)?;
+                    let deleted_versions: Vec<_> = meta
+                        .versions
+                        .iter()
+                        .map(|v| v.header.version_id.unwrap_or_default())
+                        .collect();
+                    let original_versions: Vec<_> = meta
+                        .versions
+                        .iter()
+                        .map(|v| generation(v.into_fileinfo(&volume_str, &object_str, true)?))
+                        .collect::<Result<_>>()?;
+                    let old_dir = meta.delete_version(&fi_clone)?;
+                    if old_dir.is_some_and(|dir| dir == transaction || dir.is_nil()) {
+                        return Err(DiskError::FileCorrupt);
+                    }
+                    if let Some(uuid) = old_dir {
+                        meta.data.remove(vec![fi_clone.version_id.unwrap_or_default(), uuid])?;
+                        let old_path = path_join(&[file_path.as_path(), Path::new(uuid.to_string().as_str())]);
+                        check_path_length(old_path.to_string_lossy().as_ref())?;
+                        if old_path.join(STORAGE_FORMAT_FILE).exists() {
+                            return Ok(None);
+                        }
+                    }
+                    if !meta.versions.is_empty() {
+                        return Ok(None);
+                    }
+                    let mut predecessors = BTreeSet::new();
+                    if let Ok(entries) = std::fs::read_dir(&file_path) {
+                        for (visited, entry) in entries.enumerate() {
+                            if visited >= 65536 {
+                                return Err(DiskError::DiskOngoingReq);
+                            }
+                            let entry = entry.map_err(to_file_error)?;
+                            let Some(id) = entry
+                                .file_name()
+                                .to_str()
+                                .and_then(|name| Uuid::parse_str(name).ok())
+                                .filter(|id| !id.is_nil() && *id != transaction)
+                            else {
+                                continue;
+                            };
+                            if !entry.file_type().map_err(to_file_error)?.is_dir() {
+                                return Err(DiskError::InvalidPath);
+                            }
+                            let prepared = regular_bytes(&entry.path().join(PREPARED))?;
+                            let receipt = regular_bytes(&entry.path().join(DELETE_QUORUM_COMMITTED_FILE))?;
+                            let mut records = Vec::new();
+                            if let Some(bytes) = prepared {
+                                records.push(
+                                    serde_json::from_slice::<DeleteRecoveryJournal>(&bytes)
+                                        .map_err(|_| DiskError::FileCorrupt)?,
+                                );
+                            }
+                            if let Some(bytes) = receipt.filter(|bytes| !bytes.is_empty()) {
+                                records.push(
+                                    serde_json::from_slice::<CommitReceipt>(&bytes)
+                                        .map_err(|_| DiskError::FileCorrupt)?
+                                        .journal,
+                                );
+                            }
+                            for record in records {
+                                if record.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
+                                    || record.volume != volume_str
+                                    || record.object != object_str
+                                    || record.transaction != id
+                                    || incarnation.zip(record.bucket_incarnation).is_some_and(|(a, b)| a != b)
+                                    || record.predecessors.iter().any(|before| before.is_nil() || *before == id)
+                                {
+                                    return Err(DiskError::FileCorrupt);
+                                }
+                                predecessors.insert(id);
+                            }
+                        }
+                    }
+                    let mut sparse = fi_clone;
+                    sparse.data = None;
+                    let journal = DeleteRecoveryJournal {
+                        protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
+                        volume: volume_str,
+                        object: object_str,
+                        transaction,
+                        bucket_incarnation: incarnation,
+                        original_digest: Some(digest(&buf)),
+                        target_digest: None,
+                        data_dirs: old_dir.into_iter().collect(),
+                        deleted_versions,
+                        requests: vec![sparse],
+                        original_versions,
+                        predecessors: predecessors.into_iter().collect(),
+                    };
+                    Ok(Some((old_dir, journal_bytes(&journal)?)))
+                })() {
+                    Ok(Some(pair)) => pair,
+                    Ok(None) => return Ok(Ok(None)),
+                    Err(err) => return Ok(Err(err)),
+                };
+                std::fs::create_dir_all(&staging_root)?;
+                let staged = tempfile::Builder::new()
+                    .prefix("delete-preparation-")
+                    .tempdir_in(&staging_root)?;
+                for (name, bytes) in [
+                    (STORAGE_FORMAT_FILE_BACKUP, buf.as_slice()),
+                    (PREPARED, journal_raw.as_slice()),
+                ] {
+                    let mut file = std::fs::File::create(staged.path().join(name))?;
+                    std::io::Write::write_all(&mut file, bytes)?;
+                    if sync && !bytes.is_empty() {
+                        file.sync_data()?;
+                    }
+                }
+                if sync {
+                    os::fsync_dir_std(staged.path())?;
+                }
+                match std::fs::symlink_metadata(&destination) {
+                    Ok(metadata) if metadata.is_dir() => match std::fs::remove_dir(&destination) {
+                        Ok(()) => {}
+                        Err(err) if err.kind() == ErrorKind::NotFound => {}
+                        Err(err) if is_dir_not_empty_error(&err) => return Ok(Err(DiskError::OutdatedXLMeta)),
+                        Err(err) => return Err(err),
+                    },
+                    Ok(_) => return Ok(Err(DiskError::OutdatedXLMeta)),
+                    Err(err) if err.kind() == ErrorKind::NotFound => {}
+                    Err(err) => return Err(err),
+                }
+                if let Some(parent) = destination.parent() {
+                    std::fs::create_dir_all(parent)?;
+                }
+                std::fs::rename(staged.path(), &destination)?;
+                if sync {
+                    os::fsync_dir_std(&file_path)?;
+                }
+                let mut reserved_data_path = None;
+                if let Some(uuid) = old_dir {
+                    let old_path = file_path.join(uuid.to_string());
+                    if old_path.is_dir() {
+                        let reserved_marker = old_path.join(format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}"));
+                        let marker = std::fs::File::create(&reserved_marker)?;
+                        if sync {
+                            marker.sync_all()?;
+                            os::fsync_dir_std(&old_path)?;
+                        }
+                        reserved_data_path = Some((old_path, reserved_marker));
+                    }
+                }
+                if let Err(err) = std::fs::remove_file(&xl_path)
+                    && err.kind() != ErrorKind::NotFound
+                {
+                    if let Some((_, reserved_marker)) = &reserved_data_path {
+                        let _ = std::fs::remove_file(reserved_marker);
+                    }
+                    let _ = std::fs::remove_dir_all(&destination);
+                    return Err(err);
+                }
+                if sync {
+                    os::fsync_dir_std(&file_path)?;
+                }
+                if let Some((old_path, reserved_marker)) = reserved_data_path {
+                    let recovery_marker = old_path.join(format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}"));
+                    match std::fs::rename(&reserved_marker, &recovery_marker) {
+                        Ok(()) if sync => os::fsync_dir_std(&old_path)?,
+                        Ok(()) => {}
+                        Err(err) if err.kind() == ErrorKind::NotFound => {}
+                        Err(err) => return Err(err),
+                    }
+                }
+                Ok(Ok(Some(old_dir)))
+            },
+        )
+        .await
+        .map_err(to_file_error)??;
+
+        let Some(old_dir) = res else {
+            return Ok(None);
+        };
+        if let Some(uuid) = old_dir {
+            self.io_backend
+                .invalidate_cached_fds_under(volume, &format!("{object}/{uuid}"));
+        }
+        Ok(Some(()))
+    }
+
+    pub(crate) async fn try_publish_prepared_delete_local(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: Uuid,
+        namespace_owner: Option<Arc<dyn Send + Sync>>,
+    ) -> Result<Option<(DeleteRecoveryJournal, bool, bool)>> {
+        self.reject_multipart_delete_recovery_mutation(volume, object)?;
+        if !safe_path(object, false) || transaction.is_nil() {
+            return Err(DiskError::InvalidPath);
+        }
+        let object_path = self.get_object_path(volume, object)?;
+        let metadata_owner = os::acquire_metadata_mutation_lease(&object_path, namespace_owner).await;
+        let volume_guard = os::disk_volume_mutation_lock(&self.root, volume).read_owned().await;
+        let owner: Option<Arc<dyn Send + Sync>> = Some(Arc::new((metadata_owner, volume_guard)));
+
+        let root = self.io_get_object_path(volume, object)?;
+        let rollback = root.join(transaction.to_string());
+        let receipt_path = rollback.join(DELETE_QUORUM_COMMITTED_FILE);
+        let staging_root = self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?;
+        let sync = effective_durability(volume).syncs_commit_metadata();
+        let lease = os::acquire_namespace_mutation_lease_with_owner(&receipt_path, owner).await;
+        let volume_str = volume.to_owned();
+        let object_str = object.to_owned();
+
+        os::run_blocking_namespace_operation_with_sync(
+            lease,
+            sync,
+            move || -> std::io::Result<Result<Option<(DeleteRecoveryJournal, bool, bool)>>> {
+                let prepared = match regular_bytes(&rollback.join(PREPARED)) {
+                    Ok(Some(bytes)) => bytes,
+                    Ok(None) => return Ok(Ok(None)),
+                    Err(e) => return Ok(Err(e)),
+                };
+                let Ok(journal) = serde_json::from_slice::<DeleteRecoveryJournal>(&prepared) else {
+                    return Ok(Err(DiskError::FileCorrupt));
+                };
+                if journal.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
+                    || journal.volume != volume_str
+                    || journal.object != object_str
+                    || journal.transaction != transaction
+                    || journal.transaction.is_nil()
+                    || journal.data_dirs.iter().any(|id| id.is_nil() || *id == transaction)
+                {
+                    return Ok(Err(DiskError::FileCorrupt));
+                }
+                let current_exists = match regular_file_exists(&root.join(STORAGE_FORMAT_FILE)) {
+                    Ok(exists) => exists,
+                    Err(e) => return Ok(Err(e)),
+                };
+                let unversioned_target_absent = journal.target_digest.is_none() && !current_exists;
+                if !unversioned_target_absent {
+                    return Ok(Ok(None));
+                }
+                let mut sole_transaction = journal.predecessors.is_empty();
+                if sole_transaction {
+                    match std::fs::read_dir(&root) {
+                        Ok(entries) => {
+                            for entry in entries {
+                                let entry = entry?;
+                                let Ok(name) = entry.file_name().into_string() else {
+                                    sole_transaction = false;
+                                    break;
+                                };
+                                if let Some(id) = canonical_uuid(&name).filter(|id| !id.is_nil())
+                                    && id != transaction
+                                    && !journal.data_dirs.contains(&id)
+                                {
+                                    sole_transaction = false;
+                                    break;
+                                }
+                            }
+                        }
+                        Err(err) if err.kind() == ErrorKind::NotFound => {}
+                        Err(err) => return Err(err),
+                    }
+                }
+                let receipt = CommitReceipt {
+                    journal: journal.clone(),
+                    finished: false,
+                };
+                let receipt_bytes = serde_json::to_vec(&receipt).map_err(std::io::Error::other)?;
+                std::fs::create_dir_all(&staging_root)?;
+                let mut staged = tempfile::NamedTempFile::new_in(&staging_root)?;
+                std::io::Write::write_all(staged.as_file_mut(), &receipt_bytes)?;
+                if sync && !receipt_bytes.is_empty() {
+                    staged.as_file().sync_data()?;
+                }
+                std::fs::rename(staged.path(), &receipt_path)?;
+                if sync {
+                    os::fsync_dir_std(&rollback)?;
+                }
+                Ok(Ok(Some((journal, sole_transaction, unversioned_target_absent))))
+            },
+        )
+        .await
+        .map_err(to_file_error)?
+    }
+
+    pub(crate) async fn try_finish_and_finalize_prepared_delete_local(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: Uuid,
+        journal: &DeleteRecoveryJournal,
+        namespace_owner: Option<Arc<dyn Send + Sync>>,
+    ) -> Result<bool> {
+        fn ignore_not_found(res: std::io::Result<()>) -> std::io::Result<()> {
+            match res {
+                Ok(()) => Ok(()),
+                Err(err) if err.kind() == ErrorKind::NotFound => Ok(()),
+                Err(err) => Err(err),
+            }
+        }
+        self.reject_multipart_delete_recovery_mutation(volume, object)?;
+        if !safe_path(object, false) || transaction.is_nil() {
+            return Err(DiskError::InvalidPath);
+        }
+        let object_path = self.get_object_path(volume, object)?;
+        let metadata_owner = os::acquire_metadata_mutation_lease(&object_path, namespace_owner).await;
+        let volume_guard = os::disk_volume_mutation_lock(&self.root, volume).read_owned().await;
+        let owner: Option<Arc<dyn Send + Sync>> = Some(Arc::new((metadata_owner, volume_guard)));
+
+        let mut claims: Vec<SnapshotLeaseDeleteClaim> = Vec::with_capacity(journal.data_dirs.len());
+        for data_dir in &journal.data_dirs {
+            let key = SnapshotLeaseKey {
+                volume: volume.to_owned(),
+                path: format!("{object}/{data_dir}"),
+            };
+            let opts = DeleteOptions {
+                recursive: true,
+                old_data_dir: Some(transaction),
+                ..Default::default()
+            };
+            let Some(claim) = self.claim_snapshot_delete(key, SnapshotPendingDelete::Ordinary(opts)) else {
+                for prior in claims {
+                    prior.finish(false);
+                }
+                return Ok(false);
+            };
+            claims.push(claim);
+        }
+
+        let bucket_path = self.io_get_bucket_path(volume)?;
+        let root = self.io_get_object_path(volume, object)?;
+        let rollback = root.join(transaction.to_string());
+        let receipt_path = rollback.join(DELETE_QUORUM_COMMITTED_FILE);
+        let sync = effective_durability(volume).syncs_commit_metadata();
+        let lease = os::acquire_namespace_mutation_lease_with_owner(&receipt_path, owner).await;
+        let data_dirs = journal.data_dirs.clone();
+        let prune_paths: Vec<_> = rollback
+            .ancestors()
+            .take_while(|dir| *dir != bucket_path.as_path() && dir.starts_with(&bucket_path))
+            .map(Path::to_path_buf)
+            .collect();
+
+        let res = os::run_blocking_namespace_operation_with_sync(lease, sync, move || -> std::io::Result<()> {
+            let committed = format!("{DELETE_DATA_DIR_MARKER_PREFIX}{transaction}");
+            let recovery_committed = format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}");
+            let reserved = format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}");
+            for data_dir in &data_dirs {
+                for base in [&root, &rollback] {
+                    let dir_path = base.join(data_dir.to_string());
+                    match std::fs::read_dir(&dir_path) {
+                        Ok(entries) => {
+                            let mut removable = Vec::new();
+                            let mut preserve_dir = false;
+                            for entry in entries {
+                                let entry = entry?;
+                                let name = entry.file_name();
+                                let Some(name) = name.to_str() else {
+                                    preserve_dir = true;
+                                    continue;
+                                };
+                                if entry.file_type()?.is_file()
+                                    && (delete_recovery_part_file(name)
+                                        || name == committed
+                                        || name == recovery_committed
+                                        || name == reserved
+                                        || is_delete_transaction_marker(name, RESERVED_DELETE_DATA_DIR_MARKER_PREFIX))
+                                {
+                                    removable.push(entry.path());
+                                } else {
+                                    preserve_dir = true;
+                                }
+                            }
+                            for path in removable {
+                                ignore_not_found(std::fs::remove_file(path))?;
+                            }
+                            if sync {
+                                os::fsync_dir_std(&dir_path)?;
+                            }
+                            if !preserve_dir {
+                                match std::fs::remove_dir(&dir_path) {
+                                    Ok(()) => {}
+                                    Err(err) if err.kind() == ErrorKind::NotFound || is_dir_not_empty_error(&err) => {}
+                                    Err(err) => return Err(err),
+                                }
+                            }
+                        }
+                        Err(err) if err.kind() == ErrorKind::NotFound => {}
+                        Err(err) => return Err(err),
+                    }
+                }
+            }
+            if !data_dirs.is_empty() && sync && root.is_dir() {
+                os::fsync_dir_std(&root)?;
+            }
+            for name in [
+                STORAGE_FORMAT_FILE_BACKUP,
+                DELETE_MARKER_ROLLBACK_FILE,
+                TARGET,
+                PREPARED,
+                FINISHED,
+            ] {
+                ignore_not_found(std::fs::remove_file(rollback.join(name)))?;
+            }
+            if sync {
+                os::fsync_dir_std(&rollback)?;
+            }
+            ignore_not_found(std::fs::remove_file(&receipt_path))?;
+            for dir_path in &prune_paths {
+                if let Err(err) = std::fs::remove_dir(dir_path) {
+                    if is_dir_not_empty_error(&err) {
+                        break;
+                    }
+                    if !is_benign_object_rmdir_error(&err) {
+                        return Err(std::io::Error::other(FileAccessDeniedWithContext {
+                            path: dir_path.clone(),
+                            source: err,
+                        }));
+                    }
+                }
+            }
+            Ok(())
+        })
+        .await;
+
+        let ok = res.is_ok();
+        for claim in claims {
+            claim.finish(ok);
+        }
+        res.map(|()| true).map_err(|err| to_file_error(err).into())
+    }
 }
 
 #[cfg(not(feature = "e2e-test-hooks"))]
@@ -3309,6 +3777,171 @@ mod tests {
         inspected
     }
 
+    #[tokio::test]
+    async fn fused_delete_recovery_survives_interrupted_prepare_and_publication() {
+        for published in [false, true] {
+            let (_temp, disk, _info, original) = fixture().await;
+            let transaction = Uuid::new_v4();
+            let prepared = disk
+                .try_fused_recoverable_delete_version(
+                    "recovery",
+                    "object",
+                    &request(),
+                    &DeleteOptions {
+                        recoverable_delete: true,
+                        old_data_dir: Some(transaction),
+                        ..Default::default()
+                    },
+                    transaction,
+                    None,
+                )
+                .await
+                .expect("prepare fused deletion");
+            assert_eq!(prepared, Some(()), "the test must exercise the fused path");
+
+            if published {
+                let (_, sole, target_absent) = disk
+                    .try_publish_prepared_delete_local("recovery", "object", transaction, None)
+                    .await
+                    .expect("publish fused receipt")
+                    .expect("fused publication must apply");
+                assert!(sole && target_absent);
+            }
+
+            let mut state = disk
+                .inspect_delete_transaction("recovery", "object", transaction)
+                .await
+                .expect("inspect persisted state after interruption");
+            if published {
+                assert_eq!(state.state, DeleteRecoveryState::QuorumCommitted);
+                for action in [DeleteRecoveryAction::FinishCommit, DeleteRecoveryAction::FinalizeCommit] {
+                    state = disk
+                        .settle_delete_transaction("recovery", "object", transaction, state.fingerprint, action)
+                        .await
+                        .expect("finish published deletion through recovery");
+                }
+                assert_eq!(state.state, DeleteRecoveryState::Absent);
+                assert!(disk.read_all("recovery", "object/xl.meta").await.is_err());
+            } else {
+                assert_eq!(state.state, DeleteRecoveryState::NewPrepared);
+                state = disk
+                    .settle_delete_transaction(
+                        "recovery",
+                        "object",
+                        transaction,
+                        state.fingerprint,
+                        DeleteRecoveryAction::Rollback,
+                    )
+                    .await
+                    .expect("roll back uncommitted fused deletion");
+                assert_eq!(state.state, DeleteRecoveryState::Absent);
+                assert_eq!(
+                    disk.read_all("recovery", "object/xl.meta")
+                        .await
+                        .expect("restore original metadata")
+                        .as_ref(),
+                    original
+                );
+            }
+        }
+    }
+
+    #[tokio::test]
+    async fn fused_delete_recovery_preserves_or_removes_shard_after_interruption() {
+        use crate::erasure::coding::BitrotWriter;
+
+        for mode in 0..3 {
+            let published = mode != 0;
+            let (_temp, disk, _info, _) = fixture().await;
+            let data_dir = Uuid::new_v4();
+            let payload = Bytes::from_static(b"original shard");
+            let mut info = test_file_info("object", Uuid::nil(), Some(data_dir), None);
+            info.size = i64::try_from(payload.len()).expect("payload size");
+            info.erasure.block_size = payload.len();
+            info.parts[0].size = payload.len();
+            info.parts[0].actual_size = info.size;
+            let original = test_meta(info.clone());
+            let mut writer = BitrotWriter::new(
+                std::io::Cursor::new(Vec::new()),
+                info.erasure.shard_size(),
+                HashAlgorithm::HighwayHash256S,
+            );
+            writer.write(&payload).await.expect("encode shard");
+            writer.shutdown().await.expect("finish shard encoding");
+            let encoded = writer.into_inner().into_inner();
+            let root = disk.io_get_object_path("recovery", "object").expect("object path");
+            fs::write(root.join(STORAGE_FORMAT_FILE), &original)
+                .await
+                .expect("write shard metadata");
+            let shard_dir = root.join(data_dir.to_string());
+            fs::create_dir_all(&shard_dir).await.expect("create shard directory");
+            fs::write(shard_dir.join("part.1"), &encoded)
+                .await
+                .expect("write original shard");
+            let transaction = Uuid::new_v4();
+            assert_eq!(
+                disk.try_fused_recoverable_delete_version(
+                    "recovery",
+                    "object",
+                    &request(),
+                    &DeleteOptions {
+                        recoverable_delete: true,
+                        old_data_dir: Some(transaction),
+                        ..Default::default()
+                    },
+                    transaction,
+                    None,
+                )
+                .await
+                .expect("prepare fused shard deletion"),
+                Some(()),
+                "the test must exercise the fused path"
+            );
+            if published {
+                let (journal, sole, target_absent) = disk
+                    .try_publish_prepared_delete_local("recovery", "object", transaction, None)
+                    .await
+                    .expect("publish fused receipt")
+                    .expect("fused publication must apply");
+                assert!(sole && target_absent);
+                if mode == 2 {
+                    assert!(
+                        disk.try_finish_and_finalize_prepared_delete_local("recovery", "object", transaction, &journal, None)
+                            .await
+                            .expect("finish fused deletion"),
+                        "the fused finalization must apply"
+                    );
+                    assert!(!shard_dir.exists(), "fused finalization must remove the shard");
+                    assert!(!root.join(STORAGE_FORMAT_FILE).exists());
+                    assert!(!root.join(transaction.to_string()).exists());
+                    continue;
+                }
+            }
+            let mut state = disk
+                .inspect_delete_transaction("recovery", "object", transaction)
+                .await
+                .expect("inspect persisted state");
+            let actions = if published {
+                vec![DeleteRecoveryAction::FinishCommit, DeleteRecoveryAction::FinalizeCommit]
+            } else {
+                vec![DeleteRecoveryAction::Rollback]
+            };
+            for action in actions {
+                state = disk
+                    .settle_delete_transaction("recovery", "object", transaction, state.fingerprint, action)
+                    .await
+                    .expect("recover shard deletion");
+            }
+            assert_eq!(state.state, DeleteRecoveryState::Absent);
+            if published {
+                assert!(!shard_dir.exists(), "committed deletion must remove the shard");
+            } else {
+                assert_eq!(fs::read(root.join(STORAGE_FORMAT_FILE)).await.expect("restore metadata"), original);
+                assert_eq!(fs::read(shard_dir.join("part.1")).await.expect("restore shard"), encoded);
+            }
+        }
+    }
+
     #[tokio::test]
     async fn delete_recovery_uncommitted_inline_rolls_back_exact_bytes() {
         let (_temp, disk, _info, original) = fixture().await;

crates/ecstore/src/disk/mod.rs

@@ -387,6 +387,39 @@ impl Disk {
         }
     }
 
+    pub(crate) async fn try_publish_prepared_delete_local(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: Uuid,
+        namespace_owner: Option<Arc<dyn Send + Sync>>,
+    ) -> Result<Option<(DeleteRecoveryJournal, bool, bool)>> {
+        match self {
+            Self::Local(disk) => {
+                disk.try_publish_prepared_delete_local(volume, object, transaction, namespace_owner)
+                    .await
+            }
+            Self::Remote(_) => Ok(None),
+        }
+    }
+
+    pub(crate) async fn try_finish_and_finalize_prepared_delete_local(
+        &self,
+        volume: &str,
+        object: &str,
+        transaction: Uuid,
+        journal: &DeleteRecoveryJournal,
+        namespace_owner: Option<Arc<dyn Send + Sync>>,
+    ) -> Result<bool> {
+        match self {
+            Self::Local(disk) => {
+                disk.try_finish_and_finalize_prepared_delete_local(volume, object, transaction, journal, namespace_owner)
+                    .await
+            }
+            Self::Remote(_) => Ok(false),
+        }
+    }
+
     pub(crate) async fn delete_version_recoverable_with_namespace_owner(
         &self,
         volume: &str,

crates/ecstore/src/disk/os.rs

@@ -286,6 +286,11 @@ mod dir_fsync_failures {
 
     static FAILURES: LazyLock<Mutex<HashMap<PathBuf, io::ErrorKind>>> = LazyLock::new(|| Mutex::new(HashMap::new()));
 
+    #[cfg(target_os = "linux")]
+    pub(super) fn has_any() -> bool {
+        !FAILURES.lock().is_empty()
+    }
+
     pub(super) fn set(dir: &Path, kind: io::ErrorKind) {
         FAILURES.lock().insert(dir.to_path_buf(), kind);
     }
@@ -323,6 +328,11 @@ pub fn set_local_dir_fsync_failure(disk: &crate::disk::Disk, volume: &str, path:
     Ok(())
 }
 
+#[cfg(all(target_os = "linux", any(test, feature = "test-util")))]
+pub(crate) fn has_dir_fsync_test_failures() -> bool {
+    dir_fsync_failures::has_any()
+}
+
 /// Pause a real namespace mutation inside its physical executor.
 #[cfg(all(any(test, feature = "test-util"), not(windows)))]
 pub(crate) mod prepared_publication_test_hooks {
@@ -372,6 +382,17 @@ pub(crate) mod prepared_publication_test_hooks {
         }
     }
 
+    pub(crate) fn has_any_hooks() -> bool {
+        if !BEFORE_PUBLICATION.lock().is_empty() {
+            return true;
+        }
+        #[cfg(test)]
+        if !RENAME_DESTINATIONS.lock().is_empty() || !HARDLINK_DESTINATIONS.lock().is_empty() {
+            return true;
+        }
+        false
+    }
+
     #[cfg(test)]
     type RenameDestinationHook = Box<dyn FnOnce(&Path) + Send>;
     #[cfg(test)]
@@ -758,7 +779,7 @@ fn set_file_fdatasync_group_commit_wait_for_test(wait_micros: u64) {
     file_fdatasync_group_commit_override::set_wait_micros(wait_micros);
 }
 
-fn file_fdatasync_group_commit_enabled() -> bool {
+pub(crate) fn file_fdatasync_group_commit_enabled() -> bool {
     #[cfg(test)]
     if let Some(enabled) = file_fdatasync_group_commit_override::get() {
         return enabled;
@@ -2392,6 +2413,10 @@ pub(crate) mod file_sync_probe {
         configure(root, None, false)
     }
 
+    pub(crate) fn is_active() -> bool {
+        !ROOTS.read().expect("file sync probe lock poisoned").is_empty()
+    }
+
     pub(super) fn set_failing(root: &Path) -> ProbeGuard {
         configure(root, Some(1), false)
     }
@@ -2553,18 +2578,25 @@ fn regular_files(dir: &Path) -> io::Result<Vec<PathBuf>> {
 
 /// Fdatasync every regular file directly inside `dir`, then fsync the directory
 /// itself.
-#[allow(
-    dead_code,
-    reason = "reached only through sync_dir_files, whose callers are tests (backlog#1823)"
+#[cfg_attr(
+    not(target_os = "linux"),
+    allow(
+        dead_code,
+        reason = "reached only through sync_dir_files, whose callers are tests (backlog#1823)"
+    )
 )]
 pub fn sync_dir_files_std(dir: impl AsRef<Path>) -> io::Result<()> {
-    for entry in std::fs::read_dir(dir.as_ref())? {
-        let entry = entry?;
-        if entry.file_type()?.is_file() {
-            sync_file(&entry.path())?;
-        }
-    }
-    fsync_dir_std(dir)
+    let dir = dir.as_ref();
+    let files = regular_files(dir)?;
+    rustfs_io_metrics::record_put_rename_fdatasync_batch(rustfs_io_metrics::PUT_RENAME_FDATASYNC_BATCH_MODE_SERIAL, files.len());
+    sync_files(&files)?;
+    let fsync_started = rustfs_io_metrics::put_stage_timer();
+    let result = fsync_dir_std(dir);
+    rustfs_io_metrics::record_put_object_stage_duration_from(
+        rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_SRC_DIR_FSYNC,
+        fsync_started,
+    );
+    result
 }
 
 /// Async wrapper around [`sync_dir_files_std`]. Large directories flush files
@@ -3808,6 +3840,23 @@ pub(crate) async fn rename_all_with_prepared_source(
     Ok(())
 }
 
+#[cfg(target_os = "linux")]
+pub(crate) fn rename_all_with_commit_guard_on_thread(source: &Path, destination: &Path, guard: &RenameCommitGuard) -> Result<()> {
+    guard.directories.prepare_destination().map_err(to_file_error)?;
+    #[cfg(test)]
+    prepared_publication_test_hooks::run_rename_destination(source, destination);
+    #[cfg(any(test, feature = "test-util"))]
+    {
+        prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Rename, source);
+        prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Rename, destination);
+    }
+    #[cfg(test)]
+    multipart_source_syscall_failures::check(multipart_source_syscall_failures::Stage::Rename, destination)
+        .map_err(to_file_error)?;
+    guard.directories.rename(source, destination, None).map_err(to_file_error)?;
+    Ok(())
+}
+
 #[cfg(not(windows))]
 pub(crate) async fn rename_all_with_commit_guard(
     src_file_path: impl AsRef<Path>,
@@ -6483,6 +6532,7 @@ pub(crate) fn sync_multipart_owned_record_parents(
         if !directory.starts_with(source) {
             return Err(io::Error::new(io::ErrorKind::InvalidInput, "native record barrier escaped source"));
         }
+        #[cfg_attr(not(unix), allow(unused_mut))]
         let mut pinned = pin_multipart_source_directory(directory, base, publication_root)?;
         #[cfg(unix)]
         {

crates/ecstore/src/erasure/coding/decode.rs

@@ -175,8 +175,10 @@ const DEFAULT_RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE: bool = false;
 /// stripe via their stripe handle — only when a data shard is missing or dies.
 /// This halves per-GET read bytes, IOPS, and bitrot hashing on healthy
 /// objects with 2+2 layouts.
-/// Default: false (current behavior: every live shard reader is read on every
-/// stripe).
+/// Default: false for raw `Erasure::decode` calls without deferred parity
+/// handles; `DataBlocksFirst` reader setup attaches deferred parity handles and
+/// leaves those parity reserves unengaged by default unless this env var is set
+/// to `false`.
 const ENV_RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE: &str = "RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE";
 const DEFAULT_RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE: bool = false;
 
@@ -189,6 +191,13 @@ pub(crate) fn get_lockstep_data_shards_only_enabled() -> bool {
         )
 }
 
+/// Whether `DataBlocksFirst` reader setup and `with_deferred_parity_handles`
+/// should keep parity shards deferred until needed on healthy GETs.
+pub(crate) fn get_lockstep_data_shards_only_with_deferred_parity_enabled() -> bool {
+    matches!(decode_read_policy(), DecodeReadPolicy::DemandBound)
+        || rustfs_utils::get_env_bool(ENV_RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE, true)
+}
+
 /// Get whether bitrot-decode overlap is enabled.
 fn is_bitrot_decode_overlap_enabled() -> bool {
     rustfs_utils::get_env_bool(
@@ -832,6 +841,13 @@ where
     pub(crate) fn with_deferred_parity_handles(mut self, mut handles: Vec<Option<DeferredReaderStripeHandle>>) -> Self {
         handles.resize_with(self.readers.len(), || None);
         handles.truncate(self.readers.len());
+        if get_lockstep_data_shards_only_with_deferred_parity_enabled() {
+            for (index, handle) in handles.iter().enumerate().skip(self.data_shards) {
+                if handle.is_some() {
+                    self.engaged[index] = false;
+                }
+            }
+        }
         self.deferred_handles = handles;
         self
     }
@@ -2596,7 +2612,8 @@ impl Erasure {
         // the ordinary GET read quorum. Only the opt-in GET rollout requires a
         // surplus source; available surplus shards and integrity proofs are
         // still verified before any reconstructed copy bytes are emitted.
-        let require_surplus_source = reader.demand_bound_lockstep && matches!(decode_read_policy(), DecodeReadPolicy::Default);
+        let require_surplus_source =
+            get_lockstep_data_shards_only_enabled() && matches!(decode_read_policy(), DecodeReadPolicy::Default);
 
         let start = offset / self.block_size;
         let end = end_offset.saturating_sub(1) / self.block_size;
@@ -4367,9 +4384,9 @@ mod tests {
         .await;
     }
 
-    /// Compatibility lock: with the gate off (default), the lockstep path
-    /// keeps the pre-backlog#923 behavior and reads every live shard —
-    /// including parity — on every stripe.
+    /// Compatibility lock: with the gate off (default), raw `Erasure::decode`
+    /// without deferred parity handles keeps the pre-backlog#923 behavior and
+    /// reads every live shard — including parity — on every stripe.
     #[tokio::test]
     #[serial_test::serial]
     async fn test_lockstep_default_reads_all_shards_per_stripe() {

crates/ecstore/src/object_api/types.rs

@@ -144,6 +144,8 @@ pub(crate) fn namespace_lock_signal_test_fence_is_lost(signal: &Arc<rustfs_lock:
         .is_some_and(|(_, fence)| fence.is_lock_lost())
 }
 
+type FreshPutQuota = std::result::Result<(Option<crate::bucket::quota::BucketQuota>, Uuid, OffsetDateTime), String>;
+
 #[derive(Debug)]
 pub struct ObjectLockConfigSnapshot {
     store_id: Option<Uuid>,
@@ -154,6 +156,7 @@ pub struct ObjectLockConfigSnapshot {
     lifecycle_fence: NamespaceLockFence,
     _lifecycle_guard: Option<rustfs_lock::NamespaceLockGuard>,
     metadata_transaction_guard: Option<Arc<rustfs_lock::NamespaceLockGuard>>,
+    fresh_put_quota: std::sync::Mutex<Option<FreshPutQuota>>,
 }
 
 impl ObjectLockConfigSnapshot {
@@ -167,6 +170,7 @@ impl ObjectLockConfigSnapshot {
             lifecycle_fence: NamespaceLockFence::new(),
             _lifecycle_guard: None,
             metadata_transaction_guard: None,
+            fresh_put_quota: std::sync::Mutex::new(None),
         }
     }
 
@@ -187,6 +191,7 @@ impl ObjectLockConfigSnapshot {
             lifecycle_fence: NamespaceLockFence::new(),
             _lifecycle_guard: None,
             metadata_transaction_guard: None,
+            fresh_put_quota: std::sync::Mutex::new(None),
         }
     }
 
@@ -212,6 +217,7 @@ impl ObjectLockConfigSnapshot {
             lifecycle_fence,
             _lifecycle_guard: Some(lifecycle_guard),
             metadata_transaction_guard: Some(Arc::new(metadata_transaction_guard)),
+            fresh_put_quota: std::sync::Mutex::new(None),
         }
     }
 
@@ -233,9 +239,23 @@ impl ObjectLockConfigSnapshot {
             lifecycle_fence,
             _lifecycle_guard: None,
             metadata_transaction_guard: Some(metadata_transaction_guard),
+            fresh_put_quota: std::sync::Mutex::new(None),
         }
     }
 
+    pub(crate) fn set_fresh_put_quota(&self, quota: FreshPutQuota) {
+        if self._lifecycle_guard.is_some() && self.metadata_transaction_guard.is_some() {
+            *self.fresh_put_quota.lock().unwrap_or_else(std::sync::PoisonError::into_inner) = Some(quota);
+        }
+    }
+
+    pub(crate) fn take_fresh_put_quota(&self) -> Option<FreshPutQuota> {
+        self.fresh_put_quota
+            .lock()
+            .unwrap_or_else(std::sync::PoisonError::into_inner)
+            .take()
+    }
+
     #[allow(dead_code, reason = "snapshot-scope predicate asserted by this file's tests (backlog#1823)")]
     pub(crate) fn is_for_store_bucket(
         &self,

crates/ecstore/src/set_disk/core/io_primitives.rs

@@ -1288,7 +1288,7 @@ async fn try_create_bitrot_readers_via_batch_pread(
     // parity readers unopened; preading them here would discard their bytes
     // when fill_deferred_bitrot_readers installs stripe-aligned reserves.
     let data_first = strategy != BitrotReaderSetupStrategy::AllShards
-        && crate::erasure::coding::decode::get_lockstep_data_shards_only_enabled();
+        && crate::erasure::coding::decode::get_lockstep_data_shards_only_with_deferred_parity_enabled();
     let mut setup = BitrotReaderSetup::new(disks.len());
     for (idx, disk) in disks.iter().enumerate() {
         if disk.is_none() {
@@ -2549,6 +2549,8 @@ pub(in crate::set_disk) struct RenameTailCleanup {
 pub(in crate::set_disk) struct RenameTailOutcome {
     pub(in crate::set_disk) convergence: RenameConvergence,
     pub(in crate::set_disk) cleanup: Vec<RenameTailCleanup>,
+    pub(in crate::set_disk) old_current_source_all_successful_checked: bool,
+    pub(in crate::set_disk) all_online_local_succeeded: bool,
 }
 
 const RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP_ENV: &str = "RUSTFS_PUT_RENAME_TAIL_CLEANUP_COUNTERFACTUAL_SKIP";
@@ -3572,7 +3574,7 @@ pub(in crate::set_disk) async fn finish_rename_tail_heal<
     Guards: Send + 'static,
     Finalize: FnOnce() -> FinalizeFuture + Send,
     FinalizeFuture: Future<Output = ()> + Send,
-    Cleanup: FnOnce(Guards, Vec<RenameTailCleanup>) -> CleanupFuture + Send,
+    Cleanup: FnOnce(Guards, Vec<RenameTailCleanup>, bool, bool) -> CleanupFuture + Send,
     CleanupFuture: Future<Output = ()> + Send,
     Cleanup: 'static,
     CleanupFuture: 'static,
@@ -3580,34 +3582,41 @@ pub(in crate::set_disk) async fn finish_rename_tail_heal<
     SubmitFuture: Future<Output = ()> + Send,
 {
     let tail_async_started = rustfs_io_metrics::put_stage_timer();
-    let (needs_heal, tail_cleanup, tail_complete) = match tail_drain.await {
-        Ok(Some(outcome)) => (outcome.convergence.needs_heal(), outcome.cleanup, true),
-        Ok(None) => {
-            warn!(
-                event = EVENT_SET_DISK_RENAME_TAIL_DRAIN_FAILED,
-                component = LOG_COMPONENT_ECSTORE,
-                subsystem = LOG_SUBSYSTEM_SET_DISK,
-                state = "missing_convergence",
-                bucket = %request.bucket,
-                object = request.object_prefix.as_deref().unwrap_or_default(),
-                "committed rename tail drain completed without convergence"
-            );
-            (true, Vec::new(), false)
-        }
-        Err(err) => {
-            warn!(
-                event = EVENT_SET_DISK_RENAME_TAIL_DRAIN_FAILED,
-                component = LOG_COMPONENT_ECSTORE,
-                subsystem = LOG_SUBSYSTEM_SET_DISK,
-                state = "failed",
-                bucket = %request.bucket,
-                object = request.object_prefix.as_deref().unwrap_or_default(),
-                error = %err,
-                "rename tail drain failed"
-            );
-            (true, Vec::new(), false)
-        }
-    };
+    let (needs_heal, tail_cleanup, tail_complete, old_current_source_all_successful_checked, all_online_local_succeeded) =
+        match tail_drain.await {
+            Ok(Some(outcome)) => (
+                outcome.convergence.needs_heal(),
+                outcome.cleanup,
+                true,
+                outcome.old_current_source_all_successful_checked,
+                outcome.all_online_local_succeeded,
+            ),
+            Ok(None) => {
+                warn!(
+                    event = EVENT_SET_DISK_RENAME_TAIL_DRAIN_FAILED,
+                    component = LOG_COMPONENT_ECSTORE,
+                    subsystem = LOG_SUBSYSTEM_SET_DISK,
+                    state = "missing_convergence",
+                    bucket = %request.bucket,
+                    object = request.object_prefix.as_deref().unwrap_or_default(),
+                    "committed rename tail drain completed without convergence"
+                );
+                (true, Vec::new(), false, false, false)
+            }
+            Err(err) => {
+                warn!(
+                    event = EVENT_SET_DISK_RENAME_TAIL_DRAIN_FAILED,
+                    component = LOG_COMPONENT_ECSTORE,
+                    subsystem = LOG_SUBSYSTEM_SET_DISK,
+                    state = "failed",
+                    bucket = %request.bucket,
+                    object = request.object_prefix.as_deref().unwrap_or_default(),
+                    error = %err,
+                    "rename tail drain failed"
+                );
+                (true, Vec::new(), false, false, false)
+            }
+        };
     rustfs_io_metrics::record_put_object_stage_duration_from(
         rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_ASYNC_WAIT,
         tail_async_started,
@@ -3651,7 +3660,12 @@ pub(in crate::set_disk) async fn finish_rename_tail_heal<
             let job = RenameTailCleanupJob {
                 enqueued_at: Instant::now(),
                 targets,
-                future: Box::pin(cleanup(guards, tail_cleanup)),
+                future: Box::pin(cleanup(
+                    guards,
+                    tail_cleanup,
+                    old_current_source_all_successful_checked,
+                    all_online_local_succeeded,
+                )),
             };
             if let Err(job) = rename_tail_cleanup_queue().try_enqueue(job).await {
                 rustfs_io_metrics::record_put_object_stage_duration_from(
@@ -3676,7 +3690,13 @@ pub(in crate::set_disk) async fn finish_rename_tail_heal<
                 cleanup_submit_started,
             );
             let cleanup_started = rustfs_io_metrics::put_stage_timer();
-            cleanup(guards, tail_cleanup).await;
+            cleanup(
+                guards,
+                tail_cleanup,
+                old_current_source_all_successful_checked,
+                all_online_local_succeeded,
+            )
+            .await;
             rustfs_io_metrics::record_put_object_stage_duration_from(
                 rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP,
                 cleanup_started,
@@ -4155,7 +4175,14 @@ impl SetDisks {
                         }
                     }
 
-                    if !sent_commit && success_count >= write_quorum {
+                    let pending_tasks = tasks.len();
+                    let checked_count = old_current_source_checked.iter().filter(|checked| **checked).count();
+                    let max_source_votes = Self::max_common_old_current_source_votes(&old_current_sources);
+                    let can_change_old_current_source = (checked_count < write_quorum
+                        && checked_count + pending_tasks >= write_quorum)
+                        || (max_source_votes < write_quorum && max_source_votes + pending_tasks >= write_quorum);
+
+                    if !sent_commit && success_count >= write_quorum && !can_change_old_current_source {
                         let snapshot_commit = Self::rename_data_commit_from_observations(
                             RenameDataObservations {
                                 disks: &coordinator_disks,
@@ -4174,6 +4201,8 @@ impl SetDisks {
                             cleanup_at_snapshot = commit.cleanup_disks.iter().map(Option::is_some).collect();
                         }
                         if let Some(commit_tx) = commit_tx.take() {
+                            #[cfg(test)]
+                            rename_fanout_barrier::wait_for_armed_rename_before_early_ack(&fanout_dst_object).await;
                             let _ = commit_tx.send(snapshot_commit);
                         }
                         sent_commit = true;
@@ -4296,9 +4325,21 @@ impl SetDisks {
                     }
                 }
 
+                let old_current_source_all_successful_checked = errs
+                    .iter()
+                    .enumerate()
+                    .filter(|(_, err)| err.is_none())
+                    .all(|(idx, _)| old_current_source_checked.get(idx).copied().unwrap_or(false));
+                let all_online_local_succeeded = coordinator_disks.iter().zip(errs.iter()).all(|(disk, err)| match disk {
+                    Some(d) => d.is_local() && err.is_none(),
+                    None => true,
+                });
+
                 Some(RenameTailOutcome {
                     convergence: final_convergence,
                     cleanup: tail_cleanup,
+                    old_current_source_all_successful_checked,
+                    all_online_local_succeeded,
                 })
             }
         });
@@ -4863,6 +4904,14 @@ impl SetDisks {
         })
     }
 
+    pub(in crate::set_disk) fn max_common_old_current_source_votes(old_current_sources: &[Option<FileInfo>]) -> usize {
+        let mut counts: HashMap<[u8; 32], usize> = HashMap::new();
+        for source in old_current_sources.iter().flatten() {
+            *counts.entry(Self::file_info_quorum_hash(source)).or_insert(0) += 1;
+        }
+        counts.into_values().max().unwrap_or(0)
+    }
+
     pub(in crate::set_disk) fn reduce_common_old_current_source(
         old_current_sources: &[Option<FileInfo>],
         write_quorum: usize,
@@ -7422,6 +7471,34 @@ pub(crate) mod rename_fanout_barrier {
         }
     }
 
+    pub(in crate::set_disk) async fn wait_for_armed_rename_before_early_ack(object: &str) {
+        loop {
+            let (any_armed, any_unpaused) = {
+                let reg = lock();
+                let mut any_armed = false;
+                let mut any_unpaused = false;
+                for ((obj, _, phase), armed) in &reg.armed {
+                    if obj == object && *phase == Phase::Rename {
+                        any_armed = true;
+                        if !armed.paused.load(Ordering::SeqCst) {
+                            any_unpaused = true;
+                            break;
+                        }
+                    }
+                }
+                (any_armed, any_unpaused)
+            };
+            if !any_armed {
+                return;
+            }
+            if !any_unpaused {
+                tokio::task::yield_now().await;
+                return;
+            }
+            tokio::task::yield_now().await;
+        }
+    }
+
     /// RAII scope that observes in-flight fan-out task counts for a single
     /// `object`. Counts accrue only while the scope is alive; it clears its own
     /// entry on drop.
@@ -8488,9 +8565,9 @@ mod tests {
         }
         assert_eq!(
             recorder.counter_value("rustfs_io_get_object_local_batch_shards_total", &[("outcome", "success")]),
-            4
+            2
         );
-        assert_eq!(recorder.counter_value("rustfs_io_get_object_local_batch_read_bytes_total", &[]), 14);
+        assert_eq!(recorder.counter_value("rustfs_io_get_object_local_batch_read_bytes_total", &[]), 7);
         assert_eq!(recorder.histogram_sample_count("rustfs_io_get_object_local_batch_prepare_seconds"), 8);
     }
 
@@ -10915,7 +10992,7 @@ mod tests {
             move || async move {
                 *finalize_captured.lock().expect("finalize recorder should not poison") = true;
             },
-            move |(), _| async move {
+            move |(), _, _, _| async move {
                 *cleanup_captured.lock().expect("cleanup recorder should not poison") = true;
             },
             |request| async move {

crates/ecstore/src/set_disk/ops/delete_recovery.rs

@@ -97,6 +97,14 @@ pub(crate) mod test_hooks {
             .push((bucket.to_owned(), object.to_owned(), hook));
     }
 
+    pub(super) fn has_before_gc_finalize(bucket: &str, object: &str) -> bool {
+        BEFORE_GC_FINALIZE
+            .lock()
+            .expect("GC finalize hooks")
+            .iter()
+            .any(|(hook_bucket, hook_object, _)| hook_bucket == bucket && hook_object == object)
+    }
+
     pub(super) async fn run_before_gc_finalize(bucket: &str, object: &str) {
         let hook = {
             let mut hooks = BEFORE_GC_FINALIZE.lock().expect("GC finalize hooks");
@@ -418,6 +426,59 @@ impl SetDisks {
             .unwrap_or_else(|| self.ctx.begin_namespace_commit());
         context.namespace_owner = Some(Arc::new((owner, Arc::clone(&transaction_guard))));
         let can_commit = context.can_commit;
+        let can_try_local_fast_path = disks.len() == self.set_drive_count
+            && disks.iter().all(|d| d.as_ref().is_some_and(|d| d.is_local()))
+            && !crate::disk::local::has_delete_version_test_failpoints()
+            && {
+                #[cfg(test)]
+                {
+                    !test_hooks::has_before_gc_finalize(bucket, object)
+                }
+                #[cfg(not(test))]
+                {
+                    true
+                }
+            };
+        if can_try_local_fast_path && can_commit() {
+            let fast_published = join_all(disks.iter().map(|disk| async {
+                if !can_commit() {
+                    return Err(DiskError::other("delete recovery namespace fence was lost"));
+                }
+                let disk = disk.as_ref().ok_or(DiskError::DiskNotFound)?;
+                disk.try_publish_prepared_delete_local(bucket, object, transaction, context.namespace_owner.clone())
+                    .await
+            }))
+            .await;
+            let all_fast_published = fast_published.len() == self.set_drive_count
+                && fast_published.iter().all(|res| matches!(res, Ok(Some((_, true, true)))));
+            if all_fast_published && can_commit() {
+                #[cfg(test)]
+                test_hooks::record_cleanup_attempt(bucket, object);
+                let finalized = join_all(disks.iter().zip(fast_published.iter()).map(|(disk, pub_res)| {
+                    let ns_owner = context.namespace_owner.clone();
+                    async move {
+                        if !can_commit() {
+                            return Ok(false);
+                        }
+                        let Some(disk) = disk.as_ref() else {
+                            return Ok(false);
+                        };
+                        let Ok(Some((journal, _, _))) = pub_res else {
+                            return Ok(false);
+                        };
+                        disk.try_finish_and_finalize_prepared_delete_local(bucket, object, transaction, journal, ns_owner)
+                            .await
+                    }
+                }))
+                .await;
+                if finalized.iter().all(|res| matches!(res, Ok(true))) {
+                    if let Some(ticket) = context.recovery_ticket.as_ref() {
+                        self.ctx.delete_recovery_hints().ack(ticket);
+                    }
+                    return Ok(());
+                }
+            }
+        }
         let states = inspect(disks, bucket, object, transaction).await;
         let untouched_absence = |state: &Result<DeleteRecoveryInspection>| matches!(state, Ok(state) if state.state == DeleteRecoveryState::Absent && state.current_digest.is_none());
         if states.iter().filter(|state| untouched_absence(state)).count() >= write_quorum

crates/ecstore/src/set_disk/ops/object.rs

@@ -158,13 +158,15 @@ fn rename_tail_cleanup_zero_target_tmp_delete_skip_enabled() -> bool {
 }
 
 fn tier_free_version_source_reuse_rename_data_enabled() -> bool {
-    static ENABLED: std::sync::OnceLock<bool> = std::sync::OnceLock::new();
-    *ENABLED.get_or_init(|| {
-        matches!(
-            std::env::var(TIER_FREE_VERSION_SOURCE_REUSE_RENAME_DATA_ENV).ok().as_deref(),
-            Some("1" | "true" | "TRUE" | "yes" | "YES" | "on" | "ON")
-        )
-    })
+    #[cfg(test)]
+    {
+        rustfs_utils::get_env_bool(TIER_FREE_VERSION_SOURCE_REUSE_RENAME_DATA_ENV, true)
+    }
+    #[cfg(not(test))]
+    {
+        static ENABLED: std::sync::OnceLock<bool> = std::sync::OnceLock::new();
+        *ENABLED.get_or_init(|| rustfs_utils::get_env_bool(TIER_FREE_VERSION_SOURCE_REUSE_RENAME_DATA_ENV, true))
+    }
 }
 
 fn tier_free_version_source_reuse_rename_data_assume_homogeneous_enabled() -> bool {
@@ -5523,8 +5525,10 @@ impl SetDisks {
                 );
 
                 let tier_free_version_source_lookup_started = rustfs_io_metrics::put_stage_timer();
+                let all_commit_disks_local = commit_disks.iter().flatten().all(|disk| disk.is_local());
                 let reuse_rename_data_source_for_tier_free_version = tier_free_version_source_reuse_rename_data_enabled()
-                    && (commit_set.tier_free_version_rename_data_source_capable()
+                    && (all_commit_disks_local
+                        || commit_set.tier_free_version_rename_data_source_capable()
                         || tier_free_version_source_reuse_rename_data_assume_homogeneous_enabled());
                 let mut put_tier_free_version_source = if commit_put_tier_free_version_id.is_some()
                     && commit_tier_free_version_receipt_sink.is_some()
@@ -5630,9 +5634,11 @@ impl SetDisks {
                 );
                 let lock_held_tail_handoff_started = rustfs_io_metrics::put_stage_timer();
                 if let Ok(rename_commit) = rename_result.as_mut() {
-                    commit_set.observe_tier_free_version_rename_data_source_capability(
-                        rename_commit.tail_drain.is_none() && rename_commit.old_current_source_all_successful_checked,
-                    );
+                    if rename_commit.tail_drain.is_none() || !rename_commit.old_current_source_all_successful_checked {
+                        commit_set.observe_tier_free_version_rename_data_source_capability(
+                            rename_commit.tail_drain.is_none() && rename_commit.old_current_source_all_successful_checked,
+                        );
+                    }
                     commit_set.record_capacity_scope_if_needed(commit_capacity_scope_token, &rename_commit.capacity_disks);
                     // Install the tail watcher before any post-commit await. The
                     // latch keeps namespace guards through their prior handoff point.
@@ -5700,7 +5706,12 @@ impl SetDisks {
                                 decommission_object_lock_guard,
                                 decommission_capacity_guard,
                             ),
-                                  targets| {
+                                  targets,
+                                  old_current_source_all_successful_checked,
+                                  all_online_local_succeeded| {
+                                cleanup_set.observe_tier_free_version_rename_data_source_capability(
+                                    old_current_source_all_successful_checked,
+                                );
                                 drop(object_lock_guard);
                                 drop(write_context);
                                 drop(publication_guard);
@@ -5727,7 +5738,10 @@ impl SetDisks {
                                         )
                                         .await;
                                     let tmp_delete_started = rustfs_io_metrics::put_stage_timer();
-                                    if zero_target_cleanup && rename_tail_cleanup_zero_target_tmp_delete_skip_enabled() {
+                                    if zero_target_cleanup
+                                        && (all_online_local_succeeded
+                                            || rename_tail_cleanup_zero_target_tmp_delete_skip_enabled())
+                                    {
                                         rustfs_io_metrics::record_put_object_stage_duration(
                                             rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_TAIL_CLEANUP_TMP_DELETE_COUNTERFACTUAL_SKIP,
                                             1.0,

crates/ecstore/src/store/mod.rs

@@ -1388,6 +1388,18 @@ impl crate::storage_api_contracts::object::ObjectIO for ECStore {
 }
 
 impl ECStore {
+    pub(crate) fn all_disks_local_and_online(&self) -> bool {
+        !self.disk_map.is_empty()
+            && self.disk_map.values().all(|disks| {
+                !disks.is_empty()
+                    && disks.iter().all(|disk| {
+                        disk.as_ref().is_some_and(|d| {
+                            d.is_local() && d.runtime_state() == crate::disk::health_state::RuntimeDriveHealthState::Online
+                        })
+                    })
+            })
+    }
+
     /// `put_object` plus the rename_data old-size backfill
     /// (rustfs/backlog#1009); see `SetDisks::put_object_with_old_current_size`.
     /// Post-write hooks (immediate ILM transition enqueue, list-cache
@@ -1400,19 +1412,36 @@ impl ECStore {
         data: &mut PutObjReader,
         opts: &ObjectOptions,
     ) -> Result<(ObjectInfo, Option<crate::disk::OldCurrentSize>)> {
-        let result = match self.handle_put_object(bucket, object, data, opts).await {
-            Ok((object_info, old_current_size)) => {
-                enqueue_transition_after_write(self, Ok(object_info), LcEventSrc::S3PutObject, opts)
-                    .await
-                    .map(|object_info| (object_info, old_current_size))
-            }
-            Err(err) => Err(err),
-        };
+        let result = self.handle_put_object(bucket, object, data, opts).await;
         if result.is_ok() {
             list_objects::observe_scanner_namespace_mutations(bucket, 1);
         }
         result
     }
+
+    #[allow(clippy::too_many_arguments)]
+    pub async fn copy_object_with_old_current_size(
+        &self,
+        src_bucket: &str,
+        src_object: &str,
+        dst_bucket: &str,
+        dst_object: &str,
+        src_info: &mut ObjectInfo,
+        src_opts: &ObjectOptions,
+        dst_opts: &ObjectOptions,
+    ) -> Result<(ObjectInfo, Option<crate::disk::OldCurrentSize>)> {
+        let (oi, old_current_size) = box_copy_object_handler(move || {
+            self.handle_copy_object(src_bucket, src_object, dst_bucket, dst_object, src_info, src_opts, dst_opts)
+        })
+        .await?;
+        let result = enqueue_transition_after_write(self, Ok(oi), LcEventSrc::S3CopyObject, dst_opts)
+            .await
+            .map(|oi| (oi, old_current_size));
+        if result.is_ok() {
+            list_objects::observe_scanner_namespace_mutations(dst_bucket, 1);
+        }
+        result
+    }
 }
 
 lazy_static! {
@@ -1490,20 +1519,9 @@ impl crate::storage_api_contracts::object::ObjectOperations for ECStore {
         src_opts: &ObjectOptions,
         dst_opts: &ObjectOptions,
     ) -> Result<ObjectInfo> {
-        let result = enqueue_transition_after_write(
-            self,
-            box_copy_object_handler(move || {
-                self.handle_copy_object(src_bucket, src_object, dst_bucket, dst_object, src_info, src_opts, dst_opts)
-            })
-            .await,
-            LcEventSrc::S3CopyObject,
-            dst_opts,
-        )
-        .await;
-        if result.is_ok() {
-            list_objects::observe_scanner_namespace_mutations(dst_bucket, 1);
-        }
-        result
+        self.copy_object_with_old_current_size(src_bucket, src_object, dst_bucket, dst_object, src_info, src_opts, dst_opts)
+            .await
+            .map(|(oi, _)| oi)
     }
 
     #[instrument(skip(self))]
@@ -1763,15 +1781,10 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for ECStore {
         uploaded_parts: Vec<CompletePart>,
         opts: &ObjectOptions,
     ) -> Result<ObjectInfo> {
-        let result = enqueue_transition_after_write(
-            self.as_ref(),
-            self.clone()
-                .handle_complete_multipart_upload(bucket, object, upload_id, uploaded_parts, opts)
-                .await,
-            LcEventSrc::S3CompleteMultipartUpload,
-            opts,
-        )
-        .await;
+        let result = self
+            .clone()
+            .handle_complete_multipart_upload(bucket, object, upload_id, uploaded_parts, opts)
+            .await;
         if result.is_ok() {
             list_objects::observe_scanner_namespace_mutations(bucket, 1);
         }

crates/ecstore/src/store/multipart.rs

@@ -640,6 +640,19 @@ impl ECStore {
             return Ok((res, idx, opts.expected_bucket_incarnation_id));
         }
 
+        if self.single_pool() && !self.is_suspended(0).await && !self.is_pool_rebalancing(0).await {
+            self.apply_decommission_target_mutation_fence(0, object, &mut opts, mutation_fence)
+                .await;
+            let expected_bucket_incarnation_id = opts.expected_bucket_incarnation_id;
+            let lock_object = encode_dir_object(object);
+            let res = self
+                .run_external_decommission_capacity_object_mutation(0, bucket, &lock_object, object, opts, |opts| async move {
+                    self.pools[0].new_multipart_upload(bucket, object, &opts).await
+                })
+                .await?;
+            return Ok((res, 0, expected_bucket_incarnation_id));
+        }
+
         for (idx, pool) in self.pools.iter().enumerate() {
             if self.is_suspended(idx).await || self.is_pool_rebalancing(idx).await {
                 continue;
@@ -1167,10 +1180,11 @@ impl ECStore {
         let opts = &opts;
 
         if self.single_pool() {
-            return self.pools[0]
+            let result = self.pools[0]
                 .clone()
                 .complete_multipart_upload(bucket, object, upload_id, uploaded_parts, opts)
                 .await;
+            return enqueue_transition_after_write(self.as_ref(), result, LcEventSrc::S3CompleteMultipartUpload, opts).await;
         }
         if opts.http_preconditions.is_some()
             && !opts.data_movement
@@ -1179,7 +1193,7 @@ impl ECStore {
             return Err(err);
         }
 
-        match self.multipart_upload_pool_idx(bucket, object, upload_id, opts).await {
+        let result = match self.multipart_upload_pool_idx(bucket, object, upload_id, opts).await {
             Ok(pool_idx) => {
                 self.pools[pool_idx]
                     .clone()
@@ -1190,21 +1204,29 @@ impl ECStore {
             // the pool that published it; other pools answer InvalidUploadID.
             Err(err) if is_err_invalid_upload_id(&err) => {
                 let mut missing = err;
+                let mut found = None;
                 for pool_idx in self.existing_multipart_pool_order().await {
                     match self.pools[pool_idx]
                         .clone()
                         .complete_multipart_upload(bucket, object, upload_id, uploaded_parts.clone(), opts)
                         .await
                     {
-                        Ok(info) => return Ok(info),
+                        Ok(info) => {
+                            found = Some(Ok(info));
+                            break;
+                        }
                         Err(err) if is_err_invalid_upload_id(&err) => missing = err,
-                        Err(err) => return Err(err),
+                        Err(err) => {
+                            found = Some(Err(err));
+                            break;
+                        }
                     }
                 }
-                Err(missing)
+                found.unwrap_or(Err(missing))
             }
             Err(err) => Err(err),
-        }
+        };
+        enqueue_transition_after_write(self.as_ref(), result, LcEventSrc::S3CompleteMultipartUpload, opts).await
     }
 
     #[cfg(all(test, feature = "test-util"))]
@@ -1326,6 +1348,7 @@ impl ECStore {
         // Data movement already owns the pool-meta write lease. Forward its
         // loss signal into SetDisks so commit fencing observes the same lease
         // without trying to reacquire the namespace.
+        let mut transition_opts = opts.clone();
         let result = self
             .run_decommission_capacity_admitted_mutation_with_capacity_lease(
                 target_pool_idx,
@@ -1341,8 +1364,9 @@ impl ECStore {
             )
             .await;
         drop(publication_guard);
+        transition_opts.no_lock = request_opts.no_lock;
         let result =
-            enqueue_transition_after_write(self.as_ref(), result, LcEventSrc::S3CompleteMultipartUpload, request_opts).await;
+            enqueue_transition_after_write(self.as_ref(), result, LcEventSrc::S3CompleteMultipartUpload, &transition_opts).await;
         if result.is_ok() {
             list_objects::observe_scanner_namespace_mutations(bucket, 1);
         }

crates/ecstore/src/store/object.rs

@@ -31,7 +31,7 @@ use crate::bucket::lifecycle::{
 };
 use crate::bucket::metadata_sys::{
     acquire_bucket_metadata_transaction_read_lock_in, get_cached_bucket_incarnation_id_in, get_delete_config_from_disk_in,
-    get_object_lock_config_and_incarnation_from_disk_in,
+    get_object_lock_and_quota_config_from_disk_with_guard_in, get_object_lock_config_and_incarnation_from_disk_with_guard_in,
 };
 use crate::bucket::object_lock::objectlock_sys::{
     check_object_lock_for_deletion_with_state, ensure_recursive_force_delete_allowed_for_state,
@@ -3162,7 +3162,7 @@ impl ECStore {
         let lifecycle_guard = self.acquire_bucket_lifecycle_read_lock(bucket).await?;
         let metadata_guard = acquire_bucket_metadata_transaction_read_lock_in(&self.ctx, bucket).await?;
         let (state, bucket_incarnation_id, config_revision) =
-            get_object_lock_config_and_incarnation_from_disk_in(&self.ctx, bucket).await?;
+            get_object_lock_config_and_incarnation_from_disk_with_guard_in(&self.ctx, bucket, Some(&lifecycle_guard)).await?;
         if lifecycle_guard.is_lock_lost() || metadata_guard.is_lock_lost() {
             return Err(Error::other("bucket lifecycle lock was lost while loading the Object Lock snapshot"));
         }
@@ -3177,6 +3177,30 @@ impl ECStore {
         )))
     }
 
+    async fn object_lock_config_snapshot_for_single_put(&self, bucket: &str) -> Result<Arc<ObjectLockConfigSnapshot>> {
+        check_valid_bucket_name(bucket)?;
+        let lifecycle_guard = self.acquire_bucket_lifecycle_read_lock(bucket).await?;
+        let metadata_guard = acquire_bucket_metadata_transaction_read_lock_in(&self.ctx, bucket).await?;
+        let (state, bucket_incarnation_id, config_revision, quota_res) =
+            get_object_lock_and_quota_config_from_disk_with_guard_in(&self.ctx, bucket, Some(&lifecycle_guard)).await?;
+        if lifecycle_guard.is_lock_lost() || metadata_guard.is_lock_lost() {
+            return Err(Error::other("bucket lifecycle lock was lost while loading the Object Lock snapshot"));
+        }
+        let snapshot = ObjectLockConfigSnapshot::for_guarded_store_bucket(
+            self.id,
+            bucket,
+            bucket_incarnation_id,
+            config_revision,
+            state,
+            lifecycle_guard,
+            metadata_guard,
+        );
+        if !self.ctx.lock_manager().is_disabled() {
+            snapshot.set_fresh_put_quota(quota_res);
+        }
+        Ok(Arc::new(snapshot))
+    }
+
     pub(super) async fn object_lock_config_snapshot_under_lifecycle_fence(
         &self,
         bucket: &str,
@@ -3187,7 +3211,7 @@ impl ECStore {
         }
         let metadata_guard = acquire_bucket_metadata_transaction_read_lock_in(&self.ctx, bucket).await?;
         let (state, bucket_incarnation_id, config_revision) =
-            get_object_lock_config_and_incarnation_from_disk_in(&self.ctx, bucket).await?;
+            get_object_lock_config_and_incarnation_from_disk_with_guard_in(&self.ctx, bucket, Some(&metadata_guard)).await?;
         if lifecycle_fence.is_lock_lost() || metadata_guard.is_lock_lost() {
             return Err(Error::other("bucket lock was lost while loading the Object Lock snapshot"));
         }
@@ -4652,7 +4676,7 @@ impl ECStore {
                 .expected_bucket_incarnation_id
                 .ok_or_else(|| Error::other("destructive PUT is missing its bucket incarnation"))?;
             if opts.object_lock_config_snapshot.is_none() {
-                opts.object_lock_config_snapshot = Some(self.object_lock_config_snapshot(bucket).await?);
+                opts.object_lock_config_snapshot = Some(self.object_lock_config_snapshot_for_single_put(bucket).await?);
             }
             let snapshot = match opts.object_lock_config_snapshot.as_ref() {
                 Some(snapshot) if snapshot.is_valid_for_destructive_put(self.id, bucket, expected_incarnation_id) => {
@@ -4741,6 +4765,7 @@ impl ECStore {
         self.apply_decommission_target_mutation_fence(idx, object.as_str(), &mut opts, mutation_fence.or(fixed_read_anchor))
             .await;
         let expected_data_bytes = usize::try_from(data.size()).ok();
+        let mut transition_opts = opts.clone();
         let result = self
             .run_decommission_capacity_admitted_mutation_with_capacity_lease(
                 idx,
@@ -4772,7 +4797,8 @@ impl ECStore {
                 },
             )
             .await;
-        let result = enqueue_transition_after_write(self, result, LcEventSrc::S3PutObject, request_opts).await;
+        transition_opts.no_lock = request_opts.no_lock;
+        let result = enqueue_transition_after_write(self, result, LcEventSrc::S3PutObject, &transition_opts).await;
         if result.is_ok() {
             list_objects::observe_scanner_namespace_mutations(bucket, 1);
         }
@@ -4788,7 +4814,7 @@ impl ECStore {
         data: &mut PutObjReader,
         opts: &ObjectOptions,
     ) -> Result<(ObjectInfo, Option<OldCurrentSize>)> {
-        let (object, opts) = self.prepare_put_object(bucket, object, opts).await?;
+        let (object, opts) = Box::pin(self.prepare_put_object(bucket, object, opts)).await?;
         let idx = self
             .select_put_object_pool_idx(bucket, object.as_str(), data.size(), &opts)
             .await?;
@@ -4797,11 +4823,16 @@ impl ECStore {
             opts.decommission_capacity_admission = crate::bucket::metadata_sys::object_store_if_initialized_in(&self.ctx).await;
         }
         let receipt_sink = install_tier_free_version_receipt_sink(&mut opts);
-        let result = self.pools[idx]
-            .put_object_with_old_current_size(bucket, object.as_str(), data, &opts)
-            .await;
+        let result = Box::pin(self.pools[idx].put_object_with_old_current_size(bucket, object.as_str(), data, &opts)).await;
         enqueue_recorded_tier_free_versions(self, receipt_sink).await;
-        result
+        match result {
+            Ok((object_info, old_current_size)) => {
+                Box::pin(enqueue_transition_after_write(self, Ok(object_info), LcEventSrc::S3PutObject, &opts))
+                    .await
+                    .map(|object_info| (object_info, old_current_size))
+            }
+            Err(err) => Err(err),
+        }
     }
 
     #[instrument(level = "trace", skip(self))]
@@ -4859,7 +4890,7 @@ impl ECStore {
         src_info: &mut ObjectInfo,
         src_opts: &ObjectOptions,
         dst_opts: &ObjectOptions,
-    ) -> Result<ObjectInfo> {
+    ) -> Result<(ObjectInfo, Option<OldCurrentSize>)> {
         check_copy_obj_args(src_bucket, src_object)?;
         if src_bucket == dst_bucket && src_object == dst_object {
             // In-place copies (metadata replace, restore) rewrite a key that
@@ -4938,9 +4969,17 @@ impl ECStore {
         };
 
         if cp_src_dst_same {
-            let (_, pool_idx) = self
-                .get_latest_accessible_object_info_with_idx(src_bucket, &src_object, &version_aware_lookup_opts(src_opts, true))
-                .await?;
+            let pool_idx = if self.single_pool() {
+                0
+            } else {
+                self.get_latest_accessible_object_info_with_idx(
+                    src_bucket,
+                    &src_object,
+                    &version_aware_lookup_opts(src_opts, true),
+                )
+                .await?
+                .1
+            };
             let source_pool_writable = !self.is_suspended(pool_idx).await && !self.is_pool_rebalancing(pool_idx).await;
 
             if source_pool_writable
@@ -4961,7 +5000,8 @@ impl ECStore {
                                 .await
                         },
                     )
-                    .await;
+                    .await
+                    .map(|oi| (oi, None));
             }
 
             if source_pool_writable && !dst_opts.versioned && src_opts.version_id.is_none() {
@@ -4989,7 +5029,8 @@ impl ECStore {
                                     .await
                             },
                         )
-                        .await;
+                        .await
+                        .map(|oi| (oi, None));
                 }
                 // Transitioned object self-copy: restore from tier into the same pool.
                 let mut put_opts = ObjectOptions {
@@ -5017,7 +5058,7 @@ impl ECStore {
                 return if let Some(reader) = src_info.put_object_reader.as_mut() {
                     let receipt_sink = install_tier_free_version_receipt_sink(&mut put_opts);
                     let result = self.pools[pool_idx]
-                        .put_object(dst_bucket, &dst_object, reader, &put_opts)
+                        .put_object_with_old_current_size(dst_bucket, &dst_object, reader, &put_opts)
                         .await;
                     enqueue_recorded_tier_free_versions(self, receipt_sink).await;
                     result
@@ -5061,7 +5102,7 @@ impl ECStore {
                     }
                     let receipt_sink = install_tier_free_version_receipt_sink(&mut put_opts);
                     let result = self.pools[pool_idx]
-                        .put_object(dst_bucket, &dst_object, reader, &put_opts)
+                        .put_object_with_old_current_size(dst_bucket, &dst_object, reader, &put_opts)
                         .await;
                     enqueue_recorded_tier_free_versions(self, receipt_sink).await;
                     return result;
@@ -5081,11 +5122,14 @@ impl ECStore {
                                 .await
                         },
                     )
-                    .await;
+                    .await
+                    .map(|oi| (oi, None));
             }
         }
 
-        let pool_idx = if dst_opts.no_lock {
+        let pool_idx = if self.single_pool() && !self.is_suspended(0).await && !self.is_pool_rebalancing(0).await {
+            0
+        } else if dst_opts.no_lock {
             self.get_pool_idx_no_lock(dst_bucket, &dst_object, src_info.size).await?
         } else {
             self.get_pool_idx(dst_bucket, &dst_object, src_info.size).await?
@@ -5118,7 +5162,7 @@ impl ECStore {
         if let Some(put_object_reader) = src_info.put_object_reader.as_mut() {
             let receipt_sink = install_tier_free_version_receipt_sink(&mut put_opts);
             let result = self.pools[pool_idx]
-                .put_object(dst_bucket, dst_object_name, put_object_reader, &put_opts)
+                .put_object_with_old_current_size(dst_bucket, dst_object_name, put_object_reader, &put_opts)
                 .await;
             enqueue_recorded_tier_free_versions(self, receipt_sink).await;
             return result;
@@ -5350,8 +5394,9 @@ impl ECStore {
             opts.add_namespace_lock_guard(guard);
         }
         let current_bucket_incarnation_id = if let Some(metadata_guard) = _object_lock_metadata_guard.as_ref() {
+            let guard_ref = _bucket_lifecycle_guard.as_deref().unwrap_or_else(|| metadata_guard.as_ref());
             let (state, incarnation_id, config_revision) =
-                get_delete_config_from_disk_in(&self.ctx, bucket, None, opts.lifecycle_expiry_policy.as_ref()).await?;
+                get_delete_config_from_disk_in(&self.ctx, bucket, Some(guard_ref), opts.lifecycle_expiry_policy.as_ref()).await?;
             opts.object_lock_config_snapshot = Some(Arc::new(ObjectLockConfigSnapshot::for_store_bucket_under_lifecycle_fence(
                 self.id,
                 bucket,
@@ -5591,6 +5636,42 @@ impl ECStore {
             return Ok(obj);
         }
 
+        if self.single_pool()
+            && !creates_latest_marker
+            && !opts.data_movement
+            && opts.http_preconditions.is_none()
+            && !opts.replication_request
+            && !opts.versioned
+            && !opts.version_suspended
+            && opts.version_id.is_none()
+            && !self.is_suspended(0).await
+            && !self.is_pool_rebalancing(0).await
+        {
+            let pool = self.pools[0].clone();
+            match self
+                .run_external_decommission_capacity_object_delete(0, bucket, object, object, opts.clone(), |opts| async move {
+                    pool.delete_object(bucket, object, opts).await
+                })
+                .await
+            {
+                Ok(mut res) => {
+                    res.name = decode_dir_object(res.name.as_str());
+                    return Ok(res);
+                }
+                Err(err) if is_err_read_quorum(&err) => return Err(StorageError::ErasureWriteQuorum),
+                Err(err) => {
+                    if should_purge_orphan_dir_on_missing(&err, object) && self.purge_orphan_dir_object(bucket, object).await {
+                        return Ok(ObjectInfo {
+                            bucket: bucket.to_owned(),
+                            name: decode_dir_object(object),
+                            ..Default::default()
+                        });
+                    }
+                    return Err(err);
+                }
+            }
+        }
+
         // Determine which pool contains it
         let existing_pool_info = if creates_latest_marker {
             self.get_pool_info_for_delete_marker(bucket, object, &gopts).await
@@ -9903,6 +9984,7 @@ mod tests {
                 copy_store
                     .handle_copy_object(bucket, source, bucket, target, &mut source_info, &ObjectOptions::default(), &options)
                     .await
+                    .map(|(oi, _)| oi)
             });
             let deadline = tokio::time::Instant::now() + Duration::from_secs(30);
             tokio::time::timeout_at(deadline, barrier.wait_until_paused())

crates/scanner/Cargo.toml

@@ -31,7 +31,6 @@ workspace = true
 
 [features]
 default = []
-e2e-test-hooks = []
 diagnostic-read-version-caller = ["rustfs-ecstore/diagnostic-read-version-caller"]
 hotpath = [
     "hotpath/hotpath",

crates/scanner/src/e2e_barrier.rs

@@ -1,222 +0,0 @@
-// Copyright 2026 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Test-cluster observations and bounded pauses at real scanner boundaries.
-//! This module is absent unless `e2e-test-hooks` is explicitly enabled.
-
-pub(crate) mod worker_observation;
-
-use serde::Deserialize;
-use serde_json::{Value, json};
-use std::io::{self, Write};
-use std::path::{Component, Path};
-use std::sync::atomic::{AtomicU64, Ordering};
-use std::time::{Duration, Instant};
-
-const MAX_BYTES: u64 = 1024 * 1024;
-const MAX_EVENTS: u64 = 512;
-const MAX_WAIT: Duration = Duration::from_secs(120);
-static SEQUENCE: AtomicU64 = AtomicU64::new(0);
-
-#[derive(Deserialize)]
-#[serde(deny_unknown_fields)]
-struct Session {
-    nonce: String,
-    buckets: Vec<String>,
-}
-
-#[derive(Deserialize)]
-#[serde(deny_unknown_fields)]
-struct Arm {
-    nonce: String,
-    stage: String,
-    pid: u32,
-    owner_id: Option<String>,
-    instance_id: Option<String>,
-    epoch: Option<u64>,
-    cycle: Option<u64>,
-}
-
-fn valid_component(value: &str) -> bool {
-    !value.is_empty()
-        && value.len() <= 64
-        && value
-            .bytes()
-            .all(|byte| byte.is_ascii_alphanumeric() || byte == b'-' || byte == b'_')
-}
-
-fn read_control<T: serde::de::DeserializeOwned>(path: &Path) -> io::Result<Option<T>> {
-    let metadata = match std::fs::symlink_metadata(path) {
-        Ok(metadata) => metadata,
-        Err(error) if error.kind() == io::ErrorKind::NotFound => return Ok(None),
-        Err(error) => return Err(error),
-    };
-    if !metadata.is_file() || metadata.len() > MAX_BYTES {
-        return Err(io::Error::new(io::ErrorKind::InvalidData, "invalid scanner test control file"));
-    }
-    let data = std::fs::read(path)?;
-    if data.len() > MAX_BYTES as usize {
-        return Err(io::Error::new(
-            io::ErrorKind::InvalidData,
-            "scanner test control file grew past its bound",
-        ));
-    }
-    serde_json::from_slice(&data).map(Some).map_err(io::Error::other)
-}
-
-fn write_observation(directory: &Path, name: &str, bytes: &[u8]) -> io::Result<()> {
-    let temporary = directory.join(format!(".{name}.tmp"));
-    let mut output = std::fs::OpenOptions::new().write(true).create_new(true).open(&temporary)?;
-    output.write_all(bytes)?;
-    output.sync_all()?;
-    std::fs::rename(temporary, directory.join(name))
-}
-
-/// Observe a real event and, only for an explicitly armed matching session,
-/// wait for its release file. No test control can supply scanner state.
-pub async fn checkpoint(stage: &str, detail: Value) -> io::Result<()> {
-    let Some(directory) = std::env::var_os("RUSTFS_E2E_SCANNER_BARRIER_DIR") else {
-        return Ok(());
-    };
-    let directory = Path::new(&directory);
-    if !directory.is_absolute()
-        || directory.components().any(|part| matches!(part, Component::ParentDir))
-        || !valid_component(stage)
-    {
-        return Err(io::Error::new(io::ErrorKind::InvalidInput, "invalid scanner test barrier path or stage"));
-    }
-    let Some(session) = read_control::<Session>(&directory.join("session.json"))? else {
-        return Ok(());
-    };
-    if !valid_component(&session.nonce)
-        || session.buckets.is_empty()
-        || session.buckets.len() > 32
-        || session.buckets.iter().any(|bucket| bucket.is_empty() || bucket.len() > 255)
-    {
-        return Err(io::Error::new(io::ErrorKind::InvalidInput, "invalid scanner test session"));
-    }
-    // Admission may enable the recorder while this actual start gate is held.
-    // Begin on release, before any snapshot or worker; other boundaries stay synchronous here.
-    if stage != "before_start_snapshot" {
-        worker_observation::boundary(stage, &detail);
-    }
-    if detail
-        .get("bucket")
-        .and_then(Value::as_str)
-        .is_some_and(|bucket| !session.buckets.iter().any(|allowed| allowed == bucket))
-    {
-        return Ok(());
-    }
-    let arm = read_control::<Arm>(&directory.join("arm.json"))?;
-    let armed = arm.is_some_and(|arm| {
-        arm.nonce == session.nonce
-            && arm.stage == stage
-            && arm.pid == std::process::id()
-            && arm
-                .owner_id
-                .as_ref()
-                .is_none_or(|owner| detail.get("owner_id").and_then(Value::as_str) == Some(owner.as_str()))
-            && arm
-                .instance_id
-                .as_ref()
-                .is_none_or(|instance| detail.get("instance_id").and_then(Value::as_str) == Some(instance.as_str()))
-            && arm
-                .epoch
-                .is_none_or(|epoch| detail.get("epoch").and_then(Value::as_u64) == Some(epoch))
-            && arm
-                .cycle
-                .is_none_or(|cycle| detail.get("cycle").and_then(Value::as_u64) == Some(cycle))
-    });
-    let sequence = SEQUENCE.fetch_add(1, Ordering::Relaxed);
-    if sequence >= MAX_EVENTS {
-        return Err(io::Error::other("scanner test observation limit reached"));
-    }
-    let pid = std::process::id();
-    let event = json!({
-        "nonce": session.nonce,
-        "pid": pid,
-        "sequence": sequence,
-        "stage": stage,
-        "armed": armed,
-        "detail": detail,
-    });
-    let encoded = serde_json::to_vec(&event).map_err(io::Error::other)?;
-    if encoded.len() > MAX_BYTES as usize {
-        return Err(io::Error::other("scanner test observation exceeds its byte bound"));
-    }
-    let suffix = format!("{}.{}.{}.json", session.nonce, pid, sequence);
-    write_observation(directory, &format!("observed.{suffix}"), &encoded)?;
-    if !armed {
-        if stage == "before_start_snapshot" {
-            worker_observation::boundary(stage, &event["detail"]);
-        }
-        return Ok(());
-    }
-    write_observation(directory, &format!("reached.{suffix}"), &encoded)?;
-    let release = directory.join(format!("release.{}.{}.{}", session.nonce, stage, pid));
-    let started = Instant::now();
-    while !release.try_exists()? {
-        if started.elapsed() >= MAX_WAIT {
-            return Err(io::Error::new(io::ErrorKind::TimedOut, "scanner test barrier release timed out"));
-        }
-        tokio::time::sleep(Duration::from_millis(25)).await;
-    }
-    if stage == "before_start_snapshot" {
-        worker_observation::boundary(stage, &event["detail"]);
-    }
-    Ok(())
-}
-
-pub(crate) fn hex(bytes: &[u8]) -> String {
-    hex_simd::encode_to_string(bytes, hex_simd::AsciiCase::Lower)
-}
-
-pub(crate) fn acknowledgement_observations(acknowledgements: &[crate::scanner::ScannerDirtyUsageAcknowledgement]) -> Value {
-    Value::Array(
-        acknowledgements
-            .iter()
-            .map(|acknowledgement| {
-                use crate::scanner::ScannerDirtyUsageAcknowledgementKind;
-                match &acknowledgement.kind {
-                    ScannerDirtyUsageAcknowledgementKind::Generation(generation) => json!({
-                        "host": acknowledgement.host,
-                        "instance_id": acknowledgement.instance_id,
-                        "kind": "generation",
-                        "generation": generation,
-                    }),
-                    ScannerDirtyUsageAcknowledgementKind::Scoped { owner_id, entries } => json!({
-                        "host": acknowledgement.host,
-                        "owner_id": owner_id,
-                        "instance_id": acknowledgement.instance_id,
-                        "kind": "scoped",
-                        "entries": entries.iter().map(|entry| json!({
-                            "bucket": entry.bucket,
-                            "bucket_incarnation": entry.bucket_incarnation,
-                            "generation": entry.generation,
-                        })).collect::<Vec<_>>(),
-                    }),
-                }
-            })
-            .collect(),
-    )
-}
-
-pub(crate) fn activity_observations(activity: &crate::scanner::ScannerActivitySnapshot) -> Value {
-    Value::Object(
-        activity
-            .iter()
-            .map(|(host, activity)| (host.clone(), activity.e2e_observation()))
-            .collect(),
-    )
-}

crates/scanner/src/e2e_barrier/worker_observation.rs

@@ -1,558 +0,0 @@
-// Copyright 2026 RustFS Team
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-
-//! N-client evidence only. This recorder never waits for a release or filters buckets.
-
-use super::{read_control, valid_component, write_observation};
-use serde::{Deserialize, Serialize};
-use serde_json::{Value, json};
-use sha2::{Digest, Sha256};
-use std::collections::{BTreeMap, BTreeSet};
-use std::future::Future;
-use std::io;
-use std::path::{Component, Path, PathBuf};
-use std::sync::{Mutex, OnceLock};
-use uuid::Uuid;
-
-const MAX_EVENTS: usize = 512;
-const MAX_CYCLES: usize = 8;
-const MAX_BYTES: usize = 1024 * 1024;
-static OBSERVER: OnceLock<Mutex<Option<Observer>>> = OnceLock::new();
-tokio::task_local! { static INCARNATION: Option<Uuid>; }
-
-#[derive(Deserialize)]
-#[serde(deny_unknown_fields)]
-struct Control {
-    nonce: String,
-    pid: u32,
-}
-
-#[derive(Serialize)]
-struct Request {
-    context: Value,
-    opening_sequence: usize,
-    complete_sequence: Option<usize>,
-    progress: Option<[u64; 3]>,
-}
-
-#[derive(Default)]
-struct Cycle {
-    start_sequence: usize,
-    events: Vec<Value>,
-    requests: BTreeMap<String, Request>,
-    local_walks: Vec<Value>,
-    candidate: Option<Value>,
-    closed: bool,
-}
-
-struct Observer {
-    directory: PathBuf,
-    nonce: String,
-    pid: u32,
-    sequence: usize,
-    errors: BTreeSet<&'static str>,
-    cycles: BTreeMap<(u64, u64), Cycle>,
-    manifests: Vec<Value>,
-}
-
-fn invalid(code: &'static str) -> io::Error {
-    io::Error::other(code)
-}
-
-fn cycle_key(detail: &Value) -> io::Result<(u64, u64)> {
-    Ok((
-        detail["epoch"].as_u64().ok_or_else(|| invalid("malformed"))?,
-        detail["cycle"].as_u64().ok_or_else(|| invalid("malformed"))?,
-    ))
-}
-
-fn uuid_field(detail: &Value, name: &str) -> io::Result<()> {
-    let value = detail[name].as_str().ok_or_else(|| invalid("malformed"))?;
-    if Uuid::parse_str(value).is_ok_and(|id| !id.is_nil()) {
-        Ok(())
-    } else {
-        Err(invalid("malformed"))
-    }
-}
-
-impl Observer {
-    fn new(directory: PathBuf, control: Control) -> Self {
-        Self {
-            directory,
-            nonce: control.nonce,
-            pid: control.pid,
-            sequence: 0,
-            errors: BTreeSet::new(),
-            cycles: BTreeMap::new(),
-            manifests: Vec::new(),
-        }
-    }
-
-    fn health(&self) -> Value {
-        json!({"schema": 1, "nonce": self.nonce, "pid": self.pid, "event_count": self.sequence,
-            "max_events": MAX_EVENTS, "max_cycles": MAX_CYCLES, "max_bytes": MAX_BYTES,
-            "errors": self.errors, "manifests": self.manifests})
-    }
-
-    fn persist(&self, name: &str, value: &Value) -> io::Result<String> {
-        let bytes = serde_json::to_vec(value)?;
-        if bytes.len() > MAX_BYTES {
-            return Err(invalid("overflow"));
-        }
-        write_observation(&self.directory, name, &bytes)?;
-        Ok(super::hex(&Sha256::digest(&bytes)))
-    }
-
-    fn event(&mut self, key: (u64, u64), kind: &str, detail: Value) -> io::Result<usize> {
-        if self.sequence >= MAX_EVENTS {
-            return Err(invalid("overflow"));
-        }
-        let sequence = self.sequence;
-        self.sequence += 1;
-        let event = json!({"schema": 1, "nonce": self.nonce, "pid": self.pid,
-            "sequence": sequence, "kind": kind, "epoch": key.0, "cycle": key.1, "detail": detail});
-        let name = format!("worker-observed.{}.{}.{sequence}.json", self.nonce, self.pid);
-        let sha256 = self.persist(&name, &event)?;
-        self.cycles
-            .get_mut(&key)
-            .ok_or_else(|| invalid("outside_cycle"))?
-            .events
-            .push(json!({"file": name, "sha256": sha256, "sequence": sequence, "kind": kind}));
-        Ok(sequence)
-    }
-
-    fn active(&self, key: (u64, u64)) -> io::Result<()> {
-        match self.cycles.get(&key) {
-            Some(cycle) if !cycle.closed => Ok(()),
-            Some(_) => Err(invalid("late")),
-            None => Err(invalid("outside_cycle")),
-        }
-    }
-
-    fn boundary(&mut self, stage: &str, detail: Value) -> io::Result<()> {
-        let key = cycle_key(&detail)?;
-        if stage == "before_start_snapshot" {
-            if self.cycles.len() >= MAX_CYCLES {
-                return Err(invalid("overflow"));
-            }
-            if self.cycles.contains_key(&key) {
-                return Err(invalid("duplicate"));
-            }
-            if self.cycles.values().any(|cycle| !cycle.closed) {
-                return Err(invalid("cross_cycle"));
-            }
-            self.cycles.insert(
-                key,
-                Cycle {
-                    start_sequence: self.sequence,
-                    ..Default::default()
-                },
-            );
-        }
-        self.active(key)?;
-        match stage {
-            "before_start_snapshot" => {
-                self.event(key, "cycle_begin", detail)?;
-            }
-            "bucket_walk" => {
-                if self.cycles[&key].candidate.is_some() {
-                    return Err(invalid("late"));
-                }
-                uuid_field(&detail, "bucket_incarnation")?;
-                uuid_field(&detail, "disk_uuid")?;
-                let sequence = self.event(key, "local_walk", detail.clone())?;
-                self.cycles
-                    .get_mut(&key)
-                    .ok_or_else(|| invalid("outside_cycle"))?
-                    .local_walks
-                    .push(json!({"sequence": sequence, "context": detail}));
-            }
-            "cycle_candidate" => {
-                if self.cycles[&key].candidate.is_some() {
-                    return Err(invalid("duplicate"));
-                }
-                self.event(key, "candidate", detail.clone())?;
-                self.cycles.get_mut(&key).ok_or_else(|| invalid("outside_cycle"))?.candidate = Some(detail);
-            }
-            "root_durable" => self.close(key, detail)?,
-            _ => return Err(invalid("malformed")),
-        }
-        Ok(())
-    }
-
-    fn opening(&mut self, mut context: Value, incarnation: Option<Uuid>) -> io::Result<()> {
-        let key = cycle_key(&context)?;
-        // A removed session ends the measurement window, but never hides a late event for a known cycle.
-        if !self.cycles.contains_key(&key) && !self.directory.join("session.json").try_exists()? {
-            return Ok(());
-        }
-        self.active(key)?;
-        if self.cycles[&key].candidate.is_some() {
-            return Err(invalid("late"));
-        }
-        context["bucket_incarnation"] = json!(incarnation);
-        for field in [
-            "request_id",
-            "wire_session_id",
-            "server_epoch",
-            "bucket_incarnation",
-            "disk_uuid",
-        ] {
-            uuid_field(&context, field)?;
-        }
-        for field in ["pool", "set", "disk_index", "wire_session_sequence"] {
-            context[field].as_u64().ok_or_else(|| invalid("malformed"))?;
-        }
-        for field in [
-            "bucket",
-            "worker_endpoint",
-            "worker_path",
-            "reuse_plan_sha256",
-            "coverage_plan_sha256",
-        ] {
-            if context[field].as_str().is_none_or(str::is_empty) {
-                return Err(invalid("malformed"));
-            }
-        }
-        for field in ["reuse_plan_sha256", "coverage_plan_sha256"] {
-            if context[field]
-                .as_str()
-                .is_none_or(|value| value.len() != 64 || !value.bytes().all(|byte| byte.is_ascii_hexdigit()))
-            {
-                return Err(invalid("malformed"));
-            }
-        }
-        let id = context["request_id"]
-            .as_str()
-            .ok_or_else(|| invalid("malformed"))?
-            .to_string();
-        if self.cycles.values().any(|cycle| cycle.requests.contains_key(&id)) {
-            return Err(invalid("duplicate"));
-        }
-        let sequence = self.event(key, "remote_scan_opening", context.clone())?;
-        self.cycles
-            .get_mut(&key)
-            .ok_or_else(|| invalid("outside_cycle"))?
-            .requests
-            .insert(
-                id,
-                Request {
-                    context,
-                    opening_sequence: sequence,
-                    complete_sequence: None,
-                    progress: None,
-                },
-            );
-        Ok(())
-    }
-
-    fn complete(&mut self, id: Uuid, progress: [u64; 3]) -> io::Result<()> {
-        let id = id.to_string();
-        let key = self
-            .cycles
-            .iter()
-            .find_map(|(key, cycle)| cycle.requests.contains_key(&id).then_some(*key));
-        let Some(key) = key else {
-            if !self.directory.join("session.json").try_exists()? {
-                return Ok(());
-            }
-            return Err(invalid("unpaired"));
-        };
-        self.active(key)?;
-        if self.cycles[&key].candidate.is_some() {
-            return Err(invalid("late"));
-        }
-        let request = &self.cycles[&key].requests[&id];
-        if request.complete_sequence.is_some() {
-            return Err(invalid("duplicate"));
-        }
-        let detail = json!({"request_id": id, "context": request.context, "progress": progress,
-            "eof_verified": true, "mac_source_plan_verified": true});
-        let sequence = self.event(key, "remote_scan_complete", detail)?;
-        let request = self
-            .cycles
-            .get_mut(&key)
-            .and_then(|cycle| cycle.requests.get_mut(&id))
-            .ok_or_else(|| invalid("unpaired"))?;
-        request.complete_sequence = Some(sequence);
-        request.progress = Some(progress);
-        Ok(())
-    }
-
-    // Hooks are synchronous; all worker/set tasks join before candidate/root.
-    // A closed manifest is per-cycle evidence, not a continuous live-health attestation.
-    fn close(&mut self, key: (u64, u64), root: Value) -> io::Result<()> {
-        if !self.errors.is_empty() {
-            return Err(invalid("unhealthy"));
-        }
-        let cycle = &self.cycles[&key];
-        let candidate = cycle.candidate.as_ref().ok_or_else(|| invalid("unpaired"))?;
-        for field in ["candidate_sha256", "coverage_sha256"] {
-            if root[field].as_str().is_none() || root[field] != candidate[field] {
-                return Err(invalid("coverage"));
-            }
-        }
-        let mut observed = BTreeSet::new();
-        for context in cycle
-            .requests
-            .values()
-            .map(|request| &request.context)
-            .chain(cycle.local_walks.iter().map(|walk| &walk["context"]))
-        {
-            let pool = context["pool"].as_u64().ok_or_else(|| invalid("malformed"))?;
-            let set = context["set"].as_u64().ok_or_else(|| invalid("malformed"))?;
-            let bucket = context["bucket"].as_str().ok_or_else(|| invalid("malformed"))?;
-            if !observed.insert((pool, set, bucket)) {
-                return Err(invalid("duplicate"));
-            }
-            let coverage = candidate["set_coverage"].as_array().ok_or_else(|| invalid("coverage"))?;
-            if !coverage.iter().any(|row| {
-                row["pool"].as_u64() == Some(pool)
-                    && row["set"].as_u64() == Some(set)
-                    && row["bucket_incarnations"][bucket] == context["bucket_incarnation"]
-                    && !context["bucket_incarnation"].is_null()
-            }) {
-                return Err(invalid("coverage"));
-            }
-        }
-        if cycle.requests.values().any(|request| request.complete_sequence.is_none()) {
-            return Err(invalid("unpaired"));
-        }
-        let sequence = self.event(key, "cycle_close", root.clone())?;
-        self.cycles.get_mut(&key).ok_or_else(|| invalid("outside_cycle"))?.closed = true;
-        let cycle = &self.cycles[&key];
-        let manifest = json!({"schema": 1, "nonce": self.nonce, "pid": self.pid, "epoch": key.0, "cycle": key.1,
-            "start_sequence": cycle.start_sequence, "close_sequence": sequence, "events": cycle.events,
-            "requests": cycle.requests, "local_walks": cycle.local_walks, "candidate": cycle.candidate,
-            "root": root, "errors": self.errors, "outstanding": 0, "closed": true});
-        let name = format!("worker-manifest.{}.{}.{}.{}.json", self.nonce, self.pid, key.0, key.1);
-        let sha256 = self.persist(&name, &manifest)?;
-        self.manifests
-            .push(json!({"file": name, "sha256": sha256, "epoch": key.0, "cycle": key.1,
-            "close_sequence": sequence}));
-        Ok(())
-    }
-    fn apply(&mut self, action: impl FnOnce(&mut Self) -> io::Result<()>) {
-        if let Err(error) = action(self) {
-            let code = match error.to_string().as_str() {
-                "overflow" => "overflow",
-                "late" => "late",
-                "duplicate" => "duplicate",
-                "coverage" => "coverage",
-                "unpaired" => "unpaired",
-                "outside_cycle" => "outside_cycle",
-                "cross_cycle" => "cross_cycle",
-                _ => "io_or_malformed",
-            };
-            self.errors.insert(code);
-        }
-        let name = format!("worker-health.{}.{}.json", self.nonce, self.pid);
-        if self.persist(&name, &self.health()).is_err() {
-            self.errors.insert("health_io");
-        }
-    }
-}
-
-fn control(directory: &Path) -> io::Result<Option<Control>> {
-    if !directory.is_absolute() || directory.components().any(|part| matches!(part, Component::ParentDir)) {
-        return Err(invalid("malformed"));
-    }
-    let control = read_control::<Control>(&directory.join("worker-observer.json"))?;
-    if control
-        .as_ref()
-        .is_some_and(|control| !valid_component(&control.nonce) || control.pid == 0)
-    {
-        return Err(invalid("malformed"));
-    }
-    Ok(control)
-}
-
-/// The scope covers the normal open and consume future; it never consults O again.
-pub(crate) async fn with_incarnation<F: Future>(incarnation: Option<Uuid>, future: F) -> F::Output {
-    INCARNATION.scope(incarnation, future).await
-}
-
-pub(crate) fn enabled() -> bool {
-    // Once begun, a transient control read failure must never hide a real opening or completion.
-    if OBSERVER.get().is_some_and(|state| {
-        state
-            .lock()
-            .unwrap_or_else(|poison| poison.into_inner())
-            .as_ref()
-            .is_some_and(|observer| observer.pid == std::process::id())
-    }) {
-        return true;
-    }
-    let Some(directory) = std::env::var_os("RUSTFS_E2E_SCANNER_BARRIER_DIR") else { return false };
-    control(Path::new(&directory))
-        .ok()
-        .flatten()
-        .is_some_and(|control| control.pid == std::process::id())
-}
-
-fn update(action: impl FnOnce(&mut Observer) -> io::Result<()>) {
-    let Some(directory) = std::env::var_os("RUSTFS_E2E_SCANNER_BARRIER_DIR") else { return };
-    let directory = PathBuf::from(directory);
-    let (mut guard, poisoned) = match OBSERVER.get_or_init(|| Mutex::new(None)).lock() {
-        Ok(guard) => (guard, false),
-        Err(poison) => (poison.into_inner(), true),
-    };
-    let configured = control(&directory);
-    match configured {
-        Ok(Some(control)) if control.pid == std::process::id() => {
-            if let Some(observer) = guard.as_mut() {
-                if observer.nonce != control.nonce {
-                    observer.errors.insert("session_changed");
-                }
-            } else {
-                *guard = Some(Observer::new(directory, control));
-            }
-        }
-        Ok(_) => {
-            let Some(observer) = guard.as_mut() else { return };
-            observer.errors.insert("control_changed");
-        }
-        Err(_) => {
-            if let Some(observer) = guard.as_mut() {
-                observer.errors.insert("control_io");
-            }
-        }
-    }
-    let Some(observer) = guard.as_mut() else { return };
-    if poisoned {
-        observer.errors.insert("poisoned");
-    }
-    observer.apply(action);
-}
-
-pub(crate) fn boundary(stage: &str, detail: &Value) {
-    if !matches!(stage, "before_start_snapshot" | "cycle_candidate" | "root_durable" | "bucket_walk") {
-        return;
-    }
-    if stage == "bucket_walk" && detail["worker_kind"] != "local" {
-        return;
-    }
-    update(|observer| observer.boundary(stage, detail.clone()));
-}
-
-pub(crate) fn opening(context: Value) {
-    let incarnation = INCARNATION.try_with(|incarnation| *incarnation).ok().flatten();
-    update(|observer| observer.opening(context, incarnation));
-}
-
-/// Called only after the ordinary client's authenticated terminal EOF validation.
-pub(crate) fn complete(id: Uuid, progress: [u64; 3]) {
-    update(|observer| observer.complete(id, progress));
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    fn observer(path: &Path) -> Observer {
-        std::fs::write(path.join("session.json"), b"{}").expect("active observation window");
-        Observer::new(
-            path.to_path_buf(),
-            Control {
-                nonce: "unit".into(),
-                pid: 1,
-            },
-        )
-    }
-    fn context(id: Uuid, cycle: u64) -> Value {
-        json!({"epoch": 1, "cycle": cycle, "request_id": id, "wire_session_id": Uuid::new_v4(),
-            "server_epoch": Uuid::new_v4(), "wire_session_sequence": 0, "pool": 0, "set": 0,
-            "disk_index": 0, "disk_uuid": Uuid::new_v4(), "bucket": "hot", "worker_endpoint": "http://worker:9000",
-            "worker_path": "/disk", "reuse_plan_sha256": "a".repeat(64), "coverage_plan_sha256": "b".repeat(64)})
-    }
-    fn candidate(incarnation: Uuid) -> Value {
-        json!({"epoch":1,"cycle":2,"candidate_sha256":"c".repeat(64),"coverage_sha256":"d".repeat(64),
-            "set_coverage":[{"pool":0,"set":0,"bucket_incarnations":{"hot":incarnation}}]})
-    }
-    fn root() -> Value {
-        json!({"epoch":1,"cycle":2,"candidate_sha256":"c".repeat(64),"coverage_sha256":"d".repeat(64),
-            "root_sha256":"e".repeat(64),"root_version":"root"})
-    }
-    #[test]
-    fn worker_inventory_rejects_unpaired_partial_and_late_completions() {
-        let dir = tempfile::tempdir().expect("temporary evidence directory");
-        let mut observer = observer(dir.path());
-        observer
-            .boundary("before_start_snapshot", json!({"epoch":1,"cycle":2}))
-            .expect("begin");
-        let id = Uuid::new_v4();
-        let incarnation = Uuid::new_v4();
-        observer.opening(context(id, 2), Some(incarnation)).expect("opening");
-        observer
-            .boundary("cycle_candidate", candidate(incarnation))
-            .expect("candidate");
-        assert_eq!(
-            observer
-                .close((1, 2), root())
-                .expect_err("Partial/MAC/EOF failure cannot pair opening")
-                .to_string(),
-            "unpaired"
-        );
-        observer.apply(|state| state.complete(id, [1, 0, 0]));
-        assert!(observer.errors.contains("late"));
-        assert!(observer.close((1, 2), root()).is_err());
-        assert!(observer.complete(Uuid::new_v4(), [1, 0, 0]).is_err());
-    }
-    #[test]
-    fn worker_inventory_preserves_zero_progress_and_rejects_cross_cycle_and_overflow() {
-        let dir = tempfile::tempdir().expect("temporary evidence directory");
-        let mut observer = observer(dir.path());
-        observer
-            .boundary("before_start_snapshot", json!({"epoch":1,"cycle":2}))
-            .expect("begin");
-        let id = Uuid::new_v4();
-        assert!(observer.opening(context(id, 3), Some(Uuid::new_v4())).is_err());
-        observer.opening(context(id, 2), Some(Uuid::new_v4())).expect("opening");
-        observer
-            .complete(id, [0, 0, 0])
-            .expect("verified Current may make no progress");
-        assert_eq!(observer.cycles[&(1, 2)].requests[&id.to_string()].progress, Some([0, 0, 0]));
-        let incarnation = observer.cycles[&(1, 2)].requests[&id.to_string()].context["bucket_incarnation"]
-            .as_str()
-            .expect("captured incarnation")
-            .parse()
-            .expect("UUID");
-        observer
-            .boundary("cycle_candidate", candidate(incarnation))
-            .expect("candidate");
-        observer
-            .close((1, 2), root())
-            .expect("zero-progress Current is recorded, never silently treated as a walk");
-        observer.apply(|state| state.complete(id, [1, 0, 0]));
-        assert!(observer.errors.contains("late"));
-        // Reserve exhaustion on a new active cycle; closed-cycle rejection must not mask overflow.
-        observer
-            .boundary("before_start_snapshot", json!({"epoch":1,"cycle":3}))
-            .expect("second begin");
-        observer.sequence = MAX_EVENTS;
-        observer.apply(|state| state.opening(context(Uuid::new_v4(), 3), Some(Uuid::new_v4())));
-        assert!(observer.errors.contains("overflow"));
-    }
-    #[test]
-    fn worker_inventory_write_failure_and_missing_incarnation_cannot_close() {
-        let dir = tempfile::tempdir().expect("temporary evidence directory");
-        let mut observer = observer(dir.path());
-        observer
-            .boundary("before_start_snapshot", json!({"epoch":1,"cycle":2}))
-            .expect("begin");
-        observer.apply(|state| state.opening(context(Uuid::new_v4(), 2), None));
-        assert!(observer.errors.contains("io_or_malformed"));
-        std::fs::remove_dir_all(dir.path()).expect("remove evidence destination");
-        observer.apply(|state| state.opening(context(Uuid::new_v4(), 2), Some(Uuid::new_v4())));
-        assert!(observer.errors.contains("health_io"));
-        assert_eq!(
-            observer
-                .close((1, 2), root())
-                .expect_err("sticky recorder failure prevents closure")
-                .to_string(),
-            "unhealthy"
-        );
-        assert!(observer.manifests.is_empty());
-    }
-}

crates/scanner/src/lib.rs

@@ -58,9 +58,6 @@ use tokio_util::sync::CancellationToken;
 use uuid::Uuid;
 
 pub mod data_usage_define;
-#[cfg(feature = "e2e-test-hooks")]
-#[doc(hidden)]
-pub mod e2e_barrier;
 pub mod error;
 mod heal_completion;
 pub mod prefix_usage;

crates/scanner/src/remote_scanner/stream.rs

@@ -793,16 +793,6 @@ async fn scan_and_persist_local_bucket(
     let (checkpoint_tx, mut checkpoint_rx) = mpsc::channel::<DataUsageCache>(1);
     let (node_tx, mut node_rx) = crate::workload_admission::scanner_node_request_channel();
     let scan_ctx = ctx.child_token();
-    #[cfg(feature = "e2e-test-hooks")]
-    let _ = crate::e2e_barrier::checkpoint(
-        "bucket_walk",
-        serde_json::json!({
-            "epoch": leader_epoch, "cycle": next_cycle,
-            "pool": source.pool_index, "set": source.set_index,
-            "bucket": bucket, "drive": disk.path().display().to_string(),
-        }),
-    )
-    .await;
     let scan = crate::workload_admission::with_scanner_node_requests(
         node_tx,
         ScannerIODisk::nsscanner_disk(
@@ -1087,25 +1077,6 @@ pub(crate) async fn scan_remote_bucket(
     let result = async {
         let rpc_deadline = Instant::now() + NS_SCANNER_MAX_RPC_LIFETIME;
         let open_deadline = (Instant::now() + NS_SCANNER_STALL_TIMEOUT).min(rpc_deadline);
-        #[cfg(feature = "e2e-test-hooks")]
-        if crate::e2e_barrier::worker_observation::enabled() {
-            let location = disk.get_disk_location();
-            // RemoteDisk::get_disk_id reads the same cached ID used by disk_ref; no RPC.
-            let disk_uuid = crate::storage_api::owner::EcstoreDiskAPI::get_disk_id(disk)
-                .await
-                .ok()
-                .flatten();
-            crate::e2e_barrier::worker_observation::opening(serde_json::json!({
-                "epoch": leader_epoch, "cycle": next_cycle, "request_id": request_id,
-                "wire_session_id": session_id, "wire_session_sequence": session_sequence,
-                "server_epoch": server_epoch, "bucket": bucket, "pool": expected_source.pool_index,
-                "set": expected_source.set_index, "disk_index": location.disk_idx, "disk_uuid": disk_uuid,
-                "worker_endpoint": crate::storage_api::owner::EcstoreDiskAPI::endpoint(disk).grid_host(),
-                "worker_path": disk.path().display().to_string(),
-                "reuse_plan_sha256": crate::e2e_barrier::hex(&scan_plan_digest.0),
-                "coverage_plan_sha256": crate::e2e_barrier::hex(&checkpoint_plan_digest.0),
-            }));
-        }
         let open_stream = disk.open_ns_scanner_stream(crate::NsScannerOpenRequest {
             request_id,
             server_epoch,
@@ -1457,17 +1428,6 @@ where
                     last_phase,
                 ));
             }
-            #[cfg(feature = "e2e-test-hooks")]
-            if matches!(&result, Ok(RemoteScannerOutcome::Complete { .. })) && crate::e2e_barrier::worker_observation::enabled() {
-                crate::e2e_barrier::worker_observation::complete(
-                    authenticator.request_id,
-                    [
-                        last_progress.objects_scanned,
-                        last_progress.directories_started,
-                        last_progress.entries_visited,
-                    ],
-                );
-            }
             return result;
         }
     }

crates/scanner/src/scanner.rs

@@ -2024,8 +2024,6 @@ where
             resolved_scope_observer: None,
             #[cfg(test)]
             post_walk_gate: None,
-            #[cfg(test)]
-            peer_observations: None,
         },
     )
     .await;
@@ -2166,9 +2164,6 @@ where
         scanner_post_lease_activity_defer_reason(
             result.activity_baseline(),
             probe_scanner_activity(storeapi.as_ref(), true).await,
-            result
-                .remote_dirty_usage_certification()
-                .zip(remote_publication_leases.as_ref().map(|(_, grants)| grants.as_slice())),
         )
     } else {
         None
@@ -2520,24 +2515,9 @@ where
     } else {
         "incomplete_coverage"
     };
-    #[cfg(feature = "e2e-test-hooks")]
-    let durable_observation = scan_cycle_result
-        .publication_expectation()
-        .as_ref()
-        .and_then(|expected| usage_publication_result.e2e_durable_observation(leader_epoch, cycle_info.current, expected));
-    #[cfg(feature = "e2e-test-hooks")]
-    if let Some(observation) = &durable_observation {
-        let _ = crate::e2e_barrier::checkpoint("root_durable", observation.clone()).await;
-    }
     let pending_dirty_usage_buckets = scan_cycle_result.pending_dirty_usage_buckets();
     let (completion_outcome, scanner_pending_maintenance_work, remote_dirty_usage_acknowledgements) =
         finalize_scanner_cycle_result(scan_cycle_result, usage_publication_result);
-    #[cfg(feature = "e2e-test-hooks")]
-    if let Some(mut observation) = durable_observation {
-        observation["acks"] = crate::e2e_barrier::acknowledgement_observations(&remote_dirty_usage_acknowledgements);
-        observation["pending_bucket_count"] = serde_json::json!(pending_dirty_usage_buckets);
-        let _ = crate::e2e_barrier::checkpoint("ack_send", observation).await;
-    }
     let remote_dirty_usage_pending = if remote_dirty_usage_acknowledgements.is_empty() {
         false
     } else if let Some(notification_system) = storeapi.scanner_notification_system() {
@@ -3969,26 +3949,18 @@ impl ScannerCyclePublicationPlan {
 /// Re-prove the guardrail after every peer granted a publication lease. The
 /// comparison uses the guardrail fields (`scanner_activity_guardrail_veto`),
 /// not the full digest: local generation movement in this window is the next
-/// cycle's work and its buckets stay in the dirty map. A sealed producer
-/// proof may cover peer dirty movement too, but must bind every lease owner
-/// and session without changing its candidate or acknowledged tuples.
+/// cycle's work and its buckets stay in the dirty map. A peer whose dirty
+/// usage moved keeps deferring in this phase, as it does during the walk.
 fn scanner_post_lease_activity_defer_reason(
     baseline: Option<&ScannerActivitySnapshot>,
     activity: Result<ScannerActivitySnapshot, String>,
-    certification: Option<(
-        &crate::scanner_io::ScannerRemoteDirtyUsageCertification,
-        &[crate::storage_api::EcstoreScannerPublicationLeaseGrant],
-    )>,
 ) -> Option<ScannerCycleDeferReason> {
     match activity {
         Ok(snapshot)
             if scanner_activity_allows_usage_publication(&snapshot)
                 && baseline.is_some_and(|baseline| {
                     scanner_activity_guardrail_veto(baseline, &snapshot).is_none()
-                        && certification.map_or_else(
-                            || scanner_peer_dirty_usage_moved(baseline, &snapshot).is_none(),
-                            |(proof, grants)| proof.matches_publication_leases(grants) && proof.matches_activity(&snapshot),
-                        )
+                        && scanner_peer_dirty_usage_moved(baseline, &snapshot).is_none()
                 }) =>
         {
             None
@@ -3998,13 +3970,6 @@ fn scanner_post_lease_activity_defer_reason(
                 Err(_) => "probe_failed",
                 Ok(snapshot) if !scanner_activity_allows_usage_publication(snapshot) => "publication_blocked",
                 Ok(_) if baseline.is_none() => "baseline_missing",
-                Ok(snapshot)
-                    if certification.is_some_and(|(proof, grants)| {
-                        !proof.matches_publication_leases(grants) || !proof.matches_activity(snapshot)
-                    }) =>
-                {
-                    "peer_certification"
-                }
                 Ok(snapshot) => baseline
                     .and_then(|baseline| {
                         scanner_activity_guardrail_veto(baseline, snapshot)
@@ -4168,7 +4133,6 @@ fn finalize_scanner_cycle_result(
                 .is_some_and(|proof| proof.verified_version_for(expected).is_some())
         });
     let pending_maintenance_work = scan_cycle_result.has_pending_maintenance_work()
-        || scan_cycle_result.has_remote_dirty_usage_pending()
         || (scan_cycle_result.has_dirty_usage_to_acknowledge() && !durable_complete_snapshot);
     let remote_dirty_usage_acknowledgements = if durable_complete_snapshot {
         match proof {

crates/scanner/src/scanner/activity.rs

@@ -546,24 +546,6 @@ pub(crate) struct ScannerNodeActivity {
     pub(super) publication_blocked: bool,
 }
 
-#[cfg(feature = "e2e-test-hooks")]
-impl ScannerNodeActivity {
-    pub(crate) fn e2e_observation(&self) -> serde_json::Value {
-        serde_json::json!({
-            "instance_id": self.instance_id,
-            "namespace_generation": self.namespace_generation,
-            "maintenance_generation": self.maintenance_generation,
-            "protocol_version": self.protocol_version,
-            "topology_digest": crate::e2e_barrier::hex(&self.topology_digest),
-            "data_movement_active": self.data_movement_active,
-            "dirty_usage_generation": self.dirty_usage_generation,
-            "dirty_usage_pending": self.dirty_usage_pending,
-            "movement_generation": self.movement_generation,
-            "publication_blocked": self.publication_blocked,
-        })
-    }
-}
-
 pub(crate) type ScannerActivitySnapshot = BTreeMap<String, ScannerNodeActivity>;
 
 #[cfg(test)]

crates/scanner/src/scanner/tests.rs

@@ -10362,7 +10362,7 @@ fn post_lease_activity_proof_rejects_a_peer_whose_dirty_usage_moved_before_lease
         (LOCAL_SCANNER_ACTIVITY_NODE.to_string(), scanner_node_activity("local", 5, 2)),
         ("node-2".to_string(), scanner_node_activity("epoch-a", 7, 3)),
     ]);
-    assert_eq!(scanner_post_lease_activity_defer_reason(Some(&before), Ok(before.clone()), None), None);
+    assert_eq!(scanner_post_lease_activity_defer_reason(Some(&before), Ok(before.clone())), None);
 
     // Local generation movement in the lease window is the next cycle's
     // work: its buckets stay in the dirty map and the walk remains certified.
@@ -10373,7 +10373,7 @@ fn post_lease_activity_proof_rejects_a_peer_whose_dirty_usage_moved_before_lease
     local.namespace_generation += 1;
     local.dirty_usage_generation += 1;
     local.dirty_usage_pending = true;
-    assert_eq!(scanner_post_lease_activity_defer_reason(Some(&before), Ok(local_shaded), None), None);
+    assert_eq!(scanner_post_lease_activity_defer_reason(Some(&before), Ok(local_shaded)), None);
 
     // A peer PUT tail that drained before the lease leaves the movement
     // generation accepted by the lease unchanged, but the peer's dirty state
@@ -10387,14 +10387,14 @@ fn post_lease_activity_proof_rejects_a_peer_whose_dirty_usage_moved_before_lease
         "the existing movement-only lease remains valid after a PUT tail drains"
     );
     assert!(scanner_activity_allows_usage_publication(&after));
-    let reason = scanner_post_lease_activity_defer_reason(Some(&before), Ok(after), None);
+    let reason = scanner_post_lease_activity_defer_reason(Some(&before), Ok(after));
     assert_eq!(reason, Some(ScannerCycleDeferReason::ActivityBaselineUnavailable));
 
     // A guardrail field moving in the window still defers.
     let mut moved = before.clone();
     moved.get_mut("node-2").expect("peer should be present").movement_generation += 1;
     assert_eq!(
-        scanner_post_lease_activity_defer_reason(Some(&before), Ok(moved), None),
+        scanner_post_lease_activity_defer_reason(Some(&before), Ok(moved)),
         Some(ScannerCycleDeferReason::ActivityBaselineUnavailable)
     );
 
@@ -10489,7 +10489,7 @@ fn post_lease_activity_proof_requires_a_complete_matching_baseline() {
         (Some(&blocked), Ok(blocked.clone())),
     ] {
         assert_eq!(
-            scanner_post_lease_activity_defer_reason(expected, observed, None),
+            scanner_post_lease_activity_defer_reason(expected, observed),
             Some(ScannerCycleDeferReason::ActivityBaselineUnavailable)
         );
     }

crates/scanner/src/scanner/tests/scoped_ack_publication.rs

@@ -49,7 +49,6 @@ async fn complete_candidate(store: &Arc<ECStore>, cycle: u64) -> (crate::scanner
             service_cohort: None,
             resolved_scope_observer: None,
             post_walk_gate: None,
-            peer_observations: None,
         },
     )
     .await
@@ -286,7 +285,6 @@ async fn scoped_ack_publication_observed_candidate_reuse_requires_a_new_root_pro
             service_cohort: None,
             resolved_scope_observer: Some(observer),
             post_walk_gate: None,
-            peer_observations: None,
         },
     )
     .await
@@ -458,7 +456,6 @@ async fn scoped_ack_publication_rejects_builder_mutation_after_real_root_publish
         "remote_scoped_ack_target",
         "publication_epoch",
         "remote_lease_targets",
-        "remote_pending",
     ] {
         let (_directory, store) = candidate_store().await;
         let (scan, candidate) = complete_candidate(&store, PROOF_CYCLE).await;
@@ -505,7 +502,6 @@ async fn scoped_ack_publication_rejects_builder_mutation_after_real_root_publish
                 },
             }]),
             "publication_epoch" => scan.with_publication_epoch(Some(changed_epoch)),
-            "remote_pending" => scan.with_remote_dirty_usage_pending(true),
             "remote_lease_targets" => scan.with_remote_publication_lease_targets(vec![(
                 "proof-peer:9000".to_string(),
                 crate::scanner_activity_epoch().to_string(),
@@ -528,142 +524,3 @@ async fn scoped_ack_publication_rejects_builder_mutation_after_real_root_publish
     }
     crate::scanner_io::clear_dirty_usage_buckets_for_tests();
 }
-
-#[test]
-#[serial]
-fn certified_empty_remote_ack_keeps_known_remote_work_pending() {
-    crate::scanner_io::clear_dirty_usage_buckets_for_tests();
-    let scan = ScannerCycleResult::new(ScannerCycleStatus::Complete, None).with_remote_dirty_usage_pending(true);
-    let (outcome, pending, acknowledgements) = finalize_scanner_cycle_result(scan, DataUsagePersistOutcome::Current.into());
-    assert_eq!(outcome, ScannerCycleOutcome::Completed);
-    assert!(acknowledgements.is_empty());
-    assert!(pending, "an empty ACK cannot hide the remote buckets excluded from certification");
-    crate::scanner_io::clear_dirty_usage_buckets_for_tests();
-}
-
-#[test]
-fn sealed_peer_proof_binds_every_lease_and_keeps_all_post_lease_guardrails() {
-    use crate::storage_api::scan::{
-        EcstoreScannerPeerDirtyUsageBucket, EcstoreScannerPeerDirtyUsageSnapshot, EcstoreScannerPublicationLease,
-        EcstoreScannerPublicationLeaseGrant,
-    };
-    let peer_rows = [("peer-a", 101_u128, 201_u128, true), ("peer-b", 102, 202, false)];
-    let mut before = BTreeMap::new();
-    let mut peers = Vec::new();
-    let mut grants = Vec::new();
-    for (host, owner, instance, pending) in peer_rows {
-        let owner_id = uuid::Uuid::from_u128(owner).simple().to_string();
-        let instance_id = uuid::Uuid::from_u128(instance).simple().to_string();
-        before.insert(host.to_string(), scanner_node_activity_for_tests(&instance_id, 10, 10, pending));
-        let buckets = if pending {
-            BTreeMap::from([(
-                "cold-bucket".to_string(),
-                EcstoreScannerPeerDirtyUsageBucket {
-                    bucket_incarnation: uuid::Uuid::from_u128(301),
-                    generation: 8,
-                },
-            )])
-        } else {
-            BTreeMap::new()
-        };
-        peers.push((
-            host.to_string(),
-            EcstoreScannerPeerDirtyUsageSnapshot {
-                owner_id: owner_id.clone(),
-                instance_id: instance_id.clone(),
-                generation: 10,
-                pending_bucket_count: buckets.len() as u64,
-                protocol_version: crate::SCANNER_DIRTY_USAGE_SNAPSHOT_PROTOCOL_VERSION,
-                complete: true,
-                buckets,
-            },
-        ));
-        grants.push(EcstoreScannerPublicationLeaseGrant {
-            host: host.to_string(),
-            lease: EcstoreScannerPublicationLease {
-                token: uuid::Uuid::new_v4(),
-                movement_generation: 0,
-                observational_only: false,
-                owner_id,
-                session_id: instance_id,
-                expires_at: std::time::Instant::now() + Duration::from_secs(60),
-            },
-        });
-    }
-    let proof = crate::scanner_io::ScannerRemoteDirtyUsageCertification::for_tests(before.clone(), peers)
-        .expect("complete producer evidence");
-    let mut late_write = before.clone();
-    let writer = late_write.get_mut("peer-a").expect("writer");
-    writer.namespace_generation += 1;
-    writer.dirty_usage_generation += 1;
-    assert_eq!(
-        scanner_post_lease_activity_defer_reason(Some(&before), Ok(late_write.clone()), Some((&proof, &grants))),
-        None
-    );
-
-    for case in [
-        "missing",
-        "duplicate",
-        "owner",
-        "instance",
-        "movement",
-        "expired",
-        "observational",
-        "token",
-    ] {
-        let mut invalid = grants.clone();
-        match case {
-            "missing" => {
-                invalid.pop();
-            }
-            "duplicate" => invalid[1] = invalid[0].clone(),
-            "owner" => invalid[1].lease.owner_id = uuid::Uuid::from_u128(999).to_string(),
-            "instance" => invalid[1].lease.session_id = uuid::Uuid::from_u128(999).to_string(),
-            "movement" => invalid[1].lease.movement_generation += 1,
-            "expired" => invalid[1].lease.expires_at = std::time::Instant::now() - Duration::from_secs(1),
-            "observational" => invalid[1].lease.observational_only = true,
-            "token" => invalid[1].lease.token = uuid::Uuid::nil(),
-            _ => unreachable!(),
-        }
-        assert_eq!(
-            scanner_post_lease_activity_defer_reason(Some(&before), Ok(before.clone()), Some((&proof, &invalid))),
-            Some(ScannerCycleDeferReason::ActivityBaselineUnavailable),
-            "{case}"
-        );
-    }
-    for field in [
-        "instance",
-        "protocol",
-        "topology",
-        "maintenance",
-        "data_movement",
-        "movement",
-        "blocked",
-        "missing_peer",
-        "overflow",
-        "rollback",
-    ] {
-        let mut changed = late_write.clone();
-        let peer = changed.get_mut("peer-a").expect("peer");
-        match field {
-            "instance" => peer.instance_id = uuid::Uuid::from_u128(999).to_string(),
-            "protocol" => peer.protocol_version += 1,
-            "topology" => peer.topology_digest[0] ^= 1,
-            "maintenance" => peer.maintenance_generation += 1,
-            "data_movement" => peer.data_movement_active = true,
-            "movement" => peer.movement_generation += 1,
-            "blocked" => peer.publication_blocked = true,
-            "missing_peer" => {
-                changed.remove("peer-b");
-            }
-            "overflow" => peer.dirty_usage_generation = u64::MAX,
-            "rollback" => peer.dirty_usage_generation = 9,
-            _ => unreachable!(),
-        }
-        assert_eq!(
-            scanner_post_lease_activity_defer_reason(Some(&before), Ok(changed), Some((&proof, &grants))),
-            Some(ScannerCycleDeferReason::ActivityBaselineUnavailable),
-            "{field}"
-        );
-    }
-}

crates/scanner/src/scanner/usage_store.rs

@@ -84,27 +84,6 @@ impl DataUsagePublicationResult {
         (self.outcome, self.proof)
     }
 
-    #[cfg(feature = "e2e-test-hooks")]
-    pub(super) fn e2e_durable_observation(
-        &self,
-        epoch: u64,
-        cycle: u64,
-        expected: &crate::scanner_io::ScannerPublicationExpectation,
-    ) -> Option<serde_json::Value> {
-        let outcome = match self.outcome {
-            DataUsagePersistOutcome::Saved => "saved",
-            DataUsagePersistOutcome::AlreadyDurable => "already_durable",
-            _ => return None,
-        };
-        let (root_version, root_digest) = self.proof.as_ref()?.verified_version_for(expected)?;
-        let (candidate_digest, coverage_digest) = expected.e2e_digests();
-        Some(serde_json::json!({
-            "epoch": epoch, "cycle": cycle, "outcome": outcome,
-            "candidate_sha256": candidate_digest, "coverage_sha256": coverage_digest,
-            "root_version": root_version, "root_sha256": crate::e2e_barrier::hex(root_digest),
-        }))
-    }
-
     pub(super) fn observation_for_attempt(
         &self,
         scanner_epoch: u64,
@@ -1088,23 +1067,6 @@ where
                 break DataUsagePersistOutcome::Deferred(ScannerCycleDeferReason::PublicationLeaseDeadlineExceeded);
             }
 
-            #[cfg(feature = "e2e-test-hooks")]
-            if !observational
-                && target_path == DATA_USAGE_OBJ_NAME_PATH.as_str()
-                && let (Some(expected), Some(epoch), Some(cycle)) =
-                    (ack_expectation.as_ref(), data_usage_info.scanner_epoch, data_usage_info.scanner_cycle)
-                && expected.matches_encoded_candidate(&data_digest)
-            {
-                let (candidate_digest, coverage_digest) = expected.e2e_digests();
-                let _ = crate::e2e_barrier::checkpoint(
-                    "root_before_publish",
-                    serde_json::json!({
-                        "epoch": epoch, "cycle": cycle,
-                        "candidate_sha256": candidate_digest, "coverage_sha256": coverage_digest,
-                    }),
-                )
-                .await;
-            }
             let done_save = Metrics::time(Metric::SaveUsage);
             let (save_result, commit_state) = {
                 let publication_scope = storeapi

crates/scanner/src/scanner_io.rs

@@ -938,7 +938,6 @@ pub struct ScannerBucketScanPlan {
     publication_epoch: Option<u64>,
     dirty_usage_buckets: Arc<DirtyUsageBuckets>,
     bucket_failures: ScannerBucketFailureState,
-    bucket_completions: Option<Arc<Mutex<ScannerBucketCompletions>>>,
     pending_maintenance_work: Arc<AtomicBool>,
     cache_cycle_floor: Arc<AtomicU64>,
     cold_zero_walk_reuse_observed: Arc<AtomicBool>,
@@ -1273,14 +1272,20 @@ enum ScannerCycleActivityStatus {
     Unverified,
 }
 
-fn scanner_cycle_activity_status(
+async fn scanner_cycle_activity_status<S>(
+    store: &S,
+    distributed: bool,
     before: &crate::scanner::ScannerActivitySnapshot,
-    observed: std::result::Result<crate::scanner::ScannerActivitySnapshot, String>,
-    publication_blocked: bool,
-    certification: Option<&ScannerRemoteDirtyUsageCertification>,
-    certification_required: bool,
-) -> (ScannerCycleActivityStatus, Vec<(String, String, u64)>) {
-    match observed {
+) -> (ScannerCycleActivityStatus, Vec<(String, String, u64)>)
+where
+    S: ScannerStorage,
+{
+    // Read the storage movement barrier before sampling the activity
+    // generations. A movement that starts during this await must invalidate
+    // the earlier baseline. Ordinary PUT tails are not part of the barrier:
+    // they shade their bucket and are certified per bucket below.
+    let publication_blocked = store.scanner_data_usage_publication_blocked().await;
+    match crate::scanner::probe_scanner_activity(store, distributed).await {
         Ok(after) => {
             let veto = if publication_blocked {
                 Some(crate::scanner::ScannerActivityGuardrailVeto::new(
@@ -1304,15 +1309,10 @@ fn scanner_cycle_activity_status(
                 );
                 global_metrics().record_scanner_guardrail_veto(veto.reason());
                 ScannerCycleActivityStatus::Changed
-            } else if certification_required && certification.is_none()
-                || certification.is_some_and(|proof| !proof.matches_activity(&after))
-            {
-                ScannerCycleActivityStatus::Unverified
-            } else if let Some(moved) =
-                crate::scanner::scanner_peer_dirty_usage_moved(before, &after).filter(|_| certification.is_none())
-            {
-                // Without a complete producer proof, retain the original
-                // whole-cycle fallback for a moving peer.
+            } else if let Some(moved) = crate::scanner::scanner_peer_dirty_usage_moved(before, &after) {
+                // Per-bucket certification covers the local dirty map only;
+                // a peer whose dirty state moved keeps today's whole-cycle
+                // outcome until scoped acknowledgements can name its buckets.
                 tracing::info!(
                     target: "rustfs::scanner::io",
                     event = EVENT_SCANNER_SET_STATE,
@@ -1743,8 +1743,6 @@ pub(crate) struct ScannerCycleResult {
     partial_bucket_progress: bool,
     dirty_usage_clear: Option<DirtyUsageBuckets>,
     remote_dirty_usage_acknowledgements: Vec<crate::scanner::ScannerDirtyUsageAcknowledgement>,
-    remote_dirty_usage_certification: Option<ScannerRemoteDirtyUsageCertification>,
-    remote_dirty_usage_pending: bool,
     distributed_segment_invalidation_evidence: Option<DistributedSegmentInvalidationEvidence>,
     segment_reuse_activation_preflight: ScannerSegmentReuseActivationPreflight,
     remote_publication_lease_targets: Vec<(String, String, u64)>,
@@ -1767,8 +1765,6 @@ impl ScannerCycleResult {
             partial_bucket_progress: false,
             dirty_usage_clear,
             remote_dirty_usage_acknowledgements: Vec::new(),
-            remote_dirty_usage_certification: None,
-            remote_dirty_usage_pending: false,
             distributed_segment_invalidation_evidence: None,
             segment_reuse_activation_preflight: scanner_segment_reuse_activation_preflight(),
             remote_publication_lease_targets: Vec::new(),
@@ -1892,26 +1888,6 @@ impl ScannerCycleResult {
         self
     }
 
-    fn with_remote_dirty_usage_certification(mut self, certification: Option<ScannerRemoteDirtyUsageCertification>) -> Self {
-        self.publication_expectation = None;
-        self.remote_dirty_usage_certification = certification;
-        self
-    }
-
-    pub(crate) fn remote_dirty_usage_certification(&self) -> Option<&ScannerRemoteDirtyUsageCertification> {
-        self.remote_dirty_usage_certification.as_ref()
-    }
-
-    pub(crate) fn with_remote_dirty_usage_pending(mut self, pending: bool) -> Self {
-        self.publication_expectation = None;
-        self.remote_dirty_usage_pending = pending;
-        self
-    }
-
-    pub(crate) fn has_remote_dirty_usage_pending(&self) -> bool {
-        self.remote_dirty_usage_pending
-    }
-
     fn with_distributed_segment_invalidation_evidence(
         mut self,
         evidence: Option<DistributedSegmentInvalidationEvidence>,
@@ -1993,12 +1969,9 @@ mod guards;
 pub(crate) use guards::ScannerServiceCohort;
 mod io_cache;
 mod io_cycle;
-mod peer_certification;
 #[cfg(test)]
 use io_cache::{ScannerSetCacheGeneration, prepare_scoped_set_scan};
 pub(crate) use io_cycle::{ScannerCycleRequest, nsscanner_with_storage_status_scoped};
-pub(crate) use peer_certification::ScannerRemoteDirtyUsageCertification;
-use peer_certification::{ScannerBucketCompletions, ScannerRemoteDirtyUsageWindow};
 mod io_disk;
 #[cfg(test)]
 mod publish_gate_tests;

crates/scanner/src/scanner_io/cache.rs

@@ -641,11 +641,6 @@ impl ScannerPublicationExpectation {
         Arc::ptr_eq(&self.candidate, &other.candidate) && self.candidate.1 == other.candidate.1
     }
 
-    #[cfg(feature = "e2e-test-hooks")]
-    pub(crate) fn e2e_digests(&self) -> (String, String) {
-        (crate::e2e_barrier::hex(&self.candidate.0), crate::e2e_barrier::hex(&self.candidate.1.0))
-    }
-
     #[cfg(test)]
     pub(crate) fn for_tests(candidate_digest: [u8; 32], coverage_digest: DataUsageScanPlanDigest) -> Self {
         Self {

crates/scanner/src/scanner_io/io_cache.rs

@@ -308,7 +308,6 @@ impl ScannerIOCache for SetDisks {
             publication_epoch,
             dirty_usage_buckets,
             bucket_failures,
-            bucket_completions,
             pending_maintenance_work,
             cache_cycle_floor,
             cold_zero_walk_reuse_observed,
@@ -810,8 +809,6 @@ impl ScannerIOCache for SetDisks {
             let scope_clone = scope.clone();
             let cache_cycle_floor_clone = cache_cycle_floor.clone();
             let expected_publication_epoch_clone = expected_publication_epoch;
-            #[cfg(feature = "e2e-test-hooks")]
-            let worker_incarnations = current_bucket_incarnations.clone();
             let remote_server_epoch = match worker_mode {
                 NamespaceScannerWorkerMode::RemoteV4(server_epoch) => Some(server_epoch),
                 NamespaceScannerWorkerMode::Coordinator => None,
@@ -912,7 +909,7 @@ impl ScannerIOCache for SetDisks {
                             break;
                         };
                         remote_session_sequence = next_sequence;
-                        let remote_scan = crate::remote_scanner::scan_remote_bucket(
+                        let remote_outcome = crate::remote_scanner::scan_remote_bucket(
                             &disk,
                             ctx_clone.clone(),
                             budget_clone.clone(),
@@ -928,13 +925,8 @@ impl ScannerIOCache for SetDisks {
                                 skip_healing: healing,
                                 scan_mode,
                             },
-                        );
-                        #[cfg(feature = "e2e-test-hooks")]
-                        let remote_scan = crate::e2e_barrier::worker_observation::with_incarnation(
-                            worker_incarnations.as_ref().and_then(|values| values.get(&bucket.name)).copied(),
-                            remote_scan,
-                        );
-                        let remote_outcome = remote_scan.await;
+                        )
+                        .await;
                         match remote_outcome {
                             Ok(crate::remote_scanner::RemoteScannerOutcome::Complete {
                                 usage,
@@ -1331,23 +1323,6 @@ impl ScannerIOCache for SetDisks {
                     let (node_tx, mut node_rx) = crate::workload_admission::scanner_node_request_channel();
 
                     let scan_ctx = ctx_clone.child_token();
-                    #[cfg(feature = "e2e-test-hooks")]
-                    {
-                        let disk_uuid = if crate::e2e_barrier::worker_observation::enabled() {
-                            crate::storage_api::owner::EcstoreDiskAPI::get_disk_id(disk.as_ref()).await.ok().flatten()
-                        } else { None };
-                        let location = disk.get_disk_location();
-                        let _ = crate::e2e_barrier::checkpoint("bucket_walk", serde_json::json!({
-                            "epoch": leader_epoch, "cycle": want_cycle,
-                            "pool": source.pool_index, "set": source.set_index,
-                            "bucket": bucket.name, "drive": disk.path().display().to_string(),
-                            "worker_kind": "local", "worker_path": disk.path().display().to_string(),
-                            "worker_endpoint": crate::storage_api::owner::EcstoreDiskAPI::endpoint(disk.as_ref()).grid_host(),
-                            "disk_uuid": disk_uuid,
-                            "disk_index": location.disk_idx,
-                            "bucket_incarnation": worker_incarnations.as_ref().and_then(|values| values.get(&bucket.name)),
-                        })).await;
-                    }
                     let scan = crate::workload_admission::with_scanner_node_requests(node_tx, disk.clone().nsscanner_disk(
                         scan_ctx.clone(),
                         budget_clone.clone(),
@@ -1777,22 +1752,6 @@ impl ScannerIOCache for SetDisks {
             return Err(err);
         }
 
-        // Record only actual worker completions, before adding carried values.
-        // The root's cold entries cannot certify a newly observed peer tuple.
-        if let Some(completions) = bucket_completions {
-            let completed_incarnations = completed_buckets
-                .iter()
-                .filter_map(|bucket| {
-                    current_bucket_incarnations
-                        .as_ref()?
-                        .get(bucket)
-                        .filter(|incarnation| !incarnation.is_nil())
-                        .map(|incarnation| (bucket.clone(), *incarnation))
-                })
-                .collect();
-            completions.lock().await.insert(source, completed_incarnations);
-        }
-
         let carried_buckets = carry_partial_bucket_values(
             &mut cache,
             &old_cache,

crates/scanner/src/scanner_io/io_cycle.rs

@@ -79,8 +79,6 @@ where
         resolved_scope_observer: None,
         #[cfg(test)]
         post_walk_gate: None,
-        #[cfg(test)]
-        peer_observations: None,
     };
     nsscanner_with_storage_status_scoped(store, request).await
 }
@@ -105,133 +103,10 @@ pub(crate) struct ScannerCycleRequest {
     /// certification regardless of how fast the walk finishes.
     #[cfg(test)]
     pub(crate) post_walk_gate: Option<tokio::sync::oneshot::Receiver<()>>,
-    #[cfg(test)]
-    pub(crate) peer_observations: Option<ScannerCyclePeerObservations>,
-}
-
-#[cfg(test)]
-#[derive(Clone)]
-pub(crate) struct ScannerCyclePeerObservations {
-    pub(crate) activity: Arc<dyn Fn() -> crate::scanner::ScannerActivitySnapshot + Send + Sync>,
-    pub(crate) dirty_usage: Arc<dyn Fn() -> Vec<(String, EcstoreScannerPeerDirtyUsageSnapshot)> + Send + Sync>,
-    pub(crate) scoped_ack_capable: bool,
-}
-
-pub(super) async fn probe_scanner_cycle_activity<S>(
-    store: &S,
-    distributed: bool,
-    #[cfg(test)] peer_observations: Option<&ScannerCyclePeerObservations>,
-) -> std::result::Result<crate::scanner::ScannerActivitySnapshot, String>
-where
-    S: ScannerStorage,
-{
-    #[cfg(test)]
-    if let Some(peers) = peer_observations {
-        let mut snapshot = crate::scanner::probe_scanner_activity(store, false).await?;
-        for (host, activity) in (peers.activity)() {
-            if snapshot.insert(host, activity).is_some() {
-                return Err("duplicate test peer activity".to_string());
-            }
-        }
-        return Ok(snapshot);
-    }
-    crate::scanner::probe_scanner_activity(store, distributed).await
-}
-
-async fn observe_remote_dirty_usage_window<S>(
-    store: &S,
-    before: &crate::scanner::ScannerActivitySnapshot,
-    _stage: &str,
-    _epoch: u64,
-    _cycle: u64,
-    #[cfg(test)] peer_observations: Option<&ScannerCyclePeerObservations>,
-) -> (
-    Option<ScannerRemoteDirtyUsageWindow>,
-    std::result::Result<crate::scanner::ScannerActivitySnapshot, String>,
-)
-where
-    S: ScannerStorage,
-{
-    #[cfg(feature = "e2e-test-hooks")]
-    let _ =
-        crate::e2e_barrier::checkpoint(&format!("before_{_stage}"), serde_json::json!({"epoch": _epoch, "cycle": _cycle})).await;
-    #[cfg(test)]
-    let snapshots = if let Some(peers) = peer_observations {
-        Ok((peers.dirty_usage)())
-    } else if let Some(notification) = store.scanner_notification_system() {
-        notification
-            .scanner_dirty_usage_snapshots()
-            .await
-            .map_err(|error| error.to_string())
-    } else {
-        Err("scanner peer snapshot service is unavailable".to_string())
-    };
-    #[cfg(not(test))]
-    let snapshots = if let Some(notification) = store.scanner_notification_system() {
-        notification
-            .scanner_dirty_usage_snapshots()
-            .await
-            .map_err(|error| error.to_string())
-    } else {
-        Err("scanner peer snapshot service is unavailable".to_string())
-    };
-    let after = probe_scanner_cycle_activity(
-        store,
-        true,
-        #[cfg(test)]
-        peer_observations,
-    )
-    .await;
-    let window = match (snapshots, &after) {
-        (Ok(snapshots), Ok(after)) => ScannerRemoteDirtyUsageWindow::verify(before.clone(), snapshots, after.clone()),
-        _ => None,
-    };
-    #[cfg(feature = "e2e-test-hooks")]
-    let _ = crate::e2e_barrier::checkpoint(
-        _stage,
-        window.as_ref().map_or_else(
-            || serde_json::json!({"epoch": _epoch, "cycle": _cycle, "verified": false}),
-            |window| window.observation(_epoch, _cycle),
-        ),
-    )
-    .await;
-    (window, after)
-}
-
-async fn remote_dirty_usage_window_supports_scoped_ack<S>(
-    store: &S,
-    window: &ScannerRemoteDirtyUsageWindow,
-    #[cfg(test)] peer_observations: Option<&ScannerCyclePeerObservations>,
-) -> bool
-where
-    S: ScannerStorage,
-{
-    let acknowledgements = window.acknowledgements();
-    if scanner_scoped_dirty_usage_ack_exceeds_cost_threshold(&acknowledgements) {
-        return false;
-    }
-    // Empty peers remain in the proof; they need no empty capability request.
-    if acknowledgements.is_empty() {
-        return true;
-    }
-    #[cfg(test)]
-    if let Some(peers) = peer_observations {
-        return peers.scoped_ack_capable;
-    }
-    let Some(notification) = store.scanner_notification_system() else {
-        return false;
-    };
-    matches!(
-        notification
-            .scanner_scoped_dirty_usage_capabilities(acknowledgements.into_iter().map(Into::into).collect(),)
-            .await,
-        Ok(true)
-    )
 }
 
 pub(super) struct ScannerBucketScopeResolution<'a> {
     pub(super) requested_scope: ScannerBucketScanScope,
-    pub(super) remote_window: Option<&'a ScannerRemoteDirtyUsageWindow>,
     pub(super) baseline_proof: ScannerCacheBaselineProof<'a>,
     pub(super) activity_before: &'a crate::scanner::ScannerActivitySnapshot,
     pub(super) dirty_usage_snapshot: &'a DirtyUsageSnapshot,
@@ -274,29 +149,6 @@ where
         .cloned()
         .collect::<HashSet<_>>();
     if distributed {
-        if let Some(window) = resolution.remote_window {
-            if let Some(strict_remote_usage) = window.strict_remote_usage() {
-                // Only the original quiet window can supply segment reuse evidence.
-                return resolve_remote_dirty_usage_scope(
-                    resolution.requested_scope,
-                    resolution.dirty_usage_snapshot,
-                    strict_remote_usage,
-                    resolution.all_buckets,
-                    resolution.baseline_proof,
-                );
-            }
-            let mut dirty_buckets = dirty_buckets;
-            dirty_buckets.extend(window.dirty_buckets());
-            return default_result(scoped_scan_scope_from_dirty_buckets(
-                resolution.requested_scope,
-                dirty_buckets,
-                None,
-                true,
-                false,
-                resolution.all_buckets,
-                resolution.baseline_proof,
-            ));
-        }
         let notification_system = store.scanner_notification_system();
         #[cfg(test)]
         let peer_snapshots = if let Some(peer_snapshots) = resolution.test_peer_snapshots.clone() {
@@ -470,11 +322,7 @@ where
         resolved_scope_observer,
         #[cfg(test)]
         post_walk_gate,
-        #[cfg(test)]
-        peer_observations,
     } = request;
-    #[cfg(feature = "e2e-test-hooks")]
-    let requested_scope = e2e_scope_observation(&scan_scope);
     let child_token = ctx.child_token();
     let _tier_cycle_guard = begin_tier_registry_cycle(want_cycle, leader_epoch);
 
@@ -511,17 +359,7 @@ where
     };
 
     let distributed = store.setup_is_dist_erasure().await;
-    #[cfg(test)]
-    let distributed = distributed || peer_observations.is_some();
-    let activity_before = match scanner_activity_preflight(
-        probe_scanner_cycle_activity(
-            store,
-            distributed,
-            #[cfg(test)]
-            peer_observations.as_ref(),
-        )
-        .await,
-    ) {
+    let activity_before = match scanner_activity_preflight(crate::scanner::probe_scanner_activity(store, distributed).await) {
         ScannerActivityPreflight::Ready(snapshot) => Arc::new(snapshot),
         ScannerActivityPreflight::ActivityBaselineUnavailable(err) => {
             warn!(
@@ -582,39 +420,6 @@ where
             .unwrap_or_else(|poisoned| poisoned.into_inner())
             .refresh(&buckets_by_source);
     }
-    let full_scan_required = requires_full_scan || scan_mode == HealScanMode::Deep;
-    let scope_eligible = !full_scan_required && scan_scope.is_default();
-    let mut remote_start = if distributed && scope_eligible && !all_buckets.is_empty() {
-        observe_remote_dirty_usage_window(
-            store,
-            &activity_before,
-            "start_snapshot",
-            leader_epoch,
-            want_cycle,
-            #[cfg(test)]
-            peer_observations.as_ref(),
-        )
-        .await
-        .0
-    } else {
-        None
-    };
-    if let Some(window) = remote_start.as_ref()
-        && !remote_dirty_usage_window_supports_scoped_ack(
-            store,
-            window,
-            #[cfg(test)]
-            peer_observations.as_ref(),
-        )
-        .await
-    {
-        remote_start = None;
-    }
-    // Work and Current-cache identities include writes admitted by the start
-    // window. Its earlier edge remains in the producer proof, not the cache key.
-    let activity_before = remote_start
-        .as_ref()
-        .map_or(activity_before, ScannerRemoteDirtyUsageWindow::activity_for_walk);
     let structural_scan_plan_digest =
         scanner_bucket_plan_digest(&all_buckets, crate::scanner::scanner_activity_structural_digest(&activity_before));
     let scan_plan_digest = scanner_bucket_work_digest(structural_scan_plan_digest, scan_mode, requires_full_scan);
@@ -632,21 +437,22 @@ where
         want_cycle,
         scan_plan_digest: structural_scan_plan_digest,
     };
+    let full_scan_required = requires_full_scan || scan_mode == HealScanMode::Deep;
+    let scope_eligible = !full_scan_required && scan_scope.is_default();
     let scope_resolution = resolve_scanner_bucket_scan_scope(
         store,
         distributed,
         ScannerBucketScopeResolution {
             requested_scope: scan_scope,
-            remote_window: remote_start.as_ref(),
             baseline_proof,
             activity_before: &activity_before,
             dirty_usage_snapshot: &dirty_usage_snapshot,
             all_buckets: &all_buckets,
             requires_full_scan: full_scan_required,
             #[cfg(test)]
-            test_peer_snapshots: peer_observations.as_ref().map(|peers| (peers.dirty_usage)()),
+            test_peer_snapshots: None,
             #[cfg(test)]
-            test_scoped_dirty_usage_capability: peer_observations.as_ref().map(|peers| peers.scoped_ack_capable),
+            test_scoped_dirty_usage_capability: None,
         },
     )
     .await;
@@ -690,16 +496,8 @@ where
         if !bucket_plan_complete {
             return Ok(ScannerCycleResult::new(ScannerCycleStatus::Incomplete, None).with_publication_epoch(publication_epoch));
         }
-        let publication_blocked = store.scanner_data_usage_publication_blocked().await;
-        let activity_after = probe_scanner_cycle_activity(
-            store,
-            distributed,
-            #[cfg(test)]
-            peer_observations.as_ref(),
-        )
-        .await;
         let (activity_status, remote_publication_lease_targets) =
-            scanner_cycle_activity_status(&activity_before, activity_after, publication_blocked, None, false);
+            scanner_cycle_activity_status(store, distributed, &activity_before).await;
         let dirty_usage_status = dirty_usage_snapshot_status(&dirty_usage_snapshot);
         let status = classify_nsscanner_cycle(
             true,
@@ -785,15 +583,6 @@ where
     let serial_progress_accounting = budget.requires_serial_progress_accounting();
     let set_scan_limit = scanner_budgeted_concurrency_limit(dispatch_limit, serial_progress_accounting);
     let bucket_failures = ScannerBucketFailureState::default();
-    let bucket_completions = remote_start
-        .as_ref()
-        .map(|_| Arc::new(Mutex::new(ScannerBucketCompletions::new())));
-    // Recovery sweeps have no remote certification window, but test sessions
-    // still need the actual worker completions before any values are carried.
-    #[cfg(feature = "e2e-test-hooks")]
-    let bucket_completions = bucket_completions.or_else(|| {
-        std::env::var_os("RUSTFS_E2E_SCANNER_BARRIER_DIR").map(|_| Arc::new(Mutex::new(ScannerBucketCompletions::new())))
-    });
     let pending_maintenance_work = Arc::new(AtomicBool::new(false));
     let cold_zero_walk_reuse_observed = Arc::new(AtomicBool::new(false));
     record_set_scan_concurrency_limit(set_scan_limit);
@@ -890,7 +679,6 @@ where
             publication_epoch,
             dirty_usage_buckets: dirty_usage_snapshot.buckets.clone(),
             bucket_failures: bucket_failures.clone(),
-            bucket_completions: bucket_completions.clone(),
             pending_maintenance_work: pending_maintenance_work.clone(),
             cache_cycle_floor: cache_cycle_floor.clone(),
             cold_zero_walk_reuse_observed: cold_zero_walk_reuse_observed.clone(),
@@ -1039,43 +827,8 @@ where
             !namespace_not_found_buckets.is_empty(),
         );
     let dirty_usage_status = dirty_usage_snapshot_status(&dirty_usage_snapshot);
-    // Check the movement barrier before sampling either edge. The two RPC
-    // windows are single observations; publication never waits for silence.
-    let publication_blocked = store.scanner_data_usage_publication_blocked().await;
-    let post_activity_before = probe_scanner_cycle_activity(
-        store,
-        distributed,
-        #[cfg(test)]
-        peer_observations.as_ref(),
-    )
-    .await;
-    let remote_certification_attempted = remote_start.is_some();
-    let (remote_certification, post_activity) = match (remote_start, post_activity_before) {
-        (Some(start), Ok(before)) => {
-            let (post_walk, after) = observe_remote_dirty_usage_window(
-                store,
-                &before,
-                "post_walk_snapshot",
-                leader_epoch,
-                want_cycle,
-                #[cfg(test)]
-                peer_observations.as_ref(),
-            )
-            .await;
-            (
-                post_walk.and_then(|post_walk| ScannerRemoteDirtyUsageCertification::new(start, post_walk)),
-                after,
-            )
-        }
-        (_, after) => (None, after),
-    };
-    let (activity_status, remote_publication_lease_targets) = scanner_cycle_activity_status(
-        &activity_before,
-        post_activity,
-        publication_blocked,
-        remote_certification.as_ref(),
-        remote_certification_attempted,
-    );
+    let (activity_status, remote_publication_lease_targets) =
+        scanner_cycle_activity_status(store, distributed, &activity_before).await;
     let all_bucket_names = all_buckets.iter().map(|bucket| bucket.name.clone()).collect::<Vec<_>>();
     let completed_usage = completed_usage_candidate(
         &results,
@@ -1124,93 +877,17 @@ where
     // stopped early is pending for the same reason: the publication carries
     // its last certified value, not this walk's (a `Complete` cycle has only
     // carried partial buckets, see `classify_nsscanner_cycle`).
-    let mut uncertified_buckets = failed_buckets.union(&partial_buckets).cloned().collect::<HashSet<_>>();
-    let local_pending_buckets = pending_dirty_usage_buckets(&dirty_usage_snapshot.buckets);
-    uncertified_buckets.extend(local_pending_buckets.iter().cloned());
-    let completions = match bucket_completions {
-        Some(completions) => completions.lock().await.clone(),
-        None => ScannerBucketCompletions::new(),
-    };
-    let (certified_acknowledgements, remote_pending_buckets) = remote_certification.as_ref().map_or_else(
-        || (Vec::new(), ScannerCyclePendingBuckets::new()),
-        |proof| proof.classify_buckets(&expected_sources, &completions, &uncertified_buckets),
-    );
-    let remote_dirty_usage_pending = !remote_pending_buckets.is_empty();
-    uncertified_buckets.extend(remote_pending_buckets.iter().cloned());
     let pending_buckets = if cycle_status == ScannerCycleStatus::Complete {
-        let mut pending = local_pending_buckets;
+        let mut pending = pending_dirty_usage_buckets(&dirty_usage_snapshot.buckets);
         pending.extend(partial_buckets.iter().cloned());
-        pending.extend(remote_pending_buckets);
         pending
     } else {
         ScannerCyclePendingBuckets::new()
     };
-    // A verified but empty scoped ACK is deliberate. It must never become a
-    // whole-generation ACK that could clear a hot or carried partial bucket.
-    let remote_dirty_usage_acknowledgements = if cycle_status != ScannerCycleStatus::Complete {
-        Vec::new()
-    } else if remote_certification.is_some() {
-        certified_acknowledgements
-    } else if remote_certification_attempted {
-        Vec::new()
-    } else if !remote_dirty_usage_acknowledgements.is_empty() {
-        remote_dirty_usage_acknowledgements
-    } else if scan_scope.is_default() {
-        crate::scanner::scanner_dirty_usage_acknowledgements(&activity_before)
-    } else {
-        Vec::new()
-    };
     let mut publication_expectation = None;
     let usage_update_emitted = if let Some(candidate) = completed_usage {
         let (data_usage_info, expectation) = candidate.prepare(cycle_status, &pending_buckets);
         publication_expectation = expectation;
-        #[cfg(feature = "e2e-test-hooks")]
-        {
-            let digests = publication_expectation
-                .as_ref()
-                .map(ScannerPublicationExpectation::e2e_digests);
-            let scanned_buckets = completions
-                .values()
-                .flat_map(|buckets| buckets.keys().cloned())
-                .collect::<BTreeSet<_>>();
-            let mut set_coverage = completions.iter().collect::<Vec<_>>();
-            set_coverage.sort_by_key(|(source, _)| (source.pool_index, source.set_index));
-            let _ = crate::e2e_barrier::checkpoint(
-                "cycle_candidate",
-                serde_json::json!({
-                    "epoch": leader_epoch,
-                    "cycle": want_cycle,
-                    "scan_mode": if scan_mode == HealScanMode::Deep { "deep" } else { "normal" },
-                    "requested_scope": requested_scope,
-                    "selected_scope": e2e_scope_observation(&scan_scope),
-                    "scanned_buckets": scanned_buckets,
-                    "pending_buckets": data_usage_info.usage_snapshot_pending_buckets,
-                    "converged": data_usage_info.usage_snapshot_converged,
-                    "status": match cycle_status {
-                        ScannerCycleStatus::Complete => "complete",
-                        ScannerCycleStatus::Incomplete => "incomplete",
-                        ScannerCycleStatus::Superseded => "superseded",
-                        ScannerCycleStatus::Deferred(_) => "deferred",
-                    },
-                    "candidate_sha256": digests.as_ref().map(|digests| &digests.0),
-                    "coverage_sha256": digests.as_ref().map(|digests| &digests.1),
-                    "peer_certification": remote_certification.is_some(),
-                    "certified_peers": remote_certification.as_ref().map(|proof| {
-                        proof.post_walk.peers.iter().map(|(host, peer)| serde_json::json!({
-                            "host": host, "owner_id": peer.owner_id, "instance_id": peer.instance_id,
-                        })).collect::<Vec<_>>()
-                    }).unwrap_or_default(),
-                    "set_coverage": set_coverage.into_iter().map(|(source, buckets)| serde_json::json!({
-                        "pool": source.pool_index,
-                        "set": source.set_index,
-                        "buckets": buckets.keys().cloned().collect::<BTreeSet<_>>(),
-                        "bucket_incarnations": buckets,
-                    })).collect::<Vec<_>>(),
-                    "acks": crate::e2e_barrier::acknowledgement_observations(&remote_dirty_usage_acknowledgements),
-                }),
-            )
-            .await;
-        }
         if should_publish_observational_snapshot(cycle_status) {
             publish_observational_snapshot(&updates, data_usage_info).await?
         } else {
@@ -1225,6 +902,7 @@ where
     };
     // A walk that stopped early certifies nothing: the bucket keeps its
     // dirty generation like a failed one and is walked again next cycle.
+    let uncertified_buckets = failed_buckets.union(&partial_buckets).cloned().collect::<HashSet<_>>();
     let dirty_usage_clear = should_clear_dirty_usage_snapshot(
         result.is_ok(),
         structurally_complete_snapshot,
@@ -1237,6 +915,14 @@ where
     if cycle_status == ScannerCycleStatus::Complete {
         complete_tier_registry_cycle(want_cycle, leader_epoch);
     }
+    let remote_dirty_usage_acknowledgements =
+        if cycle_status == ScannerCycleStatus::Complete && !remote_dirty_usage_acknowledgements.is_empty() {
+            remote_dirty_usage_acknowledgements
+        } else if cycle_status == ScannerCycleStatus::Complete && scan_scope.is_default() {
+            crate::scanner::scanner_dirty_usage_acknowledgements(&activity_before)
+        } else {
+            Vec::new()
+        };
     Ok(ScannerCycleResult::new(cycle_status, dirty_usage_clear)
         .with_publication_epoch(publication_epoch)
         .with_activity_digest(activity_digest)
@@ -1246,8 +932,6 @@ where
         .with_partial_bucket_progress(partial_bucket_progress)
         .with_remote_publication_lease_targets(remote_publication_lease_targets)
         .with_remote_dirty_usage_acknowledgements(remote_dirty_usage_acknowledgements)
-        .with_remote_dirty_usage_certification(remote_certification)
-        .with_remote_dirty_usage_pending(remote_dirty_usage_pending)
         .with_distributed_segment_invalidation_evidence(distributed_segment_invalidation_evidence)
         .with_segment_reuse_activation_preflight(segment_reuse_activation_preflight)
         .with_failed_dirty_usage(!failed_buckets.is_empty())
@@ -1256,11 +940,3 @@ where
         .with_scope_fallback(scope_fallback)
         .with_publication_expectation(publication_expectation))
 }
-
-#[cfg(feature = "e2e-test-hooks")]
-fn e2e_scope_observation(scope: &ScannerBucketScanScope) -> serde_json::Value {
-    match &scope.selected_buckets {
-        Some(buckets) => serde_json::json!({"kind": "buckets", "buckets": buckets.iter().collect::<BTreeSet<_>>()}),
-        None => serde_json::json!({"kind": "all"}),
-    }
-}

crates/scanner/src/scanner_io/peer_certification.rs

@@ -1,582 +0,0 @@
-// Copyright 2026 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-use super::*;
-use crate::scanner::{
-    ScannerActivitySnapshot, ScannerDirtyUsageAcknowledgement, ScannerDirtyUsageAcknowledgementKind,
-    scanner_activity_allows_usage_publication, scanner_activity_dirty_usage_state_for_host, scanner_activity_guardrail_veto,
-    scanner_activity_publication_lease_targets,
-};
-use crate::storage_api::EcstoreScannerPublicationLeaseGrant;
-
-/// Only worker results received in this cycle enter this map. A carried value
-/// or a cold bucket copied from the old set root is not a completion.
-pub(super) type ScannerBucketCompletions = HashMap<DataUsageCacheSource, HashMap<String, uuid::Uuid>>;
-
-#[derive(Debug)]
-pub(super) struct ScannerRemoteDirtyUsageWindow {
-    pub(super) before: ScannerActivitySnapshot,
-    pub(super) after: ScannerActivitySnapshot,
-    pub(super) peers: BTreeMap<String, EcstoreScannerPeerDirtyUsageSnapshot>,
-}
-
-fn remote_generations_advance(before: &ScannerActivitySnapshot, after: &ScannerActivitySnapshot) -> bool {
-    scanner_activity_allows_usage_publication(before)
-        && scanner_activity_allows_usage_publication(after)
-        && scanner_activity_guardrail_veto(before, after).is_none()
-        && scanner_activity_publication_lease_targets(before).iter().all(|(host, _, _)| {
-            let Some((_, before_generation, _)) = scanner_activity_dirty_usage_state_for_host(before, host) else {
-                return false;
-            };
-            scanner_activity_dirty_usage_state_for_host(after, host)
-                .is_some_and(|(_, after_generation, _)| before_generation <= after_generation && after_generation < u64::MAX)
-        })
-}
-
-impl ScannerRemoteDirtyUsageWindow {
-    /// The caller obtains these in order: before activity, dirty snapshot,
-    /// after activity. The generation can advance without waiting for silence.
-    pub(super) fn verify(
-        before: ScannerActivitySnapshot,
-        snapshots: Vec<(String, EcstoreScannerPeerDirtyUsageSnapshot)>,
-        after: ScannerActivitySnapshot,
-    ) -> Option<Self> {
-        let expected = scanner_activity_publication_lease_targets(&before);
-        if expected.is_empty() || expected.len() != snapshots.len() || !remote_generations_advance(&before, &after) {
-            return None;
-        }
-        let mut peers = BTreeMap::new();
-        for (host, snapshot) in snapshots {
-            let (instance_id, lower, lower_pending) = scanner_activity_dirty_usage_state_for_host(&before, &host)?;
-            let (_, upper, upper_pending) = scanner_activity_dirty_usage_state_for_host(&after, &host)?;
-            let pending = !snapshot.buckets.is_empty();
-            if host.is_empty()
-                || snapshot.instance_id != instance_id
-                || uuid::Uuid::parse_str(&snapshot.owner_id)
-                    .ok()
-                    .is_none_or(|owner| owner.is_nil())
-                || uuid::Uuid::parse_str(&snapshot.instance_id)
-                    .ok()
-                    .is_none_or(|instance| instance.is_nil())
-                || snapshot.protocol_version != crate::SCANNER_DIRTY_USAGE_SNAPSHOT_PROTOCOL_VERSION
-                || !snapshot.complete
-                || snapshot.buckets.len() > crate::SCANNER_DIRTY_USAGE_SNAPSHOT_MAX_ENTRIES
-                || snapshot.pending_bucket_count != u64::try_from(snapshot.buckets.len()).ok()?
-                || !(lower <= snapshot.generation && snapshot.generation <= upper && upper < u64::MAX)
-                || (lower == snapshot.generation && lower_pending != pending)
-                || (upper == snapshot.generation && upper_pending != pending)
-                || snapshot.buckets.iter().any(|(bucket, entry)| {
-                    bucket.is_empty()
-                        || bucket == DATA_USAGE_ROOT
-                        || entry.bucket_incarnation.is_nil()
-                        || entry.generation == 0
-                        || entry.generation > snapshot.generation
-                })
-                || peers.insert(host, snapshot).is_some()
-            {
-                return None;
-            }
-        }
-        (peers.len() == expected.len() && expected.iter().all(|(host, _, _)| peers.contains_key(host))).then_some(Self {
-            before,
-            after,
-            peers,
-        })
-    }
-
-    pub(super) fn activity_for_walk(&self) -> Arc<ScannerActivitySnapshot> {
-        Arc::new(self.after.clone())
-    }
-
-    pub(super) fn strict_remote_usage(&self) -> Option<VerifiedRemoteDirtyUsage> {
-        if crate::scanner::scanner_peer_dirty_usage_moved(&self.before, &self.after).is_some() {
-            return None;
-        }
-        let expected = self
-            .peers
-            .keys()
-            .map(|host| {
-                let (instance_id, generation, pending) = scanner_activity_dirty_usage_state_for_host(&self.before, host)?;
-                Some((
-                    host.clone(),
-                    ScannerPeerDirtyUsageExpectation {
-                        instance_id: instance_id.to_string(),
-                        generation,
-                        pending,
-                    },
-                ))
-            })
-            .collect::<Option<HashMap<_, _>>>()?;
-        verified_remote_dirty_usage(&expected, self.peers.iter().map(|(host, peer)| (host.clone(), peer.clone())).collect())
-    }
-
-    #[cfg(feature = "e2e-test-hooks")]
-    pub(super) fn observation(&self, epoch: u64, cycle: u64) -> serde_json::Value {
-        serde_json::json!({
-            "epoch": epoch,
-            "cycle": cycle,
-            "verified": true,
-            "activity_before": crate::e2e_barrier::activity_observations(&self.before),
-            "activity_after": crate::e2e_barrier::activity_observations(&self.after),
-            "peers": self.peers.iter().map(|(host, peer)| serde_json::json!({
-                "host": host,
-                "owner_id": peer.owner_id,
-                "instance_id": peer.instance_id,
-                "generation": peer.generation,
-                "protocol_version": peer.protocol_version,
-                "complete": peer.complete,
-                "pending_bucket_count": peer.pending_bucket_count,
-                "buckets": peer.buckets.iter().map(|(bucket, entry)| serde_json::json!({
-                    "bucket": bucket,
-                    "bucket_incarnation": entry.bucket_incarnation,
-                    "generation": entry.generation,
-                })).collect::<Vec<_>>(),
-            })).collect::<Vec<_>>(),
-        })
-    }
-
-    pub(super) fn dirty_buckets(&self) -> HashSet<String> {
-        self.peers.values().flat_map(|peer| peer.buckets.keys().cloned()).collect()
-    }
-
-    pub(super) fn acknowledgements(&self) -> Vec<ScannerDirtyUsageAcknowledgement> {
-        self.peers
-            .iter()
-            .filter(|(_, peer)| !peer.buckets.is_empty())
-            .map(|(host, peer)| ScannerDirtyUsageAcknowledgement {
-                host: host.clone(),
-                instance_id: peer.instance_id.clone(),
-                kind: ScannerDirtyUsageAcknowledgementKind::Scoped {
-                    owner_id: peer.owner_id.clone(),
-                    entries: peer
-                        .buckets
-                        .iter()
-                        .map(|(bucket, entry)| crate::storage_api::EcstoreScannerScopedDirtyUsageAckEntry {
-                            bucket: bucket.clone(),
-                            bucket_incarnation: entry.bucket_incarnation,
-                            generation: entry.generation,
-                        })
-                        .collect(),
-                },
-            })
-            .collect()
-    }
-}
-
-/// A complete producer proof includes clean peers even when it yields no ACK.
-/// The start and post-walk maps remain sealed until publication and lease
-/// validation; post-lease writes are retained by exact receiver comparison.
-#[derive(Debug)]
-pub(crate) struct ScannerRemoteDirtyUsageCertification {
-    pub(super) start: ScannerRemoteDirtyUsageWindow,
-    pub(super) post_walk: ScannerRemoteDirtyUsageWindow,
-}
-
-impl ScannerRemoteDirtyUsageCertification {
-    pub(super) fn new(start: ScannerRemoteDirtyUsageWindow, post_walk: ScannerRemoteDirtyUsageWindow) -> Option<Self> {
-        if !remote_generations_advance(&start.after, &post_walk.before)
-            || start.peers.len() != post_walk.peers.len()
-            || start.peers.iter().any(|(host, before)| {
-                post_walk.peers.get(host).is_none_or(|after| {
-                    before.owner_id != after.owner_id
-                        || before.instance_id != after.instance_id
-                        || before.protocol_version != after.protocol_version
-                        || before.generation > after.generation
-                })
-            })
-        {
-            return None;
-        }
-        Some(Self { start, post_walk })
-    }
-
-    #[cfg(test)]
-    pub(crate) fn for_tests(
-        activity: ScannerActivitySnapshot,
-        peers: Vec<(String, EcstoreScannerPeerDirtyUsageSnapshot)>,
-    ) -> Option<Self> {
-        let start = ScannerRemoteDirtyUsageWindow::verify(activity.clone(), peers.clone(), activity.clone())?;
-        let post = ScannerRemoteDirtyUsageWindow::verify(activity.clone(), peers, activity)?;
-        Self::new(start, post)
-    }
-
-    pub(super) fn classify_buckets(
-        &self,
-        expected_sources: &HashSet<DataUsageCacheSource>,
-        completions: &ScannerBucketCompletions,
-        uncertified: &HashSet<String>,
-    ) -> (Vec<ScannerDirtyUsageAcknowledgement>, ScannerCyclePendingBuckets) {
-        let mut buckets = self.start.dirty_buckets();
-        buckets.extend(self.post_walk.dirty_buckets());
-        let sources_complete = !expected_sources.is_empty()
-            && completions.len() == expected_sources.len()
-            && expected_sources.iter().all(|source| completions.contains_key(source));
-        let pending = buckets
-            .iter()
-            .filter(|bucket| {
-                !sources_complete
-                    || uncertified.contains(*bucket)
-                    || self.start.peers.iter().any(|(host, before)| {
-                        let start = before.buckets.get(*bucket);
-                        let post_walk = self.post_walk.peers[host].buckets.get(*bucket);
-                        start != post_walk
-                            || start.is_some_and(|entry| {
-                                expected_sources.iter().any(|source| {
-                                    completions.get(source).and_then(|set| set.get(*bucket)) != Some(&entry.bucket_incarnation)
-                                })
-                            })
-                    })
-            })
-            .cloned()
-            .collect::<ScannerCyclePendingBuckets>();
-        let acknowledgements = self
-            .start
-            .acknowledgements()
-            .into_iter()
-            .filter_map(|mut acknowledgement| {
-                let ScannerDirtyUsageAcknowledgementKind::Scoped { entries, .. } = &mut acknowledgement.kind else {
-                    return None;
-                };
-                entries.retain(|entry| !pending.contains(&entry.bucket));
-                (!entries.is_empty()).then_some(acknowledgement)
-            })
-            .collect();
-        (acknowledgements, pending)
-    }
-
-    pub(crate) fn matches_activity(&self, activity: &ScannerActivitySnapshot) -> bool {
-        remote_generations_advance(&self.post_walk.after, activity)
-    }
-
-    pub(crate) fn matches_publication_leases(&self, grants: &[EcstoreScannerPublicationLeaseGrant]) -> bool {
-        if grants.len() != self.post_walk.peers.len() {
-            return false;
-        }
-        let targets = scanner_activity_publication_lease_targets(&self.post_walk.after)
-            .into_iter()
-            .map(|(host, instance, movement)| (host, (instance, movement)))
-            .collect::<BTreeMap<_, _>>();
-        let mut hosts = HashSet::new();
-        grants.iter().all(|grant| {
-            let Some(peer) = self.post_walk.peers.get(&grant.host) else {
-                return false;
-            };
-            hosts.insert(grant.host.as_str())
-                && grant.lease.is_valid()
-                && !grant.lease.observational_only
-                && !grant.lease.token.is_nil()
-                && grant.lease.owner_id == peer.owner_id
-                && grant.lease.session_id == peer.instance_id
-                && targets.get(&grant.host).is_some_and(|(instance, movement)| {
-                    instance == &grant.lease.session_id && *movement == grant.lease.movement_generation
-                })
-        })
-    }
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    const HOT: uuid::Uuid = uuid::Uuid::from_u128(71);
-    const COLD: uuid::Uuid = uuid::Uuid::from_u128(72);
-
-    fn peers(generation: u64, hot_generation: u64) -> Vec<(String, EcstoreScannerPeerDirtyUsageSnapshot)> {
-        ["peer-a", "peer-b"]
-            .into_iter()
-            .enumerate()
-            .map(|(index, host)| {
-                let buckets = if index == 0 {
-                    BTreeMap::from([
-                        (
-                            "cold-bucket".to_string(),
-                            crate::storage_api::EcstoreScannerPeerDirtyUsageBucket {
-                                bucket_incarnation: COLD,
-                                generation: 8,
-                            },
-                        ),
-                        (
-                            "hot-bucket".to_string(),
-                            crate::storage_api::EcstoreScannerPeerDirtyUsageBucket {
-                                bucket_incarnation: HOT,
-                                generation: hot_generation,
-                            },
-                        ),
-                    ])
-                } else {
-                    BTreeMap::new()
-                };
-                (
-                    host.to_string(),
-                    EcstoreScannerPeerDirtyUsageSnapshot {
-                        owner_id: uuid::Uuid::from_u128(101 + index as u128).simple().to_string(),
-                        instance_id: uuid::Uuid::from_u128(201 + index as u128).simple().to_string(),
-                        generation: if index == 0 { generation } else { 7 },
-                        pending_bucket_count: buckets.len() as u64,
-                        protocol_version: crate::SCANNER_DIRTY_USAGE_SNAPSHOT_PROTOCOL_VERSION,
-                        complete: true,
-                        buckets,
-                    },
-                )
-            })
-            .collect()
-    }
-
-    fn activity(generation: u64) -> ScannerActivitySnapshot {
-        peers(generation, generation)
-            .into_iter()
-            .map(|(host, peer)| {
-                (
-                    host,
-                    crate::scanner::scanner_node_activity_for_tests(
-                        &peer.instance_id,
-                        peer.generation,
-                        peer.generation,
-                        !peer.buckets.is_empty(),
-                    ),
-                )
-            })
-            .collect()
-    }
-
-    fn window(lower: u64, generation: u64, upper: u64) -> ScannerRemoteDirtyUsageWindow {
-        ScannerRemoteDirtyUsageWindow::verify(activity(lower), peers(generation, generation), activity(upper))
-            .expect("ordered complete producer window")
-    }
-
-    fn completions() -> (HashSet<DataUsageCacheSource>, ScannerBucketCompletions) {
-        let sources = HashSet::from([DataUsageCacheSource::new(0, 0), DataUsageCacheSource::new(0, 1)]);
-        let completed = sources
-            .iter()
-            .map(|source| {
-                (
-                    *source,
-                    HashMap::from([("cold-bucket".to_string(), COLD), ("hot-bucket".to_string(), HOT)]),
-                )
-            })
-            .collect();
-        (sources, completed)
-    }
-
-    #[test]
-    fn hot_writes_cross_both_bounded_windows_while_the_clean_peer_stays_in_the_proof() {
-        let start = window(9, 10, 11);
-        assert!(start.strict_remote_usage().is_none(), "the relaxed window grants no segment reuse proof");
-        let proof = ScannerRemoteDirtyUsageCertification::new(start, window(12, 13, 14)).expect("same producers");
-        let (sources, completed) = completions();
-        let (acks, pending) = proof.classify_buckets(&sources, &completed, &HashSet::new());
-        assert_eq!(pending, BTreeSet::from(["hot-bucket".to_string()]));
-        assert_eq!(acks.len(), 1);
-        assert_eq!(proof.post_walk.peers.len(), 2);
-        assert!(proof.post_walk.peers["peer-b"].buckets.is_empty());
-        let ScannerDirtyUsageAcknowledgementKind::Scoped { entries, .. } = &acks[0].kind else {
-            panic!("scoped ACK")
-        };
-        assert_eq!(entries.len(), 1);
-        assert_eq!(
-            (entries[0].bucket.as_str(), entries[0].bucket_incarnation, entries[0].generation),
-            ("cold-bucket", COLD, 8)
-        );
-    }
-
-    #[test]
-    fn producer_windows_reject_missing_duplicate_stale_future_overflow_and_incomplete_evidence() {
-        for case in [
-            "missing_peer",
-            "duplicate_peer",
-            "old_snapshot",
-            "future_snapshot",
-            "overflow",
-            "owner",
-            "instance",
-            "protocol",
-            "incomplete",
-            "count",
-            "future_entry",
-            "nil_incarnation",
-            "empty_bucket",
-            "entry_limit",
-            "pending",
-        ] {
-            let mut snapshots = peers(10, 10);
-            let before = activity(9);
-            let mut after = activity(11);
-            match case {
-                "missing_peer" => {
-                    snapshots.pop();
-                }
-                "duplicate_peer" => {
-                    snapshots[1] = snapshots[0].clone();
-                }
-                "old_snapshot" => snapshots[0].1.generation = 8,
-                "future_snapshot" => snapshots[0].1.generation = 12,
-                "overflow" => after = activity(u64::MAX),
-                "owner" => snapshots[0].1.owner_id = uuid::Uuid::nil().to_string(),
-                "instance" => snapshots[0].1.instance_id = uuid::Uuid::from_u128(999).to_string(),
-                "protocol" => snapshots[0].1.protocol_version += 1,
-                "incomplete" => snapshots[0].1.complete = false,
-                "count" => snapshots[0].1.pending_bucket_count += 1,
-                "future_entry" => snapshots[0].1.buckets.get_mut("cold-bucket").expect("cold").generation = 11,
-                "nil_incarnation" => {
-                    snapshots[0]
-                        .1
-                        .buckets
-                        .get_mut("cold-bucket")
-                        .expect("cold")
-                        .bucket_incarnation = uuid::Uuid::nil()
-                }
-                "empty_bucket" => {
-                    let entry = snapshots[0].1.buckets.remove("cold-bucket").expect("cold");
-                    snapshots[0].1.buckets.insert(String::new(), entry);
-                }
-                "entry_limit" => {
-                    let peer = &mut snapshots[0].1;
-                    peer.buckets = (0..=crate::SCANNER_DIRTY_USAGE_SNAPSHOT_MAX_ENTRIES)
-                        .map(|index| {
-                            (
-                                format!("bucket-{index}"),
-                                crate::storage_api::EcstoreScannerPeerDirtyUsageBucket {
-                                    bucket_incarnation: COLD,
-                                    generation: 8,
-                                },
-                            )
-                        })
-                        .collect();
-                    peer.pending_bucket_count = peer.buckets.len() as u64;
-                }
-                "pending" => {
-                    snapshots[0].1.generation = 11;
-                    snapshots[0].1.buckets.clear();
-                    snapshots[0].1.pending_bucket_count = 0;
-                }
-                _ => unreachable!(),
-            }
-            assert!(ScannerRemoteDirtyUsageWindow::verify(before, snapshots, after).is_none(), "{case}");
-        }
-    }
-
-    #[test]
-    fn certification_rejects_owner_replacement_and_a_generation_rollback_between_windows() {
-        let mut changed = window(12, 13, 14);
-        changed.peers.get_mut("peer-b").expect("clean peer").owner_id = uuid::Uuid::from_u128(999).to_string();
-        assert!(ScannerRemoteDirtyUsageCertification::new(window(9, 10, 11), changed).is_none());
-        assert!(ScannerRemoteDirtyUsageCertification::new(window(9, 10, 11), window(8, 9, 10)).is_none());
-    }
-
-    #[test]
-    fn certification_requires_actual_coverage_and_the_same_incarnation_on_every_set() {
-        let proof = ScannerRemoteDirtyUsageCertification::new(window(10, 10, 10), window(10, 10, 10)).expect("stable peers");
-        for case in ["missing_set", "missing_bucket", "wrong_incarnation", "partial"] {
-            let (sources, mut completed) = completions();
-            let mut uncertified = HashSet::new();
-            match case {
-                "missing_set" => {
-                    completed.remove(&DataUsageCacheSource::new(0, 1));
-                }
-                "missing_bucket" => {
-                    completed
-                        .get_mut(&DataUsageCacheSource::new(0, 1))
-                        .expect("set")
-                        .remove("cold-bucket");
-                }
-                "wrong_incarnation" => {
-                    completed
-                        .get_mut(&DataUsageCacheSource::new(0, 1))
-                        .expect("set")
-                        .insert("cold-bucket".to_string(), uuid::Uuid::from_u128(999));
-                }
-                "partial" => {
-                    uncertified.insert("cold-bucket".to_string());
-                }
-                _ => unreachable!(),
-            }
-            let (acks, pending) = proof.classify_buckets(&sources, &completed, &uncertified);
-            assert!(pending.contains("cold-bucket"), "{case}");
-            assert!(acks.iter().all(|ack| matches!(&ack.kind, ScannerDirtyUsageAcknowledgementKind::Scoped { entries, .. } if entries.iter().all(|entry| entry.bucket != "cold-bucket"))), "{case}");
-        }
-    }
-
-    #[test]
-    fn a_bucket_changed_on_one_peer_cannot_be_acknowledged_on_another() {
-        let start = window(10, 10, 10);
-        let mut post_peers = peers(10, 10);
-        post_peers[1].1.generation = 8;
-        post_peers[1].1.pending_bucket_count = 1;
-        post_peers[1].1.buckets.insert(
-            "cold-bucket".to_string(),
-            crate::storage_api::EcstoreScannerPeerDirtyUsageBucket {
-                bucket_incarnation: COLD,
-                generation: 8,
-            },
-        );
-        let mut after = activity(10);
-        after.insert(
-            "peer-b".to_string(),
-            crate::scanner::scanner_node_activity_for_tests(&post_peers[1].1.instance_id, 8, 8, true),
-        );
-        let post = ScannerRemoteDirtyUsageWindow::verify(activity(10), post_peers, after).expect("clean peer became dirty");
-        let proof = ScannerRemoteDirtyUsageCertification::new(start, post).expect("same producers");
-        let (sources, completed) = completions();
-        let (acks, pending) = proof.classify_buckets(&sources, &completed, &HashSet::new());
-        assert!(pending.contains("cold-bucket"));
-        assert!(acks.iter().all(|ack| matches!(&ack.kind, ScannerDirtyUsageAcknowledgementKind::Scoped { entries, .. } if entries.iter().all(|entry| entry.bucket != "cold-bucket"))));
-    }
-
-    #[test]
-    fn start_window_trailing_activity_invalidates_same_cycle_current_cache_without_segment_reuse() {
-        let window = window(9, 10, 11);
-        let old_plan = scanner_bucket_cache_digest(
-            DataUsageScanPlanDigest(crate::scanner::scanner_activity_snapshot_digest(&window.before)),
-            None,
-        );
-        let new_plan = scanner_bucket_cache_digest(
-            DataUsageScanPlanDigest(crate::scanner::scanner_activity_snapshot_digest(&window.activity_for_walk())),
-            None,
-        );
-        let source = DataUsageCacheSource::new(0, 0);
-        let mut cache = DataUsageCache {
-            info: DataUsageCacheInfo {
-                name: "cold-bucket".to_string(),
-                next_cycle: 7,
-                leader_epoch: 11,
-                source: Some(source),
-                last_update: Some(SystemTime::UNIX_EPOCH),
-                snapshot_complete: true,
-                scan_plan_digest: Some(old_plan),
-                cache_key_format: DATA_USAGE_CACHE_KEY_FORMAT,
-                ..Default::default()
-            },
-            ..Default::default()
-        };
-        cache.replace("cold-bucket", "", DataUsageEntry::default());
-        let options = || DataUsageCacheReuseOptions {
-            require_source: true,
-            tier_registry_generation: None,
-            checkpoint_identity: None,
-            checkpoint_plan_digest: None,
-        };
-        assert!(matches!(
-            current_cache_root_or_prepare_with_generation(&mut cache.clone(), "cold-bucket", source, 7, 11, old_plan, options()),
-            DataUsageCacheScanState::Current(_)
-        ));
-        assert!(matches!(
-            current_cache_root_or_prepare_with_generation(&mut cache, "cold-bucket", source, 7, 11, new_plan, options()),
-            DataUsageCacheScanState::Prepared {
-                outcome: DataUsageCachePrepareOutcome::Reset,
-                ..
-            }
-        ));
-        assert!(window.strict_remote_usage().is_none());
-    }
-}

crates/scanner/src/scanner_io/tests.rs

@@ -1332,7 +1332,6 @@ async fn scoped_scan_production_entry_preserves_deep_and_full_maintenance_work()
                     requires_full_scan,
                     resolved_scope_observer: Some(observer),
                     post_walk_gate: None,
-                    peer_observations: None,
                     service_cohort: None,
                 },
             ),
@@ -1424,7 +1423,6 @@ async fn scoped_scan_same_cycle_maintenance_rewalks_after_root_delivery_failure(
                     requires_full_scan: false,
                     resolved_scope_observer: None,
                     post_walk_gate: None,
-                    peer_observations: None,
                     service_cohort: None,
                 },
             ),
@@ -1479,7 +1477,6 @@ async fn scoped_scan_same_cycle_maintenance_rewalks_after_root_delivery_failure(
                     requires_full_scan,
                     resolved_scope_observer: None,
                     post_walk_gate: None,
-                    peer_observations: None,
                     service_cohort: None,
                 },
             ),
@@ -2471,7 +2468,6 @@ async fn set_snapshot_reuse_requires_execution_identity_and_fences_stale_writers
             publication_epoch: Some(epoch),
             dirty_usage_buckets: Arc::new(HashMap::new()),
             bucket_failures: ScannerBucketFailureState::default(),
-            bucket_completions: None,
             pending_maintenance_work: Arc::new(AtomicBool::new(false)),
             cache_cycle_floor: Arc::new(AtomicU64::new(8)),
             cold_zero_walk_reuse_observed: Arc::new(AtomicBool::new(false)),
@@ -3389,7 +3385,6 @@ async fn distributed_scoped_scan_falls_back_when_remote_ack_exceeds_protocol_bat
         store.as_ref(),
         true,
         super::io_cycle::ScannerBucketScopeResolution {
-            remote_window: None,
             requested_scope: ScannerBucketScanScope::default(),
             baseline_proof: ScannerCacheBaselineProof {
                 authoritative_data: Some(&baseline),
@@ -3449,7 +3444,6 @@ async fn distributed_scoped_scan_falls_back_when_remote_scoped_ack_capability_is
             store.as_ref(),
             true,
             super::io_cycle::ScannerBucketScopeResolution {
-                remote_window: None,
                 requested_scope: ScannerBucketScanScope::default(),
                 baseline_proof: ScannerCacheBaselineProof {
                     authoritative_data: Some(&baseline),
@@ -5021,7 +5015,6 @@ async fn replication_maintenance_refreshes_compacted_stats_across_cycles_and_ret
                 service_cohort: None,
                 resolved_scope_observer: None,
                 post_walk_gate: None,
-                peer_observations: None,
             },
         )
         .await
@@ -5129,7 +5122,6 @@ async fn replication_maintenance_refreshes_compacted_stats_across_cycles_and_ret
                 service_cohort: None,
                 resolved_scope_observer: None,
                 post_walk_gate: None,
-                peer_observations: None,
             },
         )
         .await
@@ -5277,7 +5269,6 @@ async fn run_full_cycle(store: &Arc<ECStore>, cycle: u64) -> ScannerCycleStatus
                 resolved_scope_observer: None,
                 service_cohort: None,
                 post_walk_gate: None,
-                peer_observations: None,
             },
         ),
     )

crates/scanner/src/scanner_io/tests/checkpoint_coverage_identity.rs

@@ -54,7 +54,6 @@ async fn run_full_cycle(
                 service_cohort: None,
                 resolved_scope_observer: None,
                 post_walk_gate: None,
-                peer_observations: None,
             },
         ),
     )

crates/scanner/src/scanner_io/tests/collector_lifecycle.rs

@@ -140,7 +140,6 @@ async fn scan_plan(set: &Arc<SetDisks>, names: &[&str], cycle: u64) -> ScannerBu
         ),
         dirty_usage_buckets: Arc::new(dirty_usage_buckets_for_tests()),
         bucket_failures: ScannerBucketFailureState::default(),
-        bucket_completions: None,
         pending_maintenance_work: Arc::new(AtomicBool::new(false)),
         cache_cycle_floor: Arc::new(AtomicU64::new(cycle)),
         cold_zero_walk_reuse_observed: Arc::new(AtomicBool::new(false)),

crates/scanner/src/scanner_io/tests/listing_window_scope.rs

@@ -47,7 +47,6 @@ async fn run_cycle(
                 service_cohort: None,
                 resolved_scope_observer: Some(observer),
                 post_walk_gate: None,
-                peer_observations: None,
             },
         ),
     )

crates/scanner/src/scanner_io/tests/partial_bucket_carry.rs

@@ -57,7 +57,6 @@ async fn run_cycle(store: &Arc<ECStore>, cycle: u64, options: CycleOptions) -> (
                 service_cohort: None,
                 resolved_scope_observer: options.resolved_scope_observer,
                 post_walk_gate: None,
-                peer_observations: None,
             },
         ),
     )

crates/scanner/src/scanner_io/tests/per_bucket_certification.rs

@@ -22,7 +22,6 @@ async fn put_fixture_object(store: &Arc<ECStore>, bucket: &str, object: &str) {
 struct CycleHooks {
     resolved_scope_observer: Option<tokio::sync::oneshot::Sender<ScannerBucketScanScope>>,
     post_walk_gate: Option<tokio::sync::oneshot::Receiver<()>>,
-    peer_observations: Option<super::super::io_cycle::ScannerCyclePeerObservations>,
 }
 
 async fn run_cycle(
@@ -32,7 +31,6 @@ async fn run_cycle(
     requires_full_scan: bool,
     hooks: CycleHooks,
 ) -> (ScannerCycleResult, DataUsageInfo) {
-    let peer_cycle = hooks.peer_observations.is_some();
     let ctx = CancellationToken::new();
     let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default());
     let (updates, mut receiver) = mpsc::channel(1);
@@ -54,20 +52,12 @@ async fn run_cycle(
                 service_cohort: None,
                 resolved_scope_observer: hooks.resolved_scope_observer,
                 post_walk_gate: hooks.post_walk_gate,
-                peer_observations: hooks.peer_observations,
             },
         ),
     )
     .await
     .expect("cycle should finish within the test deadline")
     .expect("cycle should succeed");
-    if peer_cycle {
-        assert_eq!(
-            result.status,
-            ScannerCycleStatus::Complete,
-            "one remote hot bucket cannot supersede certified cold buckets"
-        );
-    }
     let snapshot = receiver.recv().await.expect("cycle should publish a snapshot");
     assert!(receiver.recv().await.is_none(), "a cycle publishes one terminal snapshot");
     (result, snapshot)
@@ -101,7 +91,6 @@ async fn write_during_walk_leaves_only_that_bucket_pending_and_scopes_the_next_c
         CycleHooks {
             resolved_scope_observer: None,
             post_walk_gate: None,
-            peer_observations: None,
         },
     )
     .await;
@@ -139,7 +128,6 @@ async fn write_during_walk_leaves_only_that_bucket_pending_and_scopes_the_next_c
         CycleHooks {
             resolved_scope_observer: Some(observer),
             post_walk_gate: Some(gate),
-            peer_observations: None,
         },
     )
     .await;
@@ -187,7 +175,6 @@ async fn write_during_walk_leaves_only_that_bucket_pending_and_scopes_the_next_c
         CycleHooks {
             resolved_scope_observer: Some(observer),
             post_walk_gate: None,
-            peer_observations: None,
         },
     )
     .await;
@@ -239,7 +226,6 @@ async fn pending_put_tail_across_the_walk_no_longer_vetoes_the_cycle() {
         CycleHooks {
             resolved_scope_observer: None,
             post_walk_gate: None,
-            peer_observations: None,
         },
     )
     .await;
@@ -299,7 +285,6 @@ async fn guardrail_veto_still_supersedes_the_cycle_onto_the_observed_route() {
         CycleHooks {
             resolved_scope_observer: Some(observer),
             post_walk_gate: Some(gate),
-            peer_observations: None,
         },
     )
     .await;
@@ -321,230 +306,3 @@ async fn guardrail_veto_still_supersedes_the_cycle_onto_the_observed_route() {
     );
     clear_dirty_usage_buckets_for_tests();
 }
-
-#[tokio::test]
-#[serial]
-async fn remote_hot_bucket_keeps_cold_buckets_certified_and_scopes_the_next_cycle() {
-    run_remote_certification_cycle(false).await;
-}
-
-#[tokio::test]
-#[serial]
-async fn a_verified_empty_ack_on_a_full_scope_keeps_the_remote_hot_bucket_pending() {
-    run_remote_certification_cycle(true).await;
-}
-
-async fn run_remote_certification_cycle(empty_ack: bool) {
-    let (_temp_dir, store) = setup_local_scanner_store(1, 1).await;
-    clear_dirty_usage_buckets_for_tests();
-    let mut incarnations = BTreeMap::new();
-    for bucket in BUCKETS {
-        store
-            .make_bucket(bucket, &MakeBucketOptions::default())
-            .await
-            .expect("fixture bucket");
-        put_fixture_object(&store, bucket, "initial").await;
-        record_dirty_usage_bucket(bucket);
-        incarnations.insert(
-            bucket.to_string(),
-            store.all_set_disks()[0]
-                .find_bucket_incarnation_id_from_disk(bucket)
-                .await
-                .expect("read actual incarnation")
-                .expect("fixture bucket has an incarnation"),
-        );
-    }
-    wait_for_namespace_commit_tails(store.as_ref()).await;
-    let peers = Arc::new(StdMutex::new(BTreeMap::from([
-        (
-            "peer-a:9000".to_string(),
-            peer_dirty_usage_snapshot(&Uuid::from_u128(101).to_string(), 7, true, &[]),
-        ),
-        (
-            "peer-b:9000".to_string(),
-            peer_dirty_usage_snapshot(&Uuid::from_u128(102).to_string(), 7, true, &[]),
-        ),
-    ])));
-    // These are the RPC observations only. Bucket listing, traversal, cache
-    // coverage, classification and candidate production use the real cycle.
-    let activity_peers = Arc::clone(&peers);
-    let dirty_peers = Arc::clone(&peers);
-    let observations = super::super::io_cycle::ScannerCyclePeerObservations {
-        activity: Arc::new(move || {
-            activity_peers
-                .lock()
-                .expect("peer fixture lock")
-                .iter()
-                .map(|(host, peer)| {
-                    (
-                        host.clone(),
-                        crate::scanner::scanner_node_activity_for_tests(
-                            &peer.instance_id,
-                            peer.generation,
-                            peer.generation,
-                            !peer.buckets.is_empty(),
-                        ),
-                    )
-                })
-                .collect()
-        }),
-        dirty_usage: Arc::new(move || {
-            dirty_peers
-                .lock()
-                .expect("peer fixture lock")
-                .iter()
-                .map(|(host, snapshot)| (host.clone(), snapshot.clone()))
-                .collect()
-        }),
-        scoped_ack_capable: true,
-    };
-    let (result, baseline) = run_cycle(
-        &store,
-        1,
-        None,
-        true,
-        CycleHooks {
-            resolved_scope_observer: None,
-            post_walk_gate: None,
-            peer_observations: Some(observations.clone()),
-        },
-    )
-    .await;
-    assert_eq!(result.status, ScannerCycleStatus::Complete);
-    assert!(result.clear_verified_usage().is_empty());
-    assert!(baseline.is_complete_bucket_usage_snapshot());
-    let initial_dirty = if empty_ack {
-        vec!["hot-bucket"]
-    } else {
-        vec!["cold-bucket", "idle-bucket"]
-    };
-    for bucket in initial_dirty {
-        put_fixture_object(&store, bucket, "second").await;
-        let mut locked = peers.lock().expect("peer fixture lock");
-        let peer = locked.get_mut("peer-a:9000").expect("remote producer exists");
-        peer.generation += 1;
-        peer.buckets.insert(
-            bucket.to_string(),
-            crate::storage_api::EcstoreScannerPeerDirtyUsageBucket {
-                bucket_incarnation: incarnations[bucket],
-                generation: peer.generation,
-            },
-        );
-        peer.pending_bucket_count = u64::try_from(peer.buckets.len()).expect("bounded fixture");
-    }
-    wait_for_namespace_commit_tails(store.as_ref()).await;
-    let (observer, resolved_scope) = tokio::sync::oneshot::channel();
-    let (release, gate) = tokio::sync::oneshot::channel();
-    let writer_store = Arc::clone(&store);
-    let writer_peers = Arc::clone(&peers);
-    let hot_incarnation = incarnations["hot-bucket"];
-    let writer = tokio::spawn(async move {
-        let scope = resolved_scope.await.expect("real cycle resolved its scope");
-        put_fixture_object(&writer_store, "hot-bucket", "during-walk").await;
-        {
-            let mut locked = writer_peers.lock().expect("peer fixture lock");
-            let peer = locked.get_mut("peer-a:9000").expect("remote producer exists");
-            peer.generation += 1;
-            peer.buckets.insert(
-                "hot-bucket".to_string(),
-                crate::storage_api::EcstoreScannerPeerDirtyUsageBucket {
-                    bucket_incarnation: hot_incarnation,
-                    generation: peer.generation,
-                },
-            );
-            peer.pending_bucket_count = u64::try_from(peer.buckets.len()).expect("bounded fixture");
-        }
-        wait_for_namespace_commit_tails(writer_store.as_ref()).await;
-        release.send(()).expect("cycle awaits its actual post-walk proof");
-        scope
-    });
-    let (result, published) = run_cycle(
-        &store,
-        2,
-        (!empty_ack).then(|| baseline_bytes(&baseline)),
-        false,
-        CycleHooks {
-            resolved_scope_observer: Some(observer),
-            post_walk_gate: Some(gate),
-            peer_observations: Some(observations.clone()),
-        },
-    )
-    .await;
-    let scope = writer.await.expect("writer completed");
-    if empty_ack {
-        assert!(scope.is_default(), "this case exercises the full-scope generation fallback");
-    } else {
-        assert_eq!(
-            scope.selected_buckets_for_tests(),
-            Some(&HashSet::from(["cold-bucket".to_string(), "idle-bucket".to_string()]))
-        );
-    }
-    assert_eq!(
-        result.status,
-        ScannerCycleStatus::Complete,
-        "one remote hot bucket cannot supersede certified cold buckets"
-    );
-    assert_eq!(published.usage_snapshot_pending_buckets, vec!["hot-bucket".to_string()]);
-    assert_eq!(published.usage_snapshot_converged, Some(false));
-    assert!(!published.usage_snapshot_observational);
-    assert_eq!(published.buckets_usage["cold-bucket"].objects_count, if empty_ack { 1 } else { 2 });
-    assert_eq!(published.buckets_usage["idle-bucket"].objects_count, if empty_ack { 1 } else { 2 });
-    if !empty_ack {
-        assert_eq!(published.buckets_usage["hot-bucket"].objects_count, 1);
-    }
-    assert!(result.has_remote_dirty_usage_pending());
-    let acknowledgements = &result.remote_dirty_usage_acknowledgements;
-    if empty_ack {
-        assert!(acknowledgements.is_empty(), "a verified empty scoped ACK cannot become a generation ACK");
-    } else {
-        assert_eq!(acknowledgements.len(), 1, "the clean peer is proved without a fabricated ACK");
-        let crate::scanner::ScannerDirtyUsageAcknowledgementKind::Scoped { owner_id, entries } = &acknowledgements[0].kind else {
-            panic!("certified remote buckets require exact scoped ACKs");
-        };
-        assert_eq!(
-            entries.iter().map(|entry| entry.bucket.as_str()).collect::<BTreeSet<_>>(),
-            BTreeSet::from(["cold-bucket", "idle-bucket"])
-        );
-        {
-            let mut locked = peers.lock().expect("peer fixture lock");
-            let peer = locked
-                .get_mut(&acknowledgements[0].host)
-                .expect("ACK targets the original peer");
-            assert_eq!(&peer.owner_id, owner_id);
-            assert_eq!(peer.instance_id, acknowledgements[0].instance_id);
-            // Model the RPC receiver only after asserting the production sender's
-            // exact scope; this does not claim runtime publication/receiver proof.
-            for entry in entries {
-                let current = peer.buckets.get(&entry.bucket).expect("ACK names an observed tuple");
-                assert_eq!(
-                    (current.bucket_incarnation, current.generation),
-                    (entry.bucket_incarnation, entry.generation)
-                );
-                peer.buckets.remove(&entry.bucket);
-            }
-            peer.pending_bucket_count = u64::try_from(peer.buckets.len()).expect("bounded fixture");
-        }
-    }
-    let _ = result.clear_verified_usage();
-    let (observer, resolved_scope) = tokio::sync::oneshot::channel();
-    let (result, next) = run_cycle(
-        &store,
-        3,
-        Some(baseline_bytes(&published)),
-        false,
-        CycleHooks {
-            resolved_scope_observer: Some(observer),
-            post_walk_gate: None,
-            peer_observations: Some(observations),
-        },
-    )
-    .await;
-    assert_eq!(
-        resolved_scope.await.expect("next cycle scope").selected_buckets_for_tests(),
-        Some(&HashSet::from(["hot-bucket".to_string()]))
-    );
-    assert_eq!(result.status, ScannerCycleStatus::Complete);
-    assert_eq!(next.buckets_usage["hot-bucket"].objects_count, if empty_ack { 3 } else { 2 });
-    assert_eq!(next.usage_snapshot_converged, Some(true));
-    clear_dirty_usage_buckets_for_tests();
-}

crates/scanner/src/scanner_io/tests/scoped_entry_fallback.rs

@@ -139,7 +139,6 @@ async fn run_entry(
                 service_cohort: None,
                 resolved_scope_observer: Some(observer),
                 post_walk_gate: None,
-                peer_observations: None,
             },
         ),
     )

crates/scanner/src/scanner_io/tests/service_cohort.rs

@@ -74,7 +74,6 @@ async fn run_cohort_cycle(
                 service_cohort: Some(cohort),
                 resolved_scope_observer: None,
                 post_walk_gate: None,
-                peer_observations: None,
             },
         ),
     )

crates/scanner/src/storage_api.rs

@@ -116,10 +116,8 @@ pub(crate) use rustfs_ecstore::api::layout::{
     EndpointServerPools as EcstoreEndpointServerPools, Endpoints as EcstoreEndpoints, PoolEndpoints as EcstorePoolEndpoints,
     SetupType as EcstoreSetupType,
 };
+pub(crate) use rustfs_ecstore::api::notification::NotificationSys as EcstoreNotificationSys;
 pub(crate) use rustfs_ecstore::api::notification::scanner_peer_transport_error_message_is_retryable;
-pub(crate) use rustfs_ecstore::api::notification::{
-    NotificationSys as EcstoreNotificationSys, ScannerPublicationLeaseGrant as EcstoreScannerPublicationLeaseGrant,
-};
 pub(crate) use rustfs_ecstore::api::object::{
     SCANNER_PUBLICATION_LEASE_FENCE_METADATA_KEY, ScannerPublicationCommitScope, ScannerPublicationCommitState,
 };
@@ -128,6 +126,8 @@ pub(crate) use rustfs_ecstore::api::rebalance::{
     RebalStatus as EcstoreRebalStatus, RebalanceInfo as EcstoreRebalanceInfo, RebalanceMeta as EcstoreRebalanceMeta,
     RebalanceStats as EcstoreRebalanceStats,
 };
+#[cfg(test)]
+pub(crate) use rustfs_ecstore::api::rpc::ScannerPeerDirtyUsageBucket as EcstoreScannerPeerDirtyUsageBucket;
 pub(crate) use rustfs_ecstore::api::rpc::{
     ScannerBucketListing as EcstoreScannerBucketListing,
     ScannerDirtyUsageAcknowledgement as EcstoreScannerDirtyUsageAcknowledgement,
@@ -135,10 +135,6 @@ pub(crate) use rustfs_ecstore::api::rpc::{
     ScannerScopedDirtyUsageAckEntry as EcstoreScannerScopedDirtyUsageAckEntry,
 };
 #[cfg(test)]
-pub(crate) use rustfs_ecstore::api::rpc::{
-    ScannerPeerDirtyUsageBucket as EcstoreScannerPeerDirtyUsageBucket, ScannerPublicationLease as EcstoreScannerPublicationLease,
-};
-#[cfg(test)]
 pub(crate) use rustfs_ecstore::api::runtime::InstanceContext as EcstoreInstanceContext;
 #[cfg(test)]
 pub(crate) use rustfs_ecstore::api::runtime::set_object_store_resolver as ecstore_set_object_store_resolver;
@@ -384,11 +380,6 @@ pub(crate) mod scan {
         SCANNER_DIRTY_USAGE_SNAPSHOT_PROTOCOL_VERSION, SCANNER_DIRTY_USAGE_SNAPSHOT_RPC_MAX_MESSAGE_SIZE,
         SCANNER_SCOPED_DIRTY_USAGE_ACK_MAX_ENTRIES,
     };
-    #[cfg(test)]
-    pub(crate) use super::{
-        EcstoreScannerPeerDirtyUsageBucket, EcstoreScannerPeerDirtyUsageSnapshot, EcstoreScannerPublicationLease,
-        EcstoreScannerPublicationLeaseGrant,
-    };
 }
 
 pub(crate) mod scanner_io {

rustfs/Cargo.toml

@@ -68,7 +68,7 @@ license = []
 io-scheduler-debug = []  # Enable debug information in I/O scheduler
 tracing-chunk-debug = []  # Enable per-chunk tracing in data plane (high noise, for debugging only)
 full = ["metrics-gpu", "ftps", "swift", "webdav", "sftp", "pyroscope", "gcs"]
-e2e-test-hooks = ["rustfs-ecstore/e2e-test-hooks", "rustfs-scanner/e2e-test-hooks"]
+e2e-test-hooks = ["rustfs-ecstore/e2e-test-hooks"]
 # Shortens Connect credentials only in debug E2E builds.
 connect-e2e-short-credentials = []
 # Builds the dedicated rustfs-cli-e2e target with a build-time public enrollment root.

rustfs/src/app/lifecycle_transition_api_test.rs

@@ -2895,8 +2895,8 @@ async fn put_object_extract_computes_replication_decision_per_entry() {
 /// replication themselves. Without it a retention or legal hold applied after upload stays
 /// local and the replica keeps its previous, unprotected state (a WORM object that is still
 /// deletable on the peer). This observes the shared decision counter: each handler must
-/// compute a replication decision exactly once, which is the step that drives both the
-/// persisted pending marker and the schedule.
+/// compute a replication decision exactly once when replication is configured, which is
+/// the step that drives both the persisted pending marker and the schedule.
 #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
 #[serial]
 #[ignore = "global-state usecase integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"]
@@ -2945,7 +2945,27 @@ async fn object_lock_handlers_schedule_replication() {
          retention change never replicates and the peer copy stays unprotected"
     );
 
-    // PutObjectLegalHold must do the same.
+    // A bucket without a replication configuration can skip the decision.
+    MUST_REPLICATE_OBJECT_CALLS.store(0, Ordering::SeqCst);
+    let no_replication_input = PutObjectLegalHoldInput::builder()
+        .bucket(bucket.clone())
+        .key(object.to_string())
+        .legal_hold(Some(ObjectLockLegalHold {
+            status: Some(ObjectLockLegalHoldStatus::from_static(ObjectLockLegalHoldStatus::OFF)),
+        }))
+        .build()
+        .unwrap();
+    fs.put_object_legal_hold(build_request(no_replication_input, Method::PUT))
+        .await
+        .expect("legal hold without replication should succeed");
+    assert_eq!(
+        MUST_REPLICATE_OBJECT_CALLS.load(Ordering::SeqCst),
+        0,
+        "a bucket without replication must skip the decision"
+    );
+
+    // A configured bucket must still evaluate the metadata replication decision.
+    install_delete_replication_config(&bucket, true, false, None).await;
     MUST_REPLICATE_OBJECT_CALLS.store(0, Ordering::SeqCst);
     let legal_hold_input = PutObjectLegalHoldInput::builder()
         .bucket(bucket.clone())

rustfs/src/app/object/copy.rs

@@ -385,12 +385,6 @@ impl DefaultObjectUsecase {
             dst_opts.object_lock_config_snapshot =
                 Some(store.object_lock_config_snapshot(&bucket).await.map_err(ApiError::from)?);
         }
-        let mut current_opts: ObjectOptions = internal_object_info_lookup_opts(
-            get_opts(&bucket, &key, dest_version_id.clone(), None, &req.headers)
-                .await
-                .map_err(ApiError::from)?,
-        );
-
         // Hold the self-copy namespace write lock before opening the source reader, including
         // expected-current historical copies. With lock optimization disabled, the source
         // stream retains its read guard until EOF; taking the write lock later in ECStore
@@ -410,28 +404,42 @@ impl DefaultObjectUsecase {
         }
         dst_opts.expected_current_version_id = expected_current_version_id.clone();
 
-        if _self_copy_lock_guard.is_some() {
-            current_opts.no_lock = true;
-        }
-        let previous_current_sizes = match store.get_object_info(&bucket, &key, &current_opts).await {
-            Ok(existing_obj_info) => {
-                validate_existing_object_lock_for_write(&object_lock_config_state, &existing_obj_info, &dst_opts)?;
-                if let Some(expected) = expected_current_version_id.as_deref()
-                    && existing_obj_info.version_id.unwrap_or_default().to_string() != expected
-                {
-                    return Err(s3_error!(PreconditionFailed));
-                }
-                Some((existing_obj_info.size.max(0) as u64, quota_object_size(&existing_obj_info)))
+        let copy_prelookup_required = self_copy_lock
+            || expected_current_version_id.is_some()
+            || dest_version_id.is_some()
+            || (!super::shared::put_like_write_creates_new_version(&dst_opts)
+                && object_lock_checks_required_for_state(&object_lock_config_state));
+        let previous_current_sizes: Option<Option<(u64, Result<u64, StorageError>)>> = if copy_prelookup_required {
+            let mut current_opts: ObjectOptions = internal_object_info_lookup_opts(
+                get_opts(&bucket, &key, dest_version_id.clone(), None, &req.headers)
+                    .await
+                    .map_err(ApiError::from)?,
+            );
+            if _self_copy_lock_guard.is_some() {
+                current_opts.no_lock = true;
             }
-            Err(err) => {
-                if expected_current_version_id.is_some() {
-                    return Err(s3_error!(PreconditionFailed));
+            Some(match store.get_object_info(&bucket, &key, &current_opts).await {
+                Ok(existing_obj_info) => {
+                    validate_existing_object_lock_for_write(&object_lock_config_state, &existing_obj_info, &dst_opts)?;
+                    if let Some(expected) = expected_current_version_id.as_deref()
+                        && existing_obj_info.version_id.unwrap_or_default().to_string() != expected
+                    {
+                        return Err(s3_error!(PreconditionFailed));
+                    }
+                    Some((existing_obj_info.size.max(0) as u64, quota_object_size(&existing_obj_info)))
                 }
-                if !is_err_object_not_found(&err) && !is_err_version_not_found(&err) {
-                    return Err(ApiError::from(err).into());
+                Err(err) => {
+                    if expected_current_version_id.is_some() {
+                        return Err(s3_error!(PreconditionFailed));
+                    }
+                    if !is_err_object_not_found(&err) && !is_err_version_not_found(&err) {
+                        return Err(ApiError::from(err).into());
+                    }
+                    None
                 }
-                None
-            }
+            })
+        } else {
+            None
         };
 
         let (mut effective_sse, mut effective_kms_key_id) = resolve_bucket_default_sse(
@@ -753,10 +761,11 @@ impl DefaultObjectUsecase {
         if let Some(quota_check) = quota_check.as_ref() {
             apply_quota_admission(&mut dst_opts, quota_check)?;
         }
-        let previous_current_size = match previous_current_sizes {
-            Some((_, Ok(logical_size))) if quota_enabled => Some(logical_size),
-            Some((_, Err(err))) if quota_enabled => return Err(ApiError::from(err).into()),
-            Some((physical_size, _)) => Some(physical_size),
+        let prelookup_previous_current_size: Option<Option<u64>> = match previous_current_sizes {
+            Some(Some((_, Ok(logical_size)))) if quota_enabled => Some(Some(logical_size)),
+            Some(Some((_, Err(err)))) if quota_enabled => return Err(ApiError::from(err).into()),
+            Some(Some((physical_size, _))) => Some(Some(physical_size)),
+            Some(None) => Some(None),
             None => None,
         };
         if let Some(quota_check) = quota_check.as_ref() {
@@ -785,8 +794,8 @@ impl DefaultObjectUsecase {
                 let _self_copy_lock_guard = _self_copy_lock_guard;
                 let _table_data_plane_publication_guards = table_data_plane_publication_guards;
 
-                let oi = store
-                    .copy_object(&src_bucket, &src_key, &bucket, &key, &mut src_info, &src_opts, &dst_opts)
+                let (oi, backfilled_old_current_size) = store
+                    .copy_object_with_old_current_size(&src_bucket, &src_key, &bucket, &key, &mut src_info, &src_opts, &dst_opts)
                     .await
                     .map_err(ApiError::from)?;
 
@@ -800,13 +809,22 @@ impl DefaultObjectUsecase {
                 maybe_enqueue_transition_immediate(&oi, LcEventSrc::S3CopyObject).await;
                 let _ = invalidate_object_data_cache_after_copy_success(&cache_adapter, &bucket, &key).await;
 
-                let dest_versioned = BucketVersioningSys::prefix_enabled(&bucket, &key).await;
+                let dest_versioned = dst_opts.versioned;
                 if has_bucket_metadata {
                     let committed_size = quota_accounting_object_size(&oi, quota_enabled)?;
-                    if dest_versioned {
-                        record_bucket_object_version_write_memory(&bucket, previous_current_size, committed_size).await;
-                    } else {
-                        record_bucket_object_write_memory(&bucket, previous_current_size, committed_size).await;
+                    match prelookup_previous_current_size
+                        .or_else(|| previous_current_size_from_backfill(backfilled_old_current_size))
+                    {
+                        Some(previous_current_size) => {
+                            if dest_versioned {
+                                record_bucket_object_version_write_memory(&bucket, previous_current_size, committed_size).await;
+                            } else {
+                                record_bucket_object_write_memory(&bucket, previous_current_size, committed_size).await;
+                            }
+                        }
+                        None => {
+                            record_bucket_object_write_unknown_previous_memory(&bucket, committed_size, dest_versioned).await;
+                        }
                     }
                 }
 

rustfs/src/app/object/delete.rs

@@ -1155,15 +1155,21 @@ impl DefaultObjectUsecase {
         let replicate_force_delete = force_delete && !replica && has_active_delete_rule(&delete_config_snapshot, &key);
         let mut force_delete_intent = None;
 
-        let existing_object_info = match store.get_object_info_for_delete(&bucket, &key, &opts).await {
-            Ok(obj_info) => Some(obj_info),
-            Err(err) => {
-                // If object not found, allow deletion to proceed (will return 204 No Content)
-                if !is_err_object_not_found(&err) && !is_err_version_not_found(&err) {
-                    return Err(ApiError::from(err).into());
+        // A version purge needs the original marker identity for replication.
+        let need_eager_pre_stat = opts.http_preconditions.is_some() || opts.version_id.is_some();
+        let existing_object_info = if need_eager_pre_stat {
+            match Box::pin(store.get_object_info_for_delete(&bucket, &key, &opts)).await {
+                Ok(obj_info) => Some(obj_info),
+                Err(err) => {
+                    // If object not found, allow deletion to proceed (will return 204 No Content)
+                    if !is_err_object_not_found(&err) && !is_err_version_not_found(&err) {
+                        return Err(ApiError::from(err).into());
+                    }
+                    None
                 }
-                None
             }
+        } else {
+            None
         };
         #[cfg(test)]
         wait_for_delete_source_test_hook(&bucket).await;
@@ -1217,16 +1223,21 @@ impl DefaultObjectUsecase {
                         return result;
                     }
 
-                    if matches!(&err, StorageError::PrefixAccessDenied(_, _))
-                        && let Some(existing_object_info) = existing_object_info.as_ref()
-                        && let Some(reason) = check_object_lock_for_deletion(
-                            &bucket,
-                            existing_object_info,
-                            has_bypass_governance_header(&req.headers),
-                        )
-                        .await
-                    {
-                        return Err(S3Error::with_message(S3ErrorCode::AccessDenied, reason.error_message()));
+                    if matches!(&err, StorageError::PrefixAccessDenied(_, _)) {
+                        let existing_object_info = match existing_object_info {
+                            Some(info) => Some(info),
+                            None => Box::pin(store.get_object_info_for_delete(&bucket, &key, &opts)).await.ok(),
+                        };
+                        if let Some(existing_object_info) = existing_object_info.as_ref()
+                            && let Some(reason) = check_object_lock_for_deletion(
+                                &bucket,
+                                existing_object_info,
+                                has_bypass_governance_header(&req.headers),
+                            )
+                            .await
+                        {
+                            return Err(S3Error::with_message(S3ErrorCode::AccessDenied, reason.error_message()));
+                        }
                     }
 
                     return Err(ApiError::from(err).into());

rustfs/src/app/object/put.rs

@@ -1734,7 +1734,9 @@ impl DefaultObjectUsecase {
         // replication), the lookup is skipped and accounting is backfilled from
         // the dst xl.meta that rename_data already reads, saving a full-disk
         // metadata fanout per PUT.
-        let prelookup_required = version_id.is_some() || object_lock_checks_required_for_state(&object_lock_config_state);
+        let prelookup_required = version_id.is_some()
+            || (!super::shared::put_like_write_creates_new_version(&opts)
+                && object_lock_checks_required_for_state(&object_lock_config_state));
         #[cfg(test)]
         let prelookup_required = prelookup_required || injected_put_storage_failure(PutStorageTestStage::Prelookup).is_some();
         // Outer None = prelookup skipped (accounting comes from the commit

rustfs/src/app/object/shared.rs

@@ -874,7 +874,7 @@ pub(crate) fn build_put_like_object_lock_metadata(
     Ok(Some(eval_metadata))
 }
 
-fn put_like_write_creates_new_version(opts: &ObjectOptions) -> bool {
+pub(super) fn put_like_write_creates_new_version(opts: &ObjectOptions) -> bool {
     opts.version_id.is_none() && opts.versioned && !opts.version_suspended
 }
 

rustfs/src/storage/ecfs.rs

@@ -1815,12 +1815,13 @@ impl S3 for FS {
             return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
         };
 
-        let _ = store
-            .get_bucket_info(&bucket, &BucketOptions::default())
-            .await
-            .map_err(ApiError::from)?;
-
-        validate_bucket_object_lock_enabled(&bucket).await?;
+        if let Err(err) = validate_bucket_object_lock_enabled(&bucket).await {
+            let _ = store
+                .get_bucket_info(&bucket, &BucketOptions::default())
+                .await
+                .map_err(ApiError::from)?;
+            return Err(err);
+        }
 
         let opts: ObjectOptions = get_opts(&bucket, &key, version_id, None, &req.headers)
             .await
@@ -1841,19 +1842,22 @@ impl S3 for FS {
         // once BEFORE the commit and persist the pending marker with it, so a restart between
         // the commit and the worker write-back leaves a Pending status on disk for the scanner
         // to re-drive. Scheduling without that marker would lose the task silently.
-        let dsc = match store.get_object_info(&bucket, &key, &opts).await.ok() {
-            Some(info) => {
-                must_replicate_metadata(
-                    &bucket,
-                    &key,
-                    &info.user_defined,
-                    info.user_tags.as_ref().clone(),
-                    popts.delete_marker_replication_status(),
-                    popts.clone(),
-                )
-                .await
-            }
-            None => ReplicateDecision::new(),
+        let dsc = if crate::app::storage_api::bucket::metadata_sys::get_replication_config(&bucket)
+            .await
+            .is_ok()
+            && let Ok(info) = store.get_object_info(&bucket, &key, &opts).await
+        {
+            must_replicate_metadata(
+                &bucket,
+                &key,
+                &info.user_defined,
+                info.user_tags.as_ref().clone(),
+                popts.delete_marker_replication_status(),
+                popts.clone(),
+            )
+            .await
+        } else {
+            ReplicateDecision::new()
         };
 
         let mut eval_metadata = parse_object_lock_legal_hold(legal_hold)?;

rustfs/src/storage/rpc/node_service.rs

@@ -1398,13 +1398,8 @@ impl scanner_control_service_server::ScannerControlService for NodeService {
                 .zip(&request.entries)
                 .map(|(guard, entry)| (guard, entry.generation))
                 .collect::<Vec<_>>();
-            #[cfg(feature = "e2e-test-hooks")]
-            observe_scoped_dirty_usage_ack("ack_before_apply", &request, None).await?;
-            let cleared = rustfs_scanner::acknowledge_scoped_dirty_usage(&request.instance_id, &entries, request.probe_only)
-                .map_err(|err| Status::failed_precondition(err.to_string()))?;
-            #[cfg(feature = "e2e-test-hooks")]
-            observe_scoped_dirty_usage_ack("ack_after_apply", &request, Some(cleared)).await?;
-            Ok::<_, Status>(cleared)
+            rustfs_scanner::acknowledge_scoped_dirty_usage(&request.instance_id, &entries, request.probe_only)
+                .map_err(|err| Status::failed_precondition(err.to_string()))
         })
         .await
         .map_err(|_| Status::deadline_exceeded("scoped dirty usage incarnation validation timed out"))??;
@@ -1427,44 +1422,6 @@ impl scanner_control_service_server::ScannerControlService for NodeService {
     }
 }
 
-/// Capture the real compare-and-remove boundary while the production
-/// incarnation fences are still held. Capability probes never pause here.
-#[cfg(feature = "e2e-test-hooks")]
-async fn observe_scoped_dirty_usage_ack(
-    stage: &str,
-    request: &ScannerScopedDirtyUsageAckRequest,
-    cleared: Option<u64>,
-) -> Result<(), Status> {
-    if request.probe_only {
-        return Ok(());
-    }
-    let dirty =
-        rustfs_scanner::scanner_dirty_usage_snapshot(rustfs_protos::scoped_dirty_usage::SCOPED_DIRTY_USAGE_MAX_ENTRIES as usize);
-    let detail = serde_json::json!({
-        "owner_id": request.owner_id,
-        "instance_id": request.instance_id,
-        "probe_only": request.probe_only,
-        "entries": request.entries.iter().map(|entry| serde_json::json!({
-            "bucket": entry.bucket,
-            "bucket_incarnation": Uuid::from_slice(entry.bucket_incarnation.as_ref()).ok().map(|value| value.to_string()),
-            "generation": entry.generation,
-        })).collect::<Vec<_>>(),
-        "cleared": cleared,
-        "dirty_snapshot": {
-            "generation": dirty.generation,
-            "pending_bucket_count": dirty.pending_bucket_count,
-            "complete": dirty.complete,
-            "buckets": dirty.buckets.iter().map(|bucket| serde_json::json!({
-                "bucket": bucket.bucket,
-                "generation": bucket.generation,
-            })).collect::<Vec<_>>(),
-        },
-    });
-    rustfs_scanner::e2e_barrier::checkpoint(stage, detail)
-        .await
-        .map_err(|error| Status::internal(format!("scoped dirty usage test barrier failed: {error}")))
-}
-
 #[tonic::async_trait]
 impl Node for NodeService {
     async fn ping(&self, request: Request<PingRequest>) -> Result<Response<PingResponse>, Status> {

scripts/ci_test_shards.py

@@ -73,7 +73,7 @@ def partition(listing, junit, protected, count=2):
             durations[key] = seconds
     require(set(durations) == expected, "duration baseline does not cover the selected tests")
     binaries = {binary for binary, _ in expected}
-    require(all(re.fullmatch(r"[A-Za-z0-9_.:-]+", binary) for binary in binaries), "unsupported binary ID")
+    require(all(re.fullmatch(r"[A-Za-z0-9_./:-]+", binary) for binary in binaries), "unsupported binary ID")
     reserved = {binary for binary, _ in protected}
     components = [reserved] if reserved else []
     components += [{binary} for binary in sorted(binaries - reserved)]
@@ -130,7 +130,7 @@ def load_plan(directory):
     protected = set(plan["protected_binaries"])
     require(protected <= expected and any(protected <= set(shard["binaries"]) for shard in shards), "split resource component")
     for shard in shards:
-        require(shard["binaries"] and all(re.fullmatch(r"[A-Za-z0-9_.:-]+", name) for name in shard["binaries"]), "invalid shard binary")
+        require(shard["binaries"] and all(re.fullmatch(r"[A-Za-z0-9_./:-]+", name) for name in shard["binaries"]), "invalid shard binary")
         require(shard["filter"] == " or ".join(f"binary_id(={name})" for name in shard["binaries"]), "shard filter mismatch")
     return plan, listing
 

scripts/test_ci_test_shards.py

@@ -77,6 +77,7 @@ class RealNextestTests(unittest.TestCase):
             (root / 'rust-toolchain.toml').write_text((shards.ROOT / 'rust-toolchain.toml').read_text())
             (root / 'Cargo.toml').write_text('[package]\nname="probe"\nversion="0.1.0"\nedition="2021"\n')
             (root / 'src/lib.rs').write_text('#[test] fn first() { assert!(std::env::var_os("SHARD_FAIL").is_none()); }\n')
+            (root / 'src/main.rs').write_text('fn main() {}\n#[test] fn binary_case() {}\n')
             (root / 'tests/second.rs').write_text('#[test] fn second() {}\n#[test] #[ignore] fn omitted() {}\n')
             (root / 'tests/third.rs').write_text('#[test] fn third() {}\n')
             (root / '.config/nextest.toml').write_text('''experimental = ["setup-scripts"]
@@ -103,6 +104,7 @@ setup = ['fixture']
                 return result
             cargo('archive', '--offline', '--archive-file', str(archive))
             full = shards.list_tests(root, ['--archive-file', str(archive), '--extract-to', str(root), '--extract-overwrite'])
+            self.assertIn('probe::bin/probe', full['rust-suites'])
             full_path = root / 'listing.json'
             shards.write_json(full_path, full)
             cargo('run', '--profile', 'ci', *shards.metadata_arguments(root))
@@ -118,7 +120,7 @@ setup = ['fixture']
                 evidence = root / 'evidence'
                 for shard in plan['shards']:
                     shards.execute(root, plan_dir, archive, shard['index'], evidence / str(shard['index']))
-                self.assertEqual(shards.verify(plan_dir, evidence), 3)
+                self.assertEqual(shards.verify(plan_dir, evidence), 4)
                 floor = root / 'migration-floor.txt'
                 floor.write_text('1\n')
                 # Older local nextest versions omit the test kind; all fixture