perf(ecstore): move speculative PUT-tail tmp cleanup off the hot path (#4389)

* perf(ecstore): move speculative tmp cleanup off the PUT hot path

On a successful PUT, rename_data has already moved the data dir out of the tmp workspace, so the delete_all(RUSTFS_META_TMP_BUCKET) at the end of SetDisks::put_object is a speculative no-op safety net. It was awaited inline on the response path, where profiling (backlog#924 / HP-3) showed the same-disk queueing behind fsync-heavy load turns a ~49us no-op into ~9ms average (p99 77ms, macOS F_FULLFSYNC amplified) added to every PUT.

Run that cleanup on a spawned task instead, keeping it as a real backstop (rename_data's remove_std only removes empty dirs and silently ignores failures). The failure path (quorum loss / rollback) keeps the cleanup inline so a failed PUT never returns with tmp shards still on disk. If the process dies before the spawned task runs, cleanup_stale_tmp_objects (24h expiry, 5-minute loop) reclaims the entry.

Scope note: ops/multipart.rs delete_all on RUSTFS_META_MULTIPART_BUCKET is intentionally untouched; it removes real leftovers and deferring it would widen CompleteMultipartUpload/Abort races.

Regression tests (hermetic SetDisks on formatted local disks, no global state): PUT success drains the tmp workspace (polling the spawned task), and PUT failure (missing bucket volume, rename_data quorum error after tmp shards were written) cleans the workspace inline before returning.

Ref: https://github.com/rustfs/backlog/issues/924

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ecstore): do not retry NotFound in reliable_rename_inner

reliable_rename_inner blindly retried the rename once on any error. A NotFound retry cannot succeed: nothing recreates the missing source or parent directory between attempts, so the second rename fails identically and speculative cleanup renames (e.g. move_to_trash on an already-removed tmp path) always paid for two syscalls.

Extract the retry decision into should_retry_rename: NotFound returns immediately, any other error keeps the existing single retry. This helper is shared by the rename_data commit path via rename_all, so behavior there is covered by a new rename_all success regression test alongside the retry-predicate tests.

Ref: https://github.com/rustfs/backlog/issues/924

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>

crates/ecstore/src/disk/os.rs

@@ -241,7 +241,7 @@ async fn reliable_rename_inner(
     let mut i = 0;
     loop {
         if let Err(e) = super::fs::rename_std(src_file_path.as_ref(), dst_file_path.as_ref()) {
-            if i == 0 {
+            if should_retry_rename(&e, i) {
                 i += 1;
                 continue;
             }
@@ -263,6 +263,19 @@ async fn reliable_rename_inner(
     Ok(())
 }
 
+/// Whether a failed `rename` in [`reliable_rename_inner`] should be retried.
+///
+/// Only the first failure is retried, and `NotFound` is never retried: the
+/// retry does not recreate the missing source or parent directory, so a second
+/// attempt is guaranteed to fail identically. Skipping it spares speculative
+/// cleanup renames (e.g. `move_to_trash` on an already-removed tmp path) a
+/// pointless second syscall. This predicate is shared by the `rename_data`
+/// commit path via `rename_all`, so any relaxation here must keep genuine
+/// transient errors retryable.
+fn should_retry_rename(err: &io::Error, attempt: usize) -> bool {
+    attempt == 0 && err.kind() != io::ErrorKind::NotFound
+}
+
 pub async fn reliable_mkdir_all(path: impl AsRef<Path>, base_dir: impl AsRef<Path>) -> io::Result<()> {
     let mut i = 0;
 
@@ -366,6 +379,38 @@ mod tests {
         assert!(!dst.exists());
     }
 
+    #[test]
+    fn rename_retry_never_retries_not_found() {
+        // NotFound is terminal for the retry loop: the retry does not recreate
+        // the missing source/parent, so a second rename would fail identically.
+        let not_found = io::Error::new(io::ErrorKind::NotFound, "missing");
+        assert!(!should_retry_rename(&not_found, 0));
+        assert!(!should_retry_rename(&not_found, 1));
+    }
+
+    #[test]
+    fn rename_retry_allows_single_retry_for_other_errors() {
+        let denied = io::Error::new(io::ErrorKind::PermissionDenied, "denied");
+        assert!(should_retry_rename(&denied, 0));
+        assert!(!should_retry_rename(&denied, 1));
+    }
+
+    #[tokio::test]
+    async fn rename_all_moves_existing_directory_tree() {
+        // Guards the rename_data commit path, which funnels through
+        // reliable_rename_inner via rename_all.
+        let temp_dir = tempdir().expect("create temp dir");
+        let src = temp_dir.path().join("src-dir");
+        std::fs::create_dir_all(src.join("nested")).expect("create src tree");
+        std::fs::write(src.join("nested").join("part.1"), b"payload").expect("write part");
+        let dst = temp_dir.path().join("dst-parent").join("dst-dir");
+
+        rename_all(&src, &dst, temp_dir.path()).await.expect("rename must succeed");
+
+        assert!(!src.exists());
+        assert_eq!(std::fs::read(dst.join("nested").join("part.1")).expect("read moved part"), b"payload");
+    }
+
     #[tokio::test]
     async fn fsync_dir_succeeds_on_directory() {
         let temp_dir = tempdir().expect("create temp dir");