fix(ecstore): make post-commit old data dir cleanup best-effort (#4386)

* fix(ecstore): make post-commit old data dir cleanup best-effort (backlog#898)

A write is authoritatively committed once rename_data returns Ok (the new
version is durable on >= write_quorum disks and immediately readable). The
subsequent reclamation of the now-dereferenced old object/<data_dir> is pure
space reclamation, yet commit_rename_data_dir propagated a below-quorum GC
failure via `?` into ErasureWriteQuorum -> 503, producing a false-negative
ACK for an already-persisted write. This is a deliberate divergence from
MinIO (erasure-object.go:1577), which couples the two; the divergence is
justified by durability semantics, not parity.

Changes:
- commit_rename_data_dir now returns a structured OldDataDirCleanup receipt
  and never returns Err. Adds an old==committed-dir anti-misdelete guard and
  a committed_data_dir parameter. Classification is extracted into pure
  functions (classify_old_data_dir_cleanup / map_cleanup_join_result /
  is_cleanup_not_found) so it is unit-testable. Task panic/cancel is mapped to
  a non-ignored DiskError::other (never DiskNotFound), and not-found is
  normalized to reclaimed.
- object.rs / multipart.rs consume the receipt instead of `?`. The result
  reverts to Ok, so the invalidate_get_object_metadata_cache self-heal and the
  capacity/compression accounting that a `?` early-return previously skipped
  now run on the cleanup-failure path too.
- On residue, report_old_data_dir_cleanup emits leak metrics and enqueues an
  object heal over the existing heal channel (disk-health signal replacing the
  503). heal_object -> reclaim_orphan_data_dirs already reclaims unreferenced
  local data dirs, closing the loop end to end.
- Adds rustfs_old_data_dir_* counters (attempted/reclaimed/leaked/below_quorum)
  as the operator-visible backstop for leaked residue.
- Adds a test-only (#[cfg(test)]) delete fault-injection seam; in production it
  inlines to a no-op None and has no behavioral effect.

Tests: pure-function A/C group + join-error mapping + actions decision; A5/A5b
real-disk guard/reclaim integration; end-to-end overwrite returning 200 while
old-data-dir cleanup fails. #864 rollback guard test remains green.

* fix(ecstore): resolve merge conflicts with origin/main in io_primitives.rs

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>

crates/ecstore/src/set_disk/ops/multipart.rs

@@ -1357,8 +1357,15 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
         .await?;
 
         if let Some(old_dir) = op_old_dir {
-            self.commit_rename_data_dir(&cleanup_disks, bucket, object, &old_dir.to_string(), write_quorum)
-                .await?;
+            let committed_dir = fi.data_dir.unwrap_or_default().to_string();
+            // backlog#898: best-effort reclaim of the dereferenced old data dir.
+            // Returns a receipt (never `Err`); a failed GC must not turn an
+            // already-committed multipart completion into a 503.
+            let cleanup = self
+                .commit_rename_data_dir(&cleanup_disks, bucket, object, &old_dir.to_string(), &committed_dir, write_quorum)
+                .await;
+            self.report_old_data_dir_cleanup(bucket, object, &old_dir.to_string(), &cleanup)
+                .await;
         }
 
         if let Some(stage_start) = complete_tail_stage_start {

crates/ecstore/src/set_disk/ops/object.rs

@@ -895,12 +895,21 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
 
             let mut cleanup_stage_ms: Option<u64> = None;
             if let Some(old_dir) = op_old_dir {
+                let committed_dir = fi.data_dir.unwrap_or_default().to_string();
                 let cleanup_stage_start = Instant::now();
-                self.commit_rename_data_dir(&cleanup_disks, bucket, object, &old_dir.to_string(), write_quorum)
-                    .await?;
+                // backlog#898: reclaiming the dereferenced old data dir is
+                // best-effort and returns a receipt (never `Err`). A failed GC
+                // here must not negate an already-committed, durable write, so we
+                // deliberately do NOT `?`-propagate it into a 503. On residue the
+                // report path emits the leak metric and enqueues a heal.
+                let cleanup = self
+                    .commit_rename_data_dir(&cleanup_disks, bucket, object, &old_dir.to_string(), &committed_dir, write_quorum)
+                    .await;
                 let cleanup_ms = cleanup_stage_start.elapsed().as_millis() as u64;
                 cleanup_stage_ms = Some(cleanup_ms);
                 rustfs_io_metrics::record_put_object_stage_duration("set_disk_old_data_cleanup", cleanup_ms as f64);
+                self.report_old_data_dir_cleanup(bucket, object, &old_dir.to_string(), &cleanup)
+                    .await;
                 if (cleanup_ms as u128) >= SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS {
                     warn!(
                         event = EVENT_SET_DISK_COMMIT_TAIL_SLOW,