fix(ecstore): make post-commit old data dir cleanup best-effort (#4386)

* fix(ecstore): make post-commit old data dir cleanup best-effort (backlog#898)

A write is authoritatively committed once rename_data returns Ok (the new
version is durable on >= write_quorum disks and immediately readable). The
subsequent reclamation of the now-dereferenced old object/<data_dir> is pure
space reclamation, yet commit_rename_data_dir propagated a below-quorum GC
failure via `?` into ErasureWriteQuorum -> 503, producing a false-negative
ACK for an already-persisted write. This is a deliberate divergence from
MinIO (erasure-object.go:1577), which couples the two; the divergence is
justified by durability semantics, not parity.

Changes:
- commit_rename_data_dir now returns a structured OldDataDirCleanup receipt
  and never returns Err. Adds an old==committed-dir anti-misdelete guard and
  a committed_data_dir parameter. Classification is extracted into pure
  functions (classify_old_data_dir_cleanup / map_cleanup_join_result /
  is_cleanup_not_found) so it is unit-testable. Task panic/cancel is mapped to
  a non-ignored DiskError::other (never DiskNotFound), and not-found is
  normalized to reclaimed.
- object.rs / multipart.rs consume the receipt instead of `?`. The result
  reverts to Ok, so the invalidate_get_object_metadata_cache self-heal and the
  capacity/compression accounting that a `?` early-return previously skipped
  now run on the cleanup-failure path too.
- On residue, report_old_data_dir_cleanup emits leak metrics and enqueues an
  object heal over the existing heal channel (disk-health signal replacing the
  503). heal_object -> reclaim_orphan_data_dirs already reclaims unreferenced
  local data dirs, closing the loop end to end.
- Adds rustfs_old_data_dir_* counters (attempted/reclaimed/leaked/below_quorum)
  as the operator-visible backstop for leaked residue.
- Adds a test-only (#[cfg(test)]) delete fault-injection seam; in production it
  inlines to a no-op None and has no behavioral effect.

Tests: pure-function A/C group + join-error mapping + actions decision; A5/A5b
real-disk guard/reclaim integration; end-to-end overwrite returning 200 while
old-data-dir cleanup fails. #864 rollback guard test remains green.

* fix(ecstore): resolve merge conflicts with origin/main in io_primitives.rs

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>

crates/io-metrics/src/capacity_metrics.rs

@@ -172,3 +172,22 @@ pub fn record_capacity_scan_disk(
         counter!("rustfs_capacity_scan_disk_partial_errors_total", "disk" => disk.to_owned()).increment(1);
     }
 }
+
+/// Record the outcome of a post-commit old-data-dir cleanup (backlog#898).
+///
+/// This is fired once per committed overwrite that had a previous data dir to
+/// reclaim. `leaked` is the number of disks whose old data dir could not be
+/// removed (a non-ignored, non-not-found failure) and is therefore a residue
+/// that leaks disk space until a heal reclaims it. The leak counter is the
+/// operator-visible backstop for that residue — see backlog#898 §3.4/§5.
+#[inline(always)]
+pub fn record_old_data_dir_cleanup(attempted: usize, reclaimed: usize, leaked: usize, below_quorum: bool) {
+    counter!("rustfs_old_data_dir_cleanup_attempted_total").increment(attempted as u64);
+    counter!("rustfs_old_data_dir_cleanup_reclaimed_total").increment(reclaimed as u64);
+    if leaked > 0 {
+        counter!("rustfs_old_data_dir_leaked_total").increment(leaked as u64);
+    }
+    if below_quorum {
+        counter!("rustfs_old_data_dir_cleanup_below_quorum_total").increment(1);
+    }
+}

crates/io-metrics/src/lib.rs

@@ -122,7 +122,7 @@ pub use capacity_metrics::{
     record_capacity_refresh_joiner, record_capacity_refresh_request, record_capacity_refresh_result,
     record_capacity_refresh_scope, record_capacity_scan_disk, record_capacity_scan_mode, record_capacity_scan_sampling,
     record_capacity_stall_detected, record_capacity_symlink, record_capacity_timeout_fallback, record_capacity_update_completed,
-    record_capacity_update_failed, record_capacity_write_operation,
+    record_capacity_update_failed, record_capacity_write_operation, record_old_data_dir_cleanup,
 };
 
 // I/O metrics exports