fix(ecstore): reclaim orphan data dirs on the healthy heal path (#4781)

PR #4356 wired `reclaim_orphan_data_dirs` only into `heal_object`'s
post-heal tail, which runs after the `disks_to_heal_count == 0` early
return. That early return is exactly the state of the objects the sweep
targets: a valid `xl.meta` with all shards present plus a leaked
pre-#3510 data dir needs no shard healing, so a healthy heal returned
before reclaim and swept nothing. On a healthy deployment (single node,
no degraded disks) the reclaim was therefore dead code — an admin heal
walked the objects, "healed" them, and reclaimed no leaked space.

Run the best-effort reclaim on the `disks_to_heal_count == 0` path as
well, gated on `!opts.dry_run`. The shared match+log block is factored
into `reclaim_orphan_data_dirs_best_effort` so both exits behave
identically. A reclaim failure still never fails the heal.

Adds an end-to-end regression: put a healthy non-inline object, plant an
unreferenced UUID data dir under it on every disk that holds the object,
then drive `heal_object`. A dry-run heal must leave the stray in place; a
real heal must reclaim it while preserving the live data dirs, `xl.meta`,
and object contents. The test fails against the pre-fix control flow.

Refs #3231, #3191, #4356.

Co-authored-by: heihutu <heihutu@gmail.com>

crates/ecstore/src/set_disk/ops/heal.rs

@@ -37,6 +37,23 @@ impl SetDisks {
         Box::pin(self.heal_object_with_regen(bucket, object, version_id, opts, true)).await
     }
 
+    /// Best-effort orphan-data-dir reclaim for an object that is healthy on this
+    /// set. Wraps [`Self::reclaim_orphan_data_dirs`] with the shared logging so
+    /// both `heal_object` exits — the already-healthy early return and the
+    /// post-heal tail — reclaim identically. Never fails the heal: delete errors
+    /// are logged and swallowed. Callers must gate this on `!opts.dry_run`.
+    async fn reclaim_orphan_data_dirs_best_effort(&self, bucket: &str, object: &str) {
+        match self.reclaim_orphan_data_dirs(bucket, object).await {
+            Ok(removed) if removed > 0 => {
+                info!(bucket, object, removed, "heal_object: reclaimed orphaned data directories");
+            }
+            Ok(_) => {}
+            Err(e) => {
+                warn!(bucket, object, error = %e, "heal_object: orphan data-dir reclaim failed");
+            }
+        }
+    }
+
     #[allow(clippy::too_many_lines)]
     async fn heal_object_with_regen(
         &self,
@@ -209,6 +226,16 @@ impl SetDisks {
                         }
 
                         if disks_to_heal_count == 0 {
+                            // The object is already healthy: no disk needs healing.
+                            // This is the common case for the very objects PR #4356
+                            // targets — a valid `xl.meta` plus a leaked pre-#3510
+                            // data dir needs no shard healing, so it would otherwise
+                            // return here and never reach the post-heal reclaim tail
+                            // below. Sweep the strays on this path too (issues #3231,
+                            // #3191). Skipped on dry-run, like every mutating step.
+                            if !opts.dry_run {
+                                self.reclaim_orphan_data_dirs_best_effort(bucket, object).await;
+                            }
                             return Ok((result, None));
                         }
 
@@ -661,15 +688,7 @@ impl SetDisks {
                         // by pre-#3510 unversioned overwrites, which the dangling paths
                         // above never touch (issues #3231, #3191). Best effort — a
                         // failure must not fail the heal.
-                        match self.reclaim_orphan_data_dirs(bucket, object).await {
-                            Ok(removed) if removed > 0 => {
-                                info!(bucket, object, removed, "heal_object: reclaimed orphaned data directories");
-                            }
-                            Ok(_) => {}
-                            Err(e) => {
-                                warn!(bucket, object, error = %e, "heal_object: orphan data-dir reclaim failed");
-                            }
-                        }
+                        self.reclaim_orphan_data_dirs_best_effort(bucket, object).await;
 
                         Ok((result, None))
                     }