feat(heal): add progress and trace observability (#6179)

* feat(heal): track erasure set progress baseline Record erasure-set heal byte progress from per-object results and seed progress totals from complete usage-cache snapshots when available. Keep usage-cache failures observational so heal execution continues without a baseline. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(heal): skip filtered erasure set versions Skip erasure-set versions written after the durable heal start time, and queue lifecycle-expired versions for expiry before skipping them. Track new-version and ILM-expired skips separately so progress can explain completed baseline work without treating these skips as retry-blocking failures. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(heal): wire abandoned data-dir cleanup check Connect check_abandoned_parts through ECStore, pool, and set layers so heal can invoke the existing orphan data-dir reclaim path instead of returning NotImplemented. Add dry-run support to the reclaim scan and cover dry-run plus scoped set behavior with regression tests. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add heal scanner trace bus Introduce an in-process broadcast trace bus with typed heal and scanner events, lazy event construction, and bounded lagged-subscriber behavior. Cover zero-subscriber publishing, subscription delivery, drop accounting, and lagged receivers with focused common-crate tests. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): stream heal trace events from admin API Wire the admin trace endpoint to the common trace bus for heal/scanner events, including kind, regex, and threshold filtering. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): emit heal trace events Publish heal task lifecycle and abandoned-parts cleanup events through the common trace bus so the admin trace stream has live heal diagnostics. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): emit scanner trace events Publish scanner folder, lifecycle action, and heal-candidate events through the common trace bus for live admin scanner diagnostics. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(heal): route data usage loader through storage api Keep ECStore data-usage facade access behind the heal storage_api boundary so architecture migration guards can validate the heal progress path. Co-Authored-By: heihutu <heihutu@gmail.com> * perf(heal): avoid lifecycle snapshots on ordinary heal pages Only request lifecycle object snapshots when the heal pass has lifecycle expiry context. This keeps ordinary listing and disk-walk pages from cloning FileInfo/ObjectInfo payloads while preserving the skip path that queues expired versions. Co-Authored-By: heihutu <heihutu@gmail.com> * test(heal): update bug-fix mocks for lifecycle snapshots Carry the lifecycle snapshot opt-in argument through the remaining heal bug-fix test mocks so all-targets clippy covers the updated storage trait. Co-Authored-By: heihutu <heihutu@gmail.com> * test(rustfs): sync heal storage mock signature Update the rustfs storage RPC test mock for the lifecycle snapshot opt-in argument and cover it with rustfs all-targets clippy. Co-Authored-By: heihutu <heihutu@gmail.com> * test(e2e): allocate smoke ports across nextest processes Serialize E2E port selection with a small /tmp allocator so nextest workers do not reuse the same just-released ephemeral port before RustFS binds it. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-19 11:06:17 +00:00 · 2026-08-18 08:29:29 +08:00
parent 7cb91a0190
commit 360bceafce
30 changed files with 2383 additions and 112 deletions
@@ -18,6 +18,7 @@ use tracing::trace;

 const LOG_COMPONENT_ECSTORE: &str = "ecstore";
 const LOG_SUBSYSTEM_HEAL: &str = "heal";
+const EVENT_HEAL_ABANDONED_PARTS: &str = "heal_abandoned_parts";
 const EVENT_HEAL_FORMAT_COMPLETED: &str = "heal_format_completed";
 const EVENT_HEAL_OBJECT_STARTED: &str = "heal_object_started";

@@ -256,13 +257,40 @@ impl ECStore {

    #[instrument(skip(self))]
    pub(super) async fn handle_check_abandoned_parts(&self, bucket: &str, object: &str, opts: &HealOpts) -> Result<()> {
-        let _ = (bucket, object, opts);
-        // Stale multipart reconciliation is already owned by the lifecycle-driven
-        // background cleanup path in `bucket_lifecycle_ops.rs`. There is currently
-        // no stable object-heal contract that should fan this request out through
-        // pool/set storage layers, so keep the placeholder explicit at the ECStore
-        // boundary instead of dispatching into lower layers.
-        Err(StorageError::NotImplemented)
+        let object = encode_dir_object(object);
+        let pools = self.get_pools_for_heal_object(opts)?;
+
+        let mut futures = Vec::with_capacity(pools.len());
+        for pool in pools.iter() {
+            futures.push(pool.check_abandoned_parts(bucket, &object, opts));
+        }
+
+        let mut first_error = None;
+        for result in join_all(futures).await {
+            if let Err(err) = result
+                && first_error.is_none()
+            {
+                first_error = Some(err);
+            }
+        }
+
+        if let Some(err) = first_error {
+            return Err(err);
+        }
+
+        trace!(
+            event = EVENT_HEAL_ABANDONED_PARTS,
+            component = LOG_COMPONENT_ECSTORE,
+            subsystem = LOG_SUBSYSTEM_HEAL,
+            state = "completed",
+            result = "ok",
+            bucket,
+            object,
+            dry_run = opts.dry_run,
+            "Heal abandoned parts completed"
+        );
+
+        Ok(())
    }
 }

@@ -34,6 +34,7 @@ impl ECStore {
        forward_to: Option<&str>,
        batch_objects: usize,
        version_budget: usize,
+        include_lifecycle_object_info: bool,
    ) -> Result<(Vec<HealWalkVersion>, Option<String>, bool)> {
        if pool_idx >= self.pools.len() || set_idx >= self.pools[pool_idx].disk_set.len() {
            return Err(Error::other(format!(
@@ -43,7 +44,7 @@ impl ECStore {
        }

        self.pools[pool_idx].disk_set[set_idx]
-            .heal_walk_versions_page(bucket, prefix, forward_to, batch_objects, version_budget)
+            .heal_walk_versions_page(bucket, prefix, forward_to, batch_objects, version_budget, include_lifecycle_object_info)
            .await
            .map_err(Error::from)
    }