* feat(heal): track erasure set progress baseline
Record erasure-set heal byte progress from per-object results and seed progress totals from complete usage-cache snapshots when available.
Keep usage-cache failures observational so heal execution continues without a baseline.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(heal): skip filtered erasure set versions
Skip erasure-set versions written after the durable heal start time, and queue lifecycle-expired versions for expiry before skipping them.
Track new-version and ILM-expired skips separately so progress can explain completed baseline work without treating these skips as retry-blocking failures.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(heal): wire abandoned data-dir cleanup check
Connect check_abandoned_parts through ECStore, pool, and set layers so heal can invoke the existing orphan data-dir reclaim path instead of returning NotImplemented.
Add dry-run support to the reclaim scan and cover dry-run plus scoped set behavior with regression tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(obs): add heal scanner trace bus
Introduce an in-process broadcast trace bus with typed heal and scanner events, lazy event construction, and bounded lagged-subscriber behavior.
Cover zero-subscriber publishing, subscription delivery, drop accounting, and lagged receivers with focused common-crate tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(obs): stream heal trace events from admin API
Wire the admin trace endpoint to the common trace bus for heal/scanner events, including kind, regex, and threshold filtering.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(obs): emit heal trace events
Publish heal task lifecycle and abandoned-parts cleanup events through the common trace bus so the admin trace stream has live heal diagnostics.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(obs): emit scanner trace events
Publish scanner folder, lifecycle action, and heal-candidate events through the common trace bus for live admin scanner diagnostics.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(heal): route data usage loader through storage api
Keep ECStore data-usage facade access behind the heal storage_api boundary so architecture migration guards can validate the heal progress path.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(heal): avoid lifecycle snapshots on ordinary heal pages
Only request lifecycle object snapshots when the heal pass has lifecycle expiry context. This keeps ordinary listing and disk-walk pages from cloning FileInfo/ObjectInfo payloads while preserving the skip path that queues expired versions.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(heal): update bug-fix mocks for lifecycle snapshots
Carry the lifecycle snapshot opt-in argument through the remaining heal bug-fix test mocks so all-targets clippy covers the updated storage trait.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rustfs): sync heal storage mock signature
Update the rustfs storage RPC test mock for the lifecycle snapshot opt-in argument and cover it with rustfs all-targets clippy.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): allocate smoke ports across nextest processes
Serialize E2E port selection with a small /tmp allocator so nextest workers do not reuse the same just-released ephemeral port before RustFS binds it.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* refactor(time): migrate audit and notify timestamps to jiff
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): initialize heal walk decode error
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(targets): parse MySQL event time with jiff
Preserve MySQL DATETIME(6) wall-time formatting for RFC3339 eventTime values while removing the direct chrono dependency from rustfs-targets.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(deps): prune unused workspace dependencies
Apply cargo shear --fix to remove unused path-clean and s3select-api tempfile entries after the scoped jiff migration.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): remove duplicate heal walk decode error init
Remove the duplicate decode_error field from the heal walk test collector initializer so lib-test clippy compiles on CI.
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(policy): emit OPA timestamps with jiff
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
When a returning node carries a stale object version that was deleted on
the quorum, the heal disk-walk partial callback used `resolve_union`
which picks only one entry from divergent disk entries. The minority
version was never enumerated and therefore never cleaned up.
Replace `resolve_union` + `ingest` with a new `ingest_merged` that
collects all unique versions from every partial entry across disks,
deduplicating by (name, version_id). This ensures stale data on a
returning node is surfaced for healing and can be deleted as dangling.
Fixes#5029
Foreground S3 listings wrapped the entire walk_dir stream in a single
wall-clock timeout (RUSTFS_DRIVE_WALKDIR_TIMEOUT_SECS, default 5s). That
budget measured how much data the walk had to produce rather than whether
the drive was still answering, so a healthy but large prefix on slow media
returned 500 InternalError / "Io error: timeout" to the client. The timer
also kept running while the walk was blocked writing to a slow consumer,
charging merge-side backpressure to the producer.
WalkDirOptions::stall_timeout_ms already carried the right semantics but was
only honored by the remote-disk RPC walk; LocalDisk::walk_dir ignored it
entirely. A single-drive deployment therefore had no way to distinguish a
hung drive from a big directory.
Teach LocalDisk::walk_dir to bound each individual drive read with the stall
budget, defaulting it from RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS when the
caller does not pin one, and let the foreground listing path skip the
wrapper-level total timeout. A walk that keeps making progress now runs to
completion; a drive that stops answering still fails with DiskError::Timeout.
Time spent blocked on the consumer stays outside the budget.
Heal and rebalance walks already skipped the total timeout and previously ran
unbounded on local drives. Give them an explicit, generous 60s stall budget so
this change does not tighten them from "no bound" to the 5s default.
Fixes#4644
Refs #2999, #3001
Co-authored-by: heihutu <heihutu@gmail.com>
B5 switched heal enumeration to list_object_versions, which only reflects the
read-quorum metadata view: a version present on fewer than read-quorum disks was
never enumerated, so it was never healed. Add a per-erasure-set disk-walk UNION
enumerator (mirrors MinIO global-heal.go objQuorum=1 listPathRaw +
mergeXLV2Versions) that surfaces every (object, version) present on ANY disk and
feeds each to the existing per-version heal_object.
- filemeta: MetaCacheEntries::resolve_union (dir_quorum=1/obj_quorum=1) yields the
cross-disk version union at one tested seam.
- ecstore: SetDisks::heal_walk_versions_page (list_path_raw fan-out, min_disks=1,
dual object/version page bound, inclusive-forward de-overlap) + ECStore delegator
+ HealWalkVersion.
- ecstore data-safety guard: before dangling-delete, try_regenerate_recoverable_meta
physically probes part files via check_parts; when >= data_blocks data shards
survive (meta lost but data recoverable) it regenerates xl.meta from a surviving
FileInfo with the correct per-disk shard index instead of dangling-deleting.
Genuine torn writes (< data_blocks) keep the current behavior — no resurrection.
- heal: dw1: forward-marker cursor codec (reuses ResumeState.resume_cursor,
idempotent restart on foreign tokens); list_versions_for_heal_page_disk_walk
trait method (default falls back to the B5 read-quorum path); heal_bucket_with_resume
selects the disk-walk enumerator when scan_mode==Deep || source==AutoHeal, else
the unchanged B5 path; anti-loop guard aborts on (empty && truncated).
Closesrustfs/backlog#920
houseme