Add an MRF pipeline regression that replays a single authoritative journal containing same-object records that differ by kind and erasure-set scope while a stale legacy mirror is present. The test proves restart replay admits each authoritative responsibility independently and never merges the stale mirror epoch.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Add a Unix process-kill fixture for the authoritative successor fsync boundary where the scoped journal is durable while the legacy mirror still contains the startup epoch. Also extend heal-control transport replay coverage so a duplicate producer after a lost response receives the canonical merged receipt without creating another task.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): remove unused digest import
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): add raw page owner index (#7375)
* feat(scanner): add raw page owner index
Add a serializable raw enumeration page owner index for scanner resume work.
The index exposes unsupported, building, and ready states, validates committed page identity by recomputing digests, and uses generation checks for CAS-style page commits.
Focused tests cover small-budget restart progress, page digest/source drift rejection, corrupt deserialized state, CAS failure, precommit crash, empty sources, and invalid entry boundaries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): persist raw page owner resume state (#7379)
Wire the scanner raw enumeration partial-cache writer to the raw page owner index so interrupted bucket walks can retain validated page-builder state across scanner restarts.
Keep complete owner sources terminal-only, add partial-source ingestion for in-progress raw directory reads, and validate the persisted page index through bucket checkpoint preparation.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): fence segment producer observations (#7381)
Require the segment observation fixture to carry source, incarnation, key-format, baseline, process epoch, generation-window, gap, overflow, and producer-coverage proof before accepting a narrowed proposal. Keep the diagnostic path fixture-only and remove its ordinary stderr output.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): isolate pool metadata read probes (#7367)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF crash successor matrix (#7369)
* test(heal): cover MRF crash successor matrix
Add process-boundary MRF replay coverage for the successor snapshot window after a retained startup journal is flushed but before cleanup deletes it. Extend the mixed authoritative/legacy reader fixture with a scoped v2 journal epoch to pin the no-merge contract.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover service-kill MRF replay (#7380)
Add a Unix process fixture that waits after publishing the pending MRF successor snapshot, then is terminated by the parent before restart replay.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover transport-lost start receipts (#7371)
Add gRPC transport fault fixtures for heal-control start admission. The tests distinguish pre-admission transport loss from post-admission response loss, then verify exact envelope retries reuse the canonical receipt while fresh forceStart requests create distinct tasks.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add crash-restart heal evidence case (#7370)
* test(scanner): add crash-restart heal evidence case
Add a distinct W21 background target crash case to the scanner/heal evidence registry and oracle path.
Keep the existing restart lane on graceful process restart, keep the crash lane on hard kill, and make the wiring checker reject evidence/oracle mismatches.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): support older Python wiring checks
Let the scanner/heal evidence wiring checker run under Python 3.9/3.10 by falling back to tomli and chunked SHA-256 hashing when the Python 3.11 standard APIs are unavailable.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): reject stale raw page source seeds (#7382)
Do not prefill a resumed raw page owner with previously indexed entries when starting a new raw directory observation pass. The next pass must observe the same prefix again before the page index can advance; otherwise the index is discarded fail-closed.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): defer raw page revalidation until observed (#7384)
A resumed raw page owner index must not prefill entries from older cache state, but it also must not discard a valid multi-entry index before the current raw directory pass has observed enough entries to prove identity. Track the persisted index floor and only run the strict owner identity check once the current pass reaches that floor.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Add an ECStore-backed child-process fixture that publishes the MRF journal in one OS process, then verifies a restarted manager can replay the complete snapshot while retaining the journal when bounded admission accepts only a prefix.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Keep the startup journal on disk when replay cannot fully re-arm or the heal manager refuses a replayed intent with Full/QueueFull. The next live snapshot can still advance the journal after the retry anchor is durable again.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
refactor(protos): move internode compat manifest send-site assertions into owning crates
Promotes the rolling-upgrade dual-write manifest from a test-only constant in rustfs-protos into the public rustfs_protos::compat_manifest module, moves the JSON-encoder send-site assertions into the crates that own the asserted sources (ecstore remote_disk.rs for requests, the rustfs binary node_service/disk.rs for responses), and splits the scanner Phase-0 overlap inventory so its heal- and ecstore-owned halves live in those crates. Adds a cross-crate include_str!/include! guard with fixture self-tests to scripts/check_layer_dependencies.sh so a library crate can never again read another crate's Rust source at compile time, and records the rule in docs/architecture/crate-boundaries.md.
Part of rustfs/backlog#1884.
Move MRF repaired-event fan-out from admission to successful terminal completion so scanner pending-heal ledgers only clear after the canonical heal task actually finishes. Track notice ownership across duplicate admission, retry merge, cancellation, and queue displacement.
Co-authored-by: heihutu <heihutu@gmail.com>
The scanner's pending-heal ledger and the MRF journal tracked the same
damaged objects with no cross-talk: once the consumer landed an intent
with the heal manager, the ledger's retry entry for that target kept
re-submitting a heal the manager already owned (backlog#1894 axis B).
Fan the acceptance out: both dispatch sites in the MRF queue (the live
consumer and the startup replay) record a compact MrfRepairedEvent
(bucket, object, version bytes) in a bounded process-wide ring owned by
rustfs-common. The scanner drains its own bucket's notices at the top
of retry_pending_scanner_heals and clears the matching Object-kind
ledger entries in one batched retain + sync (a mass-recovery first
sweep must not turn into thousands of full-table ledger clones on the
scan task), with nil notice UUIDs mapping to None per the repo-wide
defensive-UUID invariant so unversioned entries match unversioned
notices only. Notices are best-effort by design — a lost or capped-out
notice leaves the entry to expire through its own attempts/age limits,
because the ledger is a retry oracle, not a source of truth; other
buckets' notices stay queued for their own scanners. Neither persistent
format changes; old nodes that keep double-booking remain harmless.
Co-authored-by: heihutu <heihutu@gmail.com>
HealType::MRF (a #1664-era "metadata repair file" task kind) had no
production construction site left: its only builder lived in the
HealEvent -> HealRequest converter, and the HealEvent/HealEventHandler
queue itself had zero production references — both were superseded by
the MrfIntent pipeline (mrf_queue.rs), which produces Object/Metadata/
ECDecode requests and never an MRF task. The dead path nevertheless
carried ~700 lines: the whole event.rs module, the heal_mrf executor,
a dedup-key arm, an overlap arm with the "\u{0}mrf" sentinel bucket
hack, per-kind labels, and an empty MrfRuntime::record_accept shell.
Deleting the variant is compile-time safe: HealType has no Serialize
derive, the protos wire enums carry no heal-type discriminant (the
receiver rebuilds it from HealChannelRequest fields), the MRF journal
encodes MrfKind (1/2/3), and the scanner pending-heal ledger uses its
own kind enum — none of them can name an MRF task.
Also resolves the in-crate naming clash where "MRF" denoted both the
dead task kind and the live mission-repair-feed loop; the loop stays,
the task kind goes.
Co-authored-by: heihutu <heihutu@gmail.com>
`audit_runtime_facade_stops_empty_replay_workers` called the stop path and checked nothing, the same shape as the notify facade test in the previous commit. It now asserts the worker manager is empty afterwards and that a second call — which shutdown paths make — stays harmless.
The two heal timestamp tests bound their fields to `_`. Both timestamps come from `SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default()`, so a pre-epoch clock yields 0; binding to `_` could not tell that apart from a real reading, which is precisely what the tests said they were guarding. They now require the value to be past 2020-01-01, and `last_update` not to predate `start_time`.
`test_config_parsing_with_multiple_instances` is left as it was — see the issue comment. Asserting on it turned up something bigger than a missing assertion.
Refs backlog#1836
* feat(common): add MRF intent channel and Mrf request source (HS-01)
Introduce the producer-facing half of the mission repair feed: a global
bounded (8192) channel carrying lightweight MrfIntent values from IO
error paths, plus the RUSTFS_HEAL_MRF_ENABLE delivery kill-switch and
config constants for queue/journal sizing. Delivery is strictly
non-blocking (try_send, drop-on-full) so it can sit on decode-failure
and partial-write paths without adding latency. HealRequestSource grows
a 'mrf' variant so admission accounting can attribute replayed intents.
Part of backlog#1865 (option a: wire HealEvent-style intents with a
durable retry ledger).
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(heal): add MRF queue, durable journal, and intent consumer (HS-01)
Consumer half of the mission repair feed: a bounded pending queue
(100k intents / 8 MiB dual ceiling, drop-newest on overflow), a durable
journal at buckets/.heal/mrf/journal.bin holding the unaccepted pending
snapshot, and a consumer task that batches intents off the global
channel, translates them into prioritized heal requests (decode
failure -> Urgent ECDecode, metadata corruption -> High Metadata,
partial write -> Normal object heal), and retries full admissions with
a 5s backoff and a 3-attempt ceiling.
Durability: every journal record carries its own CRC32 and a
format/version header, so a torn tail truncates cleanly at replay; the
journal is deleted after a successful replay and when the pending set
drains (mirroring MinIO's post-replay list.bin unlink). Losing the last
500 ms flush window is acceptable: replayed duplicates merge via the
manager dedup key and read-repair remains the safety net.
Metrics: rustfs_heal_mrf_queue_depth/_queue_bytes, _dropped_total
{reason}, _replayed_total, _journal_bytes, _journal_fsync_total.
The consumer is wired at heal runtime bootstrap right after manager
start, honoring RUSTFS_HEAL_MRF_ENABLE (default on, rollback = off).
Tests: unit tests for the dual ceiling, record roundtrip, torn-tail
truncation, and the priority mapping; integration tests against a real
4-disk ECStore proving channel intents reach the manager queue as
Urgent/mrf-attributed requests and journal replay arms intents, drops
torn tails, and removes the file.
Part of backlog#1865 (option a).
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ecstore,scanner): deliver MRF intents from error paths (HS-01)
Wire the three production delivery points, each a single non-blocking
try_send next to the existing in-memory heal paths, which stay as the
fast path:
- read.rs decode-error branch: DecodeFailure intent beside the existing
read-repair submit, so an Urgent ECDecode request survives restarts
even when the Low-priority read-repair request was dropped or lost.
- add_partial: PartialWrite intent, giving partial-write recovery a
durable Normal-priority object heal across restarts.
- scanner_folder metadata-corruption classification: MetadataCorruption
intent beside the existing High-priority scanner heal request.
All three are on error paths only: zero cost on healthy IO.
Part of backlog#1865 (option a).
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix: include mrf heal source counts
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix: keep node heal status wire compatibility
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(heal): track erasure set progress baseline
Record erasure-set heal byte progress from per-object results and seed progress totals from complete usage-cache snapshots when available.
Keep usage-cache failures observational so heal execution continues without a baseline.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(heal): skip filtered erasure set versions
Skip erasure-set versions written after the durable heal start time, and queue lifecycle-expired versions for expiry before skipping them.
Track new-version and ILM-expired skips separately so progress can explain completed baseline work without treating these skips as retry-blocking failures.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(heal): wire abandoned data-dir cleanup check
Connect check_abandoned_parts through ECStore, pool, and set layers so heal can invoke the existing orphan data-dir reclaim path instead of returning NotImplemented.
Add dry-run support to the reclaim scan and cover dry-run plus scoped set behavior with regression tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(obs): add heal scanner trace bus
Introduce an in-process broadcast trace bus with typed heal and scanner events, lazy event construction, and bounded lagged-subscriber behavior.
Cover zero-subscriber publishing, subscription delivery, drop accounting, and lagged receivers with focused common-crate tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(obs): stream heal trace events from admin API
Wire the admin trace endpoint to the common trace bus for heal/scanner events, including kind, regex, and threshold filtering.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(obs): emit heal trace events
Publish heal task lifecycle and abandoned-parts cleanup events through the common trace bus so the admin trace stream has live heal diagnostics.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(obs): emit scanner trace events
Publish scanner folder, lifecycle action, and heal-candidate events through the common trace bus for live admin scanner diagnostics.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(heal): route data usage loader through storage api
Keep ECStore data-usage facade access behind the heal storage_api boundary so architecture migration guards can validate the heal progress path.
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(heal): avoid lifecycle snapshots on ordinary heal pages
Only request lifecycle object snapshots when the heal pass has lifecycle expiry context. This keeps ordinary listing and disk-walk pages from cloning FileInfo/ObjectInfo payloads while preserving the skip path that queues expired versions.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(heal): update bug-fix mocks for lifecycle snapshots
Carry the lifecycle snapshot opt-in argument through the remaining heal bug-fix test mocks so all-targets clippy covers the updated storage trait.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rustfs): sync heal storage mock signature
Update the rustfs storage RPC test mock for the lifecycle snapshot opt-in argument and cover it with rustfs all-targets clippy.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): allocate smoke ports across nextest processes
Serialize E2E port selection with a small /tmp allocator so nextest workers do not reuse the same just-released ephemeral port before RustFS binds it.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Add a default-off inline-only data-read metadata early-stop gate that verifies inline plaintext before cancelling pending metadata tasks.
Keep non-inline, prepared, and request-shape-sensitive reads on full fanout, and record scheduled/completed/cancelled ReadVersion lifecycle metrics for normal fanout completion.
Co-authored-by: heihutu <heihutu@gmail.com>
`test_heal_resume_across_page_boundary_e2e` panicked whenever the
erasure-set healer deferred a single object version to a later heal
cycle. Under load a per-version `heal_object` can hit a transient error;
`ErasureSetHealer::heal_bucket_with_resume` then persists its
resume/checkpoint state and returns a terminal `Failed { .. "retry
scheduled" }`, expecting a fresh heal run to finish the job. In
production the background scanner is that next run — the e2e had no such
follow-up, so the first task's `Failed` state failed the test. This is a
pre-existing rare flake (the wiped-disk heal is otherwise correct); it is
unrelated to any policy/proptest work that happened to surface it in CI.
Drive the heal to a genuine `Completed` instead: on a `Failed` carrying
the `retry scheduled` marker (retry budget still remaining) re-submit the
idempotent heal (`force_start`), mirroring the production scanner, up to a
small bound. A `Failed` without that marker (e.g. `exhausted retries`) or
any other non-`Completed` terminal state still fails the test, and the
strict per-version data-restoration assertions still run only after a
real `Completed`. `wait_for_task` is refactored onto the shared
`await_terminal_status` poller; its panic-on-failure semantics for the
unversioned-bucket heal are unchanged.
Test-only change; no production heal code is touched.
Co-authored-by: heihutu <heihutu@gmail.com>
* test(utils): add rustfs-test-utils crate, absorb heal/iam ECStore bootstrap
backlog#1153 infra-1. The ~50-line "build a real temp-disk ECStore"
bootstrap was copy-pasted (and drifting) across the heal and iam
integration tests. This adds crates/test-utils (rustfs-test-utils, a
dev-dependency-only crate) owning that bootstrap and converts the four
copies into thin wrappers:
- TestECStoreEnvBuilder: disk_count (default 4), prefix (uuid-suffixed
/tmp dir), base_dir (caller-owned dir, e.g. tempfile::TempDir),
init_bucket_metadata (default true; the iam bootstrap test opts out
to preserve its historical semantics). TestECStoreEnv exposes
temp_root/disk_paths/ecstore plus a versioned-bucket helper, and
init_tracing() replaces the per-file Once blocks.
- All rustfs_ecstore imports stay behind src/ecstore_test_compat.rs,
the sanctioned test-compat boundary pattern (mirrors
crates/iam/tests/ecstore_test_compat).
- heal: heal_integration_test / heal_b5_versioned_regression_test /
heal_b920_subquorum_union_test drop their setup_test_env{,_n} copies
for heal_env{,_n} wrappers; the tests/storage_api.rs integration
surface shrinks to what test bodies still touch.
- iam: iam_bootstrap_no_lock_test drops build_local_ecstore; its
ecstore_test_compat fixture shrinks to SetupType +
update_erasure_type.
rg 'async fn setup_test_env' crates/heal crates/iam now returns 0.
Scanner's lifecycle tests are deliberately NOT absorbed (gated on
ilm-1; 14 of 15 are #[ignore]d today). Net -230 lines.
* fix(heal): drop tokio::fs import orphaned by the b920 bootstrap move
* fix(heal): drop tokio::fs import orphaned by the b5 bootstrap move
PR #4356 wired `reclaim_orphan_data_dirs` only into `heal_object`'s
post-heal tail, which runs after the `disks_to_heal_count == 0` early
return. That early return is exactly the state of the objects the sweep
targets: a valid `xl.meta` with all shards present plus a leaked
pre-#3510 data dir needs no shard healing, so a healthy heal returned
before reclaim and swept nothing. On a healthy deployment (single node,
no degraded disks) the reclaim was therefore dead code — an admin heal
walked the objects, "healed" them, and reclaimed no leaked space.
Run the best-effort reclaim on the `disks_to_heal_count == 0` path as
well, gated on `!opts.dry_run`. The shared match+log block is factored
into `reclaim_orphan_data_dirs_best_effort` so both exits behave
identically. A reclaim failure still never fails the heal.
Adds an end-to-end regression: put a healthy non-inline object, plant an
unreferenced UUID data dir under it on every disk that holds the object,
then drive `heal_object`. A dry-run heal must leave the stray in place; a
real heal must reclaim it while preserving the live data dirs, `xl.meta`,
and object contents. The test fails against the pre-fix control flow.
Refs #3231, #3191, #4356.
Co-authored-by: heihutu <heihutu@gmail.com>
B5 switched heal enumeration to list_object_versions, which only reflects the
read-quorum metadata view: a version present on fewer than read-quorum disks was
never enumerated, so it was never healed. Add a per-erasure-set disk-walk UNION
enumerator (mirrors MinIO global-heal.go objQuorum=1 listPathRaw +
mergeXLV2Versions) that surfaces every (object, version) present on ANY disk and
feeds each to the existing per-version heal_object.
- filemeta: MetaCacheEntries::resolve_union (dir_quorum=1/obj_quorum=1) yields the
cross-disk version union at one tested seam.
- ecstore: SetDisks::heal_walk_versions_page (list_path_raw fan-out, min_disks=1,
dual object/version page bound, inclusive-forward de-overlap) + ECStore delegator
+ HealWalkVersion.
- ecstore data-safety guard: before dangling-delete, try_regenerate_recoverable_meta
physically probes part files via check_parts; when >= data_blocks data shards
survive (meta lost but data recoverable) it regenerates xl.meta from a surviving
FileInfo with the correct per-disk shard index instead of dangling-deleting.
Genuine torn writes (< data_blocks) keep the current behavior — no resurrection.
- heal: dw1: forward-marker cursor codec (reuses ResumeState.resume_cursor,
idempotent restart on foreign tokens); list_versions_for_heal_page_disk_walk
trait method (default falls back to the B5 read-quorum path); heal_bucket_with_resume
selects the disk-walk enumerator when scan_mode==Deep || source==AutoHeal, else
the unchanged B5 path; anti-loop guard aborts on (empty && truncated).
Closesrustfs/backlog#920
Disk-replacement heal previously repaired only the latest version of each
object and never enumerated objects whose latest version is a delete marker,
so old versions were left unrepaired on a replaced drive.
Switch heal enumeration from list_objects_v2 (latest-only) to
list_object_versions (every version incl. delete markers), thread the concrete
version_id into the existing per-version heal_object, and make resume
cursor-based instead of positional: an opaque (marker, version_marker) paging
token persisted in ResumeState, a length-prefixed injective per-version dedup
key, schema_version bumps (v2) migrated independently in each of the two
persisted files, and a retry that resets both managers together (fixing a
latent rescan-skips-everything defect). Adds a real-disk-wipe e2e regression
suite proving old versions and delete-marker-latest objects are physically
restored.
Fixesrustfs/backlog#918Fixesrustfs/backlog#919Closesrustfs/backlog#854
houseme