Add an ECStore-backed child-process fixture that publishes the MRF journal in one OS process, then verifies a restarted manager can replay the complete snapshot while retaining the journal when bounded admission accepts only a prefix.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Keep the startup journal on disk when replay cannot fully re-arm or the heal manager refuses a replayed intent with Full/QueueFull. The next live snapshot can still advance the journal after the retry anchor is durable again.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
fix(heal): fall back to set-wide format for directory-backed replacement targets
Since #7018 `renew_disk` routes an unformatted local endpoint through the automatic replacement heal, which requires the target to be an independently mounted disk. Directory-backed deployments (the operator set `RUSTFS_UNSAFE_BYPASS_DISK_CHECK`, which the startup disk-independence check already requires for endpoints sharing a device) can never pass that admission, so a runtime-wiped or replaced directory disk stayed unformatted forever: the heal task failed with "replacement target is not a stable mounted disk" and the auto-scan kept deferring the endpoint. This broke the Issue #1533 contract and the `heal_erasure_disk_rebuild_test` lane on main since 2026-09-02.
When the disk-check bypass is set and the replacement target fails admission, the erasure-set heal now logs a warning and downgrades to the ordinary set-wide `heal_format` path that predated replacement admission, and the auto-scan no longer defers such endpoints. The mount admission itself is unchanged and still cannot be bypassed by any environment variable; deployments without the bypass keep failing closed.
The endpoint-blackhole heal scenario now probes whether `iptables` can read the OUTPUT chain and logs an explicit skip when the host lacks `CAP_NET_ADMIN` (containerised runners report "Permission denied" from the nf_tables backend even under sudo); `RUSTFS_E2E_REQUIRE_NET_FAULT_INJECTION=1` turns that into a failure for lanes that provision the capability. The CI full-gate job surfaces the missing capability as a workflow warning, and the runtime-wipe fixture retries `remove_dir_all` on the listing race macOS surfaces as `DirectoryNotEmpty`.
Refs rustfs/backlog#2357.
Treat a negative wall-clock age as zero without bypassing count or byte eviction. Cover canonical and alias queries, terminal outcomes, exact TTL expiry, and capacity limits during rollback.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Retain failed identities in an execution-local count and byte bounded window so healthy later pages can advance. Preserve retry jitter, deadlines, terminal accounting and pressure pacing, with deterministic head-of-line and capacity regressions.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Stop task completion and legacy notices from discharging scanner retry hints. Preserve existing hints and their retry due time across admission observations, bound retry scheduling, and synchronize changed batches once even on cancellation.
Exercise the production MRF consumer, manager, event channel and scanner ledger. Document producer durability gaps without enabling successor activation or garbage collection.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
The heal manager carried its own byte-identical copy of the foreground pressure type and threshold computation that ecstore's data-movement backpressure also carries, so every change to the admission-utilization rules had to be mirrored by hand across two crates. The shared `ForegroundPressure` and `foreground_pressure` added to `rustfs-concurrency` now own that logic, and heal already depends on that crate, so this removes the duplicate without adding a crate edge.
`mainline_throttle_active` keeps the parts that are specific to this call site: the `mainline_throttle_enable` and both-thresholds-zero short circuit that avoids touching the provider at all, the optional-provider unwrap, and the heal-side threshold fields. Everything downstream is untouched — the `reason()` labels `foreground_read_pressure`, `foreground_write_pressure`, and `foreground_pressure` are byte-identical to the removed implementation, so the `rustfs_heal_mainline_throttle_total` reason label and the `heal_mainline_throttle` log fields keep their observability contract.
Refs rustfs/backlog#2049
(cherry picked from commit ec491bcbd8939e5978cd94f9a44cffb70d09fade)
(cherry picked from commit e800f29d6806591689204b3712300800b480beae)
Co-authored-by: houseme <housemecn@gmail.com>
refactor(heal): classify recoverability typed-first with documented needle fallback
Backlog#1845 step 6. Heal's retry decision leaned on substring matching of rendered messages; the typed information available in the error values now takes priority:
- Lock failures classify by LockError's own taxonomy instead of the blanket Lock(_) => recoverable: fatal variants (ResourceNotFound / PermissionDenied / Configuration) are terminal since retrying cannot fix them, while contention and transport variants (Timeout, Network, Internal, AlreadyLocked, QuorumNotReached, InsufficientNodes, ...) stay recoverable exactly as before.
- DiskError::RemoteClientUnavailable and its StorageError twin (typed in #6619) join the typed recoverable lists, so client-acquisition failures no longer depend on which needle happens to appear in the detail.
- task.rs is_transient_lock_or_timeout_error consults LockError::is_retryable / QuorumNotReached and the typed Timeout variants before falling back to needles.
- The substring list is demoted to a documented fallback: every needle now carries a producer census comment naming what emits it, with the shrink-only rule stated (delete the needle when its producer becomes typed end-to-end). heal rename incomplete remains the one needle with no typed producer.
heal gains a direct rustfs-lock dependency (already transitive via ecstore) to name LockError variants.
New tests pin each typed source: contention/transport lock variants recoverable, fatal lock variants terminal, RemoteClientUnavailable recoverable with a detail that avoids every needle. All existing recoverability tests stay green.
Ref rustfs/backlog#1845
refactor(protos): move internode compat manifest send-site assertions into owning crates
Promotes the rolling-upgrade dual-write manifest from a test-only constant in rustfs-protos into the public rustfs_protos::compat_manifest module, moves the JSON-encoder send-site assertions into the crates that own the asserted sources (ecstore remote_disk.rs for requests, the rustfs binary node_service/disk.rs for responses), and splits the scanner Phase-0 overlap inventory so its heal- and ecstore-owned halves live in those crates. Adds a cross-crate include_str!/include! guard with fixture self-tests to scripts/check_layer_dependencies.sh so a library crate can never again read another crate's Rust source at compile time, and records the rule in docs/architecture/crate-boundaries.md.
Part of rustfs/backlog#1884.
Move MRF repaired-event fan-out from admission to successful terminal completion so scanner pending-heal ledgers only clear after the canonical heal task actually finishes. Track notice ownership across duplicate admission, retry merge, cancellation, and queue displacement.
Co-authored-by: heihutu <heihutu@gmail.com>
Cache heal queue dedup keys, avoid retry request double construction, clear task aliases after terminal completion, and age out stale scanner pending-heal ledger entries during retry sweeps.
Co-authored-by: heihutu <heihutu@gmail.com>
Split the 6723-line manager.rs (44% inline tests) into a canonical
manager.rs + manager/ module tree with zero behavior change:
- manager.rs (~1830): HealManager and HealState, HealConfig, task
report/snapshot types, overlap policy, admission classification and
queue admission, submit paths, task-state queries, and the
statistics surface
- manager/queue.rs (~450): the priority heal queue, its per-key dedup
index, and the queue bookkeeping structs
- manager/scheduler.rs (~620): start_scheduler and the
process_heal_queue consumption loop with its skip/metric helpers
- manager/auto_scan.rs (~550): the automatic disk scanner
- manager/unclean_shutdown.rs (~390): unclean-shutdown recovery and
its durable replacement-intent helpers
- manager/tests.rs (~2970): the inline test module as a child module
All module paths are unchanged. The queue structs' fields and the
cross-module helpers gain pub(super), whose scope equals the old
single-module privacy domain; HealManager's private fields stay in the
root and remain reachable from child impl blocks. Code is moved
verbatim apart from those markers, heal-level super:: path fixes for
the unclean-shutdown move, per-module import headers, and rustfmt
re-wraps.
The logging-guardrail rule for the manager demote_to_debug_when! count
now sums manager.rs with its manager/*.rs children, since one
scheduler site moved with process_heal_queue; the >= 6 threshold is
unchanged and the forbidden admission info!/warn! pattern check keeps
targeting the root admission code.
Co-authored-by: heihutu <heihutu@gmail.com>
Split the 4242-line resume.rs (46% inline tests) into a canonical
foo.rs + foo/ module tree with zero behavior change:
- resume.rs (~1020): state file constants, PersistThrottle, ResumeState,
ResumeManager core (constructors, load/discovery, progress mutators,
ordinary persistence) plus root re-exports
- resume/replacement.rs (~690): replacement-intent/proof types and the
ResumeManager replacement-lifecycle methods
- resume/checkpoint.rs (~350): ResumeCheckpoint + CheckpointManager
- resume/utils.rs (~310): ResumeUtils statics
- resume/tests.rs (~1980): the inline test module as a child module
All module paths are unchanged (heal::resume::CheckpointManager and
friends resolve through root re-exports), so no consumer inside or
outside the crate changes. Items defined in child modules keep
module-private visibility; only the ten cross-module helpers gain
pub(super), which is not part of the crate API. Code is moved verbatim
apart from those visibility markers, four super::storage_api path
fixes, and the new per-module import headers.
Co-authored-by: heihutu <heihutu@gmail.com>
houseme