Do not retain Accepted or Merged replay intents as startup anchors after they have been handed to the heal manager. Only refused or still-pending replay records keep the journal on disk until a successor snapshot can persist them.
This keeps successor snapshots limited to the pending queue, which lets successful replay remove both authoritative and legacy journal paths and restores the crash-boundary tests around successor flush.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit d5b8f49c9d)
Keep replayed MRF records crash-replayable after manager admission until a later durable successor proof can tombstone them. Queue-full and transient replay submission failures now also preserve the old journal anchor instead of allowing cleanup to erase the only recovery source.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Add an MRF pipeline regression that replays a single authoritative journal containing same-object records that differ by kind and erasure-set scope while a stale legacy mirror is present. The test proves restart replay admits each authoritative responsibility independently and never merges the stale mirror epoch.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Introduce an object heal receipt wrapper so storage owners can report a positive repaired or verified result without changing legacy heal consumers.
Task-level object heal now records only receipts that match the requested object identity, version, pool, set, and carry a bucket incarnation. Legacy and mismatched receipts still fall back to unknown outcome accounting.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Add a Unix process-kill fixture for the authoritative successor fsync boundary where the scoped journal is durable while the legacy mirror still contains the startup epoch. Also extend heal-control transport replay coverage so a duplicate producer after a lost response receives the canonical merged receipt without creating another task.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): remove unused digest import
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): add raw page owner index (#7375)
* feat(scanner): add raw page owner index
Add a serializable raw enumeration page owner index for scanner resume work.
The index exposes unsupported, building, and ready states, validates committed page identity by recomputing digests, and uses generation checks for CAS-style page commits.
Focused tests cover small-budget restart progress, page digest/source drift rejection, corrupt deserialized state, CAS failure, precommit crash, empty sources, and invalid entry boundaries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): persist raw page owner resume state (#7379)
Wire the scanner raw enumeration partial-cache writer to the raw page owner index so interrupted bucket walks can retain validated page-builder state across scanner restarts.
Keep complete owner sources terminal-only, add partial-source ingestion for in-progress raw directory reads, and validate the persisted page index through bucket checkpoint preparation.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): fence segment producer observations (#7381)
Require the segment observation fixture to carry source, incarnation, key-format, baseline, process epoch, generation-window, gap, overflow, and producer-coverage proof before accepting a narrowed proposal. Keep the diagnostic path fixture-only and remove its ordinary stderr output.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): isolate pool metadata read probes (#7367)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF crash successor matrix (#7369)
* test(heal): cover MRF crash successor matrix
Add process-boundary MRF replay coverage for the successor snapshot window after a retained startup journal is flushed but before cleanup deletes it. Extend the mixed authoritative/legacy reader fixture with a scoped v2 journal epoch to pin the no-merge contract.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover service-kill MRF replay (#7380)
Add a Unix process fixture that waits after publishing the pending MRF successor snapshot, then is terminated by the parent before restart replay.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover transport-lost start receipts (#7371)
Add gRPC transport fault fixtures for heal-control start admission. The tests distinguish pre-admission transport loss from post-admission response loss, then verify exact envelope retries reuse the canonical receipt while fresh forceStart requests create distinct tasks.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add crash-restart heal evidence case (#7370)
* test(scanner): add crash-restart heal evidence case
Add a distinct W21 background target crash case to the scanner/heal evidence registry and oracle path.
Keep the existing restart lane on graceful process restart, keep the crash lane on hard kill, and make the wiring checker reject evidence/oracle mismatches.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): support older Python wiring checks
Let the scanner/heal evidence wiring checker run under Python 3.9/3.10 by falling back to tomli and chunked SHA-256 hashing when the Python 3.11 standard APIs are unavailable.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): reject stale raw page source seeds (#7382)
Do not prefill a resumed raw page owner with previously indexed entries when starting a new raw directory observation pass. The next pass must observe the same prefix again before the page index can advance; otherwise the index is discarded fail-closed.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): defer raw page revalidation until observed (#7384)
A resumed raw page owner index must not prefill entries from older cache state, but it also must not discard a valid multi-entry index before the current raw directory pass has observed enough entries to prove identity. Track the persisted index floor and only run the strict owner identity check once the current pass reaches that floor.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Add an ECStore-backed child-process fixture that publishes the MRF journal in one OS process, then verifies a restarted manager can replay the complete snapshot while retaining the journal when bounded admission accepts only a prefix.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Keep the startup journal on disk when replay cannot fully re-arm or the heal manager refuses a replayed intent with Full/QueueFull. The next live snapshot can still advance the journal after the retry anchor is durable again.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
fix(heal): fall back to set-wide format for directory-backed replacement targets
Since #7018 `renew_disk` routes an unformatted local endpoint through the automatic replacement heal, which requires the target to be an independently mounted disk. Directory-backed deployments (the operator set `RUSTFS_UNSAFE_BYPASS_DISK_CHECK`, which the startup disk-independence check already requires for endpoints sharing a device) can never pass that admission, so a runtime-wiped or replaced directory disk stayed unformatted forever: the heal task failed with "replacement target is not a stable mounted disk" and the auto-scan kept deferring the endpoint. This broke the Issue #1533 contract and the `heal_erasure_disk_rebuild_test` lane on main since 2026-09-02.
When the disk-check bypass is set and the replacement target fails admission, the erasure-set heal now logs a warning and downgrades to the ordinary set-wide `heal_format` path that predated replacement admission, and the auto-scan no longer defers such endpoints. The mount admission itself is unchanged and still cannot be bypassed by any environment variable; deployments without the bypass keep failing closed.
The endpoint-blackhole heal scenario now probes whether `iptables` can read the OUTPUT chain and logs an explicit skip when the host lacks `CAP_NET_ADMIN` (containerised runners report "Permission denied" from the nf_tables backend even under sudo); `RUSTFS_E2E_REQUIRE_NET_FAULT_INJECTION=1` turns that into a failure for lanes that provision the capability. The CI full-gate job surfaces the missing capability as a workflow warning, and the runtime-wipe fixture retries `remove_dir_all` on the listing race macOS surfaces as `DirectoryNotEmpty`.
Refs rustfs/backlog#2357.
Treat a negative wall-clock age as zero without bypassing count or byte eviction. Cover canonical and alias queries, terminal outcomes, exact TTL expiry, and capacity limits during rollback.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Retain failed identities in an execution-local count and byte bounded window so healthy later pages can advance. Preserve retry jitter, deadlines, terminal accounting and pressure pacing, with deterministic head-of-line and capacity regressions.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Stop task completion and legacy notices from discharging scanner retry hints. Preserve existing hints and their retry due time across admission observations, bound retry scheduling, and synchronize changed batches once even on cancellation.
Exercise the production MRF consumer, manager, event channel and scanner ledger. Document producer durability gaps without enabling successor activation or garbage collection.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
The heal manager carried its own byte-identical copy of the foreground pressure type and threshold computation that ecstore's data-movement backpressure also carries, so every change to the admission-utilization rules had to be mirrored by hand across two crates. The shared `ForegroundPressure` and `foreground_pressure` added to `rustfs-concurrency` now own that logic, and heal already depends on that crate, so this removes the duplicate without adding a crate edge.
`mainline_throttle_active` keeps the parts that are specific to this call site: the `mainline_throttle_enable` and both-thresholds-zero short circuit that avoids touching the provider at all, the optional-provider unwrap, and the heal-side threshold fields. Everything downstream is untouched — the `reason()` labels `foreground_read_pressure`, `foreground_write_pressure`, and `foreground_pressure` are byte-identical to the removed implementation, so the `rustfs_heal_mainline_throttle_total` reason label and the `heal_mainline_throttle` log fields keep their observability contract.
Refs rustfs/backlog#2049
(cherry picked from commit ec491bcbd8939e5978cd94f9a44cffb70d09fade)
(cherry picked from commit e800f29d6806591689204b3712300800b480beae)
Co-authored-by: houseme <housemecn@gmail.com>
refactor(heal): classify recoverability typed-first with documented needle fallback
Backlog#1845 step 6. Heal's retry decision leaned on substring matching of rendered messages; the typed information available in the error values now takes priority:
- Lock failures classify by LockError's own taxonomy instead of the blanket Lock(_) => recoverable: fatal variants (ResourceNotFound / PermissionDenied / Configuration) are terminal since retrying cannot fix them, while contention and transport variants (Timeout, Network, Internal, AlreadyLocked, QuorumNotReached, InsufficientNodes, ...) stay recoverable exactly as before.
- DiskError::RemoteClientUnavailable and its StorageError twin (typed in #6619) join the typed recoverable lists, so client-acquisition failures no longer depend on which needle happens to appear in the detail.
- task.rs is_transient_lock_or_timeout_error consults LockError::is_retryable / QuorumNotReached and the typed Timeout variants before falling back to needles.
- The substring list is demoted to a documented fallback: every needle now carries a producer census comment naming what emits it, with the shrink-only rule stated (delete the needle when its producer becomes typed end-to-end). heal rename incomplete remains the one needle with no typed producer.
heal gains a direct rustfs-lock dependency (already transitive via ecstore) to name LockError variants.
New tests pin each typed source: contention/transport lock variants recoverable, fatal lock variants terminal, RemoteClientUnavailable recoverable with a detail that avoids every needle. All existing recoverability tests stay green.
Ref rustfs/backlog#1845
refactor(protos): move internode compat manifest send-site assertions into owning crates
Promotes the rolling-upgrade dual-write manifest from a test-only constant in rustfs-protos into the public rustfs_protos::compat_manifest module, moves the JSON-encoder send-site assertions into the crates that own the asserted sources (ecstore remote_disk.rs for requests, the rustfs binary node_service/disk.rs for responses), and splits the scanner Phase-0 overlap inventory so its heal- and ecstore-owned halves live in those crates. Adds a cross-crate include_str!/include! guard with fixture self-tests to scripts/check_layer_dependencies.sh so a library crate can never again read another crate's Rust source at compile time, and records the rule in docs/architecture/crate-boundaries.md.
Part of rustfs/backlog#1884.
houseme