Record bounded first/last raw entry markers in the restart diagnostic worker and have the driver classify repeated raw windows when retained coverage does not advance.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Keep the startup journal on disk when replay cannot fully re-arm or the heal manager refuses a replayed intent with Full/QueueFull. The next live snapshot can still advance the journal after the retry anchor is durable again.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Restore the previous control-file bytes, or remove a newly created file, when the Unix compare-and-update path reaches the rename but then fails to fsync the parent directory. This keeps failed metadata CAS publications from advancing recovery anchors.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Allow the scanner/heal release evidence e2e helper to bind the active case identity and oracle file through an explicit descriptor instead of hard-coding the background target restart artifact. Extend the checker self-test so a single run can finish multiple registry-declared oracles while release gates remain pending until every required lane is complete.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Add a durable raw enumeration cursor shape to scanner usage metadata and validate it against bucket identity, source, bounds, version, and page digest before preserving it across checkpoint preparation.
Keep empty cursor metadata omitted so existing pinned .usage-cache.bin bytes stay unchanged, while legacy readers still ignore the additive field when it is present.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
The never-versioned listing fast path emitted any non-empty directory
without xl.meta as a prefix, which surfaced the data dirs deleted
versions leave behind as phantom folders that HEAD, listing and
recursive remove could not touch. Probe such directories for the
delete-residue shape before emitting them, and let any complete empty
first page of a prefix listing trigger the existing fail-closed orphan
purge so ordinary browse and remove traffic reclaims committed residue.
The admin storage walk probed a set's drives one after another, each
bounded by the disk_info timeout, so a few drives still recovering
after a power cut pushed the local snapshot past the peer probe budget
and healthy peers rendered as unknown. Probe all drives at once so the
walk costs one timeout at most, and add a test-only probe delay hook to
pin that bound.
Multipart parts shared the 250 ms direct-PutObject wait on the foreground
write permit pool, so SDK-default concurrency (many parts per upload in
flight at once) was rejected wholesale with SlowDown at stock settings.
Keep the pool that bounds in-flight bodies, but let parts wait in a
bounded queue with their own timeout before body ingest, report the
queue depth in the ForegroundWrite admission snapshot, and document the
foreground write admission environment variables.
fix(heal): fall back to set-wide format for directory-backed replacement targets
Since #7018 `renew_disk` routes an unformatted local endpoint through the automatic replacement heal, which requires the target to be an independently mounted disk. Directory-backed deployments (the operator set `RUSTFS_UNSAFE_BYPASS_DISK_CHECK`, which the startup disk-independence check already requires for endpoints sharing a device) can never pass that admission, so a runtime-wiped or replaced directory disk stayed unformatted forever: the heal task failed with "replacement target is not a stable mounted disk" and the auto-scan kept deferring the endpoint. This broke the Issue #1533 contract and the `heal_erasure_disk_rebuild_test` lane on main since 2026-09-02.
When the disk-check bypass is set and the replacement target fails admission, the erasure-set heal now logs a warning and downgrades to the ordinary set-wide `heal_format` path that predated replacement admission, and the auto-scan no longer defers such endpoints. The mount admission itself is unchanged and still cannot be bypassed by any environment variable; deployments without the bypass keep failing closed.
The endpoint-blackhole heal scenario now probes whether `iptables` can read the OUTPUT chain and logs an explicit skip when the host lacks `CAP_NET_ADMIN` (containerised runners report "Permission denied" from the nf_tables backend even under sudo); `RUSTFS_E2E_REQUIRE_NET_FAULT_INJECTION=1` turns that into a failure for lanes that provision the capability. The CI full-gate job surfaces the missing capability as a workflow warning, and the runtime-wipe fixture retries `remove_dir_all` on the listing race macOS surfaces as `DirectoryNotEmpty`.
Refs rustfs/backlog#2357.
Add a real ECStore reopen regression for scanner usage-state reset cleanup boundaries. The fixture seeds each partially completed cleanup state, recreates the store, then verifies the reset resumes without rewriting the bootstrap intent or deleting unrelated metadata.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): require root publication proof before dirty ack
Bind ACK expectations to validated scan candidates and confirm the actual primary-root revision and readback. Retain saved outcomes and dirty responsibility when stronger evidence is unavailable. Isolate CAS attempt confirmation and invalidate proof after scope mutations.
Revalidate observed candidate reuse before issuing a new publication proof, preserve the exact validated authoritative baseline work digest, and settle fixture commit tails before stable maintenance scans. Keep scoped ACK production disabled.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix: restore ILM transition and lifecycle validation (#7316)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Treat a negative wall-clock age as zero without bypassing count or byte eviction. Cover canonical and alias queries, terminal outcomes, exact TTL expiry, and capacity limits during rollback.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
feat(scanner): retain bounded bootstrap admission fairness
Keep a leader-local bounded cohort across scanner retries and preserve
waiting bucket priority during dirty arrivals and capacity overflow.
Order source permits in the dispatcher without changing result identity,
parent budgets, explicit cycle timing, or persistent coverage evidence.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Retain failed identities in an execution-local count and byte bounded window so healthy later pages can advance. Preserve retry jitter, deadlines, terminal accounting and pressure pacing, with deterministic head-of-line and capacity regressions.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Run one supervised cleanup attempt for an existing operator reset, with
strict phase and revision checks under the original leader lock. Keep v3
reset authorization and responses unchanged, report deferred status, and
bound probe and shutdown waits without aborting in-flight reset ownership.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Stop task completion and legacy notices from discharging scanner retry hints. Preserve existing hints and their retry due time across admission observations, bound retry scheduling, and synchronize changed batches once even on cancellation.
Exercise the production MRF consumer, manager, event channel and scanner ledger. Document producer durability gaps without enabling successor activation or garbage collection.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
houseme