* fix(decommission): move the free version of an expired tier version
Decommission only queued the lifecycle expiry of a transitioned version, so a source cleanup that ran first deleted the version itself and appended its tier free version after the entry's last pass. The post-check then failed the decommission until an operator re-ran it.
Apply the expiry of a transitioned version synchronously, as rebalance does. Its free version is on the source before the cleanup preflight, which reports the entry changed, and the retried entry moves the record to a target in the same pass. An expiry that cannot be applied leaves the version live, so it moves like any other version.
Refs rustfs/backlog#2873
* fix(decommission): move a delete-all expired tier version live
Applying a delete-all expiry inline made its prefix delete wait for the bucket lifecycle write lock behind the entry's own read lock until the lock timeout, blocking every write in the bucket meanwhile. Move such a transitioned version live and leave the delete-all to lifecycle on the target.
Cover a genuinely unapplied expiry and a free-version worker that consumes the source record before the entry moves it.
Refs rustfs/backlog#2873
* fix(heal): explain unverified legacy heal outcomes
Preserve storage diagnostics in canonical unknown outcomes and explain
legacy payload identity limits through the shared local/peer status path.
Borrow expected receipt identities while preserving all proof checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): keep diagnostics from pausing merged MRF repairs
Require the actual MRF executor before publishing legacy pause events.
An ordinary queued owner can retain a merged durable MRF notice without
having performed the specialized deep check. Preserve its diagnostic
without granting it authority to park the repair responsibility.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): type MRF outcomes and defer status diagnostics
Carry storage-owned legacy observations in process and return a typed
terminal result only after the qualified MRF Deep check. Publish pause
events from that result and exact identities instead of source or text.
Keep object names out of retry classification and compute only the
status diagnostic selected by the canonical execution state.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): add scanner/heal topology protection matrix
Add the S00 protection baseline for the Scanner/Heal program
(rustfs/backlog#2873, program rustfs/backlog#2872): seven
scanner_heal_topology_ cases in the distributed e2e lane that pin pool
expansion, rebalance and decommission behavior before the runtime changes.
The fixture seeds a deterministic inventory (empty, inline, external,
multipart, special and nested keys; a versioned bucket with an explicit
null version, three history versions and a delete marker per key; an
Object Lock bucket with legal hold and COMPLIANCE retention), records
every version with its requested and listed version id, length and
SHA-256, verifies the manifest through every node, reads physical
placement from xl.meta on the pool drives, and reads the strict
durability mode back from the admin API and the node logs.
Fault cases hold the mover at proven windows through file barriers that
compile only with the e2e-test-hooks feature: before the multipart target
commit and before source cleanup (shared by decommission and rebalance),
plus the existing pool-metadata before_publish cutpoint. An unreached
window fails the case. DistCluster gains capture_logs_under so a case can
read node startup events without a global environment variable.
docs/testing/scanner-heal-topology-gates.md maps the gates to the tests
and records the timeout basis.
* test(e2e): keep the barrier failure message static
The error(format!) ratchet rejects new formatted io::Error::other call
sites in ecstore. The reached observation already records the stage and
object, so the injected failure needs no per-call detail.
* test(e2e): require e2e-test-hooks for topology fault cases
The four scanner_heal_topology_ fault cases inject at server barriers that
exist only with the e2e-test-hooks feature. Against a default server they
waited for the barrier timeout and reported it as not reached. Check the
verified binary's features first so the missing prerequisite fails
immediately and is named.
* ci(e2e): build the distributed lane with e2e-test-hooks
Every other e2e lane builds and runs the fault-test hooks server. The
distributed lane alone used the default server, so the topology fault
cases could never reach their barriers there. Build and run with the same
feature so the receipt matches, and update the documented local commands.
The upgrade cases spawn the pinned previous release directly and do not
compare feature sets.
* test(e2e): keep topology artifacts under RUSTFS_E2E_LOG_DIR
The topology fixture captured node logs, barrier observations and evidence
in a temp dir removed on drop, so the distributed workflow uploaded nothing
for these cases. Fall back to RUSTFS_E2E_LOG_DIR, which the workflow
uploads, when no explicit evidence directory is set.
* test(e2e): remove retired pools before restart; race the pre-ACK start
The queue case restarted with decommissioned pools 0 and 1 still on the
command line, which the server rightly refuses. Add
remove_single_node_pools and restart without them, then expect the
remaining pools renumbered to 0 and 1.
The pre-ACK case now races the start request against the barrier. A start
that returns first, or stays in flight past the documented 60 s budget, is
reported as such instead of as an unreached barrier, and the evidence
records the reach latency.
capture_logs_under now overrides a log path assigned from
RUSTFS_E2E_LOG_DIR, so the seed node logs into the case directory too.
* test(e2e): bound fault-window waits by the movement budget
On Linux CI a busy target capacity gate delayed the first fault object by
about two minutes, past the 120 s wait borrowed from the server's release
poll. Wait for a movement window within the 240 s movement budget instead,
and fail at once when the decommission or rebalance ends without entering
the window, so an uninjected fault is never counted.
* test(e2e): bound status probes while waiting for a fault window
The admin decommission status request can wait on a mover held at a test
barrier until the barrier's own 120 s release timeout. The window wait
issued that request each loop, so a reached barrier went unnoticed until
the server timed it out and failed the decommission, which surfaced as a
false resume failure. Bound each status probe to 5 s and treat a slow probe
as still running.
* test(e2e): retry a busy decommission start
A decommission start that meets a busy movement gate can answer a
retryable 503 instead of waiting. The harness treated any non-proof
failure as fatal. Share one start helper that retries 503 within the
case's movement budget and the fleet-proof 500 for 45 s, and keeps every
other status fatal. The pre-ACK case retries 503 within its window budget.
* fix(decommission): defer source cleanup quorum failures
A quorum failure while deleting the source of an already copied entry
was fatal and marked the whole pool failed, although the copied source
is still in place and an uncommitted delete transaction is settled by
delete recovery. Carry the cleanup error typed and resolve a quorum
failure as Deferred: the entry is replayed at the end of the round, and
a deferral that persists takes the existing capacity-blocked pause path
instead of a terminal failure. Other cleanup errors and quorum errors
from other stages stay terminal.
Add set-level regressions for parent and child keys (`a/b`, `a/b/c`):
parent cleanup leaves every child file untouched, and parent cleanup
reaches quorum while the child is cleaned up or rewritten concurrently.
* test(ecstore): keep source cleanup errors typed in prefix cleanup tests
Return SourceCleanupError from the test helper instead of formatting it
into Error::other, and shrink the error(format!) ratchet baseline for
core/pools.rs, which lost one call site in the cleanup failure change.
* fix(decommission): retry cleanup quorum failures and count replays once
Retry a source cleanup quorum failure in place, up to three attempts
with the existing source-cleanup backoff, before deferring the entry.
Carry the versions an entry already counted on its deferral, so the
end-of-round replay does not add a fully counted entry to
items_decommissioned and bytes_done again.
A deferral that persists pauses the pool through the existing deferred
drain path. When source cleanup caused it, persist a reason led by
"source cleanup blocked" without the carrier error's free-drive wording,
and report waitingReason "source_cleanup" instead of "capacity".
Add a source_cleanup test fault stage and a routine test that drives a
cleanup quorum fault through deferral and replay to completion.
* fix(decommission): wait seconds between cleanup quorum retries
Give the in-place source cleanup quorum retry a fixed floor of one and
then two seconds, with the existing jitter on top, so it can ride out a
drive reconnect or health-check blip instead of retrying within a few
hundred milliseconds. The wait stays bounded and cancel-safe.
Find a source cleanup pause reason through stage wrappers, as the
capacity-blocked check does, and add a routine test where cleanup keeps
losing quorum: the decommission pauses with a source cleanup reason, the
bucket stays queued and the source stays in place.
Document the additive waitingReason value source_cleanup and its
compatibility with older nodes reading the same pool.bin.
* fix(decommission): never retry a lost source cleanup fence
The cleanup retry classifier used is_quorum_error(), which also matches
NamespaceLockQuorumUnavailable. The delete commit fence reports a lost
namespace lock or mutation fence with that variant, so a lost fence was
retried in place and only stopped later at the pre-cleanup fence check.
A lost fence means this worker no longer owns the key; it must stop
without retrying. Retry and defer only drive quorum failures:
ErasureReadQuorum, ErasureWriteQuorum, InsufficientReadQuorum and
InsufficientWriteQuorum.
---------
Co-authored-by: Hauser <housemecn@gmail.com>
* fix(decommission): let a failed start roll back its own activation
A fresh decommission start whose peer reload failed could not roll back:
the pool metadata stale-write guard from #6604 refuses any save that
clears a persisted active or queued decommission, so the pool stayed
active with no worker until the watchdog or a leader restart picked it
up.
The rollback save now carries the activation that start committed. The
guard exempts a pool only while its persisted record still equals that
activation field for field, so a worker checkpoint or a later start
keeps the rejection. A canceled record is not an active decommission
and is cleared by the rollback as well. Every other save path passes no
activation and is guarded as before.
A peer may already have reloaded the activation and started a worker
before the rollback. Saves are now fenced by decommission generation: a
write carrying a start_time is rejected when the persisted record
belongs to a later start, or has no decommission state and was updated
after that start_time, so such a worker cannot make the pool active
again. Its queued promotion and progress checkpoint also adopt the
persisted record and stop the worker.
Refs rustfs/backlog#2873
* fix(decommission): fence terminal writes of a withdrawn decommission
Terminal records carry no start_time, so the generation fence did not
cover them: after a start rollback, a peer worker of the rolled back
generation that failed (for example on a bucket completion) could still
persist a failed record over the restored pool. The merge now rejects a
failed, canceled or complete record over a persisted record without
decommission state; every attempt persists decommission state first, so
no legitimate path writes a terminal record there.
Peers also drop a withdrawn decommission sooner. A pool metadata reload
replaces a running decommission with a newer record that withdrew it,
even under an active worker; bucket completion adopts such a record like
promotion and progress checkpoints do; and a withdrawn progress
checkpoint stops the worker instead of logging a failed save and moving
more entries.
Tests cover the peer failure mark, a rollback retry that sees a worker
checkpoint land between attempts, and the reload withdrawal. A heal
fixture that wrote complete records over a fresh pool.bin now persists
the active attempt first.
Refs rustfs/backlog#2873
* fix(decommission): fence peer writes after a retry start rollback
A retry start replaces a failed or canceled record that keeps its
unresolved ledger, so its rollback restores that terminal record rather
than clearing the pool. The withdrawal checks only recognized a record
without decommission state, and the generation fence ignored a persisted
record without a start_time. A peer worker of the rolled back retry
could therefore checkpoint the pool back to active.
The generation fence now compares the persisted last_update whenever
the persisted record has no start_time of its own (cleared, terminal or
queued). The progress checkpoint also checks the generation against the
record read under the pool metadata lock, as bucket completion already
does, and the save-path withdrawal check treats a terminal record newer
than the running view as a withdrawal, so the peer adopts it and stops.
A reload still never lets a newer terminal record replace an active
worker's view, so after a retry's rollback a peer worker stops at its
next bucket completion or progress checkpoint rather than on reload.
Tests cover the retry rollback against a peer checkpoint and direct
write, and a peer bucket completion after a fresh start's rollback.
Refs rustfs/backlog#2873
* fix(decommission): keep a retry rollback's restored record intact
After a retry start rolls back, the restored record is failed, so a
peer worker's own failure mark passed every merge guard and replaced
it. A failure mark now saves through save_current_pool_meta_checked,
which runs the withdrawal check against the record read under the pool
metadata lock and adopts that record instead of writing.
The withdrawal check compared the restored record with the worker's
in-memory last_update, which a rejected save had already raised, so the
peer never adopted the rollback and the watchdog kept resuming it. It
now compares against the running generation (start_time), and a
rejected unresolved-entry save undoes its in-memory change.
save_current_pool_meta_for_decommission_start is also compiled for the
test-util feature again; a test-util helper still calls it, which broke
the workspace clippy and Connect builds.
Refs rustfs/backlog#2873
* fix(decommission): keep an unresolved entry when its own save fails
The rollback of a rejected unresolved-entry save also dropped the entry
on transient failures such as a quorum or lock-fence error, so the
following failure mark persisted a failed record without it and an
operator could clear a decommission that skipped an unreadable object.
Keep the entry and undo only the last_update bump, which is all a
withdrawn worker needs to still see the withdrawal.
Refs rustfs/backlog#2873
Under sustained writes nearly every cycle fell back to a full walk: scope
admission required the global dirty generation to stay unchanged across the
bucket listing, and after a Complete cycle the dirty-usage wake ran the next
full walk immediately, back to back.
- Track dirty-map removals with a process-local removal generation and admit
a scoped walk when no entry was removed or replayed during the listing
(`covers_listed_pending`). Writes to listed buckets in the window join the
start snapshot; unlisted buckets dirtied after the cutoff stay pending and
are never cleared by compare-and-remove certification. The strict
`covers_all_pending` still gates segment-reuse evidence.
- Classify scope fallbacks (removal during listing, generation exhausted,
dirty-bucket overflow, no listed dirty bucket, baseline unavailable, peer
scope unavailable), log `scoped_scan_fallback`, and report them in
usage_freshness as additive serde(default) fields.
- Pace the cycle after a Complete fallback while dirty buckets remain on the
capped retry schedule clamped to the cycle interval, with the dirty wake
disabled; the paced timer wake still requests a scope.
- Update the per-bucket convergence design doc.
No persisted format or internode protocol change.
Co-authored-by: Chris <anzhengchao@gmail.com>
* fix(ecstore): leave no source free version for a moved tier version
Data-movement source cleanup deletes each source version separately and
never set skip_tier_free_version, so every transitioned version that
decommission or rebalance had already copied to a target pool left a new
tier free version on the source. Decommission then failed its post-check
for any pool with tiered objects; rebalance left the record behind. The
record claims the remote object the target copy still reads and stays
blocked behind that live reference.
Decommission and rebalance now pass the identities of transitioned
versions whose copy committed on a target, and the cleanup skips the
free version for exactly those. A transitioned version that did not
move, such as one lifecycle expired during the move, still gets its free
version because that record is its only remote-delete owner.
Refs rustfs/backlog#2873, rustfs/backlog#2872
* fix(ecstore): keep the free version of a superseded moved source
Data movement treats a source null version as complete when a newer
unversioned object already sits on the target, without copying it.
That outcome returned the same Ok as a committed copy, so a restored
transitioned null version superseded this way was recorded as
transferred and lost its source free version, leaving its remote object
with no owner.
The copy paths now return DataMovementCopy::Committed or ::Superseded,
and decommission and rebalance record only committed copies of
transitioned versions. A superseded source keeps its free version.
Tests cover the superseded restored null version for rebalance and
decommission, a restored and a null transitioned version moving by
rebalance, an entry with one lifecycle-expired and one moved version,
the per-version skip with two versions of one key, and a directory
object. The tiered stale-target routine test from #8724 now expects the
decommission to complete. The ILM ownership contract and the
decommission notes describe the moved, superseded, expired and legacy
cases.
Refs rustfs/backlog#2873, rustfs/backlog#2872
* test(ecstore): build the decommission ILM fixtures without s3s
The superseded and expiry-mix decommission tests named s3s request and
lifecycle types directly, which added pools_test.rs to the ecstore
files that reference s3s and failed the footprint ratchet. Set the
restore request through ObjectOptions, and store the lifecycle rule as
bucket metadata XML and load it with get_expiry_configs, as the
decommission routine does.
* test(ecstore): pin restored decommission copies and multipart supersession
Decommission recorded a committed data copy of a restored transitioned
version without any test failing if the record were dropped; add a
routine-level decommission of a restored version that checks the post-
check passes, the source keeps nothing, and the target keeps the full
remote tuple. Add the multipart variant of the superseded restored null
version (rebalance), and a unit test that separates committed,
superseded and mismatched targets in the overwrite resume adjudication.
* fix(decommission): run the leader-side start to completion
A forwarded decommission start awaits `ECStore::decommission` inline in
the RPC handler. When the forwarding node's RPC times out, the stream is
reset and the handler future is dropped wherever it is. Dropped after
`pool.bin` records the pool as active but before the peers reload and
the worker is spawned, the start left the pool active with no worker and
peers on the old metadata, and a retry was rejected as already active.
An admin client that disconnects can do the same.
Run the start in its own task and let the caller await only its result.
The start then always reaches its end: it fails before persisting
anything, or persists the pool, has the peers reload and spawns the
local workers, rolling back if the reload fails. A retry still cannot
start twice: it waits on the start gate held by the running start and
either reports the busy gate or is rejected because the pool is active.
`decommission` now takes `self: &Arc<Self>` and spawns workers on that
store instead of looking up the global object store, which is the same
store in production and makes the path testable; the unused
`require_decommission_store` helper is removed.
Refs rustfs/backlog#2873
* fix(decommission): detach the leader-side start from its caller
9ac4c10de carried only the reproduction (the testable entry point, the
peer-reload pause point and the proof test). This adds the fix it
describes: `ECStore::decommission` spawns the start as its own task and
awaits only its result, so a caller that stops waiting, such as a
forwarded start whose internode RPC timed out, can no longer drop the
start between persisting the pool as active and having the peers reload
and the local worker start.
Also prove a retry while the first start runs cannot start the pool
twice, update the operator and architecture docs for the new behavior,
and shrink the error(format!) ratchet baseline for the removed helper.
Refs rustfs/backlog#2873
* fix(decommission): single-flight start, recheck pool state under gate
Review of 3a5276c5b:
- A start re-checks the pool state under the start gate before it
cancels routines, so a start racing another one can no longer cancel
the worker the other start just spawned.
- Only one start runs per instance at a time: another request, including
the retry of an operator whose request timed out, is rejected as
already running instead of stacking a second detached start and its
preflight heals. The start runs through `run_detached_mutation`, and a
failure nobody waits for any more is logged.
- Tests: the proof test pins its drop after the save and requires a
worker that persists a cancellation; the retry test really drops the
first caller; new tests cover the under-gate recheck and a peer reload
failure after the caller is gone. That last one shows the start
rollback cannot clear a fresh start because the pool metadata
stale-write guard refuses it; the gap is asserted and documented and
tracked in rustfs/backlog#2873.
- The start tests run on their own large-stack thread: the io_uring lane
(`uring_`, which also selects `during_`) runs them with libtest's 2 MiB
default stack, where the proof test overflowed.
- Docs: single-flight starts, status and cancel during preflight, the
rollback gap and its recovery, and the release requirement.
Refs rustfs/backlog#2873
* fix(decommission): turn away local start retries at the start slot
Second review of 29253430a:
- The leader-local admin pre-check refreshed rebalance metadata under
the start gate before the start slot was checked, so a local retry
waited out its 20 s gate budget behind a running start (which holds
the gate through its peer reload) and then got a 503. The pre-check
now checks the slot first and returns `DecommissionAlreadyRunning`,
which the handler reports as a 400 that keeps the reason.
- Prove the reload-failure path really ran in the detached start (the
injection is consumed), add a test that a fresh start's failed
rollback reports both the reload error and the stale-write refusal,
and assert an abandoned start stays tracked for shutdown.
Refs rustfs/backlog#2873
Second review of 652c56875..0ef02a485:
- The operator doc and the budget comment claimed a forwarded start that
outlives the internode RPC timeout still completes on the leader. The
leader drops it wherever it is, possibly after pool.bin records the
pool as active but before its worker starts or the peers reload.
Document that state, the watchdog that resumes the worker, and the
checks to run; the cancel-safety fix is tracked in rustfs/backlog#2873.
- Prove the budget is shared across waits (paused clock: a 60 ms wait
leaves 40 ms for the next gate).
- Prove the rebalance-refresh start gate and activation gate waits and
the decommission() entry are bounded and named.
Refs rustfs/backlog#2873
Review of 652c56875 found that the 30 s start budget was a wall-clock
deadline fixed when the start began, so slow preflight work (bucket heals,
capacity scans) could spend it before the activation gates were reached.
A gate held for a few milliseconds then failed the start at once, and a
retry repeated the preflight.
- Replace the deadline with a budget that only counts time spent actually
waiting for a gate; a free gate is always taken, and the lock timeout
reports the time waited.
- Lower the budget to 20 s, below the default 30 s internode RPC timeout,
so a forwarded start gets the leader's answer instead of an RPC timeout.
- Name each bounded wait (rebalance metadata refresh or decommission
activation, and which gate), and route every start-gate rebalance load
through one load_rebalance_meta_within.
- Move the admin pre-check error mapping into a function and test it.
- Add proof tests: a slow preflight followed by a brief gate wait still
starts; the activation wait and the admin pre-check are bounded and
named; the rebalance status read does not wait for an in-flight
rebalance entry.
- Correct the operator doc: preflight bucket heals are outside the budget,
and forwarded starts report a generic internal error.
Refs rustfs/backlog#2873
The data-movement operation gate is a write-preferring RwLock. A scanner
publication lease stores a read guard on the leasing node, and the
coordinator's lease-scoped rename, delete and validation on that node
queued a second read. A movement writer queued in between (a decommission
start, a status refresh, a peer reload) then waited for the lease while
the coordinator's second read waited for the writer, so both stalled
until the 60 s lease TTL. Admin status reads also took the movement
writer, so every status call waited for the slowest in-flight copy or
cleanup and stalled new movement reads while it queued.
- Lease-scoped target admission shares the read admission the lease
already holds instead of queueing a new read.
- Pool and rebalance status reads publish under the gates only when they
are free; otherwise they merge the persisted metadata into a private
copy and publish nothing.
- A decommission start bounds its start, activation and movement gate
waits with one 30 s budget and fails with a retryable lock timeout
before persisting anything; the admin handler maps it to 503 with
Retry-After.
Refs rustfs/backlog#2873
* test(heal): add failing regressions for a source-independent absent verdict
Add the regression tests for backlog#2900 that fail on the current code,
where the same object-level not-found is treated as deleted on the plain
heal path but as recreate_failed on the recreate path, and where the
erasure-set healer, the task predicates and the error helpers disagree on
wrapped not-found and transient inputs.
Also add the per-object disposition cache module with its unit tests, the
manager admission tests for it, the compat register entry for the display
boundary, and the admission-invariant notes. The module and the manager
tests are not declared yet, so nothing is wired into the build.
* fix(heal): decide object absence from storage evidence, not request source
Replace the five heal error classifiers (erasure-set healer outcome
mapping, task-level not-found and lock predicates, error.rs recovery
helpers and the storage existence-check matcher) with one typed
classify_heal_error. An object-level not-found from the locked storage
read is Absent for every source and every recreate_missing value; the
recreate path, the usage-observation and scanner object-dir whitelists
are gone, and display-text matching survives only in the marked
classify_error_display boundary.
Add a bounded, expiring per-object disposition memory consulted before
queue admission so a best-effort producer cannot requeue an absent or
terminally failed object on every tick; durable MRF, admin, forced,
high-priority and dry-run requests bypass it. The read-repair producer no
longer forces recreate_missing.
* fix(heal): keep rendered grace windows out of task retries
A grace window rendered into a task message belongs to an attempt that
already finished; its owner schedules the next one. Only typed grace
errors replay the task, as before the classifier merge. Update the
ecstore read-repair black-box test: the producer no longer forces
recreate_missing.
* fix(heal): classify ambiguous legacy delete residue as permanent explicitly
* fix(heal): scope disposition memory and bound absence parsing
A restarted node published FullReady from its own view while survivors still counted its drives unavailable (stale DNS for the old pod IP, then the Offline->Returning->Online hysteresis). A rolling update could then stop the next pod and cost a set its read quorum (#8669).
Add a read-only PeerDriveAdmission internode RPC that reports how a peer sees another host's drives, and gate the startup FullReady transition until a strict majority of each erasure set's peers count this node's drives online. The gate shares the startup budget, releases after 15s of unreachable peers, skips immediately when older peers cannot answer, and can be disabled with RUSTFS_STARTUP_PEER_READMISSION_ENABLE=false.
Clippy ran only after merge, so lint regressions merged freely and then
failed every main run until fixed (10 of 12 main runs on 2026-10-17/18).
Run the workspace Clippy step on every event, warm its check-mode
artifacts in cache-warm with the identical invocation so pull requests
re-check only what they touch, and pin the unconditional step in the
ci_gate contract and its self-test.
Hard quota admission is owned by the durable reservation ledger (decision
D3, rustfs/backlog#2872), but the request-path quota check still read
scanner usage on every write and mapped `UsageUnavailable` to a retryable
503 even for buckets the ledger admits. For a durable quota the checker now
treats scanner usage as reporting input: an unavailable lookup yields
`current_usage = None` and `allowed = true`, counted and logged at debug
level, and admin quota stats report the usage as null instead of zero.
Buckets without a reservation protocol keep the fail-closed snapshot path.
Quotas saved before the reservation protocols are re-saved under protocol
v2 with the same limit, type and created_at. The write goes through a new
guarded save that keeps the admin save's lifecycle/transaction guards,
incarnation check and fleet capability proof and additionally refuses when
the quota revision moved since the cached read, so a concurrent quota edit
wins. It is triggered lazily from the quota read path (background, one
attempt per bucket with a retry interval, never blocking the read) and in
bulk by `POST /rustfs/admin/v3/quota-migration`, which reports every bucket.
The ADR's Consequences section inventories the remaining scanner-usage
"authority" consumers with their meaning after D3, retirement condition and
compatibility window; the operations guide documents the migration and the
report-only usage semantics.
Refs rustfs/backlog#2904, rustfs/backlog#2872, rustfs/rustfs#8678,
rustfs/rustfs#5716, rustfs/rustfs#7350
Co-authored-by: Hauser <housemecn@gmail.com>
* fix(scanner): stop treating an ordinary PUT as data movement in cycle termination
A continuously written deployment never reached a `Complete` scanner cycle
(#8494, #8305, #7108). Three places in the termination path vetoed the cycle
for an ordinary PUT; each now lets the write shade its bucket, the path
per-bucket certification (#8542) built, and keeps the data-movement fences.
1. Usage publication is fenced by data movement only.
`scanner_data_usage_publication_blocked` no longer includes
`namespace_commits_pending()`, so the post-walk probe, the local defer
decision, the post-lease recheck and the usage-store route probes stop
superseding or deferring a cycle for a pending rename fanout. The scanner
publication leases (acquire, validate, target guard) drop the
pending-commit rejection and the namespace-generation staleness check and
keep the movement generation, the operation gate, generation exhaustion
and the purpose/target restriction. A test-util probe lets fixtures wait
for tails directly.
2. The plan identity is split into a coverage identity and a reuse identity.
`scanner_activity_coverage_digest` (peer set, protocol, topology, movement,
and a remote peer's generations) is what a bucket checkpoint is started
and resumed under, through `ScannerBucketScanPlan::checkpoint_coverage_digest`,
`DataUsageCacheReuseOptions::checkpoint_plan_digest` and
`prepare_bucket_checkpoint_with_coverage`. The reuse identity
(`scanner_activity_structural_digest`) keeps `instance_id` and
`maintenance_generation`, so a restart or a maintenance change still
invalidates the baseline and the set caches; a completed sweep is stored
under it. `publication_blocked` leaves every digest and stays a guardrail
veto. A remote worker still resumes under the single digest on the wire.
3. A bucket walk that stops early no longer makes the cycle `Incomplete`
when the bucket has a certified value. The set scan stands in with the
bucket's flattened value from its previous complete set cache
(`carry_partial_bucket_values`), the cycle reports it pending
(`ScannerBucketScanStatus::PartialCarried`), keeps it in the dirty map,
and the next scoped cycle walks it because the baseline's pending list
joins the dirty buckets. A bucket with no certified value keeps the cycle
`Incomplete`; failed buckets are unchanged.
No persisted file, generation, receipt or environment variable is added.
Refs #8494, #8305, #7108, rustfs/backlog#2903.
* chore(ecstore): gate the namespace commit probe to test builds
The scanner publication barrier no longer consults
InstanceContext::namespace_commits_pending, so it has no production
caller left; keep it for tests and the test-util facade only.
* test(rustfs): observe namespace commit ownership directly in node_service tests
The scanner usage publication barrier is movement-only now, so it no
longer reports a pending rename. The node_service tests that assert a
cancelled handler keeps its physical owner, or that wait for rename
tails to drain, read the namespace commit state through a test-only
ECStore probe instead.
* fix(scanner): preserve checkpoints on identity read failures
Distinguish confirmed missing incarnation sidecars from lookup errors before
cache preparation. Preserve durable checkpoint progress when identity reads
are unavailable and report post-scan uncertainty separately from replacement.
Reuse the transaction-fenced reader for optional and required lookups while
retaining existing missing, malformed, system-bucket and lock-loss behavior.
Verify preservation and recovery through a real coordinator with the identity
sidecar write-locked, including unchanged cache revisions and dirty work.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(ecstore): share required incarnation conversion
Keep the missing-incarnation error contract in one conversion used by both
required lookup entry points. This removes a duplicate formatted error site
and satisfies the existing error-format ratchet without changing its baseline.
Pin the full missing-identity error text in the real SetDisks regression.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): bound cross-disk MRF checkpoint I/O
Run independent journal disks with bounded concurrency while retaining each
replica's payload, companion and manifest ordering and the mirror phases.
Share encoded bodies, deduplicate local roots and collect every result
without changing receipt deadlines or persistence failure semantics.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): pin Normal mode when no bitrot start is recorded
Regression for rustfs/backlog#2905: three consecutive cycles that find no
recorded bitrot start currently run Deep through the selection window, and a
Normal cycle never records the cadence start.
* fix(scanner): anchor the bitrot cadence and budget per-disk I/O
Decide the scan mode from a recorded Deep window start only. A cycle that
finds no recorded start runs Normal and anchors the bitrot cadence in
process memory instead of opening the selection window; the Deep cycle that
follows the elapsed cadence records the window start exactly as before. A
missing record used to be read as a window that started at cycle 0, which
ran Deep for deep_window_cycles cycles whenever the start could not be read
back (rustfs/rustfs#8015). The recorded start is also remembered across a
skipped or failed save, and /v3/scanner/status reports the cadence state
under a new bitrot object.
Add a per-local-disk background I/O budget (operations and bytes per second,
defaults by detected media, auto|off|<rate>) that DynamicSleeper charges
before directory listings and object metadata reads as a hard ceiling
layered on the foreground-priority disk admission gate, with runtime config
keys, per-disk status, cycle metrics, pacing pressure, usage_freshness,
Prometheus series, Helm values, admin config help, and operator docs.
Refs rustfs/backlog#2905, rustfs/rustfs#8015, rustfs/rustfs#8377.
* test(scanner): expect Normal retries on erasure sets without a bitrot start
The pending-PUT retry fixture asserted that a 4-drive set revisits every
bucket because the cycle ran Deep. Without a recorded Deep window the cycle
now runs Normal on every layout and reuses the cold inventory.
Count only post-format rebuild failures against the replacement retry
budget; handoffs, format fences, outages, grace waits and conflicts keep
their durable reason without spending an attempt. Retired generations
(committed handoff or approved legacy migration) no longer own their set,
so fresh admission for other slots proceeds, and a committed predecessor
becomes collectible once its successor is verified.
Block discovery conflicts per generation instead of per set, re-evaluate
them on every pass, and expose the blocked entries with reasons. Report
root heal inventory conflicts per record and keep replaying the others.
Revalidate blocked checkpoints hourly with the current reader and clear
markers the reader accepts. Return a dedicated error for ambiguous legacy
delete residue instead of the retryable SlowDown.
Implements rustfs/backlog#2902. Fixesrustfs/rustfs#8664. Addresses
rustfs/rustfs#8366, rustfs/rustfs#8565, rustfs/rustfs#8645 and part of
rustfs/rustfs#8608.
Co-authored-by: Hauser <housemecn@gmail.com>
* fix(scanner): give startup gates automatic exits instead of a pause
Every fail-closed scanner gate now has a bounded automatic exit without
deleting quarantine evidence or relaxing the leader, CAS, movement or
usage-floor fences (rustfs/backlog#2901).
- Recovery retries back off instead of pausing: after the short budget the
observation reads `backoff`, stays retryable, names the next attempt and is
capped at ten minutes; one success resets the count and a 24 h window
restarts the budget (rustfs/rustfs#8297, #8670). The usage-floor failure no
longer carries the retry count across attempts.
- A corrupt `.bloomcycle.bin` is rebuilt on the iteration after its quarantine
marker is durable: the fenced leadership claim CAS-replaces the quarantined
revision under a higher epoch and the marker stays as evidence.
`future_schema` and `cleanup-pending` remain blocked.
- An unclaimable pause ledger runs the leader degraded: cycles run untracked
behind the data-movement fence, `persistence_state` reports `degraded`, and
the claim is retried at most every five minutes (rustfs/rustfs#8251).
- The usage-state full-rebuild reset starts from the empty baseline when every
usage slot is corrupt and no cycle state exists (rustfs/rustfs#8670, #6951).
- A cycle-state save reconciles non-conflict failures against durable state
and retries under the same epoch; an exhausted save reloads durable state in
place. Only a newer owner or an unreadable primary ends the iteration, so
peers see a changed fence only when ownership changed (rustfs/rustfs#8141).
* test(scanner): align usage-state reset tests with the empty-baseline rebuild
The full-rebuild reset no longer refuses corrupt or identity-less usage slots
when no cycle state exists (rustfs/rustfs#8670). The two tests that encoded
that refusal now check the new contract: the reset starts from the empty
baseline with a bootstrap-pending marker, never an authoritative zero floor,
and a valid backup still supplies the recovery floor.
---------
Co-authored-by: cxymds <cxymds@gmail.com>
#8682 replaced the cold `cargo check -p rustfs --release` with a
`cargo rustc ... -C debug-assertions=off` on the rustfs lib alone. That
leaves the dependencies on the dev-profile cfg, so the lib also fails on
items gated on one side of cfg(debug_assertions) only (E0599 on
Filenames::Embedded), and the step's "fails solely at the compile_error"
assertion can never hold; the lane has been red on main and every pull
request since. Restore the whole-graph release check, drop the two-job
limit that only a linking step needed, and record why the single-crate
variant is unsound.
Raise CARGO_BUILD_JOBS to 3 on every event in the workspace test lane and
record the closed backlog#1601 experiment: the test-profile compile took
28m21s at 2 on a pull request and 16-23 min at 3 across three successful
main runs.
Merge the two Connect boundary jobs into connect-feature-boundaries. The
offline enrollment gate no longer exports a fresh CARGO_TARGET_DIR, so it
rebuilds only the rustfs crate on top of the restored ci-dev cache; the E2E
root is isolated by the feature gate and the dedicated binary name, not by
the target directory. The short-credential release guard is now
`cargo rustc -p rustfs --lib --features connect-e2e-short-credentials --
-C debug-assertions=off`, which trips the same cfg(not(debug_assertions))
compile_error against cached dev-profile dependencies instead of a cold
release check of the whole graph. Every assertion of both lanes is kept.
Update ci_gate.py and the ci-gates.md PR matrix for the merged job.
* ci: move compile-free and prebuilt-binary lanes to hosted runners
Standard GitHub-hosted runners are free for this public repository, while
every sm-standard minute is paid self-hosted capacity. Move the jobs that
never compile (quick checks, typos, scope selection, the required-checks
aggregate, audit, closed-PR cancellation) and the two s3-tests lanes that
only run the prebuilt debug binary to ubuntu-latest, drop the wget install
the custom image needed for typos, and move the upgrade-compatibility
workflow, whose build shape already runs on hosted runners in
e2e-distributed, with renamed cache keys so the hosted and self-hosted
images never share a rust-cache entry. Gate the package summary job on
resolve having run, so development builds stop starting a pod to report a
skipped job.
* docs(ci): document runner billing and placement rules
Record in .github/workflows/README.md which runner labels GitHub bills
(none for this public repository), which ones RustFS pays for, the rule for
when a job may leave ubuntu-latest, the rust-cache single-writer and
same-image constraints, the patterns that have leaked self-hosted minutes,
and a checklist for new or moved jobs. Point to it from .github/AGENTS.md,
docs/testing/ci-gates.md and the ci.yml placement comment so the rule has a
single home.
Supersede the 2026-09-03 decision: hard quota admission is owned by the
durable reservation ledger, and scanner data usage is best-effort reporting
input. Record that a pending ordinary PUT commit is a write-barrier event
rather than a usage-publication veto, and split the scanner plan identity
into a coverage identity and a reuse identity.
Refs rustfs/backlog#2872, rustfs/backlog#2903, rustfs/backlog#2904.
* feat(oidc): make the Console session duration configurable
The OIDC callback hard-coded the Console credential lifetime at 3600
seconds, so a Console session always ended after about an hour.
RUSTFS_CONSOLE_AUTH_TIMEOUT, the documented knob next to this behaviour,
is the Console HTTP request timeout and cannot extend a session.
Resolve the lifetime from the process-level
RUSTFS_CONSOLE_OIDC_SESSION_DURATION instead and forward it to
complete_authorization_code(). Unset, unparseable, zero, and negative
values keep DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS (3600), so
existing deployments keep the previous behaviour and a misconfigured
variable can never issue an already expired credential. Values above
CONSOLE_OIDC_SESSION_DURATION_MAX_SECS (7 days) are clamped to that
maximum with a warning, which also caps how far an operator can stretch a
credential by accident.
Keep the Console logout token for that same lifetime. OidcStateStore
pinned the logout cache to a hard-coded hour, so a session configured to
last longer than that dropped its logout token while the credential was
still valid: RP-initiated logout could no longer resolve, and the browser
was sent back to the Console login page with the IdP session still
active. The logout cache now derives its TTL from the same
console_session_duration_secs(), which applies the same clamp, and the
callback reads its lifetime from that one function instead of a local
copy, so the credential lifetime and the logout-token retention cannot
drift apart. Add tests covering the default hour, an extended session,
the maximum, and out-of-range values.
The variable is deliberately not part of ENV_IDENTITY_OPENID_KEYS: that
list is suffix-scanned per provider, and this setting applies to every
provider. It has no identity_openid KVS equivalent and needs a restart,
like the other process-level settings in the same document.
Document the setting in the login flow, the process-level table, and the
troubleshooting table, and clarify that RUSTFS_CONSOLE_AUTH_TIMEOUT is a
request timeout rather than a session lifetime.
* test(ecstore): retain offline marker fixtures across reconnect
Keep simulated drive handles attached and reject I/O until explicit rejoin.
Attempt reconnect in the current-marker case so the former CI race is
deterministic while retaining physical cleanup and historical-data checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(oidc): align session diagnostics and lifetime regression tests
Use structured IAM diagnostics for clamping and move configuration tests
to their owning crate. Cover duration boundaries, logout retention and
signed credential expiration through the federation binding.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): expose mixed sweep checkpoint transitions
Correlate checkpoint admission, incomplete sweeps, and successful terminal
Partial persistence by bucket/source and cycle/leader epoch. Share the
traversal-position predicate and local/remote save diagnostic without
changing certification, acknowledgement, or persistence contracts.
Extend real-disk cache regressions across cycles and leader changes and
verify that failed terminal saves retain an older periodic frontier until
a successful terminal save permits fresh certification.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): report the aggregate incomplete sweep gate
Expose the existing unresolved_objects decision in the incomplete-sweep
event. Nested cache-entry failures can keep a sweep partial while both
the root counter and the capped diagnostic failure map are empty.
Extend the real-disk regression with diagnostics disabled and distinguish
this case from mixed-only coverage without adding another cache walk.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(table-catalog): qualify current backing contracts
* fix(table-catalog): verify smoke cleanup after object rename
* fix(ci): include the baseline writer test in full E2E selection
* fix(ci): reserve test capacity for disk-backed lock probes
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(scanner): attribute incomplete census and lease waits
Report bucket sweep blockers and correlate partial remote work and rejected
roots with their set and cycle. Keep distinct accepted bucket names in the
result collector so completion counts and missing-bucket attribution share
one source of truth, including during cancellation drain.
Distinguish worker completion from coverage, retain peer/purpose context in
lease failures without changing retry suffixes, and avoid claiming that an
unverified leader fence changed or a waiting scan was complete.
Document the census stages and preserve publication, dirty acknowledgement,
remote failure classification and checkpoint safety rules.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(scanner): move set caches through the result collector
Give the sole result collector exclusive ownership of the aggregate cache
and return it together with the accepted bucket names after channel drain.
Move that cache into complete persistence or incomplete updates instead of
sharing a mutex and cloning the terminal cache.
Return each accepted result's owned name from cache application so coverage
tracking also avoids an extra name allocation. Preserve worker joins,
cancellation drain, publication metadata and all certification fences.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): treat replication status writebacks as status-class hits
A replication status writeback (persist_replication_state_if_current in
Update mode, including the heals the scanner queues during its own walk)
marked the bucket dirty on the namespace path: it advanced the dirty-usage
generation, woke the scanner, reached the durable dirty journal and
entered the activity snapshot. Any replicated bucket therefore made every
scanner cycle Superseded, the authoritative usage baseline was never
certified and usage freshness stayed stale for as long as replication ran
(#8375).
The writeback is now status class: record_status_class_mutation records
only the process-local producer coverage bit and nothing else. There is
no rescan queue for it, because a deployment with an active replication
rule already walks every bucket on every cycle
(ScannerMaintenanceFeatures::needs_regular_cycle) and that walk refreshes
replication_stats. The routing lives in a named, unit-tested function in
rustfs/src/storage/storage_api.rs; tier-expiration cleanup removes data
and stays on the namespace path.
No persisted format, protocol, admin API or configuration changes.
This is phase 1 of docs/architecture/scanner-per-bucket-convergence.md,
which this change adds together with its index entry.
Refs #8375, #8494, rustfs/backlog#2816.
* fix(scanner): refresh replication statistics during maintenance
Refresh compacted prefixes and completed bucket caches during replication maintenance. Bind set execution reuse to the observed replication statistics so a same-cycle retry cannot restore older counters.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): avoid expanding the legacy S3 DTO surface
Decode the existing replication configuration type from a JSON fixture instead of adding direct S3 DTO references to a new file. Preserve the compacted-prefix and same-cycle assertions and the footprint ratchet.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): attribute activity guardrail vetoes per field
The post-walk cluster activity proof compared the whole activity snapshot,
so any namespace or dirty-usage generation movement vetoed publication.
Veto only on the fields that change which heap the walk described
(peer set, instance id, protocol version, topology, maintenance
generation, data movement, movement generation, publication blocked) and
name the field and host in the log and in a new usage-freshness counter.
Generation movement is a shade signal consumed per bucket. A remote peer
whose dirty-usage state moved keeps the whole-cycle outcome until scoped
acknowledgements can name its buckets.
* feat(data-usage): carry pending-bucket certification in DataUsageInfo
Add the additive fields a per-bucket certified scanner snapshot needs:
the sorted list and count of buckets whose dirty-usage generation moved
during the walk, and an explicit observational marker for snapshots whose
authoritative publication was vetoed. All three default when absent and
are skipped when empty, so older readers see the existing snapshot shape.
* feat(scanner): certify dirty usage per bucket and complete cycles with pending buckets
A structurally complete walk that the guardrail did not veto is now
`Complete` even when the dirty-usage generation moved during the walk.
Certification keeps the compare-and-remove semantics of
clear_dirty_usage_buckets: buckets whose generation still equals the start
snapshot are cleared, buckets written during the walk stay in the dirty map
and are published as the pending set, so usage_snapshot_converged becomes
`pending == 0`. The authoritative object is written for such cycles; the
observed route is selected by the explicit observational mark that vetoed
cycles carry instead of by the convergence flag, and the root publication
proof is read for every authoritative save. The post-lease activity recheck
compares the guardrail fields, with the peer dirty-usage fallback, instead
of the full snapshot digest.
* feat(scanner): accept unconverged complete publications as the cache baseline
complete_scanner_cache_baseline_plan_digest required the authoritative
snapshot to be converged, so a site whose cycles always left a bucket
pending could never scope a rescan and every cycle was a full walk. The
authoritative object is now a structural baseline whenever it carries a
convergence marker and no observational mark; bucket-cache reuse is then
governed by certification, because pending buckets stay in the dirty map
and are selected for the rescan while certified buckets reuse their set
cache. Observed candidates keep requiring the unconverged marker, and
segment reuse evidence keeps requiring a converged baseline.
Add the single-disk cycle test: a bucket written during the walk is the
only pending bucket, the others are certified, the publication is
authoritative and unconverged, and the next cycle is scoped to that
bucket; a held PUT tail still supersedes onto the observed route.
* feat(admin): expose scanner guardrail vetoes in usage freshness
Carry the attributed guardrail veto counter, timestamp and <field>@<host>
reason from the scanner metrics report into the madmin scanner status so
an operator can tell a rebalance or a PUT tail from a broken probe. The
fields are additive with serde defaults and merge like the deferral
fields.
* docs(scanner): record per-bucket certification in the design and changelog
Amend the convergence design where the implementation deviates: pending
buckets carry their walked value, maintenance_generation stays a veto,
guardrail localisation and the remaining phase-2 items ship separately,
last_cycle_result keeps its success label, and the admin surface gains the
guardrail veto fields. Update the publication contract for the post-lease
guardrail recheck and the explicit observational routing mark.
* refactor(scanner): drop the unreachable failed-bucket input from the pending set
A failed bucket never reaches a Complete cycle, so the pending set only
needs the start snapshot. Also remove a redundant clone in the data-usage
round-trip test.
* test(scanner): name the activity field mutation table type
* test(scanner): fill the post-walk gate in merged replication fixtures
The replication maintenance fixture merged from main builds a
ScannerCycleRequest literal and predates the test-only post_walk_gate
field; give it the default so the test target compiles on the rebased
branch.
* ci(nextest): run the admin target-repair fixtures on the ecstore base stack
repair_transaction_rejects_a_bucket_recreated_after_target_validation
overflowed the 2 MiB libtest default twice in a row on Linux CI for this
branch. Measured by sweeping RUST_MIN_STACK with the ci profile on macOS
arm64: main (65f30bd5c) passes at 2032 KiB and overflows at 2016 KiB,
this branch passes at 2040 KiB and overflows at 2032 KiB; on both trees
the overflow is in the test's delete_bucket step on the real store, and
the futures on that path are small (the peak is nested poll depth, not
one large future). The module was given this stack in #7272 and removed
in #7280 to check that boxing (#7277) sufficed; with the default now
exhausted on main itself, give the fixtures the same 4 MiB base stack as
the other real-storage rustfs fixtures, in both profiles.
* test(scanner): fill the post-walk gate in the walkers-per-disk fixture
The merge from main brought the walkers_per_disk cycle fixture (#8539),
whose ScannerCycleRequest initializer predates the test-only
post_walk_gate field this branch adds; the scanner test target did not
compile. Fill the default so the merged head builds.
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Separate completed traversal from persisted object health debt so leader
handoffs reuse healthy subtrees and retry failures without losing progress.
Fence old readers with a new receipt digest domain and preserve exact
accounting under diagnostic eviction, compaction and punctuation ordering.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
On Windows, URL endpoints such as http://host:9000/C:/data{1...4} (the only
form multi-node and pool-expansion layouts accept) produced the disk path
`/C:/data1`: `try_from` stripped the synthetic slash, but `Url::set_path`
put it back, and `get_file_path` only removed it for `file` endpoints.
Windows resolves `/C:/data1` as an invalid `C:` component under the current
drive, so disk validation and `resolve_local_disk_root` failed.
- `get_file_path` now restores `/X:/...` to `X:/...` for every scheme on
Windows; URL paths without a drive (`/export1`, `/C:foo`) are unchanged.
- UNC shares (`\\nas\share\d1`, `//nas/share/d1`, `\\?\UNC\...`) are rejected
with an explicit error instead of becoming a file URL whose host is the
share server and whose path drops the share name.
- A drive root (`http://host:9000/D:/`) is accepted in URL form, matching
the path-style `D:\` endpoint that was already accepted.
- The "distinct physical disks" error explains on Windows that directories
on one volume share a device identity, alongside the existing
RUSTFS_UNSAFE_BYPASS_DISK_CHECK hint.
- The Windows filesystem lane now runs on layout changes and runs the
layout::endpoint:: tests; new Windows tests carry the windows_ prefix.
- Document Windows endpoint forms, the same-volume check, and the SNSD
expansion limit in cluster-lifecycle-operations.md.
`run_process` called `block_on(async_main())` on the native main thread,
so the whole startup chain ran on whatever stack the platform hands the
main thread: 8 MiB from `ulimit -s` on Linux and macOS, but the 1 MiB PE
default on Windows. A Windows debug build overflowed before it listened.
`RUST_MIN_STACK` does not reach the native main thread.
Run the runtime on a `rustfs-main` thread with an explicit 8 MiB stack,
keeping the budget Linux and macOS already had and making it independent
of the linker, toolchain (msvc/gnu) and `ulimit`. Panics are re-raised
on the main thread so the exit status is unchanged.
Measured on macOS arm64 by lowering the main-thread stack with
`ulimit -s` before this change: debug startup overflows at 1792 KiB and
reaches /health/ready at 2048 KiB; release overflows at 512 KiB and is
ready at 768 KiB. After it, debug starts with a 256 KiB main thread.
Add a Windows `Startup Smoke` job that builds the debug server, runs
`--version`, and waits for /health/ready on a single volume.
Since 1.0.2 an encrypted object's ETag is a keyed fingerprint, not the
plaintext MD5. A target that computes its own ETag (AWS S3, Wasabi) never
reports it, so every "find the existing replica by exact ETag" step missed
encrypted objects on targets that mint their own version ids:
- a heal / MRF re-drive after a lost PUT response wrote a second version;
- an existing-object resync re-PUT a replica its ledger already named
(the ledger-located HEAD was confirmed by ETag only), and on the AWS
400 shape the ledger was not consulted at all;
- resync verification counted such replicas as failed;
- a ledger-less version purge reported the replica as already gone.
Encrypted sources now recognise their replica by shape: the byte count the
replication PUT sent (plaintext for managed SSE, ciphertext for SSE-C) and
a target write time no earlier than the source version. An ETag match
still wins (RustFS/MinIO targets). The ledger is consulted first on both
the 404 and the AWS 400 shape. Without a ledger, only a unique shape match
among the unclaimed ListObjectVersions candidates is adopted and recorded;
several shape matches mean re-PUT for convergence and a refused purge. The
AWS current-version fallback stays ETag-only, and a listing the
target refuses on the 400/403 shape keeps that fallback (re-PUT) rather
than stalling. An unversioned target's `null` version is never matched by
shape: it holds whichever generation was written last. Plaintext sources keep exact
ETag identity.
Builds on the ETag-computing fake target mode and the ComputesOwnEtags
matrix row from rustfs#8583. Adds an SSE-S3 variant of the mint-own-ids
re-drive test (red before this change: the heal left two target versions),
and both variants now also assert the purge.
* fix(scanner): treat replication status writebacks as status-class hits
A replication status writeback (persist_replication_state_if_current in
Update mode, including the heals the scanner queues during its own walk)
marked the bucket dirty on the namespace path: it advanced the dirty-usage
generation, woke the scanner, reached the durable dirty journal and
entered the activity snapshot. Any replicated bucket therefore made every
scanner cycle Superseded, the authoritative usage baseline was never
certified and usage freshness stayed stale for as long as replication ran
(#8375).
The writeback is now status class: record_status_class_mutation records
only the process-local producer coverage bit and nothing else. There is
no rescan queue for it, because a deployment with an active replication
rule already walks every bucket on every cycle
(ScannerMaintenanceFeatures::needs_regular_cycle) and that walk refreshes
replication_stats. The routing lives in a named, unit-tested function in
rustfs/src/storage/storage_api.rs; tier-expiration cleanup removes data
and stays on the namespace path.
No persisted format, protocol, admin API or configuration changes.
This is phase 1 of docs/architecture/scanner-per-bucket-convergence.md,
which this change adds together with its index entry.
Refs #8375, #8494, rustfs/backlog#2816.
* fix(scanner): refresh replication statistics during maintenance
Refresh compacted prefixes and completed bucket caches during replication maintenance. Bind set execution reuse to the observed replication statistics so a same-cycle retry cannot restore older counters.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): avoid expanding the legacy S3 DTO surface
Decode the existing replication configuration type from a JSON fixture instead of adding direct S3 DTO references to a new file. Preserve the compacted-prefix and same-cycle assertions and the footprint ratchet.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(scanner): keep raw enumeration bookkeeping linear per entry
The page writer inserted into its building page with a push-and-sort on
every entry and committed each page through a clone. Insert in sorted
position and move the committed page into the index instead.
Add a regression test that records 20,000 entries in one prefix and pins
the cost to one hash per entry and well under a second, so the quadratic
re-normalization removed by #8114 (rustfs#8113) cannot return once pacing
builds on this walker.
* feat(scanner): decouple bucket walkers from disk count
Add the scanner key walkers_per_disk (RUSTFS_SCANNER_WALKERS_PER_DISK,
1 through 16, default 1). Each local coordinator disk of a set spawns that
many bucket walkers; remote disks keep one. The walker count is what
max_concurrent_disk_scans caps and the disk count stays the floor.
The walkers of one disk share its scanner admission through
LocalDiskAdmissionShare: the first walker to start a bucket takes the
per-disk permit and the last to finish releases it, so a peer's remote
request still sees one scan session per physical disk. Without the share
the second walker was rejected as busy and retired.
The disk_concurrency_budget log line reports walkers_per_disk, walkers and
the reason for the effective limit. The default stays 1 because no local
disk reports rotational media today; the design keeps a higher flash
default as the intent.
* docs(scanner): document walkers_per_disk in the Helm chart and configuration manuals
RUSTFS_SCANNER_WALKERS_PER_DISK was only described in the scanner runtime
controls table. Add it to the Helm chart (values, configmap rendering,
README table, template test), the configuration crate's environment
reference with operator guidance, and a short section in the operations
manual on when to raise it and how to confirm it took effect.
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Hauser