* fix(ecstore): leave no source free version for a moved tier version
Data-movement source cleanup deletes each source version separately and
never set skip_tier_free_version, so every transitioned version that
decommission or rebalance had already copied to a target pool left a new
tier free version on the source. Decommission then failed its post-check
for any pool with tiered objects; rebalance left the record behind. The
record claims the remote object the target copy still reads and stays
blocked behind that live reference.
Decommission and rebalance now pass the identities of transitioned
versions whose copy committed on a target, and the cleanup skips the
free version for exactly those. A transitioned version that did not
move, such as one lifecycle expired during the move, still gets its free
version because that record is its only remote-delete owner.
Refs rustfs/backlog#2873, rustfs/backlog#2872
* fix(ecstore): keep the free version of a superseded moved source
Data movement treats a source null version as complete when a newer
unversioned object already sits on the target, without copying it.
That outcome returned the same Ok as a committed copy, so a restored
transitioned null version superseded this way was recorded as
transferred and lost its source free version, leaving its remote object
with no owner.
The copy paths now return DataMovementCopy::Committed or ::Superseded,
and decommission and rebalance record only committed copies of
transitioned versions. A superseded source keeps its free version.
Tests cover the superseded restored null version for rebalance and
decommission, a restored and a null transitioned version moving by
rebalance, an entry with one lifecycle-expired and one moved version,
the per-version skip with two versions of one key, and a directory
object. The tiered stale-target routine test from #8724 now expects the
decommission to complete. The ILM ownership contract and the
decommission notes describe the moved, superseded, expired and legacy
cases.
Refs rustfs/backlog#2873, rustfs/backlog#2872
* test(ecstore): build the decommission ILM fixtures without s3s
The superseded and expiry-mix decommission tests named s3s request and
lifecycle types directly, which added pools_test.rs to the ecstore
files that reference s3s and failed the footprint ratchet. Set the
restore request through ObjectOptions, and store the lifecycle rule as
bucket metadata XML and load it with get_expiry_configs, as the
decommission routine does.
* test(ecstore): pin restored decommission copies and multipart supersession
Decommission recorded a committed data copy of a restored transitioned
version without any test failing if the record were dropped; add a
routine-level decommission of a restored version that checks the post-
check passes, the source keeps nothing, and the target keeps the full
remote tuple. Add the multipart variant of the superseded restored null
version (rebalance), and a unit test that separates committed,
superseded and mismatched targets in the overwrite resume adjudication.
* fix(obs): skip address discovery for initialized node identity
Check the nonblocking node name before resolving the local IP, retaining the final name recheck after fallback. Add portable publication regressions and Linux worker-local netlink fault coverage.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(obs): keep identity contention checks in common
Exercise the real held-writer getter inside its owning module. Publish through the existing runtime helper from obs tests, retaining the kernel netlink failure and recovery coverage without bypassing architecture guards.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(obs): scope unsafe allowance to seccomp test helper
Document the bounded prctl FFI safety contract and allow unsafe code only
inside the Linux fault-filter installer so workspace unsafe-code denial
continues to apply everywhere else.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(decommission): run the leader-side start to completion
A forwarded decommission start awaits `ECStore::decommission` inline in
the RPC handler. When the forwarding node's RPC times out, the stream is
reset and the handler future is dropped wherever it is. Dropped after
`pool.bin` records the pool as active but before the peers reload and
the worker is spawned, the start left the pool active with no worker and
peers on the old metadata, and a retry was rejected as already active.
An admin client that disconnects can do the same.
Run the start in its own task and let the caller await only its result.
The start then always reaches its end: it fails before persisting
anything, or persists the pool, has the peers reload and spawns the
local workers, rolling back if the reload fails. A retry still cannot
start twice: it waits on the start gate held by the running start and
either reports the busy gate or is rejected because the pool is active.
`decommission` now takes `self: &Arc<Self>` and spawns workers on that
store instead of looking up the global object store, which is the same
store in production and makes the path testable; the unused
`require_decommission_store` helper is removed.
Refs rustfs/backlog#2873
* fix(decommission): detach the leader-side start from its caller
9ac4c10de carried only the reproduction (the testable entry point, the
peer-reload pause point and the proof test). This adds the fix it
describes: `ECStore::decommission` spawns the start as its own task and
awaits only its result, so a caller that stops waiting, such as a
forwarded start whose internode RPC timed out, can no longer drop the
start between persisting the pool as active and having the peers reload
and the local worker start.
Also prove a retry while the first start runs cannot start the pool
twice, update the operator and architecture docs for the new behavior,
and shrink the error(format!) ratchet baseline for the removed helper.
Refs rustfs/backlog#2873
* fix(decommission): single-flight start, recheck pool state under gate
Review of 3a5276c5b:
- A start re-checks the pool state under the start gate before it
cancels routines, so a start racing another one can no longer cancel
the worker the other start just spawned.
- Only one start runs per instance at a time: another request, including
the retry of an operator whose request timed out, is rejected as
already running instead of stacking a second detached start and its
preflight heals. The start runs through `run_detached_mutation`, and a
failure nobody waits for any more is logged.
- Tests: the proof test pins its drop after the save and requires a
worker that persists a cancellation; the retry test really drops the
first caller; new tests cover the under-gate recheck and a peer reload
failure after the caller is gone. That last one shows the start
rollback cannot clear a fresh start because the pool metadata
stale-write guard refuses it; the gap is asserted and documented and
tracked in rustfs/backlog#2873.
- The start tests run on their own large-stack thread: the io_uring lane
(`uring_`, which also selects `during_`) runs them with libtest's 2 MiB
default stack, where the proof test overflowed.
- Docs: single-flight starts, status and cancel during preflight, the
rollback gap and its recovery, and the release requirement.
Refs rustfs/backlog#2873
* fix(decommission): turn away local start retries at the start slot
Second review of 29253430a:
- The leader-local admin pre-check refreshed rebalance metadata under
the start gate before the start slot was checked, so a local retry
waited out its 20 s gate budget behind a running start (which holds
the gate through its peer reload) and then got a 503. The pre-check
now checks the slot first and returns `DecommissionAlreadyRunning`,
which the handler reports as a 400 that keeps the reason.
- Prove the reload-failure path really ran in the detached start (the
injection is consumed), add a test that a fresh start's failed
rollback reports both the reload error and the stale-write refusal,
and assert an abandoned start stays tracked for shutdown.
Refs rustfs/backlog#2873
* fix(decommission): retry every stale target selection without failing
The tier free-version copy path counted a version as failed as soon as
a capacity failure came back and then handed the error to the entry
retry, so a retry that succeeded left the decommission failed. Return
the failure without counting it, like the other copy paths.
A selected pool can also stop fitting the mutation before admission
without going over a budget: the reservation estimate is exhausted, the
pending intent no longer fits the committed-copy budget, or a temporary
write no longer fits the headroom. Report these as "no longer fits the
selected mutation" and treat them like an over-budget pool under one
StaleTargetSelection class: retry, select again, defer once the retry
budget is spent.
Also refresh the over-budget gauge of every allocation before rejecting
growth, correct the sticky-target doc on stranded markers, and cover
each copy path, the pinned path and both retry branches with tests.
Refs rustfs/backlog#2873
* fix(decommission): name the last deferral cause in the pause reason
A round that ends with deferred entries pauses the routine with a
generic "deferred entries remaining" reason, so a pinned entry that no
longer fits its pool hides its cause in the logs. The pause reason now
appends the last deferral's error.
Keep the committed-copy check on the pending intent fatal: growth and
owner admission reject any mutation that would fail it first, so it
only fires on an inconsistent ledger.
Tests:
- a multipart part that does not fit target 1's temporary-copy
headroom is deferred and pauses with the headroom shape;
- a pinned object whose remaining allocation is below its peak pauses
with the exhausted-estimate cause, and the over-budget pinned test
checks its own cause;
- the stale-target tests assert where the pause lands: the held entry
is on neither target, the anchor is already on target 1, and target
1 holds exact inflight bytes in the multipart cases;
- share the stale-target store, barrier and pinned-copy setup.
* fix(decommission): prefer a stale selection as the pause cause
The pause reason named whichever deferral of the set finished last, so
a transient busy gate could hide another entry's permanent shortfall.
A stale target selection, which waiting does not clear, now outranks
transient contention, and the reason reads "; deferral cause: <error>".
Tests:
- a pinned object that no longer fits and a later object whose gate is
busy, in the same set, pause with the pinned cause; a unit test checks
both orders;
- the tiered and intent-conflict tests check where the pause lands, and
the tiered test asserts the known post-check failure so that fixing
the source cleanup defect flips it.
Second review of 652c56875..0ef02a485:
- The operator doc and the budget comment claimed a forwarded start that
outlives the internode RPC timeout still completes on the leader. The
leader drops it wherever it is, possibly after pool.bin records the
pool as active but before its worker starts or the peers reload.
Document that state, the watchdog that resumes the worker, and the
checks to run; the cancel-safety fix is tracked in rustfs/backlog#2873.
- Prove the budget is shared across waits (paused clock: a 60 ms wait
leaves 40 ms for the next gate).
- Prove the rebalance-refresh start gate and activation gate waits and
the decommission() entry are bounded and named.
Refs rustfs/backlog#2873
Review of 652c56875 found that the 30 s start budget was a wall-clock
deadline fixed when the start began, so slow preflight work (bucket heals,
capacity scans) could spend it before the activation gates were reached.
A gate held for a few milliseconds then failed the start at once, and a
retry repeated the preflight.
- Replace the deadline with a budget that only counts time spent actually
waiting for a gate; a free gate is always taken, and the lock timeout
reports the time waited.
- Lower the budget to 20 s, below the default 30 s internode RPC timeout,
so a forwarded start gets the leader's answer instead of an RPC timeout.
- Name each bounded wait (rebalance metadata refresh or decommission
activation, and which gate), and route every start-gate rebalance load
through one load_rebalance_meta_within.
- Move the admin pre-check error mapping into a function and test it.
- Add proof tests: a slow preflight followed by a brief gate wait still
starts; the activation wait and the admin pre-check are bounded and
named; the rebalance status read does not wait for an in-flight
rebalance entry.
- Correct the operator doc: preflight bucket heals are outside the budget,
and forwarded starts report a generic internal error.
Refs rustfs/backlog#2873
The data-movement operation gate is a write-preferring RwLock. A scanner
publication lease stores a read guard on the leasing node, and the
coordinator's lease-scoped rename, delete and validation on that node
queued a second read. A movement writer queued in between (a decommission
start, a status refresh, a peer reload) then waited for the lease while
the coordinator's second read waited for the writer, so both stalled
until the 60 s lease TTL. Admin status reads also took the movement
writer, so every status call waited for the slowest in-flight copy or
cleanup and stalled new movement reads while it queued.
- Lease-scoped target admission shares the read admission the lease
already holds instead of queueing a new read.
- Pool and rebalance status reads publish under the gates only when they
are free; otherwise they merge the persisted metadata into a private
copy and publish nothing.
- A decommission start bounds its start, activation and movement gate
waits with one 30 s budget and fails with a retryable lock timeout
before persisting anything; the admin handler maps it to 503 with
Retry-After.
Refs rustfs/backlog#2873
The test derived its inputs from Zoned::now() and assumed that now - 60s is still on the same local calendar day, so it failed deterministically in the first minute after local midnight. Build the inputs from fixed local wall-clock times instead, covering same day, previous day and the 23:59:59 -> 00:00:00 crossing.
* fix(decommission): adopt target intents left by a killed leader
A decommission leader killed after persisting a target capacity intent and
before finalizing it left that intent in the exclusive pending slot of the
still-active reservation. The resumed routine derives the same reservation
identity, but every other mutation on that target was rejected as "another
mutation" until the interrupted object happened to be retried on the same
target, so the resumed decommission exhausted its retries and stalled.
At routine start, before any worker runs, the resumed routine now adopts the
reservation's pending intents: under the target fence (or the legacy pool
metadata fence) it moves each one into a mutation-scoped inflight marker that
keeps its identity and bytes. The interrupted mutation's retry settles the
marker through the existing commit, temporary-release and equivalent-target
paths, and an equivalent committed target now settles a mutation's own marker
when no pending intent remains. Adoption is pinned to the persisted
decommission start time and reservation identity, never runs mid-incarnation,
and a failure pauses the routine instead of marking the decommission failed.
* fix(decommission): keep adopted intents within budget and settle them once
Address the adversarial review of the leftover intent adoption.
- Merge an adopted intent into its mutation's staged marker by summing only
while the target's temporary-copy budget allows, never below the larger of
the two. A multipart completion moves its staged parts into the object, so
summing them could exceed the budget and pause every later routine start;
the adopted marker now exceeds the budget only where the completion's own
finalize would have. Reuse resolve and record_inflight for the move.
- Distinguish a temporary release that proved the exact committed target from
one that could not prove the target identity. When the committed mutation no
longer has a pending intent, release only the bytes the cleanup freed and
keep the adopted completion for the equivalent-target reconciliation.
- Let an equivalent target settle a mutation's own marker while another
mutation holds the pending slot, instead of failing on that unrelated intent.
- Cover real leader kills before and after the target commit at routine level
for both lock models, an injected adoption failure that pauses and then
completes, an adopted multipart completion surviving a background abort, and
a live legacy intent. Routine tests reuse the large-stack test helper.
* fix(decommission): pause instead of failing on target slot contention
The S00 leader-kill run with the null-version fix merged ended the resumed
decommission as failed. A re-run of a versioned key reconciled its delete
marker against an already charged target copy while another worker of the
same incarnation held the target's pending slot between a failed write and
its retry. The reconciliation rejected that unrelated intent, the entry
context wrapper flattened the error to a string the intent-conflict retry
could not recognize, and the bucket runner reported a sibling bucket's
OperationCanceled instead of the capacity block, so the routine marked the
decommission failed.
- An equivalent-target reconciliation, including the temporary checkpoint
variant, no longer depends on another mutation's pending intent: a target
holds one pending intent at a time, so this mutation has none to settle.
- The entry context keeps the typed cause, and the capacity-block and
intent-conflict classifiers also recognize the rendered message that an
error clone keeps.
- Concurrent bucket and listing failures keep the root cause over sibling
cancellations, so a capacity block pauses the routine.
* test(decommission): delete marker copy waits for a busy target gate
The Linux run of the S00 matrix showed the delete marker copy meeting a busy
target capacity gate without any leader kill. The copy already waits on that
gate like an ordinary object copy; pin it with an entry-level test that holds
the gate, checks the entry keeps waiting, and expects the key to migrate once
the gate is released. Share the versioned delete marker fixture with the
foreign pending slot test.
* fix(decommission): keep an adopted completion unless the target is absent
Close the remaining path where an adopted completion could lose its record
before being charged. A multipart cleanup that found a target it could not
prove as the exact owned copy still dropped the whole marker of a mutation
without a pending intent, after which the equivalent-target reconciliation
would treat the commit as already charged.
Any cleanup that did not prove the exact target absent now releases only the
bytes it freed when the mutation has no pending intent, and leaves the rest
for the reconciliation. This makes the three-state release unnecessary, so
the release proof goes back to the existing clear-pending flag.
* chore(ci): shrink the error format ratchet baseline for pools.rs
9eae47ce8 removed one Error::other(format!) call site from crates/ecstore/src/core/pools.rs, so the ratchet requires the baseline to drop from 178 to 177.
* fix(decommission): keep adopted completions in their own capacity marker
An adopted completion shared the mutation's staged marker, so whether
it survived a cleanup depended on the pool-wide free-space delta, and a
cleanup that freed little space left staged bytes behind and repeated
forever.
Record each adopted intent under a mutation id derived from the
interrupted mutation (decommission_adopted_intent_mutation_id) as a
separate temporaryMutations entry. A commit or an equivalent-target
reconciliation settles both markers, a temporary cleanup releases the
staged marker as before, and only a cleanup that proved the exact
target absent drops the adopted one. While the target holds only that
adopted completion, a cleanup keeps the delayed-release exemption of
the published pending intent it replaced. Older nodes see an ordinary
temporary mutation they never release, which only over-reserves.
Also log capacity_intents_adopted as soon as the adopted ledger is
durable, merge the root-cause helpers into decommission_root_cause,
drop the decommission_capacity_blocked: prefix fallback, and log the
already-charged equivalent-target branch at debug level.
Refs rustfs/backlog#2873, rustfs/backlog#2872
* fix(decommission): return uncommitted retries to their adopted target
A retry that had not committed yet went to the reserved target with the
most remaining allocation. When that was not the target holding the
mutation's adopted completion, the commit settled only its own target
and the adopted marker stayed inflight until the reservation was
released.
Prefer a reserved target that holds the mutation's adopted marker while
it still fits the object (decommission_reserved_target_choice).
Also cover the remaining settlement paths of the derived marker
(non-growing replacement, equivalent temporary reconciliation and its
discovery gate, scanner backlog retirement), repeated adoption of one
mutation and the zero-byte staging record in both lock models, drop two
conditions that always equal another one, and correct the adopted-id
doc comment about nodes without adoption.
Refs rustfs/backlog#2873, rustfs/backlog#2872
* fix(decommission): count a completion adopted twice only once
A mutation killed twice leaves an adopted completion and then a new
intent for the same object version. Adoption added the two, so an
11-byte object counted 22 inflight bytes and could trip the capacity
check once later commits shrank the budget. Both reserve the same
completion, so merge them by their maximum before adding staged parts;
a single adoption is unchanged.
Also assert in the three-pool sticky test that no entry failed, the
retry is charged on its adopted target and the source is cleaned, run
it for the legacy lock model too, stop saying operators rely on the
adoption log before rollback, and document that the selection fallback
cannot help an adopted completion.
Refs rustfs/backlog#2873, rustfs/backlog#2872
* fix(decommission): log where an adopted retry is placed
An e2e run could not show whether a resumed retry went back to the
target holding its adopted completion. Target selection now logs
adopted_target_reused when it does, and adopted_target_bypassed with
the adopted target pools when the object no longer fits there and the
adopted marker stays behind. Both carry the mutation id and the source
and target pools.
Refs rustfs/backlog#2873, rustfs/backlog#2872
* test(ecstore): pin null-version migration behind a delete marker
Decommission and rebalance must move a pre-versioning null version that
sits behind a delete marker, with its content and SHA-256 intact, before
removing the source entry (rustfs/backlog#2873).
* fix(ecstore): address the null version exactly during data movement
Decommission and rebalance read each source version with
`version.version_id.map(to_string)`. A null version, written before
versioning was enabled, has no version id, so the read used `None`,
which selects the latest version. Behind a delete marker that read
returned not found, the mover counted the version as already gone, and
source cleanup deleted the still-present null version with the entry.
Address the null version by the nil UUID, the existing exact selector,
on every data-movement lookup: the source read, the target put, new
multipart and complete options, the target resume lookup, and the
source revalidation before publication. The stale-target replacement
check treats the nil selector and an id-less null version as the same
identity, so unversioned buckets keep their current behavior. Target
pool selection still places a null version with the object's latest
version, as before, so a later null-slot overwrite replaces it there.
A version the mover skips as not found must also really be gone before
cleanup. It is dropped from the expected cleanup snapshot, so the
preflight reports SourceChanged while that exact version still exists
and the existing retry and exhaustion path retains the source.
Refs rustfs/backlog#2873, rustfs/backlog#2872
* fix(ecstore): prove multipart null-version staging absence exactly
The multipart staging cleanup proof in
abort_multipart_uploads_for_data_movement looked up the target with the
abort options' version id. Those options keep an id-less null version
as `None`, because the staged upload identity is derived from them, so
the proof resolved the target's latest version. Versions move newest
first, so it found the delete marker or newer version already moved and
refused every attempt with "cannot prove the exact staged identity is
absent". The legacy capacity model runs this proof on every attempt and
the exact model after any interrupted upload, so a multipart null
version behind a newer version could never be decommissioned. Look up
the exact null version for the proof and keep the abort options as they
are.
Cover the paths review found unprotected: multipart null versions under
the legacy model and after an interrupted upload, null versions behind
a newer version (lost without any not-found), the cleanup guard for
versions reported as not found while they still exist (decommission
read and delete-marker copy, rebalance read, and a control where the
version really vanished), and a retry after the null version committed
but before cleanup. The decommission fault hook can now inject a chosen
error, and rebalance gains a test-only source read fault.
Refs rustfs/backlog#2873, rustfs/backlog#2872
* test(heal): add failing regressions for a source-independent absent verdict
Add the regression tests for backlog#2900 that fail on the current code,
where the same object-level not-found is treated as deleted on the plain
heal path but as recreate_failed on the recreate path, and where the
erasure-set healer, the task predicates and the error helpers disagree on
wrapped not-found and transient inputs.
Also add the per-object disposition cache module with its unit tests, the
manager admission tests for it, the compat register entry for the display
boundary, and the admission-invariant notes. The module and the manager
tests are not declared yet, so nothing is wired into the build.
* fix(heal): decide object absence from storage evidence, not request source
Replace the five heal error classifiers (erasure-set healer outcome
mapping, task-level not-found and lock predicates, error.rs recovery
helpers and the storage existence-check matcher) with one typed
classify_heal_error. An object-level not-found from the locked storage
read is Absent for every source and every recreate_missing value; the
recreate path, the usage-observation and scanner object-dir whitelists
are gone, and display-text matching survives only in the marked
classify_error_display boundary.
Add a bounded, expiring per-object disposition memory consulted before
queue admission so a best-effort producer cannot requeue an absent or
terminally failed object on every tick; durable MRF, admin, forced,
high-priority and dry-run requests bypass it. The read-repair producer no
longer forces recreate_missing.
* fix(heal): keep rendered grace windows out of task retries
A grace window rendered into a task message belongs to an attempt that
already finished; its owner schedules the next one. Only typed grace
errors replay the task, as before the classifier merge. Update the
ecstore read-repair black-box test: the producer no longer forces
recreate_missing.
* fix(heal): classify ambiguous legacy delete residue as permanent explicitly
* fix(heal): scope disposition memory and bound absence parsing
A restarted node published FullReady from its own view while survivors still counted its drives unavailable (stale DNS for the old pod IP, then the Offline->Returning->Online hysteresis). A rolling update could then stop the next pod and cost a set its read quorum (#8669).
Add a read-only PeerDriveAdmission internode RPC that reports how a peer sees another host's drives, and gate the startup FullReady transition until a strict majority of each erasure set's peers count this node's drives online. The gate shares the startup budget, releases after 15s of unreachable peers, skips immediately when older peers cannot answer, and can be disabled with RUSTFS_STARTUP_PEER_READMISSION_ENABLE=false.
Apply the placement rule in .github/workflows/README.md to the 22 jobs
still on sm-standard-2 that only call APIs, read workflow metadata, print
summaries or scan published images, including cla.yml, the busiest
workflow. Moving the ODM cloud-source lane also stops it restoring the
ci-odm-interop key across images, since its writer already runs on
ubuntu-latest; targets-live gets a -hosted cache key for the same reason.
Clippy ran only after merge, so lint regressions merged freely and then
failed every main run until fixed (10 of 12 main runs on 2026-10-17/18).
Run the workspace Clippy step on every event, warm its check-mode
artifacts in cache-warm with the identical invocation so pull requests
re-check only what they touch, and pin the unconditional step in the
ci_gate contract and its self-test.
Hard quota admission is owned by the durable reservation ledger (decision
D3, rustfs/backlog#2872), but the request-path quota check still read
scanner usage on every write and mapped `UsageUnavailable` to a retryable
503 even for buckets the ledger admits. For a durable quota the checker now
treats scanner usage as reporting input: an unavailable lookup yields
`current_usage = None` and `allowed = true`, counted and logged at debug
level, and admin quota stats report the usage as null instead of zero.
Buckets without a reservation protocol keep the fail-closed snapshot path.
Quotas saved before the reservation protocols are re-saved under protocol
v2 with the same limit, type and created_at. The write goes through a new
guarded save that keeps the admin save's lifecycle/transaction guards,
incarnation check and fleet capability proof and additionally refuses when
the quota revision moved since the cached read, so a concurrent quota edit
wins. It is triggered lazily from the quota read path (background, one
attempt per bucket with a retry interval, never blocking the read) and in
bulk by `POST /rustfs/admin/v3/quota-migration`, which reports every bucket.
The ADR's Consequences section inventories the remaining scanner-usage
"authority" consumers with their meaning after D3, retirement condition and
compatibility window; the operations guide documents the migration and the
report-only usage semantics.
Refs rustfs/backlog#2904, rustfs/backlog#2872, rustfs/rustfs#8678,
rustfs/rustfs#5716, rustfs/rustfs#7350
Co-authored-by: Hauser <housemecn@gmail.com>
* fix(scanner): stop treating an ordinary PUT as data movement in cycle termination
A continuously written deployment never reached a `Complete` scanner cycle
(#8494, #8305, #7108). Three places in the termination path vetoed the cycle
for an ordinary PUT; each now lets the write shade its bucket, the path
per-bucket certification (#8542) built, and keeps the data-movement fences.
1. Usage publication is fenced by data movement only.
`scanner_data_usage_publication_blocked` no longer includes
`namespace_commits_pending()`, so the post-walk probe, the local defer
decision, the post-lease recheck and the usage-store route probes stop
superseding or deferring a cycle for a pending rename fanout. The scanner
publication leases (acquire, validate, target guard) drop the
pending-commit rejection and the namespace-generation staleness check and
keep the movement generation, the operation gate, generation exhaustion
and the purpose/target restriction. A test-util probe lets fixtures wait
for tails directly.
2. The plan identity is split into a coverage identity and a reuse identity.
`scanner_activity_coverage_digest` (peer set, protocol, topology, movement,
and a remote peer's generations) is what a bucket checkpoint is started
and resumed under, through `ScannerBucketScanPlan::checkpoint_coverage_digest`,
`DataUsageCacheReuseOptions::checkpoint_plan_digest` and
`prepare_bucket_checkpoint_with_coverage`. The reuse identity
(`scanner_activity_structural_digest`) keeps `instance_id` and
`maintenance_generation`, so a restart or a maintenance change still
invalidates the baseline and the set caches; a completed sweep is stored
under it. `publication_blocked` leaves every digest and stays a guardrail
veto. A remote worker still resumes under the single digest on the wire.
3. A bucket walk that stops early no longer makes the cycle `Incomplete`
when the bucket has a certified value. The set scan stands in with the
bucket's flattened value from its previous complete set cache
(`carry_partial_bucket_values`), the cycle reports it pending
(`ScannerBucketScanStatus::PartialCarried`), keeps it in the dirty map,
and the next scoped cycle walks it because the baseline's pending list
joins the dirty buckets. A bucket with no certified value keeps the cycle
`Incomplete`; failed buckets are unchanged.
No persisted file, generation, receipt or environment variable is added.
Refs #8494, #8305, #7108, rustfs/backlog#2903.
* chore(ecstore): gate the namespace commit probe to test builds
The scanner publication barrier no longer consults
InstanceContext::namespace_commits_pending, so it has no production
caller left; keep it for tests and the test-util facade only.
* test(rustfs): observe namespace commit ownership directly in node_service tests
The scanner usage publication barrier is movement-only now, so it no
longer reports a pending rename. The node_service tests that assert a
cancelled handler keeps its physical owner, or that wait for rename
tails to drain, read the namespace commit state through a test-only
ECStore probe instead.
* fix(scanner): preserve checkpoints on identity read failures
Distinguish confirmed missing incarnation sidecars from lookup errors before
cache preparation. Preserve durable checkpoint progress when identity reads
are unavailable and report post-scan uncertainty separately from replacement.
Reuse the transaction-fenced reader for optional and required lookups while
retaining existing missing, malformed, system-bucket and lock-loss behavior.
Verify preservation and recovery through a real coordinator with the identity
sidecar write-locked, including unchanged cache revisions and dirty work.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(ecstore): share required incarnation conversion
Keep the missing-incarnation error contract in one conversion used by both
required lookup entry points. This removes a duplicate formatted error site
and satisfies the existing error-format ratchet without changing its baseline.
Pin the full missing-identity error text in the real SetDisks regression.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): bound cross-disk MRF checkpoint I/O
Run independent journal disks with bounded concurrency while retaining each
replica's payload, companion and manifest ordering and the mirror phases.
Share encoded bodies, deduplicate local roots and collect every result
without changing receipt deadlines or persistence failure semantics.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): pin Normal mode when no bitrot start is recorded
Regression for rustfs/backlog#2905: three consecutive cycles that find no
recorded bitrot start currently run Deep through the selection window, and a
Normal cycle never records the cadence start.
* fix(scanner): anchor the bitrot cadence and budget per-disk I/O
Decide the scan mode from a recorded Deep window start only. A cycle that
finds no recorded start runs Normal and anchors the bitrot cadence in
process memory instead of opening the selection window; the Deep cycle that
follows the elapsed cadence records the window start exactly as before. A
missing record used to be read as a window that started at cycle 0, which
ran Deep for deep_window_cycles cycles whenever the start could not be read
back (rustfs/rustfs#8015). The recorded start is also remembered across a
skipped or failed save, and /v3/scanner/status reports the cadence state
under a new bitrot object.
Add a per-local-disk background I/O budget (operations and bytes per second,
defaults by detected media, auto|off|<rate>) that DynamicSleeper charges
before directory listings and object metadata reads as a hard ceiling
layered on the foreground-priority disk admission gate, with runtime config
keys, per-disk status, cycle metrics, pacing pressure, usage_freshness,
Prometheus series, Helm values, admin config help, and operator docs.
Refs rustfs/backlog#2905, rustfs/rustfs#8015, rustfs/rustfs#8377.
* test(scanner): expect Normal retries on erasure sets without a bitrot start
The pending-PUT retry fixture asserted that a 4-drive set revisits every
bucket because the cycle ran Deep. Without a recorded Deep window the cycle
now runs Normal on every layout and reuses the cold inventory.
Count only post-format rebuild failures against the replacement retry
budget; handoffs, format fences, outages, grace waits and conflicts keep
their durable reason without spending an attempt. Retired generations
(committed handoff or approved legacy migration) no longer own their set,
so fresh admission for other slots proceeds, and a committed predecessor
becomes collectible once its successor is verified.
Block discovery conflicts per generation instead of per set, re-evaluate
them on every pass, and expose the blocked entries with reasons. Report
root heal inventory conflicts per record and keep replaying the others.
Revalidate blocked checkpoints hourly with the current reader and clear
markers the reader accepts. Return a dedicated error for ambiguous legacy
delete residue instead of the retryable SlowDown.
Implements rustfs/backlog#2902. Fixesrustfs/rustfs#8664. Addresses
rustfs/rustfs#8366, rustfs/rustfs#8565, rustfs/rustfs#8645 and part of
rustfs/rustfs#8608.
Co-authored-by: Hauser <housemecn@gmail.com>
The first pull-request run at CARGO_BUILD_JOBS=3 that met a contended
node (39193827281) hit the 75-minute nextest timeout before any test
started. Its sampler shows cargo spinning for 70 minutes with the pod's
io.pressure "full" at 86-88% and cpu.pressure at zero, with node-level
IO pressure already at 36% when the step began: three concurrent
workspace-scale compile units saturate the pod's shared storage, the
mechanism the original #5394 mitigation named. Keep 3 on main pushes and
dispatches, where three runs succeeded, and 2 on pull requests, in both
the nextest and the serial ILM step; pin the ILM budget guard back to
that expression and record the evidence in the step comment.
* fix(scanner): give startup gates automatic exits instead of a pause
Every fail-closed scanner gate now has a bounded automatic exit without
deleting quarantine evidence or relaxing the leader, CAS, movement or
usage-floor fences (rustfs/backlog#2901).
- Recovery retries back off instead of pausing: after the short budget the
observation reads `backoff`, stays retryable, names the next attempt and is
capped at ten minutes; one success resets the count and a 24 h window
restarts the budget (rustfs/rustfs#8297, #8670). The usage-floor failure no
longer carries the retry count across attempts.
- A corrupt `.bloomcycle.bin` is rebuilt on the iteration after its quarantine
marker is durable: the fenced leadership claim CAS-replaces the quarantined
revision under a higher epoch and the marker stays as evidence.
`future_schema` and `cleanup-pending` remain blocked.
- An unclaimable pause ledger runs the leader degraded: cycles run untracked
behind the data-movement fence, `persistence_state` reports `degraded`, and
the claim is retried at most every five minutes (rustfs/rustfs#8251).
- The usage-state full-rebuild reset starts from the empty baseline when every
usage slot is corrupt and no cycle state exists (rustfs/rustfs#8670, #6951).
- A cycle-state save reconciles non-conflict failures against durable state
and retries under the same epoch; an exhausted save reloads durable state in
place. Only a newer owner or an unreadable primary ends the iteration, so
peers see a changed fence only when ownership changed (rustfs/rustfs#8141).
* test(scanner): align usage-state reset tests with the empty-baseline rebuild
The full-rebuild reset no longer refuses corrupt or identity-less usage slots
when no cycle state exists (rustfs/rustfs#8670). The two tests that encoded
that refusal now check the new contract: the reset starts from the empty
baseline with a bootstrap-pending marker, never an authoritative zero floor,
and a valid backup still supplies the recovery floor.
---------
Co-authored-by: cxymds <cxymds@gmail.com>
Decommission listing failed closed on `.rustfs.sys/buckets/.heal/`, a
directory that holds only the node-local MRF journals on one drive and
no xl.meta. The partial-listing resolver rejected the single prefix
directory below dir quorum and recorded it as unresolved object
metadata, failing the run before the worker's directory skip ran.
Treat a partial sample made only of empty-metadata directory candidates
with zero disk errors as the prefix it is and hand it to the callback
like a quorum-resolved directory; the recursive walk lists any object
beneath it on its own. Samples with disk errors, objects below the
metadata quorum, and mixed samples stay unresolved.
Fixes#8667
* fix(ci): refresh the e2e-full selection for the new policy tests
#8615 and #8671 added three e2e tests without refreshing the e2e-full
selection digest, so "Verify e2e full membership" has failed on every
main push since. Regenerate the Linux digest from the listing the failed
run uploaded; the same procedure reproduces the previously committed
value from the last green run's listing, and the name-level diff shows
three additions and no removals. The darwin digest is refreshed
separately from a real macOS listing.
* fix(ci): refresh the darwin e2e-full selection digest
Regenerated from a real `cargo nextest list --profile e2e-full -p e2e_test`
on macOS (rustc 1.99.0, cargo-nextest 0.9.140), so the local check on
macOS agrees with the three policy tests added by #8615 and #8671.
#8682 replaced the cold `cargo check -p rustfs --release` with a
`cargo rustc ... -C debug-assertions=off` on the rustfs lib alone. That
leaves the dependencies on the dev-profile cfg, so the lib also fails on
items gated on one side of cfg(debug_assertions) only (E0599 on
Filenames::Embedded), and the step's "fails solely at the compile_error"
assertion can never hold; the lane has been red on main and every pull
request since. Restore the whole-graph release check, drop the two-job
limit that only a linking step needed, and record why the single-crate
variant is unsound.
Reuse the fingerprint-validated snapshot while the metadata owner fences an unchanged key. Refresh only after absent-key preparation, retaining destructive preflights, sibling refreshes and final inspection.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
An empty storageclass.dataStorageSize used to be the way to run on
user-supplied drives (extraVolumes/extraVolumeMounts, e.g. JBOD hostPath),
but the chart now always renders data PVC templates, with an invalid empty
storage request.
When the value is empty, render no data volumeClaimTemplates and no data
volumeMounts, drop the data directory preparation from the init container
(and the init container itself when there is nothing left to prepare), and
omit volumeClaimTemplates when no PVC template remains. Standalone mode skips
the data PVC and data volume unless existingClaim.dataClaim is set. Pools may
override the value independently. Default renders are byte-identical.
Fixes#8665
Raise CARGO_BUILD_JOBS to 3 on every event in the workspace test lane and
record the closed backlog#1601 experiment: the test-profile compile took
28m21s at 2 on a pull request and 16-23 min at 3 across three successful
main runs.
Merge the two Connect boundary jobs into connect-feature-boundaries. The
offline enrollment gate no longer exports a fresh CARGO_TARGET_DIR, so it
rebuilds only the rustfs crate on top of the restored ci-dev cache; the E2E
root is isolated by the feature gate and the dedicated binary name, not by
the target directory. The short-credential release guard is now
`cargo rustc -p rustfs --lib --features connect-e2e-short-credentials --
-C debug-assertions=off`, which trips the same cfg(not(debug_assertions))
compile_error against cached dev-profile dependencies instead of a cold
release check of the whole graph. Every assertion of both lanes is kept.
Update ci_gate.py and the ci-gates.md PR matrix for the merged job.
* ci: move compile-free and prebuilt-binary lanes to hosted runners
Standard GitHub-hosted runners are free for this public repository, while
every sm-standard minute is paid self-hosted capacity. Move the jobs that
never compile (quick checks, typos, scope selection, the required-checks
aggregate, audit, closed-PR cancellation) and the two s3-tests lanes that
only run the prebuilt debug binary to ubuntu-latest, drop the wget install
the custom image needed for typos, and move the upgrade-compatibility
workflow, whose build shape already runs on hosted runners in
e2e-distributed, with renamed cache keys so the hosted and self-hosted
images never share a rust-cache entry. Gate the package summary job on
resolve having run, so development builds stop starting a pod to report a
skipped job.
* docs(ci): document runner billing and placement rules
Record in .github/workflows/README.md which runner labels GitHub bills
(none for this public repository), which ones RustFS pays for, the rule for
when a job may leave ubuntu-latest, the rust-cache single-writer and
same-image constraints, the patterns that have leaked self-hosted minutes,
and a checklist for new or moved jobs. Point to it from .github/AGENTS.md,
docs/testing/ci-gates.md and the ci.yml placement comment so the rule has a
single home.
Supersede the 2026-09-03 decision: hard quota admission is owned by the
durable reservation ledger, and scanner data usage is best-effort reporting
input. Record that a pending ordinary PUT commit is a write-barrier event
rather than a usage-publication veto, and split the scanner plan identity
into a coverage identity and a reuse identity.
Refs rustfs/backlog#2872, rustfs/backlog#2903, rustfs/backlog#2904.
* feat(policy): support object creation operation conditions
Derive the condition from the resolved S3 operation and keep multipart staging separate from object publication. Reject request and claim spoofing and cover policy evaluation and HTTP upload paths.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(protocols): preserve multipart staging authorization
Keep the trusted protocol operation until the shared policy context is built, so FTPS and SFTP honor object creation conditions on multipart staging. Reuse the server-only marker and test real backend decisions for IAM and bucket policies.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): cover ObjectCreationOperation in STS session policies
Run AWS's conditional-write example 1 as an inline AssumeRole session
policy: object publication needs a request header, while multipart
staging is exempt through s3:ObjectCreationOperation.
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: oc7o <oc7o@sloow.de>
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(heal): move impls before test module to satisfy clippy items_after_test_module lint
Clippy in Rust 1.99 flags items defined after #[cfg(test)] mod tests.
Move Serialize and Deserialize impls for CheckpointJson before the
test module in crates/heal/src/heal/resume/checkpoint/integrity.rs.
* fix(heal): restore checkpoint integrity file newline
Add the missing final newline so cargo fmt --all --check passes.
The checkpoint serializer implementations and tests remain unchanged.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* perf(filemeta): avoid clones and intermediate allocations in xl.meta codec
- Slice version headers and metadata payloads directly from the xl.meta buffer in FileMeta::unmarshal_msg instead of allocating temporary header/metadata buffers per version.
- Decode FileInfo directly from borrowed xl.meta slices in get_file_info without constructing an intermediate FileMeta, Vec<FileMetaShallowVersion>, or InlineData copy.
- Validate and search InlineData keys on borrowed slices without per-key Vec/String allocations.
- Borrow FileMetaVersion in FileMetaVersionHeader::from and stream object/delete-marker signatures directly into Xxh64Writer instead of cloning full version structs and marshaling temporary buffers.
- Encode FileMetaVersionHeader into a stack buffer in FileMeta::marshal_msg and pre-size output buffers.
- Decode msgpack struct field names onto a 16-byte stack buffer via read_small_str and unwrap PrependByteReader after reading submap headers.
* test(filemeta): pin xl.meta codec compatibility
Pin pre-optimization signatures for historical RustFS and MinIO fixtures,
and cover direct reads across selectors and decode options. Preserve null
inline keys, free-version selection, empty histories, and allocation
boundaries for large version bodies.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(policy): support s3:if-none-match and s3:if-match condition keys
Bucket, IAM, and STS session policies that enforce conditional writes
with s3:if-none-match or s3:if-match were rejected with "invalid key
name". Accept both keys and populate them from the If-None-Match and
If-Match request headers. A blank header is left out of the condition
values because the write ignores it, so it cannot satisfy a Null check
for a write that runs unconditionally.
* fix(policy): classify conditional write keys as request-derived
s3:if-match and s3:if-none-match are filled from request headers, but
is_server_derived() reported them as server-derived because their names
lack the x-amz- prefix. Add an explicit arm so header input is never
refused for these keys if they join COMMON_KEYS.
* fix(s3): reject non-wildcard If-None-Match on writes
S3 accepts If-None-Match on PutObject, CopyObject and
CompleteMultipartUpload only as "*" and answers other values with 501
NotImplemented. RustFS compared such a value as an ETag, so a write with
If-None-Match: "<other etag>" overwrote the key while satisfying a
policy that requires s3:if-none-match.
Reject ETag values in the three handlers before any I/O. The store-level
check now treats any non-blank If-None-Match as create-only, so a caller
that skips the handler check cannot weaken the precondition. Drop the
CompleteMultipartUpload ETag pre-check that the rejection supersedes.
* test: tighten conditional write policy coverage
Pin trimming and the non-UTF-8 case in the condition values, and drop
the auth test that re-ran policy evaluation already covered by the
policy crate. Run the three policy e2e scenarios on one server, assert
that ETag If-None-Match cannot overwrite under a Null: false policy and
that CompleteMultipartUpload with If-None-Match: * returns 412. Share
the AssumeRole response parser through common.rs.
* test(ecstore): isolate manual delete recovery from workers
Stop fixture background services before seeding recovery journals so the
explicit recovery pass owns the decisions its assertions inspect.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(s3): enforce request-derived conditional write guards
Validate raw If-None-Match headers as well as DTO fields so POST form
uploads cannot skip rejection of unsupported ETag preconditions. Derive
conditional policy keys after identity claims and clear colliding claims
when the actual request has no usable precondition.
Align multipart and pinned Ceph regressions with wildcard-only writes,
retaining If-Match, object-content, version and delete-marker assertions.
Document that 501 is the implementation's unsupported-value response.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(s3): eliminate conditional write request copies
Borrow request headers during policy condition collection and exclude
conditional keys at the identity-claim boundary instead of inserting and
removing them. Preserve existing aliases and multivalue header behavior.
Move the CompleteMultipartUpload input out of its request, compare strong
ETags by reference, and remove the ETag If-None-Match branch made unreachable
by the write guard. Keep both form/HTTP input checks and storage commit
preconditions intact.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(s3): align ignored conditional PUT and COPY cases
Use If-None-Match: * for the existing-object 412 regressions and assert 501
for matching and nonmatching ETag values. Cover both header-only callers
and parsed DTO-plus-header requests in the serial ILM fixtures.
Check object bytes and live version counts after each rejected operation.
Keep the existing test names and ignored-test lane selection unchanged.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(oidc): make the Console session duration configurable
The OIDC callback hard-coded the Console credential lifetime at 3600
seconds, so a Console session always ended after about an hour.
RUSTFS_CONSOLE_AUTH_TIMEOUT, the documented knob next to this behaviour,
is the Console HTTP request timeout and cannot extend a session.
Resolve the lifetime from the process-level
RUSTFS_CONSOLE_OIDC_SESSION_DURATION instead and forward it to
complete_authorization_code(). Unset, unparseable, zero, and negative
values keep DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS (3600), so
existing deployments keep the previous behaviour and a misconfigured
variable can never issue an already expired credential. Values above
CONSOLE_OIDC_SESSION_DURATION_MAX_SECS (7 days) are clamped to that
maximum with a warning, which also caps how far an operator can stretch a
credential by accident.
Keep the Console logout token for that same lifetime. OidcStateStore
pinned the logout cache to a hard-coded hour, so a session configured to
last longer than that dropped its logout token while the credential was
still valid: RP-initiated logout could no longer resolve, and the browser
was sent back to the Console login page with the IdP session still
active. The logout cache now derives its TTL from the same
console_session_duration_secs(), which applies the same clamp, and the
callback reads its lifetime from that one function instead of a local
copy, so the credential lifetime and the logout-token retention cannot
drift apart. Add tests covering the default hour, an extended session,
the maximum, and out-of-range values.
The variable is deliberately not part of ENV_IDENTITY_OPENID_KEYS: that
list is suffix-scanned per provider, and this setting applies to every
provider. It has no identity_openid KVS equivalent and needs a restart,
like the other process-level settings in the same document.
Document the setting in the login flow, the process-level table, and the
troubleshooting table, and clarify that RUSTFS_CONSOLE_AUTH_TIMEOUT is a
request timeout rather than a session lifetime.
* test(ecstore): retain offline marker fixtures across reconnect
Keep simulated drive handles attached and reject I/O until explicit rejoin.
Attempt reconnect in the current-marker case so the former CI race is
deterministic while retaining physical cleanup and historical-data checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(oidc): align session diagnostics and lifetime regression tests
Use structured IAM diagnostics for clamping and move configuration tests
to their owning crate. Cover duration boundaries, logout retention and
signed credential expiration through the federation binding.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
GatewayJ