* test(e2e): tighten scoped multipart fault acceptance
Target Abort root deletion instead of child transaction cleanup, require
actual metadata peer hits, and keep faulted APIs single-attempt. Validate
bounded ordinary recovery and safe quorum refusal for partial mutations,
with the recovery enhancement tracked in backlog#2940.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(e2e): drive concurrent TLS reads independently
Give each coordinator its own supervised task and cooperative polling
budget while retaining the existing workload, deadline and byte/hash
assertions. Preserve node logs after fixture teardown and report the
operation, response phase and received bytes when a transfer times out.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scripts): collect scanner host telemetry on macOS
Use platform-supported iostat arguments and expose optional collector
failures without interrupting scanner-status collection.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): reuse validated metadata read path buffers
Share the checked bucket/object tuple across metadata read consumers and borrow clean normalization inputs without skipping filesystem validation. Keep Windows reconstruction and the second bucket check intact.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scripts): finalize scanner telemetry for failed runs
Initialize host telemetry artifacts before external captures so reused output directories cannot retain prior successes. Share normal and error-path finalization, reap collectors on request failure, and preserve the primary exit status even if status recording fails.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): distinguish unreadable recovery inventories
Treat only the exact system-prefix read-quorum result as incomplete during bounded polling. Preserve strict final reads and the shutdown/drain assertions, record failed observations, and cover interrupted empty streaks and unrelated errors.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): avoid head-of-line blocking in delete recovery
Recover independent keys through a bounded rolling window while preserving per-key hint generations, admission budgets, cancellation retry, and the existing transaction decision engine.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): bound durable delete preparation and bootstrap recovery
Prepare independent Linux delete transactions with bounded concurrency while retaining per-key result order and serial fallback for overlapping, rollback, and legacy requests. Preserve startup discovery until bucket metadata is initialized, and add release performance and process-crash acceptance runners.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): verify clean exits in recovery acceptance
Reject forced or OOM exits when measuring clean startup and verify original ETags across mixed-layout crash recovery and a second restart.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): distinguish prepared and decided batch crashes
Observe real preparation and receipt states separately, retain non-vacuous acknowledged-delete sentinels, and require read-only physical convergence before S3 reads can trigger cleanup.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): move final delete recovery options
Move the options value into the batch-ordering regression's final case
instead of cloning it. Keep the earlier clone needed for reuse.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): replay delete receipts after interrupted proof GC
Retire only the exact empty transaction leaf before journal activation. Preserve occupied directories and prefix children, and cover partial cross-disk proof GC without a fresh delete. Tighten crash evidence to distinguish directory-only remnants from retained content.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bound implicit startup delay and publish config early
Limit warm-cache startup to the smaller of the cycle interval and one
minute before jitter, while preserving explicit startup settings.
Publish configuration gauges when applying runtime settings and retain
timeout diagnostics until an actual new cycle begins.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): unify startup waits and skip unchanged configuration
Bound read-only startup inspection and make startup and pre-leadership
retries respond to cancellation and configuration changes. Preserve one
startup deadline and discover replication behind lifecycle-only buckets.
Avoid irrelevant bucket listings and repeated per-disk reconfiguration.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): retain effective startup policy and fallback metrics
Keep the sampled startup deadline when configuration changes leave its
normalized policy unchanged. Initialize retained runtime metrics after
an invalid first refresh without suppressing the validation error or
replacing a newer accepted configuration.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* perf(rpc): remove redundant boxing from read streams
Keep the capped read stream concrete until StreamingBlob owns its type erasure. Add verified four-node TLS coverage for full, unaligned range and concurrent reads, with bounded client memory and explicit integrity checks.
Document protocol comparison commands and the old-node workaround during rolling upgrades.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): preserve TLS trust for alternate credentials
Derive credential-specific clients from the cluster configuration so they retain the generated CA. Exercise successful trusted reads and reject an invalid replacement identity instead of inheriting root credentials.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): collapse acceptance report path guard
Combine the optional report path and existing-file checks into a let chain
without changing the report or PID sidecar collision behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Recover independent keys through a bounded rolling window while preserving per-key hint generations, admission budgets, cancellation retry, and the existing transaction decision engine.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(rio): isolate internode streams with pooled HTTP/1.1
Default the internode HTTP data plane to reusable HTTP/1.1 connections so paused shard bodies cannot consume shared HTTP/2 receive credit. Retain explicit auto negotiation and the existing TLS, pooling, and recovery paths.
Refs rustfs/rustfs#8843 and rustfs/backlog#2934.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(rpc): frame bounded shard reads for connection reuse
Preserve the bounded read cap and stream-to-EOF sentinel while making connection completion independent of a chunked terminator. Cover exact reads without EOF polling, pooled TLS reuse, and partial-body truncation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(deps): migrate signer and OIDC to patched crypto
Pin the requested signer fix and upstream OIDC/OAuth2 crypto migrations to immutable commits. Preserve registry SDK type identity, remove the old bigint and elliptic-curve chains, and verify independent login state, nonce, and PKCE values.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): isolate async checkpoint log capture
Scope captured tracing subscribers to each future poll and cover the
competing-dispatcher failure without weakening checkpoint assertions.
Verify exact bounded erasure-probe accounting around the 64-entry limit.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): classify erasure candidates in one budgeted walk
Continue directory classification in cooperative 64-entry batches until
part evidence or EOF, retaining normal checkpoint and namespace work.
Defer metadata accounting and namespace reuse until classification ends.
Distinguish vanished opens from incomplete iteration and preserve quota,
sampling, failed-object and non-destructive heal behavior.
Pin test callsite interest alongside poll-scoped capture to prevent an
unsubscribed sibling from caching the checkpoint callsite as disabled.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): roll a wedged pause backlog forward to its proven ledger
For rustfs/backlog#2873 defect E.
Two interleaved writers could leave a replica state that no claim could
leave. Writer B read writer A's partial commit and selected the stable
rollback point. A's commit then completed and A verified it. B's rollback
stabilization overwrote every member it had read after the commit, and A's
stabilization reached only the remaining member. No commit survived and the
stable fields differed, so every claim and every decommission retirement
failed with "neither a surviving membership commit nor a stable rollback
point".
- When no commit survives and stable consensus fails, select the unique
newest stable ledger by (writer epoch, generation). A stable field is
written only after its ledger was selected as authority, so that ledger is
the last proven one. The next claim stabilizes it and advances normally.
- Keep failing closed for an invalid or unreadable replica, for two
different stable ledgers at the newest key, and for a missing replica that
a stored commit names as a member.
- A rollback proof written without an authoritative commit names only
members that already store a record, so a failed first write to a new
member cannot turn it into a named member that reads as lost.
- Retirement refuses a rolled-forward ledger until the writer has
stabilized it, as a retryable handoff conflict rather than a terminal
authority conflict.
- Tests replay the interleaving on real storage: it converges and retires,
it converges after another expansion with a failed first write to a new
member, and a same-key fork or a lost named member still refuses.
* fix(scanner): let retirement stabilize a rolled-forward pause backlog
Follow-up to the defect E roll-forward for rustfs/backlog#2873.
- Retirement no longer waits for a scanner writer. When no commit
survives and the native selector over every read member (survivors,
retiring source and queued pools) rolls forward, the planner commits and
stabilizes that ledger on the survivors under the writers' object fence.
The previous retryable handoff could exhaust its ~13.5 s budget long
before a writer re-claimed, and failed the decommission terminally.
- A writer's roll-forward refuses when a stored commit names a member that
is missing or outside its read, such as a retiring source excluded from
the writable set. Retirement, which reads that member, decides instead.
- A writer stabilizing a rolled-forward ledger writes missing members
first and stops at the first failure. A member that only read as missing
rejects the create, so no surviving proof can omit its newer record.
- Docs describe the actual roll-forward preconditions and the retirement
exit.
- Tests: retirement converges the wedge with no writer; a writer refuses to
roll forward past an unread source that holds the newest ledger and
retirement keeps it; a spurious missing read changes no record and the
next claim converges on the newest ledger.
* fix(decommission): create missing pause backlog survivors first
Follow-up to the defect E retirement roll-forward for rustfs/backlog#2873.
- Retirement creates survivors that read as missing one at a time, before
any stored survivor, and stops at the first failed create. A transient
create failure on a new pool's set no longer leaves every other survivor
holding a commit that names a still-missing member, which no later
selection could leave.
- A rolled-forward retirement first seeds a rollback proof that names only
the survivors that store a record; the commit still names every survivor.
- Retirement rolls forward only when the survivors cannot select a ledger
on their own or select the same one. Survivors that agree on a different
ledger keep the source-authority check.
- The scanner writer applies the same missing-first, stop-at-first-failure
order to every stabilization with a missing member, including a seed from
an old commit selected as the rollback proof.
- Tests: a failed new-pool create during retirement retries to convergence;
a spurious missing read cannot seed an old commit over a newer member;
the rolled-forward precedence over source stable records is pinned.
* docs(decommission): state the all-missing survivor limit of backlog roll-forward
* feat(scanner): configure startup recovery backoff cap
* test(scanner): compare recovery retry count without narrowing
* docs(scanner): document recovery backoff cap and expose it in Helm
List scanner.recovery_backoff_cap in the runtime controls table, drop the
fixed ten-minute wording from the follower recovery section, and map
config.rustfs.scanner.recovery_backoff_cap_secs to
RUSTFS_SCANNER_RECOVERY_BACKOFF_CAP_SECS in the chart.
* fix(scanner): recover dual corruption with retained epoch fencing
* fix(scanner): preserve recovery markers and legacy quota floors
* test(scanner): expect retained usage recovery marker state
* docs(scanner): register dual-corruption recovery evidence objects
Add the .bloomcycle.bin.dual-recovery/ evidence family to the persisted
objects table and describe its owner, write-once lifecycle and recovery
rule in Deletion And Recovery Rules.
* fix(s3): expose typed causes for unavailable responses
Keep the existing S3 retry contract while distinguishing namespace lock
timeouts, other lock refusals, and storage write quorum failures. Reuse one
bounded typed-source walk and document multipart session attribution.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(s3): preserve XML type on attributed errors
Keep application/xml when the pinned serializer replaces generated headers
with the custom cause map. Assert the media type on direct and wrapped lock
timeouts and namespace/write-quorum error responses.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): unify multipart generation fence lifetimes
Own registry registration before asynchronous incarnation validation so
cancellation cannot preserve a memo beyond real lifecycle-lock coverage.
Reuse successful validation for overlapping PUT and multipart operations,
and retain the full owner through detached part commits and orphan cleanup.
Keep cold validation concurrent and preserve authoritative snapshot,
borrowed-fence, and disabled-lock checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): remove redundant pending closure
Pass std::future::pending directly to the incarnation validator in the
cancellation regression test so strict Clippy accepts the test target.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
A set-remote-target update with the creds group on a bucket target that
belongs to a site-replication peer skipped the connection change, logged a
warning, and still returned 200. The Console's bucket replication form sends
exactly that request when an endpoint changes, so operators saw success while
the stored endpoint, credentials, and TLS settings stayed unchanged.
Return 400 InvalidRequest instead, naming the target and the evidence that
makes it peer-owned, and pointing at a site-replication edit. The ownership
probes are unchanged; the decision is extracted so it can be unit tested.
Document how to change a site's address: the local endpoint is recorded once
at setup and never re-derived, edits must reach every other peer at its stored
endpoint, so sites move one at a time (or old addresses are restored
temporarily), followed by a resync. Commands use rc because mc's resync does
not work against RustFS and its update renames the site.
* fix(decommission): let a failed start roll back its own activation
A fresh decommission start whose peer reload failed could not roll back:
the pool metadata stale-write guard from #6604 refuses any save that
clears a persisted active or queued decommission, so the pool stayed
active with no worker until the watchdog or a leader restart picked it
up.
The rollback save now carries the activation that start committed. The
guard exempts a pool only while its persisted record still equals that
activation field for field, so a worker checkpoint or a later start
keeps the rejection. A canceled record is not an active decommission
and is cleared by the rollback as well. Every other save path passes no
activation and is guarded as before.
A peer may already have reloaded the activation and started a worker
before the rollback. Saves are now fenced by decommission generation: a
write carrying a start_time is rejected when the persisted record
belongs to a later start, or has no decommission state and was updated
after that start_time, so such a worker cannot make the pool active
again. Its queued promotion and progress checkpoint also adopt the
persisted record and stop the worker.
Refs rustfs/backlog#2873
* fix(decommission): fence terminal writes of a withdrawn decommission
Terminal records carry no start_time, so the generation fence did not
cover them: after a start rollback, a peer worker of the rolled back
generation that failed (for example on a bucket completion) could still
persist a failed record over the restored pool. The merge now rejects a
failed, canceled or complete record over a persisted record without
decommission state; every attempt persists decommission state first, so
no legitimate path writes a terminal record there.
Peers also drop a withdrawn decommission sooner. A pool metadata reload
replaces a running decommission with a newer record that withdrew it,
even under an active worker; bucket completion adopts such a record like
promotion and progress checkpoints do; and a withdrawn progress
checkpoint stops the worker instead of logging a failed save and moving
more entries.
Tests cover the peer failure mark, a rollback retry that sees a worker
checkpoint land between attempts, and the reload withdrawal. A heal
fixture that wrote complete records over a fresh pool.bin now persists
the active attempt first.
Refs rustfs/backlog#2873
* fix(decommission): fence peer writes after a retry start rollback
A retry start replaces a failed or canceled record that keeps its
unresolved ledger, so its rollback restores that terminal record rather
than clearing the pool. The withdrawal checks only recognized a record
without decommission state, and the generation fence ignored a persisted
record without a start_time. A peer worker of the rolled back retry
could therefore checkpoint the pool back to active.
The generation fence now compares the persisted last_update whenever
the persisted record has no start_time of its own (cleared, terminal or
queued). The progress checkpoint also checks the generation against the
record read under the pool metadata lock, as bucket completion already
does, and the save-path withdrawal check treats a terminal record newer
than the running view as a withdrawal, so the peer adopts it and stops.
A reload still never lets a newer terminal record replace an active
worker's view, so after a retry's rollback a peer worker stops at its
next bucket completion or progress checkpoint rather than on reload.
Tests cover the retry rollback against a peer checkpoint and direct
write, and a peer bucket completion after a fresh start's rollback.
Refs rustfs/backlog#2873
* fix(decommission): keep a retry rollback's restored record intact
After a retry start rolls back, the restored record is failed, so a
peer worker's own failure mark passed every merge guard and replaced
it. A failure mark now saves through save_current_pool_meta_checked,
which runs the withdrawal check against the record read under the pool
metadata lock and adopts that record instead of writing.
The withdrawal check compared the restored record with the worker's
in-memory last_update, which a rejected save had already raised, so the
peer never adopted the rollback and the watchdog kept resuming it. It
now compares against the running generation (start_time), and a
rejected unresolved-entry save undoes its in-memory change.
save_current_pool_meta_for_decommission_start is also compiled for the
test-util feature again; a test-util helper still calls it, which broke
the workspace clippy and Connect builds.
Refs rustfs/backlog#2873
* fix(decommission): keep an unresolved entry when its own save fails
The rollback of a rejected unresolved-entry save also dropped the entry
on transient failures such as a quorum or lock-fence error, so the
following failure mark persisted a failed record without it and an
operator could clear a decommission that skipped an unreadable object.
Keep the entry and undo only the last_update bump, which is all a
withdrawn worker needs to still see the withdrawal.
Refs rustfs/backlog#2873
* fix(decommission): run the leader-side start to completion
A forwarded decommission start awaits `ECStore::decommission` inline in
the RPC handler. When the forwarding node's RPC times out, the stream is
reset and the handler future is dropped wherever it is. Dropped after
`pool.bin` records the pool as active but before the peers reload and
the worker is spawned, the start left the pool active with no worker and
peers on the old metadata, and a retry was rejected as already active.
An admin client that disconnects can do the same.
Run the start in its own task and let the caller await only its result.
The start then always reaches its end: it fails before persisting
anything, or persists the pool, has the peers reload and spawns the
local workers, rolling back if the reload fails. A retry still cannot
start twice: it waits on the start gate held by the running start and
either reports the busy gate or is rejected because the pool is active.
`decommission` now takes `self: &Arc<Self>` and spawns workers on that
store instead of looking up the global object store, which is the same
store in production and makes the path testable; the unused
`require_decommission_store` helper is removed.
Refs rustfs/backlog#2873
* fix(decommission): detach the leader-side start from its caller
9ac4c10de carried only the reproduction (the testable entry point, the
peer-reload pause point and the proof test). This adds the fix it
describes: `ECStore::decommission` spawns the start as its own task and
awaits only its result, so a caller that stops waiting, such as a
forwarded start whose internode RPC timed out, can no longer drop the
start between persisting the pool as active and having the peers reload
and the local worker start.
Also prove a retry while the first start runs cannot start the pool
twice, update the operator and architecture docs for the new behavior,
and shrink the error(format!) ratchet baseline for the removed helper.
Refs rustfs/backlog#2873
* fix(decommission): single-flight start, recheck pool state under gate
Review of 3a5276c5b:
- A start re-checks the pool state under the start gate before it
cancels routines, so a start racing another one can no longer cancel
the worker the other start just spawned.
- Only one start runs per instance at a time: another request, including
the retry of an operator whose request timed out, is rejected as
already running instead of stacking a second detached start and its
preflight heals. The start runs through `run_detached_mutation`, and a
failure nobody waits for any more is logged.
- Tests: the proof test pins its drop after the save and requires a
worker that persists a cancellation; the retry test really drops the
first caller; new tests cover the under-gate recheck and a peer reload
failure after the caller is gone. That last one shows the start
rollback cannot clear a fresh start because the pool metadata
stale-write guard refuses it; the gap is asserted and documented and
tracked in rustfs/backlog#2873.
- The start tests run on their own large-stack thread: the io_uring lane
(`uring_`, which also selects `during_`) runs them with libtest's 2 MiB
default stack, where the proof test overflowed.
- Docs: single-flight starts, status and cancel during preflight, the
rollback gap and its recovery, and the release requirement.
Refs rustfs/backlog#2873
* fix(decommission): turn away local start retries at the start slot
Second review of 29253430a:
- The leader-local admin pre-check refreshed rebalance metadata under
the start gate before the start slot was checked, so a local retry
waited out its 20 s gate budget behind a running start (which holds
the gate through its peer reload) and then got a 503. The pre-check
now checks the slot first and returns `DecommissionAlreadyRunning`,
which the handler reports as a 400 that keeps the reason.
- Prove the reload-failure path really ran in the detached start (the
injection is consumed), add a test that a fresh start's failed
rollback reports both the reload error and the stale-write refusal,
and assert an abandoned start stays tracked for shutdown.
Refs rustfs/backlog#2873
Second review of 652c56875..0ef02a485:
- The operator doc and the budget comment claimed a forwarded start that
outlives the internode RPC timeout still completes on the leader. The
leader drops it wherever it is, possibly after pool.bin records the
pool as active but before its worker starts or the peers reload.
Document that state, the watchdog that resumes the worker, and the
checks to run; the cancel-safety fix is tracked in rustfs/backlog#2873.
- Prove the budget is shared across waits (paused clock: a 60 ms wait
leaves 40 ms for the next gate).
- Prove the rebalance-refresh start gate and activation gate waits and
the decommission() entry are bounded and named.
Refs rustfs/backlog#2873
Review of 652c56875 found that the 30 s start budget was a wall-clock
deadline fixed when the start began, so slow preflight work (bucket heals,
capacity scans) could spend it before the activation gates were reached.
A gate held for a few milliseconds then failed the start at once, and a
retry repeated the preflight.
- Replace the deadline with a budget that only counts time spent actually
waiting for a gate; a free gate is always taken, and the lock timeout
reports the time waited.
- Lower the budget to 20 s, below the default 30 s internode RPC timeout,
so a forwarded start gets the leader's answer instead of an RPC timeout.
- Name each bounded wait (rebalance metadata refresh or decommission
activation, and which gate), and route every start-gate rebalance load
through one load_rebalance_meta_within.
- Move the admin pre-check error mapping into a function and test it.
- Add proof tests: a slow preflight followed by a brief gate wait still
starts; the activation wait and the admin pre-check are bounded and
named; the rebalance status read does not wait for an in-flight
rebalance entry.
- Correct the operator doc: preflight bucket heals are outside the budget,
and forwarded starts report a generic internal error.
Refs rustfs/backlog#2873
The data-movement operation gate is a write-preferring RwLock. A scanner
publication lease stores a read guard on the leasing node, and the
coordinator's lease-scoped rename, delete and validation on that node
queued a second read. A movement writer queued in between (a decommission
start, a status refresh, a peer reload) then waited for the lease while
the coordinator's second read waited for the writer, so both stalled
until the 60 s lease TTL. Admin status reads also took the movement
writer, so every status call waited for the slowest in-flight copy or
cleanup and stalled new movement reads while it queued.
- Lease-scoped target admission shares the read admission the lease
already holds instead of queueing a new read.
- Pool and rebalance status reads publish under the gates only when they
are free; otherwise they merge the persisted metadata into a private
copy and publish nothing.
- A decommission start bounds its start, activation and movement gate
waits with one 30 s budget and fails with a retryable lock timeout
before persisting anything; the admin handler maps it to 503 with
Retry-After.
Refs rustfs/backlog#2873
A restarted node published FullReady from its own view while survivors still counted its drives unavailable (stale DNS for the old pod IP, then the Offline->Returning->Online hysteresis). A rolling update could then stop the next pod and cost a set its read quorum (#8669).
Add a read-only PeerDriveAdmission internode RPC that reports how a peer sees another host's drives, and gate the startup FullReady transition until a strict majority of each erasure set's peers count this node's drives online. The gate shares the startup budget, releases after 15s of unreachable peers, skips immediately when older peers cannot answer, and can be disabled with RUSTFS_STARTUP_PEER_READMISSION_ENABLE=false.
Hard quota admission is owned by the durable reservation ledger (decision
D3, rustfs/backlog#2872), but the request-path quota check still read
scanner usage on every write and mapped `UsageUnavailable` to a retryable
503 even for buckets the ledger admits. For a durable quota the checker now
treats scanner usage as reporting input: an unavailable lookup yields
`current_usage = None` and `allowed = true`, counted and logged at debug
level, and admin quota stats report the usage as null instead of zero.
Buckets without a reservation protocol keep the fail-closed snapshot path.
Quotas saved before the reservation protocols are re-saved under protocol
v2 with the same limit, type and created_at. The write goes through a new
guarded save that keeps the admin save's lifecycle/transaction guards,
incarnation check and fleet capability proof and additionally refuses when
the quota revision moved since the cached read, so a concurrent quota edit
wins. It is triggered lazily from the quota read path (background, one
attempt per bucket with a retry interval, never blocking the read) and in
bulk by `POST /rustfs/admin/v3/quota-migration`, which reports every bucket.
The ADR's Consequences section inventories the remaining scanner-usage
"authority" consumers with their meaning after D3, retirement condition and
compatibility window; the operations guide documents the migration and the
report-only usage semantics.
Refs rustfs/backlog#2904, rustfs/backlog#2872, rustfs/rustfs#8678,
rustfs/rustfs#5716, rustfs/rustfs#7350
Co-authored-by: Hauser <housemecn@gmail.com>
* test(scanner): pin Normal mode when no bitrot start is recorded
Regression for rustfs/backlog#2905: three consecutive cycles that find no
recorded bitrot start currently run Deep through the selection window, and a
Normal cycle never records the cadence start.
* fix(scanner): anchor the bitrot cadence and budget per-disk I/O
Decide the scan mode from a recorded Deep window start only. A cycle that
finds no recorded start runs Normal and anchors the bitrot cadence in
process memory instead of opening the selection window; the Deep cycle that
follows the elapsed cadence records the window start exactly as before. A
missing record used to be read as a window that started at cycle 0, which
ran Deep for deep_window_cycles cycles whenever the start could not be read
back (rustfs/rustfs#8015). The recorded start is also remembered across a
skipped or failed save, and /v3/scanner/status reports the cadence state
under a new bitrot object.
Add a per-local-disk background I/O budget (operations and bytes per second,
defaults by detected media, auto|off|<rate>) that DynamicSleeper charges
before directory listings and object metadata reads as a hard ceiling
layered on the foreground-priority disk admission gate, with runtime config
keys, per-disk status, cycle metrics, pacing pressure, usage_freshness,
Prometheus series, Helm values, admin config help, and operator docs.
Refs rustfs/backlog#2905, rustfs/rustfs#8015, rustfs/rustfs#8377.
* test(scanner): expect Normal retries on erasure sets without a bitrot start
The pending-PUT retry fixture asserted that a 4-drive set revisits every
bucket because the cycle ran Deep. Without a recorded Deep window the cycle
now runs Normal on every layout and reuses the cold inventory.
Count only post-format rebuild failures against the replacement retry
budget; handoffs, format fences, outages, grace waits and conflicts keep
their durable reason without spending an attempt. Retired generations
(committed handoff or approved legacy migration) no longer own their set,
so fresh admission for other slots proceeds, and a committed predecessor
becomes collectible once its successor is verified.
Block discovery conflicts per generation instead of per set, re-evaluate
them on every pass, and expose the blocked entries with reasons. Report
root heal inventory conflicts per record and keep replaying the others.
Revalidate blocked checkpoints hourly with the current reader and clear
markers the reader accepts. Return a dedicated error for ambiguous legacy
delete residue instead of the retryable SlowDown.
Implements rustfs/backlog#2902. Fixesrustfs/rustfs#8664. Addresses
rustfs/rustfs#8366, rustfs/rustfs#8565, rustfs/rustfs#8645 and part of
rustfs/rustfs#8608.
Co-authored-by: Hauser <housemecn@gmail.com>
* fix(scanner): give startup gates automatic exits instead of a pause
Every fail-closed scanner gate now has a bounded automatic exit without
deleting quarantine evidence or relaxing the leader, CAS, movement or
usage-floor fences (rustfs/backlog#2901).
- Recovery retries back off instead of pausing: after the short budget the
observation reads `backoff`, stays retryable, names the next attempt and is
capped at ten minutes; one success resets the count and a 24 h window
restarts the budget (rustfs/rustfs#8297, #8670). The usage-floor failure no
longer carries the retry count across attempts.
- A corrupt `.bloomcycle.bin` is rebuilt on the iteration after its quarantine
marker is durable: the fenced leadership claim CAS-replaces the quarantined
revision under a higher epoch and the marker stays as evidence.
`future_schema` and `cleanup-pending` remain blocked.
- An unclaimable pause ledger runs the leader degraded: cycles run untracked
behind the data-movement fence, `persistence_state` reports `degraded`, and
the claim is retried at most every five minutes (rustfs/rustfs#8251).
- The usage-state full-rebuild reset starts from the empty baseline when every
usage slot is corrupt and no cycle state exists (rustfs/rustfs#8670, #6951).
- A cycle-state save reconciles non-conflict failures against durable state
and retries under the same epoch; an exhausted save reloads durable state in
place. Only a newer owner or an unreadable primary ends the iteration, so
peers see a changed fence only when ownership changed (rustfs/rustfs#8141).
* test(scanner): align usage-state reset tests with the empty-baseline rebuild
The full-rebuild reset no longer refuses corrupt or identity-less usage slots
when no cycle state exists (rustfs/rustfs#8670). The two tests that encoded
that refusal now check the new contract: the reset starts from the empty
baseline with a bootstrap-pending marker, never an authoritative zero floor,
and a valid backup still supplies the recovery floor.
---------
Co-authored-by: cxymds <cxymds@gmail.com>
* feat(oidc): make the Console session duration configurable
The OIDC callback hard-coded the Console credential lifetime at 3600
seconds, so a Console session always ended after about an hour.
RUSTFS_CONSOLE_AUTH_TIMEOUT, the documented knob next to this behaviour,
is the Console HTTP request timeout and cannot extend a session.
Resolve the lifetime from the process-level
RUSTFS_CONSOLE_OIDC_SESSION_DURATION instead and forward it to
complete_authorization_code(). Unset, unparseable, zero, and negative
values keep DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS (3600), so
existing deployments keep the previous behaviour and a misconfigured
variable can never issue an already expired credential. Values above
CONSOLE_OIDC_SESSION_DURATION_MAX_SECS (7 days) are clamped to that
maximum with a warning, which also caps how far an operator can stretch a
credential by accident.
Keep the Console logout token for that same lifetime. OidcStateStore
pinned the logout cache to a hard-coded hour, so a session configured to
last longer than that dropped its logout token while the credential was
still valid: RP-initiated logout could no longer resolve, and the browser
was sent back to the Console login page with the IdP session still
active. The logout cache now derives its TTL from the same
console_session_duration_secs(), which applies the same clamp, and the
callback reads its lifetime from that one function instead of a local
copy, so the credential lifetime and the logout-token retention cannot
drift apart. Add tests covering the default hour, an extended session,
the maximum, and out-of-range values.
The variable is deliberately not part of ENV_IDENTITY_OPENID_KEYS: that
list is suffix-scanned per provider, and this setting applies to every
provider. It has no identity_openid KVS equivalent and needs a restart,
like the other process-level settings in the same document.
Document the setting in the login flow, the process-level table, and the
troubleshooting table, and clarify that RUSTFS_CONSOLE_AUTH_TIMEOUT is a
request timeout rather than a session lifetime.
* test(ecstore): retain offline marker fixtures across reconnect
Keep simulated drive handles attached and reject I/O until explicit rejoin.
Attempt reconnect in the current-marker case so the former CI race is
deterministic while retaining physical cleanup and historical-data checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(oidc): align session diagnostics and lifetime regression tests
Use structured IAM diagnostics for clamping and move configuration tests
to their owning crate. Cover duration boundaries, logout retention and
signed credential expiration through the federation binding.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Hauser