mirror of
https://github.com/rustfs/rustfs.git
synced 2026-10-27 06:18:43 +00:00
Chris
5313b72baa
* fix(decommission): run the leader-side start to completion A forwarded decommission start awaits `ECStore::decommission` inline in the RPC handler. When the forwarding node's RPC times out, the stream is reset and the handler future is dropped wherever it is. Dropped after `pool.bin` records the pool as active but before the peers reload and the worker is spawned, the start left the pool active with no worker and peers on the old metadata, and a retry was rejected as already active. An admin client that disconnects can do the same. Run the start in its own task and let the caller await only its result. The start then always reaches its end: it fails before persisting anything, or persists the pool, has the peers reload and spawns the local workers, rolling back if the reload fails. A retry still cannot start twice: it waits on the start gate held by the running start and either reports the busy gate or is rejected because the pool is active. `decommission` now takes `self: &Arc<Self>` and spawns workers on that store instead of looking up the global object store, which is the same store in production and makes the path testable; the unused `require_decommission_store` helper is removed. Refs rustfs/backlog#2873 * fix(decommission): detach the leader-side start from its caller9ac4c10decarried only the reproduction (the testable entry point, the peer-reload pause point and the proof test). This adds the fix it describes: `ECStore::decommission` spawns the start as its own task and awaits only its result, so a caller that stops waiting, such as a forwarded start whose internode RPC timed out, can no longer drop the start between persisting the pool as active and having the peers reload and the local worker start. Also prove a retry while the first start runs cannot start the pool twice, update the operator and architecture docs for the new behavior, and shrink the error(format!) ratchet baseline for the removed helper. Refs rustfs/backlog#2873 * fix(decommission): single-flight start, recheck pool state under gate Review of3a5276c5b: - A start re-checks the pool state under the start gate before it cancels routines, so a start racing another one can no longer cancel the worker the other start just spawned. - Only one start runs per instance at a time: another request, including the retry of an operator whose request timed out, is rejected as already running instead of stacking a second detached start and its preflight heals. The start runs through `run_detached_mutation`, and a failure nobody waits for any more is logged. - Tests: the proof test pins its drop after the save and requires a worker that persists a cancellation; the retry test really drops the first caller; new tests cover the under-gate recheck and a peer reload failure after the caller is gone. That last one shows the start rollback cannot clear a fresh start because the pool metadata stale-write guard refuses it; the gap is asserted and documented and tracked in rustfs/backlog#2873. - The start tests run on their own large-stack thread: the io_uring lane (`uring_`, which also selects `during_`) runs them with libtest's 2 MiB default stack, where the proof test overflowed. - Docs: single-flight starts, status and cancel during preflight, the rollback gap and its recovery, and the release requirement. Refs rustfs/backlog#2873 * fix(decommission): turn away local start retries at the start slot Second review of29253430a: - The leader-local admin pre-check refreshed rebalance metadata under the start gate before the start slot was checked, so a local retry waited out its 20 s gate budget behind a running start (which holds the gate through its peer reload) and then got a 503. The pre-check now checks the slot first and returns `DecommissionAlreadyRunning`, which the handler reports as a 400 that keeps the reason. - Prove the reload-failure path really ran in the detached start (the injection is consumed), add a test that a fresh start's failed rollback reports both the reload error and the stale-write refusal, and assert an abandoned start stays tracked for shutdown. Refs rustfs/backlog#2873