gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-11-01 00:32:21 +00:00
Code Issues Packages Projects Releases Wiki Activity
7,595 Commits 224 Branches 164 Tags
codex/plugin-runner-integration
Commit Graph

9 Commits

Author SHA1 Message Date
Chris 3092792c3c fix(scanner): recover an interleaved pause backlog wedge after expansion (#8867) 
* fix(scanner): roll a wedged pause backlog forward to its proven ledger

For rustfs/backlog#2873 defect E.

Two interleaved writers could leave a replica state that no claim could
leave. Writer B read writer A's partial commit and selected the stable
rollback point. A's commit then completed and A verified it. B's rollback
stabilization overwrote every member it had read after the commit, and A's
stabilization reached only the remaining member. No commit survived and the
stable fields differed, so every claim and every decommission retirement
failed with "neither a surviving membership commit nor a stable rollback
point".

- When no commit survives and stable consensus fails, select the unique
  newest stable ledger by (writer epoch, generation). A stable field is
  written only after its ledger was selected as authority, so that ledger is
  the last proven one. The next claim stabilizes it and advances normally.
- Keep failing closed for an invalid or unreadable replica, for two
  different stable ledgers at the newest key, and for a missing replica that
  a stored commit names as a member.
- A rollback proof written without an authoritative commit names only
  members that already store a record, so a failed first write to a new
  member cannot turn it into a named member that reads as lost.
- Retirement refuses a rolled-forward ledger until the writer has
  stabilized it, as a retryable handoff conflict rather than a terminal
  authority conflict.
- Tests replay the interleaving on real storage: it converges and retires,
  it converges after another expansion with a failed first write to a new
  member, and a same-key fork or a lost named member still refuses.

* fix(scanner): let retirement stabilize a rolled-forward pause backlog

Follow-up to the defect E roll-forward for rustfs/backlog#2873.

- Retirement no longer waits for a scanner writer. When no commit
  survives and the native selector over every read member (survivors,
  retiring source and queued pools) rolls forward, the planner commits and
  stabilizes that ledger on the survivors under the writers' object fence.
  The previous retryable handoff could exhaust its ~13.5 s budget long
  before a writer re-claimed, and failed the decommission terminally.
- A writer's roll-forward refuses when a stored commit names a member that
  is missing or outside its read, such as a retiring source excluded from
  the writable set. Retirement, which reads that member, decides instead.
- A writer stabilizing a rolled-forward ledger writes missing members
  first and stops at the first failure. A member that only read as missing
  rejects the create, so no surviving proof can omit its newer record.
- Docs describe the actual roll-forward preconditions and the retirement
  exit.
- Tests: retirement converges the wedge with no writer; a writer refuses to
  roll forward past an unread source that holds the newest ledger and
  retirement keeps it; a spurious missing read changes no record and the
  next claim converges on the newest ledger.

* fix(decommission): create missing pause backlog survivors first

Follow-up to the defect E retirement roll-forward for rustfs/backlog#2873.

- Retirement creates survivors that read as missing one at a time, before
  any stored survivor, and stops at the first failed create. A transient
  create failure on a new pool's set no longer leaves every other survivor
  holding a commit that names a still-missing member, which no later
  selection could leave.
- A rolled-forward retirement first seeds a rollback proof that names only
  the survivors that store a record; the commit still names every survivor.
- Retirement rolls forward only when the survivors cannot select a ledger
  on their own or select the same one. Survivors that agree on a different
  ledger keep the source-authority check.
- The scanner writer applies the same missing-first, stop-at-first-failure
  order to every stabilization with a missing member, including a seed from
  an old commit selected as the rollback proof.
- Tests: a failed new-pool create during retirement retries to convergence;
  a spurious missing read cannot seed an old commit over a newer member;
  the rolled-forward precedence over source stable records is pinned.

* docs(decommission): state the all-missing survivor limit of backlog roll-forward
 2026-10-24 14:52:30 +09:00
Chris 77a36d92a3 fix(decommission): let a failed start roll back its own activation (#8752) 
* fix(decommission): let a failed start roll back its own activation

A fresh decommission start whose peer reload failed could not roll back:
the pool metadata stale-write guard from #6604 refuses any save that
clears a persisted active or queued decommission, so the pool stayed
active with no worker until the watchdog or a leader restart picked it
up.

The rollback save now carries the activation that start committed. The
guard exempts a pool only while its persisted record still equals that
activation field for field, so a worker checkpoint or a later start
keeps the rejection. A canceled record is not an active decommission
and is cleared by the rollback as well. Every other save path passes no
activation and is guarded as before.

A peer may already have reloaded the activation and started a worker
before the rollback. Saves are now fenced by decommission generation: a
write carrying a start_time is rejected when the persisted record
belongs to a later start, or has no decommission state and was updated
after that start_time, so such a worker cannot make the pool active
again. Its queued promotion and progress checkpoint also adopt the
persisted record and stop the worker.

Refs rustfs/backlog#2873

* fix(decommission): fence terminal writes of a withdrawn decommission

Terminal records carry no start_time, so the generation fence did not
cover them: after a start rollback, a peer worker of the rolled back
generation that failed (for example on a bucket completion) could still
persist a failed record over the restored pool. The merge now rejects a
failed, canceled or complete record over a persisted record without
decommission state; every attempt persists decommission state first, so
no legitimate path writes a terminal record there.

Peers also drop a withdrawn decommission sooner. A pool metadata reload
replaces a running decommission with a newer record that withdrew it,
even under an active worker; bucket completion adopts such a record like
promotion and progress checkpoints do; and a withdrawn progress
checkpoint stops the worker instead of logging a failed save and moving
more entries.

Tests cover the peer failure mark, a rollback retry that sees a worker
checkpoint land between attempts, and the reload withdrawal. A heal
fixture that wrote complete records over a fresh pool.bin now persists
the active attempt first.

Refs rustfs/backlog#2873

* fix(decommission): fence peer writes after a retry start rollback

A retry start replaces a failed or canceled record that keeps its
unresolved ledger, so its rollback restores that terminal record rather
than clearing the pool. The withdrawal checks only recognized a record
without decommission state, and the generation fence ignored a persisted
record without a start_time. A peer worker of the rolled back retry
could therefore checkpoint the pool back to active.

The generation fence now compares the persisted last_update whenever
the persisted record has no start_time of its own (cleared, terminal or
queued). The progress checkpoint also checks the generation against the
record read under the pool metadata lock, as bucket completion already
does, and the save-path withdrawal check treats a terminal record newer
than the running view as a withdrawal, so the peer adopts it and stops.

A reload still never lets a newer terminal record replace an active
worker's view, so after a retry's rollback a peer worker stops at its
next bucket completion or progress checkpoint rather than on reload.

Tests cover the retry rollback against a peer checkpoint and direct
write, and a peer bucket completion after a fresh start's rollback.

Refs rustfs/backlog#2873

* fix(decommission): keep a retry rollback's restored record intact

After a retry start rolls back, the restored record is failed, so a
peer worker's own failure mark passed every merge guard and replaced
it. A failure mark now saves through save_current_pool_meta_checked,
which runs the withdrawal check against the record read under the pool
metadata lock and adopts that record instead of writing.

The withdrawal check compared the restored record with the worker's
in-memory last_update, which a rejected save had already raised, so the
peer never adopted the rollback and the watchdog kept resuming it. It
now compares against the running generation (start_time), and a
rejected unresolved-entry save undoes its in-memory change.

save_current_pool_meta_for_decommission_start is also compiled for the
test-util feature again; a test-util helper still calls it, which broke
the workspace clippy and Connect builds.

Refs rustfs/backlog#2873

* fix(decommission): keep an unresolved entry when its own save fails

The rollback of a rejected unresolved-entry save also dropped the entry
on transient failures such as a quorum or lock-fence error, so the
following failure mark persisted a failed record without it and an
operator could clear a decommission that skipped an unreadable object.
Keep the entry and undo only the last_update bump, which is all a
withdrawn worker needs to still see the withdrawal.

Refs rustfs/backlog#2873
 2026-10-21 15:45:48 +09:00
Chris 5313b72baa fix(decommission): run the leader-side start to completion (#8723) 
* fix(decommission): run the leader-side start to completion

A forwarded decommission start awaits `ECStore::decommission` inline in
the RPC handler. When the forwarding node's RPC times out, the stream is
reset and the handler future is dropped wherever it is. Dropped after
`pool.bin` records the pool as active but before the peers reload and
the worker is spawned, the start left the pool active with no worker and
peers on the old metadata, and a retry was rejected as already active.
An admin client that disconnects can do the same.

Run the start in its own task and let the caller await only its result.
The start then always reaches its end: it fails before persisting
anything, or persists the pool, has the peers reload and spawns the
local workers, rolling back if the reload fails. A retry still cannot
start twice: it waits on the start gate held by the running start and
either reports the busy gate or is rejected because the pool is active.

`decommission` now takes `self: &Arc<Self>` and spawns workers on that
store instead of looking up the global object store, which is the same
store in production and makes the path testable; the unused
`require_decommission_store` helper is removed.

Refs rustfs/backlog#2873

* fix(decommission): detach the leader-side start from its caller

9ac4c10de carried only the reproduction (the testable entry point, the
peer-reload pause point and the proof test). This adds the fix it
describes: `ECStore::decommission` spawns the start as its own task and
awaits only its result, so a caller that stops waiting, such as a
forwarded start whose internode RPC timed out, can no longer drop the
start between persisting the pool as active and having the peers reload
and the local worker start.

Also prove a retry while the first start runs cannot start the pool
twice, update the operator and architecture docs for the new behavior,
and shrink the error(format!) ratchet baseline for the removed helper.

Refs rustfs/backlog#2873

* fix(decommission): single-flight start, recheck pool state under gate

Review of 3a5276c5b:

- A start re-checks the pool state under the start gate before it
  cancels routines, so a start racing another one can no longer cancel
  the worker the other start just spawned.
- Only one start runs per instance at a time: another request, including
  the retry of an operator whose request timed out, is rejected as
  already running instead of stacking a second detached start and its
  preflight heals. The start runs through `run_detached_mutation`, and a
  failure nobody waits for any more is logged.
- Tests: the proof test pins its drop after the save and requires a
  worker that persists a cancellation; the retry test really drops the
  first caller; new tests cover the under-gate recheck and a peer reload
  failure after the caller is gone. That last one shows the start
  rollback cannot clear a fresh start because the pool metadata
  stale-write guard refuses it; the gap is asserted and documented and
  tracked in rustfs/backlog#2873.
- The start tests run on their own large-stack thread: the io_uring lane
  (`uring_`, which also selects `during_`) runs them with libtest's 2 MiB
  default stack, where the proof test overflowed.
- Docs: single-flight starts, status and cancel during preflight, the
  rollback gap and its recovery, and the release requirement.

Refs rustfs/backlog#2873

* fix(decommission): turn away local start retries at the start slot

Second review of 29253430a:

- The leader-local admin pre-check refreshed rebalance metadata under
  the start gate before the start slot was checked, so a local retry
  waited out its 20 s gate budget behind a running start (which holds
  the gate through its peer reload) and then got a 503. The pre-check
  now checks the slot first and returns `DecommissionAlreadyRunning`,
  which the handler reports as a 400 that keeps the reason.
- Prove the reload-failure path really ran in the detached start (the
  injection is consumed), add a test that a fresh start's failed
  rollback reports both the reload error and the stale-write refusal,
  and assert an abandoned start stays tracked for shutdown.

Refs rustfs/backlog#2873
 2026-10-20 18:20:13 +09:00
overtrue f320f791a4 test(decommission): prove the shared gate budget and each bounded wait 
Second review of 652c56875..0ef02a485:

- The operator doc and the budget comment claimed a forwarded start that
  outlives the internode RPC timeout still completes on the leader. The
  leader drops it wherever it is, possibly after pool.bin records the
  pool as active but before its worker starts or the peers reload.
  Document that state, the watchdog that resumes the worker, and the
  checks to run; the cancel-safety fix is tracked in rustfs/backlog#2873.
- Prove the budget is shared across waits (paused clock: a 60 ms wait
  leaves 40 ms for the next gate).
- Prove the rebalance-refresh start gate and activation gate waits and
  the decommission() entry are bounded and named.

Refs rustfs/backlog#2873
 2026-10-20 13:29:14 +09:00
overtrue 4d2e362e23 fix(decommission): count only gate waiting in the start budget 
Review of 652c56875 found that the 30 s start budget was a wall-clock
deadline fixed when the start began, so slow preflight work (bucket heals,
capacity scans) could spend it before the activation gates were reached.
A gate held for a few milliseconds then failed the start at once, and a
retry repeated the preflight.

- Replace the deadline with a budget that only counts time spent actually
  waiting for a gate; a free gate is always taken, and the lock timeout
  reports the time waited.
- Lower the budget to 20 s, below the default 30 s internode RPC timeout,
  so a forwarded start gets the leader's answer instead of an RPC timeout.
- Name each bounded wait (rebalance metadata refresh or decommission
  activation, and which gate), and route every start-gate rebalance load
  through one load_rebalance_meta_within.
- Move the admin pre-check error mapping into a function and test it.
- Add proof tests: a slow preflight followed by a brief gate wait still
  starts; the activation wait and the admin pre-check are bounded and
  named; the rebalance status read does not wait for an in-flight
  rebalance entry.
- Correct the operator doc: preflight bucket heals are outside the budget,
  and forwarded starts report a generic internal error.

Refs rustfs/backlog#2873
 2026-10-20 13:29:14 +09:00
overtrue a55f2f671b fix(decommission): keep status reads and starts off the movement gate 
The data-movement operation gate is a write-preferring RwLock. A scanner
publication lease stores a read guard on the leasing node, and the
coordinator's lease-scoped rename, delete and validation on that node
queued a second read. A movement writer queued in between (a decommission
start, a status refresh, a peer reload) then waited for the lease while
the coordinator's second read waited for the writer, so both stalled
until the 60 s lease TTL. Admin status reads also took the movement
writer, so every status call waited for the slowest in-flight copy or
cleanup and stalled new movement reads while it queued.

- Lease-scoped target admission shares the read admission the lease
  already holds instead of queueing a new read.
- Pool and rebalance status reads publish under the gates only when they
  are free; otherwise they merge the persisted metadata into a private
  copy and publish nothing.
- A decommission start bounds its start, activation and movement gate
  waits with one 30 s budget and fails with a retryable lock timeout
  before persisting anything; the admin handler maps it to 503 with
  Retry-After.

Refs rustfs/backlog#2873
 2026-10-20 13:29:14 +09:00
唐小鸭 353e67bbf3 fix(ecstore): resolve Windows drive paths in URL endpoints (#8629) 
On Windows, URL endpoints such as http://host:9000/C:/data{1...4} (the only
form multi-node and pool-expansion layouts accept) produced the disk path
`/C:/data1`: `try_from` stripped the synthetic slash, but `Url::set_path`
put it back, and `get_file_path` only removed it for `file` endpoints.
Windows resolves `/C:/data1` as an invalid `C:` component under the current
drive, so disk validation and `resolve_local_disk_root` failed.

- `get_file_path` now restores `/X:/...` to `X:/...` for every scheme on
  Windows; URL paths without a drive (`/export1`, `/C:foo`) are unchanged.
- UNC shares (`\\nas\share\d1`, `//nas/share/d1`, `\\?\UNC\...`) are rejected
  with an explicit error instead of becoming a file URL whose host is the
  share server and whose path drops the share name.
- A drive root (`http://host:9000/D:/`) is accepted in URL form, matching
  the path-style `D:\` endpoint that was already accepted.
- The "distinct physical disks" error explains on Windows that directories
  on one volume share a device identity, alongside the existing
  RUSTFS_UNSAFE_BYPASS_DISK_CHECK hint.
- The Windows filesystem lane now runs on layout changes and runs the
  layout::endpoint:: tests; new Windows tests carry the windows_ prefix.
- Document Windows endpoint forms, the same-volume check, and the SNSD
  expansion limit in cluster-lifecycle-operations.md.
 2026-10-17 16:06:37 +08:00
Chris 641c4b3493 chore(release): merge 1.0.1 back into main and refresh installation references (#8314) 
* fix(ci): include pagination regression in full E2E selection

* fix(deps): replace yanked yoke-derive release

* fix(scanner): expose pause backlog replica diagnostics (#8258)

* fix(scanner): expose pause backlog replica diagnostics

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(connect): stabilize runtime profile lease cancellation

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(connect): tolerate delayed schedule startup in CI

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): retry quota reads during usage warmup

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): avoid meta-bucket incarnation self-deadlock

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(test): use persisted incarnation in heal fixture

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(usage): reconcile stale counters after lifecycle expiration (#8108)

* fix(usage): reconcile stale counters after lifecycle expiration

* fix(usage): account lifecycle expiry during continuous writes

* test(usage): run lifecycle usage scenarios on one scanner store

* test(usage): use a Windows-representable pre-mutation offset

* fix(usage): harden expiry accounting recovery and quota checks

Borrow expiry receipt bucket names and avoid allocating a map key on cache hits. Cover cancelled receipts, durable snapshot recovery, and legacy quota admission after scanner confirmation. Use representable timestamp offsets in the quota regression.

Refs rustfs/backlog#2689

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(usage): recover stale persisted counts through the scanner

Seed incorrect complete usage for empty and retained-object buckets, then run the real scanner and publication consumer without further object mutations. Verify durable and admin usage over two cycles instead of writing a corrected snapshot in the test.

Refs rustfs/backlog#2689
Refs rustfs/backlog#2691

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): bound delimiter pagination fixture concurrency

The 120-second smoke timeout expired after 1018 of 1200 serial fixture PUTs, before LIST ran. Prepare the same objects with at most eight concurrent requests and await every PUT. Retain the timeout and strengthen exact prefix, KeyCount, empty Contents, and continuation-token assertions with phase diagnostics.

Refs rustfs/backlog#2689

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(upgrade): establish a persisted previous-release baseline

Seed the pinned previous-release cluster and restart it once with its data intact before replacing any node. Require every old writer to pass the strict readiness probe and preserve the seed through both mixed phases and the final current cluster. Keep InternalError fail-fast behavior and all existing compatibility deadlines and assertions.

Refs rustfs/backlog#2689
Refs rustfs/backlog#2384

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): bound cancellation metadata persistence waits

Use the system-bucket incarnation boundary now supplied by main PR #8268. Bound the three cancellation waits that previously hung during pool.bin persistence, retaining their remote-generation, target-cohort, and durable-state assertions.

Refs rustfs/backlog#2697
Refs rustfs/backlog#2689

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(e2e): retain startup and shutdown failure diagnostics

* fix(test): supply CPU workload for sampler regression

* fix(ci): locate security chain scripts in the workspace

* fix(usage): recover historical counters with generation fencing (#8273)

* fix(usage): recover historical counters during continued writes

Use newer converged scanner snapshots to reconcile stale absolute usage
baselines while preserving concurrent mutation and expiry receipt fences.
Cover durable publication, admin and quota reads, and legacy generations.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(usage): fence snapshots and move preserved cache entries

Apply the cached scanner generation floor before every reconciliation path
and retain it even when an older snapshot happens to match core counts.
Move preserved usage entries instead of cloning their histogram maps under
the cache lock, retaining expiry receipt identity and cancellation fences.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): verify checkpoint takeover and repair dispatch (#8275)

* test(scanner): cover checkpoint handoff and repair dispatch

Drive runtime budget expiry, partial-cycle persistence, leadership claims,
and stale checkpoint rejection between real disk-backed fixture scans.
Verify that a metadata repair beyond the first bounded prefix is saved in
the scanner ledger and dispatched by the MRF consumer.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* ci: isolate scanner fixtures and refresh full e2e membership

Reserve nextest capacity for the real-disk scanner publication and MRF
admission fixtures. Bind both platform membership checks to the reviewed
pagination deadline test added on main.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(scanner): consolidate checkpoint fixture lifecycle

Keep one durable control store across timeout and leadership transitions,
and inject generation advancement into the shared checkpoint scenario.
Check the actual saved metadata path so late-write rejection also proves
that existing checkpoint bytes remain intact.

Centralize MRF fixture isolation and reuse nextest process isolation when
the startup environment already satisfies the test contract.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(obs): distinguish allocator counters from live memory (#8274)

* fix(obs): distinguish allocator counters from live memory

Preserve count/counter semantics and mark requested-byte attribution unavailable when live statistics or sampling are missing. Document sustained multipart memory diagnosis.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(obs): parse allocator statistics from one node

Resolve each statistic before interpreting its shape, avoiding unsupported-field tree scans and mixing data across wrapper scopes. Preserve unavailable-statistics policy and add precedence regressions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): isolate late parity recovery from metadata hedges

Use the existing object-scoped hedge timer barrier in exact-count recovery fixtures. Preserve payload and total-read assertions and verify that the omitted parity disk is read only during late refresh.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(usage): combine identical snapshot retention branches

* fix(test): await HTTP sender readiness in Top RPC fixture

* [release/1.0.1] Gate multipart copy through write admission (#8284)

Gate multipart copy through write admission

Make UploadPartCopy acquire the shared foreground write admission permit before lifecycle locks or source readers so server-side multipart copy cannot bypass the same backpressure used by UploadPart. Document the shared queue semantics and add focused coverage for saturation, cancellation, lock ordering, and disabled admission.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* Gate multipart copy through write admission (#8283)

Make UploadPartCopy acquire the shared foreground write admission permit before lifecycle locks or source readers so server-side multipart copy cannot bypass the same backpressure used by UploadPart. Document the shared queue semantics and add focused coverage for saturation, cancellation, lock ordering, and disabled admission.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix: add UploadPart OOM validation guardrails (#8287)

* docs(release): validate candidates on release branch

* fix(scanner): validate checkpoints against global cycle fence (#8278)

## Related Issues

Related to rustfs/backlog#2701.

## Summary of Changes

Route scanner checkpoint cycle and leader validation through the global store while retaining the owning set for cache persistence, CAS revisions and publication admission.

## Verification

Two independent final-diff source reviews found no issues across correctness, concurrency and durability, test coverage, compatibility, performance and simplicity on head `8b8fe51d092090b053f552ae283960e2e306be33`. Root approval `5373624714` is bound to that exact head. Regression tests cover real two-pool routing, stale fences, post-save rejection and CAS conflicts; their reported local execution belongs to the PR author, not this merge operation. Current required CI remains pending, and this authorized admin squash does not establish CI or runtime acceptance.

## Impact

Restores checkpoint progress when global cycle and leader state differ from a set-scoped view. No format, retry, timeout, assertion or scanner-policy changes are introduced by this diff. The three prior main scanner failures remain unproved repaired.

## Additional Notes

Full validation must run on the resulting exact main revision. Reverting this patch restores the earlier set-scoped fence lookup and its checkpoint rejection behavior.

* fix(ci): restore E2E membership and pagination timeouts (#8281)

* ci: locate the auto-testing checkout for lanes that run evidence from a subdirectory (#8279)

## Related Issues

Follow-up to #8229.

## Summary of Changes

Locate the private auto-testing checkout from the lane root or the workspace root so the nested security checkout can record functional-chain evidence.

## Verification

The exact PR head b66129ab9f passed one mechanical correctness and simplicity review, nine real-Git layout and provenance checks, and sixteen existing evidence/envelope tests. The baseline sibling layout failed with git exit 128; the corrected layout succeeded while revision mismatches and missing checkouts stayed rejected. Current PR checks are completed with successful or skipped conclusions, including the aggregate.

## Impact

Both lane and private-script revision checks remain intact. No time limits, assertions, production behavior, or evidence validation requirements change. The synthetic layout checks do not execute the actual scheduled security suite; integrated main CI and release acceptance remain separate gates.

## Additional Notes

Approved review 5374559393 is bound to the exact head above. Reverting the single-file change restores the previous checkout lookup.

* fix: add UploadPart OOM validation guardrails

Add a Docker validation harness for backlog#2704 so the ordinary UploadPart
low-concurrency memory workload can be reproduced with comparable case metadata,
process/cgroup sampling, TLS and metrics toggles, cache-env controls, and write
reclaim/direct-write experiments.

Warn when operators set the unrecognized RUSTFS_OBJECT_CACHE_* variables that
appeared in the reporter compose file. The variables are reported but remain
ignored, so startup does not silently change object data cache behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: AL <allan.bednarowski@gmail.com>
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): pass fence store to checkpoint fixture

* fix(s3): queue bucket operations and restore strict Clippy checks (#8290)

* fix(s3): queue concurrent bucket creation and deletion

Keep eight active bucket transactions and bound admission waiting to 128 requests and 30 seconds. Preserve detached transaction ownership and return Retry-After with overload responses.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): restore strict Clippy compatibility on Rust 1.99

Use try_update without changing atomic ordering or overflow behavior. Keep
recursive storage futures boxed once at each frame and remove the redundant
async-recursion macro, including its non-recursive SQL planner use. Remove
needless closure borrows and orphaned dependency entries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* chore: refresh dependencies and atomic update APIs

Update workspace dependencies and the lockfile. Replace deprecated
atomic fetch_update aliases with try_update while preserving closures
and memory ordering.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* improve

* fix(scanner): diagnose and verify pause backlog recovery (#8293)

* fix(scanner): diagnose and verify pause backlog recovery

Expose the retained replica snapshot and claimed membership in abnormal
admin status responses. Keep diagnostics off metrics updates and verify
single-pool recovery and conflicting-proof preservation across 24 sets.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(scanner): move replica snapshots into diagnostics

Consume the terminal admin read snapshot in a single state match and move
membership, revision, and error buffers into the response. Verify buffer
handoff and the unchanged JSON contract without altering ledger authority.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(heal): wait for held legacy responsibility in replay test

* ci: remove Docker Hub description sync

* fix: emit NextPartNumberMarker only when ListParts is truncated

ListPartsInfo.next_part_number_marker was a non-optional usize that
defaulted to 0 and was only assigned when the response was truncated.
The S3 serializer then emitted it unconditionally as Some(0), causing
AWS SDK paginators to loop infinitely on part_number_marker=0 instead
of terminating.

Change the field to Option<usize> (None by default) and set it only
inside the is_truncated branch. The S3 output layer now uses
.and_then() so NextPartNumberMarker is absent when IsTruncated=false,
matching AWS S3 behavior.

Fixes #8208

(cherry picked from commit 44de803a38)

* fix(s3): honor sparse ListParts markers and verify termination

Resume part listings at the first part above the numeric marker, even
when that marker is absent. Use binary search over the sorted part
numbers and retain the existing exact-tail empty-slice path.

Add storage, XML, and real AWS SDK paginator regressions for empty and
terminal pages, sparse markers, and multipart completion integrity.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit 673031eea1)

* fix(storage): publish delete rollback backups atomically

Stage rollback metadata outside the rollback directory and publish it only after the full write succeeds. A short write must not leave a backup that quorum rollback can rename over acknowledged version history.

Add an isolated real short-write regression and register the backported ListParts SDK test in the smoke and Linux full inventories.

* test(e2e): register paginator regression in Darwin inventory

* fix(release): install yq before Helm template checks

* chore(release): align installation references for 1.0.1

---------

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Peder Bergan <pederbe@users.noreply.github.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: AL <allan.bednarowski@gmail.com>
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: Chapman <touch65536@gmail.com>
 2026-10-03 14:40:52 +08:00
Chris aa7f55946c docs(operations): add cluster and erasure-coding lifecycle runbook (#7998) 
Consolidate the operator entry point for planning, storage classes and
EC:0 consequences, expansion, rebalance, decommission, heal, drive
replacement, restart recovery, and the rc CLI mapping. Every claim was
checked against the current handlers, storage-class validation, quorum
arithmetic, and the rustfs/cli admin reference; the runbook links the
normative contracts instead of restating them.

Refs rustfs/backlog#2607, rustfs/backlog#2608
 2026-09-18 12:50:14 +08:00