* perf(s3): borrow options during delete preflight
Pass the existing request options by reference to the metadata preflight instead of cloning them solely for that borrow. Keep the getter's mutable snapshot copy and the later owned-delete copy.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): fence backlog writes against stale pool membership
Reject native scanner publications unless their captured writable pool cohort
matches the fenced durable topology. A smaller startup view must not replace
an expanded membership commit with its older stable rollback.
Keep single-replica test injection fenced by the complete current cohort and
cover rejected startup claims, unchanged CAS revisions, and recovery of the
acknowledged expanded commit.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): use current cohort for healthy scanner CAS writes
Keep the writer's cached topology stale while explicitly capturing the
surviving membership for healthy-target publication and CAS checks. Also
assert that the old cohort rejects healthy-target writes without publishing.
Preserve retiring-source rejection, payload and ETag checks, and canceled
source membership repair without changing production admission.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Record a bounded failure boundary for each non-successful logical remote
walk while preserving the existing outcome, error source and retry behavior.
Distinguish stream open/read, downstream write/flush, wrapper admission and
deadline failures, and unfinished operations dropped before completion.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Include single-pool, status, and expansion fixtures in the existing
native scanner reservation without changing budgets or assertions.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Fence native backlog writes against their captured pool membership and retain
queued replicas as read-only proof members during retirement. Stabilize the
topology test workload and bound pre-ACK recovery separately.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* chore(deps): upgrade workspace parking_lot to 0.13
Move all six direct workspace consumers to parking_lot 0.13 while
retaining the 0.12 dependency line required by upstream crates.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(connect): stabilize client failure classification
Use an ephemeral loopback listener that explicitly rejects proxy CONNECT
and direct TLS connection establishment instead of assuming fixed ports
will promptly refuse connections. Give classification its own bounded
transport and measurement budget while retaining the timeout tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Share the existing canonical identity resolver with a non-authoritative source eligibility query. Group borrowed intents and create owned anchors only for retired generations, preserving cache and responsibility contracts and moving proof keys.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): restore marker write quorum on decommission resume
Verify equivalent target markers at write quorum and republish partial
copies before completion. Preserve exact marker creation semantics and
legacy birth identities while retaining ordinary delete and purge guards.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(odm): run delete marker regression on a bounded stack
Construct and run the deep real-storage future on the existing large-stack
test worker so the default libtest stack does not overflow. Preserve the
registered test name, serial guard, and single-part/multipart assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): use marker majority for resumed copies
Confirm existing and republished delete markers at the configured-set
majority used by their delete transaction instead of the EC data quorum.
For a twelve-drive EC4 set this accepts seven equivalent markers while
still rejecting six, including when the online inventory is shortened.
Cover named/null markers on a real rebalance retry with retained source
history. The regression fails with InsufficientReadQuorum before the fix.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): keep catch-up fixture timestamps consistent
Admit the seeded catch-up using its observation timestamp so a durable
writer claim crossing a wall-clock second cannot spuriously rate-limit it.
Model an older claim timestamp to exercise that boundary deterministically.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Use a conservative work hint for verified and typed legacy events, retaining the original capacity-maintenance pass and metrics publication. Preserve durable identities, journal retention, retries, and shutdown behavior, with isolated real-runtime and queue lifecycle regressions.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore,filemeta): batch local walk_dir metadata reads and buffer metacache streams
* fix(ecstore): stream completed listing metadata reads
Publish each result before starting the next blocking read, preserving
per-object progress and cancellation without a timed chunk queue.
Keep dispatch batching, byte limits, and namespace/error stops intact.
Cover multiple batches, page boundaries, and oversized-envelope resumption.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore,filemeta): verify production listing streams
Replace the simulated directory-walk benchmark with the public local disk
path. Validate fixture output before timing and retain metacache-only
stream benchmarks. Pin fragmented reads, read coalescing, and canonical
UUID namespace probing against the reference behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): route listing benchmark through storage API
Centralize local listing benchmark imports in the shared benchmark
storage API boundary required by the architecture migration rules.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): keep catch-up fixture timestamps consistent
Admit the seeded catch-up using its observation timestamp so a durable
writer claim crossing a wall-clock second cannot spuriously rate-limit it.
Model an older claim timestamp to exercise that boundary deterministically.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* chore(deps): refresh GCS, profiling and allocator dependencies
Update published GCS, Pyroscope, mimalloc, and PPMd crate versions.
Align the pinned erasure-fixture test recursion limit with ECStore
to compile GCS-enabled test targets.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* chore(deps): update AWS SDKs, Tokio and memmap2
Refresh compatible AWS service and Smithy dependencies, Tokio timers,
and memory mapping support with the regenerated lockfile.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): identify scanner retirement planner failure phases
Bind the existing native planner error context to each retirement phase while preserving typed retry classification and all storage fences. Exercise the actual wrapper in an isolated process across unavailable and typed failures.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): type the unavailable retirement planner error
Replace the new formatted other error with a private typed error so the error-construction ratchet remains intact. Preserve all phase messages and the existing native retry classification.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): cover unavailable planner error identity
Check the private unavailable cause for every phase through the existing IO boundary. Retain native handoff classification and verify same-phase quorum bucketing after fallible disk narrowing.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(obs): cover contended identity under netlink denial
Check that the production identity entry stays nonblocking while a name
writer is held and real netlink discovery is denied, then observes the
published name after release. Release the writer before joining so a
blocking regression reports a bounded failure.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(obs): keep identity contention behind common helpers
Move the held-writer fixture behind a test-only common helper and enable
it through the Linux x86_64 obs dev-dependency. Preserve the timeout,
writer release before reader join, and identity restoration ordering.
Exercise callback completion and unwind cleanup in the common regression.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Record the first dispatch or undispatched termination at the existing
external request boundary, with entry-gated timing and owned task scope.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* perf(heal): reuse the verified repair proof queue
Retain proof events in place under the existing registry lock and keep
the pre-consumption capacity target and exact identity matching.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(common): move the final proof mismatch fixture
Avoid a redundant clone in the last owned mismatch fixture.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* perf(filemeta,ecstore): avoid redundant decodes and clones on xl.meta write and delete paths
* perf(filemeta): remove restore parsing allocations and add guards
Borrow restore header fields instead of allocating temporary vectors and
reuse the canonical dual-prefix metadata key guard. Cover scanner/full
decoder equivalence on historical and mutated records, restore grammar,
and the complete old-current observation tuple. Add the missing Criterion
entry for unshared data-directory selection.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* bench(filemeta): isolate metadata mutation and batch deletion costs
Move the metadata-only Criterion target to filemeta so it does not build
ECStore. Add deterministic physical and inline dense/sparse deletion
cases with the same fixtures available to baseline and candidate builds.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(filemeta): index ordinary metadata deletion batches
Use a batch-local version lookup, compact directory summaries, and lazy
physical reference counts for larger ordinary V2 deletion batches. Decode
each indexed body at most once and compact the version vector once.
Flush pending removals before canonical fallback and preserve per-request
validation, partial failures, inline-key aliases, and cleanup decisions.
Keep small batches and transition/legacy/ambiguous records on the scalar
path. Cover indexed/scalar state equivalence and fail-closed cache errors.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(filemeta): enforce linear batch decoding and safe fallbacks
Count V2 scans in tests to guard the indexed path's linear work contract.
Exercise marker, retention, restore, transition, free-version, and pending
purge requests against the canonical sequential path. Compare logical
state and deterministic headers for re-encoded maps while retaining exact
byte equality for ordinary indexed deletion and opaque metadata.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Capture existing diagnostic operation, caller and first-entry context at bucket, background and tracked-mutation handoffs. Keep absent context absent and retain deferred work and admission ownership through the established storage facades.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Keep the original storage results and all required request/replication
ownership while dropping two bindings with no consumers.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Consume completed aggregation entries at their terminal conversion so
bucket and replication target keys can move into the output maps.
Preserve validation, rejection and publication behavior.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(decommission): finish multipart null versions superseded on target
A decommission entry for an unversioned multipart source whose key
already holds a newer null version on the target settles as superseded
and aborts its staged upload. The cleanup's target proof saw the newer
object, could not match it to the staged identity, and failed closed
with a capacity-blocked error, so the routine paused, the source was
never cleaned up, and the error read like a minimum-free-drive block.
The staged upload is already proven absent and the target lookup is
exact: a different version id, or a null target with another
modification time (one null slot per key), is a different object, so
the staged copy holds nothing on the target and the pending intent is
cleared. Only a versioned target with the staged version id but another
modification time stays fail-closed. The remaining identity-proof
failures use a typed error without capacity wording.
Add three-pool decommission regressions for restored-transitioned and
plain multipart null sources superseded on the target.
* test(decommission): pin multipart target proof failures as terminal
Document the accepted residual of the null-slot proof: a metadata-only
self-copy keeps a published but uncharged staged copy's data while
moving its modification time, so its marker is released and one
object's bytes stay uncharged.
Add a three-pool routine test where the target already holds an
unowned null object with the source's exact identity: the cleanup's
ownership-proof failure fails the decommission with no capacity-blocked
reason instead of pausing it.
Require every security candidate to name the lower-trust principal, the crossed boundary, the affected resource, and the observable result, and to land under exactly one of `confirmed`, `needs_validation`, or `rejected`. Only confirmed findings carry severity; a trace blocked on an unresolved deployment fact is reported as that fact instead of an assumption.
A versioned multipart completion minted a new version when the staged version id was absent or nil, so an inbound replica of a source null version written by multipart landed under a fresh UUID. Keep the null version for authorized inbound replicas, and address the null version as versionId=null on the worker convergence HEAD like the other target reads.
Refs rustfs/rustfs#8747
* fix(decommission): move the free version of an expired tier version
Decommission only queued the lifecycle expiry of a transitioned version, so a source cleanup that ran first deleted the version itself and appended its tier free version after the entry's last pass. The post-check then failed the decommission until an operator re-ran it.
Apply the expiry of a transitioned version synchronously, as rebalance does. Its free version is on the source before the cleanup preflight, which reports the entry changed, and the retried entry moves the record to a target in the same pass. An expiry that cannot be applied leaves the version live, so it moves like any other version.
Refs rustfs/backlog#2873
* fix(decommission): move a delete-all expired tier version live
Applying a delete-all expiry inline made its prefix delete wait for the bucket lifecycle write lock behind the entry's own read lock until the lock timeout, blocking every write in the bucket meanwhile. Move such a transitioned version live and leave the delete-all to lifecycle on the target.
Cover a genuinely unapplied expiry and a free-version worker that consumes the source record before the entry moves it.
Refs rustfs/backlog#2873
* fix(heal): explain unverified legacy heal outcomes
Preserve storage diagnostics in canonical unknown outcomes and explain
legacy payload identity limits through the shared local/peer status path.
Borrow expected receipt identities while preserving all proof checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): keep diagnostics from pausing merged MRF repairs
Require the actual MRF executor before publishing legacy pause events.
An ordinary queued owner can retain a merged durable MRF notice without
having performed the specialized deep check. Preserve its diagnostic
without granting it authority to park the repair responsibility.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): type MRF outcomes and defer status diagnostics
Carry storage-owned legacy observations in process and return a typed
terminal result only after the qualified MRF Deep check. Publish pause
events from that result and exact identities instead of source or text.
Keep object names out of retry classification and compute only the
status diagnostic selected by the canonical execution state.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Capture an existing ReadVersion caller and earliest entry before spawning config transactions. Restore them with the existing scoped wrapper while retaining default-feature operation propagation and detached task ownership.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(s3): classify empty multipart completion as InvalidPart
Treat absent per-drive multipart transaction directories as empty listings so
completion reaches the existing InvalidPart validation. Preserve quorum and
transaction recovery, and cover the HTTP and degraded-drive boundaries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): move multipart completion metadata in one pass
Reuse quorum-validated part metadata instead of repeatedly inserting, sorting,
finding and cloning parts. Keep legacy cleanup identities and ordering while
moving the original part allocation into the completion manifest. Compare ETags
by borrowing their normalized values.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(metrics): expose usage candidate rejection outcomes
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): borrow usage candidate fixtures
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): reserve capacity for decommission probes
Reserve all nextest slots for the two eight-disk decommission fixtures under both default and CI profiles. Keep their internal concurrency, production lock budgets, assertions and zero-retry policy intact.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): scope prewrite capacity probes to the target pool
After persisting a new capacity intent, refresh only the target pool baseline. Preserve the global admission and finalization scans, lock order, fencing, and accounting formulas. Verify the reduced probe count with real disk metrics and preserve injected snapshot/error behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Capture the caller operation before the tracked mutation task is spawned. Keep detached completion, owned guards, result errors and panic join failures intact.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): add scanner/heal topology protection matrix
Add the S00 protection baseline for the Scanner/Heal program
(rustfs/backlog#2873, program rustfs/backlog#2872): seven
scanner_heal_topology_ cases in the distributed e2e lane that pin pool
expansion, rebalance and decommission behavior before the runtime changes.
The fixture seeds a deterministic inventory (empty, inline, external,
multipart, special and nested keys; a versioned bucket with an explicit
null version, three history versions and a delete marker per key; an
Object Lock bucket with legal hold and COMPLIANCE retention), records
every version with its requested and listed version id, length and
SHA-256, verifies the manifest through every node, reads physical
placement from xl.meta on the pool drives, and reads the strict
durability mode back from the admin API and the node logs.
Fault cases hold the mover at proven windows through file barriers that
compile only with the e2e-test-hooks feature: before the multipart target
commit and before source cleanup (shared by decommission and rebalance),
plus the existing pool-metadata before_publish cutpoint. An unreached
window fails the case. DistCluster gains capture_logs_under so a case can
read node startup events without a global environment variable.
docs/testing/scanner-heal-topology-gates.md maps the gates to the tests
and records the timeout basis.
* test(e2e): keep the barrier failure message static
The error(format!) ratchet rejects new formatted io::Error::other call
sites in ecstore. The reached observation already records the stage and
object, so the injected failure needs no per-call detail.
* test(e2e): require e2e-test-hooks for topology fault cases
The four scanner_heal_topology_ fault cases inject at server barriers that
exist only with the e2e-test-hooks feature. Against a default server they
waited for the barrier timeout and reported it as not reached. Check the
verified binary's features first so the missing prerequisite fails
immediately and is named.
* ci(e2e): build the distributed lane with e2e-test-hooks
Every other e2e lane builds and runs the fault-test hooks server. The
distributed lane alone used the default server, so the topology fault
cases could never reach their barriers there. Build and run with the same
feature so the receipt matches, and update the documented local commands.
The upgrade cases spawn the pinned previous release directly and do not
compare feature sets.
* test(e2e): keep topology artifacts under RUSTFS_E2E_LOG_DIR
The topology fixture captured node logs, barrier observations and evidence
in a temp dir removed on drop, so the distributed workflow uploaded nothing
for these cases. Fall back to RUSTFS_E2E_LOG_DIR, which the workflow
uploads, when no explicit evidence directory is set.
* test(e2e): remove retired pools before restart; race the pre-ACK start
The queue case restarted with decommissioned pools 0 and 1 still on the
command line, which the server rightly refuses. Add
remove_single_node_pools and restart without them, then expect the
remaining pools renumbered to 0 and 1.
The pre-ACK case now races the start request against the barrier. A start
that returns first, or stays in flight past the documented 60 s budget, is
reported as such instead of as an unreached barrier, and the evidence
records the reach latency.
capture_logs_under now overrides a log path assigned from
RUSTFS_E2E_LOG_DIR, so the seed node logs into the case directory too.
* test(e2e): bound fault-window waits by the movement budget
On Linux CI a busy target capacity gate delayed the first fault object by
about two minutes, past the 120 s wait borrowed from the server's release
poll. Wait for a movement window within the 240 s movement budget instead,
and fail at once when the decommission or rebalance ends without entering
the window, so an uninjected fault is never counted.
* test(e2e): bound status probes while waiting for a fault window
The admin decommission status request can wait on a mover held at a test
barrier until the barrier's own 120 s release timeout. The window wait
issued that request each loop, so a reached barrier went unnoticed until
the server timed it out and failed the decommission, which surfaced as a
false resume failure. Bound each status probe to 5 s and treat a slow probe
as still running.
* test(e2e): retry a busy decommission start
A decommission start that meets a busy movement gate can answer a
retryable 503 instead of waiting. The harness treated any non-proof
failure as fatal. Share one start helper that retries 503 within the
case's movement budget and the fleet-proof 500 for 45 s, and keeps every
other status fatal. The pre-ACK case retries 503 within its window budget.
ci(pool): finalize decommission removal and verify surviving pools (rustfs#8726)
Select suite step 10 together with step 9 and pass
--finalize-decommission so the nightly exercises the post-removal
survivor path: pool ids must stay unique, bucket metadata writes must
keep succeeding across a restart, and no node may log 'pool metadata
layout changed'. Step 10 self-skips when decommission is disabled.
Co-authored-by: Hauser <housemecn@gmail.com>
Track active walks, scratch capacity and accepted bytes while preserving the existing 1 MiB stream copy behavior.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(decommission): defer source cleanup quorum failures
A quorum failure while deleting the source of an already copied entry
was fatal and marked the whole pool failed, although the copied source
is still in place and an uncommitted delete transaction is settled by
delete recovery. Carry the cleanup error typed and resolve a quorum
failure as Deferred: the entry is replayed at the end of the round, and
a deferral that persists takes the existing capacity-blocked pause path
instead of a terminal failure. Other cleanup errors and quorum errors
from other stages stay terminal.
Add set-level regressions for parent and child keys (`a/b`, `a/b/c`):
parent cleanup leaves every child file untouched, and parent cleanup
reaches quorum while the child is cleaned up or rewritten concurrently.
* test(ecstore): keep source cleanup errors typed in prefix cleanup tests
Return SourceCleanupError from the test helper instead of formatting it
into Error::other, and shrink the error(format!) ratchet baseline for
core/pools.rs, which lost one call site in the cleanup failure change.
* fix(decommission): retry cleanup quorum failures and count replays once
Retry a source cleanup quorum failure in place, up to three attempts
with the existing source-cleanup backoff, before deferring the entry.
Carry the versions an entry already counted on its deferral, so the
end-of-round replay does not add a fully counted entry to
items_decommissioned and bytes_done again.
A deferral that persists pauses the pool through the existing deferred
drain path. When source cleanup caused it, persist a reason led by
"source cleanup blocked" without the carrier error's free-drive wording,
and report waitingReason "source_cleanup" instead of "capacity".
Add a source_cleanup test fault stage and a routine test that drives a
cleanup quorum fault through deferral and replay to completion.
* fix(decommission): wait seconds between cleanup quorum retries
Give the in-place source cleanup quorum retry a fixed floor of one and
then two seconds, with the existing jitter on top, so it can ride out a
drive reconnect or health-check blip instead of retrying within a few
hundred milliseconds. The wait stays bounded and cancel-safe.
Find a source cleanup pause reason through stage wrappers, as the
capacity-blocked check does, and add a routine test where cleanup keeps
losing quorum: the decommission pauses with a source cleanup reason, the
bucket stays queued and the source stays in place.
Document the additive waitingReason value source_cleanup and its
compatibility with older nodes reading the same pool.bin.
* fix(decommission): never retry a lost source cleanup fence
The cleanup retry classifier used is_quorum_error(), which also matches
NamespaceLockQuorumUnavailable. The delete commit fence reports a lost
namespace lock or mutation fence with that variant, so a lost fence was
retried in place and only stopped later at the pre-cleanup fence check.
A lost fence means this worker no longer owns the key; it must stop
without retrying. Retry and defer only drive quorum failures:
ErasureReadQuorum, ErasureWriteQuorum, InsufficientReadQuorum and
InsufficientWriteQuorum.
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Cover actual journal and lifecycle checkpoint replay when an unknown historical source coexists with an exact source-bound repair. Reject a pre-restart lease and retain the complete unresolved record after the new repair is discharged and replayed again.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(decommission): let a failed start roll back its own activation
A fresh decommission start whose peer reload failed could not roll back:
the pool metadata stale-write guard from #6604 refuses any save that
clears a persisted active or queued decommission, so the pool stayed
active with no worker until the watchdog or a leader restart picked it
up.
The rollback save now carries the activation that start committed. The
guard exempts a pool only while its persisted record still equals that
activation field for field, so a worker checkpoint or a later start
keeps the rejection. A canceled record is not an active decommission
and is cleared by the rollback as well. Every other save path passes no
activation and is guarded as before.
A peer may already have reloaded the activation and started a worker
before the rollback. Saves are now fenced by decommission generation: a
write carrying a start_time is rejected when the persisted record
belongs to a later start, or has no decommission state and was updated
after that start_time, so such a worker cannot make the pool active
again. Its queued promotion and progress checkpoint also adopt the
persisted record and stop the worker.
Refs rustfs/backlog#2873
* fix(decommission): fence terminal writes of a withdrawn decommission
Terminal records carry no start_time, so the generation fence did not
cover them: after a start rollback, a peer worker of the rolled back
generation that failed (for example on a bucket completion) could still
persist a failed record over the restored pool. The merge now rejects a
failed, canceled or complete record over a persisted record without
decommission state; every attempt persists decommission state first, so
no legitimate path writes a terminal record there.
Peers also drop a withdrawn decommission sooner. A pool metadata reload
replaces a running decommission with a newer record that withdrew it,
even under an active worker; bucket completion adopts such a record like
promotion and progress checkpoints do; and a withdrawn progress
checkpoint stops the worker instead of logging a failed save and moving
more entries.
Tests cover the peer failure mark, a rollback retry that sees a worker
checkpoint land between attempts, and the reload withdrawal. A heal
fixture that wrote complete records over a fresh pool.bin now persists
the active attempt first.
Refs rustfs/backlog#2873
* fix(decommission): fence peer writes after a retry start rollback
A retry start replaces a failed or canceled record that keeps its
unresolved ledger, so its rollback restores that terminal record rather
than clearing the pool. The withdrawal checks only recognized a record
without decommission state, and the generation fence ignored a persisted
record without a start_time. A peer worker of the rolled back retry
could therefore checkpoint the pool back to active.
The generation fence now compares the persisted last_update whenever
the persisted record has no start_time of its own (cleared, terminal or
queued). The progress checkpoint also checks the generation against the
record read under the pool metadata lock, as bucket completion already
does, and the save-path withdrawal check treats a terminal record newer
than the running view as a withdrawal, so the peer adopts it and stops.
A reload still never lets a newer terminal record replace an active
worker's view, so after a retry's rollback a peer worker stops at its
next bucket completion or progress checkpoint rather than on reload.
Tests cover the retry rollback against a peer checkpoint and direct
write, and a peer bucket completion after a fresh start's rollback.
Refs rustfs/backlog#2873
* fix(decommission): keep a retry rollback's restored record intact
After a retry start rolls back, the restored record is failed, so a
peer worker's own failure mark passed every merge guard and replaced
it. A failure mark now saves through save_current_pool_meta_checked,
which runs the withdrawal check against the record read under the pool
metadata lock and adopts that record instead of writing.
The withdrawal check compared the restored record with the worker's
in-memory last_update, which a rejected save had already raised, so the
peer never adopted the rollback and the watchdog kept resuming it. It
now compares against the running generation (start_time), and a
rejected unresolved-entry save undoes its in-memory change.
save_current_pool_meta_for_decommission_start is also compiled for the
test-util feature again; a test-util helper still calls it, which broke
the workspace clippy and Connect builds.
Refs rustfs/backlog#2873
* fix(decommission): keep an unresolved entry when its own save fails
The rollback of a rejected unresolved-entry save also dropped the entry
on transient failures such as a quorum or lock-fence error, so the
following failure mark persisted a failed record without it and an
operator could clear a decommission that skipped an unreadable object.
Keep the entry and undo only the last_update bump, which is all a
withdrawn worker needs to still see the withdrawal.
Refs rustfs/backlog#2873
Under sustained writes nearly every cycle fell back to a full walk: scope
admission required the global dirty generation to stay unchanged across the
bucket listing, and after a Complete cycle the dirty-usage wake ran the next
full walk immediately, back to back.
- Track dirty-map removals with a process-local removal generation and admit
a scoped walk when no entry was removed or replayed during the listing
(`covers_listed_pending`). Writes to listed buckets in the window join the
start snapshot; unlisted buckets dirtied after the cutoff stay pending and
are never cleared by compare-and-remove certification. The strict
`covers_all_pending` still gates segment-reuse evidence.
- Classify scope fallbacks (removal during listing, generation exhausted,
dirty-bucket overflow, no listed dirty bucket, baseline unavailable, peer
scope unavailable), log `scoped_scan_fallback`, and report them in
usage_freshness as additive serde(default) fields.
- Pace the cycle after a Complete fallback while dirty buckets remain on the
capped retry schedule clamped to the cycle interval, with the dirty wake
disabled; the paced timer wake still requests a scope.
- Update the per-bucket convergence design doc.
No persisted format or internode protocol change.
Co-authored-by: Chris <anzhengchao@gmail.com>
Chris