* perf(ecstore): avoid head-of-line blocking in delete recovery
Recover independent keys through a bounded rolling window while preserving per-key hint generations, admission budgets, cancellation retry, and the existing transaction decision engine.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): bound durable delete preparation and bootstrap recovery
Prepare independent Linux delete transactions with bounded concurrency while retaining per-key result order and serial fallback for overlapping, rollback, and legacy requests. Preserve startup discovery until bucket metadata is initialized, and add release performance and process-crash acceptance runners.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): verify clean exits in recovery acceptance
Reject forced or OOM exits when measuring clean startup and verify original ETags across mixed-layout crash recovery and a second restart.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): distinguish prepared and decided batch crashes
Observe real preparation and receipt states separately, retain non-vacuous acknowledged-delete sentinels, and require read-only physical convergence before S3 reads can trigger cleanup.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): move final delete recovery options
Move the options value into the batch-ordering regression's final case
instead of cloning it. Keep the earlier clone needed for reuse.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): replay delete receipts after interrupted proof GC
Retire only the exact empty transaction leaf before journal activation. Preserve occupied directories and prefix children, and cover partial cross-disk proof GC without a fresh delete. Tighten crash evidence to distinguish directory-only remnants from retained content.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bound implicit startup delay and publish config early
Limit warm-cache startup to the smaller of the cycle interval and one
minute before jitter, while preserving explicit startup settings.
Publish configuration gauges when applying runtime settings and retain
timeout diagnostics until an actual new cycle begins.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): unify startup waits and skip unchanged configuration
Bound read-only startup inspection and make startup and pre-leadership
retries respond to cancellation and configuration changes. Preserve one
startup deadline and discover replication behind lifecycle-only buckets.
Avoid irrelevant bucket listings and repeated per-disk reconfiguration.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): retain effective startup policy and fallback metrics
Keep the sampled startup deadline when configuration changes leave its
normalized policy unchanged. Initialize retained runtime metrics after
an invalid first refresh without suppressing the validation error or
replacing a newer accepted configuration.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* perf(rpc): remove redundant boxing from read streams
Keep the capped read stream concrete until StreamingBlob owns its type erasure. Add verified four-node TLS coverage for full, unaligned range and concurrent reads, with bounded client memory and explicit integrity checks.
Document protocol comparison commands and the old-node workaround during rolling upgrades.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): preserve TLS trust for alternate credentials
Derive credential-specific clients from the cluster configuration so they retain the generated CA. Exercise successful trusted reads and reject an invalid replacement identity instead of inheriting root credentials.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): collapse acceptance report path guard
Combine the optional report path and existing-file checks into a let chain
without changing the report or PID sidecar collision behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): add MPU RPC quorum and recovery acceptance
Exercise separate storage and lock RPC delays across multipart phases,
including quorum boundaries, cancellation, bucket recreation, and bounded
eight-node load. Honor TMPDIR for isolated cluster drive directories.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(lock): probe healthy voters behind silent peers
Spread additional probes across the existing acquisition attempt budget
when the initial quorum-plus-spare wave stalls. Preserve quorum thresholds,
fast-path fan-out, and existing ownership and late-response cleanup.
Cover every four-node read-voter ordering, a stalled eight-node write
quorum, and normal fan-out. Extend MPU acceptance to EC12+4 at twelve and
eleven metadata acknowledgements.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Recover independent keys through a bounded rolling window while preserving per-key hint generations, admission budgets, cancellation retry, and the existing transaction decision engine.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(rio): isolate internode streams with pooled HTTP/1.1
Default the internode HTTP data plane to reusable HTTP/1.1 connections so paused shard bodies cannot consume shared HTTP/2 receive credit. Retain explicit auto negotiation and the existing TLS, pooling, and recovery paths.
Refs rustfs/rustfs#8843 and rustfs/backlog#2934.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(rpc): frame bounded shard reads for connection reuse
Preserve the bounded read cap and stream-to-EOF sentinel while making connection completion independent of a chunked terminator. Cover exact reads without EOF polling, pooled TLS reuse, and partial-body truncation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(deps): migrate signer and OIDC to patched crypto
Pin the requested signer fix and upstream OIDC/OAuth2 crypto migrations to immutable commits. Preserve registry SDK type identity, remove the old bigint and elliptic-curve chains, and verify independent login state, nonce, and PKCE values.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): isolate async checkpoint log capture
Scope captured tracing subscribers to each future poll and cover the
competing-dispatcher failure without weakening checkpoint assertions.
Verify exact bounded erasure-probe accounting around the 64-entry limit.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): classify erasure candidates in one budgeted walk
Continue directory classification in cooperative 64-entry batches until
part evidence or EOF, retaining normal checkpoint and namespace work.
Defer metadata accounting and namespace reuse until classification ends.
Distinguish vanished opens from incomplete iteration and preserve quota,
sampling, failed-object and non-destructive heal behavior.
Pin test callsite interest alongside poll-scoped capture to prevent an
unsubscribed sibling from caching the checkpoint callsite as disabled.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): roll a wedged pause backlog forward to its proven ledger
For rustfs/backlog#2873 defect E.
Two interleaved writers could leave a replica state that no claim could
leave. Writer B read writer A's partial commit and selected the stable
rollback point. A's commit then completed and A verified it. B's rollback
stabilization overwrote every member it had read after the commit, and A's
stabilization reached only the remaining member. No commit survived and the
stable fields differed, so every claim and every decommission retirement
failed with "neither a surviving membership commit nor a stable rollback
point".
- When no commit survives and stable consensus fails, select the unique
newest stable ledger by (writer epoch, generation). A stable field is
written only after its ledger was selected as authority, so that ledger is
the last proven one. The next claim stabilizes it and advances normally.
- Keep failing closed for an invalid or unreadable replica, for two
different stable ledgers at the newest key, and for a missing replica that
a stored commit names as a member.
- A rollback proof written without an authoritative commit names only
members that already store a record, so a failed first write to a new
member cannot turn it into a named member that reads as lost.
- Retirement refuses a rolled-forward ledger until the writer has
stabilized it, as a retryable handoff conflict rather than a terminal
authority conflict.
- Tests replay the interleaving on real storage: it converges and retires,
it converges after another expansion with a failed first write to a new
member, and a same-key fork or a lost named member still refuses.
* fix(scanner): let retirement stabilize a rolled-forward pause backlog
Follow-up to the defect E roll-forward for rustfs/backlog#2873.
- Retirement no longer waits for a scanner writer. When no commit
survives and the native selector over every read member (survivors,
retiring source and queued pools) rolls forward, the planner commits and
stabilizes that ledger on the survivors under the writers' object fence.
The previous retryable handoff could exhaust its ~13.5 s budget long
before a writer re-claimed, and failed the decommission terminally.
- A writer's roll-forward refuses when a stored commit names a member that
is missing or outside its read, such as a retiring source excluded from
the writable set. Retirement, which reads that member, decides instead.
- A writer stabilizing a rolled-forward ledger writes missing members
first and stops at the first failure. A member that only read as missing
rejects the create, so no surviving proof can omit its newer record.
- Docs describe the actual roll-forward preconditions and the retirement
exit.
- Tests: retirement converges the wedge with no writer; a writer refuses to
roll forward past an unread source that holds the newest ledger and
retirement keeps it; a spurious missing read changes no record and the
next claim converges on the newest ledger.
* fix(decommission): create missing pause backlog survivors first
Follow-up to the defect E retirement roll-forward for rustfs/backlog#2873.
- Retirement creates survivors that read as missing one at a time, before
any stored survivor, and stops at the first failed create. A transient
create failure on a new pool's set no longer leaves every other survivor
holding a commit that names a still-missing member, which no later
selection could leave.
- A rolled-forward retirement first seeds a rollback proof that names only
the survivors that store a record; the commit still names every survivor.
- Retirement rolls forward only when the survivors cannot select a ledger
on their own or select the same one. Survivors that agree on a different
ledger keep the source-authority check.
- The scanner writer applies the same missing-first, stop-at-first-failure
order to every stabilization with a missing member, including a seed from
an old commit selected as the rollback proof.
- Tests: a failed new-pool create during retirement retries to convergence;
a spurious missing read cannot seed an old commit over a newer member;
the rolled-forward precedence over source stable records is pinned.
* docs(decommission): state the all-missing survivor limit of backlog roll-forward
* feat(scanner): configure startup recovery backoff cap
* test(scanner): compare recovery retry count without narrowing
* docs(scanner): document recovery backoff cap and expose it in Helm
List scanner.recovery_backoff_cap in the runtime controls table, drop the
fixed ten-minute wording from the follower recovery section, and map
config.rustfs.scanner.recovery_backoff_cap_secs to
RUSTFS_SCANNER_RECOVERY_BACKOFF_CAP_SECS in the chart.
* fix(quota): retire degraded floors from durable usage reports
* docs(quota): clarify retained usage cache refresh behavior
* fix(admin): report unknown bucket usage as null in GetBucketQuota
GetBucketQuota serialized missing usage as size 0, which contradicted the
D3 rule that unknown usage is never reported as zero and, after retiring
degraded floors, made a bucket near its limit look empty. Make size
optional and stop collapsing a missing or unreadable usage snapshot to 0.
* fix(scanner): recover dual corruption with retained epoch fencing
* fix(scanner): preserve recovery markers and legacy quota floors
* test(scanner): expect retained usage recovery marker state
* docs(scanner): register dual-corruption recovery evidence objects
Add the .bloomcycle.bin.dual-recovery/ evidence family to the persisted
objects table and describe its owner, write-once lifecycle and recovery
rule in Deletion And Recovery Rules.
* fix(s3): expose typed causes for unavailable responses
Keep the existing S3 retry contract while distinguishing namespace lock
timeouts, other lock refusals, and storage write quorum failures. Reuse one
bounded typed-source walk and document multipart session attribution.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(s3): preserve XML type on attributed errors
Keep application/xml when the pinned serializer replaces generated headers
with the custom cause map. Assert the media type on direct and wrapped lock
timeouts and namespace/write-quorum error responses.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): unify multipart generation fence lifetimes
Own registry registration before asynchronous incarnation validation so
cancellation cannot preserve a memo beyond real lifecycle-lock coverage.
Reuse successful validation for overlapping PUT and multipart operations,
and retain the full owner through detached part commits and orphan cleanup.
Keep cold validation concurrent and preserve authoritative snapshot,
borrowed-fence, and disabled-lock checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): remove redundant pending closure
Pass std::future::pending directly to the incarnation validator in the
cancellation regression test so strict Clippy accepts the test target.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Picks up main through 44a73b0be. Conflicts: rustfs/src/error.rs keeps the rustfs-s3-types ApiError conversion and drops the helper main's #8816 replaced; scripts/check_s3s_footprint.sh keeps the integration script, whose baseline lives in .config (main only retuned a constant the integration script does not have). Main's new tests use the rustfs-s3-types error code in admin replication and reach s3s::BodySizeLimitExceeded through the app's test-only s3 facade, so the s3s file count stays at 118; s3_error_lines tightens to 302. The test-count baseline is regenerated; the e2e selections are unchanged.
A set-remote-target update with the creds group on a bucket target that
belongs to a site-replication peer skipped the connection change, logged a
warning, and still returned 200. The Console's bucket replication form sends
exactly that request when an endpoint changes, so operators saw success while
the stored endpoint, credentials, and TLS settings stayed unchanged.
Return 400 InvalidRequest instead, naming the target and the evidence that
makes it peer-owned, and pointing at a site-replication edit. The ownership
probes are unchanged; the decision is extracted so it can be unit tested.
Document how to change a site's address: the local endpoint is recorded once
at setup and never re-derived, edits must reach every other peer at its stored
endpoint, so sites move one at a time (or old addresses are restored
temporarily), followed by a resync. Commands use rc because mc's resync does
not work against RustFS and its update renames the site.
Picks up the 20 main commits since 62b6f5f3f. The one conflict is the tier free-version recovery test setup: it keeps the integration-only fixture isolation (background recovery disabled while the fixture starts) and main's mock tier registration (#8804 skips scans with no tier). The e2e selections are unchanged by these commits; the test-count baseline is regenerated.
The scheduled sync has stopped on conflicts since 2026-10-19. Resolved by keeping both sides: main's restructured delete-rollback path in disk/local.rs with the integration-only native-publication record, main's per-instance region with the rustfs-s3-types Region, main's new lifecycle, rebalance and admin code ported to the persisted lifecycle types and rustfs-s3-types errors, both sides' new scanner tests, and the union of the access_consumer imports. The e2e smoke, full (darwin) and repl-nightly selections and the test-count baseline are regenerated from the merged tree.
* fix(decommission): move the free version of an expired tier version
Decommission only queued the lifecycle expiry of a transitioned version, so a source cleanup that ran first deleted the version itself and appended its tier free version after the entry's last pass. The post-check then failed the decommission until an operator re-ran it.
Apply the expiry of a transitioned version synchronously, as rebalance does. Its free version is on the source before the cleanup preflight, which reports the entry changed, and the retried entry moves the record to a target in the same pass. An expiry that cannot be applied leaves the version live, so it moves like any other version.
Refs rustfs/backlog#2873
* fix(decommission): move a delete-all expired tier version live
Applying a delete-all expiry inline made its prefix delete wait for the bucket lifecycle write lock behind the entry's own read lock until the lock timeout, blocking every write in the bucket meanwhile. Move such a transitioned version live and leave the delete-all to lifecycle on the target.
Cover a genuinely unapplied expiry and a free-version worker that consumes the source record before the entry moves it.
Refs rustfs/backlog#2873
* fix(heal): explain unverified legacy heal outcomes
Preserve storage diagnostics in canonical unknown outcomes and explain
legacy payload identity limits through the shared local/peer status path.
Borrow expected receipt identities while preserving all proof checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): keep diagnostics from pausing merged MRF repairs
Require the actual MRF executor before publishing legacy pause events.
An ordinary queued owner can retain a merged durable MRF notice without
having performed the specialized deep check. Preserve its diagnostic
without granting it authority to park the repair responsibility.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): type MRF outcomes and defer status diagnostics
Carry storage-owned legacy observations in process and return a typed
terminal result only after the qualified MRF Deep check. Publish pause
events from that result and exact identities instead of source or text.
Keep object names out of retry classification and compute only the
status diagnostic selected by the canonical execution state.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* refactor(ecstore): move bucket configs onto gateway persisted types
Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.
- ecstore: BucketConfigXml is implemented by the gateway codec for every
family; metadata, metadata_sys, versioning, object lock, replication and
lifecycle consumers take the persisted types. Region moves to
rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
persisted types in storage/storage_api.rs by writing one codec's XML and
reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.
The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.
* test(app): prepare the PutBucketEncryption config from the request
The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.
* style(app): format the PutBucketEncryption test
* test(e2e): add scanner/heal topology protection matrix
Add the S00 protection baseline for the Scanner/Heal program
(rustfs/backlog#2873, program rustfs/backlog#2872): seven
scanner_heal_topology_ cases in the distributed e2e lane that pin pool
expansion, rebalance and decommission behavior before the runtime changes.
The fixture seeds a deterministic inventory (empty, inline, external,
multipart, special and nested keys; a versioned bucket with an explicit
null version, three history versions and a delete marker per key; an
Object Lock bucket with legal hold and COMPLIANCE retention), records
every version with its requested and listed version id, length and
SHA-256, verifies the manifest through every node, reads physical
placement from xl.meta on the pool drives, and reads the strict
durability mode back from the admin API and the node logs.
Fault cases hold the mover at proven windows through file barriers that
compile only with the e2e-test-hooks feature: before the multipart target
commit and before source cleanup (shared by decommission and rebalance),
plus the existing pool-metadata before_publish cutpoint. An unreached
window fails the case. DistCluster gains capture_logs_under so a case can
read node startup events without a global environment variable.
docs/testing/scanner-heal-topology-gates.md maps the gates to the tests
and records the timeout basis.
* test(e2e): keep the barrier failure message static
The error(format!) ratchet rejects new formatted io::Error::other call
sites in ecstore. The reached observation already records the stage and
object, so the injected failure needs no per-call detail.
* test(e2e): require e2e-test-hooks for topology fault cases
The four scanner_heal_topology_ fault cases inject at server barriers that
exist only with the e2e-test-hooks feature. Against a default server they
waited for the barrier timeout and reported it as not reached. Check the
verified binary's features first so the missing prerequisite fails
immediately and is named.
* ci(e2e): build the distributed lane with e2e-test-hooks
Every other e2e lane builds and runs the fault-test hooks server. The
distributed lane alone used the default server, so the topology fault
cases could never reach their barriers there. Build and run with the same
feature so the receipt matches, and update the documented local commands.
The upgrade cases spawn the pinned previous release directly and do not
compare feature sets.
* test(e2e): keep topology artifacts under RUSTFS_E2E_LOG_DIR
The topology fixture captured node logs, barrier observations and evidence
in a temp dir removed on drop, so the distributed workflow uploaded nothing
for these cases. Fall back to RUSTFS_E2E_LOG_DIR, which the workflow
uploads, when no explicit evidence directory is set.
* test(e2e): remove retired pools before restart; race the pre-ACK start
The queue case restarted with decommissioned pools 0 and 1 still on the
command line, which the server rightly refuses. Add
remove_single_node_pools and restart without them, then expect the
remaining pools renumbered to 0 and 1.
The pre-ACK case now races the start request against the barrier. A start
that returns first, or stays in flight past the documented 60 s budget, is
reported as such instead of as an unreached barrier, and the evidence
records the reach latency.
capture_logs_under now overrides a log path assigned from
RUSTFS_E2E_LOG_DIR, so the seed node logs into the case directory too.
* test(e2e): bound fault-window waits by the movement budget
On Linux CI a busy target capacity gate delayed the first fault object by
about two minutes, past the 120 s wait borrowed from the server's release
poll. Wait for a movement window within the 240 s movement budget instead,
and fail at once when the decommission or rebalance ends without entering
the window, so an uninjected fault is never counted.
* test(e2e): bound status probes while waiting for a fault window
The admin decommission status request can wait on a mover held at a test
barrier until the barrier's own 120 s release timeout. The window wait
issued that request each loop, so a reached barrier went unnoticed until
the server timed it out and failed the decommission, which surfaced as a
false resume failure. Bound each status probe to 5 s and treat a slow probe
as still running.
* test(e2e): retry a busy decommission start
A decommission start that meets a busy movement gate can answer a
retryable 503 instead of waiting. The harness treated any non-proof
failure as fatal. Share one start helper that retries 503 within the
case's movement budget and the fleet-proof 500 for 45 s, and keeps every
other status fatal. The pre-ACK case retries 503 within its window budget.
* fix(decommission): defer source cleanup quorum failures
A quorum failure while deleting the source of an already copied entry
was fatal and marked the whole pool failed, although the copied source
is still in place and an uncommitted delete transaction is settled by
delete recovery. Carry the cleanup error typed and resolve a quorum
failure as Deferred: the entry is replayed at the end of the round, and
a deferral that persists takes the existing capacity-blocked pause path
instead of a terminal failure. Other cleanup errors and quorum errors
from other stages stay terminal.
Add set-level regressions for parent and child keys (`a/b`, `a/b/c`):
parent cleanup leaves every child file untouched, and parent cleanup
reaches quorum while the child is cleaned up or rewritten concurrently.
* test(ecstore): keep source cleanup errors typed in prefix cleanup tests
Return SourceCleanupError from the test helper instead of formatting it
into Error::other, and shrink the error(format!) ratchet baseline for
core/pools.rs, which lost one call site in the cleanup failure change.
* fix(decommission): retry cleanup quorum failures and count replays once
Retry a source cleanup quorum failure in place, up to three attempts
with the existing source-cleanup backoff, before deferring the entry.
Carry the versions an entry already counted on its deferral, so the
end-of-round replay does not add a fully counted entry to
items_decommissioned and bytes_done again.
A deferral that persists pauses the pool through the existing deferred
drain path. When source cleanup caused it, persist a reason led by
"source cleanup blocked" without the carrier error's free-drive wording,
and report waitingReason "source_cleanup" instead of "capacity".
Add a source_cleanup test fault stage and a routine test that drives a
cleanup quorum fault through deferral and replay to completion.
* fix(decommission): wait seconds between cleanup quorum retries
Give the in-place source cleanup quorum retry a fixed floor of one and
then two seconds, with the existing jitter on top, so it can ride out a
drive reconnect or health-check blip instead of retrying within a few
hundred milliseconds. The wait stays bounded and cancel-safe.
Find a source cleanup pause reason through stage wrappers, as the
capacity-blocked check does, and add a routine test where cleanup keeps
losing quorum: the decommission pauses with a source cleanup reason, the
bucket stays queued and the source stays in place.
Document the additive waitingReason value source_cleanup and its
compatibility with older nodes reading the same pool.bin.
* fix(decommission): never retry a lost source cleanup fence
The cleanup retry classifier used is_quorum_error(), which also matches
NamespaceLockQuorumUnavailable. The delete commit fence reports a lost
namespace lock or mutation fence with that variant, so a lost fence was
retried in place and only stopped later at the pre-cleanup fence check.
A lost fence means this worker no longer owns the key; it must stop
without retrying. Retry and defer only drive quorum failures:
ErasureReadQuorum, ErasureWriteQuorum, InsufficientReadQuorum and
InsufficientWriteQuorum.
---------
Co-authored-by: Hauser <housemecn@gmail.com>
* fix(decommission): let a failed start roll back its own activation
A fresh decommission start whose peer reload failed could not roll back:
the pool metadata stale-write guard from #6604 refuses any save that
clears a persisted active or queued decommission, so the pool stayed
active with no worker until the watchdog or a leader restart picked it
up.
The rollback save now carries the activation that start committed. The
guard exempts a pool only while its persisted record still equals that
activation field for field, so a worker checkpoint or a later start
keeps the rejection. A canceled record is not an active decommission
and is cleared by the rollback as well. Every other save path passes no
activation and is guarded as before.
A peer may already have reloaded the activation and started a worker
before the rollback. Saves are now fenced by decommission generation: a
write carrying a start_time is rejected when the persisted record
belongs to a later start, or has no decommission state and was updated
after that start_time, so such a worker cannot make the pool active
again. Its queued promotion and progress checkpoint also adopt the
persisted record and stop the worker.
Refs rustfs/backlog#2873
* fix(decommission): fence terminal writes of a withdrawn decommission
Terminal records carry no start_time, so the generation fence did not
cover them: after a start rollback, a peer worker of the rolled back
generation that failed (for example on a bucket completion) could still
persist a failed record over the restored pool. The merge now rejects a
failed, canceled or complete record over a persisted record without
decommission state; every attempt persists decommission state first, so
no legitimate path writes a terminal record there.
Peers also drop a withdrawn decommission sooner. A pool metadata reload
replaces a running decommission with a newer record that withdrew it,
even under an active worker; bucket completion adopts such a record like
promotion and progress checkpoints do; and a withdrawn progress
checkpoint stops the worker instead of logging a failed save and moving
more entries.
Tests cover the peer failure mark, a rollback retry that sees a worker
checkpoint land between attempts, and the reload withdrawal. A heal
fixture that wrote complete records over a fresh pool.bin now persists
the active attempt first.
Refs rustfs/backlog#2873
* fix(decommission): fence peer writes after a retry start rollback
A retry start replaces a failed or canceled record that keeps its
unresolved ledger, so its rollback restores that terminal record rather
than clearing the pool. The withdrawal checks only recognized a record
without decommission state, and the generation fence ignored a persisted
record without a start_time. A peer worker of the rolled back retry
could therefore checkpoint the pool back to active.
The generation fence now compares the persisted last_update whenever
the persisted record has no start_time of its own (cleared, terminal or
queued). The progress checkpoint also checks the generation against the
record read under the pool metadata lock, as bucket completion already
does, and the save-path withdrawal check treats a terminal record newer
than the running view as a withdrawal, so the peer adopts it and stops.
A reload still never lets a newer terminal record replace an active
worker's view, so after a retry's rollback a peer worker stops at its
next bucket completion or progress checkpoint rather than on reload.
Tests cover the retry rollback against a peer checkpoint and direct
write, and a peer bucket completion after a fresh start's rollback.
Refs rustfs/backlog#2873
* fix(decommission): keep a retry rollback's restored record intact
After a retry start rolls back, the restored record is failed, so a
peer worker's own failure mark passed every merge guard and replaced
it. A failure mark now saves through save_current_pool_meta_checked,
which runs the withdrawal check against the record read under the pool
metadata lock and adopts that record instead of writing.
The withdrawal check compared the restored record with the worker's
in-memory last_update, which a rejected save had already raised, so the
peer never adopted the rollback and the watchdog kept resuming it. It
now compares against the running generation (start_time), and a
rejected unresolved-entry save undoes its in-memory change.
save_current_pool_meta_for_decommission_start is also compiled for the
test-util feature again; a test-util helper still calls it, which broke
the workspace clippy and Connect builds.
Refs rustfs/backlog#2873
* fix(decommission): keep an unresolved entry when its own save fails
The rollback of a rejected unresolved-entry save also dropped the entry
on transient failures such as a quorum or lock-fence error, so the
following failure mark persisted a failed record without it and an
operator could clear a decommission that skipped an unreadable object.
Keep the entry and undo only the last_update bump, which is all a
withdrawn worker needs to still see the withdrawal.
Refs rustfs/backlog#2873
overtrue