Keep the original storage results and all required request/replication
ownership while dropping two bindings with no consumers.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Consume completed aggregation entries at their terminal conversion so
bucket and replication target keys can move into the output maps.
Preserve validation, rejection and publication behavior.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(decommission): finish multipart null versions superseded on target
A decommission entry for an unversioned multipart source whose key
already holds a newer null version on the target settles as superseded
and aborts its staged upload. The cleanup's target proof saw the newer
object, could not match it to the staged identity, and failed closed
with a capacity-blocked error, so the routine paused, the source was
never cleaned up, and the error read like a minimum-free-drive block.
The staged upload is already proven absent and the target lookup is
exact: a different version id, or a null target with another
modification time (one null slot per key), is a different object, so
the staged copy holds nothing on the target and the pending intent is
cleared. Only a versioned target with the staged version id but another
modification time stays fail-closed. The remaining identity-proof
failures use a typed error without capacity wording.
Add three-pool decommission regressions for restored-transitioned and
plain multipart null sources superseded on the target.
* test(decommission): pin multipart target proof failures as terminal
Document the accepted residual of the null-slot proof: a metadata-only
self-copy keeps a published but uncharged staged copy's data while
moving its modification time, so its marker is released and one
object's bytes stay uncharged.
Add a three-pool routine test where the target already holds an
unowned null object with the source's exact identity: the cleanup's
ownership-proof failure fails the decommission with no capacity-blocked
reason instead of pausing it.
Require every security candidate to name the lower-trust principal, the crossed boundary, the affected resource, and the observable result, and to land under exactly one of `confirmed`, `needs_validation`, or `rejected`. Only confirmed findings carry severity; a trace blocked on an unresolved deployment fact is reported as that fact instead of an assumption.
A versioned multipart completion minted a new version when the staged version id was absent or nil, so an inbound replica of a source null version written by multipart landed under a fresh UUID. Keep the null version for authorized inbound replicas, and address the null version as versionId=null on the worker convergence HEAD like the other target reads.
Refs rustfs/rustfs#8747
* fix(decommission): move the free version of an expired tier version
Decommission only queued the lifecycle expiry of a transitioned version, so a source cleanup that ran first deleted the version itself and appended its tier free version after the entry's last pass. The post-check then failed the decommission until an operator re-ran it.
Apply the expiry of a transitioned version synchronously, as rebalance does. Its free version is on the source before the cleanup preflight, which reports the entry changed, and the retried entry moves the record to a target in the same pass. An expiry that cannot be applied leaves the version live, so it moves like any other version.
Refs rustfs/backlog#2873
* fix(decommission): move a delete-all expired tier version live
Applying a delete-all expiry inline made its prefix delete wait for the bucket lifecycle write lock behind the entry's own read lock until the lock timeout, blocking every write in the bucket meanwhile. Move such a transitioned version live and leave the delete-all to lifecycle on the target.
Cover a genuinely unapplied expiry and a free-version worker that consumes the source record before the entry moves it.
Refs rustfs/backlog#2873
* fix(heal): explain unverified legacy heal outcomes
Preserve storage diagnostics in canonical unknown outcomes and explain
legacy payload identity limits through the shared local/peer status path.
Borrow expected receipt identities while preserving all proof checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): keep diagnostics from pausing merged MRF repairs
Require the actual MRF executor before publishing legacy pause events.
An ordinary queued owner can retain a merged durable MRF notice without
having performed the specialized deep check. Preserve its diagnostic
without granting it authority to park the repair responsibility.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): type MRF outcomes and defer status diagnostics
Carry storage-owned legacy observations in process and return a typed
terminal result only after the qualified MRF Deep check. Publish pause
events from that result and exact identities instead of source or text.
Keep object names out of retry classification and compute only the
status diagnostic selected by the canonical execution state.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Capture an existing ReadVersion caller and earliest entry before spawning config transactions. Restore them with the existing scoped wrapper while retaining default-feature operation propagation and detached task ownership.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(s3): classify empty multipart completion as InvalidPart
Treat absent per-drive multipart transaction directories as empty listings so
completion reaches the existing InvalidPart validation. Preserve quorum and
transaction recovery, and cover the HTTP and degraded-drive boundaries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): move multipart completion metadata in one pass
Reuse quorum-validated part metadata instead of repeatedly inserting, sorting,
finding and cloning parts. Keep legacy cleanup identities and ordering while
moving the original part allocation into the completion manifest. Compare ETags
by borrowing their normalized values.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(metrics): expose usage candidate rejection outcomes
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): borrow usage candidate fixtures
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): reserve capacity for decommission probes
Reserve all nextest slots for the two eight-disk decommission fixtures under both default and CI profiles. Keep their internal concurrency, production lock budgets, assertions and zero-retry policy intact.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): scope prewrite capacity probes to the target pool
After persisting a new capacity intent, refresh only the target pool baseline. Preserve the global admission and finalization scans, lock order, fencing, and accounting formulas. Verify the reduced probe count with real disk metrics and preserve injected snapshot/error behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Capture the caller operation before the tracked mutation task is spawned. Keep detached completion, owned guards, result errors and panic join failures intact.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): add scanner/heal topology protection matrix
Add the S00 protection baseline for the Scanner/Heal program
(rustfs/backlog#2873, program rustfs/backlog#2872): seven
scanner_heal_topology_ cases in the distributed e2e lane that pin pool
expansion, rebalance and decommission behavior before the runtime changes.
The fixture seeds a deterministic inventory (empty, inline, external,
multipart, special and nested keys; a versioned bucket with an explicit
null version, three history versions and a delete marker per key; an
Object Lock bucket with legal hold and COMPLIANCE retention), records
every version with its requested and listed version id, length and
SHA-256, verifies the manifest through every node, reads physical
placement from xl.meta on the pool drives, and reads the strict
durability mode back from the admin API and the node logs.
Fault cases hold the mover at proven windows through file barriers that
compile only with the e2e-test-hooks feature: before the multipart target
commit and before source cleanup (shared by decommission and rebalance),
plus the existing pool-metadata before_publish cutpoint. An unreached
window fails the case. DistCluster gains capture_logs_under so a case can
read node startup events without a global environment variable.
docs/testing/scanner-heal-topology-gates.md maps the gates to the tests
and records the timeout basis.
* test(e2e): keep the barrier failure message static
The error(format!) ratchet rejects new formatted io::Error::other call
sites in ecstore. The reached observation already records the stage and
object, so the injected failure needs no per-call detail.
* test(e2e): require e2e-test-hooks for topology fault cases
The four scanner_heal_topology_ fault cases inject at server barriers that
exist only with the e2e-test-hooks feature. Against a default server they
waited for the barrier timeout and reported it as not reached. Check the
verified binary's features first so the missing prerequisite fails
immediately and is named.
* ci(e2e): build the distributed lane with e2e-test-hooks
Every other e2e lane builds and runs the fault-test hooks server. The
distributed lane alone used the default server, so the topology fault
cases could never reach their barriers there. Build and run with the same
feature so the receipt matches, and update the documented local commands.
The upgrade cases spawn the pinned previous release directly and do not
compare feature sets.
* test(e2e): keep topology artifacts under RUSTFS_E2E_LOG_DIR
The topology fixture captured node logs, barrier observations and evidence
in a temp dir removed on drop, so the distributed workflow uploaded nothing
for these cases. Fall back to RUSTFS_E2E_LOG_DIR, which the workflow
uploads, when no explicit evidence directory is set.
* test(e2e): remove retired pools before restart; race the pre-ACK start
The queue case restarted with decommissioned pools 0 and 1 still on the
command line, which the server rightly refuses. Add
remove_single_node_pools and restart without them, then expect the
remaining pools renumbered to 0 and 1.
The pre-ACK case now races the start request against the barrier. A start
that returns first, or stays in flight past the documented 60 s budget, is
reported as such instead of as an unreached barrier, and the evidence
records the reach latency.
capture_logs_under now overrides a log path assigned from
RUSTFS_E2E_LOG_DIR, so the seed node logs into the case directory too.
* test(e2e): bound fault-window waits by the movement budget
On Linux CI a busy target capacity gate delayed the first fault object by
about two minutes, past the 120 s wait borrowed from the server's release
poll. Wait for a movement window within the 240 s movement budget instead,
and fail at once when the decommission or rebalance ends without entering
the window, so an uninjected fault is never counted.
* test(e2e): bound status probes while waiting for a fault window
The admin decommission status request can wait on a mover held at a test
barrier until the barrier's own 120 s release timeout. The window wait
issued that request each loop, so a reached barrier went unnoticed until
the server timed it out and failed the decommission, which surfaced as a
false resume failure. Bound each status probe to 5 s and treat a slow probe
as still running.
* test(e2e): retry a busy decommission start
A decommission start that meets a busy movement gate can answer a
retryable 503 instead of waiting. The harness treated any non-proof
failure as fatal. Share one start helper that retries 503 within the
case's movement budget and the fleet-proof 500 for 45 s, and keeps every
other status fatal. The pre-ACK case retries 503 within its window budget.
ci(pool): finalize decommission removal and verify surviving pools (rustfs#8726)
Select suite step 10 together with step 9 and pass
--finalize-decommission so the nightly exercises the post-removal
survivor path: pool ids must stay unique, bucket metadata writes must
keep succeeding across a restart, and no node may log 'pool metadata
layout changed'. Step 10 self-skips when decommission is disabled.
Co-authored-by: Hauser <housemecn@gmail.com>
Track active walks, scratch capacity and accepted bytes while preserving the existing 1 MiB stream copy behavior.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(decommission): defer source cleanup quorum failures
A quorum failure while deleting the source of an already copied entry
was fatal and marked the whole pool failed, although the copied source
is still in place and an uncommitted delete transaction is settled by
delete recovery. Carry the cleanup error typed and resolve a quorum
failure as Deferred: the entry is replayed at the end of the round, and
a deferral that persists takes the existing capacity-blocked pause path
instead of a terminal failure. Other cleanup errors and quorum errors
from other stages stay terminal.
Add set-level regressions for parent and child keys (`a/b`, `a/b/c`):
parent cleanup leaves every child file untouched, and parent cleanup
reaches quorum while the child is cleaned up or rewritten concurrently.
* test(ecstore): keep source cleanup errors typed in prefix cleanup tests
Return SourceCleanupError from the test helper instead of formatting it
into Error::other, and shrink the error(format!) ratchet baseline for
core/pools.rs, which lost one call site in the cleanup failure change.
* fix(decommission): retry cleanup quorum failures and count replays once
Retry a source cleanup quorum failure in place, up to three attempts
with the existing source-cleanup backoff, before deferring the entry.
Carry the versions an entry already counted on its deferral, so the
end-of-round replay does not add a fully counted entry to
items_decommissioned and bytes_done again.
A deferral that persists pauses the pool through the existing deferred
drain path. When source cleanup caused it, persist a reason led by
"source cleanup blocked" without the carrier error's free-drive wording,
and report waitingReason "source_cleanup" instead of "capacity".
Add a source_cleanup test fault stage and a routine test that drives a
cleanup quorum fault through deferral and replay to completion.
* fix(decommission): wait seconds between cleanup quorum retries
Give the in-place source cleanup quorum retry a fixed floor of one and
then two seconds, with the existing jitter on top, so it can ride out a
drive reconnect or health-check blip instead of retrying within a few
hundred milliseconds. The wait stays bounded and cancel-safe.
Find a source cleanup pause reason through stage wrappers, as the
capacity-blocked check does, and add a routine test where cleanup keeps
losing quorum: the decommission pauses with a source cleanup reason, the
bucket stays queued and the source stays in place.
Document the additive waitingReason value source_cleanup and its
compatibility with older nodes reading the same pool.bin.
* fix(decommission): never retry a lost source cleanup fence
The cleanup retry classifier used is_quorum_error(), which also matches
NamespaceLockQuorumUnavailable. The delete commit fence reports a lost
namespace lock or mutation fence with that variant, so a lost fence was
retried in place and only stopped later at the pre-cleanup fence check.
A lost fence means this worker no longer owns the key; it must stop
without retrying. Retry and defer only drive quorum failures:
ErasureReadQuorum, ErasureWriteQuorum, InsufficientReadQuorum and
InsufficientWriteQuorum.
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Cover actual journal and lifecycle checkpoint replay when an unknown historical source coexists with an exact source-bound repair. Reject a pre-restart lease and retain the complete unresolved record after the new repair is discharged and replayed again.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(decommission): let a failed start roll back its own activation
A fresh decommission start whose peer reload failed could not roll back:
the pool metadata stale-write guard from #6604 refuses any save that
clears a persisted active or queued decommission, so the pool stayed
active with no worker until the watchdog or a leader restart picked it
up.
The rollback save now carries the activation that start committed. The
guard exempts a pool only while its persisted record still equals that
activation field for field, so a worker checkpoint or a later start
keeps the rejection. A canceled record is not an active decommission
and is cleared by the rollback as well. Every other save path passes no
activation and is guarded as before.
A peer may already have reloaded the activation and started a worker
before the rollback. Saves are now fenced by decommission generation: a
write carrying a start_time is rejected when the persisted record
belongs to a later start, or has no decommission state and was updated
after that start_time, so such a worker cannot make the pool active
again. Its queued promotion and progress checkpoint also adopt the
persisted record and stop the worker.
Refs rustfs/backlog#2873
* fix(decommission): fence terminal writes of a withdrawn decommission
Terminal records carry no start_time, so the generation fence did not
cover them: after a start rollback, a peer worker of the rolled back
generation that failed (for example on a bucket completion) could still
persist a failed record over the restored pool. The merge now rejects a
failed, canceled or complete record over a persisted record without
decommission state; every attempt persists decommission state first, so
no legitimate path writes a terminal record there.
Peers also drop a withdrawn decommission sooner. A pool metadata reload
replaces a running decommission with a newer record that withdrew it,
even under an active worker; bucket completion adopts such a record like
promotion and progress checkpoints do; and a withdrawn progress
checkpoint stops the worker instead of logging a failed save and moving
more entries.
Tests cover the peer failure mark, a rollback retry that sees a worker
checkpoint land between attempts, and the reload withdrawal. A heal
fixture that wrote complete records over a fresh pool.bin now persists
the active attempt first.
Refs rustfs/backlog#2873
* fix(decommission): fence peer writes after a retry start rollback
A retry start replaces a failed or canceled record that keeps its
unresolved ledger, so its rollback restores that terminal record rather
than clearing the pool. The withdrawal checks only recognized a record
without decommission state, and the generation fence ignored a persisted
record without a start_time. A peer worker of the rolled back retry
could therefore checkpoint the pool back to active.
The generation fence now compares the persisted last_update whenever
the persisted record has no start_time of its own (cleared, terminal or
queued). The progress checkpoint also checks the generation against the
record read under the pool metadata lock, as bucket completion already
does, and the save-path withdrawal check treats a terminal record newer
than the running view as a withdrawal, so the peer adopts it and stops.
A reload still never lets a newer terminal record replace an active
worker's view, so after a retry's rollback a peer worker stops at its
next bucket completion or progress checkpoint rather than on reload.
Tests cover the retry rollback against a peer checkpoint and direct
write, and a peer bucket completion after a fresh start's rollback.
Refs rustfs/backlog#2873
* fix(decommission): keep a retry rollback's restored record intact
After a retry start rolls back, the restored record is failed, so a
peer worker's own failure mark passed every merge guard and replaced
it. A failure mark now saves through save_current_pool_meta_checked,
which runs the withdrawal check against the record read under the pool
metadata lock and adopts that record instead of writing.
The withdrawal check compared the restored record with the worker's
in-memory last_update, which a rejected save had already raised, so the
peer never adopted the rollback and the watchdog kept resuming it. It
now compares against the running generation (start_time), and a
rejected unresolved-entry save undoes its in-memory change.
save_current_pool_meta_for_decommission_start is also compiled for the
test-util feature again; a test-util helper still calls it, which broke
the workspace clippy and Connect builds.
Refs rustfs/backlog#2873
* fix(decommission): keep an unresolved entry when its own save fails
The rollback of a rejected unresolved-entry save also dropped the entry
on transient failures such as a quorum or lock-fence error, so the
following failure mark persisted a failed record without it and an
operator could clear a decommission that skipped an unreadable object.
Keep the entry and undo only the last_update bump, which is all a
withdrawn worker needs to still see the withdrawal.
Refs rustfs/backlog#2873
Under sustained writes nearly every cycle fell back to a full walk: scope
admission required the global dirty generation to stay unchanged across the
bucket listing, and after a Complete cycle the dirty-usage wake ran the next
full walk immediately, back to back.
- Track dirty-map removals with a process-local removal generation and admit
a scoped walk when no entry was removed or replayed during the listing
(`covers_listed_pending`). Writes to listed buckets in the window join the
start snapshot; unlisted buckets dirtied after the cutoff stay pending and
are never cleared by compare-and-remove certification. The strict
`covers_all_pending` still gates segment-reuse evidence.
- Classify scope fallbacks (removal during listing, generation exhausted,
dirty-bucket overflow, no listed dirty bucket, baseline unavailable, peer
scope unavailable), log `scoped_scan_fallback`, and report them in
usage_freshness as additive serde(default) fields.
- Pace the cycle after a Complete fallback while dirty buckets remain on the
capped retry schedule clamped to the cycle interval, with the dirty wake
disabled; the paced timer wake still requests a scope.
- Update the per-bucket convergence design doc.
No persisted format or internode protocol change.
Co-authored-by: Chris <anzhengchao@gmail.com>
Delete-marker replication re-reads the source marker by version before sending. Multi-pool get_object_info rejects explicit marker version reads with MethodNotAllowed, so every marker failed verification and was never replicated. Add a defaulted ObjectOperations::get_object_info_for_delete, override it on ECStore with the marker-tolerant snapshot, and route both replication marker reads through it.
Fixesrustfs/rustfs#8746
* fix(ecstore): ignore child-key prefixes in delete settlement fingerprint
A batch delete of prefix-sharing keys (foo/bar and foo/bar/xyzzy) could
leave foo/bar's commit proof on disk: the child's finalization removed
foo/bar/xyzzy/ between the parent's GC inspection and settlement, the
fingerprint mismatch surfaced as OutdatedXLMeta, and publish_delete_commit
dropped the error. An immediate DeleteBucket then returned BucketNotEmpty.
Fingerprint only this key's protocol evidence (files, UUID dirs and the
transaction dir), retry published cleanup on OutdatedXLMeta with a fresh
inspection, and warn when cleanup is deferred to the recovery hint.
* test(ecstore): cover delete fingerprint narrowing and cleanup retry
Only canonical hyphenated UUID directories count as protocol evidence, so
a child key named as a simple-hex UUID cannot recreate the race. Add
negative fingerprint checks for this key's own changes, and tests for the
published-cleanup retry: success on retry, the attempt bound with the
deferred warning, and no retry after a lost fence.
* fix(ecstore): require canonical UUID names across delete recovery
Discovery, explicit legacy discard and staged-dir checks still accepted
simple, braced and URN UUID spellings. A child key named as the simple
form of a referenced data dir could then be deleted by DiscardLegacy.
Use one canonical-name predicate everywhere and register exactly one
test hook per expected cleanup attempt.
The scanner heal gate dropped Some(Uuid::nil()) along with None, so null versions in versioned buckets never reached existing-object replication. Skip only unversioned listings, matching MinIO, and recognize the nil UUID as the null version in the newer-target guard and version comparison.
Fixesrustfs/rustfs#8747
* fix(iam): derive bucket location conditions from XML
Capture the parsed CreateBucket LocationConstraint at the access boundary
and move it into shared authorization conditions independently of signing
and server regions. Cover conditional allow/deny across user, session and
anonymous requests, including absent and spoofed values.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(iam): pass bucket location directly to authorization
Replace temporary request-extension storage with an explicit owned XML
argument to the single authorization core, keeping the public entry point
compatible. Remove the marker allocation and typemap insert/remove path.
Cover Object Lock and Versioning dependent authorizations with an absent
location condition and verify their enabled state through real HTTP.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(connect): isolate bounded object probe storage
Use an independent temporary volume on Linux shared memory when available
so the real endpoint/CLI fixture does not depend on contended CI disk IO.
Await server shutdown before dropping the caller-owned storage directory.
Retain the production cleanup deadline, measurement budget, assertions,
and existing nextest capacity reservation without adding retries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(runtime): isolate configured region per store instance
Resolve the default RegionInterface through the owning ECStore context
and keep ambient fallback only for requests without an explicit context.
Preserve ambient constructors and injected interfaces, with store and
real HTTP multi-instance regression coverage.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(storage): stabilize quorum and heal crash fixtures
Keep forced-offline disks attached across reconnects and reconnect before every delete retry. Bound SIGKILL fixture I/O while preserving recovery scope, receipts, data integrity, and deadlines; include child execution phases in timeout diagnostics.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* docs(test): clarify forced-offline disk fixture behavior
Describe handle retention instead of claiming a newly reconnected health
tracker inherits the test outage. Execution and assertions are unchanged.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(rebalance): move tier free versions with their entry
Rebalance built its version snapshot with MetaCacheEntry::file_info_versions,
which drops tier free versions, while the source-cleanup preflight reads the
exact snapshot that keeps them inline. Any entry holding a free version (left
by deleting a transitioned version) therefore failed the preflight on every
pass and was deferred forever, so the source pool could never be drained of it.
Mirror decommission: keep free versions in the snapshot, skip lifecycle and
the last-delete-marker policy for them, and move them to the target as tier
metadata through decommission_tiered_object. A free version reads as a delete
marker with no transition status, so the migrator routes it explicitly;
written as a delete marker it would drop the remote owner and orphan the
remote object. A moved free version may be consumed from the source before
cleanup, like decommission; free versions hold no pool data and stay out of
the rebalance stats.
Refs rustfs/backlog#2873
* test(rebalance): cover tier free version movement edge cases
Pin the lifecycle skip, the delete-marker count, the allowed-missing and
not-found consumption paths, the stats exclusion, multiple free versions per
entry, and the unversioned superseded source's free version moving on the
next pass.
Refs rustfs/backlog#2873
* fix(s3): validate bucket location constraints
Reject explicit bucket locations outside the configured server region
before mutation admission, preserve site-default and EU alias compatibility,
and return an empty GetBucketLocation element for us-east-1.
Cover matching and rejected locations over real S3 HTTP requests, including
rejection without creating a bucket.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): refresh bucket region profile selection digests
Include both new region regression tests in the smoke and Darwin/Linux
full selection pins without changing filters or checker behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Establish the first bucket operation before fresh-task dispatch and carry its bounded value into nested config and background work. Preserve cancellation, permit ownership, lock order, and error mapping without importing diagnostic owner or sampling instrumentation.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): leave no source free version for a moved tier version
Data-movement source cleanup deletes each source version separately and
never set skip_tier_free_version, so every transitioned version that
decommission or rebalance had already copied to a target pool left a new
tier free version on the source. Decommission then failed its post-check
for any pool with tiered objects; rebalance left the record behind. The
record claims the remote object the target copy still reads and stays
blocked behind that live reference.
Decommission and rebalance now pass the identities of transitioned
versions whose copy committed on a target, and the cleanup skips the
free version for exactly those. A transitioned version that did not
move, such as one lifecycle expired during the move, still gets its free
version because that record is its only remote-delete owner.
Refs rustfs/backlog#2873, rustfs/backlog#2872
* fix(ecstore): keep the free version of a superseded moved source
Data movement treats a source null version as complete when a newer
unversioned object already sits on the target, without copying it.
That outcome returned the same Ok as a committed copy, so a restored
transitioned null version superseded this way was recorded as
transferred and lost its source free version, leaving its remote object
with no owner.
The copy paths now return DataMovementCopy::Committed or ::Superseded,
and decommission and rebalance record only committed copies of
transitioned versions. A superseded source keeps its free version.
Tests cover the superseded restored null version for rebalance and
decommission, a restored and a null transitioned version moving by
rebalance, an entry with one lifecycle-expired and one moved version,
the per-version skip with two versions of one key, and a directory
object. The tiered stale-target routine test from #8724 now expects the
decommission to complete. The ILM ownership contract and the
decommission notes describe the moved, superseded, expired and legacy
cases.
Refs rustfs/backlog#2873, rustfs/backlog#2872
* test(ecstore): build the decommission ILM fixtures without s3s
The superseded and expiry-mix decommission tests named s3s request and
lifecycle types directly, which added pools_test.rs to the ecstore
files that reference s3s and failed the footprint ratchet. Set the
restore request through ObjectOptions, and store the lifecycle rule as
bucket metadata XML and load it with get_expiry_configs, as the
decommission routine does.
* test(ecstore): pin restored decommission copies and multipart supersession
Decommission recorded a committed data copy of a restored transitioned
version without any test failing if the record were dropped; add a
routine-level decommission of a restored version that checks the post-
check passes, the source keeps nothing, and the target keeps the full
remote tuple. Add the multipart variant of the superseded restored null
version (rebalance), and a unit test that separates committed,
superseded and mismatched targets in the overwrite resume adjudication.
* fix(obs): skip address discovery for initialized node identity
Check the nonblocking node name before resolving the local IP, retaining the final name recheck after fallback. Add portable publication regressions and Linux worker-local netlink fault coverage.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(obs): keep identity contention checks in common
Exercise the real held-writer getter inside its owning module. Publish through the existing runtime helper from obs tests, retaining the kernel netlink failure and recovery coverage without bypassing architecture guards.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(obs): scope unsafe allowance to seccomp test helper
Document the bounded prctl FFI safety contract and allow unsafe code only
inside the Linux fault-filter installer so workspace unsafe-code denial
continues to apply everywhere else.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(decommission): run the leader-side start to completion
A forwarded decommission start awaits `ECStore::decommission` inline in
the RPC handler. When the forwarding node's RPC times out, the stream is
reset and the handler future is dropped wherever it is. Dropped after
`pool.bin` records the pool as active but before the peers reload and
the worker is spawned, the start left the pool active with no worker and
peers on the old metadata, and a retry was rejected as already active.
An admin client that disconnects can do the same.
Run the start in its own task and let the caller await only its result.
The start then always reaches its end: it fails before persisting
anything, or persists the pool, has the peers reload and spawns the
local workers, rolling back if the reload fails. A retry still cannot
start twice: it waits on the start gate held by the running start and
either reports the busy gate or is rejected because the pool is active.
`decommission` now takes `self: &Arc<Self>` and spawns workers on that
store instead of looking up the global object store, which is the same
store in production and makes the path testable; the unused
`require_decommission_store` helper is removed.
Refs rustfs/backlog#2873
* fix(decommission): detach the leader-side start from its caller
9ac4c10de carried only the reproduction (the testable entry point, the
peer-reload pause point and the proof test). This adds the fix it
describes: `ECStore::decommission` spawns the start as its own task and
awaits only its result, so a caller that stops waiting, such as a
forwarded start whose internode RPC timed out, can no longer drop the
start between persisting the pool as active and having the peers reload
and the local worker start.
Also prove a retry while the first start runs cannot start the pool
twice, update the operator and architecture docs for the new behavior,
and shrink the error(format!) ratchet baseline for the removed helper.
Refs rustfs/backlog#2873
* fix(decommission): single-flight start, recheck pool state under gate
Review of 3a5276c5b:
- A start re-checks the pool state under the start gate before it
cancels routines, so a start racing another one can no longer cancel
the worker the other start just spawned.
- Only one start runs per instance at a time: another request, including
the retry of an operator whose request timed out, is rejected as
already running instead of stacking a second detached start and its
preflight heals. The start runs through `run_detached_mutation`, and a
failure nobody waits for any more is logged.
- Tests: the proof test pins its drop after the save and requires a
worker that persists a cancellation; the retry test really drops the
first caller; new tests cover the under-gate recheck and a peer reload
failure after the caller is gone. That last one shows the start
rollback cannot clear a fresh start because the pool metadata
stale-write guard refuses it; the gap is asserted and documented and
tracked in rustfs/backlog#2873.
- The start tests run on their own large-stack thread: the io_uring lane
(`uring_`, which also selects `during_`) runs them with libtest's 2 MiB
default stack, where the proof test overflowed.
- Docs: single-flight starts, status and cancel during preflight, the
rollback gap and its recovery, and the release requirement.
Refs rustfs/backlog#2873
* fix(decommission): turn away local start retries at the start slot
Second review of 29253430a:
- The leader-local admin pre-check refreshed rebalance metadata under
the start gate before the start slot was checked, so a local retry
waited out its 20 s gate budget behind a running start (which holds
the gate through its peer reload) and then got a 503. The pre-check
now checks the slot first and returns `DecommissionAlreadyRunning`,
which the handler reports as a 400 that keeps the reason.
- Prove the reload-failure path really ran in the detached start (the
injection is consumed), add a test that a fresh start's failed
rollback reports both the reload error and the stale-write refusal,
and assert an abandoned start stays tracked for shutdown.
Refs rustfs/backlog#2873
* fix(decommission): retry every stale target selection without failing
The tier free-version copy path counted a version as failed as soon as
a capacity failure came back and then handed the error to the entry
retry, so a retry that succeeded left the decommission failed. Return
the failure without counting it, like the other copy paths.
A selected pool can also stop fitting the mutation before admission
without going over a budget: the reservation estimate is exhausted, the
pending intent no longer fits the committed-copy budget, or a temporary
write no longer fits the headroom. Report these as "no longer fits the
selected mutation" and treat them like an over-budget pool under one
StaleTargetSelection class: retry, select again, defer once the retry
budget is spent.
Also refresh the over-budget gauge of every allocation before rejecting
growth, correct the sticky-target doc on stranded markers, and cover
each copy path, the pinned path and both retry branches with tests.
Refs rustfs/backlog#2873
* fix(decommission): name the last deferral cause in the pause reason
A round that ends with deferred entries pauses the routine with a
generic "deferred entries remaining" reason, so a pinned entry that no
longer fits its pool hides its cause in the logs. The pause reason now
appends the last deferral's error.
Keep the committed-copy check on the pending intent fatal: growth and
owner admission reject any mutation that would fail it first, so it
only fires on an inconsistent ledger.
Tests:
- a multipart part that does not fit target 1's temporary-copy
headroom is deferred and pauses with the headroom shape;
- a pinned object whose remaining allocation is below its peak pauses
with the exhausted-estimate cause, and the over-budget pinned test
checks its own cause;
- the stale-target tests assert where the pause lands: the held entry
is on neither target, the anchor is already on target 1, and target
1 holds exact inflight bytes in the multipart cases;
- share the stale-target store, barrier and pinned-copy setup.
* fix(decommission): prefer a stale selection as the pause cause
The pause reason named whichever deferral of the set finished last, so
a transient busy gate could hide another entry's permanent shortfall.
A stale target selection, which waiting does not clear, now outranks
transient contention, and the reason reads "; deferral cause: <error>".
Tests:
- a pinned object that no longer fits and a later object whose gate is
busy, in the same set, pause with the pinned cause; a unit test checks
both orders;
- the tiered and intent-conflict tests check where the pause lands, and
the tiered test asserts the known post-check failure so that fixing
the source cleanup defect flips it.
Hauser