Cover the committed MRF manifest path through LocalDisk conditional CAS when the metadata directory fsync fails. The fixture proves the previous manifest anchor survives rollback, an unanchored first successor is removed, and the legacy MRF journal remains readable even while global durability is relaxed.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): attest fresh multi-pool bootstrap per pool creator
A fresh deployment whose pools have their first endpoint on different nodes could never publish its initial pool.bin: each node held fresh-bootstrap proof only for the pool it formatted, combine_across_pools collapsed the deployment-wide proof to None, the elected writer never wrote a pending identity, and startup died with "no durable bootstrap identity or pool.bin replica is available" once the init retry budget ran out.
Track first-hand bootstrap authority per pool. The first pool's creator mints the pending cluster identity on the pool it created, every other creator copies that nonce-bound identity onto the pool it formatted first-hand (a scoped identity write that only ever touches pools the process holds first-hand proof for), and the elected writer publishes pool.bin once it holds first-hand proof for pool 0 and every pool replica carries the same pending identity. Fresh + None is still never promoted, corrupt or disagreeing replicas still fail closed, an initialized deployment never reopens bootstrap for an expansion pool, and an elected restart without first-hand proof still cannot reuse a complete pending set.
Startup classification no longer latches the pool-metadata write gate for the two transient outcomes a healthy bootstrap passes through (a non-elected node waiting for the elected writer, the elected writer waiting for the other creators); recover_pool_meta_transaction never clears write_blocked, so a non-elected node that started before pool.bin existed stayed write-blocked for the life of the process. Genuine recovery-required states still latch.
Refs rustfs/backlog#2375, rustfs/backlog#2338
* fix(lock): contain remote lock RPC timeout storms
A lock RPC deadline evicted the shared internode HTTP/2 channel and re-dialed it unconditionally, so one slow lock endpoint produced a cluster-wide RST_STREAM / GOAWAY too_many_resets / reconnect loop (rustfs#7363).
The remote lock client now keeps a per-peer channel history: a timeout evicts only when the peer has completed no lock RPC for two deadlines, evictions and transport-failure re-dials are rate limited per peer (RUSTFS_OBJECT_LOCK_RPC_EVICTION_COOLDOWN_MS, default 5 s), and a timed-out request is detached instead of cancelled, bounded per peer by RUSTFS_OBJECT_LOCK_RPC_DETACHED_LIMIT (default 256) and by the internode RPC timeout; a lock granted after its caller gave up is released immediately. Unlocks that fail the quick retries continue on a deferred 1/2/4/8/16 s schedule before the server lease reclaims them. New rustfs_remote_lock_* metrics cover timeouts, evictions, suppressed evictions, detached streams, late completions and late releases per peer; docs/operations/lock-rpc-storm-protection.md documents the policy.
Refs rustfs/backlog#2375, rustfs#7363
* ci: refresh nightly test selection digests
The replication nightly membership guard expected the 68-test digest from #7422 while the current listing has 71 tests (additions only: test_bucket_replication_sse_c_compressed_passthrough from #7366, matrix_mint_own_version_ids_addresses_mutations_through_the_ledger and matrix_removed_replication_config_abandons_pending_purge from #7368), and the cluster fault lane expected 50 tests while #7374 added test_cluster_root_heal_recovers_remote_shards_after_background_target_crash. Both lanes have failed before running a single test since 2026-09-07. Bind the Linux digests to the listings from scheduled run 34187469350 and the Darwin e2e-nightly digest to the matching local listing.
Refs rustfs/backlog#2375
Retry heal-control RPCs once after transport auth rejects a stale replay-scope epoch, and add a distributed EC8+4 restart heal evidence case that rebuilds a replaced drive with exact shard/body assertions.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(observability): expose pool write-block diagnostics
* fix(error): merge equivalent api message branches
Combine the MaxVersionsExceeded and internal IO message branches so Clippy no longer flags identical if blocks while preserving the existing response messages.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): cleanup consumed MRF replay journals
Do not retain Accepted or Merged replay intents as startup anchors after they have been handed to the heal manager. Only refused or still-pending replay records keep the journal on disk until a successor snapshot can persist them.
This keeps successor snapshots limited to the pending queue, which lets successful replay remove both authoritative and legacy journal paths and restores the crash-boundary tests around successor flush.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit d5b8f49c9d)
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* fix(replication): resolve drifted replicas via a target version ledger
A replication target that mints its own version ids (Wasabi, AWS S3)
never answers to the source uuid, so every version-addressed mutation
after the initial PUT failed forever: permanent version deletes answered
NoSuchVersion every heal cycle, and tag / retention / legal-hold updates
re-PUT the object, minting one more target version per update
(rustfs/backlog#2340).
Record the id the target assigned as a per-target ledger on the source
version (replication-target-version-<arn>, written through the existing
status writeback) and resolve every later mutation through it: version
deletes DELETE the ledger id, metadata updates go through the
metadata-only Object Lock and tagging APIs. Replicas written before the
ledger existed are located by exact key and ETag, minus the candidates
other generations of the key already claim through their own ledgers; an
ambiguous remainder is refused with a backoff instead of guessed, since a
wrong pick would destroy a live generation. A fresh write never consults
content identity. NoSuchVersion on a version-addressed DELETE counts as
purged.
The fake target gains the Wasabi shape (404 NoSuchVersion on an unknown
id, per-version Object Lock APIs) and the matrix covers the three
mutation classes plus the same-bytes generation case.
* fix(scanner): drop the unused Digest import
Same one-line change as rustfs/rustfs#7366 (main is red with it under -D warnings); carried here so the stacked PRs' merge commits compile until that fix lands.
* fix(admin): probe replication-check mutations by the assigned version id (#7373)
On a target that mints its own version ids the DeleteMarker and
VersionDelete phases of ?replication-check were skipped: they addressed
the source id, which such a target never had. The replication worker now
addresses the id the target assigned (the target-version ledger), and
the probe already holds that id from its own PUT, so run both phases
against it. VersionFidelity keeps failing with the mismatch code and the
target stays FAILED; the phases report whether ledger-addressed purges
work against this endpoint (rustfs/backlog#2340).
* fix(replication): abandon purges to targets the bucket no longer names (#7377)
* fix(admin): probe replication-check mutations by the assigned version id
On a target that mints its own version ids the DeleteMarker and
VersionDelete phases of ?replication-check were skipped: they addressed
the source id, which such a target never had. The replication worker now
addresses the id the target assigned (the target-version ledger), and
the probe already holds that id from its own PUT, so run both phases
against it. VersionFidelity keeps failing with the mismatch code and the
target stays FAILED; the phases report whether ledger-addressed purges
work against this endpoint (rustfs/backlog#2340).
* fix(replication): abandon purges to targets the bucket no longer names
A permanent version delete whose replication keeps failing stays in
xl.meta as a PENDING purge, hidden from listings, until every target
confirms it. Once the operator removes the replication configuration or
the rule naming that target nothing ever confirms it: the heal path
derived its delete decision from the configuration (the decision string
is not persisted) and skipped the version forever, so DeleteBucket
answered BucketNotEmpty for a residue the client could neither list nor
remove (rustfs/backlog#2340).
Owe a version purge to the targets its purge state names, let the heal
path through without a configuration, and have the delete worker settle
a target the configuration no longer names as abandoned: the purge is
reported complete locally through the normal writeback, the replica on
the former target is left alone, and the event
replication_purge_abandoned plus a counter are the record.
* fix(admin): send replication-check marker creation without a version id
Running the DeleteMarker / VersionDelete phases on a target that mints
its own version ids exposed two probe-shape bugs on real Wasabi:
- the DeleteMarker phase put the assigned version id on its DELETE. A
RustFS peer reads the source-deletemarker header and creates a marker,
but a generic S3 target executes it as a permanent delete of the probe
version, so VersionDelete then answered NoSuchVersion. Use the same wire
shape as live delete replication: no versionId on a marker creation.
- cleanup treated NoSuchVersion on the version the VersionDelete phase had
already removed as a failure (RustFS/MinIO answer 204 there).
Also gate the no-configuration heal pass-through for pending purges on a
purge state that actually names targets, so a purge without a recorded
target keeps the ordinary skip (scanner unit test), and merge origin/main
(#7365 settles the pool-metadata probe test that failed in CI).
---------
Co-authored-by: houseme <housemecn@gmail.com>
* fix: reject unsupported pool expansion with actionable errors
Report singleton-pool and persisted-topology constraints before misleading startup retries. Preserve single-node multi-drive admission and existing parity policies, and cover format preservation plus operator recovery guidance for issue #6186.
* fix: keep pool layout errors typed
Preserve actionable pool layout diagnostics without adding generic formatted errors. Tighten the shrink-only baseline and assert that both typed payloads survive the I/O boundary.
---------
Co-authored-by: houseme <housemecn@gmail.com>
* fix(scanner): remove unused digest import
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): add raw page owner index (#7375)
* feat(scanner): add raw page owner index
Add a serializable raw enumeration page owner index for scanner resume work.
The index exposes unsupported, building, and ready states, validates committed page identity by recomputing digests, and uses generation checks for CAS-style page commits.
Focused tests cover small-budget restart progress, page digest/source drift rejection, corrupt deserialized state, CAS failure, precommit crash, empty sources, and invalid entry boundaries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): persist raw page owner resume state (#7379)
Wire the scanner raw enumeration partial-cache writer to the raw page owner index so interrupted bucket walks can retain validated page-builder state across scanner restarts.
Keep complete owner sources terminal-only, add partial-source ingestion for in-progress raw directory reads, and validate the persisted page index through bucket checkpoint preparation.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): fence segment producer observations (#7381)
Require the segment observation fixture to carry source, incarnation, key-format, baseline, process epoch, generation-window, gap, overflow, and producer-coverage proof before accepting a narrowed proposal. Keep the diagnostic path fixture-only and remove its ordinary stderr output.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): isolate pool metadata read probes (#7367)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF crash successor matrix (#7369)
* test(heal): cover MRF crash successor matrix
Add process-boundary MRF replay coverage for the successor snapshot window after a retained startup journal is flushed but before cleanup deletes it. Extend the mixed authoritative/legacy reader fixture with a scoped v2 journal epoch to pin the no-merge contract.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover service-kill MRF replay (#7380)
Add a Unix process fixture that waits after publishing the pending MRF successor snapshot, then is terminated by the parent before restart replay.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover transport-lost start receipts (#7371)
Add gRPC transport fault fixtures for heal-control start admission. The tests distinguish pre-admission transport loss from post-admission response loss, then verify exact envelope retries reuse the canonical receipt while fresh forceStart requests create distinct tasks.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add crash-restart heal evidence case (#7370)
* test(scanner): add crash-restart heal evidence case
Add a distinct W21 background target crash case to the scanner/heal evidence registry and oracle path.
Keep the existing restart lane on graceful process restart, keep the crash lane on hard kill, and make the wiring checker reject evidence/oracle mismatches.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): support older Python wiring checks
Let the scanner/heal evidence wiring checker run under Python 3.9/3.10 by falling back to tomli and chunked SHA-256 hashing when the Python 3.11 standard APIs are unavailable.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): reject stale raw page source seeds (#7382)
Do not prefill a resumed raw page owner with previously indexed entries when starting a new raw directory observation pass. The next pass must observe the same prefix again before the page index can advance; otherwise the index is discarded fail-closed.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): defer raw page revalidation until observed (#7384)
A resumed raw page owner index must not prefill entries from older cache state, but it also must not discard a valid multi-entry index before the current raw directory pass has observed enough entries to prove identity. Track the persisted index floor and only run the strict owner identity check once the current pass reaches that floor.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): drop the unused Digest import
* fix(replication): carry the compression layout through SSE-C passthrough (#7372)
* fix(replication): queue an in-flight version only once (#7376)
Align the suspended-owner heal regression with read-only pool metadata admission semantics. The quorum-boundary case now asserts that heal fails the current capacity admission without latching the global pool metadata writer.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Evaluate read-only pool metadata planning probes through an isolated write-state clone so transient unreadable replicas fail the current admission without permanently latching the shared writer gate.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Restore the previous control-file bytes, or remove a newly created file, when the Unix compare-and-update path reaches the rename but then fails to fsync the parent directory. This keeps failed metadata CAS publications from advancing recovery anchors.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
The never-versioned listing fast path emitted any non-empty directory
without xl.meta as a prefix, which surfaced the data dirs deleted
versions leave behind as phantom folders that HEAD, listing and
recursive remove could not touch. Probe such directories for the
delete-residue shape before emitting them, and let any complete empty
first page of a prefix listing trigger the existing fail-closed orphan
purge so ordinary browse and remove traffic reclaims committed residue.
The admin storage walk probed a set's drives one after another, each
bounded by the disk_info timeout, so a few drives still recovering
after a power cut pushed the local snapshot past the peer probe budget
and healthy peers rendered as unknown. Probe all drives at once so the
walk costs one timeout at most, and add a test-only probe delay hook to
pin that bound.
overtrue