* fix(scanner): remove unused digest import
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): add raw page owner index (#7375)
* feat(scanner): add raw page owner index
Add a serializable raw enumeration page owner index for scanner resume work.
The index exposes unsupported, building, and ready states, validates committed page identity by recomputing digests, and uses generation checks for CAS-style page commits.
Focused tests cover small-budget restart progress, page digest/source drift rejection, corrupt deserialized state, CAS failure, precommit crash, empty sources, and invalid entry boundaries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): persist raw page owner resume state (#7379)
Wire the scanner raw enumeration partial-cache writer to the raw page owner index so interrupted bucket walks can retain validated page-builder state across scanner restarts.
Keep complete owner sources terminal-only, add partial-source ingestion for in-progress raw directory reads, and validate the persisted page index through bucket checkpoint preparation.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): fence segment producer observations (#7381)
Require the segment observation fixture to carry source, incarnation, key-format, baseline, process epoch, generation-window, gap, overflow, and producer-coverage proof before accepting a narrowed proposal. Keep the diagnostic path fixture-only and remove its ordinary stderr output.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): isolate pool metadata read probes (#7367)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover MRF crash successor matrix (#7369)
* test(heal): cover MRF crash successor matrix
Add process-boundary MRF replay coverage for the successor snapshot window after a retained startup journal is flushed but before cleanup deletes it. Extend the mixed authoritative/legacy reader fixture with a scoped v2 journal epoch to pin the no-merge contract.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover service-kill MRF replay (#7380)
Add a Unix process fixture that waits after publishing the pending MRF successor snapshot, then is terminated by the parent before restart replay.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(heal): cover transport-lost start receipts (#7371)
Add gRPC transport fault fixtures for heal-control start admission. The tests distinguish pre-admission transport loss from post-admission response loss, then verify exact envelope retries reuse the canonical receipt while fresh forceStart requests create distinct tasks.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): add crash-restart heal evidence case (#7370)
* test(scanner): add crash-restart heal evidence case
Add a distinct W21 background target crash case to the scanner/heal evidence registry and oracle path.
Keep the existing restart lane on graceful process restart, keep the crash lane on hard kill, and make the wiring checker reject evidence/oracle mismatches.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): support older Python wiring checks
Let the scanner/heal evidence wiring checker run under Python 3.9/3.10 by falling back to tomli and chunked SHA-256 hashing when the Python 3.11 standard APIs are unavailable.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): reject stale raw page source seeds (#7382)
Do not prefill a resumed raw page owner with previously indexed entries when starting a new raw directory observation pass. The next pass must observe the same prefix again before the page index can advance; otherwise the index is discarded fail-closed.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): defer raw page revalidation until observed (#7384)
A resumed raw page owner index must not prefill entries from older cache state, but it also must not discard a valid multi-entry index before the current raw directory pass has observed enough entries to prove identity. Track the persisted index floor and only run the strict owner identity check once the current pass reaches that floor.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Evaluate read-only pool metadata planning probes through an isolated write-state clone so transient unreadable replicas fail the current admission without permanently latching the shared writer gate.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): retain final decommission capacity snapshot override
take_decommission_capacity_info_override_for_test used to pop the queue
to exhaustion, after which get_decommission_all_pool_capacity_infos
silently fell back to the host's real statfs numbers. Any new sampling
point added to the decommission start paths re-introduced that host
dependency and broke tests on some dev machines (#6989 patched one
instance by topping up snapshot counts, but the coupling remained).
Keep the final queued snapshot and replay it for every subsequent
sample so tests always observe injected capacity once an override is
installed. All existing injection patterns (single snapshot, repeated
identical snapshots, decreasing sequences ending at the post-operation
state) keep their semantics.
* test(ci): serialize load-sensitive heal and cache-generation tests
Under a heavily parallel nextest run (~792 ecstore tests), two tests of
set_disk::ops::heal::heal_result_report_tests failed nondeterministically
per round (different members each time; all 29 pass standalone). Every
test in the module builds a TempDir-backed 4-disk hermetic erasure set
and drives MiB-scale writes plus deep-scan heal: under load a single
disk's IO can fail while write quorum still holds, flipping per-disk
readback and aggregate-outcome assertions. The module's #[serial]
markers do not serialize across nextest's process-per-test boundary.
Verification also caught complete_multipart_generation_retires_cached_snapshot
failing once under the same load; it and its object.rs sibling carry
#[serial(metadata_cache_invalidation_probe)] and assert
get_object_metadata_cache generation semantics - the same shape that
forced the transition matrix tests into the serial group.
Add both families to the ecstore-serial-flaky test-group in the default
and ci profiles. Preventive serialization only, no retries. Three full
parallel rounds after the change: 792/792 passed each round.
* Revert "perf(ecstore): use AHashMap for FileInfo metadata fields (#6738)"
This reverts commit 13a2ae212e.
* fix(filemeta): restore standard HashMap metadata (#6742)
Remove the direct ahash dependency added for FileInfo metadata and revert the affected filemeta/ecstore call sites back to std::collections::HashMap.
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* test(pool): fix warp log path and retry rebalance start
- warp writes now use a unique mktemp log file instead of a fixed
/tmp/rustfs-warp.log: the runner user could not write the stale
root-owned file, which made the background warp process die instantly
(warp never ran). The workflow uploads /tmp/rustfs-warp.*.log.
- rebalance start is retried (6x, 20s apart): nightly builds gate
rebalance activation on a live cross-pool fence fleet capability proof
that takes ~10-20s to re-establish after a pool joins. Verified live:
attempt 1 fails with 500 'pool activation requires a live fleet
capability proof', attempt 2 succeeds.
* test(pool): annotate known server-side issues in failure output
When a node fails to start, grab the rustfs journal tail and match known
server-side error signatures (e.g. the fleet capability proof cold-start
regression, rustfs/backlog#2031), printing a hint with the tracking issue.
Also annotate the rebalance-start retry exhaustion and the rc.3 decommission
metacache-listing failure with actionable guidance.
* fix(ecstore): defer rebalance activation without fleet proof
---------
Co-authored-by: 马登山 <cxymds@qq.com>
Co-authored-by: cxymds <cxymds@gmail.com>
* fix(ecstore): fence pool metadata replica updates
* fix(ecstore): block decommission on unsafe pool metadata
* fix(ecstore): block writes after pool metadata save errors
* fix(ecstore): latch pool metadata writes before await
* fix(scanner): bootstrap pristine usage baseline
* fix(ecstore): fence pool metadata replica updates
* fix(ecstore): block decommission on unsafe pool metadata
* fix(ecstore): block writes after pool metadata save errors
* fix(ecstore): latch pool metadata writes before await
* fix(ecstore): persist unresolved decommission entries
* fix(ecstore): type decommission completion result
* fix(ecstore): allow intentional decommission listing signatures under strict clippy
The sftp/swift feature-matrix clippy gates run with -D warnings and
flag the unresolved-entry resolver (large Err payload by design, 8
context parameters) and the decommission listing driver (9 args).
Document why and align with the existing decommission_entry precedent.
cargo fmt --check has been failing since #6410 landed: the merged
decommission import groups in core/pools.rs were not canonical
rustfmt output. Apply formatting verbatim, no code changes.
A single object's SourceChanged during decommission cleanup no longer
cancels the shared worker token and fails the whole pool operation.
Cleanup preflight and source-cleanup outcomes now retry per entry with
bounded attempts and cancellation-aware backoff, applied uniformly to
ordinary versions, delete markers, and tiered copies (removing the
try-once-only branches); every retry re-lists the entry and redoes
version multiset validation before touching the source. Only quorum
loss, unrecoverable system errors, or exceeding a pool-level
SourceChanged exhaustion threshold still fails the decommission, and
exhausted entries never delete their source versions.
Retry attempts, backoff, and deferred/exhausted reasons are logged per
entry for observability. Heavy regression tests spawn on dedicated
32MiB stacks following the existing store-test pattern.
Fixesrustfs/backlog#1913
Co-authored-by: houseme <housemecn@gmail.com>
houseme