* test(ecstore): retain final decommission capacity snapshot override
take_decommission_capacity_info_override_for_test used to pop the queue
to exhaustion, after which get_decommission_all_pool_capacity_infos
silently fell back to the host's real statfs numbers. Any new sampling
point added to the decommission start paths re-introduced that host
dependency and broke tests on some dev machines (#6989 patched one
instance by topping up snapshot counts, but the coupling remained).
Keep the final queued snapshot and replay it for every subsequent
sample so tests always observe injected capacity once an override is
installed. All existing injection patterns (single snapshot, repeated
identical snapshots, decreasing sequences ending at the post-operation
state) keep their semantics.
* test(ci): serialize load-sensitive heal and cache-generation tests
Under a heavily parallel nextest run (~792 ecstore tests), two tests of
set_disk::ops::heal::heal_result_report_tests failed nondeterministically
per round (different members each time; all 29 pass standalone). Every
test in the module builds a TempDir-backed 4-disk hermetic erasure set
and drives MiB-scale writes plus deep-scan heal: under load a single
disk's IO can fail while write quorum still holds, flipping per-disk
readback and aggregate-outcome assertions. The module's #[serial]
markers do not serialize across nextest's process-per-test boundary.
Verification also caught complete_multipart_generation_retires_cached_snapshot
failing once under the same load; it and its object.rs sibling carry
#[serial(metadata_cache_invalidation_probe)] and assert
get_object_metadata_cache generation semantics - the same shape that
forced the transition matrix tests into the serial group.
Add both families to the ecstore-serial-flaky test-group in the default
and ci profiles. Preventive serialization only, no retries. Three full
parallel rounds after the change: 792/792 passed each round.
A completed multipart upload's staging cleanup prunes empty parent
directories up to the volume root, which removes shared prefixes such as
`data-movement/` and the per-object `<sha>/` while a concurrent
new_multipart_upload builds its destination chain below them. The writer
holds a descriptor to the pruned component, so its next handle-relative
mkdirat fails NotFound. Because rename never retried NotFound, the cleanup
fan-out failed several disks in the same window and broke write quorum.
Give rename preparation its own retry rule: a NotFound is retried once per
component below the base directory, so a rebuilt walk outlasts a pruning
walk, which removes ancestors monotonically upward and stops at the base.
A destination whose parent is the base keeps NotFound terminal, so
speculative cleanup renames still fail fast, and the base is only ever
opened, never created, so a genuinely missing base still fails. The rename
itself keeps its own budget and its unchanged NotFound-is-terminal rule.
The mixed-version rolling upgrade suite asserted a single list_objects_v2
snapshot seconds after restarting a node. Peers keep a restarted node's
drive in Suspect/Returning for ~probe_interval(2s) x success_threshold(3),
and while one drive is excluded the strict listing quorum (write quorum,
3 of 4) drops objects that were themselves legally written at 3/4 during
an earlier node's identical post-restart window, under-counting the
listing (observed as 254 vs 258 in CI) even though every object still
GETs correctly. Replace the snapshot asserts with a bounded convergence
poll; a real upgrade data-loss regression still fails after the deadline.
test(ecstore): assert inline fanout gate on deterministic scheduled metric
non_inline_data_read_early_stop_does_not_add_inline_fanout_on_unequal_layout
compared disk_call_counters::KIND_READ_VERSION totals between the two-phase
read-plan gate being off and on. That counter records inside each spawned
fanout task, so the single-pending inline hedge read races the early-stop
abort_all(): whether the hedge task gets its first poll before cancellation
decides a 4-vs-5 count per read. Under concurrent nextest load the two reads
can disagree (reproduced locally at ~5% when run beside one other test,
matching the CI failure on PR #6961).
Assert on the rustfs_io_get_object_metadata_fanout_scheduled histogram
instead, which records the scheduling decision synchronously in the fanout
loop and is deterministic, using the CapturingRecorder + current-thread
runtime pattern already used by the neighboring tests in this module.
Add an authenticated scanner usage-state reset endpoint that publishes a fenced bootstrap marker for full rebuilds instead of requiring operators to delete usage metadata by hand.
Guard the reset with the scanner leader lock, storage publication epoch, and per-slot revision preconditions, and make startup resumable across stale cleanup leftovers while still rejecting newer conflicting usage state.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Lower the default scanner cache save timeout so the derived usage persistence budget stays inside the effective distributed publication lease window.
Add focused regressions for the default publication budget and bootstrap-pending observational baselines, and update operator docs with the new default.
Co-authored-by: heihutu <heihutu@gmail.com>
A replicated version purge reaches the peer without the governance
bypass header, so a GOVERNANCE-retained version deleted on the source
with x-amz-bypass-governance-retention was rejected by the peer's WORM
deletion gate forever: retryStats ended at a permanent failed count and
the sites stayed diverged (issue #6850).
The source is authoritative for such a purge: the same WORM gate
already ran there, and GOVERNANCE retention with an authorized bypass
is the only lock state it can purge through. The peer's commit-time
deletion gate now treats an authorized replication delete addressed to
an explicit version as carrying that judged bypass, reusing the same
trust judgment as the replication write exemption
(ObjectOptions::replication_request, set only after the handler
authorized ReplicateDeleteAction). COMPLIANCE retention and legal hold
keep blocking replicated purges, and a plain client delete without the
bypass header stays rejected.
Under the cargo test fallback (threads in one process), tests that touch
the process-global scanner cycle recovery status or the global usage-save
metrics raced each other and failed randomly in full-suite runs.
Mark all touchers with #[serial] per docs/testing/README.md:
- 22 tests reading or writing scanner_cycle_recovery_status() via
load_scanner_cycle_state_for_startup / reset_scanner_cycle_recovery
- 24 tests mutating global_metrics() usage-save counters via
store_data_usage_in_backend*, which raced the existing serial
test_deferred_usage_save_keeps_last_real_save_metric
No-op under nextest, which isolates each test in its own process.
Align object-capacity refresh defaults with the production-oriented environment values and keep docs, script examples, and tests in sync.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): require verification source for degraded GET
Fail closed when reconstruction has only an exact decode quorum, because no surplus source remains to validate the rebuilt data. Cover both erasure engines and the data-shards-only rollout gate.
Co-authored-by: heihutu <heihutu@gmail.com>
唐小鸭