Commit Graph

446 Commits

Author SHA1 Message Date
overtrue 93a61df1cf chore(gateway): merge main into integration
Keep native publication recording in main's native_io helpers and resource
writers, and adapt merged tests to the owned Body, Bytes write input, and
fallible find_disk.
2026-10-31 06:15:49 +08:00
Chris An 0e5fe8c658 feat(storage): share physical admission across scanner and heal (#8911)
* feat(scanner): share node admission with background heal

* fix(scanner): drain candidates after listing completion

* fix(scanner): yield cache leases for shared node admission

* test(scanner): attest adaptive admission in process receipts

* fix(concurrency): declare Tokio time dependency

* feat(concurrency): retain bounded physical resource subsets

* fix(scanner): retain resource ownership in native filesystem jobs

* feat(concurrency): acquire physical groups before node work

* feat(scanner): gate background reads by physical device

* feat(concurrency): combine preallocated resource selections

* feat(scanner): share physical topology across disk plans

* fix(storage): retain only native disk resource subsets

* feat(storage): admit auxiliary physical resources without waiting

* feat(heal): admit physical resources before object repair

* fix(storage): retain resources during native format probes

* fix(storage): retain resources through direct writer I/O

* fix(storage): retain resources through buffered writer I/O

* fix(storage): retain physical resources through streamed reads

* fix(storage): retain resources through native filesystem jobs

* fix(storage): retain physical ownership during bitrot reads

* fix(ecstore): retain resources during positioned reads

* fix(ecstore): retain staged metadata write ownership

* fix(ecstore): retain native io_uring read ownership

* fix(ecstore): enforce receiver-side background admission

* fix(ecstore): retain resources through whole-file native I/O

* fix(ecstore): retain physical ownership during durable rename sync

* fix(storage): account for retained background stream buffers

* fix(storage): retain memory charges for background read results

* fix(storage): account for native io_uring read allocations

* fix(storage): retain memory charges for direct-write buffers

* fix(ecstore): account for heal and RPC reader buffers

* test(scanner): verify samples and clean concurrent collectors

* fix(storage): retain inline and HTTP writer memory reservations

* perf(rio): reuse reserved HTTP writer storage

* fix(rpc): reserve background scanner pipe storage

* perf(rpc): reuse reserved response reader storage

* feat(scanner): measure cache copy allocation requests

* fix(rpc): budget namespace scanner responses

* fix(scanner): retain the enabled supervisor task handle

* fix(scanner): budget namespace frame read staging

* fix(scanner): account namespace envelope decode storage

* fix(heal): retain and join the MRF consumer on shutdown

* test(heal): isolate process-global MRF consumer fixture

* fix(heal): join manager workers during shutdown

* test(heal): verify shutdown at native read boundaries

* fix(heal): drain channel commands before stopping MRF

* fix(heal): admit resume scans and retain replacement workers

* build: pin profiling sources and trust fixed ownership dependencies

* fix(heal): yield node admission while accepted native work drains

* fix(startup): retain shutdown work and preserve deadline failures

* fix(heal): admit inventory probes and retain replacement work

* fix(shutdown): preserve maintenance and frontend join failures

* fix(shutdown): retain HTTP child tasks through real retirement

* fix(shutdown): join actual TLS reload work before retirement

* test(multipart): preserve failures before fault barriers

* fix: keep physical admission inside repository guards

Route the multipart lock diagnostics filter through the e2e storage
compatibility module, reuse the parent module's s3s body aliases in the
namespace memory test, and share one Reed-Solomon verify error mapping
across the reserved-buffer and default paths.

* fix: reconcile physical admission with main's new call sites

Main added a source-admission metadata write layer, a CAS write fence,
fixture writers and scanner handoff tests against the old borrowed-slice
and node-permit shapes. Pass owned Bytes through the new metadata layer,
answer the scanner handoff with ScannerNodePermit, gate the test-only
disk lookup re-export, and clear the lint findings that stable 1.99
reports for the shutdown and admission code without suppressions.

* test(concurrency): name reversed attachment groups clearly

* fix(ecstore): box disk operation futures below admission layers

Every LocalDiskWrapper operation reaches the LocalDisk future through the
health, admission and timeout layers (track_cleanup_mutation, the
track_disk_health family, the io-permit and native-resource admission
split, and track_admitted_disk_operation). Each layer embedded that future,
and unoptimized builds copy it into every awaiting frame, so one local
delete cost about 1.3 MiB of stack and 16 tests overflowed libtest's
default 2 MiB stack.

Box the operation future once where track_admitted_disk_operation creates
it, through a non-async helper so the temporary never stays in a polled
frame. The layers above now hold only a pointer; the cost is one
allocation per local disk operation.

* fix(heal): box object heal futures below admission

The admission wrappers took the object heal future by value, so unoptimized
builds copied it into three awaiting frames, about 650 KiB per heal. Create
it through a non-async helper instead, leaving only a pointer in those frames.

* test(ecstore): count the I/O backend's root FD owner
2026-10-31 01:00:07 +08:00
Chris An e46877dd1c feat(scanner): certify cold peer buckets during hot writes (#8970)
* feat(scanner): certify cold peer buckets before scoped ACK

* test(e2e): keep peer evidence within JSON macro limits

* test(scanner): requalify mixed peers after rolling setup

* test(scanner): prove old-worker coverage with a fixed coordinator
2026-10-31 00:36:26 +08:00
Chris An 3fc610c8a2 fix(scanner): track heal generation and payload observations (#9008)
* fix(scanner): distinguish missing heal identity fields

* test(scanner): simplify identity case types

* fix(scanner): track heal generation and payload observations

* fix(scanner): treat heal results as observations

* test(scanner): route heal integration through storage boundary

* test(scanner): scope the heal generation fixture

* test(heal): fill observation fields in the retained outcome fixture
2026-10-31 00:19:13 +08:00
Chris An 4992034cc2 perf(scanner): batch proportional pacing in folder walkers (#8850)
* fix(heal): measure admission lock wait and hold separately

* fix(heal): exclude force-start cancellation from lock hold

* test(ecstore): drain guarded delete fixture rename tails

* feat(scanner): count object metadata visits

* test(ecstore): drain guarded delete fixture rename tails

* perf(scanner): batch proportional pacing in folder walkers

* test(ilm): isolate fixture recovery loop ownership

* test(scanner): drop the retired update cache from pacing fixtures

---------

Co-authored-by: Hauser <housemecn@gmail.com>
2026-10-31 00:08:10 +08:00
Chris An 7d45685d70 feat: classify scanner metadata calls by diagnostic prefix (#9052)
* feat: classify scanner metadata calls by diagnostic prefix

* fix: export zero scanner visit counts for diagnostic scope
2026-10-31 00:07:38 +08:00
Chris An 8ece50ff8c fix(scanner): report incomplete collector observations (#9012) 2026-10-31 00:06:59 +08:00
Chris An 7f13ebe2a5 fix(scanner): align unversioned overflow heal options (#9036)
fix(scanner): protect unversioned overflow heals
2026-10-31 00:06:47 +08:00
Chris An 696dc1496e fix(scanner): distinguish missing heal identity fields (#9000)
* fix(scanner): distinguish missing heal identity fields

* test(scanner): simplify identity case types
2026-10-31 00:06:35 +08:00
overtrue b4abe64ada chore(gateway): merge main into integration
Sync main 84e24e767 into gateway/integration. Ports main-side additions to
the integration boundaries: the tier deletion diagnostics admin handlers move
to the owned admin request/response types, and the new lifecycle rule checks
in ecstore use the persisted lifecycle types instead of s3s DTOs. Keeps the
gateway git source in deny.toml while dropping the smithy-rs pin main retired,
and refreshes the test-count baseline for the merged tree.
2026-10-29 16:01:13 +08:00
cxymds 4a63e53161 test(validation): run canonical cases and build pinned Heal tools (#8455)
* test(validation): add pinned auto-testing manual lane

* Install pinned ILM client for canonical validation lane

* Prepare Python client for pool validation cases

* Install validation client for replication cases

* test(validation): provide the Python client for Heal cases

* test(validation): build the candidate-bound Heal inspector artifact

* test(validation): expose tool artifacts through the existing manual lane

* test(validation): verify pinned Heal tools before fleet execution

* test(validation): verify Heal tools against released shard captures

* ci: audit retained validation environments before finalization

* test(validation): add optional native Heal regression tooling

* test(validation): route native tools through the existing driver

* Run validation tool resolution on hosted Linux without acquiring the fleet lock

* test(validation): bind C07 to pinned native authority evidence

* test(validation): build optional C10 fixture codec on hosted Linux

* test(ci): bind C10 execution to the verified shared codec

* feat(ci): expose read-only validation fleet inventory

* docs(ci): describe supported validation recovery

* test(ci): add isolated manager native validation builds

* test(ci): expose manager regressions through tools-only dispatch

* ci(validation): consume pinned manager native prerequisites

* fix(heal): preserve typed bucket errors for bounded retries

* fix(heal): preserve typed errors and SSE-C response headers

* ci(validation): publish isolated immutable candidate packages

* test(validation): enforce private candidate manifest schema

* test(validation): verify isolated package publication failures

* ci(validation): dispatch isolated packages through the driver

* fix(heal): deeply verify read repair payload failures

* test(ecstore): verify incarnation fence at repair commit

* ci(validation): verify C11 native storage evidence

* fix(ci): isolate validation package publication credentials

* test(ci): archive scoped F07 directory fsync evidence

* fix(ecstore): retain typed quorum errors for missing parts

* chore(ecstore): shrink typed-error ratchet baseline

* fix(ecstore): retain below-quorum data during no-remove heal

* fix(ci): allow recovery of completed cancelled validation runs

* fix(heal): retain exhausted admin batch repair receipts

* fix(observability): retain typed object error diagnostics

* fix(s3): map typed RPC unavailability to service refusal

* fix(filemeta): classify supported inline framing damage as corrupt

* test(ecstore): exercise heal during committed overwrite cleanup

* ci(validation): admit source-bound G01 native evidence before fleet execution

* test(ecstore): fence version races and bind cleanup slots

* ci(validation): require complete race native runtime preflight

* test(ecstore): fence marker and batch deletes during heal

* ci(validation): require G04 native and runtime admission

* test(ecstore): fence multipart continue complete and abort during heal

* test(ecstore): cover default heal owners and copy metadata races

* ci(validation): admit source-bound owner and copy race proofs

* test(scanner): assert typed corruption after metadata truncation

* fix(ci): require complete native admission for multipart races

* test(validation): wire I05 queued-successor native prerequisites

* fix(scanner): bind object heals to the scanning disk scope

* test(scanner): reuse storage boundary for scope regression

* style: format RPC digest precedence assertion

* fix(heal): query automatic tasks on their owning node

* test(admin): reuse error facade in heal query regression

* fix(heal): retain cancellation after successful bucket storage I/O

* fix(ci): retain validation evidence outside runner temp

* fix(heal): retain resolved object identity on terminal failures

* test(scanner): adapt scope regression to admission receipts

* ci(validation): inspect shared runner without fleet access

* ci(validation): prepare Python before creating client environments

* ci(validation): isolate each client virtual environment

* ci(validation): retain interrupted evidence without fleet recovery

* ci(validation): bind recovery to retained runner evidence

* ci(validation): wire persistence and lock source plans

* fix(validation): align read-only dispatch defaults

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
2026-10-29 08:50:42 +08:00
overtrue add363f084 chore(gateway): merge main into integration
Sync main 9601a7c4b into gateway/integration. Ports main-side additions to the
owned admin boundary (plugin delegation/attempt handlers, notification target
edit tests), drops a new ecstore s3s::dto reference in favour of the persisted
lifecycle filter, and refreshes the e2e selection digests and test-count
baseline for the merged tree.
2026-10-29 07:31:24 +08:00
RustFS 7581a70412 feat(s3): serve Veeam SOSAPI system and capacity XML (#9007)
* feat(s3): serve Veeam SOSAPI system and capacity XML

Authenticated GET and HEAD of the two well-known SOSAPI keys return
generated XML, and read preconditions use that generated ETag and
Last-Modified. Mutations of those keys are rejected while the feature
is on.

Co-authored-by: RustFS <hello@rustfs.com>

* fix(s3): simplify Veeam quota metadata lookup

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(s3): harden Veeam SOSAPI capacity and reads

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(s3): bind Veeam capacity to bucket identity

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(s3): protect virtual objects and coalesce refreshes

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): refresh Veeam profile selections

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): bound pagination fixture writes

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-28 22:04:49 +08:00
Hauser 4c997519e8 refactor(storage): replay decoded dirty journal records (#9030)
Reuse the owned frame's decoded record with bounded canonical size checks
and the existing scanner validation/application core. Keep the complete
outer parser and own the journal state fields without a JSON round trip.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-28 20:01:43 +08:00
Chris An 8e88419f3a feat(scanner): share node admission with background heal (#8903)
* feat(scanner): share node admission with background heal

* fix(scanner): drain candidates after listing completion

* fix(scanner): yield cache leases for shared node admission

* test(scanner): attest adaptive admission in process receipts

* fix(concurrency): declare Tokio time dependency

* fix(scanner): route test disk access through owner facade

* test(scanner): fix lint warnings
2026-10-28 11:57:29 +08:00
Chris An a24d305665 fix(scanner): summarize bucket root collection (#9013) 2026-10-28 04:05:44 +08:00
Chris An 3a00e9bd42 fix(scanner): add context to set cache failures (#9010) 2026-10-28 03:53:45 +08:00
Hauser be485e6cab test(scanner): add memory acceptance runner and budget probes (#8988)
* test(scanner): add memory acceptance runner and budget probes

Add owned S3 workload phases with process, endpoint, cgroup and binary
identity checks for scanner memory and scanner-disabled request bursts.
Keep scale, restart, heap-attribution and overall-acceptance limits explicit.

Characterize healthy and seeded coverage-gap cache retention through the
real folder walker and production cache codec without changing runtime code.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): bind acceptance capacity to effective storage

Validate the serving process's complete local disk inventory before load
writes and during runtime sampling. Resolve each disk inside the owned
storage root, reject nested mounts and filesystem mismatches, and bind
storage observations to phase identity for container and native runs.

Cover the unused-mount bypass, CLI-overridden environment, native paths,
symlink escapes, named volumes and stable multi-disk inventory ordering.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): fence status observations before workload writes

Close each status API window with a second storage-bound process sample
before Phase entry can authorize bucket creation or PUT requests.

Add a fail-before regression for a process change during the initial status
request, asserting no ownership or workload writes occur.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-27 16:45:37 +08:00
Hauser 11e1662c90 fix(scanner): reduce memory retention and redundant scan work (#8971)
* fix(scanner): compact completed subtrees during checkpoint sweeps

Release completed object entries while preserving resumable frontier nodes,
including heal rewalks and coverage gaps. Borrow child links during usage
aggregation and remove duplicate update-cache subtree copies.

Add real-disk restart and allocation-retention regressions for #8961.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(scanner): avoid unused progress trees and traversal copies

Bind progress caches to actual subscribers and keep durable scanner metadata
in one authoritative cache. Remove mirror-only heal synchronization, use one
tagged folder queue, and consume child ownership when deleting subtrees.

Preserve observer output, restart coverage and failed-object retry semantics.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(scanner): compact completed leaf entries in place

Keep completed-folder eligibility and frontier handling in one compaction
boundary. Borrow leaf statistics and update the cached sampling flag without
cloning, deleting, or reinserting the entry; aggregate only non-leaf nodes.

Pin allocation identity, complete metadata, failure guards and caller state
with a regression that fails under the previous compaction implementation.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-27 10:01:46 +08:00
overtrue 67fbafa13e chore(gateway): merge main into integration
Source: main 4e48c390e7
Target: gateway/integration 1000860cb8

Conflicts resolved, keeping both sides:
- crates/ecstore/src/disk/{fs,local}.rs: main's batched rollback writes,
  unified blocking write path and relocated open_file, with integration's
  native publication records restored at each publication point and the
  publication_history field kept beside main's multipart proof authority.
- crates/ecstore/src/set_disk/ops/multipart.rs: union of the imports.
- rustfs/src/storage/rpc/node_service.rs: main's MRF lifecycle reader
  capability branch; the heal capability keeps integration's restore
  copy-back probe.
- rustfs/src/admin/handlers/{audit,event,plugins_instances}.rs: main's
  target_read inventory snapshot on integration's owned request boundary;
  the new event.rs inventory test now builds owned requests.
- scripts/check_s3s_footprint.sh: integration's baseline-file version.
- .config/e2e-{full,smoke}-selection.txt: digests recomputed over the
  union of both heads' CI listings (full 700 darwin / 708 linux, smoke 232).

Also migrated main's new rebalance delete-all expiry test to the persisted
lifecycle types integration uses in ecstore, and raised the test-count
baseline to the merged counts (every decrease comes from main).
2026-10-27 08:38:17 +08:00
Jim Chen 605176fe75 fix(scanner): avoid per-set max_entries truncation before cross-set prefix usage merge (#8891)
* fix(scanner): avoid per-set max_entries truncation before cross-set prefix usage merge

compute_prefix_usage applies the caller's max_entries cap to each individual erasure set's loaded.prefix_usage(bucket, prefix, max_entries) before merging sub_prefix_map across sets, silently dropping and under-counting sub-prefix usage in multi-set deployments.

Root Cause:
In crates/scanner/src/prefix_usage.rs, compute_prefix_usage queries each erasure set with loaded.prefix_usage(bucket, prefix, max_entries) (line 178) and then merges the returned query.sub_prefixes across sets into sub_prefix_map (lines 216-218). When sets_total > 1 and the number of distinct child prefixes exceeds max_entries, any sub-prefix that ranks within the top max_entries on one set and outside the top max_entries on another set has its non-top set contributions discarded prior to merging, producing silently under-counted size, objects, versions, and delete_markers.

Fix:
Pass usize::MAX as the per-set limit when sets_total > 1 in compute_prefix_usage so each erasure set returns its full sub-prefix breakdown, and apply max_entries truncation only after merging and sorting sub_prefix_map across all sets.

Signed-off-by: czcbangkai <czcbangkai@google.com>

* test(scanner): use snake_case in prefix_usage regression test name

---------

Signed-off-by: czcbangkai <czcbangkai@google.com>
Co-authored-by: Hauser <housemecn@gmail.com>
2026-10-26 17:50:01 +08:00
Remi Cattiau 62d148b045 fix(scanner): retry unreadable cycle state in remote walk fence poll (#8939)
While a remote namespace scanner walk runs, the fence watcher reads the
persisted cycle state every 5 s and stops the walk on any error. A read
that hits the 5 s distributed-lock timeout on .bloomcycle.bin is then
reported as a fence change: the bucket scan fails, the set cache is not
published and the cycle ends without a durable usage snapshot.

Return ScannerError::CycleStateUnavailable (same message as before) when
the state cannot be read, and let the watcher tolerate up to 12
consecutive such polls before stopping, logging a warning each time and
resetting the count after a successful poll. A mismatching fence and any
other error still stop the walk at once; the fence check when the stream
opens is unchanged.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 06:09:06 +08:00
Chris An 7b29e9d65f test(scanner): verify checkpoint debt across process restarts (#8938)
test(scanner): preserve checkpoint debt across process restarts
2026-10-25 23:43:41 +08:00
Hauser 13dc635d51 feat(scanner): expose startup phases and attempt deadlines (#8931)
* feat(scanner): expose startup phases and attempt deadlines

Share one node-local startup snapshot between native status and metrics.
Track actual waits and first attempts without changing admission timing,
coverage acknowledgement, or publication semantics.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): observe startup recovery before initialization

Create the observation owner at enabled startup entry and move it through
the existing initializer, preserving admission timing and disabled cleanup.
Cover blocked recovery discovery and cancellation through the public entry.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-25 20:54:41 +08:00
overtrue 88b626e7a0 chore(gateway): merge main into integration
Source: main 0f24511535
Target: gateway/integration 8cf6187ff4

Resolve the e2e-full selection digest conflict by recomputing both
platform digests over the merged nextest listing (685 darwin / 693
linux): main adds two rpc_fault_proxy unit tests, integration keeps its
dual-stack differential and anonymous-access additions. Raise the rustfs
test-count baseline row to the merged count (5932).
2026-10-25 09:58:02 +08:00
Hauser 9517db6cf7 fix(listing): honor per-object quorum with borrowed metadata (#8901)
* fix(listing): honor persisted erasure quorum

Separate metadata admission and primary-reader sizing from each version's
persisted erasure requirements. Propagate agreed-entry uncertainty and
report LIST read-quorum failures as storage unavailability.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(listing): borrow metadata during quorum resolution

Keep validated metadata inspection and agreed-version selection in
filemeta. Resolve independent reader votes through borrowed slots and
retain owned buffers until a filtered result is required.

Preserve fixed-baseline legacy comparisons, cached latest fast paths,
quorum policy, and disk fan-out. Add ownership regressions and paired
release benchmarks for cold and cached version histories.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): remove redundant fallback clone

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-25 04:49:31 +08:00
overtrue 8cf6187ff4 chore(gateway): merge main into integration 2026-10-25 01:50:03 +08:00
Hauser 900d082764 fix(scanner): make startup responsive and skip redundant config work (#8898)
* fix(scanner): bound implicit startup delay and publish config early

Limit warm-cache startup to the smaller of the cycle interval and one
minute before jitter, while preserving explicit startup settings.
Publish configuration gauges when applying runtime settings and retain
timeout diagnostics until an actual new cycle begins.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): unify startup waits and skip unchanged configuration

Bound read-only startup inspection and make startup and pre-leadership
retries respond to cancellation and configuration changes. Preserve one
startup deadline and discover replication behind lifecycle-only buckets.
Avoid irrelevant bucket listings and repeated per-disk reconfiguration.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): retain effective startup policy and fallback metrics

Keep the sampled startup deadline when configuration changes leave its
normalized policy unchanged. Initialize retained runtime metrics after
an invalid first refresh without suppressing the validation error or
replacing a newer accepted configuration.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-25 01:01:37 +08:00
Chris e13336a116 feat(scanner): expose publication and heal observation status (#8878) 2026-10-24 20:04:57 +09:00
Chris 90c5050a0f fix(scanner): preserve unknown catch-up state on ledger failure (#8877) 2026-10-24 18:16:42 +09:00
Chris 4e64e86a29 feat(scanner): count object metadata visits (#8874)
* feat(scanner): count object metadata visits

* test(ecstore): drain guarded delete fixture rename tails
2026-10-24 17:24:17 +09:00
Hauser 0c8b05585e fix(scanner): classify erasure candidates in one budgeted walk (#8869)
* test(scanner): isolate async checkpoint log capture

Scope captured tracing subscribers to each future poll and cover the
competing-dispatcher failure without weakening checkpoint assertions.
Verify exact bounded erasure-probe accounting around the 64-entry limit.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): classify erasure candidates in one budgeted walk

Continue directory classification in cooperative 64-entry batches until
part evidence or EOF, retaining normal checkpoint and namespace work.
Defer metadata accounting and namespace reuse until classification ends.
Distinguish vanished opens from incomplete iteration and preserve quota,
sampling, failed-object and non-destructive heal behavior.

Pin test callsite interest alongside poll-scoped capture to prevent an
unsubscribed sibling from caching the checkpoint callsite as disabled.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-24 15:49:00 +08:00
Chris 3092792c3c fix(scanner): recover an interleaved pause backlog wedge after expansion (#8867)
* fix(scanner): roll a wedged pause backlog forward to its proven ledger

For rustfs/backlog#2873 defect E.

Two interleaved writers could leave a replica state that no claim could
leave. Writer B read writer A's partial commit and selected the stable
rollback point. A's commit then completed and A verified it. B's rollback
stabilization overwrote every member it had read after the commit, and A's
stabilization reached only the remaining member. No commit survived and the
stable fields differed, so every claim and every decommission retirement
failed with "neither a surviving membership commit nor a stable rollback
point".

- When no commit survives and stable consensus fails, select the unique
  newest stable ledger by (writer epoch, generation). A stable field is
  written only after its ledger was selected as authority, so that ledger is
  the last proven one. The next claim stabilizes it and advances normally.
- Keep failing closed for an invalid or unreadable replica, for two
  different stable ledgers at the newest key, and for a missing replica that
  a stored commit names as a member.
- A rollback proof written without an authoritative commit names only
  members that already store a record, so a failed first write to a new
  member cannot turn it into a named member that reads as lost.
- Retirement refuses a rolled-forward ledger until the writer has
  stabilized it, as a retryable handoff conflict rather than a terminal
  authority conflict.
- Tests replay the interleaving on real storage: it converges and retires,
  it converges after another expansion with a failed first write to a new
  member, and a same-key fork or a lost named member still refuses.

* fix(scanner): let retirement stabilize a rolled-forward pause backlog

Follow-up to the defect E roll-forward for rustfs/backlog#2873.

- Retirement no longer waits for a scanner writer. When no commit
  survives and the native selector over every read member (survivors,
  retiring source and queued pools) rolls forward, the planner commits and
  stabilizes that ledger on the survivors under the writers' object fence.
  The previous retryable handoff could exhaust its ~13.5 s budget long
  before a writer re-claimed, and failed the decommission terminally.
- A writer's roll-forward refuses when a stored commit names a member that
  is missing or outside its read, such as a retiring source excluded from
  the writable set. Retirement, which reads that member, decides instead.
- A writer stabilizing a rolled-forward ledger writes missing members
  first and stops at the first failure. A member that only read as missing
  rejects the create, so no surviving proof can omit its newer record.
- Docs describe the actual roll-forward preconditions and the retirement
  exit.
- Tests: retirement converges the wedge with no writer; a writer refuses to
  roll forward past an unread source that holds the newest ledger and
  retirement keeps it; a spurious missing read changes no record and the
  next claim converges on the newest ledger.

* fix(decommission): create missing pause backlog survivors first

Follow-up to the defect E retirement roll-forward for rustfs/backlog#2873.

- Retirement creates survivors that read as missing one at a time, before
  any stored survivor, and stops at the first failed create. A transient
  create failure on a new pool's set no longer leaves every other survivor
  holding a commit that names a still-missing member, which no later
  selection could leave.
- A rolled-forward retirement first seeds a rollback proof that names only
  the survivors that store a record; the commit still names every survivor.
- Retirement rolls forward only when the survivors cannot select a ledger
  on their own or select the same one. Survivors that agree on a different
  ledger keep the source-authority check.
- The scanner writer applies the same missing-first, stop-at-first-failure
  order to every stabilization with a missing member, including a seed from
  an old commit selected as the rollback proof.
- Tests: a failed new-pool create during retirement retries to convergence;
  a spurious missing read cannot seed an old commit over a newer member;
  the rolled-forward precedence over source stable records is pinned.

* docs(decommission): state the all-missing survivor limit of backlog roll-forward
2026-10-24 14:52:30 +09:00
Chris d90419f4dc feat(scanner): count cache save payload bytes per attempt (#8865) 2026-10-24 14:23:02 +09:00
Chris 6535c28f1a fix(scanner): recover backlog across interrupted pool expansions (#8858)
fix(scanner): recover stable membership after repeated expansion
2026-10-24 10:01:10 +09:00
Chris 76f4342ae2 feat(scanner): observe canonical heal task terminals (#8851) 2026-10-24 08:50:13 +09:00
Chris abe14e2f97 feat(scanner): configure startup recovery backoff cap (#8845)
* feat(scanner): configure startup recovery backoff cap

* test(scanner): compare recovery retry count without narrowing

* docs(scanner): document recovery backoff cap and expose it in Helm

List scanner.recovery_backoff_cap in the runtime controls table, drop the
fixed ten-minute wording from the follower recovery section, and map
config.rustfs.scanner.recovery_backoff_cap_secs to
RUSTFS_SCANNER_RECOVERY_BACKOFF_CAP_SECS in the chart.
2026-10-24 03:09:39 +09:00
Chris 3254e79efd perf(scanner): transfer owned caches without redundant clones (#8849) 2026-10-24 00:49:07 +08:00
Chris db55098aa6 fix(scanner): recover dual corruption with retained epoch fencing (#8833)
* fix(scanner): recover dual corruption with retained epoch fencing

* fix(scanner): preserve recovery markers and legacy quota floors

* test(scanner): expect retained usage recovery marker state

* docs(scanner): register dual-corruption recovery evidence objects

Add the .bloomcycle.bin.dual-recovery/ evidence family to the persisted
objects table and describe its owner, write-once lifecycle and recovery
rule in Deletion And Recovery Rules.
2026-10-24 01:43:59 +09:00
Chris 47e1850d7e fix(scanner): preserve remote checkpoint coverage identity (#8827)
* fix(scanner): preserve remote checkpoint coverage identity

* test(scanner): verify remote plan identity propagation

* test(rpc): disambiguate scanner capability body collection
2026-10-24 00:22:49 +09:00
Hauser c8cb1d86d3 fix(scanner): streamline budgeted metadata walks (#8837)
* fix(scanner): budget directory enumeration I/O

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(scanner): skip redundant metadata walk work

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(scanner): share budgeted directory traversal

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-23 21:07:09 +09:00
overtrue dd677cc121 Merge main into gateway/integration
Picks up main through 44a73b0be. Conflicts: rustfs/src/error.rs keeps the rustfs-s3-types ApiError conversion and drops the helper main's #8816 replaced; scripts/check_s3s_footprint.sh keeps the integration script, whose baseline lives in .config (main only retuned a constant the integration script does not have). Main's new tests use the rustfs-s3-types error code in admin replication and reach s3s::BodySizeLimitExceeded through the app's test-only s3 facade, so the s3s file count stays at 118; s3_error_lines tightens to 302. The test-count baseline is regenerated; the e2e selections are unchanged.
2026-10-23 08:40:02 +08:00
Hauser df10dc7864 fix(scanner): fence backlog writes against stale pool membership (#8799)
* perf(s3): borrow options during delete preflight

Pass the existing request options by reference to the metadata preflight instead of cloning them solely for that borrow. Keep the getter's mutable snapshot copy and the later owned-delete copy.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): fence backlog writes against stale pool membership

Reject native scanner publications unless their captured writable pool cohort
matches the fenced durable topology. A smaller startup view must not replace
an expanded membership commit with its older stable rollback.

Keep single-replica test injection fenced by the complete current cohort and
cover rejected startup claims, unchanged CAS revisions, and recovery of the
acknowledged expanded commit.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): use current cohort for healthy scanner CAS writes

Keep the writer's cached topology stale while explicitly capturing the
surviving membership for healthy-target publication and CAS checks. Also
assert that the old cohort rejects healthy-target writes without publishing.

Preserve retiring-source rejection, payload and ETag checks, and canceled
source membership repair without changing production admission.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 23:17:15 +08:00
overtrue b8dffa2f1d Merge main into gateway/integration
Picks up the 20 main commits since 62b6f5f3f. The one conflict is the tier free-version recovery test setup: it keeps the integration-only fixture isolation (background recovery disabled while the fixture starts) and main's mock tier registration (#8804 skips scans with no tier). The e2e selections are unchanged by these commits; the test-count baseline is regenerated.
2026-10-22 20:17:09 +08:00
Hauser 0d0e6d8497 fix(ecstore): preserve scanner authority during decommission (#8807)
Fence native backlog writes against their captured pool membership and retain
queued replicas as read-only proof members during retirement. Stabilize the
topology test workload and bound pre-ACK recovery separately.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 17:42:20 +08:00
Ramakrishna Chilaka 0bade0ca3e perf(ecstore,filemeta): batch listings and buffer metacache streams (#8791)
* perf(ecstore,filemeta): batch local walk_dir metadata reads and buffer metacache streams

* fix(ecstore): stream completed listing metadata reads

Publish each result before starting the next blocking read, preserving
per-object progress and cancellation without a timed chunk queue.
Keep dispatch batching, byte limits, and namespace/error stops intact.
Cover multiple batches, page boundaries, and oversized-envelope resumption.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore,filemeta): verify production listing streams

Replace the simulated directory-walk benchmark with the public local disk
path. Validate fixture output before timing and retain metacache-only
stream benchmarks. Pin fragmented reads, read coalescing, and canonical
UUID namespace probing against the reference behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): route listing benchmark through storage API

Centralize local listing benchmark imports in the shared benchmark
storage API boundary required by the architecture migration rules.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): keep catch-up fixture timestamps consistent

Admit the seeded catch-up using its observation timestamp so a durable
writer claim crossing a wall-clock second cannot spuriously rate-limit it.
Model an older claim timestamp to exercise that boundary deterministically.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 14:54:34 +08:00
overtrue 62b6f5f3fc Merge main into gateway/integration
The scheduled sync has stopped on conflicts since 2026-10-19. Resolved by keeping both sides: main's restructured delete-rollback path in disk/local.rs with the integration-only native-publication record, main's per-instance region with the rustfs-s3-types Region, main's new lifecycle, rebalance and admin code ported to the persisted lifecycle types and rustfs-s3-types errors, both sides' new scanner tests, and the union of the access_consumer imports. The e2e smoke, full (darwin) and repl-nightly selections and the test-count baseline are regenerated from the merged tree.
2026-10-22 10:14:41 +09:00
Hauser 4597bb91a8 feat(metrics): count scanner wakeups and cycle attempts (#8786)
Expose selected outer-wait reasons separately from runner-call attempts.
Preserve scanner scheduling, pause, rate-limit and budget boundaries.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 09:13:37 +08:00
Hauser 15b7040e78 perf(scanner): move owned usage snapshot keys (#8792)
Consume completed aggregation entries at their terminal conversion so
bucket and replication target keys can move into the output maps.
Preserve validation, rejection and publication behavior.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 09:13:06 +08:00
Hauser 65f0599c70 feat(metrics): track usage outcomes and trim decommission probes (#8781)
* feat(metrics): expose usage candidate rejection outcomes

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): borrow usage candidate fixtures

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): reserve capacity for decommission probes

Reserve all nextest slots for the two eight-disk decommission fixtures under both default and CI profiles. Keep their internal concurrency, production lock budgets, assertions and zero-retry policy intact.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): scope prewrite capacity probes to the target pool

After persisting a new capacity intent, refresh only the target pool baseline. Preserve the global admission and finalization scans, lock order, fencing, and accounting formulas. Verify the reduced probe count with real disk metrics and preserve injected snapshot/error behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 02:44:49 +08:00