* chore(obs): ReplicationStats -> ReplicationMetricsSnapshot, BucketReplicationStats -> BucketReplicationMetricsSnapshot
Rename in-obs-crate ReplicationStats and BucketReplicationStats to
ReplicationMetricsSnapshot and BucketReplicationMetricsSnapshot respectively.
No serde impact (these types are Prometheus metric collectors, not serialized).
No external consumers found outside the obs crate.
* cleanup: remove #[serial] annotations from e2e_test, scanner, lifecycle, and object-capacity crates
Remove no-op #[serial] attributes (nextest ignores serial_test) and the
serial_test dependency from four crates. All tests already use temp_env
for env-var isolation, making #[serial] purely redundant.
Crates cleaned:
- e2e_test (37 annotations, 9 imports, removed serial_test dep)
- rustfs-scanner (115 annotations across 7 files, removed serial_test dep)
- rustfs-lifecycle (46 annotations, removed serial_test dep)
- rustfs-object-capacity (38 annotations, removed serial_test dep)
Also converted scanner/tests/lifecycle_integration_test.rs
with_forced_immediate_enqueue_timeout helper from unsafe raw
env::set_var/remove_var to temp_env::async_with_vars for proper
isolation, and added async_closure feature to scanner's temp-env dep.
* fix(lifecycle): restore #[serial] on 2 tests that read env vars without temp_env
eval_inner_expires_latest_object_after_days_due and
eval_inner_does_not_panic_on_many_equal_due_events call eval_inner()
which reads ENV_ILM_PROCESS_TIME via std::env::var(). Without #[serial]
they race with other tests that set these vars via temp_env.
* style: cargo fmt
* fix(scanner): defer usage publication during pool recovery
* fix(scanner): preserve metrics when publication is deferred
* fix(scanner): route test types through storage boundary
* fix(scanner): keep cache floor deferred during movement
Move MRF repaired-event fan-out from admission to successful terminal completion so scanner pending-heal ledgers only clear after the canonical heal task actually finishes. Track notice ownership across duplicate admission, retry merge, cancellation, and queue displacement.
Co-authored-by: heihutu <heihutu@gmail.com>
Cache heal queue dedup keys, avoid retry request double construction, clear task aliases after terminal completion, and age out stale scanner pending-heal ledger entries during retry sweeps.
Co-authored-by: heihutu <heihutu@gmail.com>
Split the 8178-line scanner.rs (48% inline tests) into a canonical
scanner.rs + scanner/ module tree with zero behavior change:
- scanner.rs (~2140): cycle constants, schedule status, budget/config
helpers, startup, maintenance features, the two run loops, and
cycle-result finalization
- scanner/activity.rs (~770): wake/backoff policy and scanner activity
observation (probing, generations, topology digest)
- scanner/heal_info.rs (~110): the background-heal info object
- scanner/cycle_state.rs (~500): cycle-state codec, persisted usage
floors, and cycle-state persistence
- scanner/leadership.rs (~360): leader-lock claiming, usage-epoch
fencing, and lock-loss handling
- scanner/usage_store.rs (~480): the CAS data-usage store pipeline and
observed-snapshot cleanup
- scanner/tests.rs (~3920): the inline test module as a child module
All crate paths are unchanged: scanner::BackgroundHealInfo,
scanner::read_background_heal_info, scanner::store_data_usage_in_backend,
and scanner_topology_digest resolve through root re-exports with their
original visibilities, and the pub(crate) surface used by scanner_io and
remote_scanner re-exports at pub(crate). Cross-module items gain
pub(super), whose scope equals the old single-module privacy domain.
Code is moved verbatim apart from those markers, per-module import
headers, and rustfmt re-wraps.
Co-authored-by: heihutu <heihutu@gmail.com>
Cache fixed-label HTTP request and response metric handles so the hot request path avoids repeated recorder lookups for common counters, gauges, and histograms. Preserve the existing metric names and labels with focused mapping tests.
Co-authored-by: heihutu <heihutu@gmail.com>
Split the 6723-line manager.rs (44% inline tests) into a canonical
manager.rs + manager/ module tree with zero behavior change:
- manager.rs (~1830): HealManager and HealState, HealConfig, task
report/snapshot types, overlap policy, admission classification and
queue admission, submit paths, task-state queries, and the
statistics surface
- manager/queue.rs (~450): the priority heal queue, its per-key dedup
index, and the queue bookkeeping structs
- manager/scheduler.rs (~620): start_scheduler and the
process_heal_queue consumption loop with its skip/metric helpers
- manager/auto_scan.rs (~550): the automatic disk scanner
- manager/unclean_shutdown.rs (~390): unclean-shutdown recovery and
its durable replacement-intent helpers
- manager/tests.rs (~2970): the inline test module as a child module
All module paths are unchanged. The queue structs' fields and the
cross-module helpers gain pub(super), whose scope equals the old
single-module privacy domain; HealManager's private fields stay in the
root and remain reachable from child impl blocks. Code is moved
verbatim apart from those markers, heal-level super:: path fixes for
the unclean-shutdown move, per-module import headers, and rustfmt
re-wraps.
The logging-guardrail rule for the manager demote_to_debug_when! count
now sums manager.rs with its manager/*.rs children, since one
scheduler site moved with process_heal_queue; the >= 6 threshold is
unchanged and the forbidden admission info!/warn! pattern check keeps
targeting the root admission code.
Co-authored-by: heihutu <heihutu@gmail.com>
* refactor(scanner): split scanner_folder item actions and ledger
Split the 6345-line scanner_folder.rs (46% inline tests) into a
canonical scanner_folder.rs + scanner_folder/ module tree with zero
behavior change:
- scanner_folder.rs (~2280): scan constants, alert cooldowns, metric
accounting, resume ordering, tracing helpers, the FolderScanner
struct with failed-object bookkeeping and the scan_folder traversal,
and scan_data_folder
- scanner_folder/item_actions.rs (~890): CachedFolder, the get-size
failure policy, ScannerItem with apply_actions and the heal/ILM
admission helpers
- scanner_folder/ledger.rs (~280): the pending-scanner-heal ledger
methods and their entry helpers (record/prune/clear-for-repaired/
retry)
- scanner_folder/tests.rs (~2950): the inline test module as a child
module
The ScannerItem path used by scanner_io resolves through a root
re-export, and every other crate path is unchanged. Cross-module items
gain pub(super), whose scope equals the old single-module privacy
domain. Code is moved verbatim apart from those markers, per-module
import headers, and rustfmt re-wraps.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fmt
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Add deterministic rename_data coverage for tail-disk success/failure, cancellation serialization, and strict quorum rollback visibility after disk reopen. This establishes the safety boundary before experimenting with write-quorum early ACK and background tail completion for backlog #925.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(lifecycle): safely expire all object versions
* fix(lifecycle): preserve delete-all replication purges
* fix(lifecycle): remove dead replication journal
* fix(ci): avoid lifecycle transition test stack overflow
* fix(lifecycle): release recovery locks before tier IO
* test(lifecycle): align object-lock error assertions
* test(lifecycle): avoid scanner restore stack overflow
* test(scanner): avoid stack overflow in transition and restore flow test (#6300)
* refactor(scanner): split remote_scanner.rs into stream child module (#6289)
Split the 3080-line remote_scanner.rs (47% inline tests) into a
canonical foo.rs + foo/ module tree with zero behavior change:
- remote_scanner.rs (~320): protocol constants, process statics, and
the request decode/validate/admit/preflight/claim API plus root
re-exports
- remote_scanner/stream.rs (~1340): wire/frame types, replay cache,
FrameAuthenticator, serve path, local bucket scan + persist, client
scan, and the bounded stream plumbing
- remote_scanner/stream/tests.rs (~1470): the inline test module as a
child module of stream so it can reach both parents' private items
All crate paths are unchanged: lib.rs re-exports
(serve_remote_scanner_request, RemoteScannerRequest, ...) resolve
through root re-exports, and scanner_io's crate::remote_scanner::
{scan_remote_bucket, RemoteScannerScanSpec, RemoteScannerOutcome}
paths resolve through pub(crate) re-exports. Cross-module items gain
pub(super), whose scope equals the old single-module privacy domain;
no item's effective visibility widens. Code is moved verbatim apart
from those markers, per-module import headers, and rustfmt line
re-wraps.
Co-authored-by: heihutu <heihutu@gmail.com>
* refactor(heal): split resume.rs into focused child modules (#6290)
Split the 4242-line resume.rs (46% inline tests) into a canonical
foo.rs + foo/ module tree with zero behavior change:
- resume.rs (~1020): state file constants, PersistThrottle, ResumeState,
ResumeManager core (constructors, load/discovery, progress mutators,
ordinary persistence) plus root re-exports
- resume/replacement.rs (~690): replacement-intent/proof types and the
ResumeManager replacement-lifecycle methods
- resume/checkpoint.rs (~350): ResumeCheckpoint + CheckpointManager
- resume/utils.rs (~310): ResumeUtils statics
- resume/tests.rs (~1980): the inline test module as a child module
All module paths are unchanged (heal::resume::CheckpointManager and
friends resolve through root re-exports), so no consumer inside or
outside the crate changes. Items defined in child modules keep
module-private visibility; only the ten cross-module helpers gain
pub(super), which is not part of the crate API. Code is moved verbatim
apart from those visibility markers, four super::storage_api path
fixes, and the new per-module import headers.
Co-authored-by: heihutu <heihutu@gmail.com>
* refactor(scanner): split scanner_io.rs into child modules (#6294)
Split the 5369-line scanner_io.rs (39% inline tests) into a canonical
scanner_io.rs + scanner_io/ module tree with zero behavior change:
- scanner_io.rs (~660): constants, metadata-error constructors, the
bucket scan plan, cycle-status classification helpers, the ScannerIO /
ScannerIOCache / ScannerIODisk traits, and ScannerCycleResult
- scanner_io/dirty_usage.rs (~300): process-wide dirty-usage statics
and the acknowledgment protocol
- scanner_io/guards.rs (~270): concurrency gauges and RAII guards
- scanner_io/cache.rs (~410): scanner cache locks and the snapshot
persist/publish path
- scanner_io/io_cycle.rs (~390), io_cache.rs (~1160), io_disk.rs
(~230): the ECStore / SetDisks / Disk trait implementations
- scanner_io/publish_gate_tests.rs (~750) and tests.rs (~1340): the two
inline test modules as child modules
All crate paths are unchanged: the lib.rs scanner_io re-exports and
every crate::scanner_io:: consumer (scanner.rs, remote_scanner,
scanner_folder, and cross-crate rustfs users) resolve through root
re-exports with their original visibilities (pub stays pub, pub(crate)
stays pub(crate)). Cross-module items gain pub(super), whose scope
equals the old single-module privacy domain. Code is moved verbatim
apart from those markers, per-module import headers, and rustfmt
re-wraps.
The logging-guardrail nsscanner_disk skip-set_disks rule now points at
scanner_io/io_disk.rs where the function moved; the pattern and
thresholds are unchanged.
Co-authored-by: heihutu <heihutu@gmail.com>
* refactor(scanner): split data_usage_define persistence and tests (#6292)
Split the 3655-line data_usage_define.rs (59% inline tests) into a
canonical foo.rs + foo/ module tree with zero behavior change:
- data_usage_define.rs (~950): cache constants and revision helpers,
the data-usage tree types, DataUsageCacheInfo with its hand-written
Serialize, the in-memory tree operations, dui, and marshal/unmarshal
- data_usage_define/persistence.rs (~580): the load/backup/restore
ladder (load, try_load_inner, revision_for_path) and the CAS save
path with its retry policy and save metrics
- data_usage_define/tests.rs (~2155): the inline test module as a child
module
All module paths are unchanged (the lib.rs data_usage_define::* glob
re-export and every crate::data_usage_define:: consumer resolve as
before). The hand-written map-encoded Serialize for
DataUsageCacheInfo is moved byte-for-byte per the AGENTS.md
cross-cutting invariant; on-disk names and the cache key format const
stay in the root. Four persistence helpers used by tests gain
pub(super), whose scope equals the old single-module privacy domain.
Code is moved verbatim apart from those markers, per-module import
headers, and rustfmt re-wraps.
Co-authored-by: heihutu <heihutu@gmail.com>
* chore(deps): bump datafusion to 55.0.0 (#6288)
* refactor(heal): split task.rs per heal kind (#6293)
* feat(ecstore): batch small file fdatasync commits (#6297)
* feat(ecstore): batch small file fdatasync commits
Add a default-off experimental file fdatasync group commit path for small rename_data shard directories. The coordinator batches same-disk waiters into one blocking task while preserving per-directory source fsync after shard contents are durable.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): wait for compression S3 readiness
Reuse the shared S3 API readiness probe for compression test servers so multipart requests do not race the startup readiness gate after the TCP port opens.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): recover multi-committed mutation intents (#6296)
* fix(tier): recover multi-committed mutation intents
* fix(tier): recover committed mutations on standalone nodes
* test(scanner): avoid stack overflow in transition test
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: cxymds <cxymds@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Add PUT-stage diagnostics for file fdatasync group commit wait time, per-group outstanding depth, and rename disk completion position. These metrics keep the existing default-off PUT stage gate and do not change group commit scheduling or quorum behavior.
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(ecstore): batch small file fdatasync commits
Add a default-off experimental file fdatasync group commit path for small rename_data shard directories. The coordinator batches same-disk waiters into one blocking task while preserving per-directory source fsync after shard contents are durable.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): wait for compression S3 readiness
Reuse the shared S3 API readiness probe for compression test servers so multipart requests do not race the startup readiness gate after the TCP port opens.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Split the 3655-line data_usage_define.rs (59% inline tests) into a
canonical foo.rs + foo/ module tree with zero behavior change:
- data_usage_define.rs (~950): cache constants and revision helpers,
the data-usage tree types, DataUsageCacheInfo with its hand-written
Serialize, the in-memory tree operations, dui, and marshal/unmarshal
- data_usage_define/persistence.rs (~580): the load/backup/restore
ladder (load, try_load_inner, revision_for_path) and the CAS save
path with its retry policy and save metrics
- data_usage_define/tests.rs (~2155): the inline test module as a child
module
All module paths are unchanged (the lib.rs data_usage_define::* glob
re-export and every crate::data_usage_define:: consumer resolve as
before). The hand-written map-encoded Serialize for
DataUsageCacheInfo is moved byte-for-byte per the AGENTS.md
cross-cutting invariant; on-disk names and the cache key format const
stay in the root. Four persistence helpers used by tests gain
pub(super), whose scope equals the old single-module privacy domain.
Code is moved verbatim apart from those markers, per-module import
headers, and rustfmt re-wraps.
Co-authored-by: heihutu <heihutu@gmail.com>
Split the 5369-line scanner_io.rs (39% inline tests) into a canonical
scanner_io.rs + scanner_io/ module tree with zero behavior change:
- scanner_io.rs (~660): constants, metadata-error constructors, the
bucket scan plan, cycle-status classification helpers, the ScannerIO /
ScannerIOCache / ScannerIODisk traits, and ScannerCycleResult
- scanner_io/dirty_usage.rs (~300): process-wide dirty-usage statics
and the acknowledgment protocol
- scanner_io/guards.rs (~270): concurrency gauges and RAII guards
- scanner_io/cache.rs (~410): scanner cache locks and the snapshot
persist/publish path
- scanner_io/io_cycle.rs (~390), io_cache.rs (~1160), io_disk.rs
(~230): the ECStore / SetDisks / Disk trait implementations
- scanner_io/publish_gate_tests.rs (~750) and tests.rs (~1340): the two
inline test modules as child modules
All crate paths are unchanged: the lib.rs scanner_io re-exports and
every crate::scanner_io:: consumer (scanner.rs, remote_scanner,
scanner_folder, and cross-crate rustfs users) resolve through root
re-exports with their original visibilities (pub stays pub, pub(crate)
stays pub(crate)). Cross-module items gain pub(super), whose scope
equals the old single-module privacy domain. Code is moved verbatim
apart from those markers, per-module import headers, and rustfmt
re-wraps.
The logging-guardrail nsscanner_disk skip-set_disks rule now points at
scanner_io/io_disk.rs where the function moved; the pattern and
thresholds are unchanged.
Co-authored-by: heihutu <heihutu@gmail.com>
Split the 4242-line resume.rs (46% inline tests) into a canonical
foo.rs + foo/ module tree with zero behavior change:
- resume.rs (~1020): state file constants, PersistThrottle, ResumeState,
ResumeManager core (constructors, load/discovery, progress mutators,
ordinary persistence) plus root re-exports
- resume/replacement.rs (~690): replacement-intent/proof types and the
ResumeManager replacement-lifecycle methods
- resume/checkpoint.rs (~350): ResumeCheckpoint + CheckpointManager
- resume/utils.rs (~310): ResumeUtils statics
- resume/tests.rs (~1980): the inline test module as a child module
All module paths are unchanged (heal::resume::CheckpointManager and
friends resolve through root re-exports), so no consumer inside or
outside the crate changes. Items defined in child modules keep
module-private visibility; only the ten cross-module helpers gain
pub(super), which is not part of the crate API. Code is moved verbatim
apart from those visibility markers, four super::storage_api path
fixes, and the new per-module import headers.
Co-authored-by: heihutu <heihutu@gmail.com>
Split the 3080-line remote_scanner.rs (47% inline tests) into a
canonical foo.rs + foo/ module tree with zero behavior change:
- remote_scanner.rs (~320): protocol constants, process statics, and
the request decode/validate/admit/preflight/claim API plus root
re-exports
- remote_scanner/stream.rs (~1340): wire/frame types, replay cache,
FrameAuthenticator, serve path, local bucket scan + persist, client
scan, and the bounded stream plumbing
- remote_scanner/stream/tests.rs (~1470): the inline test module as a
child module of stream so it can reach both parents' private items
All crate paths are unchanged: lib.rs re-exports
(serve_remote_scanner_request, RemoteScannerRequest, ...) resolve
through root re-exports, and scanner_io's crate::remote_scanner::
{scan_remote_bucket, RemoteScannerScanSpec, RemoteScannerOutcome}
paths resolve through pub(crate) re-exports. Cross-module items gain
pub(super), whose scope equals the old single-module privacy domain;
no item's effective visibility widens. Code is moved verbatim apart
from those markers, per-module import headers, and rustfmt line
re-wraps.
Co-authored-by: heihutu <heihutu@gmail.com>
overtrue