Commit Graph

5679 Commits

Author SHA1 Message Date
houseme 5c81ac08e3 feat(ecstore): tune fdatasync group wait budget
Co-Authored-By: heihutu <heihutu@gmail.com>
2026-08-21 11:42:51 +08:00
cxymds dfeb732ac8 fix: make DeleteObjects idempotent for raw not-found errors (#6323)
* fix: make DeleteObjects idempotent for raw not-found errors

* fix: cover DeleteObjects raw not-found result dispatch
2026-08-21 03:12:37 +00:00
Henry Guo 1aae680373 fix(lifecycle): honor bucket default retention (#6324)
* fix(lifecycle): honor bucket retention during scanner expiry

* fix(lifecycle): reject zero object lock retention

* test(lifecycle): cover malformed retention metadata

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
1.0.0-rc.3-preview.2
2026-08-21 10:11:13 +08:00
cxymds 1b4f62d501 docs(agents): run adversarial review before pre-pr (#6325)
docs(agents): order adversarial review before pre-pr
2026-08-21 10:10:56 +08:00
houseme 4283591838 feat(ecstore): observe PUT commit lock admission (#6319)
* feat(ecstore): observe PUT commit lock admission

Co-Authored-By: heihutu <heihutu@gmail.com>

* update h2 v0.4.18

* test(e2e): box SSE-KMS negative errors

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-21 01:18:53 +00:00
Zhengchao An f2957a680d test: stabilize release-blocking full E2E checks (#6322) 2026-08-21 08:43:12 +08:00
Zhengchao An d22cb5d07a fix: resolve release-blocking integration failures (#6320)
* fix: resolve release-blocking integration failures

* fix: satisfy stable clippy lints

* fix: satisfy Rust 1.98 CI lints
2026-08-21 06:39:29 +08:00
houseme 762919b1ba perf(scanner): reduce per-object allocation churn (#6318)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-20 16:47:48 +00:00
houseme cee0d5cf9b perf(io-metrics): cache read version metric handles (#6317)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-20 16:39:26 +00:00
houseme 35af688cd9 test(obs): add metric dimension smoke harness (#6316)
test(obs): add metrics dimension smoke harness

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-20 16:26:19 +00:00
GatewayJ 205337151a fix(webdav): allow bucket-scoped root listings (#6298)
* fix(webdav): allow bucket-scoped root listings

* test(webdav): use public protocol export

* test(webdav): initialize identity inline

---------

Co-authored-by: cxymds <cxymds@gmail.com>
2026-08-21 00:15:33 +08:00
Henry Guo 105b6fbfde fix(scanner): honor explicit cycle cadence (#6313)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-08-20 23:35:03 +08:00
houseme b2e573c48b feat(ecstore): bound put commit lock admission (#6315)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-20 23:33:17 +08:00
houseme 114bf5148c refactor(heal): prune statistics label helpers (#6312)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-20 22:43:59 +08:00
houseme 830e553a3c feat(obs): complete metric dimension coverage (#6314)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-20 22:43:40 +08:00
Zhengchao An d65fba9142 chore(release): prepare 1.0.0-rc.3 1.0.0-rc.3-preview.1 2026-08-20 21:50:34 +08:00
houseme 198a07d3fa feat(ecstore): observe put commit lock wait (#6310) 2026-08-20 21:47:30 +08:00
houseme 8bd2d5d967 perf(heal): reduce scheduler skip heap churn (#6311) 2026-08-20 21:47:11 +08:00
houseme efbef700ea fix(heal): emit MRF repair notices on completion (#6309)
Move MRF repaired-event fan-out from admission to successful terminal completion so scanner pending-heal ledgers only clear after the canonical heal task actually finishes. Track notice ownership across duplicate admission, retry merge, cancellation, and queue displacement.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-20 21:23:29 +08:00
houseme a247c79359 perf(heal): trim scanner and heal queue hot paths (#6307)
Cache heal queue dedup keys, avoid retry request double construction, clear task aliases after terminal completion, and age out stale scanner pending-heal ledger entries during retry sweeps.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-20 20:28:22 +08:00
houseme cd399d1e72 feat(ecstore): add default-off rename early ack probe (#6306) 2026-08-20 19:20:01 +08:00
houseme 095bf34086 refactor(scanner): split scanner.rs cycle/leadership/persist children (#6305)
Split the 8178-line scanner.rs (48% inline tests) into a canonical
scanner.rs + scanner/ module tree with zero behavior change:

- scanner.rs (~2140): cycle constants, schedule status, budget/config
  helpers, startup, maintenance features, the two run loops, and
  cycle-result finalization
- scanner/activity.rs (~770): wake/backoff policy and scanner activity
  observation (probing, generations, topology digest)
- scanner/heal_info.rs (~110): the background-heal info object
- scanner/cycle_state.rs (~500): cycle-state codec, persisted usage
  floors, and cycle-state persistence
- scanner/leadership.rs (~360): leader-lock claiming, usage-epoch
  fencing, and lock-loss handling
- scanner/usage_store.rs (~480): the CAS data-usage store pipeline and
  observed-snapshot cleanup
- scanner/tests.rs (~3920): the inline test module as a child module

All crate paths are unchanged: scanner::BackgroundHealInfo,
scanner::read_background_heal_info, scanner::store_data_usage_in_backend,
and scanner_topology_digest resolve through root re-exports with their
original visibilities, and the pub(crate) surface used by scanner_io and
remote_scanner re-exports at pub(crate). Cross-module items gain
pub(super), whose scope equals the old single-module privacy domain.
Code is moved verbatim apart from those markers, per-module import
headers, and rustfmt re-wraps.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-20 19:12:27 +08:00
houseme 2cf0ad0f85 perf(server): cache HTTP metric handles (#6304)
Cache fixed-label HTTP request and response metric handles so the hot request path avoids repeated recorder lookups for common counters, gauges, and histograms. Preserve the existing metric names and labels with focused mapping tests.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-20 19:12:18 +08:00
houseme 2b9dcc646f refactor(heal): split manager.rs queue/scheduler/scan children (#6303)
Split the 6723-line manager.rs (44% inline tests) into a canonical
manager.rs + manager/ module tree with zero behavior change:

- manager.rs (~1830): HealManager and HealState, HealConfig, task
  report/snapshot types, overlap policy, admission classification and
  queue admission, submit paths, task-state queries, and the
  statistics surface
- manager/queue.rs (~450): the priority heal queue, its per-key dedup
  index, and the queue bookkeeping structs
- manager/scheduler.rs (~620): start_scheduler and the
  process_heal_queue consumption loop with its skip/metric helpers
- manager/auto_scan.rs (~550): the automatic disk scanner
- manager/unclean_shutdown.rs (~390): unclean-shutdown recovery and
  its durable replacement-intent helpers
- manager/tests.rs (~2970): the inline test module as a child module

All module paths are unchanged. The queue structs' fields and the
cross-module helpers gain pub(super), whose scope equals the old
single-module privacy domain; HealManager's private fields stay in the
root and remain reachable from child impl blocks. Code is moved
verbatim apart from those markers, heal-level super:: path fixes for
the unclean-shutdown move, per-module import headers, and rustfmt
re-wraps.

The logging-guardrail rule for the manager demote_to_debug_when! count
now sums manager.rs with its manager/*.rs children, since one
scheduler site moved with process_heal_queue; the >= 6 threshold is
unchanged and the forbidden admission info!/warn! pattern check keeps
targeting the root admission code.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-20 19:12:07 +08:00
houseme 129677f0b3 refactor(scanner): split scanner_folder item actions and ledger (#6302)
* refactor(scanner): split scanner_folder item actions and ledger

Split the 6345-line scanner_folder.rs (46% inline tests) into a
canonical scanner_folder.rs + scanner_folder/ module tree with zero
behavior change:

- scanner_folder.rs (~2280): scan constants, alert cooldowns, metric
  accounting, resume ordering, tracing helpers, the FolderScanner
  struct with failed-object bookkeeping and the scan_folder traversal,
  and scan_data_folder
- scanner_folder/item_actions.rs (~890): CachedFolder, the get-size
  failure policy, ScannerItem with apply_actions and the heal/ILM
  admission helpers
- scanner_folder/ledger.rs (~280): the pending-scanner-heal ledger
  methods and their entry helpers (record/prune/clear-for-repaired/
  retry)
- scanner_folder/tests.rs (~2950): the inline test module as a child
  module

The ScannerItem path used by scanner_io resolves through a root
re-export, and every other crate path is unchanged. Cross-module items
gain pub(super), whose scope equals the old single-module privacy
domain. Code is moved verbatim apart from those markers, per-module
import headers, and rustfmt re-wraps.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fmt

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-20 19:11:55 +08:00
houseme c620a74230 test(ecstore): pin rename quorum tail visibility (#6301)
Add deterministic rename_data coverage for tail-disk success/failure, cancellation serialization, and strict quorum rollback visibility after disk reopen. This establishes the safety boundary before experimenting with write-quorum early ACK and background tail completion for backlog #925.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-20 18:12:36 +08:00
cxymds 319a03e638 fix(lifecycle): safely expire all object versions (#6291)
* fix(lifecycle): safely expire all object versions

* fix(lifecycle): preserve delete-all replication purges

* fix(lifecycle): remove dead replication journal

* fix(ci): avoid lifecycle transition test stack overflow

* fix(lifecycle): release recovery locks before tier IO

* test(lifecycle): align object-lock error assertions

* test(lifecycle): avoid scanner restore stack overflow

* test(scanner): avoid stack overflow in transition and restore flow test (#6300)

* refactor(scanner): split remote_scanner.rs into stream child module (#6289)

Split the 3080-line remote_scanner.rs (47% inline tests) into a
canonical foo.rs + foo/ module tree with zero behavior change:

- remote_scanner.rs (~320): protocol constants, process statics, and
  the request decode/validate/admit/preflight/claim API plus root
  re-exports
- remote_scanner/stream.rs (~1340): wire/frame types, replay cache,
  FrameAuthenticator, serve path, local bucket scan + persist, client
  scan, and the bounded stream plumbing
- remote_scanner/stream/tests.rs (~1470): the inline test module as a
  child module of stream so it can reach both parents' private items

All crate paths are unchanged: lib.rs re-exports
(serve_remote_scanner_request, RemoteScannerRequest, ...) resolve
through root re-exports, and scanner_io's crate::remote_scanner::
{scan_remote_bucket, RemoteScannerScanSpec, RemoteScannerOutcome}
paths resolve through pub(crate) re-exports. Cross-module items gain
pub(super), whose scope equals the old single-module privacy domain;
no item's effective visibility widens. Code is moved verbatim apart
from those markers, per-module import headers, and rustfmt line
re-wraps.

Co-authored-by: heihutu <heihutu@gmail.com>

* refactor(heal): split resume.rs into focused child modules (#6290)

Split the 4242-line resume.rs (46% inline tests) into a canonical
foo.rs + foo/ module tree with zero behavior change:

- resume.rs (~1020): state file constants, PersistThrottle, ResumeState,
  ResumeManager core (constructors, load/discovery, progress mutators,
  ordinary persistence) plus root re-exports
- resume/replacement.rs (~690): replacement-intent/proof types and the
  ResumeManager replacement-lifecycle methods
- resume/checkpoint.rs (~350): ResumeCheckpoint + CheckpointManager
- resume/utils.rs (~310): ResumeUtils statics
- resume/tests.rs (~1980): the inline test module as a child module

All module paths are unchanged (heal::resume::CheckpointManager and
friends resolve through root re-exports), so no consumer inside or
outside the crate changes. Items defined in child modules keep
module-private visibility; only the ten cross-module helpers gain
pub(super), which is not part of the crate API. Code is moved verbatim
apart from those visibility markers, four super::storage_api path
fixes, and the new per-module import headers.

Co-authored-by: heihutu <heihutu@gmail.com>

* refactor(scanner): split scanner_io.rs into child modules (#6294)

Split the 5369-line scanner_io.rs (39% inline tests) into a canonical
scanner_io.rs + scanner_io/ module tree with zero behavior change:

- scanner_io.rs (~660): constants, metadata-error constructors, the
  bucket scan plan, cycle-status classification helpers, the ScannerIO /
  ScannerIOCache / ScannerIODisk traits, and ScannerCycleResult
- scanner_io/dirty_usage.rs (~300): process-wide dirty-usage statics
  and the acknowledgment protocol
- scanner_io/guards.rs (~270): concurrency gauges and RAII guards
- scanner_io/cache.rs (~410): scanner cache locks and the snapshot
  persist/publish path
- scanner_io/io_cycle.rs (~390), io_cache.rs (~1160), io_disk.rs
  (~230): the ECStore / SetDisks / Disk trait implementations
- scanner_io/publish_gate_tests.rs (~750) and tests.rs (~1340): the two
  inline test modules as child modules

All crate paths are unchanged: the lib.rs scanner_io re-exports and
every crate::scanner_io:: consumer (scanner.rs, remote_scanner,
scanner_folder, and cross-crate rustfs users) resolve through root
re-exports with their original visibilities (pub stays pub, pub(crate)
stays pub(crate)). Cross-module items gain pub(super), whose scope
equals the old single-module privacy domain. Code is moved verbatim
apart from those markers, per-module import headers, and rustfmt
re-wraps.

The logging-guardrail nsscanner_disk skip-set_disks rule now points at
scanner_io/io_disk.rs where the function moved; the pattern and
thresholds are unchanged.

Co-authored-by: heihutu <heihutu@gmail.com>

* refactor(scanner): split data_usage_define persistence and tests (#6292)

Split the 3655-line data_usage_define.rs (59% inline tests) into a
canonical foo.rs + foo/ module tree with zero behavior change:

- data_usage_define.rs (~950): cache constants and revision helpers,
  the data-usage tree types, DataUsageCacheInfo with its hand-written
  Serialize, the in-memory tree operations, dui, and marshal/unmarshal
- data_usage_define/persistence.rs (~580): the load/backup/restore
  ladder (load, try_load_inner, revision_for_path) and the CAS save
  path with its retry policy and save metrics
- data_usage_define/tests.rs (~2155): the inline test module as a child
  module

All module paths are unchanged (the lib.rs data_usage_define::* glob
re-export and every crate::data_usage_define:: consumer resolve as
before). The hand-written map-encoded Serialize for
DataUsageCacheInfo is moved byte-for-byte per the AGENTS.md
cross-cutting invariant; on-disk names and the cache key format const
stay in the root. Four persistence helpers used by tests gain
pub(super), whose scope equals the old single-module privacy domain.
Code is moved verbatim apart from those markers, per-module import
headers, and rustfmt re-wraps.

Co-authored-by: heihutu <heihutu@gmail.com>

* chore(deps): bump datafusion to 55.0.0 (#6288)

* refactor(heal): split task.rs per heal kind (#6293)

* feat(ecstore): batch small file fdatasync commits (#6297)

* feat(ecstore): batch small file fdatasync commits

Add a default-off experimental file fdatasync group commit path for small rename_data shard directories. The coordinator batches same-disk waiters into one blocking task while preserving per-directory source fsync after shard contents are durable.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(e2e): wait for compression S3 readiness

Reuse the shared S3 API readiness probe for compression test servers so multipart requests do not race the startup readiness gate after the TCP port opens.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>

* fix(tier): recover multi-committed mutation intents (#6296)

* fix(tier): recover multi-committed mutation intents

* fix(tier): recover committed mutations on standalone nodes

* test(scanner): avoid stack overflow in transition test

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: cxymds <cxymds@gmail.com>

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-20 09:42:09 +00:00
houseme 621fcb93c7 feat(ecstore): expose fdatasync group wait metrics (#6299)
Add PUT-stage diagnostics for file fdatasync group commit wait time, per-group outstanding depth, and rename disk completion position. These metrics keep the existing default-off PUT stage gate and do not change group commit scheduling or quorum behavior.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-20 16:59:37 +08:00
cxymds 76eb9c72e4 fix(tier): recover multi-committed mutation intents (#6296)
* fix(tier): recover multi-committed mutation intents

* fix(tier): recover committed mutations on standalone nodes
2026-08-20 16:26:27 +08:00
houseme 51023dc258 feat(ecstore): batch small file fdatasync commits (#6297)
* feat(ecstore): batch small file fdatasync commits

Add a default-off experimental file fdatasync group commit path for small rename_data shard directories. The coordinator batches same-disk waiters into one blocking task while preserving per-directory source fsync after shard contents are durable.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(e2e): wait for compression S3 readiness

Reuse the shared S3 API readiness probe for compression test servers so multipart requests do not race the startup readiness gate after the TCP port opens.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-20 15:22:56 +08:00
houseme 0d129ec4e7 refactor(heal): split task.rs per heal kind (#6293) 2026-08-20 12:24:16 +08:00
houseme fec9e8980a chore(deps): bump datafusion to 55.0.0 (#6288) 2026-08-20 12:23:53 +08:00
houseme c428c6615e refactor(scanner): split data_usage_define persistence and tests (#6292)
Split the 3655-line data_usage_define.rs (59% inline tests) into a
canonical foo.rs + foo/ module tree with zero behavior change:

- data_usage_define.rs (~950): cache constants and revision helpers,
  the data-usage tree types, DataUsageCacheInfo with its hand-written
  Serialize, the in-memory tree operations, dui, and marshal/unmarshal
- data_usage_define/persistence.rs (~580): the load/backup/restore
  ladder (load, try_load_inner, revision_for_path) and the CAS save
  path with its retry policy and save metrics
- data_usage_define/tests.rs (~2155): the inline test module as a child
  module

All module paths are unchanged (the lib.rs data_usage_define::* glob
re-export and every crate::data_usage_define:: consumer resolve as
before). The hand-written map-encoded Serialize for
DataUsageCacheInfo is moved byte-for-byte per the AGENTS.md
cross-cutting invariant; on-disk names and the cache key format const
stay in the root. Four persistence helpers used by tests gain
pub(super), whose scope equals the old single-module privacy domain.
Code is moved verbatim apart from those markers, per-module import
headers, and rustfmt re-wraps.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-20 12:23:09 +08:00
houseme 769e66511f refactor(scanner): split scanner_io.rs into child modules (#6294)
Split the 5369-line scanner_io.rs (39% inline tests) into a canonical
scanner_io.rs + scanner_io/ module tree with zero behavior change:

- scanner_io.rs (~660): constants, metadata-error constructors, the
  bucket scan plan, cycle-status classification helpers, the ScannerIO /
  ScannerIOCache / ScannerIODisk traits, and ScannerCycleResult
- scanner_io/dirty_usage.rs (~300): process-wide dirty-usage statics
  and the acknowledgment protocol
- scanner_io/guards.rs (~270): concurrency gauges and RAII guards
- scanner_io/cache.rs (~410): scanner cache locks and the snapshot
  persist/publish path
- scanner_io/io_cycle.rs (~390), io_cache.rs (~1160), io_disk.rs
  (~230): the ECStore / SetDisks / Disk trait implementations
- scanner_io/publish_gate_tests.rs (~750) and tests.rs (~1340): the two
  inline test modules as child modules

All crate paths are unchanged: the lib.rs scanner_io re-exports and
every crate::scanner_io:: consumer (scanner.rs, remote_scanner,
scanner_folder, and cross-crate rustfs users) resolve through root
re-exports with their original visibilities (pub stays pub, pub(crate)
stays pub(crate)). Cross-module items gain pub(super), whose scope
equals the old single-module privacy domain. Code is moved verbatim
apart from those markers, per-module import headers, and rustfmt
re-wraps.

The logging-guardrail nsscanner_disk skip-set_disks rule now points at
scanner_io/io_disk.rs where the function moved; the pattern and
thresholds are unchanged.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-20 12:18:45 +08:00
houseme e6fc661162 refactor(heal): split resume.rs into focused child modules (#6290)
Split the 4242-line resume.rs (46% inline tests) into a canonical
foo.rs + foo/ module tree with zero behavior change:

- resume.rs (~1020): state file constants, PersistThrottle, ResumeState,
  ResumeManager core (constructors, load/discovery, progress mutators,
  ordinary persistence) plus root re-exports
- resume/replacement.rs (~690): replacement-intent/proof types and the
  ResumeManager replacement-lifecycle methods
- resume/checkpoint.rs (~350): ResumeCheckpoint + CheckpointManager
- resume/utils.rs (~310): ResumeUtils statics
- resume/tests.rs (~1980): the inline test module as a child module

All module paths are unchanged (heal::resume::CheckpointManager and
friends resolve through root re-exports), so no consumer inside or
outside the crate changes. Items defined in child modules keep
module-private visibility; only the ten cross-module helpers gain
pub(super), which is not part of the crate API. Code is moved verbatim
apart from those visibility markers, four super::storage_api path
fixes, and the new per-module import headers.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-20 12:11:35 +08:00
houseme 305d291037 refactor(scanner): split remote_scanner.rs into stream child module (#6289)
Split the 3080-line remote_scanner.rs (47% inline tests) into a
canonical foo.rs + foo/ module tree with zero behavior change:

- remote_scanner.rs (~320): protocol constants, process statics, and
  the request decode/validate/admit/preflight/claim API plus root
  re-exports
- remote_scanner/stream.rs (~1340): wire/frame types, replay cache,
  FrameAuthenticator, serve path, local bucket scan + persist, client
  scan, and the bounded stream plumbing
- remote_scanner/stream/tests.rs (~1470): the inline test module as a
  child module of stream so it can reach both parents' private items

All crate paths are unchanged: lib.rs re-exports
(serve_remote_scanner_request, RemoteScannerRequest, ...) resolve
through root re-exports, and scanner_io's crate::remote_scanner::
{scan_remote_bucket, RemoteScannerScanSpec, RemoteScannerOutcome}
paths resolve through pub(crate) re-exports. Cross-module items gain
pub(super), whose scope equals the old single-module privacy domain;
no item's effective visibility widens. Code is moved verbatim apart
from those markers, per-module import headers, and rustfmt line
re-wraps.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-20 12:10:43 +08:00
唐小鸭 22b4ef9f0c fix(get): name the failing object on mid-stream GET body failures (#6284) 2026-08-20 03:33:31 +08:00
Zhengchao An c10d74c78b feat(connect): add device identity store and registration proof (#6267) 2026-08-20 03:32:59 +08:00
Zhengchao An 898aa4db95 chore: adjudicate the last 18 bare dead_code allows in the library crates (#6265)
* chore: adjudicate the last 18 bare dead_code allows in the library crates

Finishes backlog#1823 step 10 outside `rustfs/src` and `protocols`: config, s3select-query, common, madmin, heal, ecstore, signer and notify. Stripped first, then clippy asked which the compiler actually missed — 8 of the 18 were inert.

Seven items are deleted, each checked by grep as well as by clippy:

- `common/last_minute.rs`'s private `TimedAction` (with its impl) and `SizeCategory` (with its `Display` impl). The file's public surface — `AccElem`, `LastMinuteLatency` — stays; ecstore consumes it.
- `s3select-query`'s three `with_*` builders. `DefaultLogicalOptimizer::with_optimizer_rules` looks used, but the call in the same file is `SessionStateBuilder::with_optimizer_rules` from DataFusion; the local methods have no callers.
- `heal/manager.rs`'s `contains_key`. Its six apparent references are all `HashMap::contains_key`.

Three keep their code:

- `heal/storage.rs`'s `Test` variant is constructed by the `#[cfg(test)] test()` helper, which the lib target cannot see, so it takes a reasoned allow.
- `signer`'s `STREAMING_PAYLOAD_HDR` and `try_build_chunk_string_to_sign` gain the `_` prefix instead. That file already marks deliberately-unheld code that way — `_STREAMING_TRAILER_HDR`, `_PAYLOAD_CHUNK_SIZE`, and `_try_build_chunk_signature`, which is the only caller of that function. Following the existing convention removes the allow without an attribute.

`protocols` keeps its four; that crate needs `--features swift,sftp` to compile fully and is verified differently. The four `#![allow(dead_code)]` in `e2e_test` are module-root blankets in test-support files, which belong to steps 1-5 rather than step 10.

Refs backlog#1823

* chore(e2e_test): adjudicate the two dead_code allows the lib test target still needs

`cargo clippy --all-targets` compiles e2e_test's lib test target, which the earlier pass did not cover, so these two removals only surfaced in CI.

test_large_multipart_upload's allow was load-bearing: its call site in test_local_kms_multipart_upload is commented out behind "TODO: Re-enable after fixing streaming encryption issues with large files". The allow comes back with the reason string this batch uses everywhere else, so the next reader sees why it is parked instead of deleting a test we intend to run again.

TestDefinition.category was the opposite: written at all six definitions, read nowhere, and its enum's impl block is empty. The live copy of that type is crates/e2e_test/src/kms/test_runner.rs, which has an as_str; the policy copy is a vestige of it. Dropping the field, the enum, and the constructor parameter leaves the runner unchanged — it dispatches on name and filters on is_critical.

Verification: cargo clippy --all-targets -- -D warnings (workspace, the CI command) and cargo fmt --all --check both pass.

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-08-19 17:55:58 +00:00
houseme b1b4e443b2 perf(heal,scanner): single-flight MRF producers per detection event (#6282)
Two producer paths double-booked the same damage across repair records
(backlog#1894 axis A):

- The scanner's corrupt-metadata branch fired a durable MRF journal
  intent, an immediate High heal request, and a pending-ledger entry for
  the same object. When the MRF intent is accepted into the channel it
  already covers the repair durably (the consumer files a High Metadata
  heal and the journal replays it across restarts), so the immediate
  request and ledger entry are dropped in that case; on delivery failure
  (feature disabled, channel uninitialized, or full) the old immediate
  request + ledger path runs unchanged, keeping the repair safety net.
- The read path filed a journal intent before the read-repair
  reservation check, so a burst of reads failing on one object booked a
  journal record per retry. The intent now rides the submission: it is
  filed only when the sighting wins the dedup TTL, next to the Low
  request, via a new optional mrf_intent field on
  ReadRepairHealSubmission (None keeps the historical no-intent
  behavior for the other read-repair call sites).

Manager dedup-key semantics are untouched; the fix is that competing
producers stop double-booking. With RUSTFS_HEAL_MRF_ENABLE off both
paths behave exactly as before.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-19 17:32:16 +00:00
houseme d6efb65588 feat(heal,scanner): best-effort repaired notices from the MRF consumer (#6283)
The scanner's pending-heal ledger and the MRF journal tracked the same
damaged objects with no cross-talk: once the consumer landed an intent
with the heal manager, the ledger's retry entry for that target kept
re-submitting a heal the manager already owned (backlog#1894 axis B).

Fan the acceptance out: both dispatch sites in the MRF queue (the live
consumer and the startup replay) record a compact MrfRepairedEvent
(bucket, object, version bytes) in a bounded process-wide ring owned by
rustfs-common. The scanner drains its own bucket's notices at the top
of retry_pending_scanner_heals and clears the matching Object-kind
ledger entries in one batched retain + sync (a mass-recovery first
sweep must not turn into thousands of full-table ledger clones on the
scan task), with nil notice UUIDs mapping to None per the repo-wide
defensive-UUID invariant so unversioned entries match unversioned
notices only. Notices are best-effort by design — a lost or capped-out
notice leaves the entry to expire through its own attempts/age limits,
because the ledger is a retry oracle, not a source of truth; other
buckets' notices stay queued for their own scanners. Neither persistent
format changes; old nodes that keep double-booking remain harmless.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-19 17:31:26 +00:00
Zhengchao An 99c3811d93 fix(ecstore): classify decommission stage failures by type, not by message (#6269)
T2 of backlog#1827. `data_movement_stage_error` flattened every stage failure into `Error::other(format!(...))`, discarding the typed error. The cost was visible in tree: `is_decommission_target_capacity_error` had to match rendered text —

    let message = err.to_string();
    message.contains(&disk_full) || message.contains(&storage_full)

— to notice that the destination pool had filled up, and `is_decommission_copy_cleanup_safe_error` could not see a not-found that surfaced from inside a stage at all.

The wrapper now carries what it wrapped. `DataMovementStageError` renders the same string and returns the original through `source()`; `Error::other` boxes it through `std::io::Error`, so `data_movement_stage_source` recovers it by downcast. Both classifiers unwrap before matching, keeping their substring paths for errors that arrive through some other wrapper.

The rendered message is unchanged, which a test now pins against the exact string the old `format!` produced rather than against a `contains`. Three more cover the round trip for `DiskFull`, `StorageFull`, `FileNotFound` and `SlowDown`, that unrelated errors are not mistaken for stage wrappers, and — the case the issue names — that a not-found surfacing from inside a stage is judged cleanup-safe by the decommission loop exactly as a direct one is.

Refs backlog#1827
2026-08-19 16:04:29 +00:00
Zhengchao An 3a46baab13 refactor(admin): route observability handler auth through the shared gate (#6263)
Co-authored-by: houseme <housemecn@gmail.com>
2026-08-19 16:00:22 +00:00
houseme 81332718e6 perf(scanner): cut per-object allocations in the scan hot path (#6274) 2026-08-19 23:33:49 +08:00
houseme 0126f359e3 fix(heal): stop O(window) memmove in the heal result window (#6272) 2026-08-19 23:33:33 +08:00
houseme 10603d0870 chore(common): drop dead rule helpers and the s3s dependency (#6271) 2026-08-19 23:33:21 +08:00
houseme 7f8a8cdbac fix(madmin): send background-heal status over POST (#6270) 2026-08-19 23:33:06 +08:00
Zhengchao An cc0254d8de fix(admin): pass the real RemoteAddr so aws:SourceIp reaches policy evaluation (#6273)
backlog#1885. Six admin call sites hardcoded `None` for `validate_admin_request`'s `remote_addr`, so `aws:SourceIp` never entered the condition map for those endpoints.

`AddrFunc::evaluate` (crates/policy/src/policy/function/addr.rs:23-41) reads the key with `values.get(...)`; an absent key yields an empty iterator, the inner loop never runs, and the function returns `false`. That flips two policy shapes in opposite directions:

- `Allow` + an IpAddress whitelist stops matching, locking a legitimate admin out of these endpoints.
- `Deny` + an IpAddress blacklist also stops matching, so a source the policy means to block is let through. This one is a bypass, and it is the one nobody would report.

The sites now read the address the way the correct handlers do — `req.extensions.get::<Option<RemoteAddr>>()`, populated from the connection in `server/http.rs`.

A regression test covers both shapes at the policy layer, since that is where the direction is decided. The existing `test_iam_policy_source_ip` only exercised a present key matching or not matching; nothing covered an absent one.

A tree-wide sweep of all 88 `validate_admin_request*` call sites confirms these six were the only ones dropping the address. The issue asked whether more existed beyond the six it had found: they do not. Worth noting that a first pass checked the last argument and reported only three — `_with_bucket` takes `remote_addr` second-to-last — so the sweep is positional.

One caveat for operators, unchanged by this fix: admin authorization does not route the peer address through `crates/trusted-proxies`, so behind a reverse proxy these conditions match the proxy's address, not the client's.

Refs backlog#1885
2026-08-19 15:15:34 +00:00
Zhengchao An 1f23fd17b6 fix(scripts): stop the assertless-test census truncating bodies at string braces (#6280)
The census matched braces over raw source, so a `{` inside a string literal unbalanced the count and cut the test body short. `test_find_ellipses_patterns_leftover_brace_error_does_not_echo_input` was reported as assertionless because its input — `"http://:brace-secret@server/{1...2}}"` — ended the body before the `assert!` two lines below it.

Brace matching now runs over a literal-stripped view. The stripper carries state across lines, because the JSON and `r#"..."#` fixtures these tests are built from routinely span several; a per-line version falls out of phase on the first multi-line string and truncates far more than it fixes. Raw strings are closed on their own hash count, and a lone `'` is left alone so a lifetime (`&'a str`) is not mistaken for a char literal.

The candidate count is unchanged at 15, which is the interesting part: one entry left and one arrived. `utils/src/string.rs:942` drops out, correctly — it does assert. `io-metrics/src/lib.rs:3308` appears, also correctly — `test_record_get_object_path_and_stage` makes twenty-odd `record_*` calls and asserts nothing, the same shape #6238 fixed elsewhere in that file. It had been hidden behind a truncated body.

Refs backlog#1836
2026-08-19 15:07:21 +00:00
houseme be7f684718 refactor(heal): remove the dead MRF heal-type path (#6275)
HealType::MRF (a #1664-era "metadata repair file" task kind) had no
production construction site left: its only builder lived in the
HealEvent -> HealRequest converter, and the HealEvent/HealEventHandler
queue itself had zero production references — both were superseded by
the MrfIntent pipeline (mrf_queue.rs), which produces Object/Metadata/
ECDecode requests and never an MRF task. The dead path nevertheless
carried ~700 lines: the whole event.rs module, the heal_mrf executor,
a dedup-key arm, an overlap arm with the "\u{0}mrf" sentinel bucket
hack, per-kind labels, and an empty MrfRuntime::record_accept shell.

Deleting the variant is compile-time safe: HealType has no Serialize
derive, the protos wire enums carry no heal-type discriminant (the
receiver rebuilds it from HealChannelRequest fields), the MRF journal
encodes MrfKind (1/2/3), and the scanner pending-heal ledger uses its
own kind enum — none of them can name an MRF task.

Also resolves the in-crate naming clash where "MRF" denoted both the
dead task kind and the live mission-repair-feed loop; the loop stays,
the task kind goes.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-19 14:52:36 +00:00