Commit Graph

15 Commits

Author SHA1 Message Date
Henry Guo 975983abdd feat(scanner): reuse clean local bucket prefixes (#7208) 2026-09-07 05:26:56 +08:00
houseme cb72df269a fix(heal): retain hints without verified repair receipts (#7275)
Stop task completion and legacy notices from discharging scanner retry hints. Preserve existing hints and their retry due time across admission observations, bound retry scheduling, and synchronize changed batches once even on cancellation.

Exercise the production MRF consumer, manager, event channel and scanner ledger. Document producer durability gaps without enabling successor activation or garbage collection.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-06 16:44:30 +08:00
houseme fddecf0afe test(scanner): diagnose raw enumeration across restarts (#7228)
* test(scanner): diagnose raw enumeration across process restarts

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): observe the canonical synthetic disk path

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): reject unobserved enumeration budget evidence

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): remove redundant disk path clone

* fix(app): keep list-through header import test-only

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-09-06 10:51:14 +08:00
houseme 159dc13548 fix(scanner): bind resumable scans and cache publication coverage (#7210)
* chore(deps): refresh scanner heal batch dependency baseline

Regenerate compatible lockfile selections before the next implementation
batch. Cargo upgrade leaves direct requirements unchanged.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): remove duplicate local rename implementation

Keep the canonical commit module after concurrent storage changes merged.
The control-write and rollback changes are already present there.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): refresh profiling dependencies for the next batch

Update hotpath and its macro crate to the compatible patch release before
the next dependency-ready implementation tasks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(deps): preserve supported hotpath focus expressions

Keep the profiler runtime before its regex-lite compatibility regression.
Track the opt-in validation required to remove this constraint in backlog.

Refs rustfs/backlog#2302.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): require complete publication coverage

Refs rustfs/backlog#2261 and rustfs/backlog#2240.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): retain scoped partial coverage across dirty plans

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): keep stable snapshot rescan behavior

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): verify coverage receipts and scan strength

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): use valid modification times in checkpoint fixtures

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): keep maintenance cycles outside dirty bucket scopes

Force complete bucket scope for deep scans and scheduled maintenance while
preserving the existing planner for verified ordinary dirty work.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): refresh scope safety independently of idle backoff

Inspect maintenance on multi-disk startup and refresh changed or failed
evidence even when explicit bitrot configuration disables idle backoff.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): bind bucket cache reuse to scan work requirements

Carry stable scan mode and full-maintenance requirements in the existing
opaque bucket digest before local and remote cache admission. Different
requirements cannot replay a same-cycle Normal cache after root delivery
failure; matching requirements remain reusable for the same intent.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): fence set snapshot reuse with the scan work proof

Prevent same-cycle set publication from replacing freshly scanned maintenance
results with an older Normal aggregate. Recognize uniform completed
maintenance baselines when planning later ordinary dirty-bucket work.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): reproduce same-cycle dirty aggregate replay

Cover a Normal-to-Normal retry with a new dirty bucket generation after
bucket persistence and root delivery failure.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): fence same-cycle caches with full activity coverage

Keep structural baseline identity separate from the full activity coverage
required by bucket admission and set publication. Require complete set
coverage proofs while retaining revision CAS and epoch regression checks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): supply explicit coverage in publication fixtures

Keep the confirmed-empty namespace fixture authoritative under the required
coverage contract and qualify the bucket cache metadata test type.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): verify joint checkpoint coverage metadata

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): satisfy cache prefix sort lint

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
2026-09-06 07:27:18 +08:00
houseme 2e4ab045b6 test(scanner): add durable checkpoint diagnostics (#7175)
* chore(deps): refresh SDKs and pin clock skew regression coverage

Refresh compatible dependencies for Scanner/Heal V2 batch 1 and verify
the production S3 retry/signing path with a deterministic clock.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): add durable checkpoint diagnostics

Refs rustfs/backlog#2260 and rustfs/backlog#2240.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-05 07:34:29 +00:00
Zhengchao An 1585308f0f fix(test): restore #[serial] markers the fallback runner still needs (#6767) 2026-08-28 08:41:36 +08:00
Zhengchao An a96dd7d289 refactor: migrate consumers off rustfs-common heal/scanner shims (#6623)
* refactor(ecstore): import heal/scanner contracts crates directly (backlog#1843)

* refactor(heal): import heal/scanner contracts crates directly (backlog#1843)

* refactor(lifecycle): import heal/scanner contracts crates directly (backlog#1843)

* refactor(obs): import heal/scanner contracts crates directly (backlog#1843)

* refactor(protos): import heal/scanner contracts crates directly (backlog#1843)

* refactor(scanner): import heal/scanner contracts crates directly (backlog#1843)

* refactor(rustfs): import heal/scanner contracts crates directly (backlog#1843)
2026-08-26 21:13:18 +08:00
Zhengchao An 0e015360cc test(scanner): repair post-fence fixtures (#6478) 2026-08-24 09:33:21 +08:00
cxymds b2e60be647 fix(scanner): fence unknown tier accounting (#6396) 2026-08-23 19:28:43 +08:00
cxymds 76a863b3ea fix(scanner): unify unknown metadata size accounting (#6394)
* fix(scanner): unify unknown metadata size accounting

* fix(scanner): preserve restore expiry semantics

* fix(ci): resolve ecstore clippy warnings

* fix(scanner): close lifecycle review gaps

---------

Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-08-23 17:28:52 +08:00
cxymds 9cda615519 fix(scanner): discover sub-quorum heal candidates (#6384)
* fix(scanner): preserve unversioned heal retries

* fix(scanner): bound orphan heal discovery fallback

* fix(filemeta): fence unsafe heal key components

* fix(scanner): preserve exact overflow heal versions

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-08-23 16:46:36 +08:00
cxymds 32cc7c8fcf fix(heal): coalesce duplicate MRF intents (#6425)
Co-authored-by: houseme <housemecn@gmail.com>
2026-08-23 16:45:22 +08:00
Zhengchao An 5d820df79c chore(obs): rename ReplicationStats to ReplicationMetricsSnapshot (#6344)
* chore(obs): ReplicationStats -> ReplicationMetricsSnapshot, BucketReplicationStats -> BucketReplicationMetricsSnapshot

Rename in-obs-crate ReplicationStats and BucketReplicationStats to
ReplicationMetricsSnapshot and BucketReplicationMetricsSnapshot respectively.
No serde impact (these types are Prometheus metric collectors, not serialized).
No external consumers found outside the obs crate.

* cleanup: remove #[serial] annotations from e2e_test, scanner, lifecycle, and object-capacity crates

Remove no-op #[serial] attributes (nextest ignores serial_test) and the
serial_test dependency from four crates. All tests already use temp_env
for env-var isolation, making #[serial] purely redundant.

Crates cleaned:
- e2e_test (37 annotations, 9 imports, removed serial_test dep)
- rustfs-scanner (115 annotations across 7 files, removed serial_test dep)
- rustfs-lifecycle (46 annotations, removed serial_test dep)
- rustfs-object-capacity (38 annotations, removed serial_test dep)

Also converted scanner/tests/lifecycle_integration_test.rs
with_forced_immediate_enqueue_timeout helper from unsafe raw
env::set_var/remove_var to temp_env::async_with_vars for proper
isolation, and added async_closure feature to scanner's temp-env dep.

* fix(lifecycle): restore #[serial] on 2 tests that read env vars without temp_env

eval_inner_expires_latest_object_after_days_due and
eval_inner_does_not_panic_on_many_equal_due_events call eval_inner()
which reads ENV_ILM_PROCESS_TIME via std::env::var(). Without #[serial]
they race with other tests that set these vars via temp_env.

* style: cargo fmt
2026-08-22 00:30:30 +00:00
houseme a247c79359 perf(heal): trim scanner and heal queue hot paths (#6307)
Cache heal queue dedup keys, avoid retry request double construction, clear task aliases after terminal completion, and age out stale scanner pending-heal ledger entries during retry sweeps.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-20 20:28:22 +08:00
houseme 129677f0b3 refactor(scanner): split scanner_folder item actions and ledger (#6302)
* refactor(scanner): split scanner_folder item actions and ledger

Split the 6345-line scanner_folder.rs (46% inline tests) into a
canonical scanner_folder.rs + scanner_folder/ module tree with zero
behavior change:

- scanner_folder.rs (~2280): scan constants, alert cooldowns, metric
  accounting, resume ordering, tracing helpers, the FolderScanner
  struct with failed-object bookkeeping and the scan_folder traversal,
  and scan_data_folder
- scanner_folder/item_actions.rs (~890): CachedFolder, the get-size
  failure policy, ScannerItem with apply_actions and the heal/ILM
  admission helpers
- scanner_folder/ledger.rs (~280): the pending-scanner-heal ledger
  methods and their entry helpers (record/prune/clear-for-repaired/
  retry)
- scanner_folder/tests.rs (~2950): the inline test module as a child
  module

The ScannerItem path used by scanner_io resolves through a root
re-export, and every other crate path is unchanged. Cross-module items
gain pub(super), whose scope equals the old single-module privacy
domain. Code is moved verbatim apart from those markers, per-module
import headers, and rustfmt re-wraps.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fmt

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-20 19:11:55 +08:00