mirror of
https://github.com/rustfs/rustfs.git
synced 2026-10-27 06:18:43 +00:00
唐小鸭
d457378c02
* fix(scanner): treat replication status writebacks as status-class hits A replication status writeback (persist_replication_state_if_current in Update mode, including the heals the scanner queues during its own walk) marked the bucket dirty on the namespace path: it advanced the dirty-usage generation, woke the scanner, reached the durable dirty journal and entered the activity snapshot. Any replicated bucket therefore made every scanner cycle Superseded, the authoritative usage baseline was never certified and usage freshness stayed stale for as long as replication ran (#8375). The writeback is now status class: record_status_class_mutation records only the process-local producer coverage bit and nothing else. There is no rescan queue for it, because a deployment with an active replication rule already walks every bucket on every cycle (ScannerMaintenanceFeatures::needs_regular_cycle) and that walk refreshes replication_stats. The routing lives in a named, unit-tested function in rustfs/src/storage/storage_api.rs; tier-expiration cleanup removes data and stays on the namespace path. No persisted format, protocol, admin API or configuration changes. This is phase 1 of docs/architecture/scanner-per-bucket-convergence.md, which this change adds together with its index entry. Refs #8375, #8494, rustfs/backlog#2816. * fix(scanner): refresh replication statistics during maintenance Refresh compacted prefixes and completed bucket caches during replication maintenance. Bind set execution reuse to the observed replication statistics so a same-cycle retry cannot restore older counters. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(scanner): avoid expanding the legacy S3 DTO surface Decode the existing replication configuration type from a JSON fixture instead of adding direct S3 DTO references to a new file. Preserve the compacted-prefix and same-cycle assertions and the footprint ratchet. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(scanner): keep raw enumeration bookkeeping linear per entry The page writer inserted into its building page with a push-and-sort on every entry and committed each page through a clone. Insert in sorted position and move the committed page into the index instead. Add a regression test that records 20,000 entries in one prefix and pins the cost to one hash per entry and well under a second, so the quadratic re-normalization removed by #8114 (rustfs#8113) cannot return once pacing builds on this walker. * feat(scanner): decouple bucket walkers from disk count Add the scanner key walkers_per_disk (RUSTFS_SCANNER_WALKERS_PER_DISK, 1 through 16, default 1). Each local coordinator disk of a set spawns that many bucket walkers; remote disks keep one. The walker count is what max_concurrent_disk_scans caps and the disk count stays the floor. The walkers of one disk share its scanner admission through LocalDiskAdmissionShare: the first walker to start a bucket takes the per-disk permit and the last to finish releases it, so a peer's remote request still sees one scan session per physical disk. Without the share the second walker was rejected as busy and retired. The disk_concurrency_budget log line reports walkers_per_disk, walkers and the reason for the effective limit. The default stays 1 because no local disk reports rotational media today; the design keeps a higher flash default as the intent. * docs(scanner): document walkers_per_disk in the Helm chart and configuration manuals RUSTFS_SCANNER_WALKERS_PER_DISK was only described in the scanner runtime controls table. Add it to the Helm chart (values, configmap rendering, README table, template test), the configuration crate's environment reference with operator guidance, and a short section in the operations manual on when to raise it and how to confirm it took effect. --------- Co-authored-by: Hauser <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com>