feat(scanner): decouple bucket walkers from disk count (#8539)

* fix(scanner): treat replication status writebacks as status-class hits

A replication status writeback (persist_replication_state_if_current in
Update mode, including the heals the scanner queues during its own walk)
marked the bucket dirty on the namespace path: it advanced the dirty-usage
generation, woke the scanner, reached the durable dirty journal and
entered the activity snapshot. Any replicated bucket therefore made every
scanner cycle Superseded, the authoritative usage baseline was never
certified and usage freshness stayed stale for as long as replication ran
(#8375).

The writeback is now status class: record_status_class_mutation records
only the process-local producer coverage bit and nothing else. There is
no rescan queue for it, because a deployment with an active replication
rule already walks every bucket on every cycle
(ScannerMaintenanceFeatures::needs_regular_cycle) and that walk refreshes
replication_stats. The routing lives in a named, unit-tested function in
rustfs/src/storage/storage_api.rs; tier-expiration cleanup removes data
and stays on the namespace path.

No persisted format, protocol, admin API or configuration changes.

This is phase 1 of docs/architecture/scanner-per-bucket-convergence.md,
which this change adds together with its index entry.

Refs #8375, #8494, rustfs/backlog#2816.

* fix(scanner): refresh replication statistics during maintenance

Refresh compacted prefixes and completed bucket caches during replication maintenance. Bind set execution reuse to the observed replication statistics so a same-cycle retry cannot restore older counters.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): avoid expanding the legacy S3 DTO surface

Decode the existing replication configuration type from a JSON fixture instead of adding direct S3 DTO references to a new file. Preserve the compacted-prefix and same-cycle assertions and the footprint ratchet.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(scanner): keep raw enumeration bookkeeping linear per entry

The page writer inserted into its building page with a push-and-sort on
every entry and committed each page through a clone. Insert in sorted
position and move the committed page into the index instead.

Add a regression test that records 20,000 entries in one prefix and pins
the cost to one hash per entry and well under a second, so the quadratic
re-normalization removed by #8114 (rustfs#8113) cannot return once pacing
builds on this walker.

* feat(scanner): decouple bucket walkers from disk count

Add the scanner key walkers_per_disk (RUSTFS_SCANNER_WALKERS_PER_DISK,
1 through 16, default 1). Each local coordinator disk of a set spawns that
many bucket walkers; remote disks keep one. The walker count is what
max_concurrent_disk_scans caps and the disk count stays the floor.

The walkers of one disk share its scanner admission through
LocalDiskAdmissionShare: the first walker to start a bucket takes the
per-disk permit and the last to finish releases it, so a peer's remote
request still sees one scan session per physical disk. Without the share
the second walker was rejected as busy and retired.

The disk_concurrency_budget log line reports walkers_per_disk, walkers and
the reason for the effective limit. The default stays 1 because no local
disk reports rotational media today; the design keeps a higher flash
default as the intent.

* docs(scanner): document walkers_per_disk in the Helm chart and configuration manuals

RUSTFS_SCANNER_WALKERS_PER_DISK was only described in the scanner runtime
controls table. Add it to the Helm chart (values, configmap rendering,
README table, template test), the configuration crate's environment
reference with operator guidance, and a short section in the operations
manual on when to raise it and how to confirm it took effect.

---------

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

docs/architecture/scanner-per-bucket-convergence.md

@@ -253,9 +253,9 @@ for within-cycle reuse.
 
 ## Pacing and worker concurrency
 
-Today the number of bucket walkers per erasure set equals the number of
-disks in the set, capped by `max_concurrent_disk_scans`, and
-`scanner_concurrency_limit` takes the minimum of the configured value and the
+Before phase 3 the number of bucket walkers per erasure set equalled the
+number of disks in the set, capped by `max_concurrent_disk_scans`, and
+`scanner_concurrency_limit` took the minimum of the configured value and the
 disk count. Buckets queue on one channel per set and each walker takes one
 bucket at a time; inside a bucket the folder walk is sequential and
 `DynamicSleeper` sleeps after every object for the work time multiplied by
@@ -273,7 +273,14 @@ resource ceiling. The scanner adopts the same shape:
   count multiplied by a per-disk walker factor. The factor defaults from disk
   type: 1 for rotational media, higher for flash, where xl.meta decode is CPU
   bound and there is no seek penalty. Disk count stays the floor so
-  distributed deployments keep at least today's parallelism.
+  distributed deployments keep at least today's parallelism. Shipped as
+  `scanner.walkers_per_disk` (`RUSTFS_SCANNER_WALKERS_PER_DISK`, `1` through
+  `16`): each local coordinator disk gets that many walkers, which share one
+  per-disk scanner admission so a peer still sees one scan session per
+  physical disk; remote disks keep one walker. The default is `1` everywhere
+  until a local disk can report its media type
+  (`rustfs_utils::os::Info::rotational` is never populated today); the flash
+  default above `1` remains the intent.
 - **Throttle**: the `DynamicSleeper` factor is the pacer's output, not a
   fixed preset. `ScannerSpeed` presets remain as operator overrides and as
   the initial value. Foreground workload backoff in `sleeper.rs` remains the
@@ -298,7 +305,7 @@ not tuning advice.
 | Pacer raises walkers and lowers sleep | CPU and I/O ceilings are absolute. Foreground admission backoff cannot be undercut. Walker count never exceeds the ceiling-derived maximum even when the target is missed. |
 | Faster convergence discovers heal and replication work sooner | Heal admission keeps its queue bounds and `queue_full` and `dropped` accounting in `crates/scanner/src/scanner_folder/ledger.rs`; replication MRF keeps its pending cap. Convergence speed does not change retry intervals. Discovered work is swept lazily under those bounds, in the sense of a lazy sweeper: discovery and execution are decoupled. |
 | Guardrail vetoes publication | A veto is attributed to a field and localised to a peer where the field allows it; only movement and topology defer the whole cycle. Deferred retries use the capped superseded backoff. A probe failure on one peer never costs more than that peer's dirtied buckets. |
-| One bucket walk may be large | Inside a bucket, the walk takes a per-prefix quantum: a prefix that exhausts it checkpoints and yields to the next prefix or bucket, and resumes under the existing cursor and page-index fences. No single prefix may consume the whole cycle budget. The walker's own bookkeeping must stay linear in entries per prefix; `RawEnumerationProgress::record_entry` in `crates/scanner/src/scanner_folder.rs` copies the accumulated entry list on every entry today and is a prerequisite fix for pacing. |
+| One bucket walk may be large | Inside a bucket, the walk takes a per-prefix quantum: a prefix that exhausts it checkpoints and yields to the next prefix or bucket, and resumes under the existing cursor and page-index fences. No single prefix may consume the whole cycle budget. The walker's own bookkeeping must stay linear in entries per prefix. `RawEnumerationProgress::record_entry` in `crates/scanner/src/scanner_folder.rs` once re-normalized the accumulated entry list on every entry (rustfs#8113: 8,000 entries cost about 40 s); since #8114 the page writer hashes each entry once when its page commits, and a regression test pins 20,000 entries in one prefix to that cost so pacing cannot be built on a quadratic walker. |
 
 ## What stays fail-closed
 
@@ -444,10 +451,12 @@ modes.
    a hundred thousand entries no longer starves the buckets after it; a
    multi-node deployment with one peer's probe failing publishes every bucket
    that peer did not dirty.
-3. **Pacer and walker concurrency.** Fix the quadratic entry bookkeeping in
-   `RawEnumerationProgress::record_entry` first, then decouple walker count
-   from disk count and drive the throttle from the pacing target under CPU
-   and I/O ceilings. Oracle: on a single flash disk the same namespace
+3. **Pacer and walker concurrency.** Pin the entry bookkeeping in
+   `RawEnumerationProgress::record_entry` to linear cost first (the
+   clone-per-entry itself was removed by #8114; the regression test and the
+   in-place page writer are this phase's groundwork), then decouple walker
+   count from disk count (`walkers_per_disk`, shipped) and drive the throttle
+   from the pacing target under CPU and I/O ceilings (open). Oracle: on a single flash disk the same namespace
    converges in a fraction of today's cycle time with scanner CPU under the
    ceiling; on rotational media the default factor keeps today's I/O
    profile; a foreground write load at the admission threshold still backs

docs/operations/scanner-runtime-controls.md

@@ -43,7 +43,8 @@ Bitrot cycle resolution differs because the canonical persistent key belongs to
 | `scanner.idle_mode` | `RUSTFS_SCANNER_IDLE_MODE` | boolean | `true` (`DEFAULT_SCANNER_IDLE_MODE`) | Enables preset scanner sleeps and foreground-read backoff. `false` disables those sleeps; per-disk foreground-priority admission remains active. |
 | `scanner.cache_save_timeout` | `RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS` | seconds | `14` (`DEFAULT_SCANNER_CACHE_SAVE_TIMEOUT_SECS`) | Timeout for saving scanner cache; runtime enforces a minimum of `1` and keeps the default persistence budget within the distributed publication lease. |
 | `scanner.max_concurrent_set_scans` | `RUSTFS_SCANNER_MAX_CONCURRENT_SET_SCANS` | count | `4` (`DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS`) | Caps concurrent set-level scanner tasks. `0` keeps topology-derived concurrency. |
-| `scanner.max_concurrent_disk_scans` | `RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS` | count | `4` (`DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS`) | Caps concurrent disk bucket walks per set. `0` keeps disk-count-derived concurrency. |
+| `scanner.max_concurrent_disk_scans` | `RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS` | count | `4` (`DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS`) | Caps concurrent disk bucket walks per set. `0` keeps walker-count-derived concurrency. |
+| `scanner.walkers_per_disk` | `RUSTFS_SCANNER_WALKERS_PER_DISK` | count | `1` (`DEFAULT_SCANNER_WALKERS_PER_DISK`) | Bucket walkers spawned per local disk of a set. The walkers of one disk share its scanner admission, so a peer still sees one scan session per physical disk; remote disks keep one walker. The resulting walker count is what `scanner.max_concurrent_disk_scans` caps, and the disk count stays the floor. Valid range is `1` through `16` (`MAX_SCANNER_WALKERS_PER_DISK`); `1` keeps today's I/O profile and is the right value for rotational media. The `disk_concurrency_budget` log line reports `walkers_per_disk`, `walkers`, the effective `concurrency_limit`, and its `reason` (`available_walkers`, `configured_cap`, or `count_budget`). |
 | `scanner.yield_every_n_objects` | `RUSTFS_SCANNER_YIELD_EVERY_N_OBJECTS` | objects | `128` (`DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS`) | Controls how often object loops yield to the async runtime. `0` disables this extra yield. |
 | `scanner.alert_excess_versions` | `RUSTFS_SCANNER_ALERT_EXCESS_VERSIONS` | versions | `100` (`DEFAULT_SCANNER_ALERT_EXCESS_VERSIONS`) | Version count threshold for scanner alerts. |
 | `scanner.alert_excess_version_size` | `RUSTFS_SCANNER_ALERT_EXCESS_VERSION_SIZE` | bytes | `1099511627776` (`DEFAULT_SCANNER_ALERT_EXCESS_VERSION_SIZE`) | Retained version byte threshold for scanner alerts. |
@@ -61,6 +62,16 @@ Speed presets (`crates/config/src/constants/scanner.rs`) set the base sleep mult
 
 Use `scanner.delay`, `scanner.max_wait`, and `scanner.cycle` when the preset is close but one axis needs a precise override. With `idle_mode=true`, directory-level sleep is `1ms x factor` and object-level sleep is `time spent on the object x factor`, both capped at `max_wait`; a foreground-read floor of `FOREGROUND_READ_BACKOFF_PER_REQUEST_MS` (10ms) per concurrent GetObject/streaming read, capped at `FOREGROUND_READ_BACKOFF_MAX_MS` (250ms), is applied on top and can exceed the preset's `max_wait` (`crates/scanner/src/sleeper.rs`). The per-disk foreground-priority gate is independent of `idle_mode` and remains active when sleeps are disabled.
 
+### Walkers per disk
+
+The scanner walks the buckets of an erasure set with one walker per local disk by default, so a single-disk deployment walks one bucket at a time no matter how `scanner.max_concurrent_disk_scans` is set. `scanner.walkers_per_disk` (`RUSTFS_SCANNER_WALKERS_PER_DISK`) multiplies each local disk into that many walkers.
+
+- Keep the default `1` on rotational media: several walkers on one spindle compete for seeks and slow every walk down.
+- On flash, `2` to `4` is the useful range. Raise it when the scanner status shows cycles ending on the cycle budget (`metrics.pacing_pressure.last_cycle_budget_limited`) or when many buckets are never reached within a cycle, and the disk has idle I/O capacity.
+- The walkers of one disk share that disk's scanner admission, so a peer still sees one scan session per physical disk; remote disks keep one walker. `scanner.max_concurrent_disk_scans` still caps the resulting walker count, and the disk count stays the floor.
+- Confirm the setting took effect from `runtime_config.walkers_per_disk.value` in the admin scanner status and from the `disk_concurrency_budget` log line, which reports `walkers_per_disk`, `walkers`, the effective `concurrency_limit`, and the `reason` it was chosen.
+- Helm: `config.rustfs.scanner.walkers_per_disk`; Docker or systemd: the environment variable. The key is additive, so an older binary ignores it.
+
 ### Environment-only scanner knobs
 
 These have no persistent key and are read from the environment only.
@@ -144,6 +155,7 @@ runtime_config.delay.value
 runtime_config.max_wait_seconds.value
 runtime_config.cycle_interval_seconds.value
 runtime_config.bitrot_cycle_seconds.value
+runtime_config.walkers_per_disk.value
 cycle_schedule.effective_interval_seconds
 cycle_schedule.clean_idle_backoff_enabled
 cycle_schedule.clean_idle_backoff_multiplier