docs(scanner): add benchmark runbook (#3412)

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-07-26 08:18:18 +00:00 · 2026-06-13 22:39:13 +08:00
parent 6b0d7e06c4
commit 0d16a86d9a
6 changed files with 311 additions and 2 deletions
@@ -0,0 +1,298 @@
+# Scanner Benchmark Runbook
+
+This runbook describes how to collect reproducible evidence for scanner
+pressure, scanner progress, and scanner runtime tuning. It is intended for
+maintainers and operators validating scanner changes in an isolated test
+deployment.
+
+Use this runbook together with
+[Scanner Runtime Controls](scanner-runtime-controls.md). The runtime controls
+document explains each field and configuration key; this document explains how
+to run a comparable before/after observation and decide whether the scanner is
+healthier after tuning.
+
+## Scope
+
+This runbook verifies that scanner pacing and cycle controls reduce background
+pressure while preserving maintenance progress. It is useful for:
+
+- mostly idle single-node, single-disk deployments with many small objects;
+- single-node multi-disk or erasure-set deployments where scanner work is
+  spread across disks and sets;
+- distributed clusters where scanner pressure is mixed with lifecycle,
+  replication, heal, or bitrot queues;
+- lifecycle expiry or transition backlog investigations;
+- bucket replication repair backlog investigations;
+- scanner-originated heal or bitrot admission investigations;
+- pull request evidence when scanner behavior, scanner status, or scanner
+  controls change.
+
+This runbook does not prove full MinIO parity, site-replication correctness,
+or replaced-disk heal correctness. Those flows need dedicated distributed
+tests because their failure modes are not limited to scanner pacing.
+
+## Safety
+
+Run the workload only in a disposable test environment. The commands below can
+create many buckets and objects and can overwrite runtime scanner settings.
+
+Record the current scanner and heal configuration before changing anything:
+
+```bash
+mkdir -p artifacts
+mc admin config get ALIAS scanner > artifacts/scanner-config.before.txt
+mc admin config get ALIAS heal > artifacts/heal-config.before.txt
+```
+
+Replace `ALIAS`, endpoint, and credentials with values for the test
+deployment. Do not paste production credentials into saved artifacts.
+
+## Required Tools
+
+- `mc` or a compatible admin client for config changes.
+- `awscurl` or another SigV4-capable HTTP client for `/v3/scanner/status`.
+- `jq` for status extraction.
+- `pidstat`, `mpstat`, `iostat`, `top`, or equivalent host telemetry.
+- `warp`, `s3bench`, or the repository object benchmark scripts for workload
+  generation.
+
+## Test Matrix
+
+At minimum, collect two runs on the same RustFS commit and the same workload:
+
+| Run | Purpose | Example scanner settings |
+|---|---|---|
+| Baseline | Observe current behavior without additional pacing changes. | Existing config. |
+| Pacing override | Measure whether cooperative scanner sleeps reduce pressure. | `scanner.delay="30"` and `scanner.max_wait="15"`. |
+
+Add bounded-cycle runs when a single scan cycle is too long:
+
+| Run | Purpose | Example scanner settings |
+|---|---|---|
+| Duration budget | Bound wall-clock time per cycle. | `scanner.cycle_max_duration="1800"`. |
+| Object budget | Bound objects processed per cycle. | `scanner.cycle_max_objects="1000000"`. |
+| Directory budget | Bound directories entered per cycle. | `scanner.cycle_max_directories="100000"`. |
+
+When comparing runs, keep hardware, RustFS commit, object count, object size,
+bucket count, scanner-enabled state, and foreground workload constant.
+
+## Deployment Matrix
+
+Use the smallest deployment that reproduces the symptom, but do not treat
+single-node validation as the whole scanner test surface. Scanner changes that
+touch queues, admission, or cross-node maintenance should include a distributed
+run when practical.
+
+| Deployment | What it validates | Minimum evidence |
+|---|---|---|
+| Single-node, single-disk | Small-object scanner pressure, pacing, cycle interval, and basic progress. | Scanner status time series plus host CPU and disk telemetry. |
+| Single-node, multi-disk or erasure set | Set and disk scan concurrency, cycle budgets, checkpoint movement, usage cache persistence, and active path age. | Scanner status time series, per-disk host telemetry, and before/after data usage freshness. |
+| Distributed cluster | Lifecycle transition queues, bucket replication repair admission, scanner-originated heal and bitrot admission, and queue/backlog pressure under cross-node work. | Scanner status time series from the cluster, host telemetry from each node, and subsystem-specific queued/skipped/missed counters. |
+
+The single-node, single-disk run is the baseline because it is cheap and
+repeatable. It is not sufficient for PRs that claim to improve distributed
+queue behavior, replication repair, or heal/bitrot admission.
+
+## Workload Shape
+
+For the baseline small-object scanner-pressure run, use a workload that
+creates many small objects and then leaves the service mostly idle while the
+scanner walks the namespace. A useful minimum shape is:
+
+- one RustFS node;
+- one data disk;
+- several buckets;
+- at least 100,000 small objects total;
+- scanner enabled;
+- no sustained foreground workload during the observation window.
+
+For a distributed backlog run, use the same structure but add the relevant
+subsystem condition, such as lifecycle rules, a bucket replication target, or
+a configured heal/bitrot scenario. Keep the status and host telemetry cadence
+the same so the result remains comparable with the baseline run.
+
+The repository object benchmark script can generate object traffic if `warp`
+or `s3bench` is installed:
+
+```bash
+scripts/run_object_batch_bench.sh \
+  --tool warp \
+  --endpoint http://127.0.0.1:9000 \
+  --access-key "$RUSTFS_ACCESS_KEY" \
+  --secret-key "$RUSTFS_SECRET_KEY" \
+  --bucket scanner-bench \
+  --auto-new-bucket \
+  --concurrency 64 \
+  --duration 10m \
+  --sizes 1KiB,4KiB,16KiB \
+  --warp-mode put \
+  --out-dir artifacts/object-load
+```
+
+If the object generator cannot create enough objects in one run, repeat the
+same command with new buckets or prefixes and record the final object count.
+
+## Status Collection
+
+Capture scanner status before the workload, after the workload finishes, and
+throughout the idle observation window.
+
+Example status request:
+
+```bash
+awscurl \
+  --service s3 \
+  --region us-east-1 \
+  --access_key "$RUSTFS_ACCESS_KEY" \
+  --secret_key "$RUSTFS_SECRET_KEY" \
+  --request GET \
+  'http://127.0.0.1:9000/rustfs/admin/v3/scanner/status' \
+  | jq . > "artifacts/scanner-status.$(date -u +%Y%m%dT%H%M%SZ).json"
+```
+
+For a time series, sample once per minute:
+
+```bash
+mkdir -p artifacts/status
+while sleep 60; do
+  ts="$(date -u +%Y%m%dT%H%M%SZ)"
+  awscurl \
+    --service s3 \
+    --region us-east-1 \
+    --access_key "$RUSTFS_ACCESS_KEY" \
+    --secret_key "$RUSTFS_SECRET_KEY" \
+    --request GET \
+    'http://127.0.0.1:9000/rustfs/admin/v3/scanner/status' \
+    | jq . > "artifacts/status/scanner-status.${ts}.json"
+done
+```
+
+Stop the loop after the planned observation window.
+
+## Host Telemetry
+
+Collect host metrics over the same window as scanner status.
+
+```bash
+pidstat -p "$(pidof rustfs)" 60 > artifacts/pidstat.txt
+iostat -xz 60 > artifacts/iostat.txt
+mpstat 60 > artifacts/mpstat.txt
+```
+
+If `pidstat` is not available, use `top`, `ps`, or the platform monitoring
+system, but keep the sampling interval and observation window in the report.
+
+## Runtime Tuning Examples
+
+Persistent scanner config values use seconds for time fields. Use numeric
+strings instead of duration suffixes:
+
+```bash
+mc admin config set ALIAS scanner delay="30" max_wait="15"
+mc admin config set ALIAS scanner cycle="3600"
+mc admin config set ALIAS scanner cycle_max_duration="1800"
+mc admin config set ALIAS scanner cycle_max_objects="1000000"
+mc admin config set ALIAS scanner cycle_max_directories="100000"
+```
+
+The canonical persistent bitrot cadence belongs to the `heal` subsystem:
+
+```bash
+mc admin config set ALIAS heal bitrot_cycle="2592000"
+```
+
+Environment variables take precedence over persisted config and should be
+recorded separately:
+
+```bash
+RUSTFS_SCANNER_DELAY=30
+RUSTFS_SCANNER_MAX_WAIT_SECS=15
+RUSTFS_SCANNER_CYCLE=3600
+RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS=1800
+RUSTFS_SCANNER_CYCLE_MAX_OBJECTS=1000000
+RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES=100000
+RUSTFS_SCANNER_BITROT_CYCLE_SECS=2592000
+```
+
+After each config change, read scanner status and confirm the effective value
+and `source` under `runtime_config`.
+
+## Observation Window
+
+Use the same window for each run:
+
+1. Generate or verify the object namespace.
+2. Wait until foreground workload is idle.
+3. Save scanner and heal config.
+4. Save one scanner status snapshot.
+5. Collect scanner status and host telemetry for at least 30 minutes, or for
+   one complete scanner cycle when that is practical.
+6. Save one final scanner status snapshot.
+
+Longer windows are better for cycle interval comparisons. Short windows are
+acceptable for quick pressure checks only if the conclusion avoids changing
+defaults.
+
+## Fields To Compare
+
+Compare these fields between baseline and tuned runs:
+
+| Field | Why it matters |
+|---|---|
+| `runtime_config.*.value` and `runtime_config.*.source` | Confirms the tested settings actually took effect. |
+| `metrics.pacing_pressure.primary_pressure` | Shows whether pressure is from queues, budgets, pause activity, active scans, or no scanner pressure. |
+| `metrics.pacing_pressure.last_cycle_total_pause_ratio` | Shows how much of the last cycle was cooperative scanner pause time. |
+| `metrics.current_cycle_objects_scanned` | Confirms object scan progress during the current cycle. |
+| `metrics.current_cycle_directories_scanned` | Confirms directory walk progress during the current cycle. |
+| `metrics.last_cycle_result` | Confirms whether the previous cycle completed, stopped partially, or failed. |
+| `metrics.last_cycle_partial_reason` | Shows which budget stopped a partial cycle. |
+| `metrics.last_cycle_partial_source` | Shows which scanner work source consumed the stopping budget. |
+| `metrics.source_work` | Shows cumulative work found, queued, skipped, missed, executed, and failed by source. |
+| `metrics.current_cycle_source_work` | Shows which source is consuming the current scan cycle. |
+| `metrics.last_cycle_source_work` | Shows which source consumed the previous scan cycle. |
+| `metrics.lifecycle_transition.scanner_missed` | Shows scanner-discovered transition work that could not be queued. |
+| `metrics.lifecycle_transition.queue_full` | Shows transition queue pressure outside the scanner walk itself. |
+| `metrics.scan_checkpoint` | Confirms partial cycles preserve resume context. |
+| `metrics.oldest_active_path_age_seconds` | Helps identify scanner paths that may be stuck. |
+
+Do not use a single CPU spike as the conclusion. Compare average and p95 CPU
+over the same observation window.
+
+## Interpreting Results
+
+A useful tuning result has all of these properties:
+
+- average or p95 scanner-related CPU and disk pressure decreases;
+- `current_cycle_objects_scanned` or `current_cycle_directories_scanned`
+  continues to advance;
+- `source_work.missed` does not grow unexpectedly for lifecycle, replication,
+  heal, or bitrot;
+- `last_cycle_result` is either `success` or a partial result with a clear
+  budget reason and checkpoint;
+- data usage freshness remains acceptable for the tested deployment.
+
+Treat these as failure signals:
+
+- CPU drops only because the scanner stops making progress;
+- `primary_pressure` stays at `queued_scans` while queues grow;
+- `last_cycle_partial_reason` repeats forever with no checkpoint movement;
+- lifecycle transition `scanner_missed` or `queue_full` grows during a run that
+  was expected to reduce backlog;
+- heal or bitrot work moves from `queued` to `missed` after a scanner pacing
+  change.
+
+## PR Evidence Checklist
+
+For scanner behavior PRs, include this evidence when available:
+
+- RustFS commit SHA and branch.
+- Deployment shape: node count, disk count, disk type, CPU count, memory, and
+  object count.
+- Workload command or script and benchmark artifact path.
+- Scanner and heal config before and after tuning.
+- Observation window and sample interval.
+- Scanner status snapshots or time series.
+- Host CPU and disk telemetry.
+- Short conclusion that separates pressure reduction from scanner progress.
+
+For documentation-only PRs, it is enough to verify links and formatting.