From 0d16a86d9a0be9a1b601d37e3214e9dbe67412b2 Mon Sep 17 00:00:00 2001 From: Henry Guo Date: Sat, 13 Jun 2026 22:39:13 +0800 Subject: [PATCH] docs(scanner): add benchmark runbook (#3412) Co-authored-by: Henry Guo --- README.md | 4 +- crates/scanner/README.md | 2 + crates/scanner/README.zh-CN.md | 2 + docs/operations/scanner-benchmark-runbook.md | 298 +++++++++++++++++++ docs/operations/scanner-runtime-controls.md | 3 + helm/README.md | 4 +- 6 files changed, 311 insertions(+), 2 deletions(-) create mode 100644 docs/operations/scanner-benchmark-runbook.md diff --git a/README.md b/README.md index fdaa04025..c26e18e96 100644 --- a/README.md +++ b/README.md @@ -212,7 +212,9 @@ Follow the instructions in the [Helm Chart README](https://charts.rustfs.com/) t For scanner pacing, cycle budgets, bitrot cadence, lifecycle transition status, and single-node single-disk idle CPU tuning, see -[Scanner Runtime Controls](docs/operations/scanner-runtime-controls.md). +[Scanner Runtime Controls](docs/operations/scanner-runtime-controls.md). For +repeatable scanner-pressure validation, see +[Scanner Benchmark Runbook](docs/operations/scanner-benchmark-runbook.md). ### 5\. Nix Flake (Option 5) diff --git a/crates/scanner/README.md b/crates/scanner/README.md index 13f95a9a6..ad022311e 100644 --- a/crates/scanner/README.md +++ b/crates/scanner/README.md @@ -6,6 +6,8 @@ admission, scanner-driven heal/bitrot checks, and namespace alerts. For operator-facing runtime controls, status fields, and tuning workflows, see [Scanner Runtime Controls](../../docs/operations/scanner-runtime-controls.md). +For repeatable scanner-pressure validation, see +[Scanner Benchmark Runbook](../../docs/operations/scanner-benchmark-runbook.md). Chinese documentation is available in [README.zh-CN.md](README.zh-CN.md). diff --git a/crates/scanner/README.zh-CN.md b/crates/scanner/README.zh-CN.md index 3a3a43c54..2e707f2c3 100644 --- a/crates/scanner/README.zh-CN.md +++ b/crates/scanner/README.zh-CN.md @@ -4,6 +4,8 @@ RustFS Scanner 是后台维护扫描循环，负责用量统计、生命周期 面向运维人员的运行时控制项、状态字段和调参流程，请参考 [Scanner Runtime Controls](../../docs/operations/scanner-runtime-controls.md)。 +可复现的 scanner 压力验证流程请参考 +[Scanner Benchmark Runbook](../../docs/operations/scanner-benchmark-runbook.md)。 英文文档请参考 [README.md](README.md)。 diff --git a/docs/operations/scanner-benchmark-runbook.md b/docs/operations/scanner-benchmark-runbook.md new file mode 100644 index 000000000..588097b1f --- /dev/null +++ b/docs/operations/scanner-benchmark-runbook.md @@ -0,0 +1,298 @@ +# Scanner Benchmark Runbook + +This runbook describes how to collect reproducible evidence for scanner +pressure, scanner progress, and scanner runtime tuning. It is intended for +maintainers and operators validating scanner changes in an isolated test +deployment. + +Use this runbook together with +[Scanner Runtime Controls](scanner-runtime-controls.md). The runtime controls +document explains each field and configuration key; this document explains how +to run a comparable before/after observation and decide whether the scanner is +healthier after tuning. + +## Scope + +This runbook verifies that scanner pacing and cycle controls reduce background +pressure while preserving maintenance progress. It is useful for: + +- mostly idle single-node, single-disk deployments with many small objects; +- single-node multi-disk or erasure-set deployments where scanner work is + spread across disks and sets; +- distributed clusters where scanner pressure is mixed with lifecycle, + replication, heal, or bitrot queues; +- lifecycle expiry or transition backlog investigations; +- bucket replication repair backlog investigations; +- scanner-originated heal or bitrot admission investigations; +- pull request evidence when scanner behavior, scanner status, or scanner + controls change. + +This runbook does not prove full MinIO parity, site-replication correctness, +or replaced-disk heal correctness. Those flows need dedicated distributed +tests because their failure modes are not limited to scanner pacing. + +## Safety + +Run the workload only in a disposable test environment. The commands below can +create many buckets and objects and can overwrite runtime scanner settings. + +Record the current scanner and heal configuration before changing anything: + +```bash +mkdir -p artifacts +mc admin config get ALIAS scanner > artifacts/scanner-config.before.txt +mc admin config get ALIAS heal > artifacts/heal-config.before.txt +``` + +Replace `ALIAS`, endpoint, and credentials with values for the test +deployment. Do not paste production credentials into saved artifacts. + +## Required Tools + +- `mc` or a compatible admin client for config changes. +- `awscurl` or another SigV4-capable HTTP client for `/v3/scanner/status`. +- `jq` for status extraction. +- `pidstat`, `mpstat`, `iostat`, `top`, or equivalent host telemetry. +- `warp`, `s3bench`, or the repository object benchmark scripts for workload + generation. + +## Test Matrix + +At minimum, collect two runs on the same RustFS commit and the same workload: + +| Run | Purpose | Example scanner settings | +|---|---|---| +| Baseline | Observe current behavior without additional pacing changes. | Existing config. | +| Pacing override | Measure whether cooperative scanner sleeps reduce pressure. | `scanner.delay="30"` and `scanner.max_wait="15"`. | + +Add bounded-cycle runs when a single scan cycle is too long: + +| Run | Purpose | Example scanner settings | +|---|---|---| +| Duration budget | Bound wall-clock time per cycle. | `scanner.cycle_max_duration="1800"`. | +| Object budget | Bound objects processed per cycle. | `scanner.cycle_max_objects="1000000"`. | +| Directory budget | Bound directories entered per cycle. | `scanner.cycle_max_directories="100000"`. | + +When comparing runs, keep hardware, RustFS commit, object count, object size, +bucket count, scanner-enabled state, and foreground workload constant. + +## Deployment Matrix + +Use the smallest deployment that reproduces the symptom, but do not treat +single-node validation as the whole scanner test surface. Scanner changes that +touch queues, admission, or cross-node maintenance should include a distributed +run when practical. + +| Deployment | What it validates | Minimum evidence | +|---|---|---| +| Single-node, single-disk | Small-object scanner pressure, pacing, cycle interval, and basic progress. | Scanner status time series plus host CPU and disk telemetry. | +| Single-node, multi-disk or erasure set | Set and disk scan concurrency, cycle budgets, checkpoint movement, usage cache persistence, and active path age. | Scanner status time series, per-disk host telemetry, and before/after data usage freshness. | +| Distributed cluster | Lifecycle transition queues, bucket replication repair admission, scanner-originated heal and bitrot admission, and queue/backlog pressure under cross-node work. | Scanner status time series from the cluster, host telemetry from each node, and subsystem-specific queued/skipped/missed counters. | + +The single-node, single-disk run is the baseline because it is cheap and +repeatable. It is not sufficient for PRs that claim to improve distributed +queue behavior, replication repair, or heal/bitrot admission. + +## Workload Shape + +For the baseline small-object scanner-pressure run, use a workload that +creates many small objects and then leaves the service mostly idle while the +scanner walks the namespace. A useful minimum shape is: + +- one RustFS node; +- one data disk; +- several buckets; +- at least 100,000 small objects total; +- scanner enabled; +- no sustained foreground workload during the observation window. + +For a distributed backlog run, use the same structure but add the relevant +subsystem condition, such as lifecycle rules, a bucket replication target, or +a configured heal/bitrot scenario. Keep the status and host telemetry cadence +the same so the result remains comparable with the baseline run. + +The repository object benchmark script can generate object traffic if `warp` +or `s3bench` is installed: + +```bash +scripts/run_object_batch_bench.sh \ + --tool warp \ + --endpoint http://127.0.0.1:9000 \ + --access-key "$RUSTFS_ACCESS_KEY" \ + --secret-key "$RUSTFS_SECRET_KEY" \ + --bucket scanner-bench \ + --auto-new-bucket \ + --concurrency 64 \ + --duration 10m \ + --sizes 1KiB,4KiB,16KiB \ + --warp-mode put \ + --out-dir artifacts/object-load +``` + +If the object generator cannot create enough objects in one run, repeat the +same command with new buckets or prefixes and record the final object count. + +## Status Collection + +Capture scanner status before the workload, after the workload finishes, and +throughout the idle observation window. + +Example status request: + +```bash +awscurl \ + --service s3 \ + --region us-east-1 \ + --access_key "$RUSTFS_ACCESS_KEY" \ + --secret_key "$RUSTFS_SECRET_KEY" \ + --request GET \ + 'http://127.0.0.1:9000/rustfs/admin/v3/scanner/status' \ + | jq . > "artifacts/scanner-status.$(date -u +%Y%m%dT%H%M%SZ).json" +``` + +For a time series, sample once per minute: + +```bash +mkdir -p artifacts/status +while sleep 60; do + ts="$(date -u +%Y%m%dT%H%M%SZ)" + awscurl \ + --service s3 \ + --region us-east-1 \ + --access_key "$RUSTFS_ACCESS_KEY" \ + --secret_key "$RUSTFS_SECRET_KEY" \ + --request GET \ + 'http://127.0.0.1:9000/rustfs/admin/v3/scanner/status' \ + | jq . > "artifacts/status/scanner-status.${ts}.json" +done +``` + +Stop the loop after the planned observation window. + +## Host Telemetry + +Collect host metrics over the same window as scanner status. + +```bash +pidstat -p "$(pidof rustfs)" 60 > artifacts/pidstat.txt +iostat -xz 60 > artifacts/iostat.txt +mpstat 60 > artifacts/mpstat.txt +``` + +If `pidstat` is not available, use `top`, `ps`, or the platform monitoring +system, but keep the sampling interval and observation window in the report. + +## Runtime Tuning Examples + +Persistent scanner config values use seconds for time fields. Use numeric +strings instead of duration suffixes: + +```bash +mc admin config set ALIAS scanner delay="30" max_wait="15" +mc admin config set ALIAS scanner cycle="3600" +mc admin config set ALIAS scanner cycle_max_duration="1800" +mc admin config set ALIAS scanner cycle_max_objects="1000000" +mc admin config set ALIAS scanner cycle_max_directories="100000" +``` + +The canonical persistent bitrot cadence belongs to the `heal` subsystem: + +```bash +mc admin config set ALIAS heal bitrot_cycle="2592000" +``` + +Environment variables take precedence over persisted config and should be +recorded separately: + +```bash +RUSTFS_SCANNER_DELAY=30 +RUSTFS_SCANNER_MAX_WAIT_SECS=15 +RUSTFS_SCANNER_CYCLE=3600 +RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS=1800 +RUSTFS_SCANNER_CYCLE_MAX_OBJECTS=1000000 +RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES=100000 +RUSTFS_SCANNER_BITROT_CYCLE_SECS=2592000 +``` + +After each config change, read scanner status and confirm the effective value +and `source` under `runtime_config`. + +## Observation Window + +Use the same window for each run: + +1. Generate or verify the object namespace. +2. Wait until foreground workload is idle. +3. Save scanner and heal config. +4. Save one scanner status snapshot. +5. Collect scanner status and host telemetry for at least 30 minutes, or for + one complete scanner cycle when that is practical. +6. Save one final scanner status snapshot. + +Longer windows are better for cycle interval comparisons. Short windows are +acceptable for quick pressure checks only if the conclusion avoids changing +defaults. + +## Fields To Compare + +Compare these fields between baseline and tuned runs: + +| Field | Why it matters | +|---|---| +| `runtime_config.*.value` and `runtime_config.*.source` | Confirms the tested settings actually took effect. | +| `metrics.pacing_pressure.primary_pressure` | Shows whether pressure is from queues, budgets, pause activity, active scans, or no scanner pressure. | +| `metrics.pacing_pressure.last_cycle_total_pause_ratio` | Shows how much of the last cycle was cooperative scanner pause time. | +| `metrics.current_cycle_objects_scanned` | Confirms object scan progress during the current cycle. | +| `metrics.current_cycle_directories_scanned` | Confirms directory walk progress during the current cycle. | +| `metrics.last_cycle_result` | Confirms whether the previous cycle completed, stopped partially, or failed. | +| `metrics.last_cycle_partial_reason` | Shows which budget stopped a partial cycle. | +| `metrics.last_cycle_partial_source` | Shows which scanner work source consumed the stopping budget. | +| `metrics.source_work` | Shows cumulative work found, queued, skipped, missed, executed, and failed by source. | +| `metrics.current_cycle_source_work` | Shows which source is consuming the current scan cycle. | +| `metrics.last_cycle_source_work` | Shows which source consumed the previous scan cycle. | +| `metrics.lifecycle_transition.scanner_missed` | Shows scanner-discovered transition work that could not be queued. | +| `metrics.lifecycle_transition.queue_full` | Shows transition queue pressure outside the scanner walk itself. | +| `metrics.scan_checkpoint` | Confirms partial cycles preserve resume context. | +| `metrics.oldest_active_path_age_seconds` | Helps identify scanner paths that may be stuck. | + +Do not use a single CPU spike as the conclusion. Compare average and p95 CPU +over the same observation window. + +## Interpreting Results + +A useful tuning result has all of these properties: + +- average or p95 scanner-related CPU and disk pressure decreases; +- `current_cycle_objects_scanned` or `current_cycle_directories_scanned` + continues to advance; +- `source_work.missed` does not grow unexpectedly for lifecycle, replication, + heal, or bitrot; +- `last_cycle_result` is either `success` or a partial result with a clear + budget reason and checkpoint; +- data usage freshness remains acceptable for the tested deployment. + +Treat these as failure signals: + +- CPU drops only because the scanner stops making progress; +- `primary_pressure` stays at `queued_scans` while queues grow; +- `last_cycle_partial_reason` repeats forever with no checkpoint movement; +- lifecycle transition `scanner_missed` or `queue_full` grows during a run that + was expected to reduce backlog; +- heal or bitrot work moves from `queued` to `missed` after a scanner pacing + change. + +## PR Evidence Checklist + +For scanner behavior PRs, include this evidence when available: + +- RustFS commit SHA and branch. +- Deployment shape: node count, disk count, disk type, CPU count, memory, and + object count. +- Workload command or script and benchmark artifact path. +- Scanner and heal config before and after tuning. +- Observation window and sample interval. +- Scanner status snapshots or time series. +- Host CPU and disk telemetry. +- Short conclusion that separates pressure reduction from scanner progress. + +For documentation-only PRs, it is enough to verify links and formatting. diff --git a/docs/operations/scanner-runtime-controls.md b/docs/operations/scanner-runtime-controls.md index 99240756b..d568ca04b 100644 --- a/docs/operations/scanner-runtime-controls.md +++ b/docs/operations/scanner-runtime-controls.md @@ -5,6 +5,9 @@ data scanner. It is written for operators who need to reduce scanner pressure, diagnose slow scan progress, or confirm that background lifecycle, replication, heal, bitrot, and usage work is still moving. +For reproducible scanner-pressure validation and before/after evidence, see +[Scanner Benchmark Runbook](scanner-benchmark-runbook.md). + ## What the scanner does The scanner is the background maintenance loop that walks stored objects and diff --git a/helm/README.md b/helm/README.md index 67dd01dc9..f47e9b896 100644 --- a/helm/README.md +++ b/helm/README.md @@ -181,7 +181,9 @@ uer. `ClusterIssuer` or `Issuer`. | Scanner values map directly to scanner environment variables. For tuning workflow and `/v3/scanner/status` interpretation, see -[Scanner Runtime Controls](../docs/operations/scanner-runtime-controls.md). +[Scanner Runtime Controls](../docs/operations/scanner-runtime-controls.md). For +repeatable scanner-pressure validation, see +[Scanner Benchmark Runbook](../docs/operations/scanner-benchmark-runbook.md). ---