rustfs/docs/operations/scanner-benchmark-runbook.md

Scanner Benchmark Runbook

Use this when: you need reproducible before/after evidence that a scanner pacing or cycle change reduces background pressure without stalling lifecycle, replication, heal, or bitrot progress, or you are assembling evidence for a scanner-behavior PR.

Source of truth: scripts/run_scanner_validation_harness.sh (collection, scanner-summary.csv columns), scripts/run_object_batch_bench.sh (workload), and Scanner Runtime Controls for the meaning of every status field and configuration key.

Scope

This runbook verifies that scanner pacing and cycle controls reduce background pressure while preserving maintenance progress. It covers mostly idle single-node deployments with many small objects, multi-disk or erasure-set nodes, distributed clusters where scanner pressure mixes with lifecycle, replication, heal, or bitrot queues, and backlog investigations for any of those subsystems.

It does not prove full MinIO parity, site-replication correctness, or replaced-disk heal correctness. Those flows need dedicated distributed tests because their failure modes are not limited to scanner pacing.

Safety

Run the workload only in a disposable test environment. The commands below can create many buckets and objects and overwrite runtime scanner settings. Record the current scanner and heal configuration before changing anything:

mkdir -p artifacts
mc admin config get ALIAS scanner > artifacts/scanner-config.before.txt
mc admin config get ALIAS heal > artifacts/heal-config.before.txt

The scanner and heal subsystems are served by GetConfigKVHandler (rustfs/src/admin/handlers/config_admin.rs, route /v3/get-config-kv); this was confirmed by code inspection, not by running mc against a live deployment. Replace ALIAS, endpoint, and credentials with values for the test deployment. Do not paste production credentials into saved artifacts.

Required Tools

Tool	Purpose
mc or a compatible admin client	Config snapshots and changes.
awscurl or another SigV4-capable HTTP client	/v3/scanner/status and admin metrics.
jq	Status extraction.
pidstat, mpstat, iostat, top, or equivalent	Host telemetry.
warp, s3bench, or scripts/run_object_batch_bench.sh	Workload generation.

Test Matrix

Formal Scanner/Heal ABBA

The --abba mode runs five independent scenario cells: cold-hot, fresh-hot, multi-hot-new, running-heal, and mrf-replay. Each scenario runs at least three A1/B1/B2/A2 groups for both baseline/candidate with background work on, and candidate-only background off/on. A measured leg lasts at least 900 seconds; the minimum matrix contains 120 legs (30 hours before setup/oracles). The existing performance-ab.yml supplies the pattern for immutable build provenance and failure propagation, but its short Warp workload is not this scanner gate. No scheduled workflow starts this matrix automatically.

scripts/run_scanner_validation_harness.sh --abba \
  --manifest scanner-abba.json --adapter /path/to/isolated-deployment-adapter \
  --out-dir /path/to/new-artifacts --data-root /path/to/new-test-data

Both roots must be new and non-overlapping. Every leg receives a unique data directory. The runner checks disk capacity before each leg, never removes data, and stops the adapter after success or failure. Retain raw artifacts and inspect task ownership before removing any test data. The operator must reserve the target machines and map the assigned directory to separate data paths on every node; the runner cannot prove remote isolation from local path names.

The manifest has the following JSON contract (all fields are required):

Field	Value
schema, evidence	1, and measured or synthetic.
rounds, duration_seconds, min_free_bytes	3..10 groups, 900..86400 seconds for measured runs, and the independently estimated free-space reservation in bytes. Synthetic runs may use 1 second.
baseline, candidate	Each contains executable binary, full 40-character revision, and verified sha256. The runner rehashes binaries before every leg.
fixed	config_sha256, dataset_sha256, release_flags, durability, disk_type, cache_state, load_command, resource_isolation, topology (EC8+4), and positive offered_load_ops. Hashes use 64 lowercase hexadecimal characters.
oracles	A map with all five scenario names. Each value contains positive integer objects, versions, bytes, and sha256 of the independently prepared canonical object/version/content manifest.
expected_healed_objects	A map with all five scenario names and independently seeded repair counts. Running-heal and MRF-replay require a positive count.

Record exact build flags and effective durability settings, not just defaults. Use deterministic workload seeds so every isolated leg has the same expected object/version/content result. Fix the foreground arrival rate (offered load), cache preparation procedure, configuration, and hardware across every leg. Do not include credentials in the manifest, adapter output, or saved commands; the collector reads RUSTFS_ACCESS_KEY and RUSTFS_SECRET_KEY from its environment.

Deployment Adapter Contract

The runner invokes an executable as adapter ACTION request.json response.json with no shell evaluation. Actions are separate processes: prepare, measure, oracle, and stop. Every action must return zero and write a JSON object of at most 1 MiB. Logs are kept separately and require an operator-managed disk quota. Missing output, timeout, nonzero exit, unknown/missing metrics, zero samples, and request errors fail the run. Adapters must terminate their own children on failure and stop must be idempotent even after partial preparation.

The request contains the fixed manifest fields, selected build, scenario, round, leg, comparison (build or background), background mode (on or off), duration, unique data_dir, expected object oracle, and expected repair count. Adapter responsibilities:

1. prepare deploys the selected binary into an authorized isolated topology, checks actual binary/config/durability, initializes deterministic scenario data and the requested cache state, and returns {"ready": true}. For measured runs it also returns collector with exactly alias, endpoint, and comma-separated metrics_endpoints; the runner starts the existing scanner collector at 60-second cadence while measure runs.
2. measure maintains the fixed offered load for the entire requested duration. cold-hot retains cold buckets while mutating a hot bucket; fresh-hot creates a bucket after scanner startup; multi-hot-new combines several hot buckets with a newly created bucket; running-heal applies foreground load during active repair; mrf-replay replays independently seeded durable repair work. Capture same-window status for bucket-freshness issue #7108. Actual fault injection and dataset generation belong to the reviewed adapter.
3. oracle independently enumerates all objects and versions, reads and checks their complete bytes, and verifies repairs. Return complete: true, integer errors: 0, and actual matching the manifest's expected oracle. Never copy expected values into a measured oracle or infer completion from empty queues.
4. stop stops task-owned workload/server processes and returns stopped: true. Preserve data and artifacts for diagnosis. An adapter may restore previous settings but must not delete arbitrary paths or stop unrelated deployments.

The measure response echoes the observed evidence, fixed, build, data_dir, and background, plus sample_count (1..3600), elapsed_seconds, and metrics. All metrics must be finite nonnegative numbers: p99_ms, throughput_ops, rss_bytes, cpu_seconds, iops, rpc_count, cache_clone_bytes, encode_bytes, save_bytes, oldest_age_seconds, walk_objects, cold_walk_objects, healed_objects, errors, and requests. Requests, throughput, and p99 must be positive; errors must be zero. Repair counts must match the manifest when background work is on. Keep underlying request samples, counter reset checks, profiler captures, and per-node telemetry in the cell artifact directory; aggregate values alone do not establish their measurement provenance. Missing production instrumentation is a pending gate, not permission to report a fabricated zero.

For P2, measure.convergence contains booleans writes_stopped, last_mutation_observed, first_complete_publication; numeric last_mutation_time, last_mutation_observed_time, writes_stopped_time, window_start, window_end, budget_available_seconds, walk_objects, and full_walk_objects. Times use one monotonic clock. The window starts after writes stop and the final mutation is observed, and ends at the first complete publication. The reference is an independent full walk of the same static namespace. Record available budget seconds to interpret elapsed time. During continuing writes, omit this proof and report useful-work ratio and justified invalidation/re-scan work separately; the runner reports P2 pending and does not impose a fixed cumulative walk bound.

The nightly heal workflow clones rustfs/auto-testing separately and invokes auto-testing/rustfs_heal_test.sh; that script is not a local scripts/test entry point. If an adapter uses it, record and verify the external checkout's owner and full commit before use. The current workflow clones the default branch, so its contents must not be attributed to a RustFS source SHA.

Evidence Gates

report.json records each group's verdict and the raw responses remain in their cell directories. Candidate/build p99 regression must be at most 5% and throughput loss at most 3%; candidate background on/off limits are 10% and 5%. P1 requires cold-hot walk reduction of at least the baseline cold-walk share times 80%, rather than a fixed 80% reduction for every workload. P2 requires candidate post-stop work at most 1.2 times the independent full-walk reference. Missing candidate convergence proof yields inconclusive. A2/A1 or B2/B1 p99 or throughput drift above 5% also yields inconclusive, with exit code 3. Correctness errors and non-noisy performance regressions exit 1. Every group must pass; a favorable median cannot hide a failing group.

Synthetic success is explicitly synthetic_validated, with performance: pending. It validates orchestration and gate logic only. It proves no runtime, distributed, crash, mixed-version, or performance behavior and cannot close the performance acceptance gate. Run the fake-adapter self-tests with:

scripts/test_scanner_validation_harness.sh

They cover the complete 120-cell schedule, data isolation, missing builds and oracles, zero samples/requests, swallowed request errors, offered-load drift, incomplete repairs, missing metrics, noise, and P1/P2/p99 regressions. A real deployment adapter and actual ABBA artifacts remain required before any measured performance or release claim.

Collect at least two runs on the same RustFS commit and the same workload. Keep hardware, commit, object count, object size, bucket count, scanner-enabled state, and foreground workload constant between runs.

Run	Purpose	Example scanner settings
Baseline	Observe current behavior without additional pacing changes.	Existing config.
Pacing override	Measure whether cooperative scanner sleeps reduce pressure.	scanner.delay="30" and scanner.max_wait="15".
Duration budget (when one cycle is too long)	Bound wall-clock time per cycle.	scanner.cycle_max_duration="1800".
Object budget	Bound objects processed per cycle.	scanner.cycle_max_objects="1000000".
Directory budget	Bound directories entered per cycle.	scanner.cycle_max_directories="100000".

Deployment Matrix

Use the smallest deployment that reproduces the symptom. The single-node, single-disk run is the cheap, repeatable baseline; it is not sufficient for PRs that claim to improve distributed queue behavior, replication repair, or heal/bitrot admission.

Deployment	What it validates	Minimum evidence	Workload shape
Single-node, single-disk	Small-object scanner pressure, pacing, cycle interval, basic progress.	Scanner status time series plus host CPU and disk telemetry.	One node, one data disk, several buckets, at least 100,000 small objects, scanner enabled, no sustained foreground workload during observation.
Single-node, multi-disk or erasure set	Set and disk scan concurrency, cycle budgets, checkpoint movement, usage cache persistence, active path age.	Scanner status time series, per-disk host telemetry, before/after data usage freshness.	Same as above across all disks.
Distributed cluster	Lifecycle transition queues, bucket replication repair admission, scanner-originated heal and bitrot admission, queue/backlog pressure under cross-node work.	Scanner status time series from the cluster, host telemetry from each node, subsystem-specific queued/skipped/missed counters.	Same structure plus the relevant subsystem condition (lifecycle rules, a replication target, a heal/bitrot scenario); keep status and telemetry cadence identical to the baseline.

Generate object traffic with the repository script if warp or s3bench is installed; repeat with new buckets or prefixes if one run cannot create enough objects, and record the final object count:

scripts/run_object_batch_bench.sh \
  --tool warp \
  --endpoint http://127.0.0.1:9000 \
  --access-key "$RUSTFS_ACCESS_KEY" \
  --secret-key "$RUSTFS_SECRET_KEY" \
  --bucket scanner-bench \
  --auto-new-bucket \
  --concurrency 64 \
  --duration 10m \
  --sizes 1KiB,4KiB,16KiB \
  --warp-mode put \
  --out-dir artifacts/object-load

Status Collection

Capture scanner status before the workload, after the workload finishes, and throughout the idle observation window. The validation harness does this repeatably and writes scanner/heal config snapshots, scanner status samples, background heal status samples, host telemetry when available, run metadata, scanner-summary.csv, and scanner-validation-report.md:

export RUSTFS_ACCESS_KEY="<admin-access-key>"
export RUSTFS_SECRET_KEY="<admin-secret-key>"

scripts/run_scanner_validation_harness.sh \
  --alias ALIAS \
  --endpoint http://127.0.0.1:9000 \
  --deployment single-disk \
  --workload-label small-object-idle \
  --samples 30 \
  --interval-secs 60 \
  --out-dir artifacts/scanner-validation

For per-node distributed evidence pass --metrics-endpoints (comma-separated). Each sample then stores /v3/scanner/status, one /v3/background-heal/status response per listed endpoint, and one by-host admin metrics response per listed endpoint; without it, background-heal status is captured only from --endpoint:

scripts/run_scanner_validation_harness.sh \
  --alias ALIAS \
  --endpoint http://node-a:9000 \
  --deployment distributed \
  --workload-label lifecycle-replication-heal-backlog \
  --metrics-endpoints http://node-a:9000,http://node-b:9000,http://node-c:9000,http://node-d:9000 \
  --samples 30 \
  --interval-secs 60 \
  --out-dir artifacts/scanner-validation-distributed

For ad hoc per-node snapshots outside the harness window, use the by-host awscurl loop in Reading Distributed Metrics; the metrics endpoint reports only the node that handles the request.

Bucket metrics freshness validation

Use the harness around a post-start bucket creation workload to cover the timing where scanner startup sees no buckets, a bucket is created afterwards, and the first metrics collection must not confuse a cold usage cache with real zero usage:

1. Start RustFS from an empty data path.
2. Start the harness before creating buckets.
3. Create a bucket, upload objects, and keep the harness running until at least one usage save is observed.
4. Compare scanner-summary.csv with /rustfs/admin/v3/metrics?types=1&n=1 bucket metrics.

Expected evidence: dirty usage is marked, life_time_scan_cycle or life_time_scan_bucket_drive advances, life_time_scan_object advances for object workloads, and life_time_save_usage plus usage_last_save_result=success appear before non-zero bucket usage metrics are accepted as fresh.

Manual status sampling

Single snapshot:

awscurl \
  --service s3 \
  --region us-east-1 \
  --access_key "$RUSTFS_ACCESS_KEY" \
  --secret_key "$RUSTFS_SECRET_KEY" \
  --request GET \
  'http://127.0.0.1:9000/rustfs/admin/v3/scanner/status' \
  | jq . > "artifacts/scanner-status.$(date -u +%Y%m%dT%H%M%SZ).json"

Time series (stop after the planned observation window):

mkdir -p artifacts/status
while sleep 60; do
  ts="$(date -u +%Y%m%dT%H%M%SZ)"
  awscurl \
    --service s3 \
    --region us-east-1 \
    --access_key "$RUSTFS_ACCESS_KEY" \
    --secret_key "$RUSTFS_SECRET_KEY" \
    --request GET \
    'http://127.0.0.1:9000/rustfs/admin/v3/scanner/status' \
    | jq . > "artifacts/status/scanner-status.${ts}.json"
done

Host Telemetry

Collect host metrics over the same window as scanner status. If pidstat is unavailable, use top, ps, or the platform monitoring system, but record the sampling interval and window in the report.

pidstat -p "$(pidof rustfs)" 60 > artifacts/pidstat.txt
iostat -xz 60 > artifacts/iostat.txt
mpstat 60 > artifacts/mpstat.txt

Runtime Tuning Examples

Persistent scanner config values use seconds for time fields; use numeric strings, not duration suffixes. The canonical persistent bitrot cadence belongs to the heal subsystem.

mc admin config set ALIAS scanner delay="30" max_wait="15"
mc admin config set ALIAS scanner cycle="3600"
mc admin config set ALIAS scanner cycle_max_duration="1800"
mc admin config set ALIAS scanner cycle_max_objects="1000000"
mc admin config set ALIAS scanner cycle_max_directories="100000"
mc admin config set ALIAS heal bitrot_cycle="2592000"

Environment variables take precedence over persisted config and should be recorded separately:

RUSTFS_SCANNER_DELAY=30
RUSTFS_SCANNER_MAX_WAIT_SECS=15
RUSTFS_SCANNER_CYCLE=3600
RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS=1800
RUSTFS_SCANNER_CYCLE_MAX_OBJECTS=1000000
RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES=100000
RUSTFS_SCANNER_BITROT_CYCLE_SECS=2592000

After each config change, read scanner status and confirm the effective value and source under runtime_config.

Observation Window

Use the same window for each run:

1. Generate or verify the object namespace.
2. Wait until foreground workload is idle.
3. Save scanner and heal config.
4. Save one scanner status snapshot.
5. Collect scanner status and host telemetry for at least 30 minutes, or for one complete scanner cycle when practical.
6. Save one final scanner status snapshot.

Longer windows are better for cycle interval comparisons. Short windows are acceptable for quick pressure checks only if the conclusion avoids changing defaults.

Fields To Compare

Field semantics are defined in Scanner Runtime Controls; the decision fields for a before/after comparison are:

Field	Decision it supports
runtime_config.*.value and runtime_config.*.source	The tested settings actually took effect.
metrics.pacing_pressure.primary_pressure, last_cycle_total_pause_ratio	Where pressure comes from and how much of the cycle was cooperative pause.
metrics.maintenance_control.primary_control, metrics.maintenance_control.sources	Whether a maintenance source is blocked, deferred, active, or only pacing-limited.
metrics.current_cycle_objects_scanned, metrics.current_cycle_directories_scanned	Scan progress continues.
metrics.last_cycle_result, last_cycle_partial_reason, last_cycle_partial_source	Whether the previous cycle completed, which budget stopped it, and which source consumed it.
metrics.source_work, metrics.current_cycle_source_work, metrics.last_cycle_source_work	missed growth per source is a downstream admission problem, not pacing.
metrics.replication_repair (and current/last-cycle variants)	Repair kind, scanner_role, and execution_owner for replication backlog runs.
metrics.lifecycle_expiry.{current_queued,current_active,queue_missed,scanner_missed}	Expiry backlog and admission failures.
metrics.lifecycle_transition.{scanner_missed,queue_full,compensation_pending,failed}	Transition backlog, queue pressure, and worker failures.
metrics.usage_freshness.*, metrics.current_cycle_usage_saves, metrics.last_cycle_usage_saves	Bucket metrics freshness; last_usage_save_result must be success.
metrics.life_time_ops.{scan_cycle,scan_bucket_drive,scan_object,save_usage}	Cycles, bucket-drive scans, object scans, and DataUsageInfo saves actually happened after the workload.
metrics.scan_checkpoint, metrics.oldest_active_path_age_seconds	Partial cycles preserve resume context; stuck paths.

Do not use a single CPU spike as the conclusion; compare average and p95 CPU over the same observation window.

For heal or bitrot pressure investigations, also capture /v3/background-heal/status from every distributed endpoint and compare healOperations.queueLength, activeTasks, queuedBySource, activeBySource, queuedByPriority, and activeByPriority (see Reading Heal Operations).

scanner-summary.csv columns

In distributed runs the heal columns are aggregated from the background-heal snapshots captured across --metrics-endpoints.

Column	Meaning
heal_queue_length	Total queued heal requests at the same timestamp as the scanner status sample.
heal_active_tasks	Total running heal tasks.
heal_scanner_queued	Scanner-submitted heal or bitrot work waiting in the queue.
heal_admin_queued	Manual/admin heal work waiting in the queue.
heal_auto_heal_queued	Auto-heal work waiting in the queue, typically from disk/set recovery paths.
current_cycle_usage_saves	Usage saves during the current cycle.
last_cycle_usage_saves	Usage saves from the last finished or partial cycle.
usage_dirty_pending_buckets	Dirty buckets still waiting for scanner refresh.
usage_last_cycle_dirty_buckets	Dirty buckets selected by the last cycle.
usage_last_cycle_cleared_dirty_buckets	Dirty bucket marks cleared by the last cycle.
usage_last_save_result	Last DataUsageInfo save result.
usage_last_save_unix_secs	Last DataUsageInfo save timestamp.
life_time_scan_cycle	Total scanner cycles observed by the node.
life_time_scan_bucket_drive	Total bucket-drive scans completed by the node.
life_time_scan_object	Total object scan operations observed by the node.
life_time_save_usage	Total usage save operations observed by the node.

Interpreting Results

A useful tuning result has all of these properties:

• average or p95 scanner-related CPU and disk pressure decreases;
• current_cycle_objects_scanned or current_cycle_directories_scanned continues to advance;
• source_work.missed does not grow unexpectedly for lifecycle, replication, heal, or bitrot;
• last_cycle_result is either success or a partial result with a clear budget reason and checkpoint;
• data usage freshness remains acceptable for the tested deployment.

Treat these as failure signals:

Signal	Reading
CPU drops only because the scanner stops making progress	Not a tuning win.
primary_pressure stays at queued_scans while queues grow	Concurrency, not pacing, is the constraint.
last_cycle_partial_reason repeats forever with no checkpoint movement	Budget too small or checkpoint not advancing.
Lifecycle expiry queue_missed, scanner_missed, current_queued, or current_active grows during a run meant to reduce expiry backlog	Downstream expiry pressure.
Lifecycle transition scanner_missed, queue_full, compensation_pending, or failed grows during a run meant to reduce backlog	Downstream transition pressure.
Bucket metrics show zero usage after post-start uploads while dirty usage remains pending and life_time_save_usage does not advance	Usage freshness regression.
bucket_replication missed work with scanner_role=repair_admission grows while replication worker queues or target failures also grow	Downstream replication pressure, not only scanner pacing.
site_replication active_resync grows and is read as scanner-owned repair execution	Misreading: scanner_role=boundary_signal and execution_owner=site_replication_runtime mean active resync remains owned by the site replication runtime.
Heal or bitrot work moves from queued to missed after a scanner pacing change	Heal admission regression.

PR Evidence Checklist

For scanner behavior PRs, include when available:

• RustFS commit SHA and branch.
• Deployment shape: node count, disk count, disk type, CPU count, memory, object count.
• Workload command or script and benchmark artifact path.
• Scanner and heal config before and after tuning.
• Observation window and sample interval.
• Scanner status snapshots or time series.
• Host CPU and disk telemetry.
• Usage freshness fields from scanner-summary.csv when validating bucket metrics timing.
• A short conclusion that separates pressure reduction from scanner progress.
• scanner-validation-report.md from the harness when using the scripted collection path.