gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-09-06 03:59:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(knowledge-base): prune stale content and add agent-facing index (#7035)

Browse Source
This commit is contained in:
Zhengchao An
2026-09-02 08:26:59 +08:00
committed by GitHub
parent ceeff52229
commit 0a975f2fe2
99 changed files with 3312 additions and 10590 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/operations/scanner-benchmark-runbook.md
+79 -272
View File
@@ -1,42 +1,18 @@
# Scanner Benchmark Runbook
This runbook describes how to collect reproducible evidence for scanner
pressure, scanner progress, and scanner runtime tuning. It is intended for
maintainers and operators validating scanner changes in an isolated test
deployment.
**Use this when:** you need reproducible before/after evidence that a scanner pacing or cycle change reduces background pressure without stalling lifecycle, replication, heal, or bitrot progress, or you are assembling evidence for a scanner-behavior PR.
Use this runbook together with
[Scanner Runtime Controls](scanner-runtime-controls.md). The runtime controls
document explains each field and configuration key; this document explains how
to run a comparable before/after observation and decide whether the scanner is
healthier after tuning.
**Source of truth:** `scripts/run_scanner_validation_harness.sh` (collection, `scanner-summary.csv` columns), `scripts/run_object_batch_bench.sh` (workload), and [Scanner Runtime Controls](scanner-runtime-controls.md) for the meaning of every status field and configuration key.
## Scope
This runbook verifies that scanner pacing and cycle controls reduce background
pressure while preserving maintenance progress. It is useful for:
This runbook verifies that scanner pacing and cycle controls reduce background pressure while preserving maintenance progress. It covers mostly idle single-node deployments with many small objects, multi-disk or erasure-set nodes, distributed clusters where scanner pressure mixes with lifecycle, replication, heal, or bitrot queues, and backlog investigations for any of those subsystems.
- mostly idle single-node, single-disk deployments with many small objects;
- single-node multi-disk or erasure-set deployments where scanner work is
spread across disks and sets;
- distributed clusters where scanner pressure is mixed with lifecycle,
replication, heal, or bitrot queues;
- lifecycle expiry or transition backlog investigations;
- bucket replication repair backlog investigations;
- scanner-originated heal or bitrot admission investigations;
- pull request evidence when scanner behavior, scanner status, or scanner
controls change.
This runbook does not prove full MinIO parity, site-replication correctness,
or replaced-disk heal correctness. Those flows need dedicated distributed
tests because their failure modes are not limited to scanner pacing.
It does not prove full MinIO parity, site-replication correctness, or replaced-disk heal correctness. Those flows need dedicated distributed tests because their failure modes are not limited to scanner pacing.
## Safety
Run the workload only in a disposable test environment. The commands below can
create many buckets and objects and can overwrite runtime scanner settings.
Record the current scanner and heal configuration before changing anything:
Run the workload only in a disposable test environment. The commands below can create many buckets and objects and overwrite runtime scanner settings. Record the current scanner and heal configuration before changing anything:
```bash
mkdir -p artifacts
@@ -44,75 +20,41 @@ mc admin config get ALIAS scanner > artifacts/scanner-config.before.txt
mc admin config get ALIAS heal > artifacts/heal-config.before.txt
```
Replace `ALIAS`, endpoint, and credentials with values for the test
deployment. Do not paste production credentials into saved artifacts.
The `scanner` and `heal` subsystems are served by `GetConfigKVHandler` (`rustfs/src/admin/handlers/config_admin.rs`, route `/v3/get-config-kv`); this was confirmed by code inspection, not by running `mc` against a live deployment. Replace `ALIAS`, endpoint, and credentials with values for the test deployment. Do not paste production credentials into saved artifacts.
## Required Tools
- `mc` or a compatible admin client for config changes.
- `awscurl` or another SigV4-capable HTTP client for `/v3/scanner/status`.
- `jq` for status extraction.
- `pidstat`, `mpstat`, `iostat`, `top`, or equivalent host telemetry.
- `warp`, `s3bench`, or the repository object benchmark scripts for workload
generation.
| Tool | Purpose |
|---|---|
| `mc` or a compatible admin client | Config snapshots and changes. |
| `awscurl` or another SigV4-capable HTTP client | `/v3/scanner/status` and admin metrics. |
| `jq` | Status extraction. |
| `pidstat`, `mpstat`, `iostat`, `top`, or equivalent | Host telemetry. |
| `warp`, `s3bench`, or `scripts/run_object_batch_bench.sh` | Workload generation. |
## Test Matrix
At minimum, collect two runs on the same RustFS commit and the same workload:
Collect at least two runs on the same RustFS commit and the same workload. Keep hardware, commit, object count, object size, bucket count, scanner-enabled state, and foreground workload constant between runs.
| Run | Purpose | Example scanner settings |
|---|---|---|
| Baseline | Observe current behavior without additional pacing changes. | Existing config. |
| Pacing override | Measure whether cooperative scanner sleeps reduce pressure. | `scanner.delay="30"` and `scanner.max_wait="15"`. |
Add bounded-cycle runs when a single scan cycle is too long:
| Run | Purpose | Example scanner settings |
|---|---|---|
| Duration budget | Bound wall-clock time per cycle. | `scanner.cycle_max_duration="1800"`. |
| Duration budget (when one cycle is too long) | Bound wall-clock time per cycle. | `scanner.cycle_max_duration="1800"`. |
| Object budget | Bound objects processed per cycle. | `scanner.cycle_max_objects="1000000"`. |
| Directory budget | Bound directories entered per cycle. | `scanner.cycle_max_directories="100000"`. |
When comparing runs, keep hardware, RustFS commit, object count, object size,
bucket count, scanner-enabled state, and foreground workload constant.
## Deployment Matrix
Use the smallest deployment that reproduces the symptom, but do not treat
single-node validation as the whole scanner test surface. Scanner changes that
touch queues, admission, or cross-node maintenance should include a distributed
run when practical.
Use the smallest deployment that reproduces the symptom. The single-node, single-disk run is the cheap, repeatable baseline; it is not sufficient for PRs that claim to improve distributed queue behavior, replication repair, or heal/bitrot admission.
| Deployment | What it validates | Minimum evidence |
|---|---|---|
| Single-node, single-disk | Small-object scanner pressure, pacing, cycle interval, and basic progress. | Scanner status time series plus host CPU and disk telemetry. |
| Single-node, multi-disk or erasure set | Set and disk scan concurrency, cycle budgets, checkpoint movement, usage cache persistence, and active path age. | Scanner status time series, per-disk host telemetry, and before/after data usage freshness. |
| Distributed cluster | Lifecycle transition queues, bucket replication repair admission, scanner-originated heal and bitrot admission, and queue/backlog pressure under cross-node work. | Scanner status time series from the cluster, host telemetry from each node, and subsystem-specific queued/skipped/missed counters. |
| Deployment | What it validates | Minimum evidence | Workload shape |
|---|---|---|---|
| Single-node, single-disk | Small-object scanner pressure, pacing, cycle interval, basic progress. | Scanner status time series plus host CPU and disk telemetry. | One node, one data disk, several buckets, at least 100,000 small objects, scanner enabled, no sustained foreground workload during observation. |
| Single-node, multi-disk or erasure set | Set and disk scan concurrency, cycle budgets, checkpoint movement, usage cache persistence, active path age. | Scanner status time series, per-disk host telemetry, before/after data usage freshness. | Same as above across all disks. |
| Distributed cluster | Lifecycle transition queues, bucket replication repair admission, scanner-originated heal and bitrot admission, queue/backlog pressure under cross-node work. | Scanner status time series from the cluster, host telemetry from each node, subsystem-specific queued/skipped/missed counters. | Same structure plus the relevant subsystem condition (lifecycle rules, a replication target, a heal/bitrot scenario); keep status and telemetry cadence identical to the baseline. |
The single-node, single-disk run is the baseline because it is cheap and
repeatable. It is not sufficient for PRs that claim to improve distributed
queue behavior, replication repair, or heal/bitrot admission.
## Workload Shape
For the baseline small-object scanner-pressure run, use a workload that
creates many small objects and then leaves the service mostly idle while the
scanner walks the namespace. A useful minimum shape is:
- one RustFS node;
- one data disk;
- several buckets;
- at least 100,000 small objects total;
- scanner enabled;
- no sustained foreground workload during the observation window.
For a distributed backlog run, use the same structure but add the relevant
subsystem condition, such as lifecycle rules, a bucket replication target, or
a configured heal/bitrot scenario. Keep the status and host telemetry cadence
the same so the result remains comparable with the baseline run.
The repository object benchmark script can generate object traffic if `warp`
or `s3bench` is installed:
Generate object traffic with the repository script if `warp` or `s3bench` is installed; repeat with new buckets or prefixes if one run cannot create enough objects, and record the final object count:
```bash
scripts/run_object_batch_bench.sh \
@@ -129,15 +71,9 @@ scripts/run_object_batch_bench.sh \
--out-dir artifacts/object-load
```
If the object generator cannot create enough objects in one run, repeat the
same command with new buckets or prefixes and record the final object count.
## Status Collection
Capture scanner status before the workload, after the workload finishes, and
throughout the idle observation window.
The repository includes a scanner validation harness for repeatable collection:
Capture scanner status before the workload, after the workload finishes, and throughout the idle observation window. The validation harness does this repeatably and writes scanner/heal config snapshots, scanner status samples, background heal status samples, host telemetry when available, run metadata, `scanner-summary.csv`, and `scanner-validation-report.md`:
```bash
export RUSTFS_ACCESS_KEY="<admin-access-key>"
@@ -153,12 +89,7 @@ scripts/run_scanner_validation_harness.sh \
--out-dir artifacts/scanner-validation
```
The harness writes scanner/heal config snapshots, scanner status samples,
background heal status samples, host telemetry when available, run metadata,
`scanner-summary.csv`, and `scanner-validation-report.md`.
Use `--metrics-endpoints` when the validation needs per-node distributed
evidence. The value is a comma-separated list of RustFS endpoints:
For per-node distributed evidence pass `--metrics-endpoints` (comma-separated). Each sample then stores `/v3/scanner/status`, one `/v3/background-heal/status` response per listed endpoint, and one by-host admin metrics response per listed endpoint; without it, background-heal status is captured only from `--endpoint`:
```bash
scripts/run_scanner_validation_harness.sh \
@@ -172,55 +103,22 @@ scripts/run_scanner_validation_harness.sh \
--out-dir artifacts/scanner-validation-distributed
```
Each sample stores `/v3/scanner/status`, one `/v3/background-heal/status`
response per endpoint listed in `--metrics-endpoints`, and one by-host admin
metrics response per listed endpoint. When `--metrics-endpoints` is omitted,
the harness captures background-heal status only from `--endpoint`.
For ad hoc per-node snapshots outside the harness window, use the by-host `awscurl` loop in [Reading Distributed Metrics](scanner-runtime-controls.md#reading-distributed-metrics); the metrics endpoint reports only the node that handles the request.
For bucket metrics freshness validation, use the same harness around a
post-start bucket creation workload:
### Bucket metrics freshness validation
Use the harness around a post-start bucket creation workload to cover the timing where scanner startup sees no buckets, a bucket is created afterwards, and the first metrics collection must not confuse a cold usage cache with real zero usage:
1. Start RustFS from an empty data path.
2. Start the harness before creating buckets.
3. Create a bucket, upload objects, and keep the harness running until at
least one usage save is observed.
4. Compare `scanner-summary.csv` with
`/rustfs/admin/v3/metrics?types=1&n=1` bucket metrics.
3. Create a bucket, upload objects, and keep the harness running until at least one usage save is observed.
4. Compare `scanner-summary.csv` with `/rustfs/admin/v3/metrics?types=1&n=1` bucket metrics.
This covers the issue 3496 timing where scanner startup sees no buckets, the
bucket is created after startup, and the first metrics collection must not
confuse cold usage cache with real zero usage. The expected evidence is that
dirty usage is marked, `life_time_scan_cycle` or
`life_time_scan_bucket_drive` advances, `life_time_scan_object` advances for
object workloads, and `life_time_save_usage` plus
`usage_last_save_result=success` appear before accepting non-zero bucket usage
metrics as fresh.
Expected evidence: dirty usage is marked, `life_time_scan_cycle` or `life_time_scan_bucket_drive` advances, `life_time_scan_object` advances for object workloads, and `life_time_save_usage` plus `usage_last_save_result=success` appear before non-zero bucket usage metrics are accepted as fresh.
For distributed runs, capture scanner admin metrics from every node with
`by-host=true`. The metrics endpoint reports the node that handles the request;
`by-host=true` preserves that node's host view but does not collect peer nodes.
These per-node artifacts include active path age, checkpoint state, pacing
pressure, source work, and queued/skipped/missed downstream admission counters.
The validation harness can collect these artifacts automatically with
`--metrics-endpoints`; the manual loop below is useful when adding extra nodes
or collecting ad hoc snapshots outside the harness window.
### Manual status sampling
```bash
for endpoint in http://node-a:9000 http://node-b:9000 http://node-c:9000; do
node="${endpoint#http://}"
node="${node%%:*}"
awscurl \
--service s3 \
--region us-east-1 \
--access_key "$RUSTFS_ACCESS_KEY" \
--secret_key "$RUSTFS_SECRET_KEY" \
--request GET \
"${endpoint}/rustfs/admin/v3/metrics?types=1&by-host=true&n=1" \
> "artifacts/scanner-metrics.${node}.$(date -u +%Y%m%dT%H%M%SZ).ndjson"
done
```
Example status request:
Single snapshot:
```bash
awscurl \
@@ -233,7 +131,7 @@ awscurl \
| jq . > "artifacts/scanner-status.$(date -u +%Y%m%dT%H%M%SZ).json"
```
For a time series, sample once per minute:
Time series (stop after the planned observation window):
```bash
mkdir -p artifacts/status
@@ -250,11 +148,9 @@ while sleep 60; do
done
```
Stop the loop after the planned observation window.
## Host Telemetry
Collect host metrics over the same window as scanner status.
Collect host metrics over the same window as scanner status. If `pidstat` is unavailable, use `top`, `ps`, or the platform monitoring system, but record the sampling interval and window in the report.
```bash
pidstat -p "$(pidof rustfs)" 60 > artifacts/pidstat.txt
@@ -262,13 +158,9 @@ iostat -xz 60 > artifacts/iostat.txt
mpstat 60 > artifacts/mpstat.txt
```
If `pidstat` is not available, use `top`, `ps`, or the platform monitoring
system, but keep the sampling interval and observation window in the report.
## Runtime Tuning Examples
Persistent scanner config values use seconds for time fields. Use numeric
strings instead of duration suffixes:
Persistent scanner config values use seconds for time fields; use numeric strings, not duration suffixes. The canonical persistent bitrot cadence belongs to the `heal` subsystem.
```bash
mc admin config set ALIAS scanner delay="30" max_wait="15"
@@ -276,16 +168,10 @@ mc admin config set ALIAS scanner cycle="3600"
mc admin config set ALIAS scanner cycle_max_duration="1800"
mc admin config set ALIAS scanner cycle_max_objects="1000000"
mc admin config set ALIAS scanner cycle_max_directories="100000"
```
The canonical persistent bitrot cadence belongs to the `heal` subsystem:
```bash
mc admin config set ALIAS heal bitrot_cycle="2592000"
```
Environment variables take precedence over persisted config and should be
recorded separately:
Environment variables take precedence over persisted config and should be recorded separately:
```bash
RUSTFS_SCANNER_DELAY=30
@@ -297,8 +183,7 @@ RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES=100000
RUSTFS_SCANNER_BITROT_CYCLE_SECS=2592000
```
After each config change, read scanner status and confirm the effective value
and `source` under `runtime_config`.
After each config change, read scanner status and confirm the effective value and `source` under `runtime_config`.
## Observation Window
@@ -308,85 +193,45 @@ Use the same window for each run:
2. Wait until foreground workload is idle.
3. Save scanner and heal config.
4. Save one scanner status snapshot.
5. Collect scanner status and host telemetry for at least 30 minutes, or for
one complete scanner cycle when that is practical.
5. Collect scanner status and host telemetry for at least 30 minutes, or for one complete scanner cycle when practical.
6. Save one final scanner status snapshot.
Longer windows are better for cycle interval comparisons. Short windows are
acceptable for quick pressure checks only if the conclusion avoids changing
defaults.
Longer windows are better for cycle interval comparisons. Short windows are acceptable for quick pressure checks only if the conclusion avoids changing defaults.
## Fields To Compare
Compare these fields between baseline and tuned runs:
Field semantics are defined in [Scanner Runtime Controls](scanner-runtime-controls.md); the decision fields for a before/after comparison are:
| Field | Why it matters |
| Field | Decision it supports |
|---|---|
| `runtime_config.*.value` and `runtime_config.*.source` | Confirms the tested settings actually took effect. |
| `metrics.pacing_pressure.primary_pressure` | Shows whether pressure is from queues, budgets, pause activity, active scans, or no scanner pressure. |
| `metrics.pacing_pressure.last_cycle_total_pause_ratio` | Shows how much of the last cycle was cooperative scanner pause time. |
| `metrics.maintenance_control.primary_control` | Shows whether source-level maintenance is blocked, deferred, active, only pacing-limited, or idle. |
| `metrics.maintenance_control.sources` | Shows the source, state, reason, backlog, current or last-cycle missed work, and partial-cycle count for each scanner maintenance source. |
| `metrics.current_cycle_objects_scanned` | Confirms object scan progress during the current cycle. |
| `metrics.current_cycle_directories_scanned` | Confirms directory walk progress during the current cycle. |
| `metrics.last_cycle_result` | Confirms whether the previous cycle completed, stopped partially, or failed. |
| `metrics.last_cycle_partial_reason` | Shows which budget stopped a partial cycle. |
| `metrics.last_cycle_partial_source` | Shows which scanner work source consumed the stopping budget. |
| `metrics.source_work` | Shows cumulative work found, queued, skipped, missed, executed, and failed by source. |
| `metrics.current_cycle_source_work` | Shows which source is consuming the current scan cycle. |
| `metrics.last_cycle_source_work` | Shows which source consumed the previous scan cycle. |
| `metrics.replication_repair` | Splits scanner-discovered replication repair by source, kind, scanner role, and execution owner, including bucket object, delete-marker, version-purge, existing-object repair, and site replication boundary states. |
| `metrics.current_cycle_replication_repair` | Shows which replication repair kind is being discovered or admitted in the current cycle. |
| `metrics.last_cycle_replication_repair` | Shows which replication repair kind consumed the previous cycle. |
| `metrics.lifecycle_expiry.current_queued` | Shows scanner-driven expiry/delete work waiting in the expiry worker queue. |
| `metrics.lifecycle_expiry.current_active` | Shows scanner-driven expiry/delete work currently running in expiry workers. |
| `metrics.lifecycle_expiry.queue_missed` | Shows expiry/delete queue admission failures outside the scanner walk itself. |
| `metrics.lifecycle_expiry.scanner_missed` | Shows scanner-discovered expiry/delete work that could not be queued. |
| `metrics.lifecycle_transition.scanner_missed` | Shows scanner-discovered transition work that could not be queued. |
| `metrics.lifecycle_transition.queue_full` | Shows transition queue pressure outside the scanner walk itself. |
| `metrics.lifecycle_transition.compensation_pending` | Shows transition compensation still pending or running after queue pressure. |
| `metrics.lifecycle_transition.failed` | Shows transition worker failures, which should also surface as lifecycle source failure. |
| `metrics.current_cycle_usage_saves` | Shows usage cache saves produced by the active scan cycle. |
| `metrics.last_cycle_usage_saves` | Shows usage cache saves produced by the previous completed or partial scan cycle. |
| `metrics.usage_freshness.dirty_pending_buckets` | Shows whether bucket/object mutations are still waiting for usage refresh. |
| `metrics.usage_freshness.last_cycle_dirty_buckets` | Shows how many dirty buckets were picked up by the last cycle. |
| `metrics.usage_freshness.last_cycle_cleared_dirty_buckets` | Shows how many dirty bucket marks were cleared by a successful cycle. |
| `metrics.usage_freshness.last_usage_save_result` | Confirms whether the last usage save succeeded, failed, or was skipped. |
| `metrics.life_time_ops.scan_cycle` | Confirms scanner cycles actually started after the workload. |
| `metrics.life_time_ops.scan_bucket_drive` | Confirms bucket-drive scan work reached the storage layer. |
| `metrics.life_time_ops.scan_object` | Confirms object metadata scanning advanced for object workloads. |
| `metrics.life_time_ops.save_usage` | Confirms `DataUsageInfo` save work happened; this is the key freshness signal for bucket metrics. |
| `metrics.scan_checkpoint` | Confirms partial cycles preserve resume context. |
| `metrics.oldest_active_path_age_seconds` | Helps identify scanner paths that may be stuck. |
| `runtime_config.*.value` and `runtime_config.*.source` | The tested settings actually took effect. |
| `metrics.pacing_pressure.primary_pressure`, `last_cycle_total_pause_ratio` | Where pressure comes from and how much of the cycle was cooperative pause. |
| `metrics.maintenance_control.primary_control`, `metrics.maintenance_control.sources` | Whether a maintenance source is blocked, deferred, active, or only pacing-limited. |
| `metrics.current_cycle_objects_scanned`, `metrics.current_cycle_directories_scanned` | Scan progress continues. |
| `metrics.last_cycle_result`, `last_cycle_partial_reason`, `last_cycle_partial_source` | Whether the previous cycle completed, which budget stopped it, and which source consumed it. |
| `metrics.source_work`, `metrics.current_cycle_source_work`, `metrics.last_cycle_source_work` | `missed` growth per source is a downstream admission problem, not pacing. |
| `metrics.replication_repair` (and current/last-cycle variants) | Repair kind, `scanner_role`, and `execution_owner` for replication backlog runs. |
| `metrics.lifecycle_expiry.{current_queued,current_active,queue_missed,scanner_missed}` | Expiry backlog and admission failures. |
| `metrics.lifecycle_transition.{scanner_missed,queue_full,compensation_pending,failed}` | Transition backlog, queue pressure, and worker failures. |
| `metrics.usage_freshness.*`, `metrics.current_cycle_usage_saves`, `metrics.last_cycle_usage_saves` | Bucket metrics freshness; `last_usage_save_result` must be `success`. |
| `metrics.life_time_ops.{scan_cycle,scan_bucket_drive,scan_object,save_usage}` | Cycles, bucket-drive scans, object scans, and `DataUsageInfo` saves actually happened after the workload. |
| `metrics.scan_checkpoint`, `metrics.oldest_active_path_age_seconds` | Partial cycles preserve resume context; stuck paths. |
Do not use a single CPU spike as the conclusion. Compare average and p95 CPU
over the same observation window.
Do not use a single CPU spike as the conclusion; compare average and p95 CPU over the same observation window.
For heal or bitrot pressure investigations, also capture
`/v3/background-heal/status` from every distributed endpoint and compare
`healOperations.queueLength`,
`healOperations.activeTasks`, `healOperations.queuedBySource`,
`healOperations.activeBySource`, `healOperations.queuedByPriority`, and
`healOperations.activeByPriority`. These fields distinguish scanner-submitted
low-priority work from manual admin heal and auto-heal work.
For heal or bitrot pressure investigations, also capture `/v3/background-heal/status` from every distributed endpoint and compare `healOperations.queueLength`, `activeTasks`, `queuedBySource`, `activeBySource`, `queuedByPriority`, and `activeByPriority` (see [Reading Heal Operations](scanner-runtime-controls.md#reading-heal-operations)).
`scanner-summary.csv` includes the heal operation totals needed for quick
before/after comparison. In distributed runs, these fields are aggregated from
the background-heal status snapshots captured across `--metrics-endpoints`.
### `scanner-summary.csv` columns
| Field | Why it matters |
In distributed runs the heal columns are aggregated from the background-heal snapshots captured across `--metrics-endpoints`.
| Column | Meaning |
|---|---|
| `heal_queue_length` | Total queued heal requests at the same timestamp as the scanner status sample. |
| `heal_active_tasks` | Total running heal tasks. |
| `heal_scanner_queued` | Scanner-submitted heal or bitrot work waiting in the queue. |
| `heal_admin_queued` | Manual/admin heal work waiting in the queue. |
| `heal_auto_heal_queued` | Auto-heal work waiting in the queue, typically from disk/set recovery paths. |
`scanner-summary.csv` also includes usage freshness columns for quick
post-start bucket metrics validation:
| Field | Why it matters |
|---|---|
| `current_cycle_usage_saves` | Usage saves during the current cycle. |
| `last_cycle_usage_saves` | Usage saves from the last finished or partial cycle. |
| `usage_dirty_pending_buckets` | Dirty buckets still waiting for scanner refresh. |
@@ -404,74 +249,36 @@ post-start bucket metrics validation:
A useful tuning result has all of these properties:
- average or p95 scanner-related CPU and disk pressure decreases;
- `current_cycle_objects_scanned` or `current_cycle_directories_scanned`
continues to advance;
- `source_work.missed` does not grow unexpectedly for lifecycle, replication,
heal, or bitrot;
- `last_cycle_result` is either `success` or a partial result with a clear
budget reason and checkpoint;
- `current_cycle_objects_scanned` or `current_cycle_directories_scanned` continues to advance;
- `source_work.missed` does not grow unexpectedly for lifecycle, replication, heal, or bitrot;
- `last_cycle_result` is either `success` or a partial result with a clear budget reason and checkpoint;
- data usage freshness remains acceptable for the tested deployment.
Treat these as failure signals:
- CPU drops only because the scanner stops making progress;
- `primary_pressure` stays at `queued_scans` while queues grow;
- `last_cycle_partial_reason` repeats forever with no checkpoint movement;
- lifecycle expiry `queue_missed`, `scanner_missed`, `current_queued`, or
`current_active` grows during a run that was expected to reduce expiry
backlog;
- lifecycle transition `scanner_missed`, `queue_full`,
`compensation_pending`, or `failed` grows during a run that was expected to
reduce backlog;
- bucket metrics show zero usage after post-start uploads while dirty usage
remains pending and `life_time_save_usage` does not advance;
- `bucket_replication` missed work with `scanner_role=repair_admission` grows
while replication worker queues or target failures are also growing; treat
this as downstream replication pressure, not only scanner pacing pressure;
- `site_replication` `active_resync` grows and is interpreted as scanner-owned
repair execution; `scanner_role=boundary_signal` and
`execution_owner=site_replication_runtime` mean active site resync remains
owned by the site replication runtime and admin resync path;
- heal or bitrot work moves from `queued` to `missed` after a scanner pacing
change.
| Signal | Reading |
|---|---|
| CPU drops only because the scanner stops making progress | Not a tuning win. |
| `primary_pressure` stays at `queued_scans` while queues grow | Concurrency, not pacing, is the constraint. |
| `last_cycle_partial_reason` repeats forever with no checkpoint movement | Budget too small or checkpoint not advancing. |
| Lifecycle expiry `queue_missed`, `scanner_missed`, `current_queued`, or `current_active` grows during a run meant to reduce expiry backlog | Downstream expiry pressure. |
| Lifecycle transition `scanner_missed`, `queue_full`, `compensation_pending`, or `failed` grows during a run meant to reduce backlog | Downstream transition pressure. |
| Bucket metrics show zero usage after post-start uploads while dirty usage remains pending and `life_time_save_usage` does not advance | Usage freshness regression. |
| `bucket_replication` missed work with `scanner_role=repair_admission` grows while replication worker queues or target failures also grow | Downstream replication pressure, not only scanner pacing. |
| `site_replication` `active_resync` grows and is read as scanner-owned repair execution | Misreading: `scanner_role=boundary_signal` and `execution_owner=site_replication_runtime` mean active resync remains owned by the site replication runtime. |
| Heal or bitrot work moves from `queued` to `missed` after a scanner pacing change | Heal admission regression. |
## PR Evidence Checklist
For scanner behavior PRs, include this evidence when available:
For scanner behavior PRs, include when available:
- RustFS commit SHA and branch.
- Deployment shape: node count, disk count, disk type, CPU count, memory, and
object count.
- Deployment shape: node count, disk count, disk type, CPU count, memory, object count.
- Workload command or script and benchmark artifact path.
- Scanner and heal config before and after tuning.
- Observation window and sample interval.
- Scanner status snapshots or time series.
- Host CPU and disk telemetry.
- Usage freshness fields from `scanner-summary.csv` when validating bucket
metrics or issue 3496-style timing.
- Short conclusion that separates pressure reduction from scanner progress.
- `scanner-validation-report.md` from the harness when using the scripted
collection path.
## Final Parity Validation Closure
Use the final validation run to prove the scanner control plane is coherent,
not to introduce new runtime behavior. A complete closure package should have
at least these runs:
| Run | Required evidence |
|---|---|
| Single-node, single-disk small-object idle | Scanner status series, host telemetry, `scanner-summary.csv`, and a conclusion that CPU or disk pressure is lower without scan progress stopping. |
| Single-node post-start bucket metrics freshness | Empty data path startup, post-start bucket creation/upload, bucket metrics snapshots, `scanner-summary.csv` usage freshness columns, and evidence that `DataUsageInfo` save work occurred before accepting bucket usage metrics. |
| Single-node erasure or multi-disk | Checkpoint movement, active path age, set/disk scan pressure, data usage freshness, and before/after scanner config. |
| Distributed lifecycle backlog | `maintenance_control`, lifecycle expiry/transition queue fields, source work missed/failed counts, and by-host admin metrics. |
| Distributed replication backlog | Bucket replication repair kind counters, `scanner_role`, `execution_owner`, site replication passive/active boundary counters, source work queued/skipped/missed counts, and by-host admin metrics. |
| Heal or bitrot pressure | Background heal `healOperations` queued/active source and priority counts, scanner source work for heal/bitrot, and by-host admin metrics. |
The expected conclusion is MinIO-style scanner behavior at the operational
contract level: scanner remains enabled, pacing is observable and adjustable,
partial progress is explainable, maintenance work is attributed by source, and
downstream lifecycle, replication, heal, and bitrot backlog can be diagnosed
without guessing from CPU usage alone.
For documentation-only PRs, it is enough to verify links and formatting.
- Usage freshness fields from `scanner-summary.csv` when validating bucket metrics timing.
- A short conclusion that separates pressure reduction from scanner progress.
- `scanner-validation-report.md` from the harness when using the scripted collection path.