feat(scanner): expose distributed metrics (#3452)

* feat(scanner): expose distributed metrics

* docs(scanner): clarify distributed metrics collection

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
This commit is contained in:
Henry Guo
2026-06-15 07:05:43 +08:00
committed by GitHub
parent e4c1ee9941
commit 2c5615d2ea
4 changed files with 834 additions and 1 deletions
@@ -156,6 +156,27 @@ scripts/run_scanner_validation_harness.sh \
The harness writes scanner/heal config snapshots, scanner status samples, host
telemetry when available, run metadata, and `scanner-summary.csv`.
For distributed runs, capture scanner admin metrics from every node with
`by-host=true`. The metrics endpoint reports the node that handles the request;
`by-host=true` preserves that node's host view but does not collect peer nodes.
These per-node artifacts include active path age, checkpoint state, pacing
pressure, source work, and queued/skipped/missed downstream admission counters.
```bash
for endpoint in http://node-a:9000 http://node-b:9000 http://node-c:9000; do
node="${endpoint#http://}"
node="${node%%:*}"
awscurl \
--service s3 \
--region us-east-1 \
--access_key "$RUSTFS_ACCESS_KEY" \
--secret_key "$RUSTFS_SECRET_KEY" \
--request GET \
"${endpoint}/rustfs/admin/v3/metrics?types=1&by-host=true&n=1" \
> "artifacts/scanner-metrics.${node}.$(date -u +%Y%m%dT%H%M%SZ).ndjson"
done
```
Example status request:
```bash
@@ -146,6 +146,35 @@ Use these counters to decide whether scan progress is limited by scanner pacing
or by a downstream subsystem such as lifecycle transition, replication repair,
or heal admission.
## Reading Distributed Metrics
`/rustfs/admin/v3/scanner/status` and `/rustfs/admin/v3/metrics` report the
node that handles the HTTP request. The metrics endpoint does not fan out to
peer nodes. In distributed deployments, query every node explicitly and keep
`by-host=true` enabled so each response includes that node's host view:
```bash
for endpoint in http://node-a:9000 http://node-b:9000 http://node-c:9000; do
node="${endpoint#http://}"
node="${node%%:*}"
awscurl \
--service s3 \
--region us-east-1 \
--access_key "$RUSTFS_ACCESS_KEY" \
--secret_key "$RUSTFS_SECRET_KEY" \
--request GET \
"${endpoint}/rustfs/admin/v3/metrics?types=1&by-host=true&n=1" \
> "artifacts/scanner-metrics.${node}.$(date -u +%Y%m%dT%H%M%SZ).ndjson"
done
```
The `aggregated.scanner` payload preserves the same scanner progress,
checkpoint, pacing, source work, and lifecycle transition fields used by the
local scanner status, but only for the node that returned the response. The
`by_host.*.scanner` payload keeps that node's host view. Compare the per-node
artifacts externally to find old active paths, partial checkpoints, pacing
pressure, or downstream queue admission problems across the deployment.
## Reading Lifecycle Transition Status
`metrics.lifecycle_transition` focuses on scanner-driven lifecycle transition