mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-07 13:53:12 +00:00
feat(scanner): expose distributed metrics (#3452)
* feat(scanner): expose distributed metrics * docs(scanner): clarify distributed metrics collection --------- Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> Co-authored-by: houseme <housemecn@gmail.com>
This commit is contained in:
@@ -156,6 +156,27 @@ scripts/run_scanner_validation_harness.sh \
|
||||
The harness writes scanner/heal config snapshots, scanner status samples, host
|
||||
telemetry when available, run metadata, and `scanner-summary.csv`.
|
||||
|
||||
For distributed runs, capture scanner admin metrics from every node with
|
||||
`by-host=true`. The metrics endpoint reports the node that handles the request;
|
||||
`by-host=true` preserves that node's host view but does not collect peer nodes.
|
||||
These per-node artifacts include active path age, checkpoint state, pacing
|
||||
pressure, source work, and queued/skipped/missed downstream admission counters.
|
||||
|
||||
```bash
|
||||
for endpoint in http://node-a:9000 http://node-b:9000 http://node-c:9000; do
|
||||
node="${endpoint#http://}"
|
||||
node="${node%%:*}"
|
||||
awscurl \
|
||||
--service s3 \
|
||||
--region us-east-1 \
|
||||
--access_key "$RUSTFS_ACCESS_KEY" \
|
||||
--secret_key "$RUSTFS_SECRET_KEY" \
|
||||
--request GET \
|
||||
"${endpoint}/rustfs/admin/v3/metrics?types=1&by-host=true&n=1" \
|
||||
> "artifacts/scanner-metrics.${node}.$(date -u +%Y%m%dT%H%M%SZ).ndjson"
|
||||
done
|
||||
```
|
||||
|
||||
Example status request:
|
||||
|
||||
```bash
|
||||
|
||||
@@ -146,6 +146,35 @@ Use these counters to decide whether scan progress is limited by scanner pacing
|
||||
or by a downstream subsystem such as lifecycle transition, replication repair,
|
||||
or heal admission.
|
||||
|
||||
## Reading Distributed Metrics
|
||||
|
||||
`/rustfs/admin/v3/scanner/status` and `/rustfs/admin/v3/metrics` report the
|
||||
node that handles the HTTP request. The metrics endpoint does not fan out to
|
||||
peer nodes. In distributed deployments, query every node explicitly and keep
|
||||
`by-host=true` enabled so each response includes that node's host view:
|
||||
|
||||
```bash
|
||||
for endpoint in http://node-a:9000 http://node-b:9000 http://node-c:9000; do
|
||||
node="${endpoint#http://}"
|
||||
node="${node%%:*}"
|
||||
awscurl \
|
||||
--service s3 \
|
||||
--region us-east-1 \
|
||||
--access_key "$RUSTFS_ACCESS_KEY" \
|
||||
--secret_key "$RUSTFS_SECRET_KEY" \
|
||||
--request GET \
|
||||
"${endpoint}/rustfs/admin/v3/metrics?types=1&by-host=true&n=1" \
|
||||
> "artifacts/scanner-metrics.${node}.$(date -u +%Y%m%dT%H%M%SZ).ndjson"
|
||||
done
|
||||
```
|
||||
|
||||
The `aggregated.scanner` payload preserves the same scanner progress,
|
||||
checkpoint, pacing, source work, and lifecycle transition fields used by the
|
||||
local scanner status, but only for the node that returned the response. The
|
||||
`by_host.*.scanner` payload keeps that node's host view. Compare the per-node
|
||||
artifacts externally to find old active paths, partial checkpoints, pacing
|
||||
pressure, or downstream queue admission problems across the deployment.
|
||||
|
||||
## Reading Lifecycle Transition Status
|
||||
|
||||
`metrics.lifecycle_transition` focuses on scanner-driven lifecycle transition
|
||||
|
||||
Reference in New Issue
Block a user