rustfs/docs/operations/storage-metrics.md

Storage metrics and observer selection

Use this when: configuring storage dashboards, migrating from replicated per-drive metrics, or diagnosing stale and missing disk observations.

Ownership and identity

The storage collectors have two distinct scopes:

Scope	Metric families	Meaning
collection_scope="local"	rustfs_system_drive_*, rustfs_node_disk_*	Detailed metrics for drives owned by the reporting node; includes configured offline slots. Drive counts are local counts.
collection_scope="cluster"	Cluster capacity, health, objects/buckets, erasure sets, and rustfs_cluster_drive_*	One reporting node's observation of the whole cluster.

observer identifies the reporting node. server identifies the drive owner; drive alone is not a unique disk key. Keep rustfs_cluster_id, server, and drive when grouping drives, and retain pool/set/drive indices for topology. Drive counters also carry disk_id, so a physical replacement starts a separate series. Unknown IDs are empty; rustfs_system_drive_info is emitted only when the ID and topology are known. rustfs_system_drive_present preserves a configured slot even when its disk is disconnected and its ID is unavailable.

Global drive metrics contain membership, runtime state, capacity and its source. They do not duplicate remote API counters, error counters, or detailed I/O metrics. An unreachable peer remains in the global inventory with the storage layer's unknown/offline state and stale/missing capacity provenance. Missing capacity is omitted from per-drive byte metrics. Cluster capacity may still include cached observations: check capacity_stale_drives and capacity_missing_drives before interpreting it. The existing cluster offline count includes drives not currently observed online; use global per-drive runtime states to distinguish unknown.

Configure the pipeline

Give every node of a deployment the same stable, unique resource attribute:

OTEL_RESOURCE_ATTRIBUTES=rustfs.cluster.id=production-a

Keep existing resource attributes in the comma-separated value. The example Compose files accept OTEL_RESOURCE_ATTRIBUTES and default to the development cluster ID rustfs-dev; set a distinct ID for each deployment. The Collector's resource_to_telemetry_conversion.enabled: true promotes it to rustfs_cluster_id on scraped samples. Prometheus external_labels are not a substitute: they do not add a cluster label to the local time series.

Use the shipped Collector configuration with send_timestamps: true and keep Prometheus's default honor_timestamps: true. Load .docker/observability/prometheus-rules/rustfs-storage.yml. Use one ingestion route per deployment/node; scraping replicas of the same Collector requires a separate HA deduplication policy. Keep node and Prometheus clocks synchronized.

Storage export callbacks read the current snapshot from memory and stop exporting removed series. They do not probe storage or run peer RPCs. A successful collection updates rustfs_storage_snapshot_last_success_timestamp_seconds; OTLP export alone does not update this value. The source age limit is three times the larger of the collection interval and RUSTFS_OBS_METER_INTERVAL (with its normal default). rustfs_storage_snapshot_max_age_seconds exports the remaining validity budget after deducting collection time from that limit. A slow RPC or usage read therefore cannot make an already expired observation fresh by completing. An unavailable source leaves the last success unchanged; a stalled collector stops exporting its old snapshot after the source age limit.

A Collector may cache a point after RustFS stops exporting it. The recording rules therefore require both a fresh source and a raw point timestamp at least as new as that source's last successful collection. This also removes cached optional fields and old disk IDs when a new snapshot no longer contains them. The timestamp check must run on the raw selector, before label rewriting or recording the value; otherwise PromQL can substitute the query evaluation time for the original sample time. The comparison uses Prometheus millisecond precision so co-published points are not excluded by submillisecond rounding. Current recording rules preserve the original name in source_metric. When adding a storage metric, add its matching rule to that file.

Collection and export are asynchronous, not a transaction across instruments. A snapshot published during an export can briefly mix adjacent observations or withhold a value; the next complete export/collection converges. Source failure visibility is bounded by the published maximum age plus the scrape/rule intervals. Cached capacity has its own observation age and state; a fresh collection does not make that capacity live. Bucket/object counts retain the existing scanner update delay.

Queries and dashboards

The bundled dashboard requires the storage recording rules. Select one Storage cluster and one fresh Cluster observer. Global panels use that observer's complete view. If it expires, the panel shows no data; select another observer. Do not sum replicated global totals or independently take maxima/minima of fields from different observers: they may describe different moments or partitions.

For example, select the observed cluster raw capacity:

rustfs:storage:current{source_metric="rustfs_cluster_capacity_raw_total_bytes",rustfs_cluster_id="production-a",collection_scope="cluster",observer="node1:9000"}

For a per-drive API rate, select owner-scoped counters before rate, then require a current observation. Sum the resulting rates only across the desired drives:

sum by (rustfs_cluster_id, api) (
  rate(rustfs_system_drive_api_calls_total{rustfs_cluster_id="production-a",collection_scope="local"}[5m])
  and ignoring(source_metric)
  rustfs:storage:current{source_metric="rustfs_system_drive_api_calls_total",rustfs_cluster_id="production-a",collection_scope="local"}
)

A missing series is unknown, not zero. Local totals include only fresh owners and can decrease when a node becomes unreachable. Use a selected global observer for configured topology and its explicit unknown/stale indicators.

Rolling upgrades and validation

Install the rules, resource attribute, and dashboard together. During rolling upgrades, old releases have no collection_scope; strict new-scope selectors exclude their replicated series. Local panels initially contain only upgraded owners. Global panels become available when an upgraded observer publishes a fresh snapshot. Historical old-label series remain available for retrospective queries. Do not combine old and new counter histories into one rate.

Run the PromQL regression fixtures from .docker/observability/tests:

promtool test rules storage-rules.test.yml

The native pipeline test in crates/e2e_test/src/storage_metric_ownership_test.rs requires pinned Collector, Prometheus, previous-release RustFS, and current RustFS executables. Set RUSTFS_OTELCOL_BINARY, RUSTFS_PROMETHEUS_BINARY, and RUSTFS_METRICS_BASELINE_BINARY to those files; scripts/e2e_binary.py run supplies the current RustFS as CARGO_BIN_EXE_rustfs. Optionally set RUSTFS_METRICS_E2E_ARTIFACTS to retain logs and Prometheus data. Run only this external-tool test:

python3 scripts/e2e_binary.py build
python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test storage_metric_ownership_pipeline -- --ignored --nocapture

The test first reproduces duplicated global details with the previous release, rolls four nodes forward, verifies owner and disk identity, and checks node loss and recovery while the Collector remains running. SDK unit tests cover removed series, replacements, counter resets, invalid updates, and a stalled collection; PromQL fixtures cover shared paths across clusters/pools and stale cached fields.