rustfs/docs/architecture/tier-stats-contract.md

Tier Stats Contract

Use this when: changing what GET /rustfs/admin/v3/tier-stats returns, adding a tier accounting source, or wiring a metric to a remote tier request. Source of truth: rustfs/src/admin/handlers/tier.rs (GetTierInfo, tier_stats_body), crates/ecstore/src/services/notification_sys.rs (ClusterTierDailyStats), crates/ecstore/src/bucket/lifecycle/tier_last_day_stats.rs (LastDayTierStats), crates/ecstore/src/services/tier/warm_backend.rs (MeteredWarmBackend), crates/obs/src/metrics/schema/tier.rs.

Two quantities, never one

A tier carries two unrelated numbers, and reporting either as the other is the defect this contract exists to prevent.

• Stored inventory — how many bytes, objects and versions currently live in the tier. It is produced by the scanner, persisted in the data usage snapshot (DataUsageInfo::tier_stats), and is already cluster-wide. It is a level, not a rate.
• Rolling activity — how many transitions the cluster completed into the tier during the last 24 hours. Each node keeps its own 24-bin ring in memory (TransitionState::add_lastday_stats) and counts only the transitions it completed itself. It is a rate window, not a level, and it is lost on restart.

Neither substitutes for the other. An empty rolling window does not mean an empty tier, and a populated tier does not imply recent activity.

Response contract

contractVersion names the body shape.

Version 1 was a bare map of tier name to the answering process's rolling counters, with nothing in the body distinguishing a node from the cluster or a rate from a level. It remains reachable at ?format=legacy for callers pinned to it; it is not extended.

Version 2 is the default body:

• inventory.status is accounted, not-accounted, or unavailable. Per-tier inventory values are present only under accounted. An absent per-tier accounting means "not accounted", never "zero" — the scanner classifies objects by tier only once a tier exists, so a zero would be indistinguishable from an unscanned cluster.
• activity.status is complete or partial, with nodesReporting, nodesExpected, and the unavailableNodes that could not be asked, timed out, or answered with a ring this build refuses to merge. A peer that predates the TierDailyStats RPC answers UNIMPLEMENTED and is reported as unavailable rather than as zero activity.
• Each tier entry carries type only when the name is a configured remote tier. A name that carries stats without a configuration is a local storage class the scanner accounts for, or a tier removed since the snapshot.

Field names inside the counter objects are totalSize, numVersions, numObjects — the camelCase spelling admin clients expect from the madmin tier stats shape, rather than the Rust field names version 1 leaked. This aligns the spelling; it is not a claim that the envelope is byte-compatible with a specific madmin release, and a client-side check belongs with whatever client a release wants to support.

Why merging rings is not summing totals

Nodes are asked concurrently under a per-peer deadline, and each answer is merged with LastDayTierStats::merge rather than added. Merging ages the older ring forward to the newer ring's clock first, so a node that stopped transitioning yesterday contributes only the hours still inside the rolling day. Adding raw totals would keep expired hours alive for as long as the node stays up.

Double counting is prevented at the source, not at the aggregator: add_lastday_stats runs once per committed transition on the node that committed it, so a transition retried on another node is counted once, by whichever node finally committed it.

Tier request metrics

rustfs_tier_requests_success and rustfs_tier_requests_failure are updated at the two seams every remote tier request passes through: MeteredWarmBackend, which wraps every backend new_warm_backend builds, and MeteredTransitionCandidateReconciler, which wraps the separate recovery probe handle new_transition_candidate_reconciler builds. A new provider is therefore counted by construction. Two seams are deliberately delegated without a counter: validate, whose trait default performs no remote request on every backend but one, and a probe_transition_candidate that answers Unsupported, which is the same default. Counting either would report requests that were never issued.

The label set is closed by two enums in crates/scanner-metrics/src/metrics.rs: TierRequestOperation (put, get, remove, probe, in_use) and TierRequestOutcome (success, backend_error, timeout, cancelled). Tier names, endpoints and object keys must never become labels — the endpoint carries credentials in its userinfo form and the key is unbounded.

timeout and cancelled are recognised from std::io::ErrorKind, never from an error message, so a message that mentions an endpoint cannot reach a label. Until the transition client grows bounded deadlines (rustfs/backlog#2204), few failures actually carry TimedOut, so most land in backend_error; the classification is the seam that work extends, not a claim that timeouts are already distinguishable.

The outcome classifies the request only. A transition whose remote PUT succeeded and whose local commit then failed is a success here: the remote service did perform the request, and recording it as a tier failure would hide a leaked remote object behind an apparent backend outage. Local commit failure is observable through the ILM task-event metrics instead.