mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-06 03:59:14 +00:00
Zhengchao An
9863f4848d
`GET /v3/tier-stats` answered from whichever process received the request, returning that node's rolling 24-hour transition counters as if they were cluster totals, and the `TierRequestsSuccess` and `TierRequestsFailure` metric names had no producer at all. The body now separates the two quantities a tier carries. Stored inventory comes from the persisted scanner usage snapshot, which is already cluster-wide; rolling activity is summed over every member through a new read-only `TierDailyStats` peer RPC. Rings are merged rather than added, so an idle node's expired hours age out, and each node counts only its own committed transitions, so a retry is counted once. Coverage travels with the numbers: `activity.status` names the reporting members and the ones that could not be asked, timed out, or answered with a ring this build refuses to merge, and per-tier inventory is absent rather than zero when the snapshot has no accounting. The version 1 body stays reachable at `?format=legacy`. Tier request counters are recorded at the two seams every remote request passes through, so a new provider is counted by construction, with a closed operation/outcome label set that can never grow a tier name, endpoint or object key. Closes rustfs/backlog#2207 Co-authored-by: cxymds <cxymds@gmail.com>
44 lines
5.5 KiB
Markdown
44 lines
5.5 KiB
Markdown
# Tier Stats Contract
|
|
|
|
**Use this when:** changing what `GET /rustfs/admin/v3/tier-stats` returns, adding a tier accounting source, or wiring a metric to a remote tier request.
|
|
**Source of truth:** `rustfs/src/admin/handlers/tier.rs` (`GetTierInfo`, `tier_stats_body`), `crates/ecstore/src/services/notification_sys.rs` (`ClusterTierDailyStats`), `crates/ecstore/src/bucket/lifecycle/tier_last_day_stats.rs` (`LastDayTierStats`), `crates/ecstore/src/services/tier/warm_backend.rs` (`MeteredWarmBackend`), `crates/obs/src/metrics/schema/tier.rs`.
|
|
|
|
## Two quantities, never one
|
|
|
|
A tier carries two unrelated numbers, and reporting either as the other is the defect this contract exists to prevent.
|
|
|
|
- **Stored inventory** — how many bytes, objects and versions currently live in the tier. It is produced by the scanner, persisted in the data usage snapshot (`DataUsageInfo::tier_stats`), and is already cluster-wide. It is a level, not a rate.
|
|
- **Rolling activity** — how many transitions the cluster completed into the tier during the last 24 hours. Each node keeps its own 24-bin ring in memory (`TransitionState::add_lastday_stats`) and counts only the transitions it completed itself. It is a rate window, not a level, and it is lost on restart.
|
|
|
|
Neither substitutes for the other. An empty rolling window does not mean an empty tier, and a populated tier does not imply recent activity.
|
|
|
|
## Response contract
|
|
|
|
`contractVersion` names the body shape.
|
|
|
|
Version 1 was a bare map of tier name to the answering process's rolling counters, with nothing in the body distinguishing a node from the cluster or a rate from a level. It remains reachable at `?format=legacy` for callers pinned to it; it is not extended.
|
|
|
|
Version 2 is the default body:
|
|
|
|
- `inventory.status` is `accounted`, `not-accounted`, or `unavailable`. Per-tier `inventory` values are present only under `accounted`. An absent per-tier accounting means "not accounted", never "zero" — the scanner classifies objects by tier only once a tier exists, so a zero would be indistinguishable from an unscanned cluster.
|
|
- `activity.status` is `complete` or `partial`, with `nodesReporting`, `nodesExpected`, and the `unavailableNodes` that could not be asked, timed out, or answered with a ring this build refuses to merge. A peer that predates the `TierDailyStats` RPC answers `UNIMPLEMENTED` and is reported as unavailable rather than as zero activity.
|
|
- Each tier entry carries `type` only when the name is a configured remote tier. A name that carries stats without a configuration is a local storage class the scanner accounts for, or a tier removed since the snapshot.
|
|
|
|
Field names inside the counter objects are `totalSize`, `numVersions`, `numObjects` — the camelCase spelling admin clients expect from the `madmin` tier stats shape, rather than the Rust field names version 1 leaked. This aligns the spelling; it is not a claim that the envelope is byte-compatible with a specific `madmin` release, and a client-side check belongs with whatever client a release wants to support.
|
|
|
|
## Why merging rings is not summing totals
|
|
|
|
Nodes are asked concurrently under a per-peer deadline, and each answer is merged with `LastDayTierStats::merge` rather than added. Merging ages the older ring forward to the newer ring's clock first, so a node that stopped transitioning yesterday contributes only the hours still inside the rolling day. Adding raw totals would keep expired hours alive for as long as the node stays up.
|
|
|
|
Double counting is prevented at the source, not at the aggregator: `add_lastday_stats` runs once per committed transition on the node that committed it, so a transition retried on another node is counted once, by whichever node finally committed it.
|
|
|
|
## Tier request metrics
|
|
|
|
`rustfs_tier_requests_success` and `rustfs_tier_requests_failure` are updated at the two seams every remote tier request passes through: `MeteredWarmBackend`, which wraps every backend `new_warm_backend` builds, and `MeteredTransitionCandidateReconciler`, which wraps the separate recovery probe handle `new_transition_candidate_reconciler` builds. A new provider is therefore counted by construction. Two seams are deliberately delegated without a counter: `validate`, whose trait default performs no remote request on every backend but one, and a `probe_transition_candidate` that answers `Unsupported`, which is the same default. Counting either would report requests that were never issued.
|
|
|
|
The label set is closed by two enums in `crates/scanner-metrics/src/metrics.rs`: `TierRequestOperation` (`put`, `get`, `remove`, `probe`, `in_use`) and `TierRequestOutcome` (`success`, `backend_error`, `timeout`, `cancelled`). Tier names, endpoints and object keys must never become labels — the endpoint carries credentials in its userinfo form and the key is unbounded.
|
|
|
|
`timeout` and `cancelled` are recognised from `std::io::ErrorKind`, never from an error message, so a message that mentions an endpoint cannot reach a label. Until the transition client grows bounded deadlines (rustfs/backlog#2204), few failures actually carry `TimedOut`, so most land in `backend_error`; the classification is the seam that work extends, not a claim that timeouts are already distinguishable.
|
|
|
|
The outcome classifies the request only. A transition whose remote PUT succeeded and whose local commit then failed is a `success` here: the remote service did perform the request, and recording it as a tier failure would hide a leaked remote object behind an apparent backend outage. Local commit failure is observable through the ILM task-event metrics instead.
|