mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-06 12:09:12 +00:00
houseme
0e6ee3bf62
* test(scanner): wire usage and heal rebuild gates * docs(scanner): define usage authority protocol * docs(heal): clarify scanner and ecstore boundaries * refactor(scanner): split metrics from contracts * feat(scanner): use shared workload snapshots * fix(ecstore): recheck capacity before decommission drain
130 lines
7.9 KiB
Markdown
130 lines
7.9 KiB
Markdown
# Scanner Usage Publication Contract
|
|
|
|
**Use this when:** changing scanner data-usage persistence, quota-visible usage snapshots, scanner cycle state, dirty-usage catch-up, or the conditions under which an observed scanner snapshot may be served.
|
|
**Source of truth:** `crates/scanner/src/scanner/usage_store.rs`, `crates/scanner/src/scanner/cycle_state.rs`, `crates/scanner/src/scanner/backlog.rs`, `crates/scanner/src/scanner/leadership.rs`, `crates/scanner/src/data_usage_define.rs`, and quota fallback behavior in `crates/ecstore/src/bucket/quota/checker.rs`.
|
|
|
|
## Ownership Model
|
|
|
|
One scanner cycle owns an authoritative publication only after it holds the
|
|
cluster scanner leadership claim and proves that the storage publication epoch
|
|
has not moved. The leadership claim is persisted in the scanner cycle state,
|
|
while data-usage publication admission is owned by ECStore because it knows
|
|
whether rebalance, decommission, or another data-movement operation has changed
|
|
the generation that scanner results are allowed to describe.
|
|
|
|
The scanner may compute usage without publication ownership, but it must not
|
|
turn that result into authoritative quota-visible state. A complete publication
|
|
therefore has three identities:
|
|
|
|
- the scanner leader epoch that owns the cycle;
|
|
- the storage publication epoch that fences data movement;
|
|
- the per-object CAS revision on the usage object being replaced.
|
|
|
|
If any identity changes before commit, the result is a candidate for retry or
|
|
observation, not an authoritative baseline.
|
|
|
|
## Fences
|
|
|
|
The protocol uses separate fences because they exclude different stale inputs.
|
|
They must not be collapsed unless the replacement proves the same exclusions.
|
|
|
|
| Fence | Owner | Excludes |
|
|
|---|---|---|
|
|
| Scanner leadership claim | scanner | competing scanner leaders and stale cycle writers |
|
|
| Storage publication epoch | ECStore | usage computed across rebalance, decommission, or other data-movement generations |
|
|
| Publication lease | scanner peers through ECStore-facing activity probes | remote dirty-usage or maintenance state that has not acknowledged the candidate |
|
|
| CAS revision | backing config object store | lost updates to `.usage.v2.json`, `.usage.json`, or cycle-state objects |
|
|
| Per-set freshness | scanner aggregation | a merged usage snapshot that combines stale and current set results |
|
|
| Tier registry generation | scanner tier accounting | bytes classified against a different warm-tier registry |
|
|
| Usage floor identity | scanner publication and ECStore quota fallback | empty or legacy values becoming plausible authoritative quota input |
|
|
|
|
A reader that cannot prove the required fence for its surface must fail closed
|
|
or use the documented observed path below. It must not synthesize an empty usage
|
|
snapshot for a missing or corrupt authoritative object.
|
|
|
|
## Persisted Objects
|
|
|
|
The persisted objects are part of the compatibility contract. Removing one
|
|
requires a compatibility window and a dedicated cleanup entry.
|
|
|
|
| Object | Owner | Lifecycle |
|
|
|---|---|---|
|
|
| `.usage-cache.bin` under each bucket and set | scanner disk walk | Rebuilt by scanner from object metadata. Missing data causes a rescan for that bucket/set; corrupt data is not a complete baseline. |
|
|
| `.bloomcycle.bin` | scanner cycle state | CAS-updated by the leader. Missing state starts from an uninitialized cycle; corrupt or future state is quarantined before automatic retry. |
|
|
| `.usage.v2.json` and `.usage.json` | scanner authoritative publication | `.usage.v2.json` is the primary complete usage snapshot. `.usage.json` is read only as a legacy or companion baseline when it carries a valid persisted identity. Neither bypasses the v2 epoch fence, and readers may treat a snapshot as authoritative only when its baseline identity and completion fields validate. |
|
|
| `.usage.observed.json` | scanner observation path | Written when an authoritative publication cannot be proven but a diagnostic snapshot is still useful. It is never a hard-quota authority. |
|
|
| `bucket-metadata/.usage.json` | scanner usage floor | Carries the persisted per-bucket floor used by quota during a degraded authoritative-usage window. It is static until the next complete scanner publication. |
|
|
| `.bloomcycle.bin.recovery-required.json` | scanner cycle recovery | Quarantines invalid cycle state with retry evidence. Only scanner recovery code updates or clears it. |
|
|
| `.scanner-cycle.lock` | scanner runtime lock | Serializes cycle-level work. A missing lock object is not itself usage evidence. |
|
|
| `.scanner-pause-backlog.json` | scanner pause and catch-up ledger | Tracks dirty usage, discovered lifecycle work, and full-scan catch-up while authoritative publication is fenced by data movement. It never grants publication admission. |
|
|
|
|
## Observed Snapshots
|
|
|
|
Observed snapshots are a diagnostic and availability layer. They may be served
|
|
only when the snapshot explicitly reports that it is partial or observational,
|
|
and only to consumers that do not make hard quota, durability, or deletion
|
|
decisions from it. Admin usage views may expose this state with completeness
|
|
flags so operators can see progress while the authoritative publication is
|
|
blocked. Quota enforcement must not use an observed snapshot as the current
|
|
usage authority.
|
|
|
|
When an authoritative snapshot is unavailable, quota admission may use the
|
|
persisted usage floor. That is an availability fallback, not a fresh count: live
|
|
writes do not advance the floor, and overrun is bounded only by writes accepted
|
|
before the next complete scanner publication. If no valid persisted floor is
|
|
available, quota remains unavailable and fails closed.
|
|
|
|
## Availability Decision
|
|
|
|
Decision date: 2026-09-03.
|
|
|
|
RustFS keeps scanner usage as the authority for hard quota admission. The
|
|
publication protocol therefore remains necessary: leadership, storage epoch,
|
|
lease, CAS, observed snapshot, and usage-floor layers are the proof machinery
|
|
that lets a distributed background scan feed a quota decision without accepting
|
|
stale or cross-generation usage as current truth.
|
|
|
|
The availability contract is:
|
|
|
|
- the authoritative fast path reads complete in-memory or persisted scanner
|
|
usage;
|
|
- during upgrade or publication outage, quota may admit against the persisted
|
|
per-bucket usage floor;
|
|
- the floor is advisory for the outage window and must converge back to a
|
|
complete scanner publication;
|
|
- a bucket with neither authoritative usage nor a valid floor fails closed.
|
|
|
|
Changing this decision to a soft-quota model would be a product change, not a
|
|
scanner refactor. It would need a staged removal of the authority-specific
|
|
layers and compatibility handling for the persisted objects above.
|
|
|
|
## Deletion And Recovery Rules
|
|
|
|
Only the owner of an object may delete or quarantine it:
|
|
|
|
- scanner may rebuild per-set `.usage-cache.bin` after a scan proves the
|
|
replacement contents;
|
|
- scanner cycle recovery may quarantine invalid `.bloomcycle.bin` and clear the
|
|
marker only after a valid cycle state is persisted;
|
|
- scanner publication may replace `.usage.v2.json` or legacy companions only
|
|
through the publication fences above;
|
|
- quota consumers may read the usage floor but must not delete or repair it;
|
|
- operators may reset scanner usage state only through the supported scanner
|
|
reset surface, which records the reset paths and forces a full rebuild.
|
|
|
|
Missing, undecodable, or identity-less data is not converted to zero. It is
|
|
reported as uninitialized, recovery-required, observed-only, or unavailable
|
|
according to the reader's surface.
|
|
|
|
## Existing Fixes As Invariants
|
|
|
|
Several prior scanner fixes are consequences of this contract rather than
|
|
standalone patches:
|
|
|
|
- incomplete scanner usage must not become a complete admin or quota baseline,
|
|
because completeness and floor identity are part of publication ownership;
|
|
- dirty usage and maintenance acknowledgements must fence publication, because
|
|
a remote node with unacknowledged work can invalidate the candidate;
|
|
- a legacy or backup usage object may help recover availability only when it
|
|
carries a valid baseline identity and does not cross the primary epoch fence.
|