mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-05 19:55:37 +00:00
feat(scanner): coordinate usage and workload boundaries (#7093)
* test(scanner): wire usage and heal rebuild gates * docs(scanner): define usage authority protocol * docs(heal): clarify scanner and ecstore boundaries * refactor(scanner): split metrics from contracts * feat(scanner): use shared workload snapshots * fix(ecstore): recheck capacity before decommission drain
This commit is contained in:
houseme
@@ -42,6 +42,8 @@ Required headings and strings in these files are asserted by `scripts/check_arch
|
||||
| [workload-admission-contracts.md](workload-admission-contracts.md) | adding a workload class or snapshot provider, or consuming admission state from a background job |
|
||||
| [background-controller-contract.md](background-controller-contract.md) | adding a status snapshot or reconcile surface for a background service, or being tempted to fold several services into a generic controller |
|
||||
| [background-services-inventory.md](background-services-inventory.md) | you need one audited background service's desired source, current-status inputs, status surface, and declared side effects |
|
||||
| [scanner-usage-publication.md](scanner-usage-publication.md) | changing scanner data-usage cache publication, quota-visible usage snapshots, scanner cycle recovery, or the persisted scanner usage artifacts |
|
||||
| [scanner-usage-authority-decision.md](scanner-usage-authority-decision.md) | deciding whether quota admission depends on scanner data usage, removing scanner publication layers, or designing a scanner storage boundary |
|
||||
| [config-model-boundary-adr.md](config-model-boundary-adr.md) | touching the server-config model (`Config`, `KV`, `KVS`) or its persistence, or asking which crate owns which part of server configuration |
|
||||
| [admin-route-action-snapshot.md](admin-route-action-snapshot.md) | adding, moving, or re-authorizing an admin route and needing to know where the route → handler → `AdminAction` contract is enforced |
|
||||
| [kms-bulk-rekey-contract.md](kms-bulk-rekey-contract.md) | changing the bulk envelope re-wrap sweep, its admin endpoints, the re-wrap primitive, or which objects a rekey may touch |
|
||||
|
||||
@@ -49,6 +49,20 @@ Outer crates reach ECStore only through `rustfs_ecstore::api`, and only from one
|
||||
- RustFS startup internals are crate-private: only `startup_entrypoint` is a public startup module of the `rustfs` library (`rustfs/src/lib.rs`), and items inside the other `startup_*` modules use crate visibility.
|
||||
- The observability dependency baseline is [obs-ecstore-dependency-inventory.md](obs-ecstore-dependency-inventory.md); observability extraction updates it together with the guard.
|
||||
|
||||
## Scanner, Heal, And ECStore
|
||||
|
||||
Heal is split by responsibility, not by the shared word "heal". ECStore owns erasure-set repair primitives: quorum metadata arbitration, EC reconstruction, per-disk rename commit, dangling metadata classification, and orphan data-dir reclamation. These stay in ECStore because they share the same object namespace locks, rename commit model, and data-dir cleanup rules as PUT, DELETE, multipart, lifecycle expiry, rebalance, and decommission. Moving those primitives out would split the lock and commit model across crates.
|
||||
|
||||
`crates/heal` owns repair orchestration: queueing, deduplication, admission, scheduling, resume, MRF replay, replacement-disk tracking, and the admin-facing status/control surface. It reaches storage through `HealStorageAPI`; ECStore-originated repair requests flow back through typed repair channels rather than a Cargo dependency on the heal crate.
|
||||
|
||||
`crates/scanner` owns discovery, data-usage publication, lifecycle/replication scan actions, bitrot scan dispatch, and scanner-driven repair requests. Scanner may request repair through the heal channel, but it must not directly execute erasure-set repair primitives.
|
||||
|
||||
`rustfs-scanner-metrics` owns scanner telemetry DTOs, global scanner counters, lifecycle action labels consumed by metrics, and the short-window latency accumulator used by those metrics. ECStore, lifecycle, observability, admin, and scanner code may depend on this crate for metrics only. `rustfs-scanner-contracts` must not regain metrics, globals, or telemetry implementation; it is reserved for scanner storage or wire contract types.
|
||||
|
||||
`remote_scanner` remains scanner-owned for now because it carries the scanner cycle fence, replay protection, stream envelope, and per-bucket scan result protocol. A future scanner storage seam may either move remote disk scan execution behind an ECStore storage capability or move the whole remote scanner protocol with scanner; leaving the wire protocol split across both sides without a documented owner is not allowed.
|
||||
|
||||
The scanner usage authority decision is fixed in [scanner-usage-authority-decision.md](scanner-usage-authority-decision.md): scanner usage remains hard-quota authority. A future scanner storage seam must therefore model the concrete publication, cycle-lock, usage-floor, observed-snapshot, and recovery-marker capabilities described in [scanner-usage-publication.md](scanner-usage-publication.md), not a generic key-value abstraction.
|
||||
|
||||
## Loss-Prevention Coverage
|
||||
|
||||
The guard pins specific public re-export lines (its `require_source_line` entries) so contract surfaces cannot silently disappear during cleanup. The canonical lists are the guard script and the owning files, not this page:
|
||||
|
||||
@@ -3,6 +3,8 @@
|
||||
**Use this when:** changing heal, PUT/multipart commit, delete, lifecycle expiry, or data-movement code that touches the same `(bucket, object)` commit surface; or evaluating whether RustFS needs a persistent per-object healing marker like MinIO's `x-minio-healing`.
|
||||
**Source of truth:** `crates/ecstore/src/set_disk/ops/heal.rs` (`heal_object_with_explicit_version_regen`, `HealObjectLockKind`, `HEAL_RENAME_INCOMPLETE`), `crates/ecstore/src/set_disk/ops/object.rs` (PUT/DELETE lock sections, `reconcile_old_data_cleanup_receipts`), `crates/ecstore/src/set_disk/core/io_primitives.rs` (`commit_rename_data_dir`, `report_old_data_dir_cleanup`, `reclaim_orphan_data_dirs`), `crates/filemeta/src/fileinfo.rs` (`FileInfo::set_healing`), `crates/heal/src/heal/manager/queue.rs` (dedup keys).
|
||||
|
||||
For crate ownership, read [crate-boundaries.md](crate-boundaries.md): ECStore owns erasure-set repair primitives that share this lock and commit model, while `crates/heal` owns repair orchestration.
|
||||
|
||||
## Model
|
||||
|
||||
Heal and every foreground or background write path serialize on the same object-level namespace write lock (a quorum lock RPC in distributed mode, the in-process lock manager on a single node; granularity is the object, the version component is always `None`), and heal holds its guard across the whole rename commit. MinIO's `x-minio-healing` marker is an out-of-lock defence against version-cleanup logic inside `RenameData` interleaving with a heal commit; RustFS's commit model has no such interleaving, so no persistent marker exists (`x-minio-healing` does not occur in `crates/` or `rustfs/`) and none is needed. Three layers replace it:
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
# Scanner Usage Authority Decision
|
||||
|
||||
**Use this when:** deciding whether quota admission depends on scanner data usage, removing scanner publication layers, or designing a scanner storage boundary.
|
||||
**Source of truth:** [scanner-usage-publication.md](scanner-usage-publication.md), `crates/ecstore/src/bucket/quota/checker.rs`, and the scanner publication state under `crates/scanner/src/scanner/`.
|
||||
|
||||
## Decision
|
||||
|
||||
Date: 2026-09-03
|
||||
|
||||
RustFS keeps scanner data usage as authoritative for quota admission.
|
||||
|
||||
This selects option A from the backlog decision record: the scanner publication protocol remains necessary while quota admission consumes scanner usage. The cycle epoch, publication CAS, data-movement fence, tier-registry fence, observed snapshot layer, and persisted usage floor are retained and documented as protocol invariants rather than treated as removable compatibility clutter.
|
||||
|
||||
## Rationale
|
||||
|
||||
Quota is a write-path admission decision, so serving quota from best-effort scanner data would turn temporary scanner lag into under-enforcement. The current design therefore needs an availability story for authoritative usage instead of deleting the proof layers that make it authoritative.
|
||||
|
||||
The scanner usage floor provides that availability story. It is a lower bound used when a complete authoritative snapshot is not available, including cold startup, upgrade recovery, and incomplete-cycle repair. Observed snapshots remain useful for admin and observability, but they do not become quota authority.
|
||||
|
||||
## Consequences
|
||||
|
||||
#2214 is the hard design input for future usage-publication changes. A future proposal may still choose soft quota and MinIO-style best-effort usage, but that would be a product change with its own staged compatibility plan for persisted artifacts.
|
||||
|
||||
#2219 may design the scanner storage boundary against the current authoritative protocol. The interface must include the CAS key-value, cycle lock, usage-floor, observed-snapshot, and recovery-marker capabilities needed by [scanner-usage-publication.md](scanner-usage-publication.md); it must not hide those proof obligations behind a generic object-store trait.
|
||||
@@ -0,0 +1,129 @@
|
||||
# Scanner Usage Publication Contract
|
||||
|
||||
**Use this when:** changing scanner data-usage persistence, quota-visible usage snapshots, scanner cycle state, dirty-usage catch-up, or the conditions under which an observed scanner snapshot may be served.
|
||||
**Source of truth:** `crates/scanner/src/scanner/usage_store.rs`, `crates/scanner/src/scanner/cycle_state.rs`, `crates/scanner/src/scanner/backlog.rs`, `crates/scanner/src/scanner/leadership.rs`, `crates/scanner/src/data_usage_define.rs`, and quota fallback behavior in `crates/ecstore/src/bucket/quota/checker.rs`.
|
||||
|
||||
## Ownership Model
|
||||
|
||||
One scanner cycle owns an authoritative publication only after it holds the
|
||||
cluster scanner leadership claim and proves that the storage publication epoch
|
||||
has not moved. The leadership claim is persisted in the scanner cycle state,
|
||||
while data-usage publication admission is owned by ECStore because it knows
|
||||
whether rebalance, decommission, or another data-movement operation has changed
|
||||
the generation that scanner results are allowed to describe.
|
||||
|
||||
The scanner may compute usage without publication ownership, but it must not
|
||||
turn that result into authoritative quota-visible state. A complete publication
|
||||
therefore has three identities:
|
||||
|
||||
- the scanner leader epoch that owns the cycle;
|
||||
- the storage publication epoch that fences data movement;
|
||||
- the per-object CAS revision on the usage object being replaced.
|
||||
|
||||
If any identity changes before commit, the result is a candidate for retry or
|
||||
observation, not an authoritative baseline.
|
||||
|
||||
## Fences
|
||||
|
||||
The protocol uses separate fences because they exclude different stale inputs.
|
||||
They must not be collapsed unless the replacement proves the same exclusions.
|
||||
|
||||
| Fence | Owner | Excludes |
|
||||
|---|---|---|
|
||||
| Scanner leadership claim | scanner | competing scanner leaders and stale cycle writers |
|
||||
| Storage publication epoch | ECStore | usage computed across rebalance, decommission, or other data-movement generations |
|
||||
| Publication lease | scanner peers through ECStore-facing activity probes | remote dirty-usage or maintenance state that has not acknowledged the candidate |
|
||||
| CAS revision | backing config object store | lost updates to `.usage.v2.json`, `.usage.json`, or cycle-state objects |
|
||||
| Per-set freshness | scanner aggregation | a merged usage snapshot that combines stale and current set results |
|
||||
| Tier registry generation | scanner tier accounting | bytes classified against a different warm-tier registry |
|
||||
| Usage floor identity | scanner publication and ECStore quota fallback | empty or legacy values becoming plausible authoritative quota input |
|
||||
|
||||
A reader that cannot prove the required fence for its surface must fail closed
|
||||
or use the documented observed path below. It must not synthesize an empty usage
|
||||
snapshot for a missing or corrupt authoritative object.
|
||||
|
||||
## Persisted Objects
|
||||
|
||||
The persisted objects are part of the compatibility contract. Removing one
|
||||
requires a compatibility window and a dedicated cleanup entry.
|
||||
|
||||
| Object | Owner | Lifecycle |
|
||||
|---|---|---|
|
||||
| `.usage-cache.bin` under each bucket and set | scanner disk walk | Rebuilt by scanner from object metadata. Missing data causes a rescan for that bucket/set; corrupt data is not a complete baseline. |
|
||||
| `.bloomcycle.bin` | scanner cycle state | CAS-updated by the leader. Missing state starts from an uninitialized cycle; corrupt or future state is quarantined before automatic retry. |
|
||||
| `.usage.v2.json` and `.usage.json` | scanner authoritative publication | `.usage.v2.json` is the primary complete usage snapshot. `.usage.json` is read only as a legacy or companion baseline when it carries a valid persisted identity. Neither bypasses the v2 epoch fence, and readers may treat a snapshot as authoritative only when its baseline identity and completion fields validate. |
|
||||
| `.usage.observed.json` | scanner observation path | Written when an authoritative publication cannot be proven but a diagnostic snapshot is still useful. It is never a hard-quota authority. |
|
||||
| `bucket-metadata/.usage.json` | scanner usage floor | Carries the persisted per-bucket floor used by quota during a degraded authoritative-usage window. It is static until the next complete scanner publication. |
|
||||
| `.bloomcycle.bin.recovery-required.json` | scanner cycle recovery | Quarantines invalid cycle state with retry evidence. Only scanner recovery code updates or clears it. |
|
||||
| `.scanner-cycle.lock` | scanner runtime lock | Serializes cycle-level work. A missing lock object is not itself usage evidence. |
|
||||
| `.scanner-pause-backlog.json` | scanner pause and catch-up ledger | Tracks dirty usage, discovered lifecycle work, and full-scan catch-up while authoritative publication is fenced by data movement. It never grants publication admission. |
|
||||
|
||||
## Observed Snapshots
|
||||
|
||||
Observed snapshots are a diagnostic and availability layer. They may be served
|
||||
only when the snapshot explicitly reports that it is partial or observational,
|
||||
and only to consumers that do not make hard quota, durability, or deletion
|
||||
decisions from it. Admin usage views may expose this state with completeness
|
||||
flags so operators can see progress while the authoritative publication is
|
||||
blocked. Quota enforcement must not use an observed snapshot as the current
|
||||
usage authority.
|
||||
|
||||
When an authoritative snapshot is unavailable, quota admission may use the
|
||||
persisted usage floor. That is an availability fallback, not a fresh count: live
|
||||
writes do not advance the floor, and overrun is bounded only by writes accepted
|
||||
before the next complete scanner publication. If no valid persisted floor is
|
||||
available, quota remains unavailable and fails closed.
|
||||
|
||||
## Availability Decision
|
||||
|
||||
Decision date: 2026-09-03.
|
||||
|
||||
RustFS keeps scanner usage as the authority for hard quota admission. The
|
||||
publication protocol therefore remains necessary: leadership, storage epoch,
|
||||
lease, CAS, observed snapshot, and usage-floor layers are the proof machinery
|
||||
that lets a distributed background scan feed a quota decision without accepting
|
||||
stale or cross-generation usage as current truth.
|
||||
|
||||
The availability contract is:
|
||||
|
||||
- the authoritative fast path reads complete in-memory or persisted scanner
|
||||
usage;
|
||||
- during upgrade or publication outage, quota may admit against the persisted
|
||||
per-bucket usage floor;
|
||||
- the floor is advisory for the outage window and must converge back to a
|
||||
complete scanner publication;
|
||||
- a bucket with neither authoritative usage nor a valid floor fails closed.
|
||||
|
||||
Changing this decision to a soft-quota model would be a product change, not a
|
||||
scanner refactor. It would need a staged removal of the authority-specific
|
||||
layers and compatibility handling for the persisted objects above.
|
||||
|
||||
## Deletion And Recovery Rules
|
||||
|
||||
Only the owner of an object may delete or quarantine it:
|
||||
|
||||
- scanner may rebuild per-set `.usage-cache.bin` after a scan proves the
|
||||
replacement contents;
|
||||
- scanner cycle recovery may quarantine invalid `.bloomcycle.bin` and clear the
|
||||
marker only after a valid cycle state is persisted;
|
||||
- scanner publication may replace `.usage.v2.json` or legacy companions only
|
||||
through the publication fences above;
|
||||
- quota consumers may read the usage floor but must not delete or repair it;
|
||||
- operators may reset scanner usage state only through the supported scanner
|
||||
reset surface, which records the reset paths and forces a full rebuild.
|
||||
|
||||
Missing, undecodable, or identity-less data is not converted to zero. It is
|
||||
reported as uninitialized, recovery-required, observed-only, or unavailable
|
||||
according to the reader's surface.
|
||||
|
||||
## Existing Fixes As Invariants
|
||||
|
||||
Several prior scanner fixes are consequences of this contract rather than
|
||||
standalone patches:
|
||||
|
||||
- incomplete scanner usage must not become a complete admin or quota baseline,
|
||||
because completeness and floor identity are part of publication ownership;
|
||||
- dirty usage and maintenance acknowledgements must fence publication, because
|
||||
a remote node with unacknowledged work can invalidate the candidate;
|
||||
- a legacy or backup usage object may help recover availability only when it
|
||||
carries a valid baseline identity and does not cross the primary epoch fence.
|
||||
@@ -12,9 +12,9 @@
|
||||
| Class | Provider (`impl WorkloadAdmissionSnapshotProvider`) | `active` / `queued` / `limit` source | Reports `Unknown` when |
|
||||
|---|---|---|---|
|
||||
| `ForegroundRead` | `ConcurrencyManager` in `rustfs/src/storage/concurrency/manager.rs` (source of truth); re-exposed unchanged by the RustFS runtime provider | disk-read permits in use / `None` (the semaphore exposes no waiter count) / configured max concurrent disk reads | the storage registry has no entry |
|
||||
| `ForegroundWrite` | none | none | always: no write-specific admission owner exposes a read-only surface yet |
|
||||
| `ForegroundWrite` | `ConcurrencyManager` in `rustfs/src/storage/concurrency/manager.rs` (source of truth); re-exposed unchanged by the RustFS runtime provider | foreground-write permits in use or legacy active-write counter / `None` / configured or derived write-admission limit | the storage registry has no entry |
|
||||
| `Metadata` | `RustFsWorkloadAdmissionSnapshotProvider` in `rustfs/src/workload_admission.rs` | `Open` once the bucket metadata runtime handle exists; no counts | bucket metadata runtime not initialized |
|
||||
| `Scanner` | same | scanner active work-unit counter / none / none | the counter is zero (idle and uninitialized are indistinguishable) |
|
||||
| `Scanner` | same | scanner active work-unit counter / none / configured set-scan limit when nonzero | scanner runtime not initialized |
|
||||
| `Repair` | same | heal active tasks / heal queue length / `None` (limits live behind the async heal manager state) | heal manager not initialized |
|
||||
| `Replication` | same | active regular + large-object + MRF workers / site replication queue count / `None` (limits owned by the async pool and resize policy) | replication runtime not initialized, or queue stats currently locked |
|
||||
|
||||
@@ -28,6 +28,7 @@ Consumers that read the snapshot to self-throttle exist, and they do not change
|
||||
|---|---|---|
|
||||
| Data-movement backpressure (decommission, rebalance) | `crates/ecstore/src/data_movement/backpressure.rs` (`wait_for_data_movement_admission`, `foreground_pressure`) | Delays the next data-movement step while `ForegroundRead` or `ForegroundWrite` usage exceeds the configured high-water percent. ECStore receives the provider through `set_workload_admission_snapshot_provider` (`crates/ecstore/src/lib.rs`), published from `rustfs/src/startup_background.rs`; with no provider the step is admitted immediately. |
|
||||
| Heal manager mainline throttle | `crates/heal/src/heal/manager.rs` (`new_with_workload_provider`) | When `mainline_throttle_enable` is set, defers heal work while `ForegroundRead` or `ForegroundWrite` utilization exceeds the configured high-water percents; with no provider or the throttle disabled, heal pacing is unchanged. |
|
||||
| Scanner sleeper and scan fan-out | `crates/scanner/src/workload_admission.rs`, `crates/scanner/src/sleeper.rs`, and `crates/scanner/src/scanner_io/guards.rs` | Reads the same provider published from `rustfs/src/startup_background.rs` and combines it with scanner-local foreground read guards. Foreground activity increases scanner sleeps and reduces set/disk scan fan-out to one; cycle budgets still own object, directory, and duration limits. With no provider, scanner keeps the legacy local foreground-read behavior. |
|
||||
|
||||
## Boundary Rules
|
||||
|
||||
|
||||
@@ -57,6 +57,14 @@ Promotion rule: never promote a report-only lane to required from one green run.
|
||||
|
||||
e2e filters live in `.config/nextest.toml`; extend a profile instead of adding a second selector. Before a profile runs, `scripts/check_test_wiring.py` compares its listing to the committed digest in `.config/e2e-<profile>-selection.txt`, so a silent test drop fails closed.
|
||||
|
||||
Scanner usage and heal rebuild coverage are intentionally split by risk and
|
||||
cost. `data_usage_test` runs in the PR `e2e-smoke` lane so changes that affect
|
||||
authoritative scanner usage publication, quota-visible usage, or admin usage
|
||||
snapshots get an end-to-end signal before merge review. `heal_erasure_disk_rebuild_test`
|
||||
runs in `e2e-full` so core erasure heal rebuild regressions are caught no later
|
||||
than the merge queue or `main` push lane; it also remains in `e2e-nightly` with
|
||||
the serialized cluster fault-domain suites for scheduled soak signal.
|
||||
|
||||
## Scheduled validation
|
||||
|
||||
Scheduled lanes never block a PR. Their workflow-local gate fails the run, scheduled failures route to the shared failure-issue action, and `scheduled-validation-freshness.yml` fails when a workflow listed in `.github/scheduled-validations.json` has not run within its `max_age_hours` (a `never_ran_grace_until` entry covers the window before a newly enabled cron's first slot). Cadence is qualitative here; the cron lives in each workflow's `on.schedule`.
|
||||
|
||||
Reference in New Issue
Block a user