mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-28 09:08:58 +00:00
feat(scanner): define replication boundary contract (#3630)
* feat(scanner): define replication boundary contract * fix(scanner): propagate replication boundary metrics * fix(scanner): preserve repair metadata on merge --------- Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> Co-authored-by: houseme <housemecn@gmail.com>
This commit is contained in:
@@ -335,7 +335,7 @@ Compare these fields between baseline and tuned runs:
|
||||
| `metrics.source_work` | Shows cumulative work found, queued, skipped, missed, executed, and failed by source. |
|
||||
| `metrics.current_cycle_source_work` | Shows which source is consuming the current scan cycle. |
|
||||
| `metrics.last_cycle_source_work` | Shows which source consumed the previous scan cycle. |
|
||||
| `metrics.replication_repair` | Splits scanner-discovered replication repair by source and kind, including bucket object, delete-marker, version-purge, existing-object repair, and site replication boundary states. |
|
||||
| `metrics.replication_repair` | Splits scanner-discovered replication repair by source, kind, scanner role, and execution owner, including bucket object, delete-marker, version-purge, existing-object repair, and site replication boundary states. |
|
||||
| `metrics.current_cycle_replication_repair` | Shows which replication repair kind is being discovered or admitted in the current cycle. |
|
||||
| `metrics.last_cycle_replication_repair` | Shows which replication repair kind consumed the previous cycle. |
|
||||
| `metrics.lifecycle_expiry.current_queued` | Shows scanner-driven expiry/delete work waiting in the expiry worker queue. |
|
||||
@@ -425,6 +425,13 @@ Treat these as failure signals:
|
||||
reduce backlog;
|
||||
- bucket metrics show zero usage after post-start uploads while dirty usage
|
||||
remains pending and `life_time_save_usage` does not advance;
|
||||
- `bucket_replication` missed work with `scanner_role=repair_admission` grows
|
||||
while replication worker queues or target failures are also growing; treat
|
||||
this as downstream replication pressure, not only scanner pacing pressure;
|
||||
- `site_replication` `active_resync` grows and is interpreted as scanner-owned
|
||||
repair execution; `scanner_role=boundary_signal` and
|
||||
`execution_owner=site_replication_runtime` mean active site resync remains
|
||||
owned by the site replication runtime and admin resync path;
|
||||
- heal or bitrot work moves from `queued` to `missed` after a scanner pacing
|
||||
change.
|
||||
|
||||
@@ -458,7 +465,7 @@ at least these runs:
|
||||
| Single-node post-start bucket metrics freshness | Empty data path startup, post-start bucket creation/upload, bucket metrics snapshots, `scanner-summary.csv` usage freshness columns, and evidence that `DataUsageInfo` save work occurred before accepting bucket usage metrics. |
|
||||
| Single-node erasure or multi-disk | Checkpoint movement, active path age, set/disk scan pressure, data usage freshness, and before/after scanner config. |
|
||||
| Distributed lifecycle backlog | `maintenance_control`, lifecycle expiry/transition queue fields, source work missed/failed counts, and by-host admin metrics. |
|
||||
| Distributed replication backlog | Bucket replication repair kind counters, site replication passive/active boundary counters, source work queued/skipped/missed counts, and by-host admin metrics. |
|
||||
| Distributed replication backlog | Bucket replication repair kind counters, `scanner_role`, `execution_owner`, site replication passive/active boundary counters, source work queued/skipped/missed counts, and by-host admin metrics. |
|
||||
| Heal or bitrot pressure | Background heal `healOperations` queued/active source and priority counts, scanner source work for heal/bitrot, and by-host admin metrics. |
|
||||
|
||||
The expected conclusion is MinIO-style scanner behavior at the operational
|
||||
|
||||
@@ -188,6 +188,8 @@ and `missed` counters used by `source_work`, plus:
|
||||
|---|---|
|
||||
| `source` | `bucket_replication` for bucket replication repair, or `site_replication` for site replication boundary signals. |
|
||||
| `kind` | Bucket repair kinds are `object`, `delete_marker`, `version_purge`, and `existing_object`. Site replication boundary kinds are `passive_requeue` and `active_resync`. |
|
||||
| `scanner_role` | `repair_admission` means scanner found work and attempted to admit it to a worker queue. `boundary_signal` means scanner is reporting state owned by another runtime. |
|
||||
| `execution_owner` | `bucket_replication_queue` for bucket replication repair execution, or `site_replication_runtime` for site replication resync execution. |
|
||||
|
||||
For bucket replication, `queued` means scanner-discovered repair was admitted
|
||||
to the replication queue, `missed` means the queue or worker path could not
|
||||
@@ -197,6 +199,20 @@ The site replication kinds keep passive scanner discovery separate from active
|
||||
resync. Scanner status may report site replication boundary counters, but the
|
||||
scanner should not be treated as the active site replication resync controller.
|
||||
|
||||
Use this boundary when interpreting replication pressure:
|
||||
|
||||
| Scenario | Scanner source | Repair kind | Scanner role | Execution owner | Operational meaning |
|
||||
|---|---|---|---|---|---|
|
||||
| Bucket object, delete-marker, version-purge, or existing-object repair found during a scan | `bucket_replication` | `object`, `delete_marker`, `version_purge`, `existing_object` | `repair_admission` | `bucket_replication_queue` | Scanner found bucket replication repair work and attempted to admit it to the replication queue. |
|
||||
| Peer-originated or passive site replication work is observed while scanning | `site_replication` | `passive_requeue` | `boundary_signal` | `site_replication_runtime` | Scanner is reporting a passive site-replication boundary signal; it is not taking ownership of active site resync. |
|
||||
| Admin-triggered or runtime-owned site resync activity is visible in scanner metrics | `site_replication` | `active_resync` | `boundary_signal` | `site_replication_runtime` | Treat this as a boundary/status signal owned by the site replication runtime, not as scanner-controlled repair execution. |
|
||||
|
||||
If `site_replication` counters grow while bucket replication counters stay
|
||||
flat, investigate site replication status and resync state before tuning
|
||||
scanner pacing. If `bucket_replication` `missed` grows, investigate the bucket
|
||||
replication worker queue or target health before changing scanner cycle
|
||||
settings.
|
||||
|
||||
## Reading Maintenance Control
|
||||
|
||||
`metrics.maintenance_control` derives a source-level control snapshot from
|
||||
|
||||
Reference in New Issue
Block a user