mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-09 14:49:25 +00:00
安正超
ada6f7587e
* refactor: flatten test harness storage compat aliases * refactor: flatten rustfs storage compat aliases (#3597) * refactor: prune runtime storage compat surface (#3598) * refactor: flatten runtime secondary storage compat (#3599) * docs: add scheduler placement profiling baselines (#3600) * feat: add observability topology capability contracts (#3601)
98 lines
4.2 KiB
Markdown
98 lines
4.2 KiB
Markdown
# Placement And Repair Invariants
|
|
|
|
This inventory covers `G-012` for `rustfs/backlog#666`. It records the current
|
|
object placement, readiness, lock quorum, scanner, and repair boundaries that
|
|
later scheduler or topology work must preserve.
|
|
|
|
## Object To Set Hash Rule
|
|
|
|
Objects reach a set through `Sets::get_disks_by_key`, which calls
|
|
`get_hashed_set_index` on the object key:
|
|
|
|
- `DistributionAlgoVersion::V1` uses `crc_hash(input, set_count)`.
|
|
- `DistributionAlgoVersion::V2` and `V3` use
|
|
`sip_hash(input, set_count, format_id_bytes)`.
|
|
- The format ID is part of the V2/V3 distribution seed, so changing the seed,
|
|
object key, set count, or algorithm changes placement.
|
|
|
|
Preservation rule: every object read, write, list, heal, repair, and
|
|
decommission path that resolves a set for an existing object must preserve the
|
|
same object key and format distribution algorithm.
|
|
|
|
## Pool, Set, And Disk Assignment Boundary
|
|
|
|
Pool selection is separate from set hashing:
|
|
|
|
- Existing objects are discovered across pools and resolved to the best current
|
|
pool candidate before reads or updates continue.
|
|
- New object writes select an available pool from current per-pool free-space
|
|
inputs after suspended or rebalancing pools are skipped.
|
|
- Set selection inside a pool still uses the object-to-set hash rule above.
|
|
- Disk index assignment comes from endpoint and format metadata, not from a
|
|
scheduler decision.
|
|
|
|
Boundary rule: schedulers may influence admission, worker concurrency, or
|
|
buffer sizing, but they must not rewrite pool, set, or disk indexes.
|
|
|
|
## Readiness And Lock Quorum Boundary
|
|
|
|
Runtime readiness currently checks storage and lock health independently:
|
|
|
|
- Storage readiness requires every observed set to meet write quorum based on
|
|
the set drive count and storage class data/parity shape.
|
|
- Lock readiness aggregates per-set lock-client host quorum and fails fast if
|
|
any set loses quorum.
|
|
- Object and bucket mutations acquire namespace locks through the existing
|
|
storage lock wrappers before changing object or bucket state.
|
|
|
|
Boundary rule: readiness and lock quorum must stay set-aware. A global healthy
|
|
disk count or global connected-host count is not sufficient when any individual
|
|
set is below quorum.
|
|
|
|
## Scanner Budget Preservation
|
|
|
|
Scanner cycles are bounded by `ScannerCycleBudget`:
|
|
|
|
- Runtime budget cancels the child token after the configured duration.
|
|
- Object budget cancels after the configured object count.
|
|
- Directory budget rejects additional directories and cancels with the
|
|
directories reason.
|
|
- Partial-cycle metrics and checkpoints use the budget reason.
|
|
|
|
Preservation rule: later scheduler work can change how scan cycles are admitted
|
|
only if it preserves the budget reason, checkpoint reason, and child-token
|
|
cancellation behavior.
|
|
|
|
## Heal Admission Preservation
|
|
|
|
Scanner and background repair work enter the heal manager through explicit
|
|
admission:
|
|
|
|
- Scanner object heal requests are low priority and may be accepted, merged,
|
|
rejected as full, or dropped.
|
|
- Required/high-priority heal candidates escalate on non-admission instead of
|
|
silently disappearing.
|
|
- Heal queue admission deduplicates queued and active work unless the request
|
|
explicitly forces admission.
|
|
- Full queues can drop low-priority work or displace lower-priority work for a
|
|
higher-priority request according to current manager rules.
|
|
|
|
Preservation rule: repair scheduling changes must keep admission outcomes
|
|
observable and must not convert rejected or dropped repair work into silent
|
|
success.
|
|
|
|
## Behavior Change Gates
|
|
|
|
Any later placement or repair PR must use the following gates:
|
|
|
|
- Placement gate: prove object-to-set hashing is unchanged for existing object
|
|
keys and format algorithms.
|
|
- Pool gate: prove pool selection does not choose suspended or rebalancing
|
|
pools unless the existing path already allows it.
|
|
- Quorum gate: prove storage readiness and lock readiness remain per-set.
|
|
- Scanner gate: prove scan budget reason and checkpoint mapping remain stable.
|
|
- Heal gate: prove low-priority scanner heal, forced heal, duplicate merge, and
|
|
queue-full outcomes remain distinct.
|
|
- Rollback gate: if a new scheduler sidecar is disabled, placement and repair
|
|
must fall back to the current direct ECStore/scanner/heal behavior.
|