mirror of
https://github.com/rustfs/rustfs.git
synced 2026-10-26 22:07:53 +00:00
唐小鸭
222beaba00
* fix(scanner): treat replication status writebacks as status-class hits
A replication status writeback (persist_replication_state_if_current in
Update mode, including the heals the scanner queues during its own walk)
marked the bucket dirty on the namespace path: it advanced the dirty-usage
generation, woke the scanner, reached the durable dirty journal and
entered the activity snapshot. Any replicated bucket therefore made every
scanner cycle Superseded, the authoritative usage baseline was never
certified and usage freshness stayed stale for as long as replication ran
(#8375).
The writeback is now status class: record_status_class_mutation records
only the process-local producer coverage bit and nothing else. There is
no rescan queue for it, because a deployment with an active replication
rule already walks every bucket on every cycle
(ScannerMaintenanceFeatures::needs_regular_cycle) and that walk refreshes
replication_stats. The routing lives in a named, unit-tested function in
rustfs/src/storage/storage_api.rs; tier-expiration cleanup removes data
and stays on the namespace path.
No persisted format, protocol, admin API or configuration changes.
This is phase 1 of docs/architecture/scanner-per-bucket-convergence.md,
which this change adds together with its index entry.
Refs #8375, #8494, rustfs/backlog#2816.
* fix(scanner): refresh replication statistics during maintenance
Refresh compacted prefixes and completed bucket caches during replication maintenance. Bind set execution reuse to the observed replication statistics so a same-cycle retry cannot restore older counters.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): avoid expanding the legacy S3 DTO surface
Decode the existing replication configuration type from a JSON fixture instead of adding direct S3 DTO references to a new file. Preserve the compacted-prefix and same-cycle assertions and the footprint ratchet.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): attribute activity guardrail vetoes per field
The post-walk cluster activity proof compared the whole activity snapshot,
so any namespace or dirty-usage generation movement vetoed publication.
Veto only on the fields that change which heap the walk described
(peer set, instance id, protocol version, topology, maintenance
generation, data movement, movement generation, publication blocked) and
name the field and host in the log and in a new usage-freshness counter.
Generation movement is a shade signal consumed per bucket. A remote peer
whose dirty-usage state moved keeps the whole-cycle outcome until scoped
acknowledgements can name its buckets.
* feat(data-usage): carry pending-bucket certification in DataUsageInfo
Add the additive fields a per-bucket certified scanner snapshot needs:
the sorted list and count of buckets whose dirty-usage generation moved
during the walk, and an explicit observational marker for snapshots whose
authoritative publication was vetoed. All three default when absent and
are skipped when empty, so older readers see the existing snapshot shape.
* feat(scanner): certify dirty usage per bucket and complete cycles with pending buckets
A structurally complete walk that the guardrail did not veto is now
`Complete` even when the dirty-usage generation moved during the walk.
Certification keeps the compare-and-remove semantics of
clear_dirty_usage_buckets: buckets whose generation still equals the start
snapshot are cleared, buckets written during the walk stay in the dirty map
and are published as the pending set, so usage_snapshot_converged becomes
`pending == 0`. The authoritative object is written for such cycles; the
observed route is selected by the explicit observational mark that vetoed
cycles carry instead of by the convergence flag, and the root publication
proof is read for every authoritative save. The post-lease activity recheck
compares the guardrail fields, with the peer dirty-usage fallback, instead
of the full snapshot digest.
* feat(scanner): accept unconverged complete publications as the cache baseline
complete_scanner_cache_baseline_plan_digest required the authoritative
snapshot to be converged, so a site whose cycles always left a bucket
pending could never scope a rescan and every cycle was a full walk. The
authoritative object is now a structural baseline whenever it carries a
convergence marker and no observational mark; bucket-cache reuse is then
governed by certification, because pending buckets stay in the dirty map
and are selected for the rescan while certified buckets reuse their set
cache. Observed candidates keep requiring the unconverged marker, and
segment reuse evidence keeps requiring a converged baseline.
Add the single-disk cycle test: a bucket written during the walk is the
only pending bucket, the others are certified, the publication is
authoritative and unconverged, and the next cycle is scoped to that
bucket; a held PUT tail still supersedes onto the observed route.
* feat(admin): expose scanner guardrail vetoes in usage freshness
Carry the attributed guardrail veto counter, timestamp and <field>@<host>
reason from the scanner metrics report into the madmin scanner status so
an operator can tell a rebalance or a PUT tail from a broken probe. The
fields are additive with serde defaults and merge like the deferral
fields.
* docs(scanner): record per-bucket certification in the design and changelog
Amend the convergence design where the implementation deviates: pending
buckets carry their walked value, maintenance_generation stays a veto,
guardrail localisation and the remaining phase-2 items ship separately,
last_cycle_result keeps its success label, and the admin surface gains the
guardrail veto fields. Update the publication contract for the post-lease
guardrail recheck and the explicit observational routing mark.
* refactor(scanner): drop the unreachable failed-bucket input from the pending set
A failed bucket never reaches a Complete cycle, so the pending set only
needs the start snapshot. Also remove a redundant clone in the data-usage
round-trip test.
* test(scanner): name the activity field mutation table type
* test(scanner): fill the post-walk gate in merged replication fixtures
The replication maintenance fixture merged from main builds a
ScannerCycleRequest literal and predates the test-only post_walk_gate
field; give it the default so the test target compiles on the rebased
branch.
* ci(nextest): run the admin target-repair fixtures on the ecstore base stack
repair_transaction_rejects_a_bucket_recreated_after_target_validation
overflowed the 2 MiB libtest default twice in a row on Linux CI for this
branch. Measured by sweeping RUST_MIN_STACK with the ci profile on macOS
arm64: main (65f30bd5c) passes at 2032 KiB and overflows at 2016 KiB,
this branch passes at 2040 KiB and overflows at 2032 KiB; on both trees
the overflow is in the test's delete_bucket step on the real store, and
the futures on that path are small (the peak is nested poll depth, not
one large future). The module was given this stack in #7272 and removed
in #7280 to check that boxing (#7277) sufficed; with the default now
exhausted on main itself, give the fixtures the same 4 MiB base stack as
the other real-storage rustfs fixtures, in both profiles.
* test(scanner): fill the post-walk gate in the walkers-per-disk fixture
The merge from main brought the walkers_per_disk cycle fixture (#8539),
whose ScannerCycleRequest initializer predates the test-only
post_walk_gate field this branch adds; the scanner test target did not
compile. Fill the default so the merged head builds.
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>