fix(storage): harden ODM and scanner publication (#7187)

* fix(storage): harden ODM and scanner publication

* fix(app): simplify absent SSE configuration matching

* test(heal): settle PUT rename tails before disk-wipe fixtures

* fix(ecstore): remove duplicate local rename implementation

Keep the canonical commit module after concurrent storage changes merged.
The control-write and rollback changes are already present there.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ci): satisfy new clippy lints

* style(scanner): order merged test imports

* fix(scanner): invalidate bucket work after namespace completion

* fix(scanner): fence cached snapshots by scan execution

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
This commit is contained in:
Zhengchao An
2026-09-05 21:47:12 +08:00
committed by GitHub
parent 447f3c704b
commit e2a921bc16
35 changed files with 2252 additions and 274 deletions
+47 -1
View File
@@ -23,6 +23,30 @@ therefore has three identities:
If any identity changes before commit, the result is a candidate for retry or
observation, not an authoritative baseline.
Ordinary PUT rename fanouts also track instance-scoped in-flight work. A quorum
ACK does not release it: the actual disk tasks retain ownership until their
rename work ends, including when the request caller is cancelled. Scan admission
remains movement-only so sustained PUTs do not stop namespace walks and
scanner-driven lifecycle discovery. The post-walk local publication check and
remote publication leases reject pending fanouts. Begin/end namespace generations
invalidate scans and cached plans across the fanout; after acquiring remote
leases, the coordinator rechecks the full activity digest before publishing an
authoritative aggregate. This catches a tail that finishes between the scan's
last probe and lease acquisition.
This adds no namespace or movement lock. An already-verified older snapshot may
still precede a newly started write. Sustained or stalled PUT tails can delay
authoritative usage publication, which resumes through the existing retry
schedule rather than a new immediate-wakeup protocol. Intermediate per-set and
prefix cache readers retain their existing approximate-cache semantics. A
prolonged pending tail with no generation changes can also delay cycle advancement
and fresh rescans of already-current caches; this is not a guarantee of lifecycle
progress under indefinitely stalled storage I/O.
This PUT-tail protection requires every writer node to be upgraded. It does not
prove that a failed tail replica has healed, and it does not extend the same
in-flight tracking to multipart or other namespace mutation paths.
## Fences
The protocol uses separate fences because they exclude different stale inputs.
@@ -33,7 +57,7 @@ They must not be collapsed unless the replacement proves the same exclusions.
| Scanner leadership claim | scanner | competing scanner leaders and stale cycle writers |
| Storage publication epoch | ECStore | usage computed across rebalance, decommission, or other data-movement generations |
| Publication lease | scanner peers through ECStore-facing activity probes | remote dirty-usage or maintenance state that has not acknowledged the candidate |
| CAS revision | backing config object store | lost updates to `.usage.v2.json`, `.usage.json`, or cycle-state objects |
| CAS revision | backing config object store | lost updates to usage snapshots, scanner caches, or cycle-state objects |
| Per-set freshness | scanner aggregation | a merged usage snapshot that combines stale and current set results |
| Tier registry generation | scanner tier accounting | bytes classified against a different warm-tier registry |
| Usage floor identity | scanner publication and ECStore quota fallback | empty or legacy values becoming plausible authoritative quota input |
@@ -42,6 +66,28 @@ A reader that cannot prove the required fence for its surface must fail closed
or use the documented observed path below. It must not synthesize an empty usage
snapshot for a missing or corrupt authoritative object.
## Cache Execution Identity
The structural scan-plan digest can remain stable across ordinary bucket writes
so a scoped scan can retain unaffected baseline buckets. It is not sufficient
proof for reusing a completed result within the same cycle. Bucket work uses an
execution digest combining the structural plan and the full activity snapshot,
with the bucket's dirty generation included in its cache identity. Completed set
caches carry the same execution digest separately from their structural plan.
The persisted set-root fast path requires equal execution identities as well as
the existing source, cycle, leader, tier, and cache-structure checks.
A set scan also captures its starting cache revisions. When the persisted
execution differs, replacement requires those revisions to remain unchanged;
otherwise a slow scan could overwrite a newer completed result. The existing
cache lock, conditional save, and movement admission still fence the commit.
The optional `scan_execution_digest` field is appended to the map-encoded cache
metadata. Legacy caches remain readable but cannot satisfy same-cycle set-root
reuse without this identity. Older readers can ignore the added map key, but
older writers do not enforce its fence; readability is not a mixed-version
publication-safety guarantee.
## Persisted Objects
The persisted objects are part of the compatibility contract. Removing one