f17ea7f146
fix(heal): harden replacement rebuild tracking ( #5892 )
...
* fix(heal): gate auto replacement formatting
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): require replacement target outcomes
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): bind resumes to replacement targets
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): fence healing marker ownership
Co-Authored-By: heihutu <heihutu@gmail.com >
* test(heal): cover replacement target completion
Co-Authored-By: heihutu <heihutu@gmail.com >
* docs(heal): clarify replacement recovery status
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): canonicalize replacement target checks
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): satisfy marker test module lint
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): scope automatic replacement format
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): require a mounted replacement target
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(ecstore): avoid cloned ref slice in test
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): revalidate replacement before scanning
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): reset stale resume checkpoints
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): release scanner disk map before probing
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): persist replacement intent before format
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): fail closed on mountinfo read errors
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): fence replacement target identity
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): order replacement completion cleanup
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): atomically seal replacement completion
Co-Authored-By: heihutu <heihutu@gmail.com >
* test(heal): census replacement target shards
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): fence replacement recovery ownership
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): preserve replacement recovery anchors
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): satisfy replacement recovery lint gates
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(ecstore): bind replacement identity to mount lease
Co-Authored-By: heihutu <heihutu@gmail.com >
* test(heal): cover durable replacement recovery states
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): validate persisted resume task identifiers
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(ecstore): avoid blocking replacement marker CAS
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): report failed marker rollback
Co-Authored-By: heihutu <heihutu@gmail.com >
* test(heal): pin replacement resume schema compatibility
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): preserve durable recovery anchors
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(ecstore): preserve public disk path semantics
Co-Authored-By: heihutu <heihutu@gmail.com >
* test(heal): use canonical replacement task ids
Co-Authored-By: heihutu <heihutu@gmail.com >
* test(heal): cover automatic replacement in 3x4 cluster
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): verify replacement target commits
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): persist replacement completion proof
Co-Authored-By: heihutu <heihutu@gmail.com >
* feat(heal): expose durable replacement status
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): bound durable replacement discovery
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): remove replacement readiness bypass
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): retry terminal replacement cleanup
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): isolate replacement intents from legacy resume
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): migrate legacy replacement intents at startup
Co-Authored-By: heihutu <heihutu@gmail.com >
* style(heal): apply strict clippy fix
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): prioritize active replacement recovery state
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): bind readiness to the admitted mount lease
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): atomically publish replacement intents
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): isolate replacement recovery directory
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): tolerate an empty recovery directory
Co-Authored-By: heihutu <heihutu@gmail.com >
* style(heal): remove redundant disk bytes conversion
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): reconcile proof-first replacement recovery
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): fence torn intent recovery
Co-Authored-By: heihutu <heihutu@gmail.com >
* test(heal): cover replacement migration conflicts
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(ecstore): fence replacement lease mount identity
Co-Authored-By: heihutu <heihutu@gmail.com >
* test(heal): cover missing replacement path admission
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): reject conflicting legacy completion proof
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(ecstore): fall back to proc mount identity
Co-Authored-By: heihutu <heihutu@gmail.com >
* feat(admin): expose replacement recovery status
Surface the local durable replacement recovery snapshot in the background heal status response so operators can tell whether replacement cleanup is definitive or still pending.
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): keep replacement status compatible
Keep the existing background heal status response wire-compatible while retaining the Linux mount lease cleanup needed for the replacement recovery branch.
Co-Authored-By: heihutu <heihutu@gmail.com >
* style(ecstore): match linux mount lease formatting
Keep Linux rustfmt output stable for the replacement mount lease comparison.
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(ecstore): qualify mount lease test constant
Use the disk module path for the format config constant in the Linux mount lease regression test.
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(ecstore): keep procfd mount roots directory-safe
Use a procfd path with an explicit directory component so Unix directory guards can open the replacement mount lease root with O_NOFOLLOW while preserving handle-relative I/O semantics.
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(ecstore): delete empty leased buckets via dirfd
Use the held mount lease fd as the parent for non-force empty bucket deletion on Linux so procfd-rooted paths do not get rejected as BucketNotEmpty. Also make the download-part OpenOptions truncate behavior explicit and keep fsync test recording stable across procfd canonicalization.
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(ecstore): scan leased bucket paths for emptiness
Use the local disk I/O root for bucket emptiness probes before non-force bucket deletion and table-bucket metadata checks. This keeps validation on the same mount instance as the subsequent local disk delete path.
Co-Authored-By: heihutu <heihutu@gmail.com >
* test(ecstore): align lease path test probes
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): block unsafe replacement recovery restarts
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): defer blocked replacement candidates
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): retry transient replacement discovery
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): keep transient recovery errors retryable
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): block corrupt legacy replacement state
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): classify flat replacement intent corruption
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): keep transient resume loads retryable
Classify malformed legacy replacement state as blocking corruption while preserving disk and transient load failures for retry. This avoids permanently blocking replacement recovery on temporary storage errors.
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): avoid latching transient legacy publishes
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): retry blocked legacy migrations
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(heal): defer blocked startup recoveries
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(ecstore): preserve disk sync limiter across lease roots
Co-Authored-By: heihutu <heihutu@gmail.com >
---------
Co-authored-by: heihutu <heihutu@gmail.com >
Co-authored-by: zhi22915 <qiuzgang@gmail.com >
2026-08-10 08:32:47 +08:00
47369ff027
fix(heal): defer scoped repair on suspended pools ( #5876 )
2026-08-09 11:50:17 +08:00
b4133d69e6
fix(heal): respect scoped object repair limits ( #5855 )
2026-08-09 07:23:16 +08:00
ab35681928
perf(ecstore): memoize bucket-incarnation fence validation under lifecycle read-lock coverage ( #5782 )
...
* perf(ecstore): memoize bucket-incarnation fence validation under lifecycle read-lock coverage
The PUT commit fence from #5648 validated the bucket incarnation with an uncached read (a distributed metadata-transaction read lock plus an EC quorum read of bucket metadata) on every PUT commit. Under 64-concurrency 4KiB PUT load this adds two quorum round-trips per PUT and the resulting lock-manager pressure produced ~1,000 client-visible 'Lock acquisition timeout' failures per 5-minute window (see rustfs/backlog#1776 ).
Memoize the validation per node while lifecycle read-lock coverage is continuous: bucket deletion/recreation requires the lifecycle WRITE lock, so while at least one read guard on this node has been held continuously the incarnation cannot have changed. The first fenced PUT in a coverage window performs the exact authoritative disk validation as before; overlapping PUTs reuse its result. The memo clears when the node's last guard drops or any guard observes a lost lock, so the next PUT revalidates from disk. Fence semantics are unchanged; only the redundant re-validations under continuous coverage are elided.
Also right-size the s3s footprint ratchet baselines: -1 s3_error! line from this change's error-path consolidation, and +1 s3s-importing file inherited from #5763 (crates/obs/src/telemetry/filter.rs) which landed on main without the baseline bump.
* fix(ecstore): carry the bucket fence registry through the rebalance test store
The rebalance entry test constructor landed on main after this branch was cut and needs the new field.
2026-08-07 15:05:48 +00:00
ec135f8c4c
fix(heal): bound per-object logging ( #5719 )
...
Co-authored-by: Anthony Martin <949506+anthonymartin@users.noreply.github.com >
2026-08-05 02:25:59 +00:00
67904a6c18
fix(ecstore): start with unresolved Kubernetes peers ( #5460 )
...
* fix(ecstore): start with unresolved Kubernetes peers
* fix(ecstore): infer Kubernetes endpoint identity safely
* fix(ecstore): fail closed on unsafe format migration
* fix(ecstore): reject poisoned format heal candidates
* fix(ecstore): reject unsafe legacy migration outliers
* fix(ecstore): resume interrupted format migrations
* fix(ecstore): preserve Kubernetes startup compatibility
2026-07-30 11:14:38 +08:00
b38976d5ee
refactor: segment ECStore storage contracts by domain ( #3910 )
2026-06-26 17:47:36 +08:00
c9614eb7cb
refactor: route ecstore storage api boundaries ( #3892 )
2026-06-26 09:35:18 +08:00
583a23bdf2
fix(ecstore): replace panic-driven pool and set stubs ( #3753 )
...
* fix(ecstore): replace panic-driven pool and set stubs
* test(runtime): tolerate restricted local bind checks
* fix(ecstore): remove remaining trait stub placeholders
* fix(ecstore): tighten trait stub follow-up semantics
* chore: ignore local worktrees
* chore: update layer dependency baseline for resolve_* context entries
Add 7 accepted infra->app dependency entries introduced by recent
refactoring PRs (#3770 , #3771 , #3772 ) that route global state lookups
through app::context::resolve_* functions.
Co-Authored-By: heihutu <heihutu@gmail.com >
---------
Co-authored-by: heihutu <heihutu@gmail.com >
2026-06-23 12:31:17 +08:00
57403525ee
refactor: move heal and namespace contracts ( #3560 )
2026-06-18 11:00:21 +08:00
82af181dcf
refactor(logging): unify governance runtime events ( #3367 )
2026-06-11 22:26:02 +08:00
095b77795d
refactor(ecstore): split store.rs into store submodules ( #1942 )
2026-02-25 06:35:27 +08:00
houseme