Commit Graph

10 Commits

Author SHA1 Message Date
houseme ed45d1be2d fix(lock): recover stale distributed object guards (#3720)
fix(lock): reclaim expired stale object guards
2026-06-22 13:53:21 +08:00
houseme 29fbdc2dbf feat(ecstore): add object lock diagnostics (#3178)
* feat(ecstore): add object lock diagnostics

Add configurable namespace lock diagnostics for object operations so production contention can be traced by operation, owner, and key.

Wrap object read/write lock acquisition in diagnostic guards across get, head, put, delete, copy, and multipart flows, and log slow acquisition and long hold durations behind new RUSTFS_OBJECT_LOCK_DIAG_* settings.

Verification:

- make pre-commit

* feat(obs): expose object lock diagnostics metrics

Add Prometheus metrics and Grafana panels for object namespace lock diagnostics, covering slow acquire counts, slow hold counts, acquire duration, hold duration, and the diagnostics-enabled state.

Adopt PR review feedback by keeping diagnostic guards alive through the guarded operation so long-hold warnings and metrics are emitted, reusing shared env helpers, and reducing default-path overhead when diagnostics are disabled.

Verification:

- cargo check -p rustfs-ecstore -p rustfs-io-metrics

- cargo test -p rustfs-ecstore store::object -- --nocapture

- make pre-commit

* perf(obs): reduce object lock diag overhead

Avoid repeated environment parsing on hot object-lock paths by caching the diagnostics-enabled flag, and stop allocating label strings for object lock metrics by recording static labels directly.

Strengthen io-metrics tests by using a local recorder and asserting that the expected object lock diagnostic counters, gauges, and histograms are emitted.

Verification:

- cargo check -p rustfs-ecstore -p rustfs-io-metrics

- cargo test -p rustfs-io-metrics -- --nocapture

- make pre-commit
2026-06-03 02:10:27 +00:00
安正超 d9c683decc fix: retry namespace lock quorum contention (#3098)
* fix: retry namespace lock quorum contention

* fix(lock): treat remote lock rpc failures as hard

* fix(lock): classify lock contention timeout and avoid hard rollback blocks

* fix(lock): handle namespace lock quorum timeouts

* fix(lock): refine quorum contention handling

* fix(lock): refine unrecoverable quorum handling logs

* fix: address namespace lock quorum review feedback

* test: stabilize batch quorum timing tests

* fix(lock): classify remote quorum failures

* fix: drop remote lock timeout grace period

* fix: satisfy clippy on lock quorum error

* fix: classify remote lock rpc timeouts as hard failures

* fix: fast fail impossible lock quorum attempts

* fix: retry on transient timeout with pending cleanup

* fix: evict remote lock timeout connections

* refactor: avoid duplicate timeout eviction warning

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-05-28 01:35:00 +00:00
Henry Guo 0c52334480 fix(lock): retry transient distributed lock timeouts (#3101)
* fix(lock): retry transient distributed lock timeouts

* fix(lock): avoid retry during pending lock cleanup

* fix(lock): preserve acquire timeout budget

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-27 17:55:46 +00:00
weisd a0f1bb4ff0 fix(lock): prevent stale distributed object locks (#2633) 2026-04-22 02:12:33 +00:00
houseme 28edfd6190 fix(storage): harden offline drive fail-fast paths (#2564)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
2026-04-16 09:21:45 +00:00
weisd 25512e2635 perf(ecstore): batch delete object lock acquisition (#2374)
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-04-03 21:46:51 +08:00
weisd e86c6b726f fix(lock): split distributed read and write quorum (#2355) 2026-03-31 16:16:13 +08:00
weisd 05dc131a49 perf(storage): optimize internode RPC transfer path (#2262)
Co-authored-by: momoda693 <momoda693@gmail.com>
2026-03-23 17:09:49 +08:00
weisd dce117840c refactor: NamespaceLock (nslock), AHM→Heal Crate, and Lock/Clippy Fixes (#1664)
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: weisd <2057561+weisd@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-30 13:13:41 +08:00