mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-06 13:27:43 +00:00
Miguel Amador
acce8b2253
* fix(lock): let waiters hear releases and let acquisition succeed past registered waiters Same-key write contention scaled superlinearly with writer count: 8 concurrent conditional PUTs on one key cost ~340-460 ms, 16 cost ~700 ms, 32 cost ~5 s, against ~4 ms per uncontended write and ~10 ms actual lock holds (measured via RUSTFS_OBJECT_LOCK_DIAG at 1 ms thresholds). Outcomes were always correct; the cost was pure waiting. Two coupled defects in fast_lock caused it: 1. The slow path's early retries slept without subscribing to anything. notify_writer()/notify_readers() are gated on the waiter counters, which a sleeper never increments, so a release during the backoff woke nobody. The lock sat free while every loser slept out its full backoff, and the ladder compounded: successive acquires landed at the cumulative ladder offsets (10+20+40+80+100... ms). 2. try_acquire_exclusive demanded the entire packed state word be zero, including the readers_waiting/writers_waiting counter bits. A lock with registered waiters could be acquired by no one - including the waiters themselves, each blocked by the others' registration - so contended acquisition only succeeded in windows where every waiter happened to be unregistered. This is also why (1) could not be fixed by simply registering the sleepers: registration alone deadlocks acquisition until the acquire deadline. try_acquire_shared already masks correctly and preserves the counter bits in its CAS; the exclusive path now mirrors it. The fix: mask the acquisition CAS to ownership bits only (writer flag, active readers), and turn the early-retry sleep into a notification wait bounded by the same backoff, so a release wakes a waiter immediately while the bound still protects against lost or stolen wakeups exactly as NOTIFY_WAIT_CAP does for the post-retry wait. With both changes, 8 concurrent same-key CAS writers resolve in 17-29 ms (was 340-460 ms) and 32 resolve in 20-53 ms (was ~5 s), with per-racer cost now decreasing in N. Outcomes remain exactly one winner, N-1 precondition failures, zero errors at every width. cargo test -p rustfs-lock passes 113/113 at pristine-parity runtime, including test_concurrent_write_lock_contention, which previously only passed because sleepers were invisible to it. * test(lock): pin both halves of the waiter-starvation fix The fix commit touched only production files, so reverting either half left the suite green: test_concurrent_write_lock_contention only waits for five writers to finish and never asserts that acquisition happens before the backoff ladder runs out. Three tests, one per revert: * exclusive_acquisition_ignores_registered_waiters (state.rs) - a free lock with registered waiters must be acquirable, and the CAS must preserve the counters. Fails against the all-zero `expected`. * early_retry_registers_as_waiter (shard.rs) - a waiter in the early-retry backoff must appear in the writer waiter count within the ~750ms early-retry phase, since notify_writer/notify_readers are gated on those counters. Fails against a bare `sleep`, which registers nowhere. * contended_writers_drain_promptly_after_release (tests.rs) - 16 same-key writers, all registered behind one holder, must drain within 1s of the release rather than sit out their 5s acquire deadlines. Fails against the all-zero `expected` end to end. Wakeup latency is deliberately not asserted anywhere. NOTIFY_POOL is a process-global of 128 Notify slots shared by every lock, so a waiter in a concurrently-running test can consume another's notify_one and push it to the end of its rung: a 24-key latency probe measured ~150us in isolation and ~92ms - a full unexpired rung - alongside the existing 64-key missed-wakeup test. That is the stolen wakeup NOTIFY_WAIT_CAP already exists to bound, and it makes any in-suite latency budget flaky. cargo test -p rustfs-lock: 116/116. Signed-off-by: Miguel Amador <miguel@amador.one> --------- Signed-off-by: Miguel Amador <miguel@amador.one>
RustFS Lock - Distributed Locking
High-performance distributed locking system for RustFS object storage
📖 Documentation
· 🐛 Bug Reports
· 💬 Discussions
📖 Overview
RustFS Lock provides distributed locking capabilities for the RustFS distributed object storage system. For the complete RustFS experience, please visit the main RustFS repository.
✨ Features
- Distributed lock management across cluster nodes
- Read-write lock support with concurrent readers
- Lock timeout and automatic lease renewal
- Deadlock detection and prevention
- High-availability with leader election
- Performance-optimized locking algorithms
📚 Documentation
For comprehensive documentation, examples, and usage guides, please visit the main RustFS repository.
📄 License
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.