mirror of
https://github.com/rustfs/rustfs.git
synced 2026-11-08 19:43:05 +00:00
Hauser Zhi
4d46bd25f8
* perf(ecstore): probe bulk-delete write locks in parallel waves Bulk delete acquired one distributed write lock per key in serialized order, paying a full quorum round-trip per key. Probe the unique keys in bounded parallel waves with a short timeout instead; any contention drops the whole wave and the canonical serial order resolves it as before, so two overlapping batches can never deadlock. * perf(ecstore): probe bulk-delete transaction locks in parallel The set-layer delete-transaction locks were also acquired one quorum round-trip per key after the raw write locks. Apply the same optimistic all-or-nothing probe wave with a canonical serial fallback, and keep the per-key error reporting the fallback already provided. * perf(ecstore): pre-fetch bulk-delete source metadata in parallel The admission loop ran one metadata quorum fan-out per key in sequence on top of the per-key lock round-trips. Pre-fetch the source reads the loop needs with bounded parallelism under the locks already held, and derive the per-key read inputs from one shared helper so the pre-fetch and the loop cannot disagree. * test(ecstore): drop unused import in batch-delete probe test * refactor(ecstore,lock): await wave releases and unify probe waves Adversarial review of the bulk-delete lock waves found three real issues. Distributed guard release is fire-and-forget on drop, so the serial fallback could re-acquire keys while unlock RPCs were still in flight and block on locks it already owned: guards are now released with an awaited unlock before the fallback runs. The wave also probed every key after the first failure, adding up to one probe timeout per chunk of pure overhead under contention: probing now stops at the first failing chunk. Finally the two copy-pasted wave implementations are replaced by one shared probe_lock_wave, and the per-key delete read inputs are resolved once instead of being recomputed. Tests now assert the wave/fallback counters so they prove which path ran, and an overlapping-batch test pins the no-deadlock property. * refactor(ecstore,lock): batch bulk-delete lock rounds per key Replace the optimistic probe waves with the bulk lock primitive the lock layer already supports on the wire: one lock_batch RPC per lock client for a whole DeleteObjects key set, per-key quorum guards with the usual TTL heartbeat, and per-key outcomes. The batched round is deadlock-free by construction (every lock server orders concurrent batches by the same global (shard, key) order), so the probe timeout, the wave release-and-wait dance, and the canonical serial fallback all go away. Per-key lock outcomes are now reported per key end to end: a locked-out key fails alone in the response while the rest of the batch proceeds, matching the S3 DeleteObjects contract and the set layer's existing behavior. The set layer's raw-write-lock and delete-transaction-lock waves fold into the same primitive via one shared resource-folding helper, and the fast-lock batch fast path now counts acquisitions so shard metrics keep their meaning. * perf(ecstore): move journal publish off the one-worker namespace lane A bulk delete publishes one recovery transaction per key, and every journal inspection and settle-plan closure ran on the per-disk foreground namespace lane (one worker), serializing the whole publish phase per drive behind every other mutation. Route those closures onto the durable-mutation lane the fused delete prepare already uses (eight workers per disk, fsyncs coalesce in the filesystem journal), and keep 32 journal publications in flight so a 16-drive set can feed those lanes. * test(ecstore): flatten lock-error iteration in overlap test * fix(ecstore,lock): refresh duplicate keys after recovery, bound unlock Review findings on the batched lock rounds. A batch entry whose recovery rolls back an interrupted prepared delete can bring a retained version back; a later duplicate of the same key still held its pre-fetched source-missing read, skipped the object-lock check, and deleted the restored version. Recovery now evicts the pre-fetched reads of every other entry naming the same object, so those entries take the loop's live re-read and see the restored version. The batch rollback release awaited the full unlock retry schedule, including the deferred background pass, parking the request behind an unreachable peer. The synchronous phase is now a single release pass with the deferred retries spawned in the background. * fix(ecstore): release local batch locks before deferred recovery A set-layer batch delete that runs deferred key recovery re-acquires a write lock on the same keys through recover_delete_object, but the raw local batch guards were only ever released at function exit — the distributed path releases its batch ids before the deferred loop and the local path missed the mirror release, so the recovery deadlocked against the batch's own locks and surfaced as a spurious Lock(Timeout) for every deferred key. A regression test plants the interrupted prepared delete this needs and pins the end-to-end contract: a batch naming a GOVERNANCE-retained version twice must deny both entries after the first entry's recovery rolls the prepared delete back, and the restored version must survive. * test(lock): cover batch acquire partial-grant rollback The distributed acquire_guard_batch rollback path — a key granted by fewer than quorum clients must fail alone and release its partial grants before the call returns — was only reachable in distributed mode and had no coverage. A scripted per-object LockClient pins both halves: the below-quorum key reports QuorumNotReached with its partial vote, and the orphaned grant is released synchronously while the quorate key's guard stays held.
RustFS Lock - Distributed Locking
High-performance distributed locking system for RustFS object storage
📖 Documentation
· 🐛 Bug Reports
· 💬 Discussions
📖 Overview
RustFS Lock provides distributed locking capabilities for the RustFS distributed object storage system. For the complete RustFS experience, please visit the main RustFS repository.
✨ Features
- Distributed lock management across cluster nodes
- Read-write lock support with concurrent readers
- Lock timeout and automatic lease renewal
- Deadlock detection and prevention
- High-availability with leader election
- Performance-optimized locking algorithms
📚 Documentation
For comprehensive documentation, examples, and usage guides, please visit the main RustFS repository.
📄 License
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.