mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 16:28:15 +00:00
Zhengchao An
ddf197ba57
fix(lock): renew distributed locks via heartbeat; wire server refresh (backlog#899) A held distributed write lock had a fixed 30s TTL and was never renewed, so any operation exceeding it got its per-node lease reclaimed and stolen by a contender, causing split-brain writes. This implements Phase 0 + Phase 1 of the #899 design (Phase 2 abort-on-loss is deferred and tracked in code comments). Phase 0 (server refresh wiring, P0): - node_service handle_refresh was a no-op stub that parsed args then always returned success=true. Extract a testable `refresh_lock` free function that actually delegates to the node's lock backend. Not-found maps to success=false with no error_info, so RemoteClient::refresh keeps its Ok(resp.success) semantics and yields a real not_found signal to the coordinator heartbeat. Without this, client heartbeats were silently no-ops. Phase 1 (client heartbeat + safe interval + observability): - DistributedLockGuard spawns a heartbeat that refreshes every per-client lease on a derived interval; interval is derived without Duration::clamp (entries<=1 or interval>=ttl => no spawn), fixing the sub-second-ttl panic. - Add LockLostSignal: declare the lock lost when not_found exceeds entries.len() - refresh_quorum; RPC errors are not counted (absorbed by the ttl > interval margin). Expose is_lock_lost()/lock_lost() for observers. - disarm(), release(), and Drop now abort the heartbeat before releasing so no refresh races the unlock (refresh only extends, never creates, a lease). - Reclaim path stays behaviorally unchanged but now warns with owner/resource/ lease age and records a metric (#698 scavenger preserved). - Add DEFAULT_LOCK_REFRESH_INTERVAL, LockRequest.refresh_interval (serde default for RPC back-compat) + builder, and lock lifecycle metrics. Open questions adopt the design's documented defaults (marked TODO in code): refresh not-found -> Ok(false)/error_info=None; lost-quorum base entries.len(); DEFAULT_LOCK_REFRESH_INTERVAL=10s. Tests: heartbeat keepalive/quorum-loss/jitter/boundary/disarm (lock crate), server refresh delegation (rustfs), and end-to-end survives-past-ttl plus crashed-owner-reclaim regressions (namespace).
The file is empty.