mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-09 22:59:59 +00:00
Zhengchao An
ddf197ba57
fix(lock): renew distributed locks via heartbeat; wire server refresh (backlog#899) A held distributed write lock had a fixed 30s TTL and was never renewed, so any operation exceeding it got its per-node lease reclaimed and stolen by a contender, causing split-brain writes. This implements Phase 0 + Phase 1 of the #899 design (Phase 2 abort-on-loss is deferred and tracked in code comments). Phase 0 (server refresh wiring, P0): - node_service handle_refresh was a no-op stub that parsed args then always returned success=true. Extract a testable `refresh_lock` free function that actually delegates to the node's lock backend. Not-found maps to success=false with no error_info, so RemoteClient::refresh keeps its Ok(resp.success) semantics and yields a real not_found signal to the coordinator heartbeat. Without this, client heartbeats were silently no-ops. Phase 1 (client heartbeat + safe interval + observability): - DistributedLockGuard spawns a heartbeat that refreshes every per-client lease on a derived interval; interval is derived without Duration::clamp (entries<=1 or interval>=ttl => no spawn), fixing the sub-second-ttl panic. - Add LockLostSignal: declare the lock lost when not_found exceeds entries.len() - refresh_quorum; RPC errors are not counted (absorbed by the ttl > interval margin). Expose is_lock_lost()/lock_lost() for observers. - disarm(), release(), and Drop now abort the heartbeat before releasing so no refresh races the unlock (refresh only extends, never creates, a lease). - Reclaim path stays behaviorally unchanged but now warns with owner/resource/ lease age and records a metric (#698 scavenger preserved). - Add DEFAULT_LOCK_REFRESH_INTERVAL, LockRequest.refresh_interval (serde default for RPC back-compat) + builder, and lock lifecycle metrics. Open questions adopt the design's documented defaults (marked TODO in code): refresh not-found -> Ok(false)/error_info=None; lost-quorum base entries.len(); DEFAULT_LOCK_REFRESH_INTERVAL=10s. Tests: heartbeat keepalive/quorum-loss/jitter/boundary/disarm (lock crate), server refresh delegation (rustfs), and end-to-end survives-past-ttl plus crashed-owner-reclaim regressions (namespace).
76 lines
3.0 KiB
Rust
76 lines
3.0 KiB
Rust
// Copyright 2024 RustFS Team
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
//! Fast Object Lock System
|
|
//!
|
|
//! High-performance versioned object locking system optimized for object storage scenarios
|
|
//!
|
|
//! ## Core Features
|
|
//!
|
|
//! 1. **Sharded Architecture** - Hash-based object key sharding to avoid global lock contention
|
|
//! 2. **Version Awareness** - Support for multi-version object locking with fine-grained control
|
|
//! 3. **Fast Path** - Lock-free fast paths for common operations
|
|
//! 4. **Async Optimized** - True async locks that avoid thread blocking
|
|
//! 5. **Auto Cleanup** - Access-time based automatic lock reclamation
|
|
|
|
pub mod disabled_manager;
|
|
pub mod guard;
|
|
pub mod manager;
|
|
pub mod manager_trait;
|
|
pub mod metrics;
|
|
pub mod object_pool;
|
|
pub mod optimized_notify;
|
|
pub mod shard;
|
|
pub mod state;
|
|
pub mod types;
|
|
|
|
#[cfg(test)]
|
|
mod tests;
|
|
|
|
// Re-export main types
|
|
pub use disabled_manager::DisabledLockManager;
|
|
pub use guard::FastLockGuard;
|
|
pub use manager::FastObjectLockManager;
|
|
pub use manager_trait::LockManager;
|
|
use std::time::Duration;
|
|
pub use types::*;
|
|
|
|
/// Maximum acquire timeout in seconds (for slow storage / high contention; override via env)
|
|
pub(crate) const DEFAULT_RUSTFS_MAX_ACQUIRE_TIMEOUT: u64 = 60;
|
|
|
|
/// Default acquire timeout in seconds (how long to wait for a lock before giving up)
|
|
pub(crate) const DEFAULT_RUSTFS_ACQUIRE_TIMEOUT: u64 = 10;
|
|
|
|
/// Default shard count (must be power of 2)
|
|
pub const DEFAULT_SHARD_COUNT: usize = 1024;
|
|
|
|
/// Default lock timeout (lease TTL; lock is released if not refreshed within this duration)
|
|
pub const DEFAULT_LOCK_TIMEOUT: Duration = Duration::from_secs(30);
|
|
|
|
/// Default heartbeat interval for refreshing a held distributed lock (lease renewal cadence).
|
|
///
|
|
/// Kept at ttl/3 of `DEFAULT_LOCK_TIMEOUT` so a lock tolerates losing two consecutive refresh
|
|
/// cycles before its lease expires. TODO(backlog#899): maintainers to confirm this default vs
|
|
/// MinIO's 10s/20s interval-vs-validity and the #814 latency budget.
|
|
pub const DEFAULT_LOCK_REFRESH_INTERVAL: Duration = Duration::from_secs(10);
|
|
|
|
/// Default acquire timeout - common value for local/low-latency; use env to increase for slow storage
|
|
pub const DEFAULT_ACQUIRE_TIMEOUT: Duration = Duration::from_secs(DEFAULT_RUSTFS_ACQUIRE_TIMEOUT);
|
|
|
|
/// Maximum acquire timeout for high-load scenarios
|
|
pub const MAX_ACQUIRE_TIMEOUT: Duration = Duration::from_secs(DEFAULT_RUSTFS_MAX_ACQUIRE_TIMEOUT);
|
|
|
|
/// Lock cleanup interval
|
|
pub const CLEANUP_INTERVAL: Duration = Duration::from_secs(60);
|