fix(lock): refresh object write locks with a heartbeat (#4388)

fix(lock): renew distributed locks via heartbeat; wire server refresh (backlog#899)

A held distributed write lock had a fixed 30s TTL and was never renewed, so
any operation exceeding it got its per-node lease reclaimed and stolen by a
contender, causing split-brain writes. This implements Phase 0 + Phase 1 of the
#899 design (Phase 2 abort-on-loss is deferred and tracked in code comments).

Phase 0 (server refresh wiring, P0):
- node_service handle_refresh was a no-op stub that parsed args then always
  returned success=true. Extract a testable `refresh_lock` free function that
  actually delegates to the node's lock backend. Not-found maps to
  success=false with no error_info, so RemoteClient::refresh keeps its
  Ok(resp.success) semantics and yields a real not_found signal to the
  coordinator heartbeat. Without this, client heartbeats were silently no-ops.

Phase 1 (client heartbeat + safe interval + observability):
- DistributedLockGuard spawns a heartbeat that refreshes every per-client lease
  on a derived interval; interval is derived without Duration::clamp
  (entries<=1 or interval>=ttl => no spawn), fixing the sub-second-ttl panic.
- Add LockLostSignal: declare the lock lost when not_found exceeds
  entries.len() - refresh_quorum; RPC errors are not counted (absorbed by the
  ttl > interval margin). Expose is_lock_lost()/lock_lost() for observers.
- disarm(), release(), and Drop now abort the heartbeat before releasing so no
  refresh races the unlock (refresh only extends, never creates, a lease).
- Reclaim path stays behaviorally unchanged but now warns with owner/resource/
  lease age and records a metric (#698 scavenger preserved).
- Add DEFAULT_LOCK_REFRESH_INTERVAL, LockRequest.refresh_interval (serde default
  for RPC back-compat) + builder, and lock lifecycle metrics.

Open questions adopt the design's documented defaults (marked TODO in code):
refresh not-found -> Ok(false)/error_info=None; lost-quorum base entries.len();
DEFAULT_LOCK_REFRESH_INTERVAL=10s.

Tests: heartbeat keepalive/quorum-loss/jitter/boundary/disarm (lock crate),
server refresh delegation (rustfs), and end-to-end survives-past-ttl plus
crashed-owner-reclaim regressions (namespace).

crates/lock/src/client/local.rs

@@ -42,15 +42,18 @@ struct LocalGuardEntry {
     guard: FastLockGuard,
     expires_at: SystemTime,
     ttl: Duration,
+    /// Owner recorded at acquire time; used only for reclaim diagnostics (#899).
+    owner: String,
 }
 
 impl LocalGuardEntry {
-    fn new(guard: FastLockGuard, ttl: Duration) -> Self {
+    fn new(guard: FastLockGuard, ttl: Duration, owner: String) -> Self {
         let now = SystemTime::now();
         Self {
             guard,
             expires_at: now + ttl,
             ttl,
+            owner,
         }
     }
 
@@ -136,6 +139,24 @@ impl LocalClient {
             };
 
             for mut entry in expired_entries {
+                // An expired entry whose owner never refreshed it (a dead coordinator, #698) is
+                // reclaimed so a live contender can re-form quorum. With guard heartbeats in place
+                // (#899) a live owner keeps its entry from expiring, so reaching here means the
+                // lease genuinely lapsed. Surface it for observability; the reclaim decision itself
+                // is unchanged.
+                let since_last_refresh = entry
+                    .expires_at
+                    .checked_sub(entry.ttl)
+                    .and_then(|last_refresh| SystemTime::now().duration_since(last_refresh).ok())
+                    .unwrap_or(entry.ttl);
+                tracing::warn!(
+                    owner = %entry.owner,
+                    resource = %resource,
+                    ttl_ms = entry.ttl.as_millis() as u64,
+                    since_last_refresh_ms = since_last_refresh.as_millis() as u64,
+                    "reclaiming expired lock guard whose lease was not refreshed"
+                );
+                rustfs_io_metrics::record_lock_reclaimed();
                 let _ = entry.guard.release();
                 reclaimed = reclaimed.saturating_add(1);
             }
@@ -175,7 +196,7 @@ impl LockClient for LocalClient {
                     {
                         let shard = self.get_shard(&lock_id);
                         let mut guards = shard.write().await;
-                        guards.insert(lock_id.clone(), LocalGuardEntry::new(guard, request.ttl));
+                        guards.insert(lock_id.clone(), LocalGuardEntry::new(guard, request.ttl, request.owner.clone()));
                     }
 
                     let lock_info = LockInfo {