fix(lock): refresh object write locks with a heartbeat (#4388)

fix(lock): renew distributed locks via heartbeat; wire server refresh (backlog#899)

A held distributed write lock had a fixed 30s TTL and was never renewed, so
any operation exceeding it got its per-node lease reclaimed and stolen by a
contender, causing split-brain writes. This implements Phase 0 + Phase 1 of the
#899 design (Phase 2 abort-on-loss is deferred and tracked in code comments).

Phase 0 (server refresh wiring, P0):
- node_service handle_refresh was a no-op stub that parsed args then always
  returned success=true. Extract a testable `refresh_lock` free function that
  actually delegates to the node's lock backend. Not-found maps to
  success=false with no error_info, so RemoteClient::refresh keeps its
  Ok(resp.success) semantics and yields a real not_found signal to the
  coordinator heartbeat. Without this, client heartbeats were silently no-ops.

Phase 1 (client heartbeat + safe interval + observability):
- DistributedLockGuard spawns a heartbeat that refreshes every per-client lease
  on a derived interval; interval is derived without Duration::clamp
  (entries<=1 or interval>=ttl => no spawn), fixing the sub-second-ttl panic.
- Add LockLostSignal: declare the lock lost when not_found exceeds
  entries.len() - refresh_quorum; RPC errors are not counted (absorbed by the
  ttl > interval margin). Expose is_lock_lost()/lock_lost() for observers.
- disarm(), release(), and Drop now abort the heartbeat before releasing so no
  refresh races the unlock (refresh only extends, never creates, a lease).
- Reclaim path stays behaviorally unchanged but now warns with owner/resource/
  lease age and records a metric (#698 scavenger preserved).
- Add DEFAULT_LOCK_REFRESH_INTERVAL, LockRequest.refresh_interval (serde default
  for RPC back-compat) + builder, and lock lifecycle metrics.

Open questions adopt the design's documented defaults (marked TODO in code):
refresh not-found -> Ok(false)/error_info=None; lost-quorum base entries.len();
DEFAULT_LOCK_REFRESH_INTERVAL=10s.

Tests: heartbeat keepalive/quorum-loss/jitter/boundary/disarm (lock crate),
server refresh delegation (rustfs), and end-to-end survives-past-ttl plus
crashed-owner-reclaim regressions (namespace).

crates/io-metrics/src/lib.rs

@@ -157,8 +157,9 @@ pub use lock_metrics::{
 };
 
 pub use process_lock_metrics::{
-    ProcessLockSnapshot, ProcessPlatformSnapshot, record_read_lock_held_acquire, record_read_lock_held_release,
-    record_write_lock_held_acquire, record_write_lock_held_release, snapshot_process_lock_counts,
+    ProcessLockEventSnapshot, ProcessLockSnapshot, ProcessPlatformSnapshot, record_lock_reclaimed,
+    record_lock_refresh_quorum_lost, record_read_lock_held_acquire, record_read_lock_held_release,
+    record_write_lock_held_acquire, record_write_lock_held_release, snapshot_process_lock_counts, snapshot_process_lock_events,
     snapshot_process_platform_stats,
 };
 pub use s3_api_metrics::{init_s3_metrics, record_s3_op};

crates/io-metrics/src/process_lock_metrics.rs

@@ -28,6 +28,10 @@ use std::process::Command;
 
 static READ_LOCKS_HELD: AtomicU64 = AtomicU64::new(0);
 static WRITE_LOCKS_HELD: AtomicU64 = AtomicU64::new(0);
+/// Monotonic count of expired lock guards forcibly reclaimed by a contender (#899 observability).
+static LOCKS_RECLAIMED: AtomicU64 = AtomicU64::new(0);
+/// Monotonic count of guard heartbeats that observed a refresh-quorum loss (#899 observability).
+static LOCK_REFRESH_QUORUM_LOST: AtomicU64 = AtomicU64::new(0);
 
 #[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
 pub struct ProcessLockSnapshot {
@@ -35,6 +39,13 @@ pub struct ProcessLockSnapshot {
     pub write_locks_held: u64,
 }
 
+/// Monotonic lock lifecycle counters (distinct from the held gauges above).
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct ProcessLockEventSnapshot {
+    pub locks_reclaimed: u64,
+    pub lock_refresh_quorum_lost: u64,
+}
+
 #[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
 pub struct ProcessPlatformSnapshot {
     pub io_rchar_bytes: Option<u64>,
@@ -66,6 +77,16 @@ pub fn record_write_lock_held_release() {
     let _ = WRITE_LOCKS_HELD.fetch_update(Ordering::Relaxed, Ordering::Relaxed, |value| Some(value.saturating_sub(1)));
 }
 
+#[inline(always)]
+pub fn record_lock_reclaimed() {
+    LOCKS_RECLAIMED.fetch_add(1, Ordering::Relaxed);
+}
+
+#[inline(always)]
+pub fn record_lock_refresh_quorum_lost() {
+    LOCK_REFRESH_QUORUM_LOST.fetch_add(1, Ordering::Relaxed);
+}
+
 #[inline(always)]
 pub fn snapshot_process_lock_counts() -> ProcessLockSnapshot {
     ProcessLockSnapshot {
@@ -74,6 +95,14 @@ pub fn snapshot_process_lock_counts() -> ProcessLockSnapshot {
     }
 }
 
+#[inline(always)]
+pub fn snapshot_process_lock_events() -> ProcessLockEventSnapshot {
+    ProcessLockEventSnapshot {
+        locks_reclaimed: LOCKS_RECLAIMED.load(Ordering::Relaxed),
+        lock_refresh_quorum_lost: LOCK_REFRESH_QUORUM_LOST.load(Ordering::Relaxed),
+    }
+}
+
 #[inline]
 pub fn snapshot_process_platform_stats() -> ProcessPlatformSnapshot {
     platform::snapshot()