perf(get,heal): fix GET hot-path overhead and heal checkpoint scaling (#4237)

perf(get,heal): land verified fixes for backlog #800-#804

Five fixes from the GET-path performance audit and scanner/heal
completeness audit (rustfs/backlog#800..#804), each verified locally:

- backlog#800 (heal checkpoint O(N^2)): ResumeCheckpoint object sets are
  now HashSet (Vec::contains was O(n) per healed object; 1.5ms at n=1M
  vs 11ns measured), and per-object checkpoint/resume-state persistence
  is batched (1000 mutations / 5s) instead of rewriting the whole file
  per object. complete_page() prunes the sets at page boundaries so
  memory stays bounded; positions still persist unconditionally and
  legacy Vec-format checkpoints still deserialize.

- backlog#801 (DiskInfo.healing never set): erasure-set heal now writes
  a healing marker (.rustfs.sys/healing.bin) on the disks it rebuilds
  (endpoints plumbed via HealRequest/HealTask.heal_endpoints from the
  auto disk scanner) and clears it on success. LocalDisk::disk_info
  surfaces the marker, so scanner heal coordination, lock selection and
  admin/metrics healing counts see the rebuild.

- backlog#802 (cache probe after data read): new GetObjectBodyCacheHook
  in ecstore lets the app-layer object data cache serve the body inside
  get_object_reader, after metadata quorum resolution (etag known) but
  before the erasure shard read/decode. Previously a hit still paid the
  full disk read. Hook is None/no-op when the cache is disabled.

- backlog#803 (GET hot-path redundant work): ObjectInfo is cloned for
  event notification only when an event will actually be built (GET and
  HEAD paths; events are currently suppressed so the clone was pure
  waste); get_opts/put_opts/del_opts resolve bucket versioning with one
  metadata-sys lookup instead of two; skip_verify_bitrot and
  get_lock_acquire_timeout env reads are cached via OnceLock; the
  io-priority metric is no longer double-counted; GetObject input fields
  are cloned selectively instead of cloning the whole input.

- backlog#804 (disk permit starvation): the disk-read permit wait is now
  bounded (RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT, default 5s, 0 =
  previous unbounded behavior); on timeout the GET proceeds without a
  permit and the bypass is counted. DiskReadPermitReader also releases
  the permit at body EOF instead of holding it until the client drops
  the stream.

Verification: make pre-commit; cargo clippy -D warnings on the four
changed crates; full rustfs lib suite (2096 tests) green; rustfs-heal
lib suite green with new unit tests for checkpoint pruning/legacy
format/throttle, permit EOF release, and the cache hook (hit + SSE
skip). The heal_integration_test and one set_disk listing test fail
identically on unmodified main (pre-existing global-state ordering
flakes, verified via git stash A/B).

Co-authored-by: houseme <housemecn@gmail.com>

crates/ecstore/src/set_disk/mod.rs

@@ -27,7 +27,7 @@ use crate::client::{object_api_utils::get_raw_etag, transition_api::ReaderImpl};
 use crate::cluster::rpc::heal_bucket_local_on_disks;
 use crate::data_usage::record_compression_total_memory;
 use crate::diagnostics::get::{
-    GET_CODEC_STREAMING_OBJECT_CLASS_PLAIN_SINGLE_PART, GET_OBJECT_PATH_CODEC_STREAMING,
+    GET_CODEC_STREAMING_OBJECT_CLASS_PLAIN_SINGLE_PART, GET_OBJECT_PATH_BODY_CACHE, GET_OBJECT_PATH_CODEC_STREAMING,
     GET_OBJECT_PATH_CODEC_STREAMING_LEGACY_ENGINE, GET_OBJECT_PATH_CODEC_STREAMING_RUSTFS_ENGINE, GET_OBJECT_PATH_DIRECT_MEMORY,
     GET_OBJECT_PATH_EMPTY, GET_OBJECT_PATH_INLINE_DIRECT, GET_OBJECT_PATH_LEGACY_DUPLEX, GET_OBJECT_PATH_REMOTE_TRANSITION,
     GET_OBJECT_PATH_SET_DISK, GET_STAGE_DECODE, GET_STAGE_EMIT, GET_STAGE_INLINE_PREPARE, GET_STAGE_LOCK_ACQUIRE,
@@ -51,6 +51,7 @@ use crate::error::{Error, Result, is_err_version_not_found};
 use crate::error::{GenericError, ObjectApiError, is_err_object_not_found};
 use crate::io_support::bitrot::{create_bitrot_reader, create_bitrot_reader_from_bytes, create_bitrot_writer};
 use crate::object_api::ObjectOptions;
+use crate::object_api::get_object_body_cache_hook;
 use crate::runtime::sources as runtime_sources;
 use crate::services::batch_processor::AsyncBatchProcessor;
 use crate::storage_api_contracts::{
@@ -453,11 +454,27 @@ mod write;
 
 /// Get lock acquire timeout from environment variable RUSTFS_LOCK_ACQUIRE_TIMEOUT (in seconds)
 /// Defaults to 30 seconds if not set or invalid
+/// Lock acquisition timeout. Cached: this is consulted on every object
+/// lock acquisition and `std::env::var` takes a process-global lock. In test
+/// builds the env var is read directly so `temp_env` overrides take effect.
 pub fn get_lock_acquire_timeout() -> Duration {
-    Duration::from_secs(rustfs_utils::get_env_u64(
-        rustfs_config::ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT,
-        rustfs_config::DEFAULT_OBJECT_LOCK_ACQUIRE_TIMEOUT,
-    ))
+    #[cfg(test)]
+    {
+        Duration::from_secs(rustfs_utils::get_env_u64(
+            rustfs_config::ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT,
+            rustfs_config::DEFAULT_OBJECT_LOCK_ACQUIRE_TIMEOUT,
+        ))
+    }
+    #[cfg(not(test))]
+    {
+        static CACHED: OnceLock<Duration> = OnceLock::new();
+        *CACHED.get_or_init(|| {
+            Duration::from_secs(rustfs_utils::get_env_u64(
+                rustfs_config::ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT,
+                rustfs_config::DEFAULT_OBJECT_LOCK_ACQUIRE_TIMEOUT,
+            ))
+        })
+    }
 }
 
 pub fn is_object_lock_diag_enabled() -> bool {
@@ -2332,6 +2349,27 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
             ));
         }
 
+        // App-layer object data cache probe: metadata (etag/size) is resolved
+        // but no data shards have been read yet, so a hit skips the erasure
+        // read, bitrot verify and decode entirely. The hook validates object
+        // identity and rejects anything it cannot serve byte-identically.
+        if range.is_none()
+            && opts.part_number.is_none()
+            && let Some(hook) = get_object_body_cache_hook()
+            && let Some(body) = hook.lookup(bucket, object, &object_info).await
+        {
+            record_get_object_reader_path_observation(GET_OBJECT_PATH_BODY_CACHE, object_class, size_bucket);
+            let reader = GetObjectReader {
+                stream: Box::new(Cursor::new(body.clone())),
+                object_info,
+                buffered_body: Some(body),
+            };
+            if lock_optimization_enabled {
+                release_materialized_read_lock(bucket, object, read_lock_guard.take());
+            }
+            return Ok(reader);
+        }
+
         if is_get_small_object_direct_memory_eligible(&range, &object_info, &fi, opts) {
             let object_size = usize::try_from(object_info.size)
                 .map_err(|_| to_object_err(Error::other("direct-memory GET object size is invalid"), vec![bucket, object]))?;