test(cache): end-to-end regressions for the body-cache hook P0s (#4675)

fix(ecstore): make body-cache hook re-registrable + add e2e regressions

ODC-21 (backlog#1126): the GET body-cache hook lived in a first-wins
OnceLock. When AppContext is rebuilt (config reload, test re-init) a fresh
ObjectDataCacheAdapter is constructed and re-registered, but the OnceLock
kept ecstore's GET probe pointed at adapter #1 while every usecase-layer
fill and invalidation targeted adapter #2 — silently degrading the feature
to a 0% hit rate with no error, log, or metric, and stranding entries in the
unreachable cache until their TTL.

Replace the slot with RwLock<Option<Arc<dyn GetObjectBodyCacheHook>>> so
re-registration atomically swaps to the newest adapter, and log at WARN when
a swap replaces a *different* instance (Arc::ptr_eq). RwLock over
ArcSwapOption because arc-swap's RefCnt is impl<T> (Sized, thin *mut T) and
cannot hold an Arc<dyn Trait> without a sized newtype wrapper; the probe
reads the slot once per full-object GET but only clones an Arc, negligible
next to the metadata quorum fan-out already done before the probe. Add a
test-only clear_get_object_body_cache_hook so tests register/unregister
deterministically.

With the hook now re-registrable, add true end-to-end regressions that drive
get_object_reader (not the full_object_plaintext_len predicate) against a
real erasure-coded, genuinely-compressed object via the blackbox
make_local_set_disks harness, with a stand-in hook playing the app-layer
cache (the injection point production uses; the adapter itself lives above
ecstore). These close the gap the predicate-only tests left — a caller that
opens a new shortcut serving the cached body directly, the original form of
both P0s:

- backlog#1108: a raw_data_movement_read must yield the STORED (compressed)
  bytes, never the cached plaintext.
- backlog#1109: a compressed cache hit must publish the DECOMPRESSED length
  as object_info.size (the UploadPartCopy invariant), with the streamed
  length matching.
- backlog#1146: a restore read (restore_request.days) must serve STORED
  bytes, not the cache.

Mutation-verified each e2e test bites: dropping the raw_data_movement_read
gate serves plaintext (fails #1108); removing the hit-site size republication
publishes 2972 vs 660000 (fails #1109); dropping the restore gate serves
plaintext (fails #1146).

Co-authored-by: heihutu <heihutu@gmail.com>

crates/ecstore/src/object_api/body_cache_hook.rs

@@ -22,7 +22,7 @@
 
 use crate::object_api::ObjectInfo;
 use bytes::Bytes;
-use std::sync::{Arc, OnceLock};
+use std::sync::{Arc, RwLock};
 
 /// Serves full-object GET bodies from a cache keyed by object identity.
 ///
@@ -35,15 +35,48 @@ pub trait GetObjectBodyCacheHook: Send + Sync + 'static {
     async fn lookup(&self, bucket: &str, object: &str, info: &ObjectInfo) -> Option<Bytes>;
 }
 
-static GET_OBJECT_BODY_CACHE_HOOK: OnceLock<Arc<dyn GetObjectBodyCacheHook>> = OnceLock::new();
+// `RwLock<Option<Arc<dyn ...>>>` rather than `ArcSwapOption`: arc-swap's
+// `RefCnt` is implemented only for the sized `Arc<T>` (it stores a thin
+// `*mut T`), so it cannot hold an `Arc<dyn GetObjectBodyCacheHook>` without a
+// sized newtype wrapper. The probe reads this slot once per full-object GET,
+// but the read guard only clones an `Arc`, which is negligible next to the
+// metadata quorum fan-out already completed before the probe. Registration is
+// a startup / config-reload event, so writer contention is a non-issue.
+static GET_OBJECT_BODY_CACHE_HOOK: RwLock<Option<Arc<dyn GetObjectBodyCacheHook>>> = RwLock::new(None);
 
-/// Register the process-wide GET body cache hook. First registration wins;
-/// later calls are ignored so tests and re-inits cannot swap the hook midway.
+/// Register (or re-register) the process-wide GET body cache hook.
+///
+/// Re-registration atomically swaps to `hook`, so a rebuilt `AppContext`
+/// (config reload, or the test re-init pattern) leaves ecstore's GET probe
+/// pointed at the newest adapter. A first-wins slot would instead pin the probe
+/// to the original adapter while every usecase-layer fill and invalidation
+/// targeted the replacement, silently degrading the feature to a 0% hit rate
+/// and stranding entries in the unreachable cache until their TTL (backlog#1126).
+///
+/// Replacing a *different* hook instance is logged at WARN: in production the
+/// hook is installed exactly once per process, so a swap to a distinct instance
+/// signals an unexpected re-init and orphans the previous adapter's cache.
 pub fn register_get_object_body_cache_hook(hook: Arc<dyn GetObjectBodyCacheHook>) {
-    let _ = GET_OBJECT_BODY_CACHE_HOOK.set(hook);
+    let mut slot = GET_OBJECT_BODY_CACHE_HOOK.write().unwrap_or_else(|e| e.into_inner());
+    if let Some(previous) = slot.as_ref()
+        && !Arc::ptr_eq(previous, &hook)
+    {
+        tracing::warn!(
+            "GET object body cache hook re-registered with a different instance; \
+             the previous adapter's cache is now unreachable by ecstore's GET probe"
+        );
+    }
+    *slot = Some(hook);
 }
 
 /// The registered hook, if any.
-pub(crate) fn get_object_body_cache_hook() -> Option<&'static Arc<dyn GetObjectBodyCacheHook>> {
-    GET_OBJECT_BODY_CACHE_HOOK.get()
+pub(crate) fn get_object_body_cache_hook() -> Option<Arc<dyn GetObjectBodyCacheHook>> {
+    GET_OBJECT_BODY_CACHE_HOOK.read().unwrap_or_else(|e| e.into_inner()).clone()
+}
+
+/// Test-only: unregister the hook so tests can register and clear the slot
+/// deterministically without leaking a hook into unrelated tests.
+#[cfg(test)]
+pub(crate) fn clear_get_object_body_cache_hook() {
+    *GET_OBJECT_BODY_CACHE_HOOK.write().unwrap_or_else(|e| e.into_inner()) = None;
 }