feat(object-data-cache): close write-side invalidation gaps and add an admin surface (#4694)

* feat(object-data-cache): close write/delete-side invalidation gaps

The object data cache exposed only a single per-(bucket,object)
invalidation primitive and no write-side ecstore hook, so several
delete paths left dead bodies resident until TTL (hygiene/capacity, not
stale-serving: lookups follow a fresh metadata quorum and cannot serve a
gone object). This adds the missing primitives and wires them in.

ODC-26 (backlog#1131): add an `ObjectMutationHook` trait beside the GET
body hook, registered next to it at startup, and call it from the
ecstore-internal delete paths (`apply_expiry_on_non_transitioned_objects`,
`expire_transitioned_object` including the restored-copy branch, and
`delete_object_versions`). The app impl is one `invalidate_object` call
under a new `AfterLifecycleExpiry` reason.

ODC-27 (backlog#1132): force prefix delete now invalidates the whole
prefix, not just the prefix string. `store.delete_object(delete_prefix)`
returns no deleted-name list, so this uses a new prefix primitive rather
than the batch path.

ODC-28 (backlog#1133): DeleteBucket now flushes the bucket via a new
bucket-scope primitive (covers force and non-force, which share the
delete_bucket call).

ODC-C2 (backlog#1143): add `ObjectDataCache::clear()` and two admin
handlers (GET stats, POST flush) routed through admin runtime_sources.

The starshard identity index gains a single `remove_matching` full-scan
API backing prefix/bucket/clear; it is documented as admin/delete-path
only and never runs on the GET or fill hot path. New invalidation
reasons and metric labels added; outcome (removed/noop) labelling kept
correct for every new primitive.

Also fixes a pre-existing broken intra-doc link in memory.rs.

Co-Authored-By: heihutu <heihutu@gmail.com>

* refactor(ecstore): extract the shared HookSlot behind both cache hooks

This PR introduced object_mutation_hook.rs by mirroring body_cache_hook.rs,
which left two process-global registration slots whose register/get/clear
bodies were line-for-line identical except the trait type and the WARN string:
a RwLock<Option<Arc<dyn _>>>, an Arc::ptr_eq "different instance" warning, the
poison-recovery closure, and the same read-lock-and-clone read. Two copies of
the same swap-vs-warn logic can drift apart under maintenance.

Hoist it into a generic HookSlot<T: ?Sized> that owns the logic once. Each hook
module keeps its `static HOOK: HookSlot<dyn XxxHook>` and its thin, unchanged
public wrappers (register_/get_/clear_), so the crate's public surface and
every call site are untouched — this is an internal consolidation, not a
contract change.

The load-bearing #1126 guarantee (newest registration wins, so a rebuilt
AppContext is never stranded on a first-wins slot) previously had no direct
test — the hook tests only covered register-then-notify. HookSlot now has its
own unit tests including re_registration_swaps_to_the_latest_instance;
mutation-testing confirms a first-wins regression fails exactly that test.

No behavior change: the two hooks' existing tests, the P0 body_cache_hook_e2e
regressions, and the app-layer mutation-hook tests all pass unchanged.

Refs: backlog#1126, backlog#1131

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(admin): register the object-data-cache routes in the policy inventory

This PR added GET /object-data-cache/stats and POST /object-data-cache/flush
but did not list them in the two registries that must account for every
admin route: the route-policy inventory (route_policy.rs) and the route
matrix (route_registration_test.rs). Their coverage tests —
route_policy_inventory_covers_registered_routes and
test_admin_route_matrix_matches_registered_routes — failed on CI because a
registered route had no policy/matrix entry.

These two tests are not part of `make pre-commit` (which runs fmt + arch +
quick-check, not the full suite), so the gap passed local pre-commit and
only surfaced in the CI Test-and-Lint lane.

stats is a read (ServerInfoAdminAction, Sensitive); flush mutates
(ConfigUpdateAdminAction, High) — matching the actions the handlers already
enforce. The MinIO-alias matrix test is unaffected: these are native rustfs
endpoints with no MinIO equivalent.

Refs: backlog#1143

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>

crates/ecstore/src/object_api/hook_slot.rs

@@ -0,0 +1,126 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! A process-wide, re-registrable slot for a single `Arc` hook.
+//!
+//! The GET body cache hook and the object mutation hook both need the same
+//! shape: one global slot that starts empty, is (re-)registered from the app
+//! layer at startup, is read on the hot/delete paths, and warns when a
+//! re-registration swaps in a *different* instance (an unexpected re-init that
+//! orphans the previous adapter's cache). This type holds that logic once so
+//! the two callers cannot drift apart.
+
+use std::sync::{Arc, RwLock};
+
+/// A global slot holding at most one `Arc<T>` hook.
+///
+/// Registration atomically swaps the stored hook. Poisoned locks recover in
+/// place: a panicked writer cannot leave an `Option<Arc<_>>` in an unsound
+/// state, so there is nothing to salvage.
+pub(crate) struct HookSlot<T: ?Sized> {
+    inner: RwLock<Option<Arc<T>>>,
+}
+
+impl<T: ?Sized> HookSlot<T> {
+    /// Creates an empty slot. `const` so it can initialize a `static`.
+    pub(crate) const fn new() -> Self {
+        Self {
+            inner: RwLock::new(None),
+        }
+    }
+
+    /// Registers (or re-registers) the hook.
+    ///
+    /// Replacing a *different* instance logs `warn_on_replace` at WARN: in
+    /// production a hook is installed exactly once per process, so a swap to a
+    /// distinct instance signals an unexpected re-init that leaves the previous
+    /// adapter's cache unreachable.
+    pub(crate) fn register(&self, hook: Arc<T>, warn_on_replace: &str) {
+        let mut slot = self.inner.write().unwrap_or_else(|poisoned| poisoned.into_inner());
+        if let Some(previous) = slot.as_ref()
+            && !Arc::ptr_eq(previous, &hook)
+        {
+            tracing::warn!("{warn_on_replace}");
+        }
+        *slot = Some(hook);
+    }
+
+    /// Returns the registered hook, if any.
+    pub(crate) fn get(&self) -> Option<Arc<T>> {
+        self.inner.read().unwrap_or_else(|poisoned| poisoned.into_inner()).clone()
+    }
+
+    /// Clears the slot. Test-only: production never unregisters a hook.
+    #[cfg(test)]
+    pub(crate) fn clear(&self) {
+        *self.inner.write().unwrap_or_else(|poisoned| poisoned.into_inner()) = None;
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::HookSlot;
+    use std::sync::Arc;
+
+    trait Marker: Send + Sync {
+        fn id(&self) -> u32;
+    }
+    struct Impl(u32);
+    impl Marker for Impl {
+        fn id(&self) -> u32 {
+            self.0
+        }
+    }
+
+    #[test]
+    fn empty_slot_reads_none() {
+        let slot: HookSlot<dyn Marker> = HookSlot::new();
+        assert!(slot.get().is_none());
+    }
+
+    #[test]
+    fn register_then_get_returns_the_hook() {
+        let slot: HookSlot<dyn Marker> = HookSlot::new();
+        slot.register(Arc::new(Impl(7)), "unused");
+        assert_eq!(slot.get().map(|h| h.id()), Some(7));
+    }
+
+    #[test]
+    fn re_registration_swaps_to_the_latest_instance() {
+        // The load-bearing #1126 guarantee: the newest registration wins, so a
+        // rebuilt AppContext leaves ecstore pointed at the current adapter
+        // rather than a stranded first-wins one.
+        let slot: HookSlot<dyn Marker> = HookSlot::new();
+        slot.register(Arc::new(Impl(1)), "unused");
+        slot.register(Arc::new(Impl(2)), "unused");
+        assert_eq!(slot.get().map(|h| h.id()), Some(2));
+    }
+
+    #[test]
+    fn re_registering_the_same_arc_is_idempotent() {
+        let slot: HookSlot<dyn Marker> = HookSlot::new();
+        let hook: Arc<dyn Marker> = Arc::new(Impl(9));
+        slot.register(Arc::clone(&hook), "unused");
+        slot.register(hook, "unused");
+        assert_eq!(slot.get().map(|h| h.id()), Some(9));
+    }
+
+    #[test]
+    fn clear_empties_the_slot() {
+        let slot: HookSlot<dyn Marker> = HookSlot::new();
+        slot.register(Arc::new(Impl(3)), "unused");
+        slot.clear();
+        assert!(slot.get().is_none());
+    }
+}