feat(object-data-cache): close write-side invalidation gaps and add an admin surface (#4694)

* feat(object-data-cache): close write/delete-side invalidation gaps

The object data cache exposed only a single per-(bucket,object)
invalidation primitive and no write-side ecstore hook, so several
delete paths left dead bodies resident until TTL (hygiene/capacity, not
stale-serving: lookups follow a fresh metadata quorum and cannot serve a
gone object). This adds the missing primitives and wires them in.

ODC-26 (backlog#1131): add an `ObjectMutationHook` trait beside the GET
body hook, registered next to it at startup, and call it from the
ecstore-internal delete paths (`apply_expiry_on_non_transitioned_objects`,
`expire_transitioned_object` including the restored-copy branch, and
`delete_object_versions`). The app impl is one `invalidate_object` call
under a new `AfterLifecycleExpiry` reason.

ODC-27 (backlog#1132): force prefix delete now invalidates the whole
prefix, not just the prefix string. `store.delete_object(delete_prefix)`
returns no deleted-name list, so this uses a new prefix primitive rather
than the batch path.

ODC-28 (backlog#1133): DeleteBucket now flushes the bucket via a new
bucket-scope primitive (covers force and non-force, which share the
delete_bucket call).

ODC-C2 (backlog#1143): add `ObjectDataCache::clear()` and two admin
handlers (GET stats, POST flush) routed through admin runtime_sources.

The starshard identity index gains a single `remove_matching` full-scan
API backing prefix/bucket/clear; it is documented as admin/delete-path
only and never runs on the GET or fill hot path. New invalidation
reasons and metric labels added; outcome (removed/noop) labelling kept
correct for every new primitive.

Also fixes a pre-existing broken intra-doc link in memory.rs.

Co-Authored-By: heihutu <heihutu@gmail.com>

* refactor(ecstore): extract the shared HookSlot behind both cache hooks

This PR introduced object_mutation_hook.rs by mirroring body_cache_hook.rs,
which left two process-global registration slots whose register/get/clear
bodies were line-for-line identical except the trait type and the WARN string:
a RwLock<Option<Arc<dyn _>>>, an Arc::ptr_eq "different instance" warning, the
poison-recovery closure, and the same read-lock-and-clone read. Two copies of
the same swap-vs-warn logic can drift apart under maintenance.

Hoist it into a generic HookSlot<T: ?Sized> that owns the logic once. Each hook
module keeps its `static HOOK: HookSlot<dyn XxxHook>` and its thin, unchanged
public wrappers (register_/get_/clear_), so the crate's public surface and
every call site are untouched — this is an internal consolidation, not a
contract change.

The load-bearing #1126 guarantee (newest registration wins, so a rebuilt
AppContext is never stranded on a first-wins slot) previously had no direct
test — the hook tests only covered register-then-notify. HookSlot now has its
own unit tests including re_registration_swaps_to_the_latest_instance;
mutation-testing confirms a first-wins regression fails exactly that test.

No behavior change: the two hooks' existing tests, the P0 body_cache_hook_e2e
regressions, and the app-layer mutation-hook tests all pass unchanged.

Refs: backlog#1126, backlog#1131

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(admin): register the object-data-cache routes in the policy inventory

This PR added GET /object-data-cache/stats and POST /object-data-cache/flush
but did not list them in the two registries that must account for every
admin route: the route-policy inventory (route_policy.rs) and the route
matrix (route_registration_test.rs). Their coverage tests —
route_policy_inventory_covers_registered_routes and
test_admin_route_matrix_matches_registered_routes — failed on CI because a
registered route had no policy/matrix entry.

These two tests are not part of `make pre-commit` (which runs fmt + arch +
quick-check, not the full suite), so the gap passed local pre-commit and
only surfaced in the CI Test-and-Lint lane.

stats is a read (ServerInfoAdminAction, Sensitive); flush mutates
(ConfigUpdateAdminAction, High) — matching the actions the handlers already
enforce. The MinIO-alias matrix test is unaffected: these are native rustfs
endpoints with no MinIO equivalent.

Refs: backlog#1143

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>

crates/object-data-cache/src/starshard_index.rs

@@ -98,6 +98,42 @@ impl StarshardIdentityIndex {
             .map_or_else(Vec::new, |set| set.cloned())
     }
 
+    /// Removes every tracked identity whose key matches `predicate`, returning
+    /// all keys that were dropped so the caller can evict them from the cache.
+    ///
+    /// This performs a **full shard scan** (`keys()` snapshots every identity,
+    /// then each match is removed under its shard write lock). It is therefore
+    /// restricted to the rare prefix-delete, bucket-delete, and admin
+    /// `clear()`/flush paths — it must never run on the GET or fill hot path.
+    /// A `true`-returning predicate clears the whole index (used by `clear()`).
+    ///
+    /// The snapshot/remove split leaves a small window: an identity inserted
+    /// after the snapshot but before removal is not visited, and a key added to
+    /// a matched identity between snapshot and its `remove` is still dropped
+    /// from the index (via `remove`) but returned for cache eviction, so no
+    /// tracked body is stranded. This is acceptable hygiene slack on these rare
+    /// paths (backlog#1132/#1133/#1143).
+    pub async fn remove_matching<F>(&self, predicate: F) -> Vec<ObjectDataCacheKey>
+    where
+        F: Fn(&ObjectDataCacheIdentity) -> bool,
+    {
+        let identities: Vec<ObjectDataCacheIdentity> = self
+            .by_object
+            .keys()
+            .await
+            .into_iter()
+            .filter(|identity| predicate(identity))
+            .collect();
+
+        let mut removed = Vec::new();
+        for identity in identities {
+            if let Some(key_set) = self.by_object.remove(&identity).await {
+                removed.extend(key_set.cloned());
+            }
+        }
+        removed
+    }
+
     /// Removes a single evicted key tracked under an identity, but only when the
     /// evicted entry's generation token still matches the tracked one.
     ///
@@ -259,6 +295,53 @@ mod tests {
         assert_eq!(invalidated, vec![key_a]);
     }
 
+    fn bucketed_key(bucket: &str, object: &str) -> ObjectDataCacheKey {
+        ObjectDataCacheKey::new(bucket, object, Some("v1"), "etag", 1, ObjectDataCacheBodyVariant::FullObjectPlainV1)
+    }
+
+    #[tokio::test]
+    async fn remove_matching_returns_only_predicate_matches() {
+        let index = StarshardIdentityIndex::new(4);
+        let id_photos = ObjectDataCacheIdentity::new("bucket", "photos/a.jpg");
+        let id_photos2 = ObjectDataCacheIdentity::new("bucket", "photos/b.jpg");
+        let id_videos = ObjectDataCacheIdentity::new("bucket", "videos/c.mp4");
+
+        let _ = index
+            .insert(id_photos.clone(), bucketed_key("bucket", "photos/a.jpg"), 1)
+            .await;
+        let _ = index
+            .insert(id_photos2.clone(), bucketed_key("bucket", "photos/b.jpg"), 2)
+            .await;
+        let _ = index
+            .insert(id_videos.clone(), bucketed_key("bucket", "videos/c.mp4"), 3)
+            .await;
+
+        let removed = index
+            .remove_matching(|identity| identity.bucket.as_ref() == "bucket" && identity.object.starts_with("photos/"))
+            .await;
+
+        assert_eq!(removed.len(), 2, "only the two photos/ identities match the prefix");
+        // The unmatched identity is still tracked; the matched ones are gone.
+        assert!(index.remove_identity(&id_videos).await.len() == 1);
+        assert!(index.remove_identity(&id_photos).await.is_empty());
+    }
+
+    #[tokio::test]
+    async fn remove_matching_true_predicate_clears_index() {
+        let index = StarshardIdentityIndex::new(4);
+        let _ = index
+            .insert(ObjectDataCacheIdentity::new("b1", "o1"), bucketed_key("b1", "o1"), 1)
+            .await;
+        let _ = index
+            .insert(ObjectDataCacheIdentity::new("b2", "o2"), bucketed_key("b2", "o2"), 2)
+            .await;
+
+        let removed = index.remove_matching(|_| true).await;
+
+        assert_eq!(removed.len(), 2);
+        assert_eq!(index.identity_count().await, 0, "a true predicate clears every identity");
+    }
+
     #[tokio::test]
     async fn identity_index_matching_eviction_removes_key() {
         let index = StarshardIdentityIndex::new(4);