feat(object-data-cache): close write-side invalidation gaps and add an admin surface (#4694)

* feat(object-data-cache): close write/delete-side invalidation gaps

The object data cache exposed only a single per-(bucket,object)
invalidation primitive and no write-side ecstore hook, so several
delete paths left dead bodies resident until TTL (hygiene/capacity, not
stale-serving: lookups follow a fresh metadata quorum and cannot serve a
gone object). This adds the missing primitives and wires them in.

ODC-26 (backlog#1131): add an `ObjectMutationHook` trait beside the GET
body hook, registered next to it at startup, and call it from the
ecstore-internal delete paths (`apply_expiry_on_non_transitioned_objects`,
`expire_transitioned_object` including the restored-copy branch, and
`delete_object_versions`). The app impl is one `invalidate_object` call
under a new `AfterLifecycleExpiry` reason.

ODC-27 (backlog#1132): force prefix delete now invalidates the whole
prefix, not just the prefix string. `store.delete_object(delete_prefix)`
returns no deleted-name list, so this uses a new prefix primitive rather
than the batch path.

ODC-28 (backlog#1133): DeleteBucket now flushes the bucket via a new
bucket-scope primitive (covers force and non-force, which share the
delete_bucket call).

ODC-C2 (backlog#1143): add `ObjectDataCache::clear()` and two admin
handlers (GET stats, POST flush) routed through admin runtime_sources.

The starshard identity index gains a single `remove_matching` full-scan
API backing prefix/bucket/clear; it is documented as admin/delete-path
only and never runs on the GET or fill hot path. New invalidation
reasons and metric labels added; outcome (removed/noop) labelling kept
correct for every new primitive.

Also fixes a pre-existing broken intra-doc link in memory.rs.

Co-Authored-By: heihutu <heihutu@gmail.com>

* refactor(ecstore): extract the shared HookSlot behind both cache hooks

This PR introduced object_mutation_hook.rs by mirroring body_cache_hook.rs,
which left two process-global registration slots whose register/get/clear
bodies were line-for-line identical except the trait type and the WARN string:
a RwLock<Option<Arc<dyn _>>>, an Arc::ptr_eq "different instance" warning, the
poison-recovery closure, and the same read-lock-and-clone read. Two copies of
the same swap-vs-warn logic can drift apart under maintenance.

Hoist it into a generic HookSlot<T: ?Sized> that owns the logic once. Each hook
module keeps its `static HOOK: HookSlot<dyn XxxHook>` and its thin, unchanged
public wrappers (register_/get_/clear_), so the crate's public surface and
every call site are untouched — this is an internal consolidation, not a
contract change.

The load-bearing #1126 guarantee (newest registration wins, so a rebuilt
AppContext is never stranded on a first-wins slot) previously had no direct
test — the hook tests only covered register-then-notify. HookSlot now has its
own unit tests including re_registration_swaps_to_the_latest_instance;
mutation-testing confirms a first-wins regression fails exactly that test.

No behavior change: the two hooks' existing tests, the P0 body_cache_hook_e2e
regressions, and the app-layer mutation-hook tests all pass unchanged.

Refs: backlog#1126, backlog#1131

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(admin): register the object-data-cache routes in the policy inventory

This PR added GET /object-data-cache/stats and POST /object-data-cache/flush
but did not list them in the two registries that must account for every
admin route: the route-policy inventory (route_policy.rs) and the route
matrix (route_registration_test.rs). Their coverage tests —
route_policy_inventory_covers_registered_routes and
test_admin_route_matrix_matches_registered_routes — failed on CI because a
registered route had no policy/matrix entry.

These two tests are not part of `make pre-commit` (which runs fmt + arch +
quick-check, not the full suite), so the gap passed local pre-commit and
only surfaced in the CI Test-and-Lint lane.

stats is a read (ServerInfoAdminAction, Sensitive); flush mutates
(ConfigUpdateAdminAction, High) — matching the actions the handlers already
enforce. The MinIO-alias matrix test is unaffected: these are native rustfs
endpoints with no MinIO equivalent.

Refs: backlog#1143

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>

crates/ecstore/src/api/mod.rs

@@ -351,8 +351,8 @@ pub mod notification {
 
 pub mod object {
     pub use crate::object_api::{
-        BLOCK_SIZE_V2, ERASURE_ALGORITHM, GetObjectBodyCacheHook, GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader,
-        RangedDecompressReader, StreamConsumer, register_get_object_body_cache_hook,
+        BLOCK_SIZE_V2, ERASURE_ALGORITHM, GetObjectBodyCacheHook, GetObjectReader, ObjectInfo, ObjectMutationHook, ObjectOptions,
+        PutObjReader, RangedDecompressReader, StreamConsumer, register_get_object_body_cache_hook, register_object_mutation_hook,
     };
 }
 

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -2261,6 +2261,9 @@ pub async fn expire_transitioned_object(
         opts.transition.expire_restored = true;
         return match api.delete_object(&oi.bucket, &oi.name, opts).await {
             Ok(dobj) => {
+                // Drop any cached restored-copy body so it does not sit resident
+                // until TTL after the copy is expired (ODC-26).
+                crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await;
                 //audit_log_lifecycle(*oi, ILMExpiry, tags, traceFn);
                 Ok(dobj)
             }
@@ -2293,6 +2296,10 @@ pub async fn expire_transitioned_object(
 
     schedule_lifecycle_replication_delete_if_needed(oi, &dobj).await;
 
+    // The transitioned version is gone; evict any cached body for this object
+    // so it does not linger until TTL (ODC-26).
+    crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await;
+
     //audit_log_lifecycle(oi, ILMExpiry, tags);
 
     emit_transitioned_expiration_event(oi, &dobj);
@@ -2802,6 +2809,13 @@ pub async fn apply_expiry_on_non_transitioned_objects(
         }
     };
     schedule_lifecycle_replication_delete_if_needed(oi, &dobj).await;
+
+    // The object (or all its versions, for delete_all) was expired; evict any
+    // cached body so dead bytes do not sit resident until TTL (ODC-26). The
+    // cache identity is the decoded object name used by GET, not the
+    // encode_dir_object form passed to delete_object.
+    crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await;
+
     //debug!("dobj: {:?}", dobj);
     if dobj.name.is_empty() {
         dobj = oi.clone();

crates/ecstore/src/client/object_handlers_common.rs

@@ -103,6 +103,11 @@ pub async fn delete_object_versions(api: &Arc<ECStore>, bucket: &str, to_del: &[
             if errors.get(i).and_then(|err| err.as_ref()).is_some() {
                 continue;
             }
+            // Evict any cached body for the successfully deleted noncurrent
+            // version so it does not sit resident until TTL (ODC-26).
+            if let Some(target) = to_del.get(i) {
+                crate::object_api::notify_object_mutation(bucket, &target.object_name).await;
+            }
             let Some(replication_state) = replication_candidates.get(i).and_then(|c| c.clone()) else {
                 continue;
             };

crates/ecstore/src/object_api/body_cache_hook.rs

@@ -21,8 +21,9 @@
 //! (after the reader is built) means a hit no longer saves any disk I/O.
 
 use crate::object_api::ObjectInfo;
+use crate::object_api::hook_slot::HookSlot;
 use bytes::Bytes;
-use std::sync::{Arc, RwLock};
+use std::sync::Arc;
 
 /// Serves full-object GET bodies from a cache keyed by object identity.
 ///
@@ -35,14 +36,15 @@ pub trait GetObjectBodyCacheHook: Send + Sync + 'static {
     async fn lookup(&self, bucket: &str, object: &str, info: &ObjectInfo) -> Option<Bytes>;
 }
 
-// `RwLock<Option<Arc<dyn ...>>>` rather than `ArcSwapOption`: arc-swap's
-// `RefCnt` is implemented only for the sized `Arc<T>` (it stores a thin
-// `*mut T`), so it cannot hold an `Arc<dyn GetObjectBodyCacheHook>` without a
-// sized newtype wrapper. The probe reads this slot once per full-object GET,
-// but the read guard only clones an `Arc`, which is negligible next to the
-// metadata quorum fan-out already completed before the probe. Registration is
-// a startup / config-reload event, so writer contention is a non-issue.
-static GET_OBJECT_BODY_CACHE_HOOK: RwLock<Option<Arc<dyn GetObjectBodyCacheHook>>> = RwLock::new(None);
+// A `HookSlot` (RwLock<Option<Arc<dyn ...>>>) rather than `ArcSwapOption`:
+// arc-swap's `RefCnt` is implemented only for the sized `Arc<T>` (it stores a
+// thin `*mut T`), so it cannot hold an `Arc<dyn GetObjectBodyCacheHook>`
+// without a sized newtype wrapper. The probe reads this slot once per
+// full-object GET, but the read guard only clones an `Arc`, which is negligible
+// next to the metadata quorum fan-out already completed before the probe.
+// Registration is a startup / config-reload event, so writer contention is a
+// non-issue.
+static GET_OBJECT_BODY_CACHE_HOOK: HookSlot<dyn GetObjectBodyCacheHook> = HookSlot::new();
 
 /// Register (or re-register) the process-wide GET body cache hook.
 ///
@@ -52,31 +54,22 @@ static GET_OBJECT_BODY_CACHE_HOOK: RwLock<Option<Arc<dyn GetObjectBodyCacheHook>
 /// to the original adapter while every usecase-layer fill and invalidation
 /// targeted the replacement, silently degrading the feature to a 0% hit rate
 /// and stranding entries in the unreachable cache until their TTL (backlog#1126).
-///
-/// Replacing a *different* hook instance is logged at WARN: in production the
-/// hook is installed exactly once per process, so a swap to a distinct instance
-/// signals an unexpected re-init and orphans the previous adapter's cache.
 pub fn register_get_object_body_cache_hook(hook: Arc<dyn GetObjectBodyCacheHook>) {
-    let mut slot = GET_OBJECT_BODY_CACHE_HOOK.write().unwrap_or_else(|e| e.into_inner());
-    if let Some(previous) = slot.as_ref()
-        && !Arc::ptr_eq(previous, &hook)
-    {
-        tracing::warn!(
-            "GET object body cache hook re-registered with a different instance; \
-             the previous adapter's cache is now unreachable by ecstore's GET probe"
-        );
-    }
-    *slot = Some(hook);
+    GET_OBJECT_BODY_CACHE_HOOK.register(
+        hook,
+        "GET object body cache hook re-registered with a different instance; \
+         the previous adapter's cache is now unreachable by ecstore's GET probe",
+    );
 }
 
 /// The registered hook, if any.
 pub(crate) fn get_object_body_cache_hook() -> Option<Arc<dyn GetObjectBodyCacheHook>> {
-    GET_OBJECT_BODY_CACHE_HOOK.read().unwrap_or_else(|e| e.into_inner()).clone()
+    GET_OBJECT_BODY_CACHE_HOOK.get()
 }
 
 /// Test-only: unregister the hook so tests can register and clear the slot
 /// deterministically without leaking a hook into unrelated tests.
 #[cfg(test)]
 pub(crate) fn clear_get_object_body_cache_hook() {
-    *GET_OBJECT_BODY_CACHE_HOOK.write().unwrap_or_else(|e| e.into_inner()) = None;
+    GET_OBJECT_BODY_CACHE_HOOK.clear();
 }

crates/ecstore/src/object_api/hook_slot.rs

@@ -0,0 +1,126 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! A process-wide, re-registrable slot for a single `Arc` hook.
+//!
+//! The GET body cache hook and the object mutation hook both need the same
+//! shape: one global slot that starts empty, is (re-)registered from the app
+//! layer at startup, is read on the hot/delete paths, and warns when a
+//! re-registration swaps in a *different* instance (an unexpected re-init that
+//! orphans the previous adapter's cache). This type holds that logic once so
+//! the two callers cannot drift apart.
+
+use std::sync::{Arc, RwLock};
+
+/// A global slot holding at most one `Arc<T>` hook.
+///
+/// Registration atomically swaps the stored hook. Poisoned locks recover in
+/// place: a panicked writer cannot leave an `Option<Arc<_>>` in an unsound
+/// state, so there is nothing to salvage.
+pub(crate) struct HookSlot<T: ?Sized> {
+    inner: RwLock<Option<Arc<T>>>,
+}
+
+impl<T: ?Sized> HookSlot<T> {
+    /// Creates an empty slot. `const` so it can initialize a `static`.
+    pub(crate) const fn new() -> Self {
+        Self {
+            inner: RwLock::new(None),
+        }
+    }
+
+    /// Registers (or re-registers) the hook.
+    ///
+    /// Replacing a *different* instance logs `warn_on_replace` at WARN: in
+    /// production a hook is installed exactly once per process, so a swap to a
+    /// distinct instance signals an unexpected re-init that leaves the previous
+    /// adapter's cache unreachable.
+    pub(crate) fn register(&self, hook: Arc<T>, warn_on_replace: &str) {
+        let mut slot = self.inner.write().unwrap_or_else(|poisoned| poisoned.into_inner());
+        if let Some(previous) = slot.as_ref()
+            && !Arc::ptr_eq(previous, &hook)
+        {
+            tracing::warn!("{warn_on_replace}");
+        }
+        *slot = Some(hook);
+    }
+
+    /// Returns the registered hook, if any.
+    pub(crate) fn get(&self) -> Option<Arc<T>> {
+        self.inner.read().unwrap_or_else(|poisoned| poisoned.into_inner()).clone()
+    }
+
+    /// Clears the slot. Test-only: production never unregisters a hook.
+    #[cfg(test)]
+    pub(crate) fn clear(&self) {
+        *self.inner.write().unwrap_or_else(|poisoned| poisoned.into_inner()) = None;
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::HookSlot;
+    use std::sync::Arc;
+
+    trait Marker: Send + Sync {
+        fn id(&self) -> u32;
+    }
+    struct Impl(u32);
+    impl Marker for Impl {
+        fn id(&self) -> u32 {
+            self.0
+        }
+    }
+
+    #[test]
+    fn empty_slot_reads_none() {
+        let slot: HookSlot<dyn Marker> = HookSlot::new();
+        assert!(slot.get().is_none());
+    }
+
+    #[test]
+    fn register_then_get_returns_the_hook() {
+        let slot: HookSlot<dyn Marker> = HookSlot::new();
+        slot.register(Arc::new(Impl(7)), "unused");
+        assert_eq!(slot.get().map(|h| h.id()), Some(7));
+    }
+
+    #[test]
+    fn re_registration_swaps_to_the_latest_instance() {
+        // The load-bearing #1126 guarantee: the newest registration wins, so a
+        // rebuilt AppContext leaves ecstore pointed at the current adapter
+        // rather than a stranded first-wins one.
+        let slot: HookSlot<dyn Marker> = HookSlot::new();
+        slot.register(Arc::new(Impl(1)), "unused");
+        slot.register(Arc::new(Impl(2)), "unused");
+        assert_eq!(slot.get().map(|h| h.id()), Some(2));
+    }
+
+    #[test]
+    fn re_registering_the_same_arc_is_idempotent() {
+        let slot: HookSlot<dyn Marker> = HookSlot::new();
+        let hook: Arc<dyn Marker> = Arc::new(Impl(9));
+        slot.register(Arc::clone(&hook), "unused");
+        slot.register(hook, "unused");
+        assert_eq!(slot.get().map(|h| h.id()), Some(9));
+    }
+
+    #[test]
+    fn clear_empties_the_slot() {
+        let slot: HookSlot<dyn Marker> = HookSlot::new();
+        slot.register(Arc::new(Impl(3)), "unused");
+        slot.clear();
+        assert!(slot.get().is_none());
+    }
+}

crates/ecstore/src/object_api/mod.rs

@@ -53,6 +53,8 @@ pub const ERASURE_ALGORITHM: &str = "rs-vandermonde";
 pub const BLOCK_SIZE_V2: usize = 1024 * 1024; // 1M
 
 mod body_cache_hook;
+mod hook_slot;
+mod object_mutation_hook;
 mod readers;
 mod types;
 
@@ -60,5 +62,7 @@ mod types;
 pub(crate) use body_cache_hook::clear_get_object_body_cache_hook;
 pub(crate) use body_cache_hook::get_object_body_cache_hook;
 pub use body_cache_hook::{GetObjectBodyCacheHook, register_get_object_body_cache_hook};
+pub(crate) use object_mutation_hook::notify_object_mutation;
+pub use object_mutation_hook::{ObjectMutationHook, register_object_mutation_hook};
 pub use readers::*;
 pub use types::*;

crates/ecstore/src/object_api/object_mutation_hook.rs

@@ -0,0 +1,119 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Write-side counterpart to [`super::body_cache_hook`].
+//!
+//! ecstore terminates object bodies from paths that never pass through the
+//! app-layer usecases: lifecycle/scanner expiry, non-current version cleanup,
+//! and restored-copy expiry all delete objects directly on the store. The GET
+//! body cache is keyed on `(bucket, object, versionId, etag, size, variant)`
+//! and every lookup follows a fresh metadata quorum, so a stale entry can never
+//! be *served* after its object is gone (the GET fails at metadata resolution
+//! first). What lingers is dead body bytes resident until TTL, which evict live
+//! hot entries — a hygiene and capacity problem, not a correctness one.
+//!
+//! This hook lets the app-layer cache adapter drop those bodies from its index
+//! the moment ecstore removes the object (ODC-26, backlog#1131).
+
+use crate::object_api::hook_slot::HookSlot;
+use std::sync::Arc;
+
+/// Invalidates cached object bodies after an ecstore-internal mutation removed
+/// them. Keyed on `(bucket, object)`; the implementation invalidates every
+/// cached version/etag under that identity.
+#[async_trait::async_trait]
+pub trait ObjectMutationHook: Send + Sync + 'static {
+    async fn after_object_mutation(&self, bucket: &str, object: &str);
+}
+
+// A `HookSlot`, for the same reason as the GET hook slot: arc-swap cannot hold
+// an unsized `Arc<dyn ObjectMutationHook>`. Registration is a startup event and
+// each delete-path invocation only clones an `Arc` under the read guard,
+// negligible next to the delete it accompanies.
+static OBJECT_MUTATION_HOOK: HookSlot<dyn ObjectMutationHook> = HookSlot::new();
+
+/// Register (or re-register) the process-wide object mutation hook.
+///
+/// Re-registration atomically swaps to `hook`, mirroring the GET body hook so a
+/// rebuilt `AppContext` leaves ecstore's delete paths pointed at the newest
+/// adapter.
+pub fn register_object_mutation_hook(hook: Arc<dyn ObjectMutationHook>) {
+    OBJECT_MUTATION_HOOK.register(
+        hook,
+        "object mutation cache hook re-registered with a different instance; \
+         the previous adapter's cache is now unreachable by ecstore's delete paths",
+    );
+}
+
+/// The registered hook, if any.
+fn object_mutation_hook() -> Option<Arc<dyn ObjectMutationHook>> {
+    OBJECT_MUTATION_HOOK.get()
+}
+
+/// Invoke the registered hook for `(bucket, object)`, if one is installed.
+///
+/// A single `None` branch when the cache feature is off, so the ecstore delete
+/// paths pay nothing beyond one relaxed lock read when unconfigured.
+pub(crate) async fn notify_object_mutation(bucket: &str, object: &str) {
+    if let Some(hook) = object_mutation_hook() {
+        hook.after_object_mutation(bucket, object).await;
+    }
+}
+
+/// Test-only: unregister the hook so tests can register and clear the slot
+/// deterministically without leaking a hook into unrelated tests.
+#[cfg(test)]
+pub(crate) fn clear_object_mutation_hook() {
+    OBJECT_MUTATION_HOOK.clear();
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::sync::Mutex;
+
+    struct RecordingHook {
+        calls: Arc<Mutex<Vec<(String, String)>>>,
+    }
+
+    #[async_trait::async_trait]
+    impl ObjectMutationHook for RecordingHook {
+        async fn after_object_mutation(&self, bucket: &str, object: &str) {
+            self.calls.lock().unwrap().push((bucket.to_string(), object.to_string()));
+        }
+    }
+
+    #[tokio::test]
+    #[serial_test::serial(object_mutation_hook)]
+    async fn notify_invokes_registered_hook_with_identity() {
+        clear_object_mutation_hook();
+        let calls = Arc::new(Mutex::new(Vec::new()));
+        register_object_mutation_hook(Arc::new(RecordingHook {
+            calls: Arc::clone(&calls),
+        }));
+
+        notify_object_mutation("bucket", "photos/a.jpg").await;
+
+        assert_eq!(&*calls.lock().unwrap(), &[("bucket".to_string(), "photos/a.jpg".to_string())]);
+        clear_object_mutation_hook();
+    }
+
+    #[tokio::test]
+    #[serial_test::serial(object_mutation_hook)]
+    async fn notify_without_registered_hook_is_noop() {
+        clear_object_mutation_hook();
+        // Must not panic when no hook is installed (the cache feature is off).
+        notify_object_mutation("bucket", "object").await;
+    }
+}

crates/object-data-cache/src/cache.rs

@@ -258,25 +258,80 @@ impl ObjectDataCache {
     /// Invalidates all cache entries associated with the object identity.
     pub async fn invalidate_object(
         &self,
-        _identity: ObjectDataCacheIdentity,
-        _reason: ObjectDataCacheInvalidationReason,
+        identity: ObjectDataCacheIdentity,
+        reason: ObjectDataCacheInvalidationReason,
     ) -> ObjectDataCacheInvalidationResult {
         let result = match &self.backend {
             ObjectDataCacheBackendKind::Noop(backend) => backend.invalidate_object().await,
-            ObjectDataCacheBackendKind::Moka(backend) => backend.invalidate_object(&_identity).await,
+            ObjectDataCacheBackendKind::Moka(backend) => backend.invalidate_object(&identity).await,
         };
+        self.finish_invalidation(result, reason)
+    }
 
+    /// Invalidates every cached body under `bucket`/`prefix`.
+    ///
+    /// Drops the cached bodies of every identity in `bucket` whose object key
+    /// starts with `prefix`. Backed by a full identity-index scan, so it must
+    /// stay on the rare force-delete and admin paths and never touch the GET or
+    /// fill hot path (ODC-27, backlog#1132).
+    pub async fn invalidate_prefix(
+        &self,
+        bucket: &str,
+        prefix: &str,
+        reason: ObjectDataCacheInvalidationReason,
+    ) -> ObjectDataCacheInvalidationResult {
+        let result = match &self.backend {
+            ObjectDataCacheBackendKind::Noop(backend) => backend.invalidate_prefix().await,
+            ObjectDataCacheBackendKind::Moka(backend) => backend.invalidate_prefix(bucket, prefix).await,
+        };
+        self.finish_invalidation(result, reason)
+    }
+
+    /// Invalidates every cached body in `bucket`.
+    ///
+    /// Backed by a full identity-index scan; keep it on the rare bucket-delete
+    /// and admin paths only (ODC-28, backlog#1133).
+    pub async fn invalidate_bucket(
+        &self,
+        bucket: &str,
+        reason: ObjectDataCacheInvalidationReason,
+    ) -> ObjectDataCacheInvalidationResult {
+        let result = match &self.backend {
+            ObjectDataCacheBackendKind::Noop(backend) => backend.invalidate_bucket().await,
+            ObjectDataCacheBackendKind::Moka(backend) => backend.invalidate_bucket(bucket).await,
+        };
+        self.finish_invalidation(result, reason)
+    }
+
+    /// Drops every cached body and resets the identity index.
+    ///
+    /// The only production remediation for a poisoned or stale entry short of a
+    /// node restart (ODC-C2, backlog#1143). Rare admin path only.
+    pub async fn clear(&self, reason: ObjectDataCacheInvalidationReason) -> ObjectDataCacheInvalidationResult {
+        let result = match &self.backend {
+            ObjectDataCacheBackendKind::Noop(backend) => backend.clear().await,
+            ObjectDataCacheBackendKind::Moka(backend) => backend.clear().await,
+        };
+        self.finish_invalidation(result, reason)
+    }
+
+    /// Shared post-processing for every invalidation primitive: bump the
+    /// invalidation counter, refresh the cache-state gauge only when something
+    /// was removed, and emit the outcome-labelled metric. A mutating op
+    /// invalidates twice by design (before + after) and the vast majority of
+    /// those touch identities that were never cached, so the no-op path skips
+    /// the gauge refresh (backlog#1141).
+    fn finish_invalidation(
+        &self,
+        result: ObjectDataCacheInvalidationResult,
+        reason: ObjectDataCacheInvalidationReason,
+    ) -> ObjectDataCacheInvalidationResult {
         self.stats.record_invalidation();
         let outcome = invalidation_outcome(&result);
-        // A mutating op invalidates twice by design (before + after); the vast
-        // majority of those touch identities that were never cached. Only refresh
-        // the cache-state gauge when something was actually removed so the
-        // no-op path stays cheap (backlog#1141).
         if outcome != INVALIDATION_OUTCOME_NOOP {
             self.refresh_entry_count();
         }
-        record_invalidation(self.backend.as_metric_label(), _reason.as_metric_label(), outcome);
-
+        record_invalidation(self.backend.as_metric_label(), reason.as_metric_label(), outcome);
         result
     }
 
@@ -285,6 +340,11 @@ impl ObjectDataCache {
         self.stats.snapshot()
     }
 
+    /// Returns the configured runtime mode, for admin status reporting.
+    pub fn mode(&self) -> crate::config::ObjectDataCacheMode {
+        self.config.mode
+    }
+
     /// Returns true when the cache facade is fully disabled.
     pub fn is_disabled(&self) -> bool {
         self.config.is_disabled()
@@ -438,6 +498,15 @@ pub enum ObjectDataCacheInvalidationReason {
     AfterCopySuccess,
     /// Invalidation after a successful complete multipart upload.
     AfterCompleteMultipartSuccess,
+    /// Invalidation after an ecstore-internal lifecycle/scanner expiry deleted
+    /// the object body (ODC-26).
+    AfterLifecycleExpiry,
+    /// Invalidation after a forced prefix delete removed every object under a
+    /// prefix (ODC-27).
+    AfterPrefixDelete,
+    /// Invalidation after a bucket delete removed every object in the bucket
+    /// (ODC-28).
+    AfterBucketDelete,
     /// Manual invalidation requested by the caller.
     Manual,
 }
@@ -450,6 +519,9 @@ impl ObjectDataCacheInvalidationReason {
             Self::AfterDeleteSuccess => "after_delete_success",
             Self::AfterCopySuccess => "after_copy_success",
             Self::AfterCompleteMultipartSuccess => "after_complete_multipart_success",
+            Self::AfterLifecycleExpiry => "after_lifecycle_expiry",
+            Self::AfterPrefixDelete => "after_prefix_delete",
+            Self::AfterBucketDelete => "after_bucket_delete",
             Self::Manual => "manual",
         }
     }
@@ -471,7 +543,7 @@ pub enum ObjectDataCacheInvalidationResult {
 mod tests {
     use super::{
         ObjectDataCache, ObjectDataCacheFillResult, ObjectDataCacheGetPlan, ObjectDataCacheGetRequest,
-        ObjectDataCacheInvalidationReason, ObjectDataCacheLookup,
+        ObjectDataCacheInvalidationReason, ObjectDataCacheInvalidationResult, ObjectDataCacheLookup,
     };
     use crate::config::{ObjectDataCacheConfig, ObjectDataCacheMode};
     use crate::key::{ObjectDataCacheBodyVariant, ObjectDataCacheIdentity};
@@ -678,6 +750,96 @@ mod tests {
         );
     }
 
+    #[test]
+    fn new_invalidation_reasons_map_to_distinct_labels() {
+        assert_eq!(
+            ObjectDataCacheInvalidationReason::AfterLifecycleExpiry.as_metric_label(),
+            "after_lifecycle_expiry"
+        );
+        assert_eq!(
+            ObjectDataCacheInvalidationReason::AfterPrefixDelete.as_metric_label(),
+            "after_prefix_delete"
+        );
+        assert_eq!(
+            ObjectDataCacheInvalidationReason::AfterBucketDelete.as_metric_label(),
+            "after_bucket_delete"
+        );
+    }
+
+    #[test]
+    fn prefix_invalidation_of_cached_identity_labels_reason_and_removed() {
+        // ODC-27: a prefix flush that drops a cached body is labelled with its
+        // own reason and outcome=removed so dashboards attribute the churn.
+        let cache = fill_enabled_cache();
+        let metrics = capture_metrics(|| async {
+            let plan = cache.plan_get(plain_request("bucket", "photos/a", "etag", 5));
+            assert_eq!(
+                cache.fill_body(&plan, Bytes::from_static(b"hello")).await,
+                ObjectDataCacheFillResult::Inserted
+            );
+            let result = cache
+                .invalidate_prefix("bucket", "photos/", ObjectDataCacheInvalidationReason::AfterPrefixDelete)
+                .await;
+            assert_eq!(result, ObjectDataCacheInvalidationResult::Removed { keys: 1 });
+        });
+
+        assert!(has_counter_with_label(
+            &metrics,
+            "rustfs_object_data_cache_invalidations_total",
+            ("reason", "after_prefix_delete")
+        ));
+        assert!(has_counter_with_label(
+            &metrics,
+            "rustfs_object_data_cache_invalidations_total",
+            ("outcome", "removed")
+        ));
+    }
+
+    #[test]
+    fn bucket_invalidation_of_uncached_bucket_labels_noop() {
+        // ODC-28: a bucket flush that matched nothing is a no-op and must not
+        // refresh the entries gauge.
+        let cache = fill_enabled_cache();
+        let metrics = capture_metrics(|| async {
+            let result = cache
+                .invalidate_bucket("empty-bucket", ObjectDataCacheInvalidationReason::AfterBucketDelete)
+                .await;
+            assert_eq!(result, ObjectDataCacheInvalidationResult::NoOp);
+        });
+
+        assert!(has_counter_with_label(
+            &metrics,
+            "rustfs_object_data_cache_invalidations_total",
+            ("outcome", "noop")
+        ));
+        assert!(
+            !has_gauge(&metrics, "rustfs_object_data_cache_entries"),
+            "a no-op bucket flush must not refresh the entries gauge"
+        );
+    }
+
+    #[test]
+    fn clear_of_cached_cache_labels_manual_and_removed() {
+        // ODC-C2: an admin clear reuses the Manual reason and reports removed.
+        let cache = fill_enabled_cache();
+        let metrics = capture_metrics(|| async {
+            let plan = cache.plan_get(plain_request("bucket", "object", "etag", 5));
+            assert_eq!(
+                cache.fill_body(&plan, Bytes::from_static(b"hello")).await,
+                ObjectDataCacheFillResult::Inserted
+            );
+            let result = cache.clear(ObjectDataCacheInvalidationReason::Manual).await;
+            assert_eq!(result, ObjectDataCacheInvalidationResult::Removed { keys: 1 });
+            assert!(matches!(cache.lookup_body(&plan).await, ObjectDataCacheLookup::Miss));
+        });
+
+        assert!(has_counter_with_label(
+            &metrics,
+            "rustfs_object_data_cache_invalidations_total",
+            ("reason", "manual")
+        ));
+    }
+
     #[tokio::test]
     async fn fill_body_rejects_size_mismatch() {
         let cache = fill_enabled_cache();

crates/object-data-cache/src/config.rs

@@ -54,6 +54,11 @@ pub enum ObjectDataCacheMode {
 }
 
 impl ObjectDataCacheMode {
+    /// Stable lowercase identifier for this mode, for admin status reporting.
+    pub const fn as_str(self) -> &'static str {
+        self.as_metric_label()
+    }
+
     pub(crate) const fn as_metric_label(self) -> &'static str {
         match self {
             Self::Disabled => "disabled",

crates/object-data-cache/src/memory.rs

@@ -153,7 +153,7 @@ impl Drop for RefresherGuard {
 /// Memory gate that keeps a cheap, lock-free snapshot for fill-path checks.
 ///
 /// The snapshot is sampled off the fill path by a dedicated periodic refresher
-/// (see [`spawn_refresher`](ObjectDataCacheMemoryGate::spawn_refresher)) that
+/// (the private `spawn_refresher` task) that
 /// runs the blocking `sysinfo` read on a `spawn_blocking` thread. `allows_fill`
 /// only reads atomics, so it never blocks a tokio worker and concurrent fills
 /// never serialize on a refresh.

crates/object-data-cache/src/moka_backend.rs

@@ -293,6 +293,48 @@ impl MokaBackend {
             self.cache.remove(&key).await;
         }
 
+        Self::invalidation_result(removed)
+    }
+
+    /// Invalidates every cached key of an identity in `bucket` whose object key
+    /// starts with `prefix`. Full index scan; rare force-delete/admin path only.
+    pub async fn invalidate_prefix(&self, bucket: &str, prefix: &str) -> ObjectDataCacheInvalidationResult {
+        let keys_to_remove = self
+            .index
+            .remove_matching(|identity| identity.bucket.as_ref() == bucket && identity.object.starts_with(prefix))
+            .await;
+        let removed = keys_to_remove.len();
+        for key in keys_to_remove {
+            self.cache.remove(&key).await;
+        }
+        Self::invalidation_result(removed)
+    }
+
+    /// Invalidates every cached key of every identity in `bucket`. Full index
+    /// scan; rare bucket-delete/admin path only.
+    pub async fn invalidate_bucket(&self, bucket: &str) -> ObjectDataCacheInvalidationResult {
+        let keys_to_remove = self
+            .index
+            .remove_matching(|identity| identity.bucket.as_ref() == bucket)
+            .await;
+        let removed = keys_to_remove.len();
+        for key in keys_to_remove {
+            self.cache.remove(&key).await;
+        }
+        Self::invalidation_result(removed)
+    }
+
+    /// Drops every cached body and resets the identity index. The index scan
+    /// yields the tracked key count for the outcome label; `invalidate_all`
+    /// then clears moka in one call (also dropping any entry not tracked in the
+    /// index). Rare admin `clear()` path only.
+    pub async fn clear(&self) -> ObjectDataCacheInvalidationResult {
+        let removed = self.index.remove_matching(|_| true).await.len();
+        self.cache.invalidate_all();
+        Self::invalidation_result(removed)
+    }
+
+    const fn invalidation_result(removed: usize) -> ObjectDataCacheInvalidationResult {
         if removed > 0 {
             ObjectDataCacheInvalidationResult::Removed { keys: removed }
         } else {
@@ -402,6 +444,109 @@ mod tests {
         assert_eq!(result, ObjectDataCacheInvalidationResult::NoOp);
     }
 
+    fn bucketed_plan(bucket: &str, object: &str, etag: &str) -> ObjectDataCacheGetPlan {
+        ObjectDataCacheGetPlan::Cacheable {
+            key: ObjectDataCacheKey::new(bucket, object, None, etag, 5, ObjectDataCacheBodyVariant::FullObjectPlainV1),
+        }
+    }
+
+    // ODC-27: prefix invalidation drops only the identities under the prefix and
+    // leaves every other cached body intact.
+    #[tokio::test]
+    async fn moka_backend_invalidate_prefix_removes_only_matching_prefix() {
+        let mut config = enabled_config();
+        config.ttl = Duration::from_secs(30);
+        config.time_to_idle = Duration::from_secs(30);
+        let backend = MokaBackend::new(&config, Arc::new(ObjectDataCacheStats::default())).expect("moka backend should build");
+        let plan_a = cacheable_plan("photos/a.jpg", "etag-a");
+        let plan_b = cacheable_plan("photos/b.jpg", "etag-b");
+        let plan_c = cacheable_plan("videos/c.mp4", "etag-c");
+
+        let _ = backend.fill_body(&plan_a, Bytes::from_static(b"aaaaa")).await;
+        let _ = backend.fill_body(&plan_b, Bytes::from_static(b"bbbbb")).await;
+        let _ = backend.fill_body(&plan_c, Bytes::from_static(b"ccccc")).await;
+
+        let result = backend.invalidate_prefix("bucket", "photos/").await;
+
+        assert_eq!(result, ObjectDataCacheInvalidationResult::Removed { keys: 2 });
+        assert!(matches!(backend.lookup_body(&plan_a).await, ObjectDataCacheLookup::Miss));
+        assert!(matches!(backend.lookup_body(&plan_b).await, ObjectDataCacheLookup::Miss));
+        assert!(
+            matches!(backend.lookup_body(&plan_c).await, ObjectDataCacheLookup::Hit(_)),
+            "an object outside the prefix must survive"
+        );
+    }
+
+    // ODC-27: a prefix that matches nothing cached is a no-op.
+    #[tokio::test]
+    async fn moka_backend_invalidate_prefix_uncached_is_noop() {
+        let backend =
+            MokaBackend::new(&enabled_config(), Arc::new(ObjectDataCacheStats::default())).expect("moka backend should build");
+        let _ = backend
+            .fill_body(&cacheable_plan("photos/a.jpg", "e"), Bytes::from_static(b"aaaaa"))
+            .await;
+
+        let result = backend.invalidate_prefix("bucket", "videos/").await;
+
+        assert_eq!(result, ObjectDataCacheInvalidationResult::NoOp);
+    }
+
+    // ODC-28: bucket invalidation drops every identity in the bucket and leaves
+    // other buckets untouched.
+    #[tokio::test]
+    async fn moka_backend_invalidate_bucket_removes_only_that_bucket() {
+        let mut config = enabled_config();
+        config.ttl = Duration::from_secs(30);
+        config.time_to_idle = Duration::from_secs(30);
+        let backend = MokaBackend::new(&config, Arc::new(ObjectDataCacheStats::default())).expect("moka backend should build");
+        let plan_a = bucketed_plan("bucket-a", "o1", "etag-a");
+        let plan_b = bucketed_plan("bucket-a", "o2", "etag-b");
+        let plan_other = bucketed_plan("bucket-b", "o3", "etag-c");
+
+        let _ = backend.fill_body(&plan_a, Bytes::from_static(b"aaaaa")).await;
+        let _ = backend.fill_body(&plan_b, Bytes::from_static(b"bbbbb")).await;
+        let _ = backend.fill_body(&plan_other, Bytes::from_static(b"ccccc")).await;
+
+        let result = backend.invalidate_bucket("bucket-a").await;
+
+        assert_eq!(result, ObjectDataCacheInvalidationResult::Removed { keys: 2 });
+        assert!(matches!(backend.lookup_body(&plan_a).await, ObjectDataCacheLookup::Miss));
+        assert!(matches!(backend.lookup_body(&plan_b).await, ObjectDataCacheLookup::Miss));
+        assert!(
+            matches!(backend.lookup_body(&plan_other).await, ObjectDataCacheLookup::Hit(_)),
+            "an object in another bucket must survive a bucket flush"
+        );
+    }
+
+    // ODC-C2: clear drops every cached body and empties the identity index.
+    #[tokio::test]
+    async fn moka_backend_clear_removes_everything() {
+        let mut config = enabled_config();
+        config.ttl = Duration::from_secs(30);
+        config.time_to_idle = Duration::from_secs(30);
+        let backend = MokaBackend::new(&config, Arc::new(ObjectDataCacheStats::default())).expect("moka backend should build");
+        let plan_a = bucketed_plan("bucket-a", "o1", "etag-a");
+        let plan_b = bucketed_plan("bucket-b", "o2", "etag-b");
+
+        let _ = backend.fill_body(&plan_a, Bytes::from_static(b"aaaaa")).await;
+        let _ = backend.fill_body(&plan_b, Bytes::from_static(b"bbbbb")).await;
+
+        let result = backend.clear().await;
+
+        assert_eq!(result, ObjectDataCacheInvalidationResult::Removed { keys: 2 });
+        assert!(matches!(backend.lookup_body(&plan_a).await, ObjectDataCacheLookup::Miss));
+        assert!(matches!(backend.lookup_body(&plan_b).await, ObjectDataCacheLookup::Miss));
+        assert_eq!(backend.index.identity_count().await, 0, "clear must empty the identity index");
+    }
+
+    #[tokio::test]
+    async fn moka_backend_clear_empty_cache_is_noop() {
+        let backend =
+            MokaBackend::new(&enabled_config(), Arc::new(ObjectDataCacheStats::default())).expect("moka backend should build");
+
+        assert_eq!(backend.clear().await, ObjectDataCacheInvalidationResult::NoOp);
+    }
+
     #[tokio::test]
     async fn moka_backend_expires_entries_by_ttl() {
         let backend =

crates/object-data-cache/src/noop.rs

@@ -34,6 +34,21 @@ impl NoopBackend {
     pub async fn invalidate_object(&self) -> ObjectDataCacheInvalidationResult {
         ObjectDataCacheInvalidationResult::NoOp
     }
+
+    /// Prefix invalidation on a disabled cache removes nothing.
+    pub async fn invalidate_prefix(&self) -> ObjectDataCacheInvalidationResult {
+        ObjectDataCacheInvalidationResult::NoOp
+    }
+
+    /// Bucket invalidation on a disabled cache removes nothing.
+    pub async fn invalidate_bucket(&self) -> ObjectDataCacheInvalidationResult {
+        ObjectDataCacheInvalidationResult::NoOp
+    }
+
+    /// Clearing a disabled cache removes nothing.
+    pub async fn clear(&self) -> ObjectDataCacheInvalidationResult {
+        ObjectDataCacheInvalidationResult::NoOp
+    }
 }
 
 #[cfg(test)]

crates/object-data-cache/src/starshard_index.rs

@@ -98,6 +98,42 @@ impl StarshardIdentityIndex {
             .map_or_else(Vec::new, |set| set.cloned())
     }
 
+    /// Removes every tracked identity whose key matches `predicate`, returning
+    /// all keys that were dropped so the caller can evict them from the cache.
+    ///
+    /// This performs a **full shard scan** (`keys()` snapshots every identity,
+    /// then each match is removed under its shard write lock). It is therefore
+    /// restricted to the rare prefix-delete, bucket-delete, and admin
+    /// `clear()`/flush paths — it must never run on the GET or fill hot path.
+    /// A `true`-returning predicate clears the whole index (used by `clear()`).
+    ///
+    /// The snapshot/remove split leaves a small window: an identity inserted
+    /// after the snapshot but before removal is not visited, and a key added to
+    /// a matched identity between snapshot and its `remove` is still dropped
+    /// from the index (via `remove`) but returned for cache eviction, so no
+    /// tracked body is stranded. This is acceptable hygiene slack on these rare
+    /// paths (backlog#1132/#1133/#1143).
+    pub async fn remove_matching<F>(&self, predicate: F) -> Vec<ObjectDataCacheKey>
+    where
+        F: Fn(&ObjectDataCacheIdentity) -> bool,
+    {
+        let identities: Vec<ObjectDataCacheIdentity> = self
+            .by_object
+            .keys()
+            .await
+            .into_iter()
+            .filter(|identity| predicate(identity))
+            .collect();
+
+        let mut removed = Vec::new();
+        for identity in identities {
+            if let Some(key_set) = self.by_object.remove(&identity).await {
+                removed.extend(key_set.cloned());
+            }
+        }
+        removed
+    }
+
     /// Removes a single evicted key tracked under an identity, but only when the
     /// evicted entry's generation token still matches the tracked one.
     ///
@@ -259,6 +295,53 @@ mod tests {
         assert_eq!(invalidated, vec![key_a]);
     }
 
+    fn bucketed_key(bucket: &str, object: &str) -> ObjectDataCacheKey {
+        ObjectDataCacheKey::new(bucket, object, Some("v1"), "etag", 1, ObjectDataCacheBodyVariant::FullObjectPlainV1)
+    }
+
+    #[tokio::test]
+    async fn remove_matching_returns_only_predicate_matches() {
+        let index = StarshardIdentityIndex::new(4);
+        let id_photos = ObjectDataCacheIdentity::new("bucket", "photos/a.jpg");
+        let id_photos2 = ObjectDataCacheIdentity::new("bucket", "photos/b.jpg");
+        let id_videos = ObjectDataCacheIdentity::new("bucket", "videos/c.mp4");
+
+        let _ = index
+            .insert(id_photos.clone(), bucketed_key("bucket", "photos/a.jpg"), 1)
+            .await;
+        let _ = index
+            .insert(id_photos2.clone(), bucketed_key("bucket", "photos/b.jpg"), 2)
+            .await;
+        let _ = index
+            .insert(id_videos.clone(), bucketed_key("bucket", "videos/c.mp4"), 3)
+            .await;
+
+        let removed = index
+            .remove_matching(|identity| identity.bucket.as_ref() == "bucket" && identity.object.starts_with("photos/"))
+            .await;
+
+        assert_eq!(removed.len(), 2, "only the two photos/ identities match the prefix");
+        // The unmatched identity is still tracked; the matched ones are gone.
+        assert!(index.remove_identity(&id_videos).await.len() == 1);
+        assert!(index.remove_identity(&id_photos).await.is_empty());
+    }
+
+    #[tokio::test]
+    async fn remove_matching_true_predicate_clears_index() {
+        let index = StarshardIdentityIndex::new(4);
+        let _ = index
+            .insert(ObjectDataCacheIdentity::new("b1", "o1"), bucketed_key("b1", "o1"), 1)
+            .await;
+        let _ = index
+            .insert(ObjectDataCacheIdentity::new("b2", "o2"), bucketed_key("b2", "o2"), 2)
+            .await;
+
+        let removed = index.remove_matching(|_| true).await;
+
+        assert_eq!(removed.len(), 2);
+        assert_eq!(index.identity_count().await, 0, "a true predicate clears every identity");
+    }
+
     #[tokio::test]
     async fn identity_index_matching_eviction_removes_key() {
         let index = StarshardIdentityIndex::new(4);