perf(object-data-cache): take fill off the GET path and fix the metrics (#4678)

* fix(object-data-cache): make cache metrics one-increment-per-GET

Rework the object data cache observability so each counter carries one
clear meaning, and drop dead per-entry state.

ODC-17 (backlog#1122): split requests_total, which was incremented by
both plan_get and lookup_body, into plan_total{decision,reason,size_class}
(emitted only by plan_get) and lookup_total{result,size_class} (emitted
only by lookup_body). Each is now incremented exactly once per GET per
layer; help text states this.

ODC-18 (backlog#1123): add a JoinedInflightFill result (label
joined_inflight) so a singleflight waiter is no longer counted as an
insert. record_fill_result now always counts the outcome but records
fill bytes only when non-zero and the duration histogram only when
present, so joined_inflight / skipped_by_mode / skipped_size_mismatch
no longer inflate fill_bytes_total or add non-fill duration samples.
The waiter->JoinedInflightFill mapping lives in MokaBackend (owned by a
concurrent branch); cache.rs handles the variant already.

ODC-29 (backlog#1134): stop refreshing the cache-state gauge on every
lookup, and debounce it on fill/invalidate to at most once per second
via an AtomicU64 millis timestamp, since moka's entry_count is a
settling approximation.

ODC-36 (backlog#1141): give invalidations_total an outcome label
(removed|noop) and skip the gauge refresh on the no-op path. Extend
ObjectDataCacheInvalidationResult with Removed{keys}/NoOp (Success kept
as a transitional variant until MokaBackend reports the removal count);
NoopBackend now reports NoOp.

ODC-30 (backlog#1141): drop the dead content_length/etag/inserted_at
fields (and getters) from ObjectDataCacheEntry, which are redundant with
the moka key identity; the constructor keeps its arity so the
out-of-scope MokaBackend caller still compiles. Gate is_null_version
behind cfg(test).

Tests use a thread-local metrics recorder to avoid the global-singleton
flakiness. Fill-enabled test configs set min_free_memory_percent=0 so
the cgroup gate does not refuse fills in CI pods.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(object-data-cache): move fill off GET path, bound & harden the fill

Implements five object-data-cache audit findings.

ODC-15 (backlog#1120): the GET response no longer blocks on cache fill.
The buffered/materialized body is already in hand, so the fill now runs
in a detached task and the response is built immediately. Singleflight is
made non-blocking: `try_acquire` returns Leader or Busy, and a non-leader
skips its own fill (SkippedSingleflightBusy) instead of waiting on another
request's leader. The inner fill spawn is KEPT: once the index key is
registered, the recheck/undo must complete even if the enclosing fill
future is aborted, so removing it would weaken the cancellation-safety
guarantee (ODC-13 test) and the invalidation-race undo.

ODC-11 (backlog#1116): enforce the fill-concurrency knobs. MokaBackend
now holds a Semaphore sized min(per_cpu * parallelism, max), acquired
after winning leadership and before the memory gate. On saturation it
rejects (try_acquire_owned) with SkippedFillConcurrency rather than
queueing, so the fill path never reintroduces GET-latency coupling.

ODC-14 (backlog#1119): the memory gate no longer does a blocking sysinfo
refresh under a mutex on the async fill path. A dedicated periodic
refresher (tokio interval + spawn_blocking) updates an atomic snapshot
every 5s; allows_fill is now lock-free. The min_free_memory_percent == 0
short-circuit still runs before any snapshot read. No runtime at
construction (sync unit tests) simply keeps the seed snapshot.

ODC-32 (backlog#1137): singleflight no longer emits metrics while holding
the map mutex. try_acquire/remove_entry capture the map length under the
guard, drop it, then emit the gauge/counter.

ODC-07 (backlog#1112): the materialize read is bounded via
take(capacity + 1) so an over-long stream cannot grow the buffer past the
in-memory GET threshold, and any length mismatch is now a hard error
matching the direct-memory GET path, instead of warn-and-serve.

cache.rs is owned by a concurrent branch; this commit only appends the
SkippedFillConcurrency and SkippedSingleflightBusy variants + label arms.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(object-data-cache): report JoinedInflightFill and invalidation outcome

Reconciles the moka_backend half of two metrics-semantics findings after
merging fix/odc-metrics-semantics (which landed the cache.rs half).

ODC-18 (backlog#1123): the singleflight non-leader path now returns
JoinedInflightFill instead of counting as an insert, so N concurrent GETs
of one cold key record one insert, not N. This composes with the ODC-15
redesign: the non-leader does NOT wait for the leader (it already owns the
body and never re-serves from the fill result), so no blocking wait is
reintroduced. Drops the redundant local SkippedSingleflightBusy variant in
favor of the canonical JoinedInflightFill (mapped to no bytes / no duration
by the facade). SkippedFillConcurrency (ODC-11) is retained.

ODC-36 (backlog#1141): MokaBackend::invalidate_object now returns
Removed { keys } / NoOp instead of the transitional Success, so the facade
labels invalidations_total{outcome=removed|noop} and skips the cache-state
gauge refresh on the no-op path.

Tests: duplicate-fill test asserts JoinedInflightFill (bites when reverted
to Inserted); new no-op invalidation test; matching-identity test asserts
Removed { keys: 1 }.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(object-data-cache): drop unused mut on materialize stream binding

The ODC-07 change moves `final_stream` into `AsyncReadExt::take`, so the
`build_get_object_body_with_cache` parameter is no longer mutated in place.

Co-Authored-By: heihutu <heihutu@gmail.com>

* refactor(object-data-cache): close the cross-branch coordination seams

Merging the metrics and hot-path branches left four transitional shims that
only existed because each side could not edit the other's files. With both
sides present they are dead weight, and two of them actively misreport.

- ObjectDataCacheEntry::new took content_length and etag only to keep the
  caller in moka_backend.rs compiling, then discarded both. Drop them; the
  entry is a Bytes wrapper and the caller now passes only the body.

- SkippedSingleflightClosed lost its only producer when the waiter model was
  replaced by Leader/Busy election. Remove the variant.

- The fill-accounting match ended in a wildcard that charged fill_bytes and a
  duration to every non-Inserted outcome, so a fill rejected by the memory
  gate or the concurrency semaphore — which never touched the backend and
  wrote nothing — still inflated fill_bytes_total. Enumerate every variant
  explicitly: only outcomes that wrote the body report bytes and duration.
  Exhaustiveness also forces a future variant to state its accounting rather
  than inherit a wrong default.

- InvalidationResult::Success mapped a no-op to outcome=removed, the exact lie
  backlog#1141 set out to fix, and its doc comment claimed MokaBackend still
  returned it after that backend had been widened. It has no producer left.
  Remove it, and tighten the app-layer test from "any successful variant" to
  the real contract: the pre-mutation call reports Removed{keys:1}, the
  post-delete call NoOp.

Refs: backlog#1123, backlog#1141, backlog#1135

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>

crates/object-data-cache/src/cache.rs

@@ -17,22 +17,39 @@ use crate::config::ObjectDataCacheConfig;
 use crate::error::ObjectDataCacheConfigError;
 use crate::key::{ObjectDataCacheBodyVariant, ObjectDataCacheIdentity, ObjectDataCacheKey};
 use crate::metrics::{
-    describe_metrics_once, publish_cache_state, record_fill_result, record_hit_bytes, record_invalidation,
-    record_request_decision,
+    describe_metrics_once, publish_cache_state, record_fill_result, record_hit_bytes, record_invalidation, record_lookup_result,
+    record_plan_decision,
 };
 use crate::moka_backend::MokaBackend;
 use crate::noop::NoopBackend;
 use crate::stats::{ObjectDataCacheStats, ObjectDataCacheStatsSnapshot};
 use bytes::Bytes;
 use std::sync::Arc;
+use std::sync::atomic::{AtomicU64, Ordering};
 use std::time::Instant;
 
+/// Minimum spacing between cache-state gauge publishes. Moka's `entry_count`
+/// and `weighted_size` are cross-segment approximations that only settle after
+/// pending tasks run, so republishing on every fill/invalidate lands the same
+/// stale value thousands of times between scrapes (backlog#1134).
+const ENTRY_COUNT_PUBLISH_DEBOUNCE_MS: u64 = 1000;
+
+/// Invalidation outcome label: keys were dropped from the cache.
+const INVALIDATION_OUTCOME_REMOVED: &str = "removed";
+/// Invalidation outcome label: the identity was not cached, nothing removed.
+const INVALIDATION_OUTCOME_NOOP: &str = "noop";
+
 /// Protocol-neutral cache facade for object body reuse.
 #[derive(Debug)]
 pub struct ObjectDataCache {
     backend: ObjectDataCacheBackendKind,
     config: Arc<ObjectDataCacheConfig>,
     stats: Arc<ObjectDataCacheStats>,
+    /// Monotonic origin for the cache-state publish debounce.
+    created_at: Instant,
+    /// Millis since `created_at` of the last cache-state gauge publish, or `0`
+    /// when it has never published.
+    last_entry_publish_ms: AtomicU64,
 }
 
 impl ObjectDataCache {
@@ -45,6 +62,8 @@ impl ObjectDataCache {
             backend: ObjectDataCacheBackendKind::Noop(NoopBackend),
             config,
             stats,
+            created_at: Instant::now(),
+            last_entry_publish_ms: AtomicU64::new(0),
         }
     }
 
@@ -63,13 +82,15 @@ impl ObjectDataCache {
             backend,
             config: Arc::new(config),
             stats,
+            created_at: Instant::now(),
+            last_entry_publish_ms: AtomicU64::new(0),
         })
     }
 
     /// Produces a lightweight GET plan from request metadata.
     pub fn plan_get(&self, request: ObjectDataCacheGetRequest<'_>) -> ObjectDataCacheGetPlan {
         if self.config.is_disabled() {
-            record_request_decision(
+            record_plan_decision(
                 self.backend.as_metric_label(),
                 self.config.mode,
                 "disabled",
@@ -80,11 +101,11 @@ impl ObjectDataCache {
         }
 
         if request.size > self.config.max_entry_bytes {
-            record_request_decision(self.backend.as_metric_label(), self.config.mode, "skip", "too_large", request.size);
+            record_plan_decision(self.backend.as_metric_label(), self.config.mode, "skip", "too_large", request.size);
             return ObjectDataCacheGetPlan::SkipTooLarge;
         }
 
-        record_request_decision(self.backend.as_metric_label(), self.config.mode, "cacheable", "eligible", request.size);
+        record_plan_decision(self.backend.as_metric_label(), self.config.mode, "cacheable", "eligible", request.size);
 
         ObjectDataCacheGetPlan::Cacheable {
             key: ObjectDataCacheKey::new(
@@ -106,11 +127,13 @@ impl ObjectDataCache {
         };
 
         self.stats.record_lookup(matches!(lookup, ObjectDataCacheLookup::Hit(_)));
-        self.refresh_entry_count();
+        // Do not refresh the cache-state gauge on the lookup hot path: moka's
+        // approximations do not change on a read, so it would only republish the
+        // same stale value on every GET (backlog#1134).
         match &lookup {
             ObjectDataCacheLookup::Hit(bytes) => {
                 let size_bytes = u64::try_from(bytes.len()).unwrap_or(u64::MAX);
-                record_request_decision(self.backend.as_metric_label(), self.config.mode, "hit", "cache_hit", size_bytes);
+                record_lookup_result(self.backend.as_metric_label(), self.config.mode, "hit", size_bytes);
                 record_hit_bytes(self.backend.as_metric_label(), self.config.mode, size_bytes);
             }
             ObjectDataCacheLookup::Miss => {
@@ -118,13 +141,13 @@ impl ObjectDataCache {
                     ObjectDataCacheGetPlan::Cacheable { key } => key.size,
                     _ => 0,
                 };
-                record_request_decision(self.backend.as_metric_label(), self.config.mode, "miss", "cache_miss", size_bytes);
+                record_lookup_result(self.backend.as_metric_label(), self.config.mode, "miss", size_bytes);
             }
             ObjectDataCacheLookup::SkipDisabled => {
-                record_request_decision(self.backend.as_metric_label(), self.config.mode, "skip", "lookup_disabled", 0);
+                record_lookup_result(self.backend.as_metric_label(), self.config.mode, "skip_disabled", 0);
             }
             ObjectDataCacheLookup::SkipNotCacheable => {
-                record_request_decision(self.backend.as_metric_label(), self.config.mode, "skip", "lookup_not_cacheable", 0);
+                record_lookup_result(self.backend.as_metric_label(), self.config.mode, "skip_not_cacheable", 0);
             }
         }
 
@@ -135,21 +158,19 @@ impl ObjectDataCache {
     pub async fn fill_body(&self, plan: &ObjectDataCacheGetPlan, bytes: Bytes) -> ObjectDataCacheFillResult {
         let fill_bytes = u64::try_from(bytes.len()).unwrap_or(u64::MAX);
         if !self.config.fill_enabled() {
-            record_fill_result(self.backend.as_metric_label(), self.config.mode, "skipped_by_mode", fill_bytes, 0.0);
-            return ObjectDataCacheFillResult::SkippedByMode;
+            // Never reached the backend: count the outcome, but record no bytes
+            // (nothing was submitted) and no duration (there was no fill work).
+            let result = ObjectDataCacheFillResult::SkippedByMode;
+            record_fill_result(self.backend.as_metric_label(), self.config.mode, result.as_metric_label(), 0, None);
+            return result;
         }
 
         if let ObjectDataCacheGetPlan::Cacheable { key } = plan
             && fill_bytes != key.size
         {
+            // Never reached the backend either: count only.
             let result = ObjectDataCacheFillResult::SkippedSizeMismatch;
-            record_fill_result(
-                self.backend.as_metric_label(),
-                self.config.mode,
-                result.as_metric_label(),
-                fill_bytes,
-                0.0,
-            );
+            record_fill_result(self.backend.as_metric_label(), self.config.mode, result.as_metric_label(), 0, None);
             return result;
         }
 
@@ -159,16 +180,39 @@ impl ObjectDataCache {
             ObjectDataCacheBackendKind::Moka(backend) => backend.fill_body(plan, bytes).await,
         };
 
-        if matches!(result, ObjectDataCacheFillResult::Inserted) {
-            self.stats.record_fill();
-        }
-        self.refresh_entry_count();
+        // Fill bytes and duration describe work the backend actually performed,
+        // so each outcome is listed explicitly rather than caught by a wildcard:
+        // a rejected fill wrote nothing and must not inflate fill_bytes_total.
+        // See backlog#1123.
+        let (recorded_bytes, duration) = match &result {
+            // Inserted the body: the only outcome that moves the entry count.
+            ObjectDataCacheFillResult::Inserted => {
+                self.stats.record_fill();
+                self.refresh_entry_count();
+                (fill_bytes, Some(fill_start.elapsed().as_secs_f64()))
+            }
+            // Reached the backend and wrote the body, then undid it. The bytes
+            // were written, so both are real.
+            ObjectDataCacheFillResult::SkippedInvalidationRace | ObjectDataCacheFillResult::SkippedIdentityOverflow => {
+                (fill_bytes, Some(fill_start.elapsed().as_secs_f64()))
+            }
+            // Rejected before writing anything: count the outcome, nothing else.
+            // A `JoinedInflightFill` is already counted by singleflight_joins,
+            // and its elapsed time would be wait time, not fill work.
+            ObjectDataCacheFillResult::JoinedInflightFill
+            | ObjectDataCacheFillResult::SkippedFillConcurrency
+            | ObjectDataCacheFillResult::SkippedMemoryPressure
+            | ObjectDataCacheFillResult::SkippedDisabled
+            | ObjectDataCacheFillResult::SkippedByMode
+            | ObjectDataCacheFillResult::SkippedNotCacheable
+            | ObjectDataCacheFillResult::SkippedSizeMismatch => (0, None),
+        };
         record_fill_result(
             self.backend.as_metric_label(),
             self.config.mode,
             result.as_metric_label(),
-            fill_bytes,
-            fill_start.elapsed().as_secs_f64(),
+            recorded_bytes,
+            duration,
         );
 
         result
@@ -186,8 +230,15 @@ impl ObjectDataCache {
         };
 
         self.stats.record_invalidation();
-        self.refresh_entry_count();
-        record_invalidation(self.backend.as_metric_label(), _reason.as_metric_label());
+        let outcome = invalidation_outcome(&result);
+        // A mutating op invalidates twice by design (before + after); the vast
+        // majority of those touch identities that were never cached. Only refresh
+        // the cache-state gauge when something was actually removed so the
+        // no-op path stays cheap (backlog#1141).
+        if outcome != INVALIDATION_OUTCOME_NOOP {
+            self.refresh_entry_count();
+        }
+        record_invalidation(self.backend.as_metric_label(), _reason.as_metric_label(), outcome);
 
         result
     }
@@ -207,7 +258,22 @@ impl ObjectDataCache {
         matches!(self.config.mode, crate::config::ObjectDataCacheMode::FillMaterializeEnabled)
     }
 
+    /// Publishes the cache-state gauge (and mirrors it into stats), debounced so
+    /// it fires at most once per [`ENTRY_COUNT_PUBLISH_DEBOUNCE_MS`]. Moka's
+    /// `entry_count`/`weighted_size` are approximations that only settle after
+    /// pending tasks run, so publishing on every fill/invalidate would restore
+    /// the same stale value thousands of times between scrapes (backlog#1134).
     fn refresh_entry_count(&self) {
+        let now_ms = u64::try_from(self.created_at.elapsed().as_millis())
+            .unwrap_or(u64::MAX)
+            // Reserve 0 as the "never published" sentinel.
+            .max(1);
+        let last = self.last_entry_publish_ms.load(Ordering::Relaxed);
+        if last != 0 && now_ms.saturating_sub(last) < ENTRY_COUNT_PUBLISH_DEBOUNCE_MS {
+            return;
+        }
+        self.last_entry_publish_ms.store(now_ms, Ordering::Relaxed);
+
         let (entries, weighted_bytes) = match &self.backend {
             ObjectDataCacheBackendKind::Noop(_) => (0, 0),
             ObjectDataCacheBackendKind::Moka(backend) => (backend.entry_count(), backend.weighted_size()),
@@ -217,6 +283,18 @@ impl ObjectDataCache {
     }
 }
 
+/// Maps an invalidation result to its metric outcome label.
+const fn invalidation_outcome(result: &ObjectDataCacheInvalidationResult) -> &'static str {
+    match result {
+        ObjectDataCacheInvalidationResult::Removed { keys } if *keys > 0 => INVALIDATION_OUTCOME_REMOVED,
+        ObjectDataCacheInvalidationResult::Removed { .. } | ObjectDataCacheInvalidationResult::NoOp => INVALIDATION_OUTCOME_NOOP,
+        // Transitional: a backend that has not yet been widened to report the
+        // removal count (`MokaBackend`) still returns `Success`. Treat it as a
+        // removal so the gauge keeps refreshing until the backend reports
+        // `Removed`/`NoOp` (see report / backlog#1141).
+    }
+}
+
 /// Protocol-neutral GET request metadata for cache planning.
 #[derive(Debug, Clone)]
 pub struct ObjectDataCacheGetRequest<'a> {
@@ -276,12 +354,17 @@ pub enum ObjectDataCacheFillResult {
     SkippedIdentityOverflow,
     /// Fill was skipped because the provided body length did not match the cache key identity.
     SkippedSizeMismatch,
-    /// Fill waiters were released without a published leader result.
-    SkippedSingleflightClosed,
     /// Fill was undone because an invalidation raced with the insert.
     SkippedInvalidationRace,
+    /// The caller joined an in-flight leader fill instead of performing one, so
+    /// it did no fill work of its own (the join is counted by
+    /// `singleflight_joins`). Distinguishes waiters from true inserts so N
+    /// concurrent GETs of one cold key do not record N inserts (backlog#1123).
+    JoinedInflightFill,
     /// The cache entry was inserted successfully.
     Inserted,
+    /// Fill was skipped because the fill-concurrency limiter was saturated.
+    SkippedFillConcurrency,
 }
 
 impl ObjectDataCacheFillResult {
@@ -293,9 +376,10 @@ impl ObjectDataCacheFillResult {
             Self::SkippedMemoryPressure => "skipped_memory_pressure",
             Self::SkippedIdentityOverflow => "skipped_identity_overflow",
             Self::SkippedSizeMismatch => "skipped_size_mismatch",
-            Self::SkippedSingleflightClosed => "skipped_singleflight_closed",
             Self::SkippedInvalidationRace => "skipped_invalidation_race",
+            Self::JoinedInflightFill => "joined_inflight",
             Self::Inserted => "inserted",
+            Self::SkippedFillConcurrency => "skipped_fill_concurrency",
         }
     }
 }
@@ -333,16 +417,26 @@ impl ObjectDataCacheInvalidationReason {
 /// Result of an invalidation request.
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
 pub enum ObjectDataCacheInvalidationResult {
-    /// Invalidation completed successfully.
-    Success,
+    /// Cache keys were removed for the identity.
+    Removed {
+        /// Number of cache keys dropped.
+        keys: usize,
+    },
+    /// The identity was not cached, so nothing was removed.
+    NoOp,
 }
 
 #[cfg(test)]
 mod tests {
-    use super::{ObjectDataCache, ObjectDataCacheFillResult, ObjectDataCacheGetRequest, ObjectDataCacheLookup};
+    use super::{
+        ObjectDataCache, ObjectDataCacheFillResult, ObjectDataCacheGetPlan, ObjectDataCacheGetRequest,
+        ObjectDataCacheInvalidationReason, ObjectDataCacheLookup,
+    };
     use crate::config::{ObjectDataCacheConfig, ObjectDataCacheMode};
-    use crate::key::ObjectDataCacheBodyVariant;
+    use crate::key::{ObjectDataCacheBodyVariant, ObjectDataCacheIdentity};
     use bytes::Bytes;
+    use metrics_util::MetricKind;
+    use metrics_util::debugging::{DebugValue, DebuggingRecorder};
 
     fn fill_enabled_cache() -> ObjectDataCache {
         let config = ObjectDataCacheConfig {
@@ -356,6 +450,192 @@ mod tests {
         ObjectDataCache::new(config).expect("fill-enabled cache config should initialize")
     }
 
+    fn hit_only_cache() -> ObjectDataCache {
+        let config = ObjectDataCacheConfig {
+            mode: ObjectDataCacheMode::HitOnly,
+            max_bytes: 8_388_608,
+            ..ObjectDataCacheConfig::default()
+        };
+        ObjectDataCache::new(config).expect("hit-only cache config should initialize")
+    }
+
+    fn plain_request<'a>(bucket: &'a str, object: &'a str, etag: &'a str, size: u64) -> ObjectDataCacheGetRequest<'a> {
+        ObjectDataCacheGetRequest {
+            bucket,
+            object,
+            version_id: None,
+            etag,
+            size,
+            body_variant: ObjectDataCacheBodyVariant::FullObjectPlainV1,
+        }
+    }
+
+    struct CapturedMetric {
+        kind: MetricKind,
+        name: String,
+        labels: Vec<(String, String)>,
+        value: DebugValue,
+    }
+
+    /// Runs `f` under a thread-local debugging recorder and a current-thread
+    /// runtime, so every metric the async body emits is captured without
+    /// touching the process-global registry. A current-thread runtime keeps the
+    /// spawned fill tasks on the recorder's thread.
+    fn capture_metrics<F, Fut>(f: F) -> Vec<CapturedMetric>
+    where
+        F: FnOnce() -> Fut,
+        Fut: std::future::Future<Output = ()>,
+    {
+        let recorder = DebuggingRecorder::new();
+        let snapshotter = recorder.snapshotter();
+        let runtime = tokio::runtime::Builder::new_current_thread()
+            .enable_all()
+            .build()
+            .expect("current-thread runtime should build");
+        metrics::with_local_recorder(&recorder, || runtime.block_on(f()));
+        snapshotter
+            .snapshot()
+            .into_vec()
+            .into_iter()
+            .map(|(composite, _unit, _desc, value)| {
+                let labels = composite
+                    .key()
+                    .labels()
+                    .map(|label| (label.key().to_string(), label.value().to_string()))
+                    .collect();
+                CapturedMetric {
+                    kind: composite.kind(),
+                    name: composite.key().name().to_string(),
+                    labels,
+                    value,
+                }
+            })
+            .collect()
+    }
+
+    fn counter_total(metrics: &[CapturedMetric], name: &str) -> Option<u64> {
+        let mut found = false;
+        let mut sum = 0u64;
+        for metric in metrics {
+            if metric.kind == MetricKind::Counter && metric.name == name {
+                found = true;
+                if let DebugValue::Counter(v) = metric.value {
+                    sum += v;
+                }
+            }
+        }
+        found.then_some(sum)
+    }
+
+    fn has_gauge(metrics: &[CapturedMetric], name: &str) -> bool {
+        metrics.iter().any(|m| m.kind == MetricKind::Gauge && m.name == name)
+    }
+
+    fn has_counter_with_label(metrics: &[CapturedMetric], name: &str, label: (&str, &str)) -> bool {
+        metrics.iter().any(|m| {
+            m.kind == MetricKind::Counter && m.name == name && m.labels.iter().any(|(k, v)| k == label.0 && v == label.1)
+        })
+    }
+
+    #[test]
+    fn single_get_records_one_plan_and_one_lookup_increment() {
+        // ODC-17: one GET must produce exactly one plan increment and one lookup
+        // increment, no longer conflated on a single requests_total counter.
+        let cache = hit_only_cache();
+        let metrics = capture_metrics(|| async {
+            let plan = cache.plan_get(plain_request("bucket", "object", "etag", 1024));
+            let _ = cache.lookup_body(&plan).await;
+        });
+
+        assert_eq!(
+            counter_total(&metrics, "rustfs_object_data_cache_plan_total"),
+            Some(1),
+            "exactly one plan decision per GET"
+        );
+        assert_eq!(
+            counter_total(&metrics, "rustfs_object_data_cache_lookup_total"),
+            Some(1),
+            "exactly one lookup outcome per GET"
+        );
+    }
+
+    #[test]
+    fn lookup_does_not_publish_entry_count_gauge() {
+        // ODC-29: the lookup hot path must not republish the cache-state gauge.
+        let cache = hit_only_cache();
+        let metrics = capture_metrics(|| async {
+            let plan = cache.plan_get(plain_request("bucket", "object", "etag", 1024));
+            let _ = cache.lookup_body(&plan).await;
+        });
+
+        assert!(
+            !has_gauge(&metrics, "rustfs_object_data_cache_entries"),
+            "lookup must not publish the entries gauge"
+        );
+    }
+
+    #[test]
+    fn joined_inflight_result_maps_to_metric_label() {
+        // ODC-18: the waiter outcome has its own, distinct metric label.
+        assert_eq!(ObjectDataCacheFillResult::JoinedInflightFill.as_metric_label(), "joined_inflight");
+        assert_ne!(
+            ObjectDataCacheFillResult::JoinedInflightFill.as_metric_label(),
+            ObjectDataCacheFillResult::Inserted.as_metric_label()
+        );
+    }
+
+    #[test]
+    fn disabled_cache_invalidation_labels_noop_and_skips_gauge() {
+        // ODC-36: invalidating an identity that was never cached is a no-op; it
+        // is labeled outcome=noop and must not refresh the cache-state gauge.
+        let cache = ObjectDataCache::disabled();
+        let metrics = capture_metrics(|| async {
+            let _ = cache
+                .invalidate_object(
+                    ObjectDataCacheIdentity::new("bucket", "object"),
+                    ObjectDataCacheInvalidationReason::BeforeMutation,
+                )
+                .await;
+        });
+
+        assert!(
+            has_counter_with_label(&metrics, "rustfs_object_data_cache_invalidations_total", ("outcome", "noop")),
+            "a no-op invalidation must be labeled outcome=noop"
+        );
+        assert!(
+            !has_gauge(&metrics, "rustfs_object_data_cache_entries"),
+            "a no-op invalidation must not refresh the entries gauge"
+        );
+    }
+
+    #[test]
+    fn invalidation_of_cached_identity_labels_removed() {
+        // ODC-36: invalidating an identity that held cached keys is labeled
+        // outcome=removed.
+        let cache = fill_enabled_cache();
+        let metrics = capture_metrics(|| async {
+            let plan = cache.plan_get(plain_request("bucket", "object", "etag", 5));
+            let ObjectDataCacheGetPlan::Cacheable { .. } = &plan else {
+                panic!("plan should be cacheable");
+            };
+            assert_eq!(
+                cache.fill_body(&plan, Bytes::from_static(b"hello")).await,
+                ObjectDataCacheFillResult::Inserted
+            );
+            let _ = cache
+                .invalidate_object(
+                    ObjectDataCacheIdentity::new("bucket", "object"),
+                    ObjectDataCacheInvalidationReason::AfterDeleteSuccess,
+                )
+                .await;
+        });
+
+        assert!(
+            has_counter_with_label(&metrics, "rustfs_object_data_cache_invalidations_total", ("outcome", "removed")),
+            "invalidating a cached identity must be labeled outcome=removed"
+        );
+    }
+
     #[tokio::test]
     async fn fill_body_rejects_size_mismatch() {
         let cache = fill_enabled_cache();