fix(object-data-cache): make the GET key write-unique and dedup the lookup (#4693)

fix(object-data-cache): make GET body cache key write-unique and dedup lookups

Address four object-data-cache GET-path findings (backlog#1107 batch):

ODC-06 (backlog#1111): the cache key was content-unique, not write-unique.
Extend ObjectDataCacheKey with the resolved version's modification time
(i128 unix nanoseconds, None -> 0), derived once in the shared planner so the
ecstore hook and the usecase layer produce an identical key. An unversioned
overwrite advances mod_time, so a stale node can no longer serve old bytes for
up to the TTL under an MD5 collision; etag + size stay as belt-and-braces.

ODC-16 (backlog#1121): every cacheable GET planned and looked up twice (once in
the ecstore hook, once in the usecase layer), double-counting hits, hit_bytes
and lookups. GetObjectReader now carries a GetObjectBodySource marker
(Unprobed / HookMissed / HookServed); the hook stamps it, and
build_get_object_body_with_cache serves a hook-served body directly and skips
its lookup whenever the hook already probed. One hook-served GET now records
exactly one lookup.

ODC-19 (backlog#1124): ENABLE=true with no explicit mode defaulted to HitOnly,
which never fills and keeps a permanent 0% hit rate. Default to
FillBufferedOnly, log the resolved mode at startup, and warn when HitOnly is
selected explicitly.

ODC-24 (backlog#1129): max_entry_bytes above the in-memory GET fill limits was
silently inert. Clamp the planner's size eligibility to
min(max_entry_bytes, seek-support threshold, 64 MiB buffer cap) so ineligible
sizes plan SkipTooLarge instead of being reported eligible, and warn at startup
when the excess is inert.

Co-authored-by: heihutu <heihutu@gmail.com>

crates/object-data-cache/src/key.rs

@@ -26,6 +26,16 @@ pub enum ObjectDataCacheBodyVariant {
 }
 
 /// Stable cache key for a reusable object body.
+///
+/// The key is *write-unique*, not merely *content-unique* (backlog#1111 /
+/// ODC-06). `etag + size` alone identify content: for an unversioned overwrite
+/// two same-length payloads that collide on MD5 would derive the identical key,
+/// so a GET on a node that never observed the overwrite could serve the old
+/// bytes for up to the TTL, and the same collision turns the
+/// fill-after-invalidation race (backlog#1118) into a serving bug. Including
+/// the resolved version's modification time distinguishes two writes even under
+/// an MD5 collision, because an overwrite advances `mod_time`. `etag + size`
+/// stay in the key as belt-and-braces.
 #[derive(Debug, Clone, PartialEq, Eq, Hash)]
 pub struct ObjectDataCacheKey {
     /// Bucket name.
@@ -38,6 +48,11 @@ pub struct ObjectDataCacheKey {
     pub etag: Arc<str>,
     /// Object size in bytes.
     pub size: u64,
+    /// Resolved version's modification time as Unix nanoseconds
+    /// (`OffsetDateTime::unix_timestamp_nanos`), or `0` when absent. This is the
+    /// write-unique component: an overwrite advances `mod_time`, so the key
+    /// changes even when `etag + size` are unchanged (an MD5 collision).
+    pub mod_time_unix_nanos: i128,
     /// Cached body semantics.
     pub body_variant: ObjectDataCacheBodyVariant,
 }
@@ -52,13 +67,17 @@ pub struct ObjectDataCacheIdentity {
 }
 
 impl ObjectDataCacheKey {
-    /// Creates a new stable object data cache key.
-    pub fn new(
+    /// Creates a stable object data cache key with an explicit modification
+    /// time. This is the full constructor; the production GET planner uses it so
+    /// the key is write-unique (backlog#1111 / ODC-06).
+    #[allow(clippy::too_many_arguments)]
+    pub fn with_mod_time(
         bucket: impl Into<Arc<str>>,
         object: impl Into<Arc<str>>,
         version_id: Option<&str>,
         etag: impl Into<Arc<str>>,
         size: u64,
+        mod_time_unix_nanos: i128,
         body_variant: ObjectDataCacheBodyVariant,
     ) -> Self {
         Self {
@@ -67,10 +86,25 @@ impl ObjectDataCacheKey {
             version_id: version_id.map_or_else(|| Arc::<str>::from(NULL_VERSION_ID), Arc::<str>::from),
             etag: etag.into(),
             size,
+            mod_time_unix_nanos,
             body_variant,
         }
     }
 
+    /// Creates a stable object data cache key with no modification time
+    /// (`mod_time_unix_nanos == 0`). Retained for callers that key purely by
+    /// content identity (index/backend tests).
+    pub fn new(
+        bucket: impl Into<Arc<str>>,
+        object: impl Into<Arc<str>>,
+        version_id: Option<&str>,
+        etag: impl Into<Arc<str>>,
+        size: u64,
+        body_variant: ObjectDataCacheBodyVariant,
+    ) -> Self {
+        Self::with_mod_time(bucket, object, version_id, etag, size, 0, body_variant)
+    }
+
     /// Returns true when this key targets the canonical unversioned body variant.
     ///
     /// Only exercised by tests; gated so it is not compiled into the shipping
@@ -112,6 +146,51 @@ mod tests {
         assert_ne!(latest, versioned);
     }
 
+    #[test]
+    fn key_distinguishes_writes_by_mod_time() {
+        // ODC-06 (backlog#1111): two writes with identical etag + size (an MD5
+        // collision on an unversioned overwrite) must derive different keys once
+        // the modification time differs, so a stale node cannot serve old bytes.
+        let old = ObjectDataCacheKey::with_mod_time(
+            "bucket",
+            "object",
+            None,
+            "etag",
+            42,
+            1_000,
+            ObjectDataCacheBodyVariant::FullObjectPlainV1,
+        );
+        let new = ObjectDataCacheKey::with_mod_time(
+            "bucket",
+            "object",
+            None,
+            "etag",
+            42,
+            2_000,
+            ObjectDataCacheBodyVariant::FullObjectPlainV1,
+        );
+
+        assert_ne!(old, new, "keys differing only by mod_time must not collide");
+        // Same mod_time still collapses to one key (a true re-read of one write).
+        let new_again = ObjectDataCacheKey::with_mod_time(
+            "bucket",
+            "object",
+            None,
+            "etag",
+            42,
+            2_000,
+            ObjectDataCacheBodyVariant::FullObjectPlainV1,
+        );
+        assert_eq!(new, new_again);
+    }
+
+    #[test]
+    fn key_new_defaults_mod_time_to_zero() {
+        let key = ObjectDataCacheKey::new("bucket", "object", None, "etag", 42, ObjectDataCacheBodyVariant::FullObjectPlainV1);
+
+        assert_eq!(key.mod_time_unix_nanos, 0);
+    }
+
     #[test]
     fn identity_new_preserves_bucket_and_object() {
         let identity = ObjectDataCacheIdentity::new("bucket", "object");