fix(object-data-cache): make memory container-aware and bound cache config (#4655)

* fix(object-data-cache): bound cache config and make memory container-aware Harden the object data cache configuration path so a single bad input degrades to the disabled adapter instead of OOM-killing pods, panicking at boot, or silently mis-sizing the cache (backlog#1110/1113/1114/1115/ 1127/1140/1130). - ODC-05: resolve capacity and the fill gate from the effective (container-aware) memory. Prefer sysinfo cgroup_limits() and use min(host, cgroup) as the total plus cgroup-derived availability, falling back to host values when no cgroup limit constrains. Log the resolved capacity and its basis once at startup. - ODC-08: cap ttl/time_to_idle at 30 days in validate() so an unbounded Duration can no longer trip moka's ~1000-year builder assertion. - ODC-09: drop the max_entry_bytes floor from the derived-capacity clamp so MAX_ENTRY_BYTES can no longer inflate total capacity above the safety clamp; reject an entry larger than the resolved capacity instead. - ODC-10: require an explicit max_bytes to clear max_entry_bytes plus the weigher overhead so a fillable-but-unretainable cache is rejected. - ODC-22: reject max_entry_bytes at/above the u32 weigher boundary. - ODC-35: warn (not reject) when time_to_idle exceeds ttl and document the min(ttl, time_to_idle) expiry interaction. - ODC-25: give numeric env overrides two-valued semantics via a new rustfs_utils::get_env_parse_outcome; a malformed value now disables the whole cache with one aggregated warning instead of silently keeping defaults. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(object-data-cache): require identity_keys_max of at least 2 The identity index admits a new key by evicting the oldest one, so a budget of 1 evicts the previous key on every fill and can never hold two live keys of one object at once — a versioned bucket alternating two versions then hits a permanent 0% hit rate. Reject the degenerate value at config validation time, where the adapter turns it into a disabled cache with a warning, since the bounded-eviction policy cannot rescue it at runtime. Handed off from the identity-index batch (backlog#1128), which changed the overflow policy but could not touch validate(). Refs: backlog#1115, backlog#1128 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(object-data-cache): let a zero free-memory floor opt out of the gate Making the memory gate container-aware turned it live in CI: the runners are Kubernetes pods, so the gate now reads the pod's cgroup free memory, finds it below the 20% floor, and refuses the fill. Tests that assert a fill succeeds then failed on CI while passing on a developer host, which has no cgroup and falls back to host memory. The gate is behaving correctly — the tests were the ones depending on a live memory reading. Treat min_free_memory_percent = 0 as a deliberate opt-out rather than an invalid value: allows_fill returns early before it touches any snapshot, so admission becomes independent of where the suite runs. Operators gain the same escape hatch. Every test that requires a fill to succeed now sets the floor to 0. The tests that exercise the gate keep it enabled via memory_gated_config, so the ODC-05 coverage they provide is preserved rather than short-circuited. Refs: backlog#1110 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(object-data-cache): opt the usecase fill tests out of the memory gate The previous commit exempted the fill-dependent tests it could find, but missed the six adapters built inside object_usecase.rs. Fixing the first batch moved nextest's fail-fast point forward and CI surfaced them: build_get_object_body_with_cache_materializes_once_and_hits_later and ..._uses_cached_body_without_reader_preread both assert a fill lands, and both ran with the default 20% free-memory floor. Set the floor to 0 on all six, matching the sibling test modules. Verified by pinning the gate's snapshot to 0% available — harsher than any CI pod — and confirming these tests still pass, which shows the exemption path never reads memory at all. An exhaustive sweep over every fill-enabled ObjectDataCacheConfig in the tree now shows no remaining site: the only unexempted ones are fill_enabled()'s matches! arm and two adapter tests that assert the adapter is disabled and therefore never fill. Refs: backlog#1110 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-19 19:16:17 +00:00 · 2026-07-10 17:17:32 +08:00
parent 15b8b13698
commit 904554b417
13 changed files with 686 additions and 41 deletions
@@ -13,12 +13,32 @@
 // limitations under the License.

 use crate::error::ObjectDataCacheConfigError;
+use crate::memory::{MemoryBasis, resolve_effective_memory};
+use std::sync::Once;
 use std::time::Duration;
-use sysinfo::System;

 const DEFAULT_DERIVED_MAX_MEMORY_PERCENT_CAP: u64 = 10;
 const DEFAULT_DERIVED_MAX_BYTES_CAP: u64 = 64 * 1024 * 1024 * 1024;

+/// Upper bound (seconds) for `ttl` / `time_to_idle`. Kept far below moka's
+/// ~1000-year builder assertion while remaining a sane operational cap so a
+/// bad env var degrades to the disabled adapter instead of panicking at boot.
+const MAX_DURATION_SECS: u64 = 30 * 24 * 60 * 60;
+
+/// Overhead reserved on top of a cached body when validating an explicit
+/// `max_bytes`. moka's weigher charges key bytes + a small per-entry overhead
+/// on top of the body, so `max_bytes` must clear `max_entry_bytes` by at least
+/// this margin for the entry to ever be retained.
+const ENTRY_WEIGHT_OVERHEAD_BYTES: u64 = 4096;
+
+/// Upper bound for `max_entry_bytes`. moka weighers return `u32`, so an entry
+/// above ~4 GiB would be under-weighted and bypass capacity accounting; stay
+/// below `u32::MAX` with room for the weigher overhead.
+const MAX_ENTRY_BYTES_LIMIT: u64 = u32::MAX as u64 - ENTRY_WEIGHT_OVERHEAD_BYTES;
+
+/// Guards the one-shot startup log of the resolved cache capacity.
+static RESOLVED_CAPACITY_LOGGED: Once = Once::new();
+
 /// Runtime mode for the object data cache.
 #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
 pub enum ObjectDataCacheMode {
@@ -56,16 +76,25 @@ pub struct ObjectDataCacheConfig {
    /// Maximum cacheable entry size in bytes.
    pub max_entry_bytes: u64,
    /// Time-to-live for a cache entry.
+    ///
+    /// moka expires an entry at `min(ttl, time_to_idle-since-last-access)`, so
+    /// a `time_to_idle` larger than `ttl` never takes effect.
    pub ttl: Duration,
    /// Time-to-idle for a cache entry.
+    ///
+    /// See [`ttl`](Self::ttl): expiration uses `min(ttl, time_to_idle)`, so
+    /// setting `time_to_idle` above `ttl` is inert.
    pub time_to_idle: Duration,
-    /// Minimum free memory percent before fill is paused.
+    /// Minimum free memory percent before fill is paused. Zero disables the
+    /// memory gate entirely, which makes fill admission independent of the
+    /// host's or container's live memory reading.
    pub min_free_memory_percent: u8,
    /// Fill concurrency multiplier applied to CPU count.
    pub fill_concurrency_per_cpu: u16,
    /// Absolute fill concurrency cap.
    pub fill_concurrency_max: u16,
-    /// Conservative cap for keys attached to one object identity.
+    /// Conservative cap for keys attached to one object identity. Must be at
+    /// least 2: the index admits a new key by evicting the oldest one.
    pub identity_keys_max: u16,
 }

@@ -111,16 +140,27 @@ impl ObjectDataCacheConfig {
            return Ok(self.max_bytes);
        }

-        let mut system = System::new();
-        system.refresh_memory();
-        let total_memory = system.total_memory();
+        // Resolve capacity from the effective (container-aware) total memory so
+        // a pod with a cgroup limit far below the node RAM does not size the
+        // cache to the node.
+        let effective = resolve_effective_memory();
+        let total_memory = effective.total_bytes;
        let derived = total_memory.saturating_mul(u64::from(self.max_memory_percent)) / 100;
-        let resolved = clamp_derived_max_bytes(derived, total_memory, self.max_entry_bytes);
+        let resolved = clamp_derived_max_bytes(derived, total_memory);

        if resolved == 0 {
            return Err(ObjectDataCacheConfigError::ZeroResolvedMaxBytes);
        }

+        // The derived capacity is no longer floored by `max_entry_bytes` (that
+        // used to silently inflate the cache above the safety clamp). If a
+        // single entry cannot fit, reject rather than inflate.
+        if self.max_entry_bytes > resolved {
+            return Err(ObjectDataCacheConfigError::MaxEntryBytesExceedsCapacity);
+        }
+
+        log_resolved_capacity_once(resolved, total_memory, effective.basis);
+
        Ok(resolved)
    }

@@ -134,15 +174,46 @@ impl ObjectDataCacheConfig {
            return Err(ObjectDataCacheConfigError::ZeroMaxEntryBytes);
        }

+        if self.max_entry_bytes > MAX_ENTRY_BYTES_LIMIT {
+            return Err(ObjectDataCacheConfigError::MaxEntryBytesTooLarge);
+        }
+
+        // An explicit capacity must leave room for a full entry plus the
+        // weigher overhead, otherwise moka can never retain the entry while
+        // fills still report success.
+        if self.max_bytes > 0 && self.max_bytes < self.max_entry_bytes.saturating_add(ENTRY_WEIGHT_OVERHEAD_BYTES) {
+            return Err(ObjectDataCacheConfigError::MaxEntryBytesExceedsMaxBytes);
+        }
+
        if self.ttl.is_zero() {
            return Err(ObjectDataCacheConfigError::ZeroTimeToLiveSecs);
        }

+        if self.ttl.as_secs() > MAX_DURATION_SECS {
+            return Err(ObjectDataCacheConfigError::TimeToLiveTooLarge);
+        }
+
        if self.time_to_idle.is_zero() {
            return Err(ObjectDataCacheConfigError::ZeroTimeToIdleSecs);
        }

-        if self.min_free_memory_percent == 0 || self.min_free_memory_percent > 100 {
+        if self.time_to_idle.as_secs() > MAX_DURATION_SECS {
+            return Err(ObjectDataCacheConfigError::TimeToIdleTooLarge);
+        }
+
+        // moka expires at min(ttl, time_to_idle); a larger time_to_idle is
+        // inert. Warn instead of rejecting so a benign misconfiguration still
+        // starts the cache.
+        if self.time_to_idle > self.ttl {
+            tracing::warn!(
+                time_to_idle_secs = self.time_to_idle.as_secs(),
+                ttl_secs = self.ttl.as_secs(),
+                "object data cache time_to_idle exceeds ttl; moka expires at min(ttl, time_to_idle) so the larger time_to_idle has no effect"
+            );
+        }
+
+        // Zero is a deliberate opt-out of the memory gate, not an invalid value.
+        if self.min_free_memory_percent > 100 {
            return Err(ObjectDataCacheConfigError::InvalidMinFreeMemoryPercent);
        }

@@ -162,15 +233,33 @@ impl ObjectDataCacheConfig {
            return Err(ObjectDataCacheConfigError::ZeroIdentityKeysMax);
        }

+        // The identity index evicts the oldest key to admit a new one, so a
+        // budget of 1 evicts the previous key on every fill and can never hold
+        // two live keys of one object at once.
+        if self.identity_keys_max < 2 {
+            return Err(ObjectDataCacheConfigError::IdentityKeysMaxTooSmall);
+        }
+
        Ok(())
    }
 }

-fn clamp_derived_max_bytes(derived: u64, total_memory: u64, max_entry_bytes: u64) -> u64 {
+fn clamp_derived_max_bytes(derived: u64, total_memory: u64) -> u64 {
    let percent_cap = total_memory.saturating_mul(DEFAULT_DERIVED_MAX_MEMORY_PERCENT_CAP) / 100;
-    let safe_cap = percent_cap.min(DEFAULT_DERIVED_MAX_BYTES_CAP).max(max_entry_bytes);
+    let safe_cap = percent_cap.min(DEFAULT_DERIVED_MAX_BYTES_CAP);

-    derived.min(safe_cap).max(max_entry_bytes)
+    derived.min(safe_cap)
+}
+
+fn log_resolved_capacity_once(resolved_max_bytes: u64, effective_total_bytes: u64, basis: MemoryBasis) {
+    RESOLVED_CAPACITY_LOGGED.call_once(|| {
+        tracing::info!(
+            resolved_max_bytes,
+            effective_total_bytes,
+            basis = basis.as_str(),
+            "object data cache resolved capacity"
+        );
+    });
 }

 #[cfg(test)]
@@ -260,6 +349,54 @@ mod tests {
        assert_eq!(err, ObjectDataCacheConfigError::FillConcurrencyMaxTooSmall);
    }

+    #[test]
+    fn validate_accepts_zero_min_free_memory_percent_as_gate_opt_out() {
+        let config = ObjectDataCacheConfig {
+            min_free_memory_percent: 0,
+            ..ObjectDataCacheConfig::default()
+        };
+
+        assert!(config.validate().is_ok());
+    }
+
+    #[test]
+    fn validate_rejects_min_free_memory_percent_above_100() {
+        let config = ObjectDataCacheConfig {
+            min_free_memory_percent: 101,
+            ..ObjectDataCacheConfig::default()
+        };
+
+        let err = config
+            .validate()
+            .expect_err("a free-memory floor above 100% is unsatisfiable");
+
+        assert_eq!(err, ObjectDataCacheConfigError::InvalidMinFreeMemoryPercent);
+    }
+
+    #[test]
+    fn validate_rejects_single_key_identity_budget() {
+        let config = ObjectDataCacheConfig {
+            identity_keys_max: 1,
+            ..ObjectDataCacheConfig::default()
+        };
+
+        let err = config
+            .validate()
+            .expect_err("a one-key identity budget evicts the previous key on every fill");
+
+        assert_eq!(err, ObjectDataCacheConfigError::IdentityKeysMaxTooSmall);
+    }
+
+    #[test]
+    fn validate_accepts_two_key_identity_budget() {
+        let config = ObjectDataCacheConfig {
+            identity_keys_max: 2,
+            ..ObjectDataCacheConfig::default()
+        };
+
+        assert!(config.validate().is_ok());
+    }
+
    #[test]
    fn validate_accepts_explicit_byte_cap() {
        let config = ObjectDataCacheConfig {
@@ -287,25 +424,129 @@ mod tests {
    }

    #[test]
-    fn resolved_max_bytes_is_at_least_max_entry_bytes() {
+    fn derived_capacity_is_not_inflated_by_max_entry_bytes() {
+        // 512 MiB effective memory with a tiny percent yields a small derived
+        // capacity. A large max_entry_bytes must NOT raise the total capacity
+        // (the old `.max(max_entry_bytes)` floor did exactly that).
+        let host = 512_u64 * 1024 * 1024;
+        let derived = host / 100;
+        let resolved = clamp_derived_max_bytes(derived, host);
+
+        assert_eq!(resolved, derived);
+        assert!(resolved < 1024 * 1024 * 1024);
+    }
+
+    #[test]
+    fn resolved_max_bytes_rejects_entry_larger_than_capacity() {
+        // Derived capacity is clamped to at most 64 GiB, so a 128 GiB entry cap
+        // can never fit regardless of the test host's memory.
        let config = ObjectDataCacheConfig {
            max_bytes: 0,
-            max_memory_percent: 1,
-            max_entry_bytes: 8_388_608,
+            max_memory_percent: 100,
+            max_entry_bytes: 128 * 1024 * 1024 * 1024,
            ..ObjectDataCacheConfig::default()
        };

-        let resolved = config.resolved_max_bytes().expect("derived capacity should stay positive");
+        let err = config
+            .resolved_max_bytes()
+            .expect_err("entry cap above the derived capacity must be rejected");

-        assert!(resolved >= config.max_entry_bytes);
+        assert_eq!(err, ObjectDataCacheConfigError::MaxEntryBytesExceedsCapacity);
    }

    #[test]
    fn derived_max_bytes_clamps_to_v3_safe_cap() {
        let one_tib = 1024_u64 * 1024 * 1024 * 1024;
        let derived = one_tib / 2;
-        let resolved = clamp_derived_max_bytes(derived, one_tib, 1_048_576);
+        let resolved = clamp_derived_max_bytes(derived, one_tib);

        assert_eq!(resolved, DEFAULT_DERIVED_MAX_BYTES_CAP);
    }
+
+    #[test]
+    fn validate_rejects_ttl_above_upper_bound() {
+        let config = ObjectDataCacheConfig {
+            ttl: Duration::from_secs(u64::MAX),
+            ..ObjectDataCacheConfig::default()
+        };
+
+        let err = config.validate().expect_err("ttl above the operational cap must be rejected");
+
+        assert_eq!(err, ObjectDataCacheConfigError::TimeToLiveTooLarge);
+    }
+
+    #[test]
+    fn validate_rejects_time_to_idle_above_upper_bound() {
+        let config = ObjectDataCacheConfig {
+            time_to_idle: Duration::from_secs(u64::MAX),
+            ..ObjectDataCacheConfig::default()
+        };
+
+        let err = config
+            .validate()
+            .expect_err("time-to-idle above the operational cap must be rejected");
+
+        assert_eq!(err, ObjectDataCacheConfigError::TimeToIdleTooLarge);
+    }
+
+    #[test]
+    fn validate_rejects_entry_above_weigher_limit() {
+        let config = ObjectDataCacheConfig {
+            max_entry_bytes: u32::MAX as u64,
+            ..ObjectDataCacheConfig::default()
+        };
+
+        let err = config
+            .validate()
+            .expect_err("entry size at the u32 weigher boundary must be rejected");
+
+        assert_eq!(err, ObjectDataCacheConfigError::MaxEntryBytesTooLarge);
+    }
+
+    #[test]
+    fn validate_rejects_entry_larger_than_explicit_max_bytes() {
+        let config = ObjectDataCacheConfig {
+            mode: ObjectDataCacheMode::HitOnly,
+            max_bytes: 2 * 1024 * 1024,
+            max_memory_percent: 0,
+            max_entry_bytes: 8 * 1024 * 1024,
+            ..ObjectDataCacheConfig::default()
+        };
+
+        let err = config
+            .validate()
+            .expect_err("an entry larger than max_bytes can never be retained");
+
+        assert_eq!(err, ObjectDataCacheConfigError::MaxEntryBytesExceedsMaxBytes);
+    }
+
+    #[test]
+    fn validate_rejects_entry_equal_to_explicit_max_bytes() {
+        // Equal fails because the weigher adds key bytes + per-entry overhead.
+        let config = ObjectDataCacheConfig {
+            mode: ObjectDataCacheMode::HitOnly,
+            max_bytes: 4 * 1024 * 1024,
+            max_memory_percent: 0,
+            max_entry_bytes: 4 * 1024 * 1024,
+            ..ObjectDataCacheConfig::default()
+        };
+
+        let err = config
+            .validate()
+            .expect_err("max_entry_bytes must clear max_bytes by the weigher overhead");
+
+        assert_eq!(err, ObjectDataCacheConfigError::MaxEntryBytesExceedsMaxBytes);
+    }
+
+    #[test]
+    fn validate_accepts_time_to_idle_greater_than_ttl() {
+        // A time_to_idle above ttl is inert (warned, not rejected).
+        let config = ObjectDataCacheConfig {
+            ttl: Duration::from_secs(30),
+            time_to_idle: Duration::from_secs(60),
+            ..ObjectDataCacheConfig::default()
+        };
+
+        assert!(config.validate().is_ok());
+    }
 }