mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-07 05:43:14 +00:00
perf(object-data-cache): take fill off the GET path and fix the metrics (#4678)
* fix(object-data-cache): make cache metrics one-increment-per-GET
Rework the object data cache observability so each counter carries one
clear meaning, and drop dead per-entry state.
ODC-17 (backlog#1122): split requests_total, which was incremented by
both plan_get and lookup_body, into plan_total{decision,reason,size_class}
(emitted only by plan_get) and lookup_total{result,size_class} (emitted
only by lookup_body). Each is now incremented exactly once per GET per
layer; help text states this.
ODC-18 (backlog#1123): add a JoinedInflightFill result (label
joined_inflight) so a singleflight waiter is no longer counted as an
insert. record_fill_result now always counts the outcome but records
fill bytes only when non-zero and the duration histogram only when
present, so joined_inflight / skipped_by_mode / skipped_size_mismatch
no longer inflate fill_bytes_total or add non-fill duration samples.
The waiter->JoinedInflightFill mapping lives in MokaBackend (owned by a
concurrent branch); cache.rs handles the variant already.
ODC-29 (backlog#1134): stop refreshing the cache-state gauge on every
lookup, and debounce it on fill/invalidate to at most once per second
via an AtomicU64 millis timestamp, since moka's entry_count is a
settling approximation.
ODC-36 (backlog#1141): give invalidations_total an outcome label
(removed|noop) and skip the gauge refresh on the no-op path. Extend
ObjectDataCacheInvalidationResult with Removed{keys}/NoOp (Success kept
as a transitional variant until MokaBackend reports the removal count);
NoopBackend now reports NoOp.
ODC-30 (backlog#1141): drop the dead content_length/etag/inserted_at
fields (and getters) from ObjectDataCacheEntry, which are redundant with
the moka key identity; the constructor keeps its arity so the
out-of-scope MokaBackend caller still compiles. Gate is_null_version
behind cfg(test).
Tests use a thread-local metrics recorder to avoid the global-singleton
flakiness. Fill-enabled test configs set min_free_memory_percent=0 so
the cgroup gate does not refuse fills in CI pods.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): move fill off GET path, bound & harden the fill
Implements five object-data-cache audit findings.
ODC-15 (backlog#1120): the GET response no longer blocks on cache fill.
The buffered/materialized body is already in hand, so the fill now runs
in a detached task and the response is built immediately. Singleflight is
made non-blocking: `try_acquire` returns Leader or Busy, and a non-leader
skips its own fill (SkippedSingleflightBusy) instead of waiting on another
request's leader. The inner fill spawn is KEPT: once the index key is
registered, the recheck/undo must complete even if the enclosing fill
future is aborted, so removing it would weaken the cancellation-safety
guarantee (ODC-13 test) and the invalidation-race undo.
ODC-11 (backlog#1116): enforce the fill-concurrency knobs. MokaBackend
now holds a Semaphore sized min(per_cpu * parallelism, max), acquired
after winning leadership and before the memory gate. On saturation it
rejects (try_acquire_owned) with SkippedFillConcurrency rather than
queueing, so the fill path never reintroduces GET-latency coupling.
ODC-14 (backlog#1119): the memory gate no longer does a blocking sysinfo
refresh under a mutex on the async fill path. A dedicated periodic
refresher (tokio interval + spawn_blocking) updates an atomic snapshot
every 5s; allows_fill is now lock-free. The min_free_memory_percent == 0
short-circuit still runs before any snapshot read. No runtime at
construction (sync unit tests) simply keeps the seed snapshot.
ODC-32 (backlog#1137): singleflight no longer emits metrics while holding
the map mutex. try_acquire/remove_entry capture the map length under the
guard, drop it, then emit the gauge/counter.
ODC-07 (backlog#1112): the materialize read is bounded via
take(capacity + 1) so an over-long stream cannot grow the buffer past the
in-memory GET threshold, and any length mismatch is now a hard error
matching the direct-memory GET path, instead of warn-and-serve.
cache.rs is owned by a concurrent branch; this commit only appends the
SkippedFillConcurrency and SkippedSingleflightBusy variants + label arms.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): report JoinedInflightFill and invalidation outcome
Reconciles the moka_backend half of two metrics-semantics findings after
merging fix/odc-metrics-semantics (which landed the cache.rs half).
ODC-18 (backlog#1123): the singleflight non-leader path now returns
JoinedInflightFill instead of counting as an insert, so N concurrent GETs
of one cold key record one insert, not N. This composes with the ODC-15
redesign: the non-leader does NOT wait for the leader (it already owns the
body and never re-serves from the fill result), so no blocking wait is
reintroduced. Drops the redundant local SkippedSingleflightBusy variant in
favor of the canonical JoinedInflightFill (mapped to no bytes / no duration
by the facade). SkippedFillConcurrency (ODC-11) is retained.
ODC-36 (backlog#1141): MokaBackend::invalidate_object now returns
Removed { keys } / NoOp instead of the transitional Success, so the facade
labels invalidations_total{outcome=removed|noop} and skips the cache-state
gauge refresh on the no-op path.
Tests: duplicate-fill test asserts JoinedInflightFill (bites when reverted
to Inserted); new no-op invalidation test; matching-identity test asserts
Removed { keys: 1 }.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): drop unused mut on materialize stream binding
The ODC-07 change moves `final_stream` into `AsyncReadExt::take`, so the
`build_get_object_body_with_cache` parameter is no longer mutated in place.
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(object-data-cache): close the cross-branch coordination seams
Merging the metrics and hot-path branches left four transitional shims that
only existed because each side could not edit the other's files. With both
sides present they are dead weight, and two of them actively misreport.
- ObjectDataCacheEntry::new took content_length and etag only to keep the
caller in moka_backend.rs compiling, then discarded both. Drop them; the
entry is a Bytes wrapper and the caller now passes only the body.
- SkippedSingleflightClosed lost its only producer when the waiter model was
replaced by Leader/Busy election. Remove the variant.
- The fill-accounting match ended in a wildcard that charged fill_bytes and a
duration to every non-Inserted outcome, so a fill rejected by the memory
gate or the concurrency semaphore — which never touched the backend and
wrote nothing — still inflated fill_bytes_total. Enumerate every variant
explicitly: only outcomes that wrote the body report bytes and duration.
Exhaustiveness also forces a future variant to state its accounting rather
than inherit a wrong default.
- InvalidationResult::Success mapped a no-op to outcome=removed, the exact lie
backlog#1141 set out to fix, and its doc comment claimed MokaBackend still
returned it after that backend had been widened. It has no producer left.
Remove it, and tighten the app-layer test from "any successful variant" to
the real contract: the pre-mutation call reports Removed{keys:1}, the
post-delete call NoOp.
Refs: backlog#1123, backlog#1141, backlog#1135
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
@@ -15,9 +15,11 @@
|
||||
use crate::config::ObjectDataCacheConfig;
|
||||
use crate::metrics::record_memory_pressure;
|
||||
use crate::stats::ObjectDataCacheStats;
|
||||
use std::sync::Arc;
|
||||
#[cfg(test)]
|
||||
use std::sync::Mutex;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::{Duration, Instant};
|
||||
use std::time::Duration;
|
||||
use sysinfo::System;
|
||||
|
||||
const DEFAULT_REFRESH_INTERVAL: Duration = Duration::from_secs(5);
|
||||
@@ -110,16 +112,60 @@ impl ObjectDataCacheMemorySnapshot {
|
||||
}
|
||||
}
|
||||
|
||||
/// Memory gate that keeps a cheap snapshot for fill-path checks.
|
||||
/// Lock-free memory snapshot shared between the gate and its refresher task.
|
||||
#[derive(Debug)]
|
||||
struct MemorySnapshotCell {
|
||||
total_bytes: AtomicU64,
|
||||
available_bytes: AtomicU64,
|
||||
}
|
||||
|
||||
impl MemorySnapshotCell {
|
||||
fn new(snapshot: ObjectDataCacheMemorySnapshot) -> Self {
|
||||
Self {
|
||||
total_bytes: AtomicU64::new(snapshot.total_bytes),
|
||||
available_bytes: AtomicU64::new(snapshot.available_bytes),
|
||||
}
|
||||
}
|
||||
|
||||
fn store(&self, snapshot: ObjectDataCacheMemorySnapshot) {
|
||||
self.total_bytes.store(snapshot.total_bytes, Ordering::Relaxed);
|
||||
self.available_bytes.store(snapshot.available_bytes, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
fn load(&self) -> ObjectDataCacheMemorySnapshot {
|
||||
ObjectDataCacheMemorySnapshot {
|
||||
total_bytes: self.total_bytes.load(Ordering::Relaxed),
|
||||
available_bytes: self.available_bytes.load(Ordering::Relaxed),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Aborts the periodic refresher when the gate (and thus the cache) is dropped.
|
||||
#[derive(Debug)]
|
||||
struct RefresherGuard(tokio::task::JoinHandle<()>);
|
||||
|
||||
impl Drop for RefresherGuard {
|
||||
fn drop(&mut self) {
|
||||
self.0.abort();
|
||||
}
|
||||
}
|
||||
|
||||
/// Memory gate that keeps a cheap, lock-free snapshot for fill-path checks.
|
||||
///
|
||||
/// The snapshot is sampled off the fill path by a dedicated periodic refresher
|
||||
/// (see [`spawn_refresher`](ObjectDataCacheMemoryGate::spawn_refresher)) that
|
||||
/// runs the blocking `sysinfo` read on a `spawn_blocking` thread. `allows_fill`
|
||||
/// only reads atomics, so it never blocks a tokio worker and concurrent fills
|
||||
/// never serialize on a refresh.
|
||||
#[derive(Debug)]
|
||||
pub struct ObjectDataCacheMemoryGate {
|
||||
system: Mutex<System>,
|
||||
last_refresh: Mutex<Instant>,
|
||||
snapshot_total_bytes: AtomicU64,
|
||||
snapshot_available_bytes: AtomicU64,
|
||||
snapshot: Arc<MemorySnapshotCell>,
|
||||
min_free_memory_percent: u8,
|
||||
refresh_interval: Duration,
|
||||
stats: Arc<ObjectDataCacheStats>,
|
||||
/// Kept alive so the refresher runs for the gate's lifetime; `None` when the
|
||||
/// gate is opted out (`min_free_memory_percent == 0`) or no tokio runtime is
|
||||
/// available at construction (e.g. synchronous unit tests).
|
||||
_refresher: Option<RefresherGuard>,
|
||||
#[cfg(test)]
|
||||
test_override: Mutex<Option<ObjectDataCacheMemorySnapshot>>,
|
||||
}
|
||||
@@ -127,27 +173,59 @@ pub struct ObjectDataCacheMemoryGate {
|
||||
impl ObjectDataCacheMemoryGate {
|
||||
/// Creates a new memory gate.
|
||||
pub fn new(config: &ObjectDataCacheConfig, stats: Arc<ObjectDataCacheStats>) -> Self {
|
||||
let mut system = System::new();
|
||||
system.refresh_memory();
|
||||
let effective = effective_memory_from_system(&system);
|
||||
let snapshot = ObjectDataCacheMemorySnapshot {
|
||||
// Seed the snapshot once at construction. This is the only blocking
|
||||
// sysinfo read on any caller's stack; all later refreshes run off-path.
|
||||
let effective = resolve_effective_memory();
|
||||
let snapshot = Arc::new(MemorySnapshotCell::new(ObjectDataCacheMemorySnapshot {
|
||||
total_bytes: effective.total_bytes,
|
||||
available_bytes: effective.available_bytes,
|
||||
}));
|
||||
|
||||
let min_free_memory_percent = config.min_free_memory_percent;
|
||||
// A zero floor opts out of the gate entirely, so there is nothing to
|
||||
// refresh; skip the background task in that case.
|
||||
let refresher = if min_free_memory_percent == 0 {
|
||||
None
|
||||
} else {
|
||||
Self::spawn_refresher(Arc::clone(&snapshot), DEFAULT_REFRESH_INTERVAL)
|
||||
};
|
||||
|
||||
Self {
|
||||
system: Mutex::new(system),
|
||||
last_refresh: Mutex::new(Instant::now()),
|
||||
snapshot_total_bytes: AtomicU64::new(snapshot.total_bytes),
|
||||
snapshot_available_bytes: AtomicU64::new(snapshot.available_bytes),
|
||||
min_free_memory_percent: config.min_free_memory_percent,
|
||||
refresh_interval: DEFAULT_REFRESH_INTERVAL,
|
||||
snapshot,
|
||||
min_free_memory_percent,
|
||||
stats,
|
||||
_refresher: refresher,
|
||||
#[cfg(test)]
|
||||
test_override: Mutex::new(None),
|
||||
}
|
||||
}
|
||||
|
||||
/// Spawns the periodic refresher, returning `None` when no tokio runtime is
|
||||
/// available (the gate then keeps its construction-time seed snapshot).
|
||||
fn spawn_refresher(snapshot: Arc<MemorySnapshotCell>, interval: Duration) -> Option<RefresherGuard> {
|
||||
let handle = tokio::runtime::Handle::try_current().ok()?;
|
||||
let task = handle.spawn(async move {
|
||||
let mut ticker = tokio::time::interval(interval);
|
||||
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
|
||||
// The first tick fires immediately; the seed snapshot is already in
|
||||
// place, so refresh from the second tick onward.
|
||||
ticker.tick().await;
|
||||
loop {
|
||||
ticker.tick().await;
|
||||
// Run the blocking /proc/meminfo read off the async worker. A
|
||||
// join error only happens if the runtime is shutting down; keep
|
||||
// the last snapshot rather than clobbering it in that case.
|
||||
if let Ok(effective) = tokio::task::spawn_blocking(resolve_effective_memory).await {
|
||||
snapshot.store(ObjectDataCacheMemorySnapshot {
|
||||
total_bytes: effective.total_bytes,
|
||||
available_bytes: effective.available_bytes,
|
||||
});
|
||||
}
|
||||
}
|
||||
});
|
||||
Some(RefresherGuard(task))
|
||||
}
|
||||
|
||||
/// Returns the current atomic memory snapshot.
|
||||
pub fn snapshot(&self) -> ObjectDataCacheMemorySnapshot {
|
||||
#[cfg(test)]
|
||||
@@ -157,44 +235,21 @@ impl ObjectDataCacheMemoryGate {
|
||||
}
|
||||
}
|
||||
|
||||
ObjectDataCacheMemorySnapshot {
|
||||
total_bytes: self.snapshot_total_bytes.load(Ordering::Relaxed),
|
||||
available_bytes: self.snapshot_available_bytes.load(Ordering::Relaxed),
|
||||
}
|
||||
}
|
||||
|
||||
/// Refreshes the snapshot if it is stale.
|
||||
pub fn refresh_if_stale(&self) {
|
||||
{
|
||||
let last_refresh = lock_or_recover(&self.last_refresh);
|
||||
if last_refresh.elapsed() < self.refresh_interval {
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
let mut system = lock_or_recover(&self.system);
|
||||
system.refresh_memory();
|
||||
let effective = effective_memory_from_system(&system);
|
||||
let snapshot = ObjectDataCacheMemorySnapshot {
|
||||
total_bytes: effective.total_bytes,
|
||||
available_bytes: effective.available_bytes,
|
||||
};
|
||||
|
||||
self.snapshot_total_bytes.store(snapshot.total_bytes, Ordering::Relaxed);
|
||||
self.snapshot_available_bytes
|
||||
.store(snapshot.available_bytes, Ordering::Relaxed);
|
||||
*lock_or_recover(&self.last_refresh) = Instant::now();
|
||||
self.snapshot.load()
|
||||
}
|
||||
|
||||
/// Returns true when the fill path may proceed under current memory pressure.
|
||||
///
|
||||
/// This is lock-free and does no blocking sysinfo read: it only reads the
|
||||
/// atomic snapshot maintained by the periodic refresher.
|
||||
pub fn allows_fill(&self, required_bytes: u64) -> bool {
|
||||
// A zero floor opts out of the gate, so fill admission never depends on
|
||||
// a live memory reading — which differs between a host and a container.
|
||||
// This must short-circuit before any snapshot read (see 51a97a81c).
|
||||
if self.min_free_memory_percent == 0 {
|
||||
return true;
|
||||
}
|
||||
|
||||
self.refresh_if_stale();
|
||||
let snapshot = self.snapshot();
|
||||
if snapshot.total_bytes == 0 {
|
||||
return true;
|
||||
@@ -216,8 +271,22 @@ impl ObjectDataCacheMemoryGate {
|
||||
pub fn set_test_snapshot(&self, snapshot: Option<ObjectDataCacheMemorySnapshot>) {
|
||||
*lock_or_recover(&self.test_override) = snapshot;
|
||||
}
|
||||
|
||||
/// Writes the atomic snapshot directly, bypassing the `test_override` read
|
||||
/// path so a test can observe whether `allows_fill` mutates the snapshot.
|
||||
#[cfg(test)]
|
||||
fn store_raw_snapshot_for_test(&self, snapshot: ObjectDataCacheMemorySnapshot) {
|
||||
self.snapshot.store(snapshot);
|
||||
}
|
||||
|
||||
/// Reads the atomic snapshot directly, bypassing `test_override`.
|
||||
#[cfg(test)]
|
||||
fn raw_snapshot_for_test(&self) -> ObjectDataCacheMemorySnapshot {
|
||||
self.snapshot.load()
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
fn lock_or_recover<T>(mutex: &Mutex<T>) -> std::sync::MutexGuard<'_, T> {
|
||||
match mutex.lock() {
|
||||
Ok(guard) => guard,
|
||||
@@ -231,6 +300,7 @@ mod tests {
|
||||
use crate::config::ObjectDataCacheConfig;
|
||||
use crate::stats::ObjectDataCacheStats;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
const GIB: u64 = 1024 * 1024 * 1024;
|
||||
|
||||
@@ -331,4 +401,57 @@ mod tests {
|
||||
assert!(!gate.allows_fill(128));
|
||||
assert_eq!(stats.snapshot().memory_pressure_events, 1);
|
||||
}
|
||||
|
||||
// ODC-14: `allows_fill` must read the atomic snapshot without performing an
|
||||
// inline (blocking) refresh. A synchronous test has no tokio runtime, so no
|
||||
// refresher task exists; if `allows_fill` refreshed inline it would clobber
|
||||
// the seeded atomic snapshot with the real host reading. Comparing exact
|
||||
// byte values makes the assertion independent of the host's actual memory.
|
||||
#[test]
|
||||
fn allows_fill_reads_snapshot_without_refreshing_inline() {
|
||||
let stats = Arc::new(ObjectDataCacheStats::default());
|
||||
let gate = ObjectDataCacheMemoryGate::new(&ObjectDataCacheConfig::default(), Arc::clone(&stats));
|
||||
|
||||
let sentinel = ObjectDataCacheMemorySnapshot {
|
||||
total_bytes: 4_242,
|
||||
available_bytes: 2_121,
|
||||
};
|
||||
gate.store_raw_snapshot_for_test(sentinel);
|
||||
|
||||
// Exercise the gate on the atomic path (no test_override installed).
|
||||
let _ = gate.allows_fill(1);
|
||||
|
||||
let after = gate.raw_snapshot_for_test();
|
||||
assert_eq!(
|
||||
after, sentinel,
|
||||
"allows_fill must not mutate the snapshot; an inline refresh would overwrite it"
|
||||
);
|
||||
}
|
||||
|
||||
// ODC-14: the periodic refresher samples memory off the fill path and
|
||||
// updates the atomic snapshot, so a stale seed is replaced without any
|
||||
// fill ever blocking on a refresh.
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn periodic_refresher_updates_snapshot_off_path() {
|
||||
let stats = Arc::new(ObjectDataCacheStats::default());
|
||||
let snapshot = Arc::new(super::MemorySnapshotCell::new(ObjectDataCacheMemorySnapshot {
|
||||
total_bytes: 1,
|
||||
available_bytes: 1,
|
||||
}));
|
||||
let _guard = ObjectDataCacheMemoryGate::spawn_refresher(Arc::clone(&snapshot), Duration::from_millis(20))
|
||||
.expect("a tokio runtime is available in an async test");
|
||||
|
||||
// The seed is a bogus 1/1; wait for the refresher to overwrite it with a
|
||||
// real host reading (total memory is always far above 1 byte).
|
||||
let mut refreshed = false;
|
||||
for _ in 0..200 {
|
||||
if snapshot.load().total_bytes > 1 {
|
||||
refreshed = true;
|
||||
break;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(10)).await;
|
||||
}
|
||||
assert!(refreshed, "the periodic refresher must replace the seed snapshot off the fill path");
|
||||
let _ = stats;
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user