14 Commits

Author	SHA1	Message	Date
Zhengchao An	5a4cf1d4b5	fix: repair flaky moka clear drain loop from #4759 (#4763) ... fix(cache): drain pending removals until entry_count reaches zero in clear() The previous clear() implementation used a single run_pending_tasks() call after invalidate_all(), which was insufficient under concurrent fill pressure — moka processes invalidations lazily in batches, so entries can linger after a single maintenance pass. Replace the fixed single call with a drain loop (up to 256 rounds) that calls run_pending_tasks() and yields between iterations until entry_count() reaches zero. This ensures the concurrency storm test (moka_backend_concurrency_storm_leaves_no_leaked_state) passes reliably. The earlier fix (#4759) added a pre-invalidate_all drain and an 8-pass loop but reordered operations in a way that introduced a new race. This commit keeps the original invalidate_all-first ordering and only adds the drain loop after the initial run_pending_tasks() call.	2026-07-12 04:26:38 +00:00
Zhengchao An	3832f1d270	fix(cache): drain pending removals during clear (#4759)	2026-07-12 09:46:13 +08:00
Zhengchao An	af831bde4b	fix(cache): drain entries before clear returns (#4751)	2026-07-12 04:32:21 +08:00
houseme	9bf102f965	fix(cache): reserve admitted bytes in the memory gate to bound burst overshoot (#4718) ... The fill gate compared each request against a snapshot refreshed at most every 5 s, with no accounting for what it had already let through. A burst arriving while the snapshot still read high therefore all passed the same check-then-act test and over-allocated far past the real headroom before the next refresh — a gap the burst stress test could expose but not close. Track admitted bytes since the last refresh in the shared snapshot cell and subtract them from available memory in `allows_fill`, reserving the request's size on each admission. Cumulative admission is now bounded to the real budget even though every fill reads the same stale snapshot; the refresh resets the counter because the fresh reading already reflects those allocations. The `min_free_memory_percent == 0` opt-out still short-circuits first, and the already-low-memory path is unchanged. New test `moka_backend_gate_reservation_bounds_burst_under_stale_snapshot`: 20 concurrent 40 KiB fills against a 500 KiB stale-high snapshot (300 KiB budget) admit only a bounded handful, not the whole storm. Passed 10/10 runs; mutation-verified — dropping the reservation admits all 20 and fails the test. Refs: backlog#1107 Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-11 05:36:20 +00:00
houseme	d5d6f1160d	test(object-data-cache): add concurrency stress tests and GET benchmarks (#4711) ... * test(object-data-cache): add concurrency stress tests for the fill machinery The race coverage so far pins specific interleavings with an injected barrier. These add sustained, real-parallelism contention to catch what pinned tests cannot — a leaked singleflight leader, a stranded semaphore permit, an index/cache divergence that only surfaces under volume: - moka_backend_concurrency_storm_leaves_no_leaked_state: 48 tasks x 400 mixed fill/lookup/invalidate ops on a 6-key pool over 4 worker threads, then asserts inflight_fills == 0 (no leaked in-flight fill), a clean post-storm fill/hit/invalidate sequence still works (state not corrupted), and clear() drains the cache. - moka_backend_gate_bounds_admission_under_concurrent_burst: a low-memory snapshot must reject an entire 32-fill burst — admission is bounded, not a check-then-act race that lets the burst through. (This does not cover the separate 5s-stale-snapshot overshoot, which needs byte-based reservation.) Both are mutation-verified: neutering invalidate_object fails the storm's "invalidation must win" assertion, and making allows_fill always-true fails the burst's full-rejection assertion. The storm passed 20/20 runs. `rt-multi-thread` is added to dev-deps so the tasks run on real worker threads. Refs: backlog#1107 Co-Authored-By: heihutu <heihutu@gmail.com> * bench(object-data-cache): measure the GET-path cost the audit argued statically Adds a criterion benchmark driving the public ObjectDataCache facade, so the "hot path is clean / high throughput" claim rests on numbers, not analysis: - plan_get ~104-112 ns (per-GET planning + metric label-set hash) - lookup_hit ~143-147 ns (a resident-body hit: moka get + Bytes clone + metric) - fill_distinct ~2.2-2.4 us (cold fill: plan + singleflight + index + moka + inner spawn) A hit at ~143 ns against a multi-millisecond erasure read is the ~1000x saving the cache exists for; the per-GET metric cost is real but ~100 ns, not a bottleneck; the fill cost (incl. the cancellation-safety spawn kept on purpose) is negligible on a background task. Run with `cargo bench -p rustfs-object-data-cache`. Refs: backlog#1107 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-11 02:30:19 +00:00
houseme	7437f99c45	fix(cache): key the object body cache on data_dir for write-uniqueness (#4703) ... * refactor(object-data-cache): derive Default for ObjectDataCacheGetRequest The GET request literal is hand-listed field-by-field across ~13 test sites in two crates. Adding `mod_time_unix_nanos` in backlog#1111 had to touch every one and still missed a literal, producing a compile error caught only in a later CI lane. Derive `Default` and spread the engine-crate literals so the next field addition is absorbed rather than fanned out. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(cache): key the object body cache on data_dir for write-uniqueness The cache key's correctness rested on being write-unique, but its only write-scoped component was `mod_time` — a wall-clock timestamp that is not monotonic and can be absent. Two writes that collide on MD5 (same etag+size) and land on an equal mod_time (clock skew, same-tick, or absent) derived the same key, so a node that never saw the overwrite could serve the previous body for up to the TTL, and the same collision turned the fill-after-invalidation race into a serving bug. This is the store's strong-read-after-write guarantee leaning on a probabilistic argument. Add `data_dir` — the xl.meta directory UUID ecstore regenerates on every body write — as the primary write-unique anchor: - surface `data_dir: Option<Uuid>` on ObjectInfo, copied from FileInfo in the single GET-path constructor `from_file_info`; - carry it into ObjectDataCacheKey as `data_dir_u128` (held as u128 to keep the engine crate free of a uuid dependency), derived in the one planner site both the ecstore hook and the usecase layer share, so both produce an identical key by construction; - keep `mod_time` as a second anchor and `etag+size` as belt-and-braces; an absent data_dir falls back to the prior behavior — strict improvement, no regression. Two writes distinct only by data_dir now derive different keys even under an MD5 collision with identical mod_time — the case mod_time alone cannot cover. Blast radius is compiler-guarded: ObjectInfo derives Default and every real construction site uses `..Default::default()`, so only from_file_info and one full-literal test needed the field. The three P0 body_cache_hook_e2e regressions, engine (80), and app (36) suites pass unchanged; a mutation that severs the planner wiring fails planner_key_changes_with_data_dir. Refs: backlog#1111, backlog#1118 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(cache): thread data_dir field through the merged mutation-hook test Merging main (which landed the object-mutation-hook work, backlog#1131) brought in a GetRequest test literal that predates the data_dir field. Spread it via `..Default::default()` — the derive(Default) added here means this is the last such hand-listed literal to need touching. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-11 08:00:52 +08:00
houseme	85fd824581	feat(object-data-cache): close write-side invalidation gaps and add an admin surface (#4694) ... * feat(object-data-cache): close write/delete-side invalidation gaps The object data cache exposed only a single per-(bucket,object) invalidation primitive and no write-side ecstore hook, so several delete paths left dead bodies resident until TTL (hygiene/capacity, not stale-serving: lookups follow a fresh metadata quorum and cannot serve a gone object). This adds the missing primitives and wires them in. ODC-26 (backlog#1131): add an `ObjectMutationHook` trait beside the GET body hook, registered next to it at startup, and call it from the ecstore-internal delete paths (`apply_expiry_on_non_transitioned_objects`, `expire_transitioned_object` including the restored-copy branch, and `delete_object_versions`). The app impl is one `invalidate_object` call under a new `AfterLifecycleExpiry` reason. ODC-27 (backlog#1132): force prefix delete now invalidates the whole prefix, not just the prefix string. `store.delete_object(delete_prefix)` returns no deleted-name list, so this uses a new prefix primitive rather than the batch path. ODC-28 (backlog#1133): DeleteBucket now flushes the bucket via a new bucket-scope primitive (covers force and non-force, which share the delete_bucket call). ODC-C2 (backlog#1143): add `ObjectDataCache::clear()` and two admin handlers (GET stats, POST flush) routed through admin runtime_sources. The starshard identity index gains a single `remove_matching` full-scan API backing prefix/bucket/clear; it is documented as admin/delete-path only and never runs on the GET or fill hot path. New invalidation reasons and metric labels added; outcome (removed/noop) labelling kept correct for every new primitive. Also fixes a pre-existing broken intra-doc link in memory.rs. Co-Authored-By: heihutu <heihutu@gmail.com> * refactor(ecstore): extract the shared HookSlot behind both cache hooks This PR introduced object_mutation_hook.rs by mirroring body_cache_hook.rs, which left two process-global registration slots whose register/get/clear bodies were line-for-line identical except the trait type and the WARN string: a RwLock<Option<Arc<dyn _>>>, an Arc::ptr_eq "different instance" warning, the poison-recovery closure, and the same read-lock-and-clone read. Two copies of the same swap-vs-warn logic can drift apart under maintenance. Hoist it into a generic HookSlot<T: ?Sized> that owns the logic once. Each hook module keeps its `static HOOK: HookSlot<dyn XxxHook>` and its thin, unchanged public wrappers (register_/get_/clear_), so the crate's public surface and every call site are untouched — this is an internal consolidation, not a contract change. The load-bearing #1126 guarantee (newest registration wins, so a rebuilt AppContext is never stranded on a first-wins slot) previously had no direct test — the hook tests only covered register-then-notify. HookSlot now has its own unit tests including re_registration_swaps_to_the_latest_instance; mutation-testing confirms a first-wins regression fails exactly that test. No behavior change: the two hooks' existing tests, the P0 body_cache_hook_e2e regressions, and the app-layer mutation-hook tests all pass unchanged. Refs: backlog#1126, backlog#1131 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(admin): register the object-data-cache routes in the policy inventory This PR added GET /object-data-cache/stats and POST /object-data-cache/flush but did not list them in the two registries that must account for every admin route: the route-policy inventory (route_policy.rs) and the route matrix (route_registration_test.rs). Their coverage tests — route_policy_inventory_covers_registered_routes and test_admin_route_matrix_matches_registered_routes — failed on CI because a registered route had no policy/matrix entry. These two tests are not part of `make pre-commit` (which runs fmt + arch + quick-check, not the full suite), so the gap passed local pre-commit and only surfaced in the CI Test-and-Lint lane. stats is a read (ServerInfoAdminAction, Sensitive); flush mutates (ConfigUpdateAdminAction, High) — matching the actions the handlers already enforce. The MinIO-alias matrix test is unaffected: these are native rustfs endpoints with no MinIO equivalent. Refs: backlog#1143 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-10 20:04:05 +00:00
houseme	95819cb986	docs(object-data-cache): state the correctness boundary and timing channel (#4695) ... docs(object-data-cache): state the correctness boundary and the timing channel The crate doc still described itself as "a minimal skeleton for the initial rollout phase", which stopped being true several releases ago, and neither the crate nor the operator-facing env constant said anything about what the cache does or does not guarantee. Record two things a reader has to know. The correctness boundary: a hit is sound because the key matched metadata the caller just resolved from a read quorum, not because invalidation ran. Process-local invalidation is hygiene that frees capacity; the key is what keeps a stale body from being served. Anyone tempted to weaken the key should meet this sentence first. The timing side channel: a hit skips the erasure read, bitrot verify and decode, so it is reliably faster than a miss. A principal authorized to read an object can time a single GET and learn whether someone read that object within the entry's lifetime. This crosses no authorization boundary — the probe needs read access to that exact object, checked before the cache is consulted — but it does disclose a co-tenant's recent access pattern. Say so where operators look: on RUSTFS_OBJECT_DATA_CACHE_ENABLE. Timing noise would cost precisely the latency the cache exists to save, so the mitigation is to leave the cache off for buckets where access-pattern confidentiality matters. Refs: backlog#1139 Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-11 02:37:34 +08:00
houseme	6780140318	fix(object-data-cache): make the GET key write-unique and dedup the lookup (#4693) ... fix(object-data-cache): make GET body cache key write-unique and dedup lookups Address four object-data-cache GET-path findings (backlog#1107 batch): ODC-06 (backlog#1111): the cache key was content-unique, not write-unique. Extend ObjectDataCacheKey with the resolved version's modification time (i128 unix nanoseconds, None -> 0), derived once in the shared planner so the ecstore hook and the usecase layer produce an identical key. An unversioned overwrite advances mod_time, so a stale node can no longer serve old bytes for up to the TTL under an MD5 collision; etag + size stay as belt-and-braces. ODC-16 (backlog#1121): every cacheable GET planned and looked up twice (once in the ecstore hook, once in the usecase layer), double-counting hits, hit_bytes and lookups. GetObjectReader now carries a GetObjectBodySource marker (Unprobed / HookMissed / HookServed); the hook stamps it, and build_get_object_body_with_cache serves a hook-served body directly and skips its lookup whenever the hook already probed. One hook-served GET now records exactly one lookup. ODC-19 (backlog#1124): ENABLE=true with no explicit mode defaulted to HitOnly, which never fills and keeps a permanent 0% hit rate. Default to FillBufferedOnly, log the resolved mode at startup, and warn when HitOnly is selected explicitly. ODC-24 (backlog#1129): max_entry_bytes above the in-memory GET fill limits was silently inert. Clamp the planner's size eligibility to min(max_entry_bytes, seek-support threshold, 64 MiB buffer cap) so ineligible sizes plan SkipTooLarge instead of being reported eligible, and warn at startup when the excess is inert. Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-10 18:22:20 +00:00
houseme	b604074217	perf(object-data-cache): take fill off the GET path and fix the metrics (#4678) ... * fix(object-data-cache): make cache metrics one-increment-per-GET Rework the object data cache observability so each counter carries one clear meaning, and drop dead per-entry state. ODC-17 (backlog#1122): split requests_total, which was incremented by both plan_get and lookup_body, into plan_total{decision,reason,size_class} (emitted only by plan_get) and lookup_total{result,size_class} (emitted only by lookup_body). Each is now incremented exactly once per GET per layer; help text states this. ODC-18 (backlog#1123): add a JoinedInflightFill result (label joined_inflight) so a singleflight waiter is no longer counted as an insert. record_fill_result now always counts the outcome but records fill bytes only when non-zero and the duration histogram only when present, so joined_inflight / skipped_by_mode / skipped_size_mismatch no longer inflate fill_bytes_total or add non-fill duration samples. The waiter->JoinedInflightFill mapping lives in MokaBackend (owned by a concurrent branch); cache.rs handles the variant already. ODC-29 (backlog#1134): stop refreshing the cache-state gauge on every lookup, and debounce it on fill/invalidate to at most once per second via an AtomicU64 millis timestamp, since moka's entry_count is a settling approximation. ODC-36 (backlog#1141): give invalidations_total an outcome label (removed|noop) and skip the gauge refresh on the no-op path. Extend ObjectDataCacheInvalidationResult with Removed{keys}/NoOp (Success kept as a transitional variant until MokaBackend reports the removal count); NoopBackend now reports NoOp. ODC-30 (backlog#1141): drop the dead content_length/etag/inserted_at fields (and getters) from ObjectDataCacheEntry, which are redundant with the moka key identity; the constructor keeps its arity so the out-of-scope MokaBackend caller still compiles. Gate is_null_version behind cfg(test). Tests use a thread-local metrics recorder to avoid the global-singleton flakiness. Fill-enabled test configs set min_free_memory_percent=0 so the cgroup gate does not refuse fills in CI pods. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(object-data-cache): move fill off GET path, bound & harden the fill Implements five object-data-cache audit findings. ODC-15 (backlog#1120): the GET response no longer blocks on cache fill. The buffered/materialized body is already in hand, so the fill now runs in a detached task and the response is built immediately. Singleflight is made non-blocking: `try_acquire` returns Leader or Busy, and a non-leader skips its own fill (SkippedSingleflightBusy) instead of waiting on another request's leader. The inner fill spawn is KEPT: once the index key is registered, the recheck/undo must complete even if the enclosing fill future is aborted, so removing it would weaken the cancellation-safety guarantee (ODC-13 test) and the invalidation-race undo. ODC-11 (backlog#1116): enforce the fill-concurrency knobs. MokaBackend now holds a Semaphore sized min(per_cpu * parallelism, max), acquired after winning leadership and before the memory gate. On saturation it rejects (try_acquire_owned) with SkippedFillConcurrency rather than queueing, so the fill path never reintroduces GET-latency coupling. ODC-14 (backlog#1119): the memory gate no longer does a blocking sysinfo refresh under a mutex on the async fill path. A dedicated periodic refresher (tokio interval + spawn_blocking) updates an atomic snapshot every 5s; allows_fill is now lock-free. The min_free_memory_percent == 0 short-circuit still runs before any snapshot read. No runtime at construction (sync unit tests) simply keeps the seed snapshot. ODC-32 (backlog#1137): singleflight no longer emits metrics while holding the map mutex. try_acquire/remove_entry capture the map length under the guard, drop it, then emit the gauge/counter. ODC-07 (backlog#1112): the materialize read is bounded via take(capacity + 1) so an over-long stream cannot grow the buffer past the in-memory GET threshold, and any length mismatch is now a hard error matching the direct-memory GET path, instead of warn-and-serve. cache.rs is owned by a concurrent branch; this commit only appends the SkippedFillConcurrency and SkippedSingleflightBusy variants + label arms. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(object-data-cache): report JoinedInflightFill and invalidation outcome Reconciles the moka_backend half of two metrics-semantics findings after merging fix/odc-metrics-semantics (which landed the cache.rs half). ODC-18 (backlog#1123): the singleflight non-leader path now returns JoinedInflightFill instead of counting as an insert, so N concurrent GETs of one cold key record one insert, not N. This composes with the ODC-15 redesign: the non-leader does NOT wait for the leader (it already owns the body and never re-serves from the fill result), so no blocking wait is reintroduced. Drops the redundant local SkippedSingleflightBusy variant in favor of the canonical JoinedInflightFill (mapped to no bytes / no duration by the facade). SkippedFillConcurrency (ODC-11) is retained. ODC-36 (backlog#1141): MokaBackend::invalidate_object now returns Removed { keys } / NoOp instead of the transitional Success, so the facade labels invalidations_total{outcome=removed|noop} and skips the cache-state gauge refresh on the no-op path. Tests: duplicate-fill test asserts JoinedInflightFill (bites when reverted to Inserted); new no-op invalidation test; matching-identity test asserts Removed { keys: 1 }. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(object-data-cache): drop unused mut on materialize stream binding The ODC-07 change moves `final_stream` into `AsyncReadExt::take`, so the `build_get_object_body_with_cache` parameter is no longer mutated in place. Co-Authored-By: heihutu <heihutu@gmail.com> * refactor(object-data-cache): close the cross-branch coordination seams Merging the metrics and hot-path branches left four transitional shims that only existed because each side could not edit the other's files. With both sides present they are dead weight, and two of them actively misreport. - ObjectDataCacheEntry::new took content_length and etag only to keep the caller in moka_backend.rs compiling, then discarded both. Drop them; the entry is a Bytes wrapper and the caller now passes only the body. - SkippedSingleflightClosed lost its only producer when the waiter model was replaced by Leader/Busy election. Remove the variant. - The fill-accounting match ended in a wildcard that charged fill_bytes and a duration to every non-Inserted outcome, so a fill rejected by the memory gate or the concurrency semaphore — which never touched the backend and wrote nothing — still inflated fill_bytes_total. Enumerate every variant explicitly: only outcomes that wrote the body report bytes and duration. Exhaustiveness also forces a future variant to state its accounting rather than inherit a wrong default. - InvalidationResult::Success mapped a no-op to outcome=removed, the exact lie backlog#1141 set out to fix, and its doc comment claimed MokaBackend still returned it after that backend had been widened. It has no producer left. Remove it, and tighten the app-layer test from "any successful variant" to the real contract: the pre-mutation call reports Removed{keys:1}, the post-delete call NoOp. Refs: backlog#1123, backlog#1141, backlog#1135 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-10 12:25:23 +00:00
houseme	904554b417	fix(object-data-cache): make memory container-aware and bound cache config (#4655) ... * fix(object-data-cache): bound cache config and make memory container-aware Harden the object data cache configuration path so a single bad input degrades to the disabled adapter instead of OOM-killing pods, panicking at boot, or silently mis-sizing the cache (backlog#1110/1113/1114/1115/ 1127/1140/1130). - ODC-05: resolve capacity and the fill gate from the effective (container-aware) memory. Prefer sysinfo cgroup_limits() and use min(host, cgroup) as the total plus cgroup-derived availability, falling back to host values when no cgroup limit constrains. Log the resolved capacity and its basis once at startup. - ODC-08: cap ttl/time_to_idle at 30 days in validate() so an unbounded Duration can no longer trip moka's ~1000-year builder assertion. - ODC-09: drop the max_entry_bytes floor from the derived-capacity clamp so MAX_ENTRY_BYTES can no longer inflate total capacity above the safety clamp; reject an entry larger than the resolved capacity instead. - ODC-10: require an explicit max_bytes to clear max_entry_bytes plus the weigher overhead so a fillable-but-unretainable cache is rejected. - ODC-22: reject max_entry_bytes at/above the u32 weigher boundary. - ODC-35: warn (not reject) when time_to_idle exceeds ttl and document the min(ttl, time_to_idle) expiry interaction. - ODC-25: give numeric env overrides two-valued semantics via a new rustfs_utils::get_env_parse_outcome; a malformed value now disables the whole cache with one aggregated warning instead of silently keeping defaults. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(object-data-cache): require identity_keys_max of at least 2 The identity index admits a new key by evicting the oldest one, so a budget of 1 evicts the previous key on every fill and can never hold two live keys of one object at once — a versioned bucket alternating two versions then hits a permanent 0% hit rate. Reject the degenerate value at config validation time, where the adapter turns it into a disabled cache with a warning, since the bounded-eviction policy cannot rescue it at runtime. Handed off from the identity-index batch (backlog#1128), which changed the overflow policy but could not touch validate(). Refs: backlog#1115, backlog#1128 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(object-data-cache): let a zero free-memory floor opt out of the gate Making the memory gate container-aware turned it live in CI: the runners are Kubernetes pods, so the gate now reads the pod's cgroup free memory, finds it below the 20% floor, and refuses the fill. Tests that assert a fill succeeds then failed on CI while passing on a developer host, which has no cgroup and falls back to host memory. The gate is behaving correctly — the tests were the ones depending on a live memory reading. Treat min_free_memory_percent = 0 as a deliberate opt-out rather than an invalid value: allows_fill returns early before it touches any snapshot, so admission becomes independent of where the suite runs. Operators gain the same escape hatch. Every test that requires a fill to succeed now sets the floor to 0. The tests that exercise the gate keep it enabled via memory_gated_config, so the ODC-05 coverage they provide is preserved rather than short-circuited. Refs: backlog#1110 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(object-data-cache): opt the usecase fill tests out of the memory gate The previous commit exempted the fill-dependent tests it could find, but missed the six adapters built inside object_usecase.rs. Fixing the first batch moved nextest's fail-fast point forward and CI surfaced them: build_get_object_body_with_cache_materializes_once_and_hits_later and ..._uses_cached_body_without_reader_preread both assert a fill lands, and both ran with the default 20% free-memory floor. Set the floor to 0 on all six, matching the sibling test modules. Verified by pinning the gate's snapshot to 0% available — harsher than any CI pod — and confirming these tests still pass, which shows the exemption path never reads memory at all. An exhaustive sweep over every fill-enabled ObjectDataCacheConfig in the tree now shows no remaining site: the only unexempted ones are fill_enabled()'s matches! arm and two adapter tests that assert the adapter is disabled and therefore never fill. Refs: backlog#1110 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-10 09:17:32 +00:00
houseme	d29e637890	fix(object-data-cache): close identity-index races in fill/invalidate (#4657) ... fix(object-data-cache): close identity-index races in fill/invalidate (backlog#1117/#1118/#1125/#1128/#1136) The object body cache's fill-vs-invalidation contract (index.insert before cache.insert, then re-check index membership and undo the fill on mismatch) had several races where the identity index and the cache could drift, either poisoning the race metric, breaking invalidation silently, or letting a principal evict a hot body on demand. ODC-12 (backlog#1117) — generation-aware index removal. The eviction listener removed (identity, key) by key equality with no way to tell an evicted old entry from a freshly refilled one under the same key. Verified against moka 0.12.15: an insert over a TTL/TTI-expired but not yet purged entry runs do_insert_with_hash inline during insert().await, and do_post_update_steps awaits notify_upsert INLINE with cause Expired (was_evicted() == true), so the listener deleted the index key the refill just registered and the post-insert recheck then self-destructed the fresh entry (SkippedInvalidationRace). A deferred Size notification for an old generation had the same effect on a cleanly refilled key. Chose Option A (generation token) over Option B (re-insert after cache.contains_key): B removes the key first and only reconciles afterward, which reopens a narrow window where a concurrent invalidate_object misses the key while it is transiently absent from the index. A never removes a key whose generation does not match, so the decision is atomic under the shard lock. The token is the cache entry's Arc pointer captured at fill time (readable from the value moka hands the listener), so no change to the entry type is needed. ODC-13 (backlog#1118) — cancellation-safe insert/recheck/undo. The recheck+undo sat after two awaits inside the S3 GET future; a client disconnect could cancel the fill at the recheck await after cache.insert made the entry visible, stranding a stale body with no index entry. The register/insert/recheck/undo sequence now runs in a spawned task whose JoinHandle the leader awaits: cancelling the await detaches the task, which still finishes the recheck and undo. The dropped leader unblocks waiters as before (SkippedSingleflightClosed), so no waiter is stranded. ODC-20 (backlog#1125) — do not prune in-flight sibling keys. prune_missing could remove another in-flight fill's index key (a different key of the same identity that had registered but not yet published its cache entry), making that fill's recheck fail and delete its own valid entry. Added a read-only ObjectDataCacheSingleflight::has_inflight and consult it in the prune predicate alongside cache.contains_key. ODC-23 (backlog#1128) — bounded eviction instead of clear-and-reject. Exceeding identity_keys_max drained the whole key set and rejected the incoming key, so a principal able to GET identity_keys_max+1 distinct versions could evict an object's hot body on demand, and identity_keys_max=1 on a versioned bucket meant a permanent 0% hit rate. The key set now evicts only the oldest key(s) (the Vec preserves insertion order) and inserts the new key. The insert result carries evicted_keys so the backend removes exactly those from the cache and still caches the newest body. The Overflow variant and the now-unused SkippedIdentityOverflow construction are gone. NOTE: the audit also asks for identity_keys_max >= 2 validation; that lives in config.rs (out of scope here) and is deferred to the config batch. ODC-31 (backlog#1136) — deterministic race coverage. Added a #[cfg(test)] fill barrier between the index insert and the cache insert so the fill-vs-invalidation recheck can be driven deterministically, plus a companion test asserting the identity-budget eviction drops the evicted keys' cache entries. New tests: - index: key_set_bounded_eviction_evicts_oldest_and_inserts_new, key_set_removes_only_matching_generation_token, key_set_duplicate_refreshes_generation_token - starshard_index: identity_index_bounded_eviction_evicts_oldest, identity_index_stale_eviction_preserves_refreshed_key, identity_index_matching_eviction_removes_key - moka_backend: moka_backend_refill_after_expiry_without_maintenance_inserts, moka_backend_concurrent_fills_same_identity_both_insert, moka_backend_identity_budget_evicts_oldest_and_caches_newest, moka_backend_fill_loses_race_to_invalidation, moka_backend_cancelled_fill_still_undoes_lost_race cargo fmt --all, cargo check/clippy/test -p rustfs-object-data-cache all pass (44 tests). Verified the refill-after-expiry and concurrent-fill tests fail without their respective fixes. Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-10 08:04:57 +00:00
Zhengchao An	afa3935ebb	fix(object-data-cache): prune identity index on moka eviction (#4458) ... Moka evicts entries by TTL, time-to-idle, and capacity-LRU without going through invalidate_object, so the per-object identity index (by_object) never dropped the evicted keys and grew without bound, bypassing the memory cap. Register an async eviction listener that removes truly evicted keys from the identity index, holding only a Weak reference to avoid an Arc cycle. Refs rustfs/backlog#1031	2026-07-08 09:14:46 +00:00
houseme	eebd16d8a4	feat(cache): add object data cache engine and app flow (#4187) ... * feat(cache): add object data cache engine * feat(cache): wire app-layer object cache flow * refactor(cache): streamline app-layer cache flow * refactor(cache): tighten cache flow internals * refactor: address final clippy cleanup * chore(deps): update quick-xml to 0.41.0 * feat(cache): wire object data cache env config * fix(cache): gate materialize fill by cache plan * chore(cache): add object data cache benchmark gate * fix(cache): guard object cache fill size mismatches * refactor(cache): streamline object cache body planning * fix(cache): align object cache rollout config * test(cache): cover buffered object cache benchmark * test(cache): isolate object cache benchmark metrics * test(cache): mark materialize rollout experimental * test(cache): tighten object cache benchmark gate * fix(cache): address review findings for object data cache - singleflight: clean up leader entry on cancellation (Drop impl) so a dropped GET future can no longer wedge all subsequent fills for the same key; switch the fill map to a std Mutex and add a regression test - adapter: honor RUSTFS_OBJECT_DATA_CACHE_ENABLE=true by defaulting to hit_only when no explicit mode is set (explicit mode still wins) - planner: treat nil version UUIDs as "no value" per repo convention so unversioned objects key under the canonical "null" instead of fragmenting the key space - multipart: invalidate the object cache on the quota-exceeded rollback delete after complete-multipart, closing a stale-cache window - layering: move the disabled-cache fallback into app::context and drop the new infra->app layer-dependency baseline entry * fix(cache): close invalidation races and drop full-cache scan on writes - index: make identity-index insert/remove/prune atomic via starshard compute_if_present/compute_if_absent so concurrent fills can no longer drop each other's keys (lost keys made entries unreachable to invalidation until TTL); add a concurrency regression test - fill: register the key in the identity index before the entry becomes visible in the cache and re-check the index afterwards, undoing the fill when an invalidation raced in between (new skipped_invalidation_race fill result) - invalidate: with the index now authoritative, remove the full-cache iter() fallback that made every PUT/DELETE of a never-cached object O(total cache entries) (two scans per PUT, 2N per batch delete) - materialize-fill: fail the GET instead of falling back to the partially consumed stream after a mid-read error (the fallback would send a body missing its prefix under a full-length Content-Length), and log the same size-mismatch warning as the sibling buffering paths Co-Authored-By: heihutu <heihutu@gmail.com> * test(storage): fix media-dependent buffer clamp expectation test_concurrency_manager_multi_factor_strategy_buffer_clamp asserted media_cap.min(MI_B), but the implementation's final safety clamp is [32KiB, media_cap.max(MI_B)] — deliberately so a media cap above 1MiB (NVMe's 2MiB default) stays effective. The test only passed on machines detected as SSD/Unknown (cap == 1MiB) and failed on NVMe-backed CI runners with 2MiB != 1MiB. Assert the media cap itself, which is what the strategy actually guarantees on every environment. Co-Authored-By: heihutu <heihutu@gmail.com> * test(storage): format buffer clamp assertion * chore(logging): update tier guardrail path --------- Signed-off-by: houseme <housemecn@gmail.com> Co-authored-by: cxymds <cxymds@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-03 18:11:14 +08:00