* test(object-data-cache): add concurrency stress tests for the fill machinery
The race coverage so far pins specific interleavings with an injected barrier.
These add sustained, real-parallelism contention to catch what pinned tests
cannot — a leaked singleflight leader, a stranded semaphore permit, an
index/cache divergence that only surfaces under volume:
- moka_backend_concurrency_storm_leaves_no_leaked_state: 48 tasks x 400 mixed
fill/lookup/invalidate ops on a 6-key pool over 4 worker threads, then
asserts inflight_fills == 0 (no leaked in-flight fill), a clean post-storm
fill/hit/invalidate sequence still works (state not corrupted), and clear()
drains the cache.
- moka_backend_gate_bounds_admission_under_concurrent_burst: a low-memory
snapshot must reject an entire 32-fill burst — admission is bounded, not a
check-then-act race that lets the burst through. (This does not cover the
separate 5s-stale-snapshot overshoot, which needs byte-based reservation.)
Both are mutation-verified: neutering invalidate_object fails the storm's
"invalidation must win" assertion, and making allows_fill always-true fails the
burst's full-rejection assertion. The storm passed 20/20 runs. `rt-multi-thread`
is added to dev-deps so the tasks run on real worker threads.
Refs: backlog#1107
Co-Authored-By: heihutu <heihutu@gmail.com>
* bench(object-data-cache): measure the GET-path cost the audit argued statically
Adds a criterion benchmark driving the public ObjectDataCache facade, so the
"hot path is clean / high throughput" claim rests on numbers, not analysis:
- plan_get ~104-112 ns (per-GET planning + metric label-set hash)
- lookup_hit ~143-147 ns (a resident-body hit: moka get + Bytes clone + metric)
- fill_distinct ~2.2-2.4 us (cold fill: plan + singleflight + index + moka + inner spawn)
A hit at ~143 ns against a multi-millisecond erasure read is the ~1000x saving
the cache exists for; the per-GET metric cost is real but ~100 ns, not a
bottleneck; the fill cost (incl. the cancellation-safety spawn kept on purpose)
is negligible on a background task. Run with
`cargo bench -p rustfs-object-data-cache`.
Refs: backlog#1107
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
houseme