mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-18 10:43:15 +00:00
feat(object-data-cache): close write-side invalidation gaps and add an admin surface (#4694)
* feat(object-data-cache): close write/delete-side invalidation gaps The object data cache exposed only a single per-(bucket,object) invalidation primitive and no write-side ecstore hook, so several delete paths left dead bodies resident until TTL (hygiene/capacity, not stale-serving: lookups follow a fresh metadata quorum and cannot serve a gone object). This adds the missing primitives and wires them in. ODC-26 (backlog#1131): add an `ObjectMutationHook` trait beside the GET body hook, registered next to it at startup, and call it from the ecstore-internal delete paths (`apply_expiry_on_non_transitioned_objects`, `expire_transitioned_object` including the restored-copy branch, and `delete_object_versions`). The app impl is one `invalidate_object` call under a new `AfterLifecycleExpiry` reason. ODC-27 (backlog#1132): force prefix delete now invalidates the whole prefix, not just the prefix string. `store.delete_object(delete_prefix)` returns no deleted-name list, so this uses a new prefix primitive rather than the batch path. ODC-28 (backlog#1133): DeleteBucket now flushes the bucket via a new bucket-scope primitive (covers force and non-force, which share the delete_bucket call). ODC-C2 (backlog#1143): add `ObjectDataCache::clear()` and two admin handlers (GET stats, POST flush) routed through admin runtime_sources. The starshard identity index gains a single `remove_matching` full-scan API backing prefix/bucket/clear; it is documented as admin/delete-path only and never runs on the GET or fill hot path. New invalidation reasons and metric labels added; outcome (removed/noop) labelling kept correct for every new primitive. Also fixes a pre-existing broken intra-doc link in memory.rs. Co-Authored-By: heihutu <heihutu@gmail.com> * refactor(ecstore): extract the shared HookSlot behind both cache hooks This PR introduced object_mutation_hook.rs by mirroring body_cache_hook.rs, which left two process-global registration slots whose register/get/clear bodies were line-for-line identical except the trait type and the WARN string: a RwLock<Option<Arc<dyn _>>>, an Arc::ptr_eq "different instance" warning, the poison-recovery closure, and the same read-lock-and-clone read. Two copies of the same swap-vs-warn logic can drift apart under maintenance. Hoist it into a generic HookSlot<T: ?Sized> that owns the logic once. Each hook module keeps its `static HOOK: HookSlot<dyn XxxHook>` and its thin, unchanged public wrappers (register_/get_/clear_), so the crate's public surface and every call site are untouched — this is an internal consolidation, not a contract change. The load-bearing #1126 guarantee (newest registration wins, so a rebuilt AppContext is never stranded on a first-wins slot) previously had no direct test — the hook tests only covered register-then-notify. HookSlot now has its own unit tests including re_registration_swaps_to_the_latest_instance; mutation-testing confirms a first-wins regression fails exactly that test. No behavior change: the two hooks' existing tests, the P0 body_cache_hook_e2e regressions, and the app-layer mutation-hook tests all pass unchanged. Refs: backlog#1126, backlog#1131 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(admin): register the object-data-cache routes in the policy inventory This PR added GET /object-data-cache/stats and POST /object-data-cache/flush but did not list them in the two registries that must account for every admin route: the route-policy inventory (route_policy.rs) and the route matrix (route_registration_test.rs). Their coverage tests — route_policy_inventory_covers_registered_routes and test_admin_route_matrix_matches_registered_routes — failed on CI because a registered route had no policy/matrix entry. These two tests are not part of `make pre-commit` (which runs fmt + arch + quick-check, not the full suite), so the gap passed local pre-commit and only surfaced in the CI Test-and-Lint lane. stats is a read (ServerInfoAdminAction, Sensitive); flush mutates (ConfigUpdateAdminAction, High) — matching the actions the handlers already enforce. The MinIO-alias matrix test is unaffected: these are native rustfs endpoints with no MinIO equivalent. Refs: backlog#1143 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
@@ -293,6 +293,48 @@ impl MokaBackend {
|
||||
self.cache.remove(&key).await;
|
||||
}
|
||||
|
||||
Self::invalidation_result(removed)
|
||||
}
|
||||
|
||||
/// Invalidates every cached key of an identity in `bucket` whose object key
|
||||
/// starts with `prefix`. Full index scan; rare force-delete/admin path only.
|
||||
pub async fn invalidate_prefix(&self, bucket: &str, prefix: &str) -> ObjectDataCacheInvalidationResult {
|
||||
let keys_to_remove = self
|
||||
.index
|
||||
.remove_matching(|identity| identity.bucket.as_ref() == bucket && identity.object.starts_with(prefix))
|
||||
.await;
|
||||
let removed = keys_to_remove.len();
|
||||
for key in keys_to_remove {
|
||||
self.cache.remove(&key).await;
|
||||
}
|
||||
Self::invalidation_result(removed)
|
||||
}
|
||||
|
||||
/// Invalidates every cached key of every identity in `bucket`. Full index
|
||||
/// scan; rare bucket-delete/admin path only.
|
||||
pub async fn invalidate_bucket(&self, bucket: &str) -> ObjectDataCacheInvalidationResult {
|
||||
let keys_to_remove = self
|
||||
.index
|
||||
.remove_matching(|identity| identity.bucket.as_ref() == bucket)
|
||||
.await;
|
||||
let removed = keys_to_remove.len();
|
||||
for key in keys_to_remove {
|
||||
self.cache.remove(&key).await;
|
||||
}
|
||||
Self::invalidation_result(removed)
|
||||
}
|
||||
|
||||
/// Drops every cached body and resets the identity index. The index scan
|
||||
/// yields the tracked key count for the outcome label; `invalidate_all`
|
||||
/// then clears moka in one call (also dropping any entry not tracked in the
|
||||
/// index). Rare admin `clear()` path only.
|
||||
pub async fn clear(&self) -> ObjectDataCacheInvalidationResult {
|
||||
let removed = self.index.remove_matching(|_| true).await.len();
|
||||
self.cache.invalidate_all();
|
||||
Self::invalidation_result(removed)
|
||||
}
|
||||
|
||||
const fn invalidation_result(removed: usize) -> ObjectDataCacheInvalidationResult {
|
||||
if removed > 0 {
|
||||
ObjectDataCacheInvalidationResult::Removed { keys: removed }
|
||||
} else {
|
||||
@@ -402,6 +444,109 @@ mod tests {
|
||||
assert_eq!(result, ObjectDataCacheInvalidationResult::NoOp);
|
||||
}
|
||||
|
||||
fn bucketed_plan(bucket: &str, object: &str, etag: &str) -> ObjectDataCacheGetPlan {
|
||||
ObjectDataCacheGetPlan::Cacheable {
|
||||
key: ObjectDataCacheKey::new(bucket, object, None, etag, 5, ObjectDataCacheBodyVariant::FullObjectPlainV1),
|
||||
}
|
||||
}
|
||||
|
||||
// ODC-27: prefix invalidation drops only the identities under the prefix and
|
||||
// leaves every other cached body intact.
|
||||
#[tokio::test]
|
||||
async fn moka_backend_invalidate_prefix_removes_only_matching_prefix() {
|
||||
let mut config = enabled_config();
|
||||
config.ttl = Duration::from_secs(30);
|
||||
config.time_to_idle = Duration::from_secs(30);
|
||||
let backend = MokaBackend::new(&config, Arc::new(ObjectDataCacheStats::default())).expect("moka backend should build");
|
||||
let plan_a = cacheable_plan("photos/a.jpg", "etag-a");
|
||||
let plan_b = cacheable_plan("photos/b.jpg", "etag-b");
|
||||
let plan_c = cacheable_plan("videos/c.mp4", "etag-c");
|
||||
|
||||
let _ = backend.fill_body(&plan_a, Bytes::from_static(b"aaaaa")).await;
|
||||
let _ = backend.fill_body(&plan_b, Bytes::from_static(b"bbbbb")).await;
|
||||
let _ = backend.fill_body(&plan_c, Bytes::from_static(b"ccccc")).await;
|
||||
|
||||
let result = backend.invalidate_prefix("bucket", "photos/").await;
|
||||
|
||||
assert_eq!(result, ObjectDataCacheInvalidationResult::Removed { keys: 2 });
|
||||
assert!(matches!(backend.lookup_body(&plan_a).await, ObjectDataCacheLookup::Miss));
|
||||
assert!(matches!(backend.lookup_body(&plan_b).await, ObjectDataCacheLookup::Miss));
|
||||
assert!(
|
||||
matches!(backend.lookup_body(&plan_c).await, ObjectDataCacheLookup::Hit(_)),
|
||||
"an object outside the prefix must survive"
|
||||
);
|
||||
}
|
||||
|
||||
// ODC-27: a prefix that matches nothing cached is a no-op.
|
||||
#[tokio::test]
|
||||
async fn moka_backend_invalidate_prefix_uncached_is_noop() {
|
||||
let backend =
|
||||
MokaBackend::new(&enabled_config(), Arc::new(ObjectDataCacheStats::default())).expect("moka backend should build");
|
||||
let _ = backend
|
||||
.fill_body(&cacheable_plan("photos/a.jpg", "e"), Bytes::from_static(b"aaaaa"))
|
||||
.await;
|
||||
|
||||
let result = backend.invalidate_prefix("bucket", "videos/").await;
|
||||
|
||||
assert_eq!(result, ObjectDataCacheInvalidationResult::NoOp);
|
||||
}
|
||||
|
||||
// ODC-28: bucket invalidation drops every identity in the bucket and leaves
|
||||
// other buckets untouched.
|
||||
#[tokio::test]
|
||||
async fn moka_backend_invalidate_bucket_removes_only_that_bucket() {
|
||||
let mut config = enabled_config();
|
||||
config.ttl = Duration::from_secs(30);
|
||||
config.time_to_idle = Duration::from_secs(30);
|
||||
let backend = MokaBackend::new(&config, Arc::new(ObjectDataCacheStats::default())).expect("moka backend should build");
|
||||
let plan_a = bucketed_plan("bucket-a", "o1", "etag-a");
|
||||
let plan_b = bucketed_plan("bucket-a", "o2", "etag-b");
|
||||
let plan_other = bucketed_plan("bucket-b", "o3", "etag-c");
|
||||
|
||||
let _ = backend.fill_body(&plan_a, Bytes::from_static(b"aaaaa")).await;
|
||||
let _ = backend.fill_body(&plan_b, Bytes::from_static(b"bbbbb")).await;
|
||||
let _ = backend.fill_body(&plan_other, Bytes::from_static(b"ccccc")).await;
|
||||
|
||||
let result = backend.invalidate_bucket("bucket-a").await;
|
||||
|
||||
assert_eq!(result, ObjectDataCacheInvalidationResult::Removed { keys: 2 });
|
||||
assert!(matches!(backend.lookup_body(&plan_a).await, ObjectDataCacheLookup::Miss));
|
||||
assert!(matches!(backend.lookup_body(&plan_b).await, ObjectDataCacheLookup::Miss));
|
||||
assert!(
|
||||
matches!(backend.lookup_body(&plan_other).await, ObjectDataCacheLookup::Hit(_)),
|
||||
"an object in another bucket must survive a bucket flush"
|
||||
);
|
||||
}
|
||||
|
||||
// ODC-C2: clear drops every cached body and empties the identity index.
|
||||
#[tokio::test]
|
||||
async fn moka_backend_clear_removes_everything() {
|
||||
let mut config = enabled_config();
|
||||
config.ttl = Duration::from_secs(30);
|
||||
config.time_to_idle = Duration::from_secs(30);
|
||||
let backend = MokaBackend::new(&config, Arc::new(ObjectDataCacheStats::default())).expect("moka backend should build");
|
||||
let plan_a = bucketed_plan("bucket-a", "o1", "etag-a");
|
||||
let plan_b = bucketed_plan("bucket-b", "o2", "etag-b");
|
||||
|
||||
let _ = backend.fill_body(&plan_a, Bytes::from_static(b"aaaaa")).await;
|
||||
let _ = backend.fill_body(&plan_b, Bytes::from_static(b"bbbbb")).await;
|
||||
|
||||
let result = backend.clear().await;
|
||||
|
||||
assert_eq!(result, ObjectDataCacheInvalidationResult::Removed { keys: 2 });
|
||||
assert!(matches!(backend.lookup_body(&plan_a).await, ObjectDataCacheLookup::Miss));
|
||||
assert!(matches!(backend.lookup_body(&plan_b).await, ObjectDataCacheLookup::Miss));
|
||||
assert_eq!(backend.index.identity_count().await, 0, "clear must empty the identity index");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn moka_backend_clear_empty_cache_is_noop() {
|
||||
let backend =
|
||||
MokaBackend::new(&enabled_config(), Arc::new(ObjectDataCacheStats::default())).expect("moka backend should build");
|
||||
|
||||
assert_eq!(backend.clear().await, ObjectDataCacheInvalidationResult::NoOp);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn moka_backend_expires_entries_by_ttl() {
|
||||
let backend =
|
||||
|
||||
Reference in New Issue
Block a user