mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-06 21:33:14 +00:00
feat(object-data-cache): close write-side invalidation gaps and add an admin surface (#4694)
* feat(object-data-cache): close write/delete-side invalidation gaps The object data cache exposed only a single per-(bucket,object) invalidation primitive and no write-side ecstore hook, so several delete paths left dead bodies resident until TTL (hygiene/capacity, not stale-serving: lookups follow a fresh metadata quorum and cannot serve a gone object). This adds the missing primitives and wires them in. ODC-26 (backlog#1131): add an `ObjectMutationHook` trait beside the GET body hook, registered next to it at startup, and call it from the ecstore-internal delete paths (`apply_expiry_on_non_transitioned_objects`, `expire_transitioned_object` including the restored-copy branch, and `delete_object_versions`). The app impl is one `invalidate_object` call under a new `AfterLifecycleExpiry` reason. ODC-27 (backlog#1132): force prefix delete now invalidates the whole prefix, not just the prefix string. `store.delete_object(delete_prefix)` returns no deleted-name list, so this uses a new prefix primitive rather than the batch path. ODC-28 (backlog#1133): DeleteBucket now flushes the bucket via a new bucket-scope primitive (covers force and non-force, which share the delete_bucket call). ODC-C2 (backlog#1143): add `ObjectDataCache::clear()` and two admin handlers (GET stats, POST flush) routed through admin runtime_sources. The starshard identity index gains a single `remove_matching` full-scan API backing prefix/bucket/clear; it is documented as admin/delete-path only and never runs on the GET or fill hot path. New invalidation reasons and metric labels added; outcome (removed/noop) labelling kept correct for every new primitive. Also fixes a pre-existing broken intra-doc link in memory.rs. Co-Authored-By: heihutu <heihutu@gmail.com> * refactor(ecstore): extract the shared HookSlot behind both cache hooks This PR introduced object_mutation_hook.rs by mirroring body_cache_hook.rs, which left two process-global registration slots whose register/get/clear bodies were line-for-line identical except the trait type and the WARN string: a RwLock<Option<Arc<dyn _>>>, an Arc::ptr_eq "different instance" warning, the poison-recovery closure, and the same read-lock-and-clone read. Two copies of the same swap-vs-warn logic can drift apart under maintenance. Hoist it into a generic HookSlot<T: ?Sized> that owns the logic once. Each hook module keeps its `static HOOK: HookSlot<dyn XxxHook>` and its thin, unchanged public wrappers (register_/get_/clear_), so the crate's public surface and every call site are untouched — this is an internal consolidation, not a contract change. The load-bearing #1126 guarantee (newest registration wins, so a rebuilt AppContext is never stranded on a first-wins slot) previously had no direct test — the hook tests only covered register-then-notify. HookSlot now has its own unit tests including re_registration_swaps_to_the_latest_instance; mutation-testing confirms a first-wins regression fails exactly that test. No behavior change: the two hooks' existing tests, the P0 body_cache_hook_e2e regressions, and the app-layer mutation-hook tests all pass unchanged. Refs: backlog#1126, backlog#1131 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(admin): register the object-data-cache routes in the policy inventory This PR added GET /object-data-cache/stats and POST /object-data-cache/flush but did not list them in the two registries that must account for every admin route: the route-policy inventory (route_policy.rs) and the route matrix (route_registration_test.rs). Their coverage tests — route_policy_inventory_covers_registered_routes and test_admin_route_matrix_matches_registered_routes — failed on CI because a registered route had no policy/matrix entry. These two tests are not part of `make pre-commit` (which runs fmt + arch + quick-check, not the full suite), so the gap passed local pre-commit and only surfaced in the CI Test-and-Lint lane. stats is a read (ServerInfoAdminAction, Sensitive); flush mutates (ConfigUpdateAdminAction, High) — matching the actions the handlers already enforce. The MinIO-alias matrix test is unaffected: these are native rustfs endpoints with no MinIO equivalent. Refs: backlog#1143 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
@@ -258,25 +258,80 @@ impl ObjectDataCache {
|
||||
/// Invalidates all cache entries associated with the object identity.
|
||||
pub async fn invalidate_object(
|
||||
&self,
|
||||
_identity: ObjectDataCacheIdentity,
|
||||
_reason: ObjectDataCacheInvalidationReason,
|
||||
identity: ObjectDataCacheIdentity,
|
||||
reason: ObjectDataCacheInvalidationReason,
|
||||
) -> ObjectDataCacheInvalidationResult {
|
||||
let result = match &self.backend {
|
||||
ObjectDataCacheBackendKind::Noop(backend) => backend.invalidate_object().await,
|
||||
ObjectDataCacheBackendKind::Moka(backend) => backend.invalidate_object(&_identity).await,
|
||||
ObjectDataCacheBackendKind::Moka(backend) => backend.invalidate_object(&identity).await,
|
||||
};
|
||||
self.finish_invalidation(result, reason)
|
||||
}
|
||||
|
||||
/// Invalidates every cached body under `bucket`/`prefix`.
|
||||
///
|
||||
/// Drops the cached bodies of every identity in `bucket` whose object key
|
||||
/// starts with `prefix`. Backed by a full identity-index scan, so it must
|
||||
/// stay on the rare force-delete and admin paths and never touch the GET or
|
||||
/// fill hot path (ODC-27, backlog#1132).
|
||||
pub async fn invalidate_prefix(
|
||||
&self,
|
||||
bucket: &str,
|
||||
prefix: &str,
|
||||
reason: ObjectDataCacheInvalidationReason,
|
||||
) -> ObjectDataCacheInvalidationResult {
|
||||
let result = match &self.backend {
|
||||
ObjectDataCacheBackendKind::Noop(backend) => backend.invalidate_prefix().await,
|
||||
ObjectDataCacheBackendKind::Moka(backend) => backend.invalidate_prefix(bucket, prefix).await,
|
||||
};
|
||||
self.finish_invalidation(result, reason)
|
||||
}
|
||||
|
||||
/// Invalidates every cached body in `bucket`.
|
||||
///
|
||||
/// Backed by a full identity-index scan; keep it on the rare bucket-delete
|
||||
/// and admin paths only (ODC-28, backlog#1133).
|
||||
pub async fn invalidate_bucket(
|
||||
&self,
|
||||
bucket: &str,
|
||||
reason: ObjectDataCacheInvalidationReason,
|
||||
) -> ObjectDataCacheInvalidationResult {
|
||||
let result = match &self.backend {
|
||||
ObjectDataCacheBackendKind::Noop(backend) => backend.invalidate_bucket().await,
|
||||
ObjectDataCacheBackendKind::Moka(backend) => backend.invalidate_bucket(bucket).await,
|
||||
};
|
||||
self.finish_invalidation(result, reason)
|
||||
}
|
||||
|
||||
/// Drops every cached body and resets the identity index.
|
||||
///
|
||||
/// The only production remediation for a poisoned or stale entry short of a
|
||||
/// node restart (ODC-C2, backlog#1143). Rare admin path only.
|
||||
pub async fn clear(&self, reason: ObjectDataCacheInvalidationReason) -> ObjectDataCacheInvalidationResult {
|
||||
let result = match &self.backend {
|
||||
ObjectDataCacheBackendKind::Noop(backend) => backend.clear().await,
|
||||
ObjectDataCacheBackendKind::Moka(backend) => backend.clear().await,
|
||||
};
|
||||
self.finish_invalidation(result, reason)
|
||||
}
|
||||
|
||||
/// Shared post-processing for every invalidation primitive: bump the
|
||||
/// invalidation counter, refresh the cache-state gauge only when something
|
||||
/// was removed, and emit the outcome-labelled metric. A mutating op
|
||||
/// invalidates twice by design (before + after) and the vast majority of
|
||||
/// those touch identities that were never cached, so the no-op path skips
|
||||
/// the gauge refresh (backlog#1141).
|
||||
fn finish_invalidation(
|
||||
&self,
|
||||
result: ObjectDataCacheInvalidationResult,
|
||||
reason: ObjectDataCacheInvalidationReason,
|
||||
) -> ObjectDataCacheInvalidationResult {
|
||||
self.stats.record_invalidation();
|
||||
let outcome = invalidation_outcome(&result);
|
||||
// A mutating op invalidates twice by design (before + after); the vast
|
||||
// majority of those touch identities that were never cached. Only refresh
|
||||
// the cache-state gauge when something was actually removed so the
|
||||
// no-op path stays cheap (backlog#1141).
|
||||
if outcome != INVALIDATION_OUTCOME_NOOP {
|
||||
self.refresh_entry_count();
|
||||
}
|
||||
record_invalidation(self.backend.as_metric_label(), _reason.as_metric_label(), outcome);
|
||||
|
||||
record_invalidation(self.backend.as_metric_label(), reason.as_metric_label(), outcome);
|
||||
result
|
||||
}
|
||||
|
||||
@@ -285,6 +340,11 @@ impl ObjectDataCache {
|
||||
self.stats.snapshot()
|
||||
}
|
||||
|
||||
/// Returns the configured runtime mode, for admin status reporting.
|
||||
pub fn mode(&self) -> crate::config::ObjectDataCacheMode {
|
||||
self.config.mode
|
||||
}
|
||||
|
||||
/// Returns true when the cache facade is fully disabled.
|
||||
pub fn is_disabled(&self) -> bool {
|
||||
self.config.is_disabled()
|
||||
@@ -438,6 +498,15 @@ pub enum ObjectDataCacheInvalidationReason {
|
||||
AfterCopySuccess,
|
||||
/// Invalidation after a successful complete multipart upload.
|
||||
AfterCompleteMultipartSuccess,
|
||||
/// Invalidation after an ecstore-internal lifecycle/scanner expiry deleted
|
||||
/// the object body (ODC-26).
|
||||
AfterLifecycleExpiry,
|
||||
/// Invalidation after a forced prefix delete removed every object under a
|
||||
/// prefix (ODC-27).
|
||||
AfterPrefixDelete,
|
||||
/// Invalidation after a bucket delete removed every object in the bucket
|
||||
/// (ODC-28).
|
||||
AfterBucketDelete,
|
||||
/// Manual invalidation requested by the caller.
|
||||
Manual,
|
||||
}
|
||||
@@ -450,6 +519,9 @@ impl ObjectDataCacheInvalidationReason {
|
||||
Self::AfterDeleteSuccess => "after_delete_success",
|
||||
Self::AfterCopySuccess => "after_copy_success",
|
||||
Self::AfterCompleteMultipartSuccess => "after_complete_multipart_success",
|
||||
Self::AfterLifecycleExpiry => "after_lifecycle_expiry",
|
||||
Self::AfterPrefixDelete => "after_prefix_delete",
|
||||
Self::AfterBucketDelete => "after_bucket_delete",
|
||||
Self::Manual => "manual",
|
||||
}
|
||||
}
|
||||
@@ -471,7 +543,7 @@ pub enum ObjectDataCacheInvalidationResult {
|
||||
mod tests {
|
||||
use super::{
|
||||
ObjectDataCache, ObjectDataCacheFillResult, ObjectDataCacheGetPlan, ObjectDataCacheGetRequest,
|
||||
ObjectDataCacheInvalidationReason, ObjectDataCacheLookup,
|
||||
ObjectDataCacheInvalidationReason, ObjectDataCacheInvalidationResult, ObjectDataCacheLookup,
|
||||
};
|
||||
use crate::config::{ObjectDataCacheConfig, ObjectDataCacheMode};
|
||||
use crate::key::{ObjectDataCacheBodyVariant, ObjectDataCacheIdentity};
|
||||
@@ -678,6 +750,96 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn new_invalidation_reasons_map_to_distinct_labels() {
|
||||
assert_eq!(
|
||||
ObjectDataCacheInvalidationReason::AfterLifecycleExpiry.as_metric_label(),
|
||||
"after_lifecycle_expiry"
|
||||
);
|
||||
assert_eq!(
|
||||
ObjectDataCacheInvalidationReason::AfterPrefixDelete.as_metric_label(),
|
||||
"after_prefix_delete"
|
||||
);
|
||||
assert_eq!(
|
||||
ObjectDataCacheInvalidationReason::AfterBucketDelete.as_metric_label(),
|
||||
"after_bucket_delete"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn prefix_invalidation_of_cached_identity_labels_reason_and_removed() {
|
||||
// ODC-27: a prefix flush that drops a cached body is labelled with its
|
||||
// own reason and outcome=removed so dashboards attribute the churn.
|
||||
let cache = fill_enabled_cache();
|
||||
let metrics = capture_metrics(|| async {
|
||||
let plan = cache.plan_get(plain_request("bucket", "photos/a", "etag", 5));
|
||||
assert_eq!(
|
||||
cache.fill_body(&plan, Bytes::from_static(b"hello")).await,
|
||||
ObjectDataCacheFillResult::Inserted
|
||||
);
|
||||
let result = cache
|
||||
.invalidate_prefix("bucket", "photos/", ObjectDataCacheInvalidationReason::AfterPrefixDelete)
|
||||
.await;
|
||||
assert_eq!(result, ObjectDataCacheInvalidationResult::Removed { keys: 1 });
|
||||
});
|
||||
|
||||
assert!(has_counter_with_label(
|
||||
&metrics,
|
||||
"rustfs_object_data_cache_invalidations_total",
|
||||
("reason", "after_prefix_delete")
|
||||
));
|
||||
assert!(has_counter_with_label(
|
||||
&metrics,
|
||||
"rustfs_object_data_cache_invalidations_total",
|
||||
("outcome", "removed")
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bucket_invalidation_of_uncached_bucket_labels_noop() {
|
||||
// ODC-28: a bucket flush that matched nothing is a no-op and must not
|
||||
// refresh the entries gauge.
|
||||
let cache = fill_enabled_cache();
|
||||
let metrics = capture_metrics(|| async {
|
||||
let result = cache
|
||||
.invalidate_bucket("empty-bucket", ObjectDataCacheInvalidationReason::AfterBucketDelete)
|
||||
.await;
|
||||
assert_eq!(result, ObjectDataCacheInvalidationResult::NoOp);
|
||||
});
|
||||
|
||||
assert!(has_counter_with_label(
|
||||
&metrics,
|
||||
"rustfs_object_data_cache_invalidations_total",
|
||||
("outcome", "noop")
|
||||
));
|
||||
assert!(
|
||||
!has_gauge(&metrics, "rustfs_object_data_cache_entries"),
|
||||
"a no-op bucket flush must not refresh the entries gauge"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn clear_of_cached_cache_labels_manual_and_removed() {
|
||||
// ODC-C2: an admin clear reuses the Manual reason and reports removed.
|
||||
let cache = fill_enabled_cache();
|
||||
let metrics = capture_metrics(|| async {
|
||||
let plan = cache.plan_get(plain_request("bucket", "object", "etag", 5));
|
||||
assert_eq!(
|
||||
cache.fill_body(&plan, Bytes::from_static(b"hello")).await,
|
||||
ObjectDataCacheFillResult::Inserted
|
||||
);
|
||||
let result = cache.clear(ObjectDataCacheInvalidationReason::Manual).await;
|
||||
assert_eq!(result, ObjectDataCacheInvalidationResult::Removed { keys: 1 });
|
||||
assert!(matches!(cache.lookup_body(&plan).await, ObjectDataCacheLookup::Miss));
|
||||
});
|
||||
|
||||
assert!(has_counter_with_label(
|
||||
&metrics,
|
||||
"rustfs_object_data_cache_invalidations_total",
|
||||
("reason", "manual")
|
||||
));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn fill_body_rejects_size_mismatch() {
|
||||
let cache = fill_enabled_cache();
|
||||
|
||||
Reference in New Issue
Block a user