Files
rustfs/crates/object-data-cache/src/key.rs
T
houseme eebd16d8a4 feat(cache): add object data cache engine and app flow (#4187)
* feat(cache): add object data cache engine

* feat(cache): wire app-layer object cache flow

* refactor(cache): streamline app-layer cache flow

* refactor(cache): tighten cache flow internals

* refactor: address final clippy cleanup

* chore(deps): update quick-xml to 0.41.0

* feat(cache): wire object data cache env config

* fix(cache): gate materialize fill by cache plan

* chore(cache): add object data cache benchmark gate

* fix(cache): guard object cache fill size mismatches

* refactor(cache): streamline object cache body planning

* fix(cache): align object cache rollout config

* test(cache): cover buffered object cache benchmark

* test(cache): isolate object cache benchmark metrics

* test(cache): mark materialize rollout experimental

* test(cache): tighten object cache benchmark gate

* fix(cache): address review findings for object data cache

- singleflight: clean up leader entry on cancellation (Drop impl) so a dropped GET future can no longer wedge all subsequent fills for the same key; switch the fill map to a std Mutex and add a regression test

- adapter: honor RUSTFS_OBJECT_DATA_CACHE_ENABLE=true by defaulting to hit_only when no explicit mode is set (explicit mode still wins)

- planner: treat nil version UUIDs as "no value" per repo convention so unversioned objects key under the canonical "null" instead of fragmenting the key space

- multipart: invalidate the object cache on the quota-exceeded rollback delete after complete-multipart, closing a stale-cache window

- layering: move the disabled-cache fallback into app::context and drop the new infra->app layer-dependency baseline entry

* fix(cache): close invalidation races and drop full-cache scan on writes

- index: make identity-index insert/remove/prune atomic via starshard compute_if_present/compute_if_absent so concurrent fills can no longer drop each other's keys (lost keys made entries unreachable to invalidation until TTL); add a concurrency regression test

- fill: register the key in the identity index before the entry becomes visible in the cache and re-check the index afterwards, undoing the fill when an invalidation raced in between (new skipped_invalidation_race fill result)

- invalidate: with the index now authoritative, remove the full-cache iter() fallback that made every PUT/DELETE of a never-cached object O(total cache entries) (two scans per PUT, 2N per batch delete)

- materialize-fill: fail the GET instead of falling back to the partially consumed stream after a mid-read error (the fallback would send a body missing its prefix under a full-length Content-Length), and log the same size-mismatch warning as the sibling buffering paths

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(storage): fix media-dependent buffer clamp expectation

test_concurrency_manager_multi_factor_strategy_buffer_clamp asserted media_cap.min(MI_B), but the implementation's final safety clamp is [32KiB, media_cap.max(MI_B)] — deliberately so a media cap above 1MiB (NVMe's 2MiB default) stays effective. The test only passed on machines detected as SSD/Unknown (cap == 1MiB) and failed on NVMe-backed CI runners with 2MiB != 1MiB. Assert the media cap itself, which is what the strategy actually guarantees on every environment.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(storage): format buffer clamp assertion

* chore(logging): update tier guardrail path

---------

Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-03 18:11:14 +08:00

119 lines
3.8 KiB
Rust

// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::sync::Arc;
/// Canonical synthetic version id for unversioned or latest-only object bodies.
pub const NULL_VERSION_ID: &str = "null";
/// Response body variant supported by the cache.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
pub enum ObjectDataCacheBodyVariant {
/// Full plain object body.
#[default]
FullObjectPlainV1,
}
/// Stable cache key for a reusable object body.
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct ObjectDataCacheKey {
/// Bucket name.
pub bucket: Arc<str>,
/// Object key.
pub object: Arc<str>,
/// Canonical version id, using `"null"` for unversioned bodies.
pub version_id: Arc<str>,
/// Object ETag.
pub etag: Arc<str>,
/// Object size in bytes.
pub size: u64,
/// Cached body semantics.
pub body_variant: ObjectDataCacheBodyVariant,
}
/// Identity used for conservative invalidation.
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct ObjectDataCacheIdentity {
/// Bucket name.
pub bucket: Arc<str>,
/// Object key.
pub object: Arc<str>,
}
impl ObjectDataCacheKey {
/// Creates a new stable object data cache key.
pub fn new(
bucket: impl Into<Arc<str>>,
object: impl Into<Arc<str>>,
version_id: Option<&str>,
etag: impl Into<Arc<str>>,
size: u64,
body_variant: ObjectDataCacheBodyVariant,
) -> Self {
Self {
bucket: bucket.into(),
object: object.into(),
version_id: version_id.map_or_else(|| Arc::<str>::from(NULL_VERSION_ID), Arc::<str>::from),
etag: etag.into(),
size,
body_variant,
}
}
/// Returns true when this key targets the canonical unversioned body variant.
pub fn is_null_version(&self) -> bool {
self.version_id.as_ref() == NULL_VERSION_ID
}
}
impl ObjectDataCacheIdentity {
/// Creates a conservative invalidation identity.
pub fn new(bucket: impl Into<Arc<str>>, object: impl Into<Arc<str>>) -> Self {
Self {
bucket: bucket.into(),
object: object.into(),
}
}
}
#[cfg(test)]
mod tests {
use super::{NULL_VERSION_ID, ObjectDataCacheBodyVariant, ObjectDataCacheIdentity, ObjectDataCacheKey};
#[test]
fn key_uses_canonical_null_version_for_missing_version_id() {
let key = ObjectDataCacheKey::new("bucket", "object", None, "etag", 42, ObjectDataCacheBodyVariant::FullObjectPlainV1);
assert_eq!(key.version_id.as_ref(), NULL_VERSION_ID);
assert!(key.is_null_version());
}
#[test]
fn key_distinguishes_explicit_version_ids() {
let latest = ObjectDataCacheKey::new("bucket", "object", None, "etag", 42, ObjectDataCacheBodyVariant::FullObjectPlainV1);
let versioned =
ObjectDataCacheKey::new("bucket", "object", Some("3d2"), "etag", 42, ObjectDataCacheBodyVariant::FullObjectPlainV1);
assert_ne!(latest, versioned);
}
#[test]
fn identity_new_preserves_bucket_and_object() {
let identity = ObjectDataCacheIdentity::new("bucket", "object");
assert_eq!(identity.bucket.as_ref(), "bucket");
assert_eq!(identity.object.as_ref(), "object");
}
}