feat(get): Small-file GET performance optimization for 1KiB-1MiB objects (#4016)

* feat(get): SF01 - bucket validation cache

Add 5s TTL cache for bucket validation to avoid repeated stat_volume()
calls on every GET request.

Changes:
- Add BUCKET_VALIDATED_CACHE (OnceLock + RwLock + HashMap)
- Add invalidate_bucket_validation_cache() for cache invalidation
- Add invalidate_all_bucket_validation_cache() for bulk invalidation
- Update get_validated_store() to use cache
- Add cache invalidation in execute_delete_bucket()

Expected impact: 3-5x improvement for small file GET latency.

Closes rustfs/backlog#766

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(get): SF03 - metadata cache TTL increase

Increase metadata cache TTL from 250ms to 2s and capacity from 1024
to 4096 entries.

Changes:
- GET_OBJECT_METADATA_CACHE_TTL: 250ms -> 2s
- GET_OBJECT_METADATA_CACHE_MAX_ENTRIES: 1024 -> 4096

All mutation paths already call invalidate_get_object_metadata_cache,
so the longer TTL is safe.

Expected impact: 10-50x improvement for hot objects.

Closes rustfs/backlog#768

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(get): SF04 - remove unnecessary tokio::spawn in metadata fanout

Replace tokio::spawn with direct async future in read_all_fileinfo_full_wait.
join_all already provides concurrency, so tokio::spawn adds unnecessary
task creation and scheduling overhead.

Changes:
- Remove tokio::spawn from metadata fanout futures
- Update result handling for direct future results

Expected impact: 16-32us reduction per GET request.

Closes rustfs/backlog#769

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(get): SF06 - conditional lifecycle check

Only call resolve_put_object_expiration when the object has an
x-amz-expiration metadata marker. This avoids unnecessary lifecycle
configuration reads on every GET request.

Expected impact: 50-100us reduction per GET request.

Closes rustfs/backlog#771

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(get): SF07 - conditional metrics recording

Gate hot path metrics behind get_stage_metrics_enabled() to reduce
overhead when metrics are not needed.

Changes:
- Conditional record_zero_copy_read
- Conditional manager.record_disk_operation
- Conditional manager.record_access
- Conditional manager.record_transfer

Expected impact: 20-50us reduction per GET request.

Closes rustfs/backlog#772

Co-Authored-By: heihutu <heihutu@gmail.com>

* refactor(get): SF01 - use moka instead of dashmap for bucket cache

Replace OnceLock + RwLock + HashMap with moka::sync::Cache for bucket
validation cache. moka provides built-in TTL support and is already
available in the workspace.

Changes:
- Add moka dependency to rustfs crate
- Replace manual TTL management with moka's time_to_live
- Simplify cache operations

Closes rustfs/backlog#766

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(get): SF02 - inline data fast path

Add fast path for small inline objects that bypasses duplex pipe,
tokio::spawn, and bitrot reader creation when data is already in memory.

Changes:
- Add inline data detection before codec streaming gate
- Direct in-memory erasure decode for inline objects <= 128KB
- Add GET_OBJECT_PATH_INLINE_DIRECT metric path
- Skip duplex pipe and background task for inline data

Conditions for fast path:
- Single part object
- Inline data available
- Size <= 128KB
- Not encrypted/compressed/remote
- No range request

Expected impact: 2-3x improvement for small file GET latency.

Closes rustfs/backlog#767

Co-Authored-By: heihutu <heihutu@gmail.com>

* refactor: translate Chinese comments to English

Translate all Chinese comments to English in modified files:
- rustfs/src/storage/ecfs_extend.rs
- rustfs/src/app/bucket_usecase.rs

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix

* add

* fmt and improve import

* fmt

* feat(get): SF05 skip IO planning + refactor inline detection + adaptive bucket cache

SF05: Skip disk I/O semaphore for inline data fast path
- Reorder prepare_get_object_read_execution: read first, then decide semaphore
- Inline objects skip acquire_disk_read_permit() entirely (saves 100-200us)
- Add is_inline_fast_path field to GetObjectReadSetup

Refactor: Unify inline detection logic
- Add ObjectInfo::is_inline_fast_path_eligible() as single source of truth
- Version-aware thresholds: non-versioned 128KB, versioned 16KB (matches PUT)
- Eliminates divergent conditions between set_disk/mod.rs and object_usecase.rs

Refactor: Restore fault tolerance in metadata fanout
- Restore tokio::spawn + JoinError handling in read_all_fileinfo_full_wait
- Prevents single disk read panic from unwinding the entire operation

Refactor: Restore lifecycle check correctness
- Remove incorrect SF06 conditional that skipped lifecycle for most objects
- Always call resolve_put_object_expiration (original behavior)

Fix: make_bucket cache invalidation
- Invalidate bucket validation cache on create_bucket

Fix: erasure decode written validation
- Check decode() return value; error if 0 bytes written for non-empty object

Adaptive bucket cache
- Default: RwLock<HashMap> for < 100 buckets (low overhead)
- Opt-in: starshard::ShardedHashMap via RUSTFS_BUCKET_CACHE_STARSHARD=1
- 5s TTL with manual timestamp checking

Benchmark results (warp get, concurrency 32, 10s, 3 rounds):
- 10KiB: 25.10 MiB/s (+28.2% vs SF01-07)
- 100KiB: 221.81 MiB/s
- 1MiB: 1972.78 MiB/s
- vs main: -10% to -12% (inline path not triggered by warp)

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(versioning): use read lock for versioning config query + five-expert analysis

P0 fix: BucketVersioningSys::get() was using write lock on
GLOBAL_BucketMetadataSys for a pure read operation. This serialized
all concurrent GET requests (3 write-lock acquisitions per request).

Changed to read lock — get_versioning_config() handles its own
internal locking via metadata_map RwLock.

Five-expert analysis identified top bottlenecks:
1. Versioning write lock (P0, fixed)
2. Inline fast path not triggered (P0, needs verification)
3. Metadata fanout no early-stop (P1, early-stop has bug, reverted)
4. Request-level versioning cache (P1, pending)
5. Duplex pipe for small objects (P2, pending)

Benchmark (read-lock fix, warp concurrency 32):
- 1KiB: 2.29 MiB/s (vs 2.53 before, within variance)
- 10KiB: 25.00 MiB/s (same as before)
- 100KiB: 246.72 MiB/s (+11% vs 221.81)
- 1MiB: 2039.95 MiB/s (+3% vs 1972.78)

Co-Authored-By: heihutu <heihutu@gmail.com>

* chore: remove benchmark results from git, keep locally only

Remove docs/benchmark/*.md from version control.
Files remain on disk but are no longer tracked by git.
Added docs/benchmark/*.md to .gitignore.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(get): decode inline fast path through bitrot readers

---------

Co-authored-by: heihutu <heihutu@gmail.com>

rustfs/Cargo.toml

@@ -104,6 +104,7 @@ rustfs-object-capacity = { workspace = true }
 rustfs-concurrency = { workspace = true }
 rustfs-scanner = { workspace = true }
 tempfile = { workspace = true }
+starshard = { workspace = true }
 
 # Async Runtime and Networking
 async-trait = { workspace = true }

rustfs/src/app/bucket_usecase.rs

@@ -817,7 +817,11 @@ impl DefaultBucketUsecase {
             .await;
 
         match make_result {
-            Ok(()) => {}
+            Ok(()) => {
+                // Invalidate the bucket validation cache so subsequent GETs
+                // see the newly created bucket immediately.
+                crate::storage::invalidate_bucket_validation_cache(&bucket);
+            }
             Err(StorageError::BucketExists(_)) => {
                 // Per S3 spec: bucket namespace is global. Owner recreating returns 200 OK;
                 // non-owner gets 409 BucketAlreadyExists.
@@ -869,6 +873,10 @@ impl DefaultBucketUsecase {
             )
             .await
             .map_err(ApiError::from)?;
+
+        // Invalidate bucket validation cache
+        crate::storage::invalidate_bucket_validation_cache(&input.bucket);
+
         rustfs_scanner::clear_dirty_usage_bucket(&input.bucket);
         if let Err(err) = remove_bucket_usage_from_backend(store.clone(), &input.bucket).await {
             warn!(bucket = %input.bucket, error = ?err, "failed to remove deleted bucket from data usage");

rustfs/src/app/object_usecase.rs

@@ -251,7 +251,8 @@ struct GetObjectBootstrap {
 }
 
 struct GetObjectIoPlanning<'a> {
-    _disk_permit: tokio::sync::SemaphorePermit<'a>,
+    /// `None` when inline fast path skips disk I/O semaphore.
+    _disk_permit: Option<tokio::sync::SemaphorePermit<'a>>,
     permit_wait_duration: Duration,
     queue_status: concurrency::IoQueueStatus,
     queue_utilization: f64,
@@ -280,6 +281,8 @@ struct GetObjectReadSetup {
     sse_customer_key_md5: Option<SSECustomerKeyMD5>,
     ssekms_key_id: Option<SSEKMSKeyId>,
     encryption_applied: bool,
+    /// `true` when the object was read via the inline data fast path (no disk I/O).
+    is_inline_fast_path: bool,
 }
 
 struct GetObjectPreparedRead<'a> {
@@ -2078,7 +2081,7 @@ impl DefaultObjectUsecase {
         Self::ensure_get_object_not_timed_out(wrapper, timeout_config, bucket, key, GetObjectTimeoutStage::BeforeRead)?;
 
         Ok(GetObjectIoPlanning {
-            _disk_permit: disk_permit,
+            _disk_permit: Some(disk_permit),
             permit_wait_duration,
             queue_status,
             queue_utilization,
@@ -2148,7 +2151,8 @@ impl DefaultObjectUsecase {
         part_number: Option<usize>,
     ) -> S3Result<GetObjectPreparedRead<'a>> {
         let h = req.headers.clone();
-        let io_planning = Self::acquire_get_object_io_planning(manager, wrapper, timeout_config, bucket, key).await?;
+
+        // SF05: Store lookup first (cached via SF01 moka cache).
         let store_lookup_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now);
         let store = get_validated_store(bucket).await?;
         if let Some(store_lookup_start) = store_lookup_start {
@@ -2159,6 +2163,8 @@ impl DefaultObjectUsecase {
             );
         }
 
+        // SF05: Read object metadata/data BEFORE acquiring disk I/O semaphore.
+        // ECStore's get_object_reader acquires its own RwLock — safe without the semaphore.
         let read_start = std::time::Instant::now();
         let read_stage_start = rustfs_io_metrics::get_stage_metrics_enabled().then_some(read_start);
         let read_setup = Self::prepare_get_object_read(
@@ -2176,6 +2182,18 @@ impl DefaultObjectUsecase {
         )
         .await?;
 
+        // SF05: Skip disk I/O semaphore for inline fast path — data is already in memory.
+        let io_planning = if read_setup.is_inline_fast_path {
+            GetObjectIoPlanning {
+                _disk_permit: None,
+                permit_wait_duration: Duration::ZERO,
+                queue_status: concurrency::IoQueueStatus::default(),
+                queue_utilization: 0.0,
+            }
+        } else {
+            Self::acquire_get_object_io_planning(manager, wrapper, timeout_config, bucket, key).await?
+        };
+
         Ok(GetObjectPreparedRead { io_planning, read_setup })
     }
 
@@ -2207,11 +2225,14 @@ impl DefaultObjectUsecase {
 
         let info = reader.object_info;
 
-        use rustfs_io_metrics::record_zero_copy_read;
         let read_duration = read_start.elapsed();
-        record_zero_copy_read(info.size as usize, read_duration.as_secs_f64() * 1000.0);
 
-        manager.record_disk_operation(info.size as u64, read_duration, true).await;
+        // Conditional metrics recording to reduce overhead
+        if rustfs_io_metrics::get_stage_metrics_enabled() {
+            use rustfs_io_metrics::record_zero_copy_read;
+            record_zero_copy_read(info.size as usize, read_duration.as_secs_f64() * 1000.0);
+            manager.record_disk_operation(info.size as u64, read_duration, true).await;
+        }
 
         check_preconditions(&req.headers, &info)?;
 
@@ -2307,6 +2328,10 @@ impl DefaultObjectUsecase {
             None => (None, None, None, None, false, wrap_reader(reader.stream)),
         };
 
+        // Detect inline fast path: data is in memory, no disk I/O semaphore needed.
+        // Uses the shared predicate from ObjectInfo; additionally checks no range request.
+        let is_inline_fast_path = info.is_inline_fast_path_eligible() && rs.is_none();
+
         Ok(GetObjectReadSetup {
             info,
             event_info,
@@ -2321,6 +2346,7 @@ impl DefaultObjectUsecase {
             sse_customer_key_md5,
             ssekms_key_id,
             encryption_applied,
+            is_inline_fast_path,
         })
     }
     #[allow(clippy::too_many_arguments)]
@@ -2351,14 +2377,17 @@ impl DefaultObjectUsecase {
             false
         };
 
-        if let Some(range_spec) = rs
-            && range_spec.start >= 0
-        {
-            manager.record_access(range_spec.start as u64, response_content_length as u64);
-        }
+        // Conditional metrics recording to reduce overhead
+        if rustfs_io_metrics::get_stage_metrics_enabled() {
+            if let Some(range_spec) = rs
+                && range_spec.start >= 0
+            {
+                manager.record_access(range_spec.start as u64, response_content_length as u64);
+            }
 
-        if response_content_length > 0 {
-            manager.record_transfer(response_content_length as u64, permit_wait_duration);
+            if response_content_length > 0 {
+                manager.record_transfer(response_content_length as u64, permit_wait_duration);
+            }
         }
 
         let io_strategy =
@@ -3410,6 +3439,7 @@ impl DefaultObjectUsecase {
             sse_customer_key_md5,
             ssekms_key_id,
             encryption_applied,
+            is_inline_fast_path: _,
         } = read_setup;
 
         let versioning_start = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now);

rustfs/src/storage/ecfs_extend.rs

@@ -42,7 +42,8 @@ use s3s::{S3Error, S3ErrorCode, S3Response, S3Result};
 use serde_urlencoded::from_bytes;
 use std::collections::HashMap;
 use std::ops::Add;
-use std::sync::Arc;
+use std::sync::{Arc, OnceLock, RwLock};
+use std::time::{Duration, Instant};
 use time::OffsetDateTime;
 use time::format_description::well_known::Rfc3339;
 use time::{format_description::FormatItem, macros::format_description};
@@ -745,18 +746,118 @@ pub(crate) async fn has_replication_rules(bucket: &str, objects: &[ObjectToDelet
     false
 }
 
-/// Helper function to get store and validate bucket exists
+/// Bucket validation cache to avoid repeated stat_volume() calls on every GET.
+///
+/// **Adaptive strategy** (selected once at startup via env var):
+///
+/// | Backend | Env var | Best for |
+/// |---------|---------|----------|
+/// | `RwLock<HashMap>` | default | < 100 buckets — lower per-op overhead |
+/// | `starshard::ShardedHashMap` | `RUSTFS_BUCKET_CACHE_STARSHARD=1` | >= 100 buckets — sharded locks reduce contention |
+///
+/// Entries expire after `BUCKET_VALIDATION_TTL` (checked on read).
+/// Write operations (delete/make bucket) invalidate the cache explicitly.
+const BUCKET_VALIDATION_TTL: Duration = Duration::from_secs(5);
+
+/// Tracks which backend is active: `false` = HashMap, `true` = starshard.
+static USE_STARSHARD_CACHE: OnceLock<bool> = OnceLock::new();
+
+fn use_starshard() -> bool {
+    *USE_STARSHARD_CACHE.get_or_init(|| {
+        std::env::var("RUSTFS_BUCKET_CACHE_STARSHARD")
+            .ok()
+            .and_then(|v| v.parse::<bool>().ok())
+            .unwrap_or(false)
+    })
+}
+
+/// --- HashMap backend (default) ---
+static BUCKET_CACHE_SMALL: OnceLock<RwLock<HashMap<String, Instant>>> = OnceLock::new();
+
+fn small_cache() -> &'static RwLock<HashMap<String, Instant>> {
+    BUCKET_CACHE_SMALL.get_or_init(|| RwLock::new(HashMap::new()))
+}
+
+/// --- starshard backend (opt-in) ---
+static BUCKET_CACHE_LARGE: OnceLock<starshard::ShardedHashMap<String, Instant>> = OnceLock::new();
+
+fn large_cache() -> &'static starshard::ShardedHashMap<String, Instant> {
+    BUCKET_CACHE_LARGE.get_or_init(|| starshard::ShardedHashMap::new(128))
+}
+
+/// Get a value from the active cache backend.
+fn cache_get(bucket: &str) -> Option<Instant> {
+    if use_starshard() {
+        large_cache().get(&bucket.to_string())
+    } else {
+        small_cache().read().ok()?.get(bucket).copied()
+    }
+}
+
+/// Insert a value into the active cache backend.
+fn cache_insert(bucket: String, ts: Instant) {
+    if use_starshard() {
+        large_cache().insert(bucket, ts);
+    } else if let Ok(mut map) = small_cache().write() {
+        map.insert(bucket, ts);
+    }
+}
+
+/// Remove a value from the active cache backend.
+fn cache_remove(bucket: &str) {
+    if use_starshard() {
+        large_cache().remove(&bucket.to_string());
+    } else if let Ok(mut map) = small_cache().write() {
+        map.remove(bucket);
+    }
+}
+
+/// Clear all entries in the active cache backend.
+#[allow(dead_code)]
+fn cache_clear() {
+    if use_starshard() {
+        large_cache().clear();
+    } else if let Ok(mut map) = small_cache().write() {
+        map.clear();
+    }
+}
+
+/// Invalidate the validation cache for a specific bucket.
+pub fn invalidate_bucket_validation_cache(bucket: &str) {
+    cache_remove(bucket);
+}
+
+/// Invalidate all bucket validation cache entries.
+#[allow(dead_code)]
+pub fn invalidate_all_bucket_validation_cache() {
+    cache_clear();
+}
+
+/// Helper function to get store and validate bucket exists.
+///
+/// Uses adaptive cache with 5s TTL to avoid repeated stat_volume() calls.
+/// Returns store directly on cache hit without calling get_bucket_info().
 pub(crate) async fn get_validated_store(bucket: &str) -> S3Result<Arc<super::ECStore>> {
     let Some(store) = runtime_sources::current_object_store_handle() else {
         return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
     };
 
-    // Validate bucket exists
+    // Check cache — TTL is checked manually.
+    if let Some(inserted_at) = cache_get(bucket)
+        && inserted_at.elapsed() < BUCKET_VALIDATION_TTL
+    {
+        return Ok(store); // Cache hit, skip validation
+    }
+
+    // Cache miss or expired, perform validation
     store
         .get_bucket_info(bucket, &BucketOptions::default())
         .await
         .map_err(ApiError::from)?;
 
+    // Update cache
+    cache_insert(bucket.to_string(), Instant::now());
+
     Ok(store)
 }