feat(get): Small-file GET performance optimization for 1KiB-1MiB objects (#4016)

* feat(get): SF01 - bucket validation cache

Add 5s TTL cache for bucket validation to avoid repeated stat_volume()
calls on every GET request.

Changes:
- Add BUCKET_VALIDATED_CACHE (OnceLock + RwLock + HashMap)
- Add invalidate_bucket_validation_cache() for cache invalidation
- Add invalidate_all_bucket_validation_cache() for bulk invalidation
- Update get_validated_store() to use cache
- Add cache invalidation in execute_delete_bucket()

Expected impact: 3-5x improvement for small file GET latency.

Closes rustfs/backlog#766

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(get): SF03 - metadata cache TTL increase

Increase metadata cache TTL from 250ms to 2s and capacity from 1024
to 4096 entries.

Changes:
- GET_OBJECT_METADATA_CACHE_TTL: 250ms -> 2s
- GET_OBJECT_METADATA_CACHE_MAX_ENTRIES: 1024 -> 4096

All mutation paths already call invalidate_get_object_metadata_cache,
so the longer TTL is safe.

Expected impact: 10-50x improvement for hot objects.

Closes rustfs/backlog#768

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(get): SF04 - remove unnecessary tokio::spawn in metadata fanout

Replace tokio::spawn with direct async future in read_all_fileinfo_full_wait.
join_all already provides concurrency, so tokio::spawn adds unnecessary
task creation and scheduling overhead.

Changes:
- Remove tokio::spawn from metadata fanout futures
- Update result handling for direct future results

Expected impact: 16-32us reduction per GET request.

Closes rustfs/backlog#769

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(get): SF06 - conditional lifecycle check

Only call resolve_put_object_expiration when the object has an
x-amz-expiration metadata marker. This avoids unnecessary lifecycle
configuration reads on every GET request.

Expected impact: 50-100us reduction per GET request.

Closes rustfs/backlog#771

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(get): SF07 - conditional metrics recording

Gate hot path metrics behind get_stage_metrics_enabled() to reduce
overhead when metrics are not needed.

Changes:
- Conditional record_zero_copy_read
- Conditional manager.record_disk_operation
- Conditional manager.record_access
- Conditional manager.record_transfer

Expected impact: 20-50us reduction per GET request.

Closes rustfs/backlog#772

Co-Authored-By: heihutu <heihutu@gmail.com>

* refactor(get): SF01 - use moka instead of dashmap for bucket cache

Replace OnceLock + RwLock + HashMap with moka::sync::Cache for bucket
validation cache. moka provides built-in TTL support and is already
available in the workspace.

Changes:
- Add moka dependency to rustfs crate
- Replace manual TTL management with moka's time_to_live
- Simplify cache operations

Closes rustfs/backlog#766

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(get): SF02 - inline data fast path

Add fast path for small inline objects that bypasses duplex pipe,
tokio::spawn, and bitrot reader creation when data is already in memory.

Changes:
- Add inline data detection before codec streaming gate
- Direct in-memory erasure decode for inline objects <= 128KB
- Add GET_OBJECT_PATH_INLINE_DIRECT metric path
- Skip duplex pipe and background task for inline data

Conditions for fast path:
- Single part object
- Inline data available
- Size <= 128KB
- Not encrypted/compressed/remote
- No range request

Expected impact: 2-3x improvement for small file GET latency.

Closes rustfs/backlog#767

Co-Authored-By: heihutu <heihutu@gmail.com>

* refactor: translate Chinese comments to English

Translate all Chinese comments to English in modified files:
- rustfs/src/storage/ecfs_extend.rs
- rustfs/src/app/bucket_usecase.rs

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix

* add

* fmt and improve import

* fmt

* feat(get): SF05 skip IO planning + refactor inline detection + adaptive bucket cache

SF05: Skip disk I/O semaphore for inline data fast path
- Reorder prepare_get_object_read_execution: read first, then decide semaphore
- Inline objects skip acquire_disk_read_permit() entirely (saves 100-200us)
- Add is_inline_fast_path field to GetObjectReadSetup

Refactor: Unify inline detection logic
- Add ObjectInfo::is_inline_fast_path_eligible() as single source of truth
- Version-aware thresholds: non-versioned 128KB, versioned 16KB (matches PUT)
- Eliminates divergent conditions between set_disk/mod.rs and object_usecase.rs

Refactor: Restore fault tolerance in metadata fanout
- Restore tokio::spawn + JoinError handling in read_all_fileinfo_full_wait
- Prevents single disk read panic from unwinding the entire operation

Refactor: Restore lifecycle check correctness
- Remove incorrect SF06 conditional that skipped lifecycle for most objects
- Always call resolve_put_object_expiration (original behavior)

Fix: make_bucket cache invalidation
- Invalidate bucket validation cache on create_bucket

Fix: erasure decode written validation
- Check decode() return value; error if 0 bytes written for non-empty object

Adaptive bucket cache
- Default: RwLock<HashMap> for < 100 buckets (low overhead)
- Opt-in: starshard::ShardedHashMap via RUSTFS_BUCKET_CACHE_STARSHARD=1
- 5s TTL with manual timestamp checking

Benchmark results (warp get, concurrency 32, 10s, 3 rounds):
- 10KiB: 25.10 MiB/s (+28.2% vs SF01-07)
- 100KiB: 221.81 MiB/s
- 1MiB: 1972.78 MiB/s
- vs main: -10% to -12% (inline path not triggered by warp)

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(versioning): use read lock for versioning config query + five-expert analysis

P0 fix: BucketVersioningSys::get() was using write lock on
GLOBAL_BucketMetadataSys for a pure read operation. This serialized
all concurrent GET requests (3 write-lock acquisitions per request).

Changed to read lock — get_versioning_config() handles its own
internal locking via metadata_map RwLock.

Five-expert analysis identified top bottlenecks:
1. Versioning write lock (P0, fixed)
2. Inline fast path not triggered (P0, needs verification)
3. Metadata fanout no early-stop (P1, early-stop has bug, reverted)
4. Request-level versioning cache (P1, pending)
5. Duplex pipe for small objects (P2, pending)

Benchmark (read-lock fix, warp concurrency 32):
- 1KiB: 2.29 MiB/s (vs 2.53 before, within variance)
- 10KiB: 25.00 MiB/s (same as before)
- 100KiB: 246.72 MiB/s (+11% vs 221.81)
- 1MiB: 2039.95 MiB/s (+3% vs 1972.78)

Co-Authored-By: heihutu <heihutu@gmail.com>

* chore: remove benchmark results from git, keep locally only

Remove docs/benchmark/*.md from version control.
Files remain on disk but are no longer tracked by git.
Added docs/benchmark/*.md to .gitignore.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(get): decode inline fast path through bitrot readers

---------

Co-authored-by: heihutu <heihutu@gmail.com>

crates/ecstore/src/object_api/readers.rs

@@ -1,3 +1,17 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
 use super::*;
 #[cfg(feature = "rio-v2")]
 use aes_gcm::aead::Payload;
@@ -122,7 +136,7 @@ fn restore_request_active(opts: &ObjectOptions) -> bool {
     restore.type_.is_some() || restore.days.is_some() || restore.output_location.is_some() || restore.select_parameters.is_some()
 }
 
-fn decode_compression_index(index: Option<&bytes::Bytes>) -> Option<Index> {
+fn decode_compression_index(index: Option<&Bytes>) -> Option<Index> {
     crate::io_support::rio::decode_compression_index_bytes(index?)
 }
 
@@ -833,11 +847,7 @@ impl<R: AsyncRead + Unpin + Send + Sync + 'static> RangedDecompressReader<R> {
 }
 
 impl<R: AsyncRead + Unpin + Send + Sync + 'static> AsyncRead for RangedDecompressReader<R> {
-    fn poll_read(
-        mut self: std::pin::Pin<&mut Self>,
-        cx: &mut std::task::Context<'_>,
-        buf: &mut tokio::io::ReadBuf<'_>,
-    ) -> std::task::Poll<std::io::Result<()>> {
+    fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<std::io::Result<()>> {
         use std::pin::Pin;
         use std::task::Poll;
         use tokio::io::ReadBuf;
@@ -985,11 +995,7 @@ impl<R: AsyncRead + Unpin + Send + 'static> StreamConsumer<R> {
 }
 
 impl<R: AsyncRead + Unpin + Send + 'static> AsyncRead for StreamConsumer<R> {
-    fn poll_read(
-        mut self: std::pin::Pin<&mut Self>,
-        cx: &mut std::task::Context<'_>,
-        buf: &mut tokio::io::ReadBuf<'_>,
-    ) -> std::task::Poll<std::io::Result<()>> {
+    fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<std::io::Result<()>> {
         use std::pin::Pin;
         use std::task::Poll;
 
@@ -1032,7 +1038,7 @@ fn encrypted_plaintext_size(oi: &ObjectInfo, is_multipart: bool, is_compressed:
     oi.decrypted_size().map_err(Into::into)
 }
 
-fn is_multipart_encrypted_object(parts: &[rustfs_filemeta::ObjectPartInfo], etag: Option<&str>) -> bool {
+fn is_multipart_encrypted_object(parts: &[ObjectPartInfo], etag: Option<&str>) -> bool {
     if parts.len() > 1 {
         return true;
     }
@@ -1040,13 +1046,13 @@ fn is_multipart_encrypted_object(parts: &[rustfs_filemeta::ObjectPartInfo], etag
     etag.map(|etag| etag.trim_matches('"').len() != 32).unwrap_or(false)
 }
 
-fn multipart_plaintext_size(parts: &[rustfs_filemeta::ObjectPartInfo], fallback: i64) -> i64 {
+fn multipart_plaintext_size(parts: &[ObjectPartInfo], fallback: i64) -> i64 {
     let total: i64 = parts.iter().map(part_plaintext_size).sum();
 
     if total > 0 { total } else { fallback }
 }
 
-fn multipart_part_numbers(parts: &[rustfs_filemeta::ObjectPartInfo]) -> Vec<usize> {
+fn multipart_part_numbers(parts: &[ObjectPartInfo]) -> Vec<usize> {
     parts.iter().map(|part| part.number).collect()
 }
 
@@ -1605,13 +1611,13 @@ mod tests {
 
     fn ssec_headers_from_key(key_bytes: [u8; 32]) -> HeaderMap<HeaderValue> {
         let mut headers = HeaderMap::new();
-        headers.insert(rustfs_utils::http::SSEC_ALGORITHM_HEADER, HeaderValue::from_static("AES256"));
+        headers.insert(SSEC_ALGORITHM_HEADER, HeaderValue::from_static("AES256"));
         headers.insert(
-            rustfs_utils::http::SSEC_KEY_HEADER,
+            SSEC_KEY_HEADER,
             HeaderValue::from_str(&BASE64_STANDARD.encode(key_bytes)).expect("valid base64 header"),
         );
         headers.insert(
-            rustfs_utils::http::SSEC_KEY_MD5_HEADER,
+            SSEC_KEY_MD5_HEADER,
             HeaderValue::from_str(&BASE64_STANDARD.encode(md5_bytes(key_bytes))).expect("valid md5 header"),
         );
         headers
@@ -2616,7 +2622,7 @@ mod tests {
 
     #[tokio::test]
     async fn test_get_object_reader_compressed_range_returns_physical_offset_from_index() {
-        let mut index = crate::io_support::rio::Index::new();
+        let mut index = Index::new();
         index.add(0, 0).unwrap();
         index.add(1_048_576, 2_097_152).unwrap();
 
@@ -2670,7 +2676,7 @@ mod tests {
 
     #[tokio::test]
     async fn test_read_plan_compressed_range_tracks_storage_and_visible_offsets() {
-        let mut index = crate::io_support::rio::Index::new();
+        let mut index = Index::new();
         index.add(0, 0).unwrap();
         index.add(1_048_576, 2_097_152).unwrap();
 
@@ -2746,7 +2752,7 @@ mod tests {
     #[cfg(feature = "rio-v2")]
     #[tokio::test]
     async fn test_read_plan_accepts_minio_headerless_compression_index() {
-        let mut index = crate::io_support::rio::Index::new();
+        let mut index = Index::new();
         index.add(0, 0).unwrap();
         index.add(1_048_576, 2_097_152).unwrap();
         let headerless_index = crate::io_support::rio::compression_index_storage_bytes(&index);
@@ -2800,7 +2806,7 @@ mod tests {
     #[cfg(feature = "rio-v2")]
     #[test]
     fn test_get_compressed_offsets_aligns_encrypted_ranges_to_dare_packages() {
-        let mut index = crate::io_support::rio::Index::new();
+        let mut index = Index::new();
         index.add(0, 0).unwrap();
         index.add(200_000, 2_097_152).unwrap();
         let stored_index = crate::io_support::rio::compression_index_storage_bytes(&index);

crates/ecstore/src/object_api/types.rs

@@ -1,3 +1,17 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
 use super::*;
 use crate::storage_api_contracts::{
     list::VersionMarker,
@@ -263,6 +277,46 @@ impl ObjectInfo {
         })
     }
 
+    /// Maximum inline size for non-versioned objects (128 KiB).
+    /// Matches `DEFAULT_INLINE_BLOCK` in `storageclass.rs`.
+    pub const INLINE_MAX_SIZE: i64 = 128 * 1024;
+
+    /// Maximum inline size for versioned objects (16 KiB).
+    /// Matches `DEFAULT_INLINE_BLOCK / 8` in `storageclass.rs`.
+    pub const INLINE_MAX_SIZE_VERSIONED: i64 = 16 * 1024;
+
+    /// Returns `true` when this object qualifies for the inline data fast path.
+    ///
+    /// The inline fast path decodes erasure-coded data entirely in memory,
+    /// bypassing disk I/O, duplex pipes, and the disk-read semaphore.
+    ///
+    /// The `inlined` flag is the primary signal — it is set during PUT by
+    /// `storage_class_should_inline()` which already applies the correct
+    /// version-aware threshold (128 KiB non-versioned, 16 KiB versioned).
+    /// The size check below is a safety net using the same thresholds.
+    ///
+    /// Additional conditions:
+    /// - Single part
+    /// - Not encrypted
+    /// - Not compressed
+    /// - Not transitioned to remote tier
+    pub fn is_inline_fast_path_eligible(&self) -> bool {
+        if !self.inlined {
+            return false;
+        }
+        // Apply the same version-aware threshold as PUT (storageclass.rs).
+        let max_size = if self.version_id.is_some() {
+            Self::INLINE_MAX_SIZE_VERSIONED
+        } else {
+            Self::INLINE_MAX_SIZE
+        };
+        self.parts.len() == 1
+            && self.size <= max_size
+            && !self.is_encrypted()
+            && !self.is_compressed()
+            && self.transitioned_object.tier.is_empty()
+    }
+
     pub fn encryption_original_size(&self) -> std::io::Result<Option<i64>> {
         let actual_size = rustfs_utils::http::get_str(&self.user_defined, rustfs_utils::http::SUFFIX_ACTUAL_SIZE);
         if let Some(size_str) = self
@@ -354,14 +408,14 @@ impl ObjectInfo {
         // Parse expires from metadata (HTTP date format RFC 7231 or ISO 8601)
         let expires = fi.metadata.get("expires").and_then(|s| {
             // Try parsing as ISO 8601 first
-            time::OffsetDateTime::parse(s, &time::format_description::well_known::Iso8601::DEFAULT)
+            OffsetDateTime::parse(s, &time::format_description::well_known::Iso8601::DEFAULT)
                 .or_else(|_| {
                     // Try RFC 2822 format
-                    time::OffsetDateTime::parse(s, &time::format_description::well_known::Rfc2822)
+                    OffsetDateTime::parse(s, &time::format_description::well_known::Rfc2822)
                 })
                 .or_else(|_| {
                     // Try RFC 3339 format
-                    time::OffsetDateTime::parse(s, &time::format_description::well_known::Rfc3339)
+                    OffsetDateTime::parse(s, &time::format_description::well_known::Rfc3339)
                 })
                 .ok()
         });
@@ -775,7 +829,7 @@ mod tests {
     #[tokio::test]
     async fn versions_listing_applies_version_marker_only_to_first_entry() {
         let metadata = rustfs_filemeta::test_data::create_real_xlmeta().expect("test metadata should be valid");
-        let entries = rustfs_filemeta::MetaCacheEntriesSorted {
+        let entries = MetaCacheEntriesSorted {
             o: rustfs_filemeta::MetaCacheEntries(vec![
                 Some(rustfs_filemeta::MetaCacheEntry {
                     name: "obj-a".to_owned(),
@@ -878,7 +932,7 @@ mod tests {
 
     #[test]
     fn from_file_info_preserves_replication_decision() {
-        let fi = rustfs_filemeta::FileInfo {
+        let fi = FileInfo {
             replication_state_internal: Some(ReplicationState {
                 replicate_decision_str: "arn=true;false;arn:replication::1:dest;rule-id".to_string(),
                 ..Default::default()
@@ -904,11 +958,11 @@ mod tests {
             actual_size: 0,
             user_defined: Arc::new(user_defined),
             parts: Arc::new(vec![
-                rustfs_filemeta::ObjectPartInfo {
+                ObjectPartInfo {
                     actual_size: 4,
                     ..Default::default()
                 },
-                rustfs_filemeta::ObjectPartInfo {
+                ObjectPartInfo {
                     actual_size: 5,
                     ..Default::default()
                 },
@@ -1016,13 +1070,13 @@ mod tests {
             user_defined: Arc::new(ud),
             user_tags: Arc::new("env=prod&team=storage".to_string()),
             parts: Arc::new(vec![
-                rustfs_filemeta::ObjectPartInfo {
+                ObjectPartInfo {
                     number: 1,
                     size: 1024,
                     actual_size: 1024,
                     ..Default::default()
                 },
-                rustfs_filemeta::ObjectPartInfo {
+                ObjectPartInfo {
                     number: 2,
                     size: 512,
                     actual_size: 512,