feat(get): Small-file GET performance optimization for 1KiB-1MiB objects (#4016)

* feat(get): SF01 - bucket validation cache

Add 5s TTL cache for bucket validation to avoid repeated stat_volume()
calls on every GET request.

Changes:
- Add BUCKET_VALIDATED_CACHE (OnceLock + RwLock + HashMap)
- Add invalidate_bucket_validation_cache() for cache invalidation
- Add invalidate_all_bucket_validation_cache() for bulk invalidation
- Update get_validated_store() to use cache
- Add cache invalidation in execute_delete_bucket()

Expected impact: 3-5x improvement for small file GET latency.

Closes rustfs/backlog#766

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(get): SF03 - metadata cache TTL increase

Increase metadata cache TTL from 250ms to 2s and capacity from 1024
to 4096 entries.

Changes:
- GET_OBJECT_METADATA_CACHE_TTL: 250ms -> 2s
- GET_OBJECT_METADATA_CACHE_MAX_ENTRIES: 1024 -> 4096

All mutation paths already call invalidate_get_object_metadata_cache,
so the longer TTL is safe.

Expected impact: 10-50x improvement for hot objects.

Closes rustfs/backlog#768

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(get): SF04 - remove unnecessary tokio::spawn in metadata fanout

Replace tokio::spawn with direct async future in read_all_fileinfo_full_wait.
join_all already provides concurrency, so tokio::spawn adds unnecessary
task creation and scheduling overhead.

Changes:
- Remove tokio::spawn from metadata fanout futures
- Update result handling for direct future results

Expected impact: 16-32us reduction per GET request.

Closes rustfs/backlog#769

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(get): SF06 - conditional lifecycle check

Only call resolve_put_object_expiration when the object has an
x-amz-expiration metadata marker. This avoids unnecessary lifecycle
configuration reads on every GET request.

Expected impact: 50-100us reduction per GET request.

Closes rustfs/backlog#771

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(get): SF07 - conditional metrics recording

Gate hot path metrics behind get_stage_metrics_enabled() to reduce
overhead when metrics are not needed.

Changes:
- Conditional record_zero_copy_read
- Conditional manager.record_disk_operation
- Conditional manager.record_access
- Conditional manager.record_transfer

Expected impact: 20-50us reduction per GET request.

Closes rustfs/backlog#772

Co-Authored-By: heihutu <heihutu@gmail.com>

* refactor(get): SF01 - use moka instead of dashmap for bucket cache

Replace OnceLock + RwLock + HashMap with moka::sync::Cache for bucket
validation cache. moka provides built-in TTL support and is already
available in the workspace.

Changes:
- Add moka dependency to rustfs crate
- Replace manual TTL management with moka's time_to_live
- Simplify cache operations

Closes rustfs/backlog#766

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(get): SF02 - inline data fast path

Add fast path for small inline objects that bypasses duplex pipe,
tokio::spawn, and bitrot reader creation when data is already in memory.

Changes:
- Add inline data detection before codec streaming gate
- Direct in-memory erasure decode for inline objects <= 128KB
- Add GET_OBJECT_PATH_INLINE_DIRECT metric path
- Skip duplex pipe and background task for inline data

Conditions for fast path:
- Single part object
- Inline data available
- Size <= 128KB
- Not encrypted/compressed/remote
- No range request

Expected impact: 2-3x improvement for small file GET latency.

Closes rustfs/backlog#767

Co-Authored-By: heihutu <heihutu@gmail.com>

* refactor: translate Chinese comments to English

Translate all Chinese comments to English in modified files:
- rustfs/src/storage/ecfs_extend.rs
- rustfs/src/app/bucket_usecase.rs

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix

* add

* fmt and improve import

* fmt

* feat(get): SF05 skip IO planning + refactor inline detection + adaptive bucket cache

SF05: Skip disk I/O semaphore for inline data fast path
- Reorder prepare_get_object_read_execution: read first, then decide semaphore
- Inline objects skip acquire_disk_read_permit() entirely (saves 100-200us)
- Add is_inline_fast_path field to GetObjectReadSetup

Refactor: Unify inline detection logic
- Add ObjectInfo::is_inline_fast_path_eligible() as single source of truth
- Version-aware thresholds: non-versioned 128KB, versioned 16KB (matches PUT)
- Eliminates divergent conditions between set_disk/mod.rs and object_usecase.rs

Refactor: Restore fault tolerance in metadata fanout
- Restore tokio::spawn + JoinError handling in read_all_fileinfo_full_wait
- Prevents single disk read panic from unwinding the entire operation

Refactor: Restore lifecycle check correctness
- Remove incorrect SF06 conditional that skipped lifecycle for most objects
- Always call resolve_put_object_expiration (original behavior)

Fix: make_bucket cache invalidation
- Invalidate bucket validation cache on create_bucket

Fix: erasure decode written validation
- Check decode() return value; error if 0 bytes written for non-empty object

Adaptive bucket cache
- Default: RwLock<HashMap> for < 100 buckets (low overhead)
- Opt-in: starshard::ShardedHashMap via RUSTFS_BUCKET_CACHE_STARSHARD=1
- 5s TTL with manual timestamp checking

Benchmark results (warp get, concurrency 32, 10s, 3 rounds):
- 10KiB: 25.10 MiB/s (+28.2% vs SF01-07)
- 100KiB: 221.81 MiB/s
- 1MiB: 1972.78 MiB/s
- vs main: -10% to -12% (inline path not triggered by warp)

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(versioning): use read lock for versioning config query + five-expert analysis

P0 fix: BucketVersioningSys::get() was using write lock on
GLOBAL_BucketMetadataSys for a pure read operation. This serialized
all concurrent GET requests (3 write-lock acquisitions per request).

Changed to read lock — get_versioning_config() handles its own
internal locking via metadata_map RwLock.

Five-expert analysis identified top bottlenecks:
1. Versioning write lock (P0, fixed)
2. Inline fast path not triggered (P0, needs verification)
3. Metadata fanout no early-stop (P1, early-stop has bug, reverted)
4. Request-level versioning cache (P1, pending)
5. Duplex pipe for small objects (P2, pending)

Benchmark (read-lock fix, warp concurrency 32):
- 1KiB: 2.29 MiB/s (vs 2.53 before, within variance)
- 10KiB: 25.00 MiB/s (same as before)
- 100KiB: 246.72 MiB/s (+11% vs 221.81)
- 1MiB: 2039.95 MiB/s (+3% vs 1972.78)

Co-Authored-By: heihutu <heihutu@gmail.com>

* chore: remove benchmark results from git, keep locally only

Remove docs/benchmark/*.md from version control.
Files remain on disk but are no longer tracked by git.
Added docs/benchmark/*.md to .gitignore.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(get): decode inline fast path through bitrot readers

---------

Co-authored-by: heihutu <heihutu@gmail.com>

crates/ecstore/src/object_api/readers.rs

@@ -1,3 +1,17 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
 use super::*;
 #[cfg(feature = "rio-v2")]
 use aes_gcm::aead::Payload;
@@ -122,7 +136,7 @@ fn restore_request_active(opts: &ObjectOptions) -> bool {
     restore.type_.is_some() || restore.days.is_some() || restore.output_location.is_some() || restore.select_parameters.is_some()
 }
 
-fn decode_compression_index(index: Option<&bytes::Bytes>) -> Option<Index> {
+fn decode_compression_index(index: Option<&Bytes>) -> Option<Index> {
     crate::io_support::rio::decode_compression_index_bytes(index?)
 }
 
@@ -833,11 +847,7 @@ impl<R: AsyncRead + Unpin + Send + Sync + 'static> RangedDecompressReader<R> {
 }
 
 impl<R: AsyncRead + Unpin + Send + Sync + 'static> AsyncRead for RangedDecompressReader<R> {
-    fn poll_read(
-        mut self: std::pin::Pin<&mut Self>,
-        cx: &mut std::task::Context<'_>,
-        buf: &mut tokio::io::ReadBuf<'_>,
-    ) -> std::task::Poll<std::io::Result<()>> {
+    fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<std::io::Result<()>> {
         use std::pin::Pin;
         use std::task::Poll;
         use tokio::io::ReadBuf;
@@ -985,11 +995,7 @@ impl<R: AsyncRead + Unpin + Send + 'static> StreamConsumer<R> {
 }
 
 impl<R: AsyncRead + Unpin + Send + 'static> AsyncRead for StreamConsumer<R> {
-    fn poll_read(
-        mut self: std::pin::Pin<&mut Self>,
-        cx: &mut std::task::Context<'_>,
-        buf: &mut tokio::io::ReadBuf<'_>,
-    ) -> std::task::Poll<std::io::Result<()>> {
+    fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<std::io::Result<()>> {
         use std::pin::Pin;
         use std::task::Poll;
 
@@ -1032,7 +1038,7 @@ fn encrypted_plaintext_size(oi: &ObjectInfo, is_multipart: bool, is_compressed:
     oi.decrypted_size().map_err(Into::into)
 }
 
-fn is_multipart_encrypted_object(parts: &[rustfs_filemeta::ObjectPartInfo], etag: Option<&str>) -> bool {
+fn is_multipart_encrypted_object(parts: &[ObjectPartInfo], etag: Option<&str>) -> bool {
     if parts.len() > 1 {
         return true;
     }
@@ -1040,13 +1046,13 @@ fn is_multipart_encrypted_object(parts: &[rustfs_filemeta::ObjectPartInfo], etag
     etag.map(|etag| etag.trim_matches('"').len() != 32).unwrap_or(false)
 }
 
-fn multipart_plaintext_size(parts: &[rustfs_filemeta::ObjectPartInfo], fallback: i64) -> i64 {
+fn multipart_plaintext_size(parts: &[ObjectPartInfo], fallback: i64) -> i64 {
     let total: i64 = parts.iter().map(part_plaintext_size).sum();
 
     if total > 0 { total } else { fallback }
 }
 
-fn multipart_part_numbers(parts: &[rustfs_filemeta::ObjectPartInfo]) -> Vec<usize> {
+fn multipart_part_numbers(parts: &[ObjectPartInfo]) -> Vec<usize> {
     parts.iter().map(|part| part.number).collect()
 }
 
@@ -1605,13 +1611,13 @@ mod tests {
 
     fn ssec_headers_from_key(key_bytes: [u8; 32]) -> HeaderMap<HeaderValue> {
         let mut headers = HeaderMap::new();
-        headers.insert(rustfs_utils::http::SSEC_ALGORITHM_HEADER, HeaderValue::from_static("AES256"));
+        headers.insert(SSEC_ALGORITHM_HEADER, HeaderValue::from_static("AES256"));
         headers.insert(
-            rustfs_utils::http::SSEC_KEY_HEADER,
+            SSEC_KEY_HEADER,
             HeaderValue::from_str(&BASE64_STANDARD.encode(key_bytes)).expect("valid base64 header"),
         );
         headers.insert(
-            rustfs_utils::http::SSEC_KEY_MD5_HEADER,
+            SSEC_KEY_MD5_HEADER,
             HeaderValue::from_str(&BASE64_STANDARD.encode(md5_bytes(key_bytes))).expect("valid md5 header"),
         );
         headers
@@ -2616,7 +2622,7 @@ mod tests {
 
     #[tokio::test]
     async fn test_get_object_reader_compressed_range_returns_physical_offset_from_index() {
-        let mut index = crate::io_support::rio::Index::new();
+        let mut index = Index::new();
         index.add(0, 0).unwrap();
         index.add(1_048_576, 2_097_152).unwrap();
 
@@ -2670,7 +2676,7 @@ mod tests {
 
     #[tokio::test]
     async fn test_read_plan_compressed_range_tracks_storage_and_visible_offsets() {
-        let mut index = crate::io_support::rio::Index::new();
+        let mut index = Index::new();
         index.add(0, 0).unwrap();
         index.add(1_048_576, 2_097_152).unwrap();
 
@@ -2746,7 +2752,7 @@ mod tests {
     #[cfg(feature = "rio-v2")]
     #[tokio::test]
     async fn test_read_plan_accepts_minio_headerless_compression_index() {
-        let mut index = crate::io_support::rio::Index::new();
+        let mut index = Index::new();
         index.add(0, 0).unwrap();
         index.add(1_048_576, 2_097_152).unwrap();
         let headerless_index = crate::io_support::rio::compression_index_storage_bytes(&index);
@@ -2800,7 +2806,7 @@ mod tests {
     #[cfg(feature = "rio-v2")]
     #[test]
     fn test_get_compressed_offsets_aligns_encrypted_ranges_to_dare_packages() {
-        let mut index = crate::io_support::rio::Index::new();
+        let mut index = Index::new();
         index.add(0, 0).unwrap();
         index.add(200_000, 2_097_152).unwrap();
         let stored_index = crate::io_support::rio::compression_index_storage_bytes(&index);