perf(put): comprehensive PUT performance optimization (#3514)

* perf(put): add eager path metrics and isolation tooling * fix(decommission): persist progress adaptively (#3497) Persist decommission progress after either the existing time interval or a migrated-item threshold, and flush progress baselines after bucket and terminal-state saves. Also stabilize the OIDC discovery mock used by the pre-commit gate. * refactor: move bucket operations contract (#3507) * fix(s3): handle multipart flexible checksums (#3508) * fix(io-core): avoid blocking on pooled buffer return * perf(put): add slow inflight diagnostics * perf(put): fix 16KiB regression with threshold and pool bypass - Lower SMALL_EAGER_PUT_MAX_SIZE from 256KB to 8KB so objects >8KiB use the streaming BufReader path (matches baseline behavior) - Add POOL_BYPASE_MAX_SIZE (16KiB) to bypass BytesPool for very small objects, avoiding Small-tier Mutex contention under high concurrency - Add read_small_put_body_exact_direct() for direct Vec<u8> allocation - Fix stale test assertions to match new 8KB threshold Root cause analysis: the 16KiB regression was primarily caused by instrumentation overhead in set_disk.rs (4x Instant::now() + metrics per PUT), not BytesPool contention. Lowering the threshold eliminates the eager-path overhead for 16KiB+ objects. * perf(put): gate stage metrics behind observability flag Add put_stage_metrics_enabled() AtomicBool switch in io-metrics crate. When disabled (default), record_put_object_path() and record_put_object_stage_duration() are no-ops, avoiding unnecessary histogram/counter macro overhead in the PUT hot path. The flag is set to true during startup when OTEL metric export is enabled (rustfs_obs::observability_metric_enabled() == true). This eliminates the per-request metrics overhead that contributed to the 16KiB PUT regression when metrics collection is not active. * perf(put): comprehensive optimization - restore eager path, cache env, remove UUID Change 1: Restore SMALL_EAGER_PUT_MAX_SIZE from 8KB to 1MB - The try_lock() fix (d13a189e3) eliminates the blocking that caused service health timeouts under 512KiB c64 load - Eager path with BytesPool is now safe for objects up to 1MB - Recovers the eager path benefit for 32KiB-256KiB objects Change 2: Adjust POOL_BYPASE_MAX_SIZE from 16KB to 4KB - With eager path restored to 1MB, objects 4KB-1MB benefit from pool reuse - Only ≤4KB objects bypass the pool (allocation cost negligible) Change 3: Cache RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES via OnceLock - Eliminates per-encode std::env::var() syscall - Env var still works (read once at first use) Change 4: Replace Uuid::new_v4() with Uuid::nil() in Erasure construction - _id field is unused in hot paths (documented in code) - Eliminates CSPRNG syscall per PUT request Change 5: Add concurrency-aware buffer sizing to PUT path - Reuses get_concurrency_aware_buffer_size() from GET path - Reduces buffer size under high concurrency (0.4x at >8 concurrent) - Lowers memory pressure for >1MB streaming PUTs * chore: add pyroscope feature flag and clean up imports - Add pyroscope feature flag forwarding to rustfs-obs - Remove unused allow(non_upper_case_globals) in globals.rs - Sort imports and fix Cargo.toml formatting consistency * style: fix import ordering and code formatting - Sort imports alphabetically in globals.rs, encode.rs - Fix indentation in erasure_coding encode/erasure - Clean up HashReader formatting in object_usecase.rs * fix(test): use tokio::test for request_logging_layer tests The tests call tokio::spawn via RequestContextLayer, which requires a Tokio runtime. Changed from #[test] + futures::executor::block_on to #[tokio::test] + .await, and replaced tracing::subscriber::with_default with tracing::subscriber::set_default to support async. * fix(bench): normalize no-space throughput/latency parsing in to_bps/to_ms When a benchmark tool prints throughput without a separator (e.g. 123MiB/s), awk '{print $2}' returns empty because the whole string is one field, causing to_bps to return N/A and losing valid measurements in CSV output. Insert a space between number and unit via sed before awk field splitting. Same fix applied to to_ms for latency values like '50ms'. Also add TODO comment on PUT path noting that get_concurrency_aware_buffer_size reads ACTIVE_GET_REQUESTS instead of PUT concurrency (PR #3514 review). Refs: PR #3514 review comments by chatgpt-codex-connector * fix(metrics): correct POOL_BYPASS comments and separate PUT vs generic stage metrics - Fix 3 comment-code mismatches: POOL_BYPASS_MAX_SIZE is 4KiB, not 16KiB - Add generic record_stage_duration() with separate histogram (rustfs_internal_stage_duration_ms) for non-PUT paths - Replace record_put_object_stage_duration with record_stage_duration in metacache_set, store_list_objects, and bucket_lifecycle_ops to avoid polluting PUT-specific dashboards with listing/lifecycle timings - Fix flaky test: serialize tests mutating PUT_STAGE_METRICS_ENABLED with METRICS_FLAG_LOCK mutex and explicitly set desired state at test start Refs: PR #3514 review comments by chatgpt-codex-connector * style: apply cargo fmt to metacache_set.rs --------- Co-authored-by: cxymds <cxymds@gmail.com> Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-08-05 12:57:42 +00:00 · 2026-06-17 21:19:11 +08:00
parent a58692f550
commit 8d24d9133b
16 changed files with 769 additions and 135 deletions
@@ -55,6 +55,7 @@ tracing-chunk-debug = []  # Enable per-chunk tracing in data plane (high noise,
 full = ["metrics-gpu", "ftps", "swift", "webdav", "sftp"]
 manual-test-runners = []
 rio-v2 = ["rustfs-ecstore/rio-v2"]
+pyroscope = ["rustfs-obs/pyroscope"]

 [lints]
 workspace = true
@@ -79,7 +80,7 @@ rustfs-obs = { workspace = true }
 rustfs-policy = { workspace = true }
 rustfs-protocols = { workspace = true }
 rustfs-protos = { workspace = true }
-rustfs-rio.workspace = true
+rustfs-rio = { workspace = true }
 rustfs-s3-types = { workspace = true }
 rustfs-s3-ops = { workspace = true }
 rustfs-security-governance = { workspace = true }
@@ -86,6 +86,7 @@ use rustfs_filemeta::{
    ReplicationType, RestoreStatusOps, VersionPurgeStatusType, parse_restore_obj_status, replication_statuses_map,
    version_purge_statuses_map,
 };
+use rustfs_io_core::{BytesPool, PooledBuffer};
 use rustfs_io_metrics;
 use rustfs_lock::NamespaceLockGuard;
 use rustfs_notify::EventArgsBuilder;
@@ -118,6 +119,8 @@ use s3s::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error};
 use std::collections::HashMap;
 use std::ops::Add;
 use std::path::{Component, Path};
+use std::pin::Pin;
+use std::task::{Context, Poll};

 use std::str::FromStr;
 use std::sync::atomic::{AtomicBool, Ordering};
@@ -133,6 +136,11 @@ use uuid::Uuid;

 const ACCEPT_RANGES_BYTES: &str = "bytes";
 const MAX_GET_OBJECT_MEMORY_BUFFER_BYTES: i64 = 64 * 1024 * 1024;
+const LOG_COMPONENT_APP: &str = "app";
+const LOG_SUBSYSTEM_OBJECT: &str = "object";
+const EVENT_PUT_OBJECT_STORE_INFLIGHT_SLOW: &str = "put_object_store_inflight_slow";
+const EVENT_PUT_OBJECT_STORE_RETURNED: &str = "put_object_store_returned";
+const PUT_OBJECT_STORE_WARN_THRESHOLD: Duration = Duration::from_secs(5);
 static GET_OBJECT_BUFFER_THRESHOLD_WARNED: AtomicBool = AtomicBool::new(false);

 fn decoded_content_length_from_headers(headers: &HeaderMap) -> S3Result<Option<i64>> {
@@ -374,6 +382,33 @@ impl<R: AsyncRead> AsyncRead for ExtractArchiveEtagReader<R> {
    }
 }

+struct PooledBufferReader {
+    buffer: PooledBuffer,
+    len: usize,
+    pos: usize,
+}
+
+impl PooledBufferReader {
+    fn new(buffer: PooledBuffer, len: usize) -> Self {
+        Self { buffer, len, pos: 0 }
+    }
+}
+
+impl AsyncRead for PooledBufferReader {
+    fn poll_read(mut self: Pin<&mut Self>, _cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<std::io::Result<()>> {
+        if self.pos >= self.len {
+            return Poll::Ready(Ok(()));
+        }
+
+        let remaining = self.len - self.pos;
+        let to_read = remaining.min(buf.remaining());
+        buf.put_slice(&self.buffer[self.pos..self.pos + to_read]);
+        self.pos += to_read;
+
+        Poll::Ready(Ok(()))
+    }
+}
+
 /// Determine if zero-copy write should be used for this PutObject operation.
 ///
 /// Zero-copy is beneficial for large objects without encryption or compression.
@@ -429,6 +464,105 @@ fn should_use_zero_copy(size: i64, headers: &HeaderMap) -> bool {
    true
 }

+fn has_put_sse_request_headers(headers: &HeaderMap) -> bool {
+    headers.get(AMZ_SERVER_SIDE_ENCRYPTION).is_some()
+        || headers.get(AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM).is_some()
+        || headers.get(AMZ_SERVER_SIDE_ENCRYPTION_KMS_ID).is_some()
+}
+
+fn should_use_small_eager_put_path(
+    size: i64,
+    headers: &HeaderMap,
+    server_side_encryption_requested: bool,
+    should_compress: bool,
+    is_extract: bool,
+) -> bool {
+    const SMALL_EAGER_PUT_MAX_SIZE: i64 = 1024 * 1024;
+
+    if is_extract || should_compress || server_side_encryption_requested {
+        return false;
+    }
+
+    if size <= 0 || size > SMALL_EAGER_PUT_MAX_SIZE {
+        return false;
+    }
+
+    if has_put_sse_request_headers(headers) {
+        return false;
+    }
+
+    if request_uses_aws_chunked(headers) && decoded_content_length_from_headers(headers).ok().flatten().is_none() {
+        return false;
+    }
+
+    true
+}
+
+/// Objects at or below this size bypass BytesPool and use direct allocation.
+/// This avoids Small-tier Mutex contention under high concurrency for tiny objects
+/// where the allocation cost is negligible (≤4KiB memcpy).
+const POOL_BYPASS_MAX_SIZE: usize = 4 * 1024;
+
+async fn read_small_put_body_exact_pooled<R>(mut body: R, size: usize, pool: &BytesPool) -> S3Result<PooledBuffer>
+where
+    R: AsyncRead + Unpin,
+{
+    let mut buf = pool.acquire_buffer(size).await;
+    buf.resize(size, 0);
+    let mut filled = 0;
+
+    while filled < size {
+        let read = tokio::io::AsyncReadExt::read(&mut body, &mut buf[filled..size])
+            .await
+            .map_err(|err| ApiError::from(StorageError::other(err.to_string())))?;
+        if read == 0 {
+            return Err(s3_error!(IncompleteBody));
+        }
+        filled += read;
+    }
+
+    let mut extra = [0u8; 1];
+    let extra_read = tokio::io::AsyncReadExt::read(&mut body, &mut extra)
+        .await
+        .map_err(|err| ApiError::from(StorageError::other(err.to_string())))?;
+    if extra_read != 0 {
+        return Err(s3_error!(UnexpectedContent));
+    }
+
+    Ok(buf)
+}
+
+/// Read small PUT body into a directly-allocated buffer, bypassing BytesPool.
+/// Used for objects ≤4KiB where pool contention under high concurrency
+/// outweighs the allocation cost.
+async fn read_small_put_body_exact_direct<R>(mut body: R, size: usize) -> S3Result<std::io::Cursor<Vec<u8>>>
+where
+    R: AsyncRead + Unpin,
+{
+    let mut buf = vec![0u8; size];
+    let mut filled = 0;
+
+    while filled < size {
+        let read = tokio::io::AsyncReadExt::read(&mut body, &mut buf[filled..size])
+            .await
+            .map_err(|err| ApiError::from(StorageError::other(err.to_string())))?;
+        if read == 0 {
+            return Err(s3_error!(IncompleteBody));
+        }
+        filled += read;
+    }
+
+    let mut extra = [0u8; 1];
+    let extra_read = tokio::io::AsyncReadExt::read(&mut body, &mut extra)
+        .await
+        .map_err(|err| ApiError::from(StorageError::other(err.to_string())))?;
+    if extra_read != 0 {
+        return Err(s3_error!(UnexpectedContent));
+    }
+
+    Ok(std::io::Cursor::new(buf))
+}
+
 fn object_seek_support_threshold() -> usize {
    static OBJECT_SEEK_SUPPORT_THRESHOLD: OnceLock<usize> = OnceLock::new();
    *OBJECT_SEEK_SUPPORT_THRESHOLD.get_or_init(|| {
@@ -1814,10 +1948,21 @@ impl DefaultObjectUsecase {
            return Err(s3_error!(UnexpectedContent));
        }

+        let ingress_stage_start = std::time::Instant::now();
+        let should_compress = is_disk_compressible(&req.headers, &key) && size > MIN_DISK_COMPRESSIBLE_SIZE as i64;
+        let server_side_encryption_requested =
+            server_side_encryption.is_some() || sse_customer_algorithm.is_some() || ssekms_key_id.is_some();
+
        // Apply adaptive buffer sizing based on file size for optimal streaming performance.
        // Uses workload profile configuration (enabled by default) to select appropriate buffer size.
        // Buffer sizes range from 32KB to 4MB depending on file size and configured workload profile.
-        let buffer_size = get_buffer_size_opt_in(size);
+        // Concurrency-aware adjustment reduces buffer size under high concurrency to lower memory pressure.
+        // TODO: get_concurrency_aware_buffer_size reads ACTIVE_GET_REQUESTS (GET concurrency tracker),
+        // not PUT concurrency. Under pure PUT load the counter stays zero so buffers never shrink;
+        // unrelated GET load can shrink PUT buffers instead. Fix by adding ACTIVE_PUT_REQUESTS +
+        // PutObjectGuard and using PUT concurrency here. See PR #3514 review comment.
+        let base_buffer_size = get_buffer_size_opt_in(size);
+        let buffer_size = get_concurrency_aware_buffer_size(size, base_buffer_size);

        // Detect zero-copy opportunity before encryption/compression decisions
        // Zero-copy is beneficial for large unencrypted, uncompressed objects
@@ -1830,10 +1975,15 @@ impl DefaultObjectUsecase {
            debug!("Zero-copy write enabled for {} byte object (bucket={}, key={})", size, bucket, key);
        }

-        let body = tokio::io::BufReader::with_capacity(
-            buffer_size,
-            StreamReader::new(body.map(|f| f.map_err(|e| std::io::Error::other(e.to_string())))),
-        );
+        let use_small_eager_put_path =
+            should_use_small_eager_put_path(size, &req.headers, server_side_encryption_requested, should_compress, false);
+        let put_path = if should_compress {
+            "stream_compressed"
+        } else if use_small_eager_put_path {
+            "small_eager"
+        } else {
+            "streaming"
+        };

        let store = get_validated_store(&bucket).await?;

@@ -1955,9 +2105,12 @@ impl DefaultObjectUsecase {

        let mut sha256hex = get_content_sha256_with_query(&req.headers, req.uri.query());

-        let should_compress = is_disk_compressible(&req.headers, &key) && size > MIN_DISK_COMPRESSIBLE_SIZE as i64;
        let mut write_plan = WritePlan::new();
        let mut reader = if should_compress {
+            let body = tokio::io::BufReader::with_capacity(
+                buffer_size,
+                StreamReader::new(body.map(|f| f.map_err(|e| std::io::Error::other(e.to_string())))),
+            );
            let algorithm = CompressionAlgorithm::default();
            insert_str(
                &mut metadata,
@@ -1985,7 +2138,35 @@ impl DefaultObjectUsecase {
            write_plan = write_plan.with_compression(algorithm);
            hrd
        } else {
-            HashReader::from_stream(body, size, actual_size, md5hex, sha256hex, false).map_err(ApiError::from)?
+            if use_small_eager_put_path {
+                if (actual_size as usize) <= POOL_BYPASS_MAX_SIZE {
+                    // Bypass BytesPool for very small objects to avoid Small-tier
+                    // Mutex contention under high concurrency. Direct allocation
+                    // for ≤4KiB is negligible cost.
+                    let eager_body = read_small_put_body_exact_direct(
+                        StreamReader::new(body.map(|f| f.map_err(|e| std::io::Error::other(e.to_string())))),
+                        actual_size as usize,
+                    )
+                    .await?;
+                    HashReader::from_stream(eager_body, size, actual_size, md5hex, sha256hex, false).map_err(ApiError::from)?
+                } else {
+                    let pool = get_concurrency_manager().bytes_pool();
+                    let eager_body = read_small_put_body_exact_pooled(
+                        StreamReader::new(body.map(|f| f.map_err(|e| std::io::Error::other(e.to_string())))),
+                        actual_size as usize,
+                        pool.as_ref(),
+                    )
+                    .await?;
+                    let eager_reader = PooledBufferReader::new(eager_body, actual_size as usize);
+                    HashReader::from_stream(eager_reader, size, actual_size, md5hex, sha256hex, false).map_err(ApiError::from)?
+                }
+            } else {
+                let body = tokio::io::BufReader::with_capacity(
+                    buffer_size,
+                    StreamReader::new(body.map(|f| f.map_err(|e| std::io::Error::other(e.to_string())))),
+                );
+                HashReader::from_stream(body, size, actual_size, md5hex, sha256hex, false).map_err(ApiError::from)?
+            }
        };

        if size >= 0 {
@@ -1995,6 +2176,11 @@ impl DefaultObjectUsecase {

            opts.want_checksum = reader.checksum();
        }
+        rustfs_io_metrics::record_put_object_path(put_path);
+        rustfs_io_metrics::record_put_object_stage_duration(
+            "ingress_prepare",
+            ingress_stage_start.elapsed().as_secs_f64() * 1000.0,
+        );

        let mut helper = OperationHelper::new(&req, event_name, S3Operation::PutObject);
        let ssekms_context = extract_ssekms_context_from_headers(&req.headers)?;
@@ -2038,6 +2224,11 @@ impl DefaultObjectUsecase {

        let mt2 = metadata.clone();
        opts.user_defined.extend(metadata);
+        let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
+        let request_id = request_context
+            .as_ref()
+            .map(|ctx| ctx.request_id.clone())
+            .unwrap_or_else(|| request_context::RequestContext::fallback().request_id);

        let repoptions =
            get_must_replicate_options(&mt2, "".to_string(), ReplicationStatusType::Empty, ReplicationType::Object, opts.clone());
@@ -2052,13 +2243,76 @@ impl DefaultObjectUsecase {
            );
        }

+        let store_put_watchdog = tokio_util::sync::CancellationToken::new();
+        spawn_traced({
+            let store_put_watchdog = store_put_watchdog.clone();
+            let request_id = request_id.clone();
+            let bucket = bucket.clone();
+            let key = key.clone();
+            let put_path = put_path.to_string();
+            async move {
+                tokio::select! {
+                    _ = store_put_watchdog.cancelled() => {}
+                    _ = tokio::time::sleep(PUT_OBJECT_STORE_WARN_THRESHOLD) => {
+                        warn!(
+                            target: "rustfs::app::object_usecase",
+                            event = EVENT_PUT_OBJECT_STORE_INFLIGHT_SLOW,
+                            component = LOG_COMPONENT_APP,
+                            subsystem = LOG_SUBSYSTEM_OBJECT,
+                            request_id = %request_id,
+                            bucket = %bucket,
+                            key = %key,
+                            put_path = %put_path,
+                            object_size = actual_size,
+                            threshold_ms = PUT_OBJECT_STORE_WARN_THRESHOLD.as_millis() as u64,
+                            state = "store_put_pending",
+                            "PutObject store write remains in flight"
+                        );
+                    }
+                }
+            }
+        });
+
        let obj_info = match store
            .put_object(&bucket, &key, &mut reader, &opts)
            .await
            .map_err(ApiError::from)
        {
-            Ok(obj_info) => obj_info,
+            Ok(obj_info) => {
+                store_put_watchdog.cancel();
+                debug!(
+                    target: "rustfs::app::object_usecase",
+                    event = EVENT_PUT_OBJECT_STORE_RETURNED,
+                    component = LOG_COMPONENT_APP,
+                    subsystem = LOG_SUBSYSTEM_OBJECT,
+                    request_id = %request_id,
+                    bucket = %bucket,
+                    key = %key,
+                    put_path = put_path,
+                    object_size = actual_size,
+                    duration_ms = start_time.elapsed().as_millis() as u64,
+                    result = "success",
+                    "PutObject store write returned"
+                );
+                obj_info
+            }
            Err(err) => {
+                store_put_watchdog.cancel();
+                warn!(
+                    target: "rustfs::app::object_usecase",
+                    event = EVENT_PUT_OBJECT_STORE_RETURNED,
+                    component = LOG_COMPONENT_APP,
+                    subsystem = LOG_SUBSYSTEM_OBJECT,
+                    request_id = %request_id,
+                    bucket = %bucket,
+                    key = %key,
+                    put_path = put_path,
+                    object_size = actual_size,
+                    duration_ms = start_time.elapsed().as_millis() as u64,
+                    result = "error",
+                    error = %err,
+                    "PutObject store write returned"
+                );
                let result: S3Result<S3Response<PutObjectOutput>> = Err(err.into());
                let _ = helper.complete(&result);
                return result;
@@ -5056,6 +5310,86 @@ mod tests {
        assert!(!should_use_zero_copy(2 * 1024 * 1024, &headers));
    }

+    #[test]
+    fn should_use_small_eager_put_path_allows_up_to_1mb() {
+        let headers = HeaderMap::new();
+
+        assert!(should_use_small_eager_put_path(1024, &headers, false, false, false));
+        assert!(should_use_small_eager_put_path(1024 * 1024, &headers, false, false, false));
+        assert!(!should_use_small_eager_put_path(1024 * 1024 + 1, &headers, false, false, false));
+    }
+
+    #[test]
+    fn should_use_small_eager_put_path_rejects_sse_requests() {
+        let headers = HeaderMap::new();
+
+        assert!(!should_use_small_eager_put_path(1024, &headers, true, false, false));
+    }
+
+    #[test]
+    fn should_use_small_eager_put_path_rejects_compressible_objects() {
+        let headers = HeaderMap::new();
+
+        assert!(!should_use_small_eager_put_path(1024, &headers, false, true, false));
+    }
+
+    #[test]
+    fn should_use_small_eager_put_path_rejects_extract_requests() {
+        let headers = HeaderMap::new();
+
+        assert!(!should_use_small_eager_put_path(1024, &headers, false, false, true));
+    }
+
+    #[test]
+    fn should_use_small_eager_put_path_rejects_large_or_empty_objects() {
+        let headers = HeaderMap::new();
+
+        assert!(!should_use_small_eager_put_path(0, &headers, false, false, false));
+        assert!(!should_use_small_eager_put_path(1024 * 1024 + 1, &headers, false, false, false));
+    }
+
+    #[tokio::test]
+    async fn read_small_put_body_exact_pooled_reads_exact_bytes() {
+        let pool = get_concurrency_manager().bytes_pool();
+        let body = std::io::Cursor::new(b"hello".to_vec());
+
+        let buffer = read_small_put_body_exact_pooled(body, 5, pool.as_ref())
+            .await
+            .expect("pooled exact read should succeed");
+
+        assert_eq!(&buffer[..5], b"hello");
+    }
+
+    #[tokio::test]
+    async fn read_small_put_body_exact_pooled_rejects_short_body() {
+        let pool = get_concurrency_manager().bytes_pool();
+        let body = std::io::Cursor::new(b"hell".to_vec());
+
+        let err = match read_small_put_body_exact_pooled(body, 5, pool.as_ref()).await {
+            Ok(_) => panic!("short pooled body should fail"),
+            Err(err) => err,
+        };
+
+        assert_eq!(err.code(), &S3ErrorCode::IncompleteBody);
+    }
+
+    #[tokio::test]
+    async fn pooled_buffer_reader_keeps_buffer_alive_until_consumed() {
+        use tokio::io::AsyncReadExt;
+
+        let pool = get_concurrency_manager().bytes_pool();
+        let body = std::io::Cursor::new(b"hello".to_vec());
+        let buffer = read_small_put_body_exact_pooled(body, 5, pool.as_ref())
+            .await
+            .expect("pooled exact read should succeed");
+        let mut reader = PooledBufferReader::new(buffer, 5);
+        let mut out = Vec::new();
+
+        reader.read_to_end(&mut out).await.expect("pooled reader should be readable");
+
+        assert_eq!(out, b"hello");
+    }
+
    #[test]
    fn should_use_zero_copy_allows_large_unencrypted_binary_objects() {
        let mut headers = HeaderMap::new();
@@ -275,13 +275,31 @@ pub async fn start_http_server(config: &config::Config, readiness: Arc<GlobalRea

        // Helper to configure socket with optimized parameters
        let configure_socket = |socket: &socket2::Socket| -> Result<()> {
-            socket.set_reuse_address(true)?;
+            if let Err(e) = socket.set_reuse_address(true) {
+                debug!(
+                    event = "socket_option_unavailable",
+                    component = LOG_COMPONENT_SERVER,
+                    subsystem = LOG_SUBSYSTEM_STARTUP,
+                    option = "SO_REUSEADDR",
+                    error = %e,
+                    "Socket option is unavailable"
+                );
+            }

            // Set the socket to non-blocking before passing it to Tokio.
            socket.set_nonblocking(true)?;

            // 1. Disable Nagle algorithm: Critical for 4KB Payload, achieving ultra-low latency
-            socket.set_tcp_nodelay(true)?;
+            if let Err(e) = socket.set_tcp_nodelay(true) {
+                debug!(
+                    event = "socket_option_unavailable",
+                    component = LOG_COMPONENT_SERVER,
+                    subsystem = LOG_SUBSYSTEM_STARTUP,
+                    option = "TCP_NODELAY",
+                    error = %e,
+                    "Socket option is unavailable"
+                );
+            }

            // 2. Enable SO_REUSEPORT for better multi-core scalability on supported platforms
            #[cfg(all(unix, not(target_os = "solaris"), not(target_os = "illumos")))]
@@ -297,11 +315,40 @@ pub async fn start_http_server(config: &config::Config, readiness: Arc<GlobalRea
            }

            // 3. Set system-level TCP KeepAlive to protect long connections
-            socket.set_tcp_keepalive(&keepalive)?;
+            if let Err(e) = socket.set_tcp_keepalive(&keepalive) {
+                debug!(
+                    event = "socket_option_unavailable",
+                    component = LOG_COMPONENT_SERVER,
+                    subsystem = LOG_SUBSYSTEM_STARTUP,
+                    option = "TCP_KEEPALIVE",
+                    error = %e,
+                    "Socket option is unavailable"
+                );
+            }

-            // 4. Increase receive/send buffer to support BDP at GB-level throughput
-            socket.set_recv_buffer_size(4 * rustfs_config::MI_B)?;
-            socket.set_send_buffer_size(4 * rustfs_config::MI_B)?;
+            // 4. Increase receive/send buffer to support BDP at GB-level throughput.
+            // Some constrained local environments reject these socket options with
+            // EPERM/ENOPROTOOPT-style failures; log and continue in that case.
+            if let Err(e) = socket.set_recv_buffer_size(4 * rustfs_config::MI_B) {
+                debug!(
+                    event = "socket_option_unavailable",
+                    component = LOG_COMPONENT_SERVER,
+                    subsystem = LOG_SUBSYSTEM_STARTUP,
+                    option = "SO_RCVBUF",
+                    error = %e,
+                    "Socket option is unavailable"
+                );
+            }
+            if let Err(e) = socket.set_send_buffer_size(4 * rustfs_config::MI_B) {
+                debug!(
+                    event = "socket_option_unavailable",
+                    component = LOG_COMPONENT_SERVER,
+                    subsystem = LOG_SUBSYSTEM_STARTUP,
+                    option = "SO_SNDBUF",
+                    error = %e,
+                    "Socket option is unavailable"
+                );
+            }

            Ok(())
        };
@@ -24,7 +24,9 @@ use crate::server::{
    has_path_prefix, is_admin_path, is_table_catalog_path,
 };
 use crate::storage::apply_cors_headers;
-use crate::storage::request_context::{RequestContext, extract_request_id_from_headers, extract_trace_context_ids_from_headers};
+use crate::storage::request_context::{
+    RequestContext, extract_request_id_from_headers, extract_trace_context_ids_from_headers, spawn_traced,
+};
 use bytes::Bytes;
 use http::{HeaderMap, HeaderValue, Method, Request as HttpRequest, Response, StatusCode, Uri};
 use http_body::Body;
@@ -38,17 +40,20 @@ use std::future::Future;
 use std::pin::Pin;
 use std::sync::Arc;
 use std::task::{Context, Poll};
-use std::time::Instant;
+use std::time::{Duration, Instant};
+use tokio_util::sync::CancellationToken;
 use tower::{Layer, Service};
-use tracing::{debug, error, info};
+use tracing::{debug, error, info, warn};
 use url::form_urlencoded;

 const HTTP_REQUEST_COMPLETED_EVENT: &str = "http_request_completed";
 const HTTP_REQUEST_FAILED_EVENT: &str = "http_request_failed";
+const HTTP_REQUEST_INFLIGHT_SLOW_EVENT: &str = "http_request_inflight_slow";
 const LOG_COMPONENT_SERVER: &str = "server";
 const LOG_SUBSYSTEM_HTTP: &str = "http";
 const REDACTED_QUERY_VALUE: &str = "redacted";
 const OBJECT_ZIP_DOWNLOADS_PATH: &str = "/v3/object-zip-downloads/";
+const HTTP_REQUEST_INFLIGHT_WARN_THRESHOLD: Duration = Duration::from_secs(5);

 pub(crate) fn redact_sensitive_uri_query(uri: &http::Uri) -> String {
    let path = uri.path();
@@ -322,9 +327,39 @@ where
    fn call(&mut self, req: HttpRequest<B>) -> Self::Future {
        let context = RequestLogContext::from_request(&req);
        let mut inner = self.inner.clone();
+        let watchdog = CancellationToken::new();
+        let watchdog_context = context.clone();
+
+        spawn_traced({
+            let watchdog = watchdog.clone();
+            async move {
+                tokio::select! {
+                    _ = watchdog.cancelled() => {}
+                    _ = tokio::time::sleep(HTTP_REQUEST_INFLIGHT_WARN_THRESHOLD) => {
+                        warn!(
+                            event = HTTP_REQUEST_INFLIGHT_SLOW_EVENT,
+                            component = LOG_COMPONENT_SERVER,
+                            subsystem = LOG_SUBSYSTEM_HTTP,
+                            request_id = %watchdog_context.request_id,
+                            trace_id = %watchdog_context.trace_id.as_deref().unwrap_or("unknown"),
+                            span_id = %watchdog_context.span_id.as_deref().unwrap_or("unknown"),
+                            peer_addr = %watchdog_context.peer_addr,
+                            method = %watchdog_context.method,
+                            uri = %watchdog_context.uri,
+                            duration_ms = watchdog_context.duration_ms(),
+                            active_requests = active_http_requests(),
+                            threshold_ms = HTTP_REQUEST_INFLIGHT_WARN_THRESHOLD.as_millis() as u64,
+                            state = "response_pending",
+                            "HTTP request remains in flight"
+                        );
+                    }
+                }
+            }
+        });

        Box::pin(async move {
            let result = inner.call(req).await;
+            watchdog.cancel();
            match &result {
                Ok(response) => context.log_response(response),
                Err(error) => context.log_failure(error),
@@ -3054,8 +3089,8 @@ mod tests {
        assert_eq!(redact_sensitive_uri_query(&uri), "/rustfs/admin/v3/users?token=not-a-download-token");
    }

-    #[test]
-    fn request_logging_layer_emits_single_completion_event_with_standard_fields() {
+    #[tokio::test]
+    async fn request_logging_layer_emits_single_completion_event_with_standard_fields() {
        let writer = SharedWriter::default();
        let captured = writer.buffer.clone();
        let subscriber = Registry::default().with(
@@ -3067,25 +3102,25 @@ mod tests {
                .with_writer(writer),
        );

-        tracing::subscriber::with_default(subscriber, || {
-            let mut service = tower::ServiceBuilder::new()
-                .layer(RequestContextLayer)
-                .layer(RequestLoggingLayer)
-                .service(StatusService::new(StatusCode::OK));
+        let _guard = tracing::subscriber::set_default(subscriber);

-            let mut request: Request<Full<Bytes>> = Request::builder()
-                .method(Method::GET)
-                .uri("/bucket/object.txt")
-                .header("x-request-id", "req-123")
-                .body(Full::from(Bytes::new()))
-                .expect("request");
-            request
-                .extensions_mut()
-                .insert(RemoteAddr("127.0.0.1:9000".parse().expect("socket addr")));
+        let mut service = tower::ServiceBuilder::new()
+            .layer(RequestContextLayer)
+            .layer(RequestLoggingLayer)
+            .service(StatusService::new(StatusCode::OK));

-            let response = futures::executor::block_on(service.call(request)).expect("response");
-            assert_eq!(response.status(), StatusCode::OK);
-        });
+        let mut request: Request<Full<Bytes>> = Request::builder()
+            .method(Method::GET)
+            .uri("/bucket/object.txt")
+            .header("x-request-id", "req-123")
+            .body(Full::from(Bytes::new()))
+            .expect("request");
+        request
+            .extensions_mut()
+            .insert(RemoteAddr("127.0.0.1:9000".parse().expect("socket addr")));
+
+        let response = service.call(request).await.expect("response");
+        assert_eq!(response.status(), StatusCode::OK);

        let output = String::from_utf8(captured.lock().expect("captured logs").clone()).expect("utf8 logs");
        assert_eq!(output.matches("HTTP request completed").count(), 1, "{output}");
@@ -3110,8 +3145,8 @@ mod tests {
        assert!(output.contains("duration_ms"), "{output}");
    }

-    #[test]
-    fn request_logging_layer_uses_request_context_trace_fields() {
+    #[tokio::test]
+    async fn request_logging_layer_uses_request_context_trace_fields() {
        let writer = SharedWriter::default();
        let captured = writer.buffer.clone();
        let subscriber = Registry::default().with(
@@ -3123,28 +3158,28 @@ mod tests {
                .with_writer(writer),
        );

-        tracing::subscriber::with_default(subscriber, || {
-            let mut service = RequestLoggingLayer.layer(StatusService::new(StatusCode::INTERNAL_SERVER_ERROR));
+        let _guard = tracing::subscriber::set_default(subscriber);

-            let mut request = Request::builder()
-                .method(Method::GET)
-                .uri("/bucket/object.txt")
-                .body(())
-                .expect("request");
-            request.extensions_mut().insert(RequestContext {
-                request_id: "req-ctx".to_string(),
-                x_amz_request_id: "amz-ctx".to_string(),
-                trace_id: Some("trace-ctx".to_string()),
-                span_id: Some("span-ctx".to_string()),
-                start_time: Instant::now(),
-            });
-            request
-                .extensions_mut()
-                .insert(RemoteAddr("127.0.0.1:9000".parse().expect("socket addr")));
+        let mut service = RequestLoggingLayer.layer(StatusService::new(StatusCode::INTERNAL_SERVER_ERROR));

-            let response = futures::executor::block_on(service.call(request)).expect("response");
-            assert_eq!(response.status(), StatusCode::INTERNAL_SERVER_ERROR);
+        let mut request = Request::builder()
+            .method(Method::GET)
+            .uri("/bucket/object.txt")
+            .body(())
+            .expect("request");
+        request.extensions_mut().insert(RequestContext {
+            request_id: "req-ctx".to_string(),
+            x_amz_request_id: "amz-ctx".to_string(),
+            trace_id: Some("trace-ctx".to_string()),
+            span_id: Some("span-ctx".to_string()),
+            start_time: Instant::now(),
        });
+        request
+            .extensions_mut()
+            .insert(RemoteAddr("127.0.0.1:9000".parse().expect("socket addr")));
+
+        let response = service.call(request).await.expect("response");
+        assert_eq!(response.status(), StatusCode::INTERNAL_SERVER_ERROR);

        let output = String::from_utf8(captured.lock().expect("captured logs").clone()).expect("utf8 logs");
        assert!(output.contains("http_request_completed"), "{output}");
@@ -355,6 +355,7 @@ async fn init_observability_runtime(ctx: CancellationToken) {
    crate::allocator_reclaim::init_allocator_reclaim(ctx.clone());

    if rustfs_obs::observability_metric_enabled() {
+        rustfs_io_metrics::set_put_stage_metrics_enabled(true);
        init_metrics_runtime(ctx.clone());
        crate::memory_observability::init_memory_observability(ctx.clone());
        init_auto_tuner(ctx).await;