perf(put): comprehensive PUT performance optimization (#3514)

* perf(put): add eager path metrics and isolation tooling

* fix(decommission): persist progress adaptively (#3497)

Persist decommission progress after either the existing time interval or a migrated-item threshold, and flush progress baselines after bucket and terminal-state saves.

Also stabilize the OIDC discovery mock used by the pre-commit gate.

* refactor: move bucket operations contract (#3507)

* fix(s3): handle multipart flexible checksums (#3508)

* fix(io-core): avoid blocking on pooled buffer return

* perf(put): add slow inflight diagnostics

* perf(put): fix 16KiB regression with threshold and pool bypass

- Lower SMALL_EAGER_PUT_MAX_SIZE from 256KB to 8KB so objects >8KiB
  use the streaming BufReader path (matches baseline behavior)
- Add POOL_BYPASE_MAX_SIZE (16KiB) to bypass BytesPool for very small
  objects, avoiding Small-tier Mutex contention under high concurrency
- Add read_small_put_body_exact_direct() for direct Vec<u8> allocation
- Fix stale test assertions to match new 8KB threshold

Root cause analysis: the 16KiB regression was primarily caused by
instrumentation overhead in set_disk.rs (4x Instant::now() + metrics
per PUT), not BytesPool contention. Lowering the threshold eliminates
the eager-path overhead for 16KiB+ objects.

* perf(put): gate stage metrics behind observability flag

Add put_stage_metrics_enabled() AtomicBool switch in io-metrics crate.
When disabled (default), record_put_object_path() and
record_put_object_stage_duration() are no-ops, avoiding unnecessary
histogram/counter macro overhead in the PUT hot path.

The flag is set to true during startup when OTEL metric export is
enabled (rustfs_obs::observability_metric_enabled() == true).

This eliminates the per-request metrics overhead that contributed
to the 16KiB PUT regression when metrics collection is not active.

* perf(put): comprehensive optimization - restore eager path, cache env, remove UUID

Change 1: Restore SMALL_EAGER_PUT_MAX_SIZE from 8KB to 1MB
- The try_lock() fix (d13a189e3) eliminates the blocking that caused
  service health timeouts under 512KiB c64 load
- Eager path with BytesPool is now safe for objects up to 1MB
- Recovers the eager path benefit for 32KiB-256KiB objects

Change 2: Adjust POOL_BYPASE_MAX_SIZE from 16KB to 4KB
- With eager path restored to 1MB, objects 4KB-1MB benefit from pool reuse
- Only ≤4KB objects bypass the pool (allocation cost negligible)

Change 3: Cache RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES via OnceLock
- Eliminates per-encode std::env::var() syscall
- Env var still works (read once at first use)

Change 4: Replace Uuid::new_v4() with Uuid::nil() in Erasure construction
- _id field is unused in hot paths (documented in code)
- Eliminates CSPRNG syscall per PUT request

Change 5: Add concurrency-aware buffer sizing to PUT path
- Reuses get_concurrency_aware_buffer_size() from GET path
- Reduces buffer size under high concurrency (0.4x at >8 concurrent)
- Lowers memory pressure for >1MB streaming PUTs

* chore: add pyroscope feature flag and clean up imports

- Add pyroscope feature flag forwarding to rustfs-obs
- Remove unused allow(non_upper_case_globals) in globals.rs
- Sort imports and fix Cargo.toml formatting consistency

* style: fix import ordering and code formatting

- Sort imports alphabetically in globals.rs, encode.rs
- Fix indentation in erasure_coding encode/erasure
- Clean up HashReader formatting in object_usecase.rs

* fix(test): use tokio::test for request_logging_layer tests

The tests call tokio::spawn via RequestContextLayer, which requires a
Tokio runtime. Changed from #[test] + futures::executor::block_on to
#[tokio::test] + .await, and replaced tracing::subscriber::with_default
with tracing::subscriber::set_default to support async.

* fix(bench): normalize no-space throughput/latency parsing in to_bps/to_ms

When a benchmark tool prints throughput without a separator (e.g. 123MiB/s),
awk '{print $2}' returns empty because the whole string is one field,
causing to_bps to return N/A and losing valid measurements in CSV output.

Insert a space between number and unit via sed before awk field splitting.
Same fix applied to to_ms for latency values like '50ms'.

Also add TODO comment on PUT path noting that get_concurrency_aware_buffer_size
reads ACTIVE_GET_REQUESTS instead of PUT concurrency (PR #3514 review).

Refs: PR #3514 review comments by chatgpt-codex-connector

* fix(metrics): correct POOL_BYPASS comments and separate PUT vs generic stage metrics

- Fix 3 comment-code mismatches: POOL_BYPASS_MAX_SIZE is 4KiB, not 16KiB
- Add generic record_stage_duration() with separate histogram
  (rustfs_internal_stage_duration_ms) for non-PUT paths
- Replace record_put_object_stage_duration with record_stage_duration in
  metacache_set, store_list_objects, and bucket_lifecycle_ops to avoid
  polluting PUT-specific dashboards with listing/lifecycle timings
- Fix flaky test: serialize tests mutating PUT_STAGE_METRICS_ENABLED with
  METRICS_FLAG_LOCK mutex and explicitly set desired state at test start

Refs: PR #3514 review comments by chatgpt-codex-connector

* style: apply cargo fmt to metacache_set.rs

---------

Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>

crates/ecstore/src/erasure_coding/encode.rs

@@ -31,6 +31,10 @@ const ENV_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES: &str = "RUSTFS_ERASURE_ENCOD
 const DEFAULT_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES: usize = 32 * 1024 * 1024;
 const DEFAULT_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BLOCKS: usize = 32;
 
+/// Cached value of `RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES` env var.
+/// Read once at first use via `OnceLock` to avoid per-encode syscall.
+static CACHED_MAX_INFLIGHT_BYTES: std::sync::OnceLock<usize> = std::sync::OnceLock::new();
+
 fn encode_channel_capacity(expanded_block_bytes: usize, max_inflight_bytes: usize) -> usize {
     if expanded_block_bytes == 0 {
         return 1;
@@ -273,10 +277,12 @@ impl Erasure {
 
         // Bound queued encoded blocks by memory budget to avoid per-request spikes.
         let expanded_block_bytes = self.shard_size().saturating_mul(self.total_shard_count());
-        let max_inflight_bytes = rustfs_utils::get_env_usize(
-            ENV_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES,
-            DEFAULT_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES,
-        );
+        let max_inflight_bytes = *CACHED_MAX_INFLIGHT_BYTES.get_or_init(|| {
+            rustfs_utils::get_env_usize(
+                ENV_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES,
+                DEFAULT_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES,
+            )
+        });
         let inflight_blocks = encode_channel_capacity(expanded_block_bytes, max_inflight_bytes);
         let (tx, mut rx) = mpsc::channel::<Vec<Bytes>>(inflight_blocks);
 
@@ -443,7 +449,7 @@ mod tests {
         ))];
 
         let erasure = Arc::new(Erasure::new(1, 0, 16));
-        let reader = tokio::io::BufReader::new(std::io::Cursor::new(b"small payload".to_vec()));
+        let reader = tokio::io::BufReader::new(Cursor::new(b"small payload".to_vec()));
         let (_reader, written) = erasure.encode(reader, &mut writers, 1).await.unwrap();
 
         assert_eq!(written, b"small payload".len());
@@ -482,7 +488,7 @@ mod tests {
         ))];
 
         let erasure = Arc::new(Erasure::new(1, 0, 0));
-        let reader = tokio::io::BufReader::new(std::io::Cursor::new(b"payload".to_vec()));
+        let reader = tokio::io::BufReader::new(Cursor::new(b"payload".to_vec()));
         let err = erasure
             .encode(reader, &mut writers, 1)
             .await
@@ -505,7 +511,7 @@ mod tests {
         ))];
 
         let erasure = Arc::new(Erasure::new(1, 0, 16));
-        let reader = tokio::io::BufReader::new(std::io::Cursor::new(Vec::<u8>::new()));
+        let reader = tokio::io::BufReader::new(Cursor::new(Vec::<u8>::new()));
         let (_reader, total) = erasure.encode_inline_small(reader, &mut writers, 1).await.unwrap();
 
         assert_eq!(total, 0);
@@ -537,7 +543,7 @@ mod tests {
 
         let payload = b"hello inline small";
         let erasure = Arc::new(Erasure::new(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE));
-        let reader = tokio::io::BufReader::new(std::io::Cursor::new(payload.to_vec()));
+        let reader = tokio::io::BufReader::new(Cursor::new(payload.to_vec()));
         let (_reader, total) = erasure.encode_inline_small(reader, &mut writers, DATA_SHARDS).await.unwrap();
 
         assert_eq!(total, payload.len());
@@ -569,7 +575,7 @@ mod tests {
 
         let payload = b"hello single block";
         let erasure = Arc::new(Erasure::new(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE));
-        let reader = tokio::io::BufReader::new(std::io::Cursor::new(payload.to_vec()));
+        let reader = tokio::io::BufReader::new(Cursor::new(payload.to_vec()));
         let (_reader, total) = erasure
             .encode_single_block_non_inline(reader, &mut writers, DATA_SHARDS)
             .await
@@ -603,7 +609,7 @@ mod tests {
 
         let payload = vec![1u8; BLOCK_SIZE + 1];
         let erasure = Arc::new(Erasure::new(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE));
-        let reader = tokio::io::BufReader::new(std::io::Cursor::new(payload));
+        let reader = tokio::io::BufReader::new(Cursor::new(payload));
         let err = erasure
             .encode_single_block_non_inline(reader, &mut writers, DATA_SHARDS)
             .await

crates/ecstore/src/erasure_coding/erasure.rs

@@ -394,7 +394,7 @@ impl Erasure {
             encoder,
             legacy_encoder,
             uses_legacy,
-            _id: Uuid::new_v4(),
+            _id: Uuid::nil(), // Unused in hot paths; avoid CSPRNG syscall
         }
     }
 
@@ -631,15 +631,11 @@ impl Erasure {
     ) -> Result<usize, E>
     where
         R: AsyncRead + Send + Sync + Unpin,
-        F: FnMut(std::io::Result<Vec<Bytes>>) -> Fut + Send,
-        Fut: std::future::Future<Output = Result<(), E>> + Send,
+        F: FnMut(io::Result<Vec<Bytes>>) -> Fut + Send,
+        Fut: Future<Output = Result<(), E>> + Send,
     {
         if self.block_size == 0 {
-            on_block(Err(std::io::Error::new(
-                std::io::ErrorKind::InvalidInput,
-                "erasure block_size must be non-zero",
-            )))
-            .await?;
+            on_block(Err(io::Error::new(io::ErrorKind::InvalidInput, "erasure block_size must be non-zero"))).await?;
             return Ok(0);
         }
 
@@ -662,7 +658,7 @@ impl Erasure {
                     {
                         Ok(result) => result,
                         Err(err) => {
-                            on_block(Err(std::io::Error::other(format!("EC encode task failed: {err}")))).await?;
+                            on_block(Err(io::Error::other(format!("EC encode task failed: {err}")))).await?;
                             break;
                         }
                     };
@@ -673,7 +669,7 @@ impl Erasure {
                     warn!("encode_stream_callback_async read unexpected ok");
                     break;
                 }
-                Err(e) if e.kind() == std::io::ErrorKind::UnexpectedEof => {
+                Err(e) if e.kind() == io::ErrorKind::UnexpectedEof => {
                     warn!("encode_stream_callback_async read unexpected eof");
                     break;
                 }
@@ -690,7 +686,6 @@ impl Erasure {
 
 #[cfg(test)]
 mod tests {
-
     use super::*;
 
     fn optional_shards(shards: &[Bytes]) -> Vec<Option<Vec<u8>>> {
@@ -1140,7 +1135,7 @@ mod tests {
         assert_eq!(total, 0);
         let observed = observed.lock().unwrap();
         let (kind, message) = observed.as_ref().expect("callback should be invoked once");
-        assert_eq!(*kind, std::io::ErrorKind::InvalidInput);
+        assert_eq!(*kind, io::ErrorKind::InvalidInput);
         assert!(message.contains("block_size"));
     }