gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-09 14:49:25 +00:00
Code Issues Packages Projects Releases Wiki Activity

perf(put): comprehensive PUT performance optimization (#3514)

Browse Source 
* perf(put): add eager path metrics and isolation tooling

* fix(decommission): persist progress adaptively (#3497)

Persist decommission progress after either the existing time interval or a migrated-item threshold, and flush progress baselines after bucket and terminal-state saves.

Also stabilize the OIDC discovery mock used by the pre-commit gate.

* refactor: move bucket operations contract (#3507)

* fix(s3): handle multipart flexible checksums (#3508)

* fix(io-core): avoid blocking on pooled buffer return

* perf(put): add slow inflight diagnostics

* perf(put): fix 16KiB regression with threshold and pool bypass

- Lower SMALL_EAGER_PUT_MAX_SIZE from 256KB to 8KB so objects >8KiB
  use the streaming BufReader path (matches baseline behavior)
- Add POOL_BYPASE_MAX_SIZE (16KiB) to bypass BytesPool for very small
  objects, avoiding Small-tier Mutex contention under high concurrency
- Add read_small_put_body_exact_direct() for direct Vec<u8> allocation
- Fix stale test assertions to match new 8KB threshold

Root cause analysis: the 16KiB regression was primarily caused by
instrumentation overhead in set_disk.rs (4x Instant::now() + metrics
per PUT), not BytesPool contention. Lowering the threshold eliminates
the eager-path overhead for 16KiB+ objects.

* perf(put): gate stage metrics behind observability flag

Add put_stage_metrics_enabled() AtomicBool switch in io-metrics crate.
When disabled (default), record_put_object_path() and
record_put_object_stage_duration() are no-ops, avoiding unnecessary
histogram/counter macro overhead in the PUT hot path.

The flag is set to true during startup when OTEL metric export is
enabled (rustfs_obs::observability_metric_enabled() == true).

This eliminates the per-request metrics overhead that contributed
to the 16KiB PUT regression when metrics collection is not active.

* perf(put): comprehensive optimization - restore eager path, cache env, remove UUID

Change 1: Restore SMALL_EAGER_PUT_MAX_SIZE from 8KB to 1MB
- The try_lock() fix (d13a189e3) eliminates the blocking that caused
  service health timeouts under 512KiB c64 load
- Eager path with BytesPool is now safe for objects up to 1MB
- Recovers the eager path benefit for 32KiB-256KiB objects

Change 2: Adjust POOL_BYPASE_MAX_SIZE from 16KB to 4KB
- With eager path restored to 1MB, objects 4KB-1MB benefit from pool reuse
- Only ≤4KB objects bypass the pool (allocation cost negligible)

Change 3: Cache RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES via OnceLock
- Eliminates per-encode std::env::var() syscall
- Env var still works (read once at first use)

Change 4: Replace Uuid::new_v4() with Uuid::nil() in Erasure construction
- _id field is unused in hot paths (documented in code)
- Eliminates CSPRNG syscall per PUT request

Change 5: Add concurrency-aware buffer sizing to PUT path
- Reuses get_concurrency_aware_buffer_size() from GET path
- Reduces buffer size under high concurrency (0.4x at >8 concurrent)
- Lowers memory pressure for >1MB streaming PUTs

* chore: add pyroscope feature flag and clean up imports

- Add pyroscope feature flag forwarding to rustfs-obs
- Remove unused allow(non_upper_case_globals) in globals.rs
- Sort imports and fix Cargo.toml formatting consistency

* style: fix import ordering and code formatting

- Sort imports alphabetically in globals.rs, encode.rs
- Fix indentation in erasure_coding encode/erasure
- Clean up HashReader formatting in object_usecase.rs

* fix(test): use tokio::test for request_logging_layer tests

The tests call tokio::spawn via RequestContextLayer, which requires a
Tokio runtime. Changed from #[test] + futures::executor::block_on to
#[tokio::test] + .await, and replaced tracing::subscriber::with_default
with tracing::subscriber::set_default to support async.

* fix(bench): normalize no-space throughput/latency parsing in to_bps/to_ms

When a benchmark tool prints throughput without a separator (e.g. 123MiB/s),
awk '{print $2}' returns empty because the whole string is one field,
causing to_bps to return N/A and losing valid measurements in CSV output.

Insert a space between number and unit via sed before awk field splitting.
Same fix applied to to_ms for latency values like '50ms'.

Also add TODO comment on PUT path noting that get_concurrency_aware_buffer_size
reads ACTIVE_GET_REQUESTS instead of PUT concurrency (PR #3514 review).

Refs: PR #3514 review comments by chatgpt-codex-connector

* fix(metrics): correct POOL_BYPASS comments and separate PUT vs generic stage metrics

- Fix 3 comment-code mismatches: POOL_BYPASS_MAX_SIZE is 4KiB, not 16KiB
- Add generic record_stage_duration() with separate histogram
  (rustfs_internal_stage_duration_ms) for non-PUT paths
- Replace record_put_object_stage_duration with record_stage_duration in
  metacache_set, store_list_objects, and bucket_lifecycle_ops to avoid
  polluting PUT-specific dashboards with listing/lifecycle timings
- Fix flaky test: serialize tests mutating PUT_STAGE_METRICS_ENABLED with
  METRICS_FLAG_LOCK mutex and explicitly set desired state at test start

Refs: PR #3514 review comments by chatgpt-codex-connector

* style: apply cargo fmt to metacache_set.rs

---------

Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
This commit is contained in:
houseme
2026-06-17 21:19:11 +08:00
committed by GitHub
parent a58692f550
commit 8d24d9133b
16 changed files with 769 additions and 135 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/common/src/globals.rs
-2
View File
@@ -12,8 +12,6 @@
// See the License for the specific language governing permissions and
// limitations under the License.
#![allow(non_upper_case_globals)] // FIXME
use chrono::{DateTime, Utc};
use std::collections::HashMap;
use std::sync::LazyLock;
crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs
+4
View File
@@ -1416,6 +1416,10 @@ fn spawn_tier_free_version_recovery_once(api: Arc<ECStore>) {
);
}
Err(err) => {
rustfs_io_metrics::record_stage_duration(
"lifecycle_free_version_recovery_failed",
started_at.elapsed().as_secs_f64() * 1000.0,
);
warn!(
event = EVENT_LIFECYCLE_WORKER_STATE,
component = LOG_COMPONENT_ECSTORE,
crates/ecstore/src/cache_value/metacache_set.rs
+23 -1
View File
@@ -184,9 +184,19 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
let mut need_fallback = false;
let mut last_err = None;
if let Some(disk) = opdisk {
let primary_walk_started = std::time::Instant::now();
match disk.walk_dir(wakl_opts, &mut wr).await {
Ok(_res) => {}
Ok(_res) => {
rustfs_io_metrics::record_stage_duration(
"metacache_walk_dir_primary",
primary_walk_started.elapsed().as_secs_f64() * 1000.0,
);
}
Err(err) => {
rustfs_io_metrics::record_stage_duration(
"metacache_walk_dir_primary_failed",
primary_walk_started.elapsed().as_secs_f64() * 1000.0,
);
warn!(
event = EVENT_METACACHE_LISTING,
component = LOG_COMPONENT_ECSTORE,
@@ -237,6 +247,7 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
return Err(err);
};
let fallback_walk_started = std::time::Instant::now();
match disk
.as_ref()
.walk_dir(
@@ -256,10 +267,18 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
.await
{
Ok(_r) => {
rustfs_io_metrics::record_stage_duration(
"metacache_walk_dir_fallback",
fallback_walk_started.elapsed().as_secs_f64() * 1000.0,
);
need_fallback = false;
last_err = None;
}
Err(err) => {
rustfs_io_metrics::record_stage_duration(
"metacache_walk_dir_fallback_failed",
fallback_walk_started.elapsed().as_secs_f64() * 1000.0,
);
error!(
event = EVENT_METACACHE_LISTING,
component = LOG_COMPONENT_ECSTORE,
@@ -538,7 +557,9 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
Ok(())
});
let merge_started = std::time::Instant::now();
if let Err(err) = revjob.await.map_err(std::io::Error::other)? {
rustfs_io_metrics::record_stage_duration("metacache_merge_failed", merge_started.elapsed().as_secs_f64() * 1000.0);
error!(
event = EVENT_METACACHE_LISTING,
component = LOG_COMPONENT_ECSTORE,
@@ -556,6 +577,7 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
return Err(err);
}
rustfs_io_metrics::record_stage_duration("metacache_merge", merge_started.elapsed().as_secs_f64() * 1000.0);
// The merge consumer can finish successfully before every producer finishes
// (for example after reaching EOF quorum while a tolerated drive is stalled,
crates/ecstore/src/erasure_coding/encode.rs
+16 -10
View File
@@ -31,6 +31,10 @@ const ENV_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES: &str = "RUSTFS_ERASURE_ENCOD
const DEFAULT_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES: usize = 32 * 1024 * 1024;
const DEFAULT_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BLOCKS: usize = 32;
/// Cached value of `RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES` env var.
/// Read once at first use via `OnceLock` to avoid per-encode syscall.
static CACHED_MAX_INFLIGHT_BYTES: std::sync::OnceLock<usize> = std::sync::OnceLock::new();
fn encode_channel_capacity(expanded_block_bytes: usize, max_inflight_bytes: usize) -> usize {
if expanded_block_bytes == 0 {
return 1;
@@ -273,10 +277,12 @@ impl Erasure {
// Bound queued encoded blocks by memory budget to avoid per-request spikes.
let expanded_block_bytes = self.shard_size().saturating_mul(self.total_shard_count());
let max_inflight_bytes = rustfs_utils::get_env_usize(
ENV_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES,
DEFAULT_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES,
);
let max_inflight_bytes = *CACHED_MAX_INFLIGHT_BYTES.get_or_init(|| {
rustfs_utils::get_env_usize(
ENV_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES,
DEFAULT_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES,
)
});
let inflight_blocks = encode_channel_capacity(expanded_block_bytes, max_inflight_bytes);
let (tx, mut rx) = mpsc::channel::<Vec<Bytes>>(inflight_blocks);
@@ -443,7 +449,7 @@ mod tests {
))];
let erasure = Arc::new(Erasure::new(1, 0, 16));
let reader = tokio::io::BufReader::new(std::io::Cursor::new(b"small payload".to_vec()));
let reader = tokio::io::BufReader::new(Cursor::new(b"small payload".to_vec()));
let (_reader, written) = erasure.encode(reader, &mut writers, 1).await.unwrap();
assert_eq!(written, b"small payload".len());
@@ -482,7 +488,7 @@ mod tests {
))];
let erasure = Arc::new(Erasure::new(1, 0, 0));
let reader = tokio::io::BufReader::new(std::io::Cursor::new(b"payload".to_vec()));
let reader = tokio::io::BufReader::new(Cursor::new(b"payload".to_vec()));
let err = erasure
.encode(reader, &mut writers, 1)
.await
@@ -505,7 +511,7 @@ mod tests {
))];
let erasure = Arc::new(Erasure::new(1, 0, 16));
let reader = tokio::io::BufReader::new(std::io::Cursor::new(Vec::<u8>::new()));
let reader = tokio::io::BufReader::new(Cursor::new(Vec::<u8>::new()));
let (_reader, total) = erasure.encode_inline_small(reader, &mut writers, 1).await.unwrap();
assert_eq!(total, 0);
@@ -537,7 +543,7 @@ mod tests {
let payload = b"hello inline small";
let erasure = Arc::new(Erasure::new(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE));
let reader = tokio::io::BufReader::new(std::io::Cursor::new(payload.to_vec()));
let reader = tokio::io::BufReader::new(Cursor::new(payload.to_vec()));
let (_reader, total) = erasure.encode_inline_small(reader, &mut writers, DATA_SHARDS).await.unwrap();
assert_eq!(total, payload.len());
@@ -569,7 +575,7 @@ mod tests {
let payload = b"hello single block";
let erasure = Arc::new(Erasure::new(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE));
let reader = tokio::io::BufReader::new(std::io::Cursor::new(payload.to_vec()));
let reader = tokio::io::BufReader::new(Cursor::new(payload.to_vec()));
let (_reader, total) = erasure
.encode_single_block_non_inline(reader, &mut writers, DATA_SHARDS)
.await
@@ -603,7 +609,7 @@ mod tests {
let payload = vec![1u8; BLOCK_SIZE + 1];
let erasure = Arc::new(Erasure::new(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE));
let reader = tokio::io::BufReader::new(std::io::Cursor::new(payload));
let reader = tokio::io::BufReader::new(Cursor::new(payload));
let err = erasure
.encode_single_block_non_inline(reader, &mut writers, DATA_SHARDS)
.await
crates/ecstore/src/erasure_coding/erasure.rs
+7 -12
View File
@@ -394,7 +394,7 @@ impl Erasure {
encoder,
legacy_encoder,
uses_legacy,
_id: Uuid::new_v4(),
_id: Uuid::nil(), // Unused in hot paths; avoid CSPRNG syscall
}
}
@@ -631,15 +631,11 @@ impl Erasure {
) -> Result<usize, E>
where
R: AsyncRead + Send + Sync + Unpin,
F: FnMut(std::io::Result<Vec<Bytes>>) -> Fut + Send,
Fut: std::future::Future<Output = Result<(), E>> + Send,
F: FnMut(io::Result<Vec<Bytes>>) -> Fut + Send,
Fut: Future<Output = Result<(), E>> + Send,
{
if self.block_size == 0 {
on_block(Err(std::io::Error::new(
std::io::ErrorKind::InvalidInput,
"erasure block_size must be non-zero",
)))
.await?;
on_block(Err(io::Error::new(io::ErrorKind::InvalidInput, "erasure block_size must be non-zero"))).await?;
return Ok(0);
}
@@ -662,7 +658,7 @@ impl Erasure {
{
Ok(result) => result,
Err(err) => {
on_block(Err(std::io::Error::other(format!("EC encode task failed: {err}")))).await?;
on_block(Err(io::Error::other(format!("EC encode task failed: {err}")))).await?;
break;
}
};
@@ -673,7 +669,7 @@ impl Erasure {
warn!("encode_stream_callback_async read unexpected ok");
break;
}
Err(e) if e.kind() == std::io::ErrorKind::UnexpectedEof => {
Err(e) if e.kind() == io::ErrorKind::UnexpectedEof => {
warn!("encode_stream_callback_async read unexpected eof");
break;
}
@@ -690,7 +686,6 @@ impl Erasure {
#[cfg(test)]
mod tests {
use super::*;
fn optional_shards(shards: &[Bytes]) -> Vec<Option<Vec<u8>>> {
@@ -1140,7 +1135,7 @@ mod tests {
assert_eq!(total, 0);
let observed = observed.lock().unwrap();
let (kind, message) = observed.as_ref().expect("callback should be invoked once");
assert_eq!(*kind, std::io::ErrorKind::InvalidInput);
assert_eq!(*kind, io::ErrorKind::InvalidInput);
assert!(message.contains("block_size"));
}
crates/ecstore/src/set_disk.rs
+83
View File
@@ -137,6 +137,8 @@ const LOG_SUBSYSTEM_SET_DISK: &str = "set_disk";
const EVENT_SET_DISK_MULTIPART: &str = "set_disk_multipart";
const EVENT_SET_DISK_WRITE: &str = "set_disk_write";
const EVENT_SET_DISK_HEAL: &str = "set_disk_heal";
const EVENT_SET_DISK_COMMIT_TAIL_SLOW: &str = "set_disk_commit_tail_slow";
const SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS: u128 = 5_000;
use crate::rio::{EtagResolvable, HashReader, HashReaderMut, TryGetIndex as _};
@@ -804,6 +806,16 @@ enum SmallWritePath {
Pipeline,
}
impl SmallWritePath {
fn metric_label(&self) -> &'static str {
match self {
SmallWritePath::Inline => "write_inline",
SmallWritePath::SingleBlockNonInline => "write_single_block_non_inline",
SmallWritePath::Pipeline => "write_pipeline",
}
}
}
fn classify_small_write_path(is_inline_buffer: bool, object_size: i64, block_size: usize) -> SmallWritePath {
if should_use_inline_small_fast_path(is_inline_buffer, object_size, block_size) {
SmallWritePath::Inline
@@ -1061,6 +1073,7 @@ impl ObjectIO for SetDisks {
let shard_file_size = erasure.shard_file_size(data.size());
let shard_size = erasure.shard_size();
let writer_setup_stage_start = Instant::now();
let writer_futs: Vec<_> = shuffle_disks
.iter()
.map(|disk_op| {
@@ -1107,6 +1120,10 @@ impl ObjectIO for SetDisks {
writers.push(w);
errors.push(e);
}
rustfs_io_metrics::record_put_object_stage_duration(
"set_disk_writer_setup",
writer_setup_stage_start.elapsed().as_secs_f64() * 1000.0,
);
let nil_count = errors.iter().filter(|&e| e.is_none()).count();
if nil_count < write_quorum {
@@ -1135,7 +1152,9 @@ impl ObjectIO for SetDisks {
);
let write_path = classify_small_write_path(is_inline_buffer, data.size(), fi.erasure.block_size);
rustfs_io_metrics::record_put_object_path(write_path.metric_label());
let encode_stage_start = Instant::now();
let (reader, w_size) = match write_path {
SmallWritePath::Inline => match Arc::new(erasure)
.encode_inline_small(stream, &mut writers, write_quorum)
@@ -1165,6 +1184,10 @@ impl ObjectIO for SetDisks {
}
},
};
rustfs_io_metrics::record_put_object_stage_duration(
"set_disk_encode",
encode_stage_start.elapsed().as_secs_f64() * 1000.0,
);
let _ = mem::replace(&mut data.stream, reader);
// if let Err(err) = close_bitrot_writers(&mut writers).await {
@@ -1259,6 +1282,7 @@ impl ObjectIO for SetDisks {
object_lock_guard = Some(self.acquire_write_lock_diag("put_object_commit", bucket, object).await?);
}
let rename_stage_start = Instant::now();
let (online_disks, _, op_old_dir, cleanup_disks) = Self::rename_data(
&shuffle_disks,
RUSTFS_META_TMP_BUCKET,
@@ -1269,10 +1293,52 @@ impl ObjectIO for SetDisks {
write_quorum,
)
.await?;
rustfs_io_metrics::record_put_object_stage_duration(
"set_disk_rename",
rename_stage_start.elapsed().as_secs_f64() * 1000.0,
);
let rename_stage_ms = rename_stage_start.elapsed().as_millis();
if rename_stage_ms >= SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS {
warn!(
event = EVENT_SET_DISK_COMMIT_TAIL_SLOW,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_SET_DISK,
stage = "rename_data",
bucket = %bucket,
object = %object,
tmp_dir = %tmp_dir,
duration_ms = rename_stage_ms as u64,
write_quorum,
state = "slow",
"SetDisk commit tail stage is slow"
);
}
if let Some(old_dir) = op_old_dir {
let cleanup_stage_start = Instant::now();
self.commit_rename_data_dir(&cleanup_disks, bucket, object, &old_dir.to_string(), write_quorum)
.await?;
rustfs_io_metrics::record_put_object_stage_duration(
"set_disk_old_data_cleanup",
cleanup_stage_start.elapsed().as_secs_f64() * 1000.0,
);
let cleanup_stage_ms = cleanup_stage_start.elapsed().as_millis();
if cleanup_stage_ms >= SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS {
warn!(
event = EVENT_SET_DISK_COMMIT_TAIL_SLOW,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_SET_DISK,
stage = "commit_rename_data_dir",
bucket = %bucket,
object = %object,
tmp_dir = %tmp_dir,
old_dir = %old_dir,
duration_ms = cleanup_stage_ms as u64,
write_quorum,
state = "slow",
"SetDisk commit tail stage is slow"
);
}
}
drop(object_lock_guard); // drop object lock guard to release the lock
@@ -1307,6 +1373,23 @@ impl ObjectIO for SetDisks {
);
}
let total_commit_tail_ms = rename_stage_start.elapsed().as_millis();
if total_commit_tail_ms >= SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS {
warn!(
event = EVENT_SET_DISK_COMMIT_TAIL_SLOW,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_SET_DISK,
stage = "put_object_commit_tail",
bucket = %bucket,
object = %object,
tmp_dir = %tmp_dir,
duration_ms = total_commit_tail_ms as u64,
write_quorum,
state = "slow",
"SetDisk commit tail is slow"
);
}
Ok(ObjectInfo::from_file_info(&fi, bucket, object, opts.versioned || opts.version_suspended))
}
.await;
crates/ecstore/src/store_list_objects.rs
+5
View File
@@ -1158,6 +1158,7 @@ impl ECStore {
tokio::spawn(async move { merge_entry_channels(rx, inputs, merge_tx, 1).await }.instrument(tracing::Span::current()));
let walk_started = std::time::Instant::now();
let walk_results = join_all(futures).await;
let mut errs = Vec::new();
for walk_result in walk_results {
@@ -1166,6 +1167,10 @@ impl ECStore {
Err(err) => errs.push(Some(err.into())),
}
}
rustfs_io_metrics::record_stage_duration(
"store_list_objects_walk_internal",
walk_started.elapsed().as_secs_f64() * 1000.0,
);
let result = walk_result_from_set_errors(&errs);
if let Err(err) = &result {
crates/io-core/src/pool.rs
+16 -11
View File
@@ -423,23 +423,31 @@ impl PoolTier {
})
}
/// Return a buffer to the pool for reuse.
/// Return a buffer to the pool for reuse without ever blocking the caller.
fn return_buffer(&self, buffer: BytesMut) {
let mut available = self.available_buffers.lock().unwrap_or_else(|e| e.into_inner());
// Limit the size of the pool to prevent unbounded growth
if available.len() < self.max_buffers {
available.push(buffer);
if let Some(ref metrics) = *self.metrics.lock().unwrap_or_else(|e| e.into_inner()) {
let mut buffer = Some(buffer);
if let Ok(mut available) = self.available_buffers.try_lock()
&& available.len() < self.max_buffers
{
available.push(buffer.take().expect("buffer should be present until returned"));
if let Ok(metrics) = self.metrics.try_lock()
&& let Some(metrics) = metrics.as_ref()
{
metrics.available_buffers.fetch_add(1, Ordering::Relaxed);
}
} else {
}
if let Some(buffer) = buffer {
let released_bytes = buffer.capacity() as u64;
self.tier_current_allocated_bytes
.fetch_update(Ordering::Relaxed, Ordering::Relaxed, |current| {
Some(current.saturating_sub(released_bytes))
})
.ok();
if let Some(ref metrics) = *self.metrics.lock().unwrap_or_else(|e| e.into_inner()) {
if let Ok(metrics) = self.metrics.try_lock()
&& let Some(metrics) = metrics.as_ref()
{
metrics
.current_allocated_bytes
.fetch_update(Ordering::Relaxed, Ordering::Relaxed, |current| {
@@ -448,7 +456,6 @@ impl PoolTier {
.ok();
}
}
// If pool is full, buffer is dropped and memory is freed
rustfs_io_metrics::record_bytes_pool_allocated(self.name, self.tier_current_allocated_bytes.load(Ordering::Relaxed));
}
}
@@ -458,8 +465,6 @@ impl Drop for PooledBuffer {
// buffer moves it exactly once into the pool when a tier still owns it.
#[allow(unsafe_code)]
fn drop(&mut self) {
// Return buffer to pool if tier reference exists.
// Otherwise, drop the standalone fallback buffer normally.
let buffer = unsafe { ManuallyDrop::take(&mut self.buffer) };
if let Some(ref tier) = self.tier {
tier.return_buffer(buffer);
crates/io-metrics/src/lib.rs
+87 -4
View File
@@ -49,7 +49,30 @@
#[macro_use]
extern crate metrics;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
/// Global switch for detailed per-stage PUT metrics (path label, stage durations).
/// When `false`, `record_put_object_path` and `record_put_object_stage_duration`
/// become no-ops, and callers can skip the `Instant::now()` syscalls entirely.
///
/// Set to `true` during startup when OTEL metric export is enabled.
static PUT_STAGE_METRICS_ENABLED: AtomicBool = AtomicBool::new(false);
/// Enable or disable detailed per-stage PUT metrics.
///
/// Called once during startup, typically gated by `rustfs_obs::observability_metric_enabled()`.
pub fn set_put_stage_metrics_enabled(enabled: bool) {
PUT_STAGE_METRICS_ENABLED.store(enabled, Ordering::Relaxed);
}
/// Returns `true` if detailed per-stage PUT metrics are enabled.
///
/// Callers should check this before calling `Instant::now()` for stage timing
/// to avoid unnecessary syscalls when metrics are disabled.
#[inline(always)]
pub fn put_stage_metrics_enabled() -> bool {
PUT_STAGE_METRICS_ENABLED.load(Ordering::Relaxed)
}
// Public modules
pub mod adaptive_ttl;
@@ -409,9 +432,9 @@ pub fn record_get_object(duration_ms: f64, size_bytes: i64) {
///
/// * `duration_ms` - Operation duration in milliseconds
/// * `size_bytes` - Object size in bytes
/// * `zero_copy_enabled` - Whether zero-copy was enabled for this operation
/// * `zero_copy_eligible` - Whether the request was eligible for a zero-copy path
#[inline(always)]
pub fn record_put_object(duration_ms: f64, size_bytes: i64, zero_copy_enabled: bool) {
pub fn record_put_object(duration_ms: f64, size_bytes: i64, zero_copy_eligible: bool) {
counter!("rustfs_s3_put_object_total").increment(1);
histogram!("rustfs_s3_put_object_duration_ms").record(duration_ms);
@@ -419,11 +442,37 @@ pub fn record_put_object(duration_ms: f64, size_bytes: i64, zero_copy_enabled: b
histogram!("rustfs_s3_put_object_size_bytes").record(size_bytes as f64);
}
if zero_copy_enabled {
if zero_copy_eligible {
// Backward-compatible alias for historical dashboards.
counter!("rustfs_s3_put_object_zero_copy_enabled_total").increment(1);
counter!("rustfs_s3_put_object_zero_copy_eligible_total").increment(1);
}
}
#[inline(always)]
pub fn record_put_object_path(path: &'static str) {
if !put_stage_metrics_enabled() {
return;
}
counter!("rustfs_s3_put_object_path_total", "path" => path).increment(1);
}
#[inline(always)]
pub fn record_put_object_stage_duration(stage: &'static str, duration_ms: f64) {
if !put_stage_metrics_enabled() {
return;
}
histogram!("rustfs_s3_put_object_stage_duration_ms", "stage" => stage).record(duration_ms);
}
/// Record generic internal operation stage duration (non-PUT paths).
/// Use this for metacache walks, listing, lifecycle, and other background
/// operations that are NOT part of the PUT object hot path.
#[inline(always)]
pub fn record_stage_duration(stage: &'static str, duration_ms: f64) {
histogram!("rustfs_internal_stage_duration_ms", "stage" => stage).record(duration_ms);
}
/// Record ListObjects operation metrics.
///
/// # Arguments
@@ -788,6 +837,10 @@ pub fn record_io_latency_p99(latency_ms: f64) {
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Mutex;
// Serialize tests that mutate the process-global PUT_STAGE_METRICS_ENABLED flag.
static METRICS_FLAG_LOCK: Mutex<()> = Mutex::new(());
#[test]
fn test_record_zero_copy_read() {
@@ -824,6 +877,36 @@ mod tests {
record_put_object(100.0, 512, false);
}
#[test]
fn test_record_put_object_path_and_stage() {
let _guard = METRICS_FLAG_LOCK.lock().unwrap_or_else(|e| e.into_inner());
set_put_stage_metrics_enabled(true);
record_put_object_path("small_eager");
record_put_object_path("write_inline");
record_put_object_stage_duration("ingress_prepare", 12.5);
record_put_object_stage_duration("set_disk_encode", 8.0);
set_put_stage_metrics_enabled(false);
}
#[test]
fn test_put_stage_metrics_disabled_by_default() {
let _guard = METRICS_FLAG_LOCK.lock().unwrap_or_else(|e| e.into_inner());
set_put_stage_metrics_enabled(false);
// These should be no-ops (no panic, no recording)
record_put_object_path("small_eager");
record_put_object_stage_duration("set_disk_encode", 5.0);
// Still disabled
assert!(!put_stage_metrics_enabled());
}
#[test]
fn test_record_stage_duration_generic() {
// Generic stage duration should always record (no gating flag)
record_stage_duration("metacache_walk_dir_primary", 15.0);
record_stage_duration("store_list_objects_walk_internal", 8.5);
record_stage_duration("lifecycle_free_version_recovery_failed", 120.0);
}
#[test]
fn test_record_list_objects() {
record_list_objects(50.0, 100, false);
crates/object-capacity/src/scan.rs
+2
View File
@@ -556,6 +556,7 @@ async fn get_dir_size_async(path: &Path) -> Result<CapacityScanResult, std::io::
subsystem = LOG_SUBSYSTEM_SCAN,
result = "partial",
root_path = ?path,
file_count,
error = %err,
"capacity scan traversal failed"
);
@@ -590,6 +591,7 @@ async fn get_dir_size_async(path: &Path) -> Result<CapacityScanResult, std::io::
subsystem = LOG_SUBSYSTEM_SCAN,
result = "partial",
entry_path = ?entry.path(),
file_count,
error = %err,
"capacity scan metadata failed"
);