* perf(ecstore): seek encrypted multipart Range GETs to the covering part boundary on the Legacy rio v1 backend
Phase A of https://github.com/rustfs/backlog/issues/1316. Encrypted Range GETs on the default (non rio-v2) backend previously planned storage_offset=0, storage_length=oi.size and discarded the decrypted prefix, so a small Range on a large SSE object read, erasure-decoded, and decrypted the whole object. Eligible multipart objects now seek to the covering part boundary using only the per-part size/actual_size metadata facts; the multipart decrypt reader already handles streams starting at any part boundary, so rio and the on-disk format are untouched. Single-part objects, compressed payloads, defective parts tables, and zero-length ranges keep the previous full read, and RUSTFS_ENCRYPTED_RANGE_SEEK=false restores it globally. A new histogram rustfs_get_encrypted_range_read_amplification plus a full|part_seek path counter record the physical/plaintext amplification at the ReadPlan decision point.
* fix(ecstore): guard encrypted multipart range seeks
* fix(io-metrics): align encrypted range metric names with the rustfs_io_ prefix
Every other metric in rustfs-io-metrics uses the rustfs_io_ prefix; the
two encrypted-range-seek metrics were the only exception. Rename before
first release so dashboards never see the unprefixed names.
Record local read_dir entry counts and duration for live-walker ListObjects scans so wide root/prefix amplification can be measured below the cross-set merge layer.
Add a focused LocalDisk scan_dir test showing a page limit of one still observes the whole parent directory enumeration, plus metric helper coverage.
Co-authored-by: heihutu <heihutu@gmail.com>
The `metrics` macros re-run the recorder's `register_*` on every emission — a
`RwLock` read, a name-key hash, and an `Arc` clone — even for a metric that
never varies its key. For the hot, label-less recorders on the per-IO path
(`record_data_transfer`, `record_io_latency`, `record_io_latency_p95/p99`,
`record_io_queue_congestion`) that lookup is pure overhead once observability is
on.
Add `counter_increment_cached!` / `gauge_set_cached!` / `histogram_record_cached!`
that resolve the handle once via `LazyLock` in production and reuse it. Under
`cfg(test)` they re-resolve on every call, because the `metrics` crate resolves
against a thread-local recorder that `with_local_recorder` swaps per test — a
process-global cached handle would bind to whichever recorder was active first
and break test capture. The macros only wrap FIXED (label-less) keys, and the
`metrics_enabled()` gate still short-circuits before any emission when disabled.
Verified: the only callers of these functions are the collector (io-metrics'
own cfg(test) tests, which re-resolve) and production code; no cross-crate test
captures them. rustfs-io-metrics builds on both cfg paths, 147 unit + 4 doctests
pass, clippy clean.
Addresses rustfs/backlog#1185 (P3, per-emission handle caching).
Co-authored-by: heihutu <heihutu@gmail.com>
The io-metrics crate already had per-stage GET/PUT switches, but the request
headers/summaries and ~40 general recorders (I/O scheduler, bytes-pool,
zero-copy, bandwidth, system-resource, error/timeout/retry) emitted
unconditionally, paying their label allocations and arithmetic even when no
metric recorder is installed.
Add `METRICS_ENABLED: AtomicBool` (default false) with `set_metrics_enabled()`
/ `metrics_enabled()`, isomorphic to the existing stage switches, and gate:
- GET headers/summaries via `get_stage_metrics_enabled()`:
record_get_object_request_start / _request_started / _request_result /
_timeout / _completion / _total_duration_with_path, plus legacy
record_get_object.
- PUT headers via `put_stage_metrics_enabled()`:
record_put_object_request_start / _request_result, plus legacy
record_put_object.
- 41 general recorders via `metrics_enabled()`.
Startup enables all three switches together under
`observability_metric_enabled()` (startup_observability.rs), with a passthrough
in startup_runtime_sources.rs. The global recorder is itself only installed
when observability metric export is on, so gating off is pure cost savings when
export is disabled and a no-op when it is enabled.
Deliberately NOT gated (would break function, not just observation): the EC
encode in-flight and GET buffered-bytes accounting guards
(add/remove_ec_encode_inflight_bytes, track_get_object_buffered_bytes) whose
values are read back by current_*; and the stateful sibling modules
(lock/deadlock/backpressure/autotuner/adaptive_ttl/collector/internode). The
console realtime metrics endpoint (/admin/v3/metrics) samples system state and
internode metrics directly, so it does not depend on the gated recorders.
Adds a toggle test exercising both the enabled and disabled paths, serialized
on the existing METRICS_FLAG_LOCK to stay robust under nextest's shared-process
model.
Co-authored-by: heihutu <heihutu@gmail.com>
The tracker never influenced traversal: walkdir's follow behavior is
fixed up front by follow_links(), and should_follow() only gated the
tracker's own bookkeeping. Its 'depth limit' compared tree depth (not
symlink chain depth), so RUSTFS_CAPACITY_MAX_SYMLINK_DEPTH was a
complete no-op while its telemetry claimed symlinks were skipped that
walkdir had in fact followed and counted; record_symlink was always
called with size 0, so tracked_bytes never left zero (S12).
Remove the tracker, its skipped/summary events, the symlink metric and
the depth env knob end to end (the env's 'as u8' truncation goes with
it), and document the real semantics at the walker: follow_links(true)
counts targets with walkdir's ancestor-loop detection breaking cycles,
follow_links(false) — the default — counts no symlink targets. The scan
root itself is pre-resolved since backlog#1015.
Ref: rustfs/backlog#1018 (S12 from audit rustfs/backlog#1010)
Progress/timeout checks only ran when file_count advanced, so trees of
directory-only or error-dense entries (dirs, traversal errors, symlinks
never increment file_count) bypassed the entire cooperative time budget
and held the blocking thread for unbounded time, while each error entry
logged its own warn — permission-dense trees flooded the log (S13).
The stall detector was structurally unreachable: it fired on 'no file
progress between checks', but checks themselves only ran when file
progress happened, so RUSTFS_CAPACITY_STALL_TIMEOUT never did anything
since its introduction (S09).
- Progress checks (timeout + early-sampling entry) are now driven by a
visited-entry counter that also advances on directories and errors, so
every tree shape reaches the budget checks.
- Remove the unreachable stall detector and its plumbing end to end
(ProgressMonitor fields, ScanLimits, config getter, env const, stall
metric). Genuine walker wedges are handled by the hard outer
wall-clock budget from backlog#1017; no decorative protection is left.
- Cap per-entry error warns at 10 per scan with an explicit suppression
notice; had_partial_errors still records the condition.
Ref: rustfs/backlog#1016 (S09+S13 from audit rustfs/backlog#1010)
* fix(obs): isolate process sampler windows
Refs rustfs/backlog#1004
Refs rustfs/backlog#986
- add a reusable ProcessSampler so callers can own independent sysinfo refresh windows
- wire separate sampler instances for obs metrics scheduling and memory observability
- keep compatibility helpers while avoiding cross-task CPU and disk delta interference
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): import process sampler bundle helper
Refs rustfs/backlog#1004
Refs rustfs/backlog#986
- import collect_process_metric_bundle_with in the metrics scheduler
- drop the stale collect_process_metric_bundle import after switching scheduler sampling to independent process samplers
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): move process sampler into blocking task
Refs rustfs/backlog#1004
Refs rustfs/backlog#986
- move the memory observability process sampler into the spawn_blocking closure
- satisfy the closure static lifetime required by tokio while keeping the isolated sampler design intact
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ecstore): make post-commit old data dir cleanup best-effort (backlog#898)
A write is authoritatively committed once rename_data returns Ok (the new
version is durable on >= write_quorum disks and immediately readable). The
subsequent reclamation of the now-dereferenced old object/<data_dir> is pure
space reclamation, yet commit_rename_data_dir propagated a below-quorum GC
failure via `?` into ErasureWriteQuorum -> 503, producing a false-negative
ACK for an already-persisted write. This is a deliberate divergence from
MinIO (erasure-object.go:1577), which couples the two; the divergence is
justified by durability semantics, not parity.
Changes:
- commit_rename_data_dir now returns a structured OldDataDirCleanup receipt
and never returns Err. Adds an old==committed-dir anti-misdelete guard and
a committed_data_dir parameter. Classification is extracted into pure
functions (classify_old_data_dir_cleanup / map_cleanup_join_result /
is_cleanup_not_found) so it is unit-testable. Task panic/cancel is mapped to
a non-ignored DiskError::other (never DiskNotFound), and not-found is
normalized to reclaimed.
- object.rs / multipart.rs consume the receipt instead of `?`. The result
reverts to Ok, so the invalidate_get_object_metadata_cache self-heal and the
capacity/compression accounting that a `?` early-return previously skipped
now run on the cleanup-failure path too.
- On residue, report_old_data_dir_cleanup emits leak metrics and enqueues an
object heal over the existing heal channel (disk-health signal replacing the
503). heal_object -> reclaim_orphan_data_dirs already reclaims unreferenced
local data dirs, closing the loop end to end.
- Adds rustfs_old_data_dir_* counters (attempted/reclaimed/leaked/below_quorum)
as the operator-visible backstop for leaked residue.
- Adds a test-only (#[cfg(test)]) delete fault-injection seam; in production it
inlines to a no-op None and has no behavioral effect.
Tests: pure-function A/C group + join-error mapping + actions decision; A5/A5b
real-disk guard/reclaim integration; end-to-end overwrite returning 200 while
old-data-dir cleanup fails. #864 rollback guard test remains green.
* fix(ecstore): resolve merge conflicts with origin/main in io_primitives.rs
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
fix(lock): renew distributed locks via heartbeat; wire server refresh (backlog#899)
A held distributed write lock had a fixed 30s TTL and was never renewed, so
any operation exceeding it got its per-node lease reclaimed and stolen by a
contender, causing split-brain writes. This implements Phase 0 + Phase 1 of the
#899 design (Phase 2 abort-on-loss is deferred and tracked in code comments).
Phase 0 (server refresh wiring, P0):
- node_service handle_refresh was a no-op stub that parsed args then always
returned success=true. Extract a testable `refresh_lock` free function that
actually delegates to the node's lock backend. Not-found maps to
success=false with no error_info, so RemoteClient::refresh keeps its
Ok(resp.success) semantics and yields a real not_found signal to the
coordinator heartbeat. Without this, client heartbeats were silently no-ops.
Phase 1 (client heartbeat + safe interval + observability):
- DistributedLockGuard spawns a heartbeat that refreshes every per-client lease
on a derived interval; interval is derived without Duration::clamp
(entries<=1 or interval>=ttl => no spawn), fixing the sub-second-ttl panic.
- Add LockLostSignal: declare the lock lost when not_found exceeds
entries.len() - refresh_quorum; RPC errors are not counted (absorbed by the
ttl > interval margin). Expose is_lock_lost()/lock_lost() for observers.
- disarm(), release(), and Drop now abort the heartbeat before releasing so no
refresh races the unlock (refresh only extends, never creates, a lease).
- Reclaim path stays behaviorally unchanged but now warns with owner/resource/
lease age and records a metric (#698 scavenger preserved).
- Add DEFAULT_LOCK_REFRESH_INTERVAL, LockRequest.refresh_interval (serde default
for RPC back-compat) + builder, and lock lifecycle metrics.
Open questions adopt the design's documented defaults (marked TODO in code):
refresh not-found -> Ok(false)/error_info=None; lost-quorum base entries.len();
DEFAULT_LOCK_REFRESH_INTERVAL=10s.
Tests: heartbeat keepalive/quorum-loss/jitter/boundary/disarm (lock crate),
server refresh delegation (rustfs), and end-to-end survives-past-ttl plus
crashed-owner-reclaim regressions (namespace).
* fix(rio): propagate http writer shutdown errors
* fix(ecstore): unify remote lock rpc deadlines
* fix(storage): reject corrupt read multiple payloads
* feat(rio): add internode http tuning profiles
* feat(metrics): add internode baseline signals
* feat(ecstore): observe shard locality topology
* feat(ecstore): gate shard locality scheduling
* feat(ecstore): gate batch read version rpc
* feat(ecstore): observe batch processor adaptation
* feat(ecstore): gate batch processor observation
* docs: add get benchmark regression analysis
* docs: add issue 797 execution plan status
* fix(ecstore): require explicit batch rpc support
* fix(ecstore): honor documented batch read gate
* fix(ecstore): keep batch read gate stable per call
* chore: update workspace dependencies
* feat(ecstore): log batch read gate decisions
* feat(ecstore): count batch read gate decisions
* test(issue-797): add local internode A/B runner
* test(rio): fix tuning profile spelling fixture
* fix(protocols): adapt sftp channel open callbacks
* fix(metrics): wrap batch processor observation args
* chore(docs): keep issue notes local only
* fix(storage): address internode review feedback
* fix(storage): address internode data-path review findings
- Run the BatchReadVersion auto-mode unary fallback outside the batch
RPC deadline so each read_version keeps its own per-op timeout and
health accounting instead of racing the whole batch against one
drive timeout.
- Cap adaptive batch-processor concurrency growth at a hard multiple
of the configured baseline so sustained fast batches cannot ratchet
past the configured limit.
- Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE,
and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading
the environment on hot paths.
- Skip shard read-cost collection in observe mode when stage metrics
are disabled, and cache the local endpoint host list instead of
rebuilding it on every read.
- Allow --warp-extra-args values starting with -- and drop the unused
warp_hosts_csv helper in the issue-797 A/B runner.
* fix(storage): address internode data-path review findings
- Run the BatchReadVersion auto-mode unary fallback outside the batch
RPC deadline so each read_version keeps its own per-op timeout and
health accounting instead of racing the whole batch against one
drive timeout.
- Cap adaptive batch-processor concurrency growth at a hard multiple
of the configured baseline so sustained fast batches cannot ratchet
past the configured limit.
- Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE,
and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading
the environment on hot paths.
- Skip shard read-cost collection in observe mode when stage metrics
are disabled, and cache the local endpoint host list instead of
rebuilding it on every read.
- Allow --warp-extra-args values starting with -- and drop the unused
warp_hosts_csv helper in the issue-797 A/B runner.
Co-Authored-By: heihutu<heihutu@gmail.com>
* fix(storage): align buffer clamp test with media cap
* fix(ecstore): release optimized read locks before streaming
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* perf(put): add eager path metrics and isolation tooling
* fix(decommission): persist progress adaptively (#3497)
Persist decommission progress after either the existing time interval or a migrated-item threshold, and flush progress baselines after bucket and terminal-state saves.
Also stabilize the OIDC discovery mock used by the pre-commit gate.
* refactor: move bucket operations contract (#3507)
* fix(s3): handle multipart flexible checksums (#3508)
* fix(io-core): avoid blocking on pooled buffer return
* perf(put): add slow inflight diagnostics
* perf(put): fix 16KiB regression with threshold and pool bypass
- Lower SMALL_EAGER_PUT_MAX_SIZE from 256KB to 8KB so objects >8KiB
use the streaming BufReader path (matches baseline behavior)
- Add POOL_BYPASE_MAX_SIZE (16KiB) to bypass BytesPool for very small
objects, avoiding Small-tier Mutex contention under high concurrency
- Add read_small_put_body_exact_direct() for direct Vec<u8> allocation
- Fix stale test assertions to match new 8KB threshold
Root cause analysis: the 16KiB regression was primarily caused by
instrumentation overhead in set_disk.rs (4x Instant::now() + metrics
per PUT), not BytesPool contention. Lowering the threshold eliminates
the eager-path overhead for 16KiB+ objects.
* perf(put): gate stage metrics behind observability flag
Add put_stage_metrics_enabled() AtomicBool switch in io-metrics crate.
When disabled (default), record_put_object_path() and
record_put_object_stage_duration() are no-ops, avoiding unnecessary
histogram/counter macro overhead in the PUT hot path.
The flag is set to true during startup when OTEL metric export is
enabled (rustfs_obs::observability_metric_enabled() == true).
This eliminates the per-request metrics overhead that contributed
to the 16KiB PUT regression when metrics collection is not active.
* perf(put): comprehensive optimization - restore eager path, cache env, remove UUID
Change 1: Restore SMALL_EAGER_PUT_MAX_SIZE from 8KB to 1MB
- The try_lock() fix (d13a189e3) eliminates the blocking that caused
service health timeouts under 512KiB c64 load
- Eager path with BytesPool is now safe for objects up to 1MB
- Recovers the eager path benefit for 32KiB-256KiB objects
Change 2: Adjust POOL_BYPASE_MAX_SIZE from 16KB to 4KB
- With eager path restored to 1MB, objects 4KB-1MB benefit from pool reuse
- Only ≤4KB objects bypass the pool (allocation cost negligible)
Change 3: Cache RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES via OnceLock
- Eliminates per-encode std::env::var() syscall
- Env var still works (read once at first use)
Change 4: Replace Uuid::new_v4() with Uuid::nil() in Erasure construction
- _id field is unused in hot paths (documented in code)
- Eliminates CSPRNG syscall per PUT request
Change 5: Add concurrency-aware buffer sizing to PUT path
- Reuses get_concurrency_aware_buffer_size() from GET path
- Reduces buffer size under high concurrency (0.4x at >8 concurrent)
- Lowers memory pressure for >1MB streaming PUTs
* chore: add pyroscope feature flag and clean up imports
- Add pyroscope feature flag forwarding to rustfs-obs
- Remove unused allow(non_upper_case_globals) in globals.rs
- Sort imports and fix Cargo.toml formatting consistency
* style: fix import ordering and code formatting
- Sort imports alphabetically in globals.rs, encode.rs
- Fix indentation in erasure_coding encode/erasure
- Clean up HashReader formatting in object_usecase.rs
* fix(test): use tokio::test for request_logging_layer tests
The tests call tokio::spawn via RequestContextLayer, which requires a
Tokio runtime. Changed from #[test] + futures::executor::block_on to
#[tokio::test] + .await, and replaced tracing::subscriber::with_default
with tracing::subscriber::set_default to support async.
* fix(bench): normalize no-space throughput/latency parsing in to_bps/to_ms
When a benchmark tool prints throughput without a separator (e.g. 123MiB/s),
awk '{print $2}' returns empty because the whole string is one field,
causing to_bps to return N/A and losing valid measurements in CSV output.
Insert a space between number and unit via sed before awk field splitting.
Same fix applied to to_ms for latency values like '50ms'.
Also add TODO comment on PUT path noting that get_concurrency_aware_buffer_size
reads ACTIVE_GET_REQUESTS instead of PUT concurrency (PR #3514 review).
Refs: PR #3514 review comments by chatgpt-codex-connector
* fix(metrics): correct POOL_BYPASS comments and separate PUT vs generic stage metrics
- Fix 3 comment-code mismatches: POOL_BYPASS_MAX_SIZE is 4KiB, not 16KiB
- Add generic record_stage_duration() with separate histogram
(rustfs_internal_stage_duration_ms) for non-PUT paths
- Replace record_put_object_stage_duration with record_stage_duration in
metacache_set, store_list_objects, and bucket_lifecycle_ops to avoid
polluting PUT-specific dashboards with listing/lifecycle timings
- Fix flaky test: serialize tests mutating PUT_STAGE_METRICS_ENABLED with
METRICS_FLAG_LOCK mutex and explicitly set desired state at test start
Refs: PR #3514 review comments by chatgpt-codex-connector
* style: apply cargo fmt to metacache_set.rs
---------
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
* feat(ecstore): add object lock diagnostics
Add configurable namespace lock diagnostics for object operations so production contention can be traced by operation, owner, and key.
Wrap object read/write lock acquisition in diagnostic guards across get, head, put, delete, copy, and multipart flows, and log slow acquisition and long hold durations behind new RUSTFS_OBJECT_LOCK_DIAG_* settings.
Verification:
- make pre-commit
* feat(obs): expose object lock diagnostics metrics
Add Prometheus metrics and Grafana panels for object namespace lock diagnostics, covering slow acquire counts, slow hold counts, acquire duration, hold duration, and the diagnostics-enabled state.
Adopt PR review feedback by keeping diagnostic guards alive through the guarded operation so long-hold warnings and metrics are emitted, reusing shared env helpers, and reducing default-path overhead when diagnostics are disabled.
Verification:
- cargo check -p rustfs-ecstore -p rustfs-io-metrics
- cargo test -p rustfs-ecstore store::object -- --nocapture
- make pre-commit
* perf(obs): reduce object lock diag overhead
Avoid repeated environment parsing on hot object-lock paths by caching the diagnostics-enabled flag, and stop allocating label strings for object lock metrics by recording static labels directly.
Strengthen io-metrics tests by using a local recorder and asserting that the expected object lock diagnostic counters, gauges, and histograms are emitted.
Verification:
- cargo check -p rustfs-ecstore -p rustfs-io-metrics
- cargo test -p rustfs-io-metrics -- --nocapture
- make pre-commit
houseme