perf(capacity): remove per-PUT global lock and per-disk allocation from write dirty-scope
Every successful write recorded its capacity dirty scope by allocating an endpoint/path String per online disk, deduplicating through a HashSet, entering the global dirty-scope Mutex, and — in the app response path — taking a global async RwLock to record the write frequency. Under small-object high concurrency this created a global serialization point and O(disks) allocation on the hot path (https://github.com/rustfs/backlog/issues/1315).
This change makes the steady-state write path allocation-free and lock-free without altering capacity accounting semantics:
- Memoize the per-set dirty scope. Each set resolves its disks' immutable endpoint/path identity lazily into a slot-indexed cache and reuses a shared `Arc<CapacityScope>`; steady-state writes clone the Arc under a read lock instead of rebuilding String/HashSet. The heal path keeps an ad-hoc scope builder because it passes disks in erasure-distribution order rather than physical-slot order.
- Add a monotonic generation to the global dirty-scope registry, advanced only when a non-empty drain removes disks. A set upgrades the global registry mutex only on the first write of each generation and then skips it while the generation is unchanged; the observed generation is read under the registry lock so a concurrent drain forces a re-mark, preventing lost updates. The write commits its bytes before recording the scope, so any drain that could remove the mark is ordered after the commit and the following refresh reads the committed bytes.
- Replace the write-frequency `RwLock<WriteRecord>` with lock-free atomics: per-second CAS buckets, an atomic last-write timestamp, and an atomic total counter. The frequency window and debounce semantics the refresh scheduler relies on are unchanged.
Capacity marking remains a conservative superset of the disks actually written, so admin/scan totals are byte-for-byte identical: extra dirty marks only trigger a re-read of a disk whose usage is unchanged. White-box tests assert the memoized scope equals the previous ad-hoc construction, that the global registry is upgraded exactly once per generation and re-marked after a drain, and that the lock-free write record is exact under concurrent contention.
Ref: https://github.com/rustfs/backlog/issues/1315
* chore(deps): remove redundant dependency features
Remove manifest feature entries that are implied by other requested features in the same dependency declaration.
Verified that the resolved Cargo feature graph is unchanged after the cleanup.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(deps): narrow tokio and reqwest features
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Move workspace-level dependency feature lists into the member crates that consume each dependency while keeping required default-features flags at the workspace root.
Also refresh starshard to 2.2.2 via cargo update and cargo upgrade --exclude ratelimit.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(object-capacity): harden clock handling in write window, background intervals and scope registry
Three low-severity robustness gaps (S32+S33+#35):
- WriteRecord keyed its 60-bucket write window by wall-clock unix
seconds while the debounce used Instant. An NTP step backwards marked
recent buckets as future and silently suppressed write-triggered
refreshes until the wall clock caught up; a forward step aged the
whole window at once. Bucket keys now derive from a monotonic
per-record epoch, immune to clock steps.
- Background refresh/metrics intervals were only clamped at zero;
RUSTFS_CAPACITY_SCHEDULED_INTERVAL=u64::MAX overflowed
Instant + Duration and panicked the spawned task, silently killing
scheduled refreshes. Intervals now clamp into [1s, 30 days] via one
helper with a structured warn.
- record_capacity_scope merged new disks into an expired entry and
refreshed its recorded_at, resurrecting a scope take_capacity_scope
would have discarded (and the merge path never pruned). Expired
entries are now replaced, not merged into.
Ref: rustfs/backlog#1022 (S32+S33+#35 from audit rustfs/backlog#1010)
The tracker never influenced traversal: walkdir's follow behavior is
fixed up front by follow_links(), and should_follow() only gated the
tracker's own bookkeeping. Its 'depth limit' compared tree depth (not
symlink chain depth), so RUSTFS_CAPACITY_MAX_SYMLINK_DEPTH was a
complete no-op while its telemetry claimed symlinks were skipped that
walkdir had in fact followed and counted; record_symlink was always
called with size 0, so tracked_bytes never left zero (S12).
Remove the tracker, its skipped/summary events, the symlink metric and
the depth env knob end to end (the env's 'as u8' truncation goes with
it), and document the real semantics at the walker: follow_links(true)
counts targets with walkdir's ancestor-loop detection breaking cycles,
follow_links(false) — the default — counts no symlink targets. The scan
root itself is pre-resolved since backlog#1015.
Ref: rustfs/backlog#1018 (S12 from audit rustfs/backlog#1010)
walkdir defaults to follow_root_links=true precisely so a root that is
itself a symlink still descends, but the scanner overrode it with the
general follow_symlinks flag (default false). For the common indirection
layout (/data/rustfs0 -> /mnt/vol0) the walk yielded a single skipped
symlink entry and returned used_bytes=0, file_count=0, is_estimated=false
with no error and no log — an exact 0 committed straight into the
per-disk baseline (S05).
Canonicalize the scan root before walking (also covering nested
indirection and keeping the dedicated-mount statvfs check on the real
path); resolution failures surface as scan errors instead of a silent
zero. Inner-symlink policy is unchanged.
Ref: rustfs/backlog#1015 (S05 from audit rustfs/backlog#1010)
RUSTFS_CAPACITY_MAX_FILES_THRESHOLD=0 made every file count as
overflow with an empty exact prefix, so disks with fewer files than the
sample rate committed 0 bytes as the cluster total; STAT_TIMEOUT=0 with
dynamic timeout disabled made every scan fail at the first entry. Both
were accepted unvalidated, unlike the already-clamped sample_rate and
interval values.
Zero values for the threshold and the stat/min/max timeouts now fall
back to their defaults with a single structured warn at config load
(S11). The symlink-depth knob is left untouched here: it is removed
entirely by the backlog#1018 fix.
Ref: rustfs/backlog#1019 (S11 from audit rustfs/backlog#1010)
Progress/timeout checks only ran when file_count advanced, so trees of
directory-only or error-dense entries (dirs, traversal errors, symlinks
never increment file_count) bypassed the entire cooperative time budget
and held the blocking thread for unbounded time, while each error entry
logged its own warn — permission-dense trees flooded the log (S13).
The stall detector was structurally unreachable: it fired on 'no file
progress between checks', but checks themselves only ran when file
progress happened, so RUSTFS_CAPACITY_STALL_TIMEOUT never did anything
since its introduction (S09).
- Progress checks (timeout + early-sampling entry) are now driven by a
visited-entry counter that also advances on directories and errors, so
every tree shape reaches the budget checks.
- Remove the unreachable stall detector and its plumbing end to end
(ProgressMonitor fields, ScanLimits, config getter, env const, stall
metric). Genuine walker wedges are handled by the hard outer
wall-clock budget from backlog#1017; no decorative protection is left.
- Cap per-entry error warns at 10 per scan with an explicit suppression
notice; had_partial_errors still records the condition.
Ref: rustfs/backlog#1016 (S09+S13 from audit rustfs/backlog#1010)
Two dirty-mark lifecycle gaps (S19+S30):
- A commit cleared every dirty mark for the disks it scanned, even marks
recorded while the walker was already past the written prefix, so the
next dirty-subset refresh served stale cached bytes as exact until the
next full scan. Dirty marks now carry the instant they were last
recorded and a commit only clears marks that predate its scan start
(CapacityUpdate::scan_started_at, set by refresh_capacity_with_scope).
- The write side marks every disk of an EC set dirty, including remote
peers, while scans and clearing only cover local disks — remote or
removed disks stayed marked forever, keeping the dirty-disk gauge
permanently non-zero with bounded memory stuck behind it. The refresh
disk selection now prunes dirty entries outside the current local
topology before consulting them.
Ref: rustfs/backlog#1020 (S19+S30 from audit rustfs/backlog#1010)
Two defensive gaps left the refresh singleflight vulnerable to a
permanent wedge (running=true forever: scheduled refreshes silently
stop, admin capacity joiners hang until restart):
- spawn_refresh_if_needed's spawned task had no leader guard, unlike
refresh_or_join: a panic after the refresh future completes (commit,
metrics) killed the task before the reset block. The task now holds
the same RAII RefreshLeaderGuard, disarmed only after the state is
reset and the result published (S20).
- refresh_fn() was evaluated before AssertUnwindSafe wrapping in both
paths, so a panic while constructing the future escaped catch_unwind;
construction now happens inside the wrapped future.
- RefreshLeaderGuard::drop silently skipped the reset when try_lock was
contended and no tokio runtime was current; it now falls back to a
blocking reset, which is safe precisely because there is no executor
to stall in that context (S31).
Ref: rustfs/backlog#1021 (S20+S31 from audit rustfs/backlog#1010)
The whole disk traversal runs inside spawn_blocking with no timeout
anywhere on the async side; the in-scan ProgressMonitor checks are
cooperative and only run between walker entries. A stat/readdir blocked
on a dying disk or hung NFS mount therefore never returns: the blocking
thread leaks, the refresh singleflight stays running forever, every
subsequent scheduled refresh is skipped as inflight, and every admin
capacity query joins an unbounded wait until process restart (S02).
- Wrap each disk scan in tokio::time::timeout with a hard wall-clock
ceiling of 2x the cooperative budget (min 5s). On expiry the caller
fails the disk scan (releasing the singleflight through the normal
error path, where the degraded/partial machinery from backlog#1014
keeps the failed disk's last-known value) and a shared AtomicBool asks
the blocking walker to exit at its next entry, bounding the thread
leak to the single wedged syscall.
- Bound refresh_or_join joiner waits at 5 minutes so admin queries
degrade into a clear error instead of hanging if the leader wedges in
a way the drop/panic guards don't cover.
Ref: rustfs/backlog#1017 (S02 from audit rustfs/backlog#1010)
The capacity scan's timeout fallback required the exact prefix
(threshold + sample_rate files, default ~200k) to be fully enumerated
before any sample existed. Cold-cache HDDs cannot stat that many files
inside the 15s budget, so large slow disks — exactly the disks sampling
was designed for — always timed out with sampled_count == 0, discarded
the accumulated exact-prefix work with a hard error, and re-ran 15s of
useless full-disk I/O every 120s (S03). When a sample did exist, the
estimate only extrapolated over files the walker had seen, silently
under-reporting disks whose tail was never reached (S10).
- Enter sampling early: once half the (possibly dynamic) time budget is
consumed and the exact prefix hasn't filled, freeze the threshold at
the current position so the remaining budget collects samples and the
timeout fallback is always reachable.
- Compensate the unseen tail: when the scan root is a dedicated mount
point (stat st_dev differs from the parent), take the max of the
seen-files extrapolation and the filesystem-level used bytes. On
shared filesystems (multi-disk dev layouts) statvfs would overcount,
so it is not trusted and behavior is unchanged.
- sampled_count == 0 at timeout no longer hard-fails when a dedicated
mount provides filesystem usage; the exact prefix is kept as a floor.
Without any estimator the original error still surfaces.
- Extract ScanLimits from env reading so the blocking scanner is
parameterizable in tests.
Ref: rustfs/backlog#1013 (S03+S10 from audit rustfs/backlog#1010)
A full refresh with partial disk failures used to commit the surviving
subset's sum as a fresh exact cluster total (no field carried the
partial-failure fact), while the complete disk cache kept the failed
disk's old value — so the reported capacity oscillated between the
partial sum and the cache-merged total on alternating refreshes.
- Add a degraded flag to CapacityUpdate/CachedCapacity, set when the
scan behind the update had partial errors; expose it in refresh logs,
admin capacity logs and a new rustfs_capacity_degraded_readings_total
counter.
- On a degraded full refresh, surface only the disks whose own scan
fully succeeded and merge them over a complete disk cache, so failed
disks keep their last-known values and the published total no longer
dips and bounces back. The disk cache is never replaced from a
degraded refresh.
- Without a complete cache, keep the partial sum (unchanged #805
non-pollution behavior) but mark the reading degraded.
Ref: rustfs/backlog#1014 (S06 from audit rustfs/backlog#1010)
A dirty-subset refresh scans only the dirty disks, so its raw
`CapacityUpdate` carries just that subset's `total_used`/`file_count`.
`update_capacity` recomputed the correct cluster-wide total into a local
variable and wrote it to the cache, but never wrote it back to the
`CapacityUpdate` that `refresh_or_join`/`spawn_refresh_if_needed` return
and publish to joiners. The admin blocking path (`resolve_admin_used_capacity`
-> `refresh_or_join_admin_disks(allow_dirty_subset=true)`) consumes
`update.total_used`, so a single dirty disk in an N-disk cluster made
admin StorageInfo report only the scanned subset's bytes (a large
transient undercount for that request and same-cycle joiners).
`CachedDiskCapacity` also dropped each disk's `file_count`/`is_estimated`,
so subset refreshes could not recompute a correct cluster file count and
would launder an estimated per-disk value into an exact cluster total.
Fix:
- `update_capacity` now reconciles `total_used`, `file_count`, and
`is_estimated` from the full per-disk cache and returns the corrected
`CapacityUpdate`.
- `CachedDiskCapacity` stores `file_count`/`is_estimated` per disk;
`file_count` sums the cache and `is_estimated` is the OR across disks.
- `refresh_or_join` and `spawn_refresh_if_needed` rebind their result to
the reconciled update so the leader return value and the value
published to joiners both carry cluster totals.
Tests: extend the subset-refresh test to assert the returned update's
reconciled `total_used`/`file_count`/`is_estimated`, and add a
`refresh_or_join` dirty-subset test asserting the leader returns the
merged cluster total (not the subset sum) and matches the cache.
Refs: https://github.com/rustfs/backlog/issues/1011
fix(object-capacity): use u128 intermediate to stop sampling estimate overflow (backlog#1012)
The three sampling extrapolation sites in scan.rs computed
`overflow_sampled_bytes.saturating_mul(overflow_count) / sampled_count`,
multiplying in u64 before dividing. On very large disks the product exceeds
u64::MAX and saturates to a constant, so dividing by a sampled_count that grows
with disk size yields an estimate that monotonically shrinks — bigger disks
report less capacity. It only sets is_estimated, with no alert.
Extract a shared `estimate_overflow_bytes` helper that performs the
multiplication in u128, divides, then clamps to u64::MAX, so the intermediate
product can never overflow. It also guards division by zero with `.max(1)` even
though callers only enter the branch when sampled_count > 0.
Adds unit tests covering the realistic ~105 TB regression scenario (asserting
the inputs actually overflow u64 and the fixed estimate dwarfs the saturated
value), monotonic scaling across disk sizes, extreme-value clamping, small
non-overflowing inputs (unchanged), and the zero-sampled-count guard, all
compared against a u128 reference implementation.
Refs: https://github.com/rustfs/backlog/issues/1012
* fix(object-capacity): stop cancelled/remote-disk refreshes from corrupting capacity
Two independent capacity-refresh bugs (backlog rustfs/backlog#805):
- refresh_or_join set the singleflight `running` flag then awaited the
refresh future with no drop guard. When the admin request that became
leader was cancelled (client disconnect) mid-await, `running` stayed true
forever: joiners blocked indefinitely and the 120s scheduled refresh could
never start again. catch_unwind covered panics but not cancellation. A
RefreshLeaderGuard now resets the state and publishes an error on drop.
- capacity_disk_refs mapped the cluster-wide storage_info disk list without
filtering non-local disks, so admin-triggered refreshes ran a local WalkDir
over remote disks' drive_path. On multi-node clusters this double-counted
local bytes (shared mount layout) or hit NotFound (per-node layouts), and
poisoned the per-disk cache the scheduled local-only refresh depends on,
making the cached total oscillate. Filter to local disks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(rio): harden internode HTTP client build, cache, and PUT body integrity
Follow-ups from the internode HTTP review (backlog rustfs/backlog#805):
- handle_put_file accepted a truncated body as success: a HttpWriter dropped
mid-stream closes the chunked body cleanly, indistinguishable from EOF, and
the server never compared bytes copied against the declared size. Reject
size mismatches on the create path (append/unknown-size writes send size<=0
and are exempt).
- build_http_client used `.expect()` on ClientBuilder::build(), which runs
lazily on the first request and on every TLS generation bump (cert
rotation), so a build failure panicked a serving task. It now returns an
io::Error; get_http_client falls back to the previous TLS generation when a
rebuild fails instead of failing the request.
- CLIENT_CACHE was a tokio::Mutex taken on every stream open (data_shards
times per GET). Replaced with arc_swap::ArcSwapOption for lock-free reads on
the hot path; the generation-monotonic replacement guard is preserved via rcu.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Remove or consolidate 57 test cases that cannot catch regressions
(literal-constant asserts, construct-then-assert, derived-serde
round-trips, near-duplicate env/getter matrices) in common, config,
iam, madmin, and object-capacity, keeping all wire-format and
error-path guards. Add 13 tests for previously uncovered high-risk
behavior: filemeta version-sort determinism and merge resilience to
garbage headers, zip extraction path-traversal rejection and exact
limit boundaries, JWT tampered-signature rejection, and the bytes
variant of dual-key (rustfs/minio) metadata fallback and precedence.
Test-only change; no production code touched.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* perf(put): add eager path metrics and isolation tooling
* fix(decommission): persist progress adaptively (#3497)
Persist decommission progress after either the existing time interval or a migrated-item threshold, and flush progress baselines after bucket and terminal-state saves.
Also stabilize the OIDC discovery mock used by the pre-commit gate.
* refactor: move bucket operations contract (#3507)
* fix(s3): handle multipart flexible checksums (#3508)
* fix(io-core): avoid blocking on pooled buffer return
* perf(put): add slow inflight diagnostics
* perf(put): fix 16KiB regression with threshold and pool bypass
- Lower SMALL_EAGER_PUT_MAX_SIZE from 256KB to 8KB so objects >8KiB
use the streaming BufReader path (matches baseline behavior)
- Add POOL_BYPASE_MAX_SIZE (16KiB) to bypass BytesPool for very small
objects, avoiding Small-tier Mutex contention under high concurrency
- Add read_small_put_body_exact_direct() for direct Vec<u8> allocation
- Fix stale test assertions to match new 8KB threshold
Root cause analysis: the 16KiB regression was primarily caused by
instrumentation overhead in set_disk.rs (4x Instant::now() + metrics
per PUT), not BytesPool contention. Lowering the threshold eliminates
the eager-path overhead for 16KiB+ objects.
* perf(put): gate stage metrics behind observability flag
Add put_stage_metrics_enabled() AtomicBool switch in io-metrics crate.
When disabled (default), record_put_object_path() and
record_put_object_stage_duration() are no-ops, avoiding unnecessary
histogram/counter macro overhead in the PUT hot path.
The flag is set to true during startup when OTEL metric export is
enabled (rustfs_obs::observability_metric_enabled() == true).
This eliminates the per-request metrics overhead that contributed
to the 16KiB PUT regression when metrics collection is not active.
* perf(put): comprehensive optimization - restore eager path, cache env, remove UUID
Change 1: Restore SMALL_EAGER_PUT_MAX_SIZE from 8KB to 1MB
- The try_lock() fix (d13a189e3) eliminates the blocking that caused
service health timeouts under 512KiB c64 load
- Eager path with BytesPool is now safe for objects up to 1MB
- Recovers the eager path benefit for 32KiB-256KiB objects
Change 2: Adjust POOL_BYPASE_MAX_SIZE from 16KB to 4KB
- With eager path restored to 1MB, objects 4KB-1MB benefit from pool reuse
- Only ≤4KB objects bypass the pool (allocation cost negligible)
Change 3: Cache RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES via OnceLock
- Eliminates per-encode std::env::var() syscall
- Env var still works (read once at first use)
Change 4: Replace Uuid::new_v4() with Uuid::nil() in Erasure construction
- _id field is unused in hot paths (documented in code)
- Eliminates CSPRNG syscall per PUT request
Change 5: Add concurrency-aware buffer sizing to PUT path
- Reuses get_concurrency_aware_buffer_size() from GET path
- Reduces buffer size under high concurrency (0.4x at >8 concurrent)
- Lowers memory pressure for >1MB streaming PUTs
* chore: add pyroscope feature flag and clean up imports
- Add pyroscope feature flag forwarding to rustfs-obs
- Remove unused allow(non_upper_case_globals) in globals.rs
- Sort imports and fix Cargo.toml formatting consistency
* style: fix import ordering and code formatting
- Sort imports alphabetically in globals.rs, encode.rs
- Fix indentation in erasure_coding encode/erasure
- Clean up HashReader formatting in object_usecase.rs
* fix(test): use tokio::test for request_logging_layer tests
The tests call tokio::spawn via RequestContextLayer, which requires a
Tokio runtime. Changed from #[test] + futures::executor::block_on to
#[tokio::test] + .await, and replaced tracing::subscriber::with_default
with tracing::subscriber::set_default to support async.
* fix(bench): normalize no-space throughput/latency parsing in to_bps/to_ms
When a benchmark tool prints throughput without a separator (e.g. 123MiB/s),
awk '{print $2}' returns empty because the whole string is one field,
causing to_bps to return N/A and losing valid measurements in CSV output.
Insert a space between number and unit via sed before awk field splitting.
Same fix applied to to_ms for latency values like '50ms'.
Also add TODO comment on PUT path noting that get_concurrency_aware_buffer_size
reads ACTIVE_GET_REQUESTS instead of PUT concurrency (PR #3514 review).
Refs: PR #3514 review comments by chatgpt-codex-connector
* fix(metrics): correct POOL_BYPASS comments and separate PUT vs generic stage metrics
- Fix 3 comment-code mismatches: POOL_BYPASS_MAX_SIZE is 4KiB, not 16KiB
- Add generic record_stage_duration() with separate histogram
(rustfs_internal_stage_duration_ms) for non-PUT paths
- Replace record_put_object_stage_duration with record_stage_duration in
metacache_set, store_list_objects, and bucket_lifecycle_ops to avoid
polluting PUT-specific dashboards with listing/lifecycle timings
- Fix flaky test: serialize tests mutating PUT_STAGE_METRICS_ENABLED with
METRICS_FLAG_LOCK mutex and explicitly set desired state at test start
Refs: PR #3514 review comments by chatgpt-codex-connector
* style: apply cargo fmt to metacache_set.rs
---------
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Treat empty ping bodies as liveness probes, add a startup cleanup barrier for early walk_dir calls, and delay immediate background cleanup/capacity timers to reduce transient restart-time VolumeNotFound noise.
Also downgrade expected missing-path producer results during startup from generic errors to warnings while preserving existing storage semantics.
Zhengchao An