The capacity scan's timeout fallback required the exact prefix
(threshold + sample_rate files, default ~200k) to be fully enumerated
before any sample existed. Cold-cache HDDs cannot stat that many files
inside the 15s budget, so large slow disks — exactly the disks sampling
was designed for — always timed out with sampled_count == 0, discarded
the accumulated exact-prefix work with a hard error, and re-ran 15s of
useless full-disk I/O every 120s (S03). When a sample did exist, the
estimate only extrapolated over files the walker had seen, silently
under-reporting disks whose tail was never reached (S10).
- Enter sampling early: once half the (possibly dynamic) time budget is
consumed and the exact prefix hasn't filled, freeze the threshold at
the current position so the remaining budget collects samples and the
timeout fallback is always reachable.
- Compensate the unseen tail: when the scan root is a dedicated mount
point (stat st_dev differs from the parent), take the max of the
seen-files extrapolation and the filesystem-level used bytes. On
shared filesystems (multi-disk dev layouts) statvfs would overcount,
so it is not trusted and behavior is unchanged.
- sampled_count == 0 at timeout no longer hard-fails when a dedicated
mount provides filesystem usage; the exact prefix is kept as a floor.
Without any estimator the original error still surfaces.
- Extract ScanLimits from env reading so the blocking scanner is
parameterizable in tests.
Ref: rustfs/backlog#1013 (S03+S10 from audit rustfs/backlog#1010)
A full refresh with partial disk failures used to commit the surviving
subset's sum as a fresh exact cluster total (no field carried the
partial-failure fact), while the complete disk cache kept the failed
disk's old value — so the reported capacity oscillated between the
partial sum and the cache-merged total on alternating refreshes.
- Add a degraded flag to CapacityUpdate/CachedCapacity, set when the
scan behind the update had partial errors; expose it in refresh logs,
admin capacity logs and a new rustfs_capacity_degraded_readings_total
counter.
- On a degraded full refresh, surface only the disks whose own scan
fully succeeded and merge them over a complete disk cache, so failed
disks keep their last-known values and the published total no longer
dips and bounces back. The disk cache is never replaced from a
degraded refresh.
- Without a complete cache, keep the partial sum (unchanged #805
non-pollution behavior) but mark the reading degraded.
Ref: rustfs/backlog#1014 (S06 from audit rustfs/backlog#1010)
A dirty-subset refresh scans only the dirty disks, so its raw
`CapacityUpdate` carries just that subset's `total_used`/`file_count`.
`update_capacity` recomputed the correct cluster-wide total into a local
variable and wrote it to the cache, but never wrote it back to the
`CapacityUpdate` that `refresh_or_join`/`spawn_refresh_if_needed` return
and publish to joiners. The admin blocking path (`resolve_admin_used_capacity`
-> `refresh_or_join_admin_disks(allow_dirty_subset=true)`) consumes
`update.total_used`, so a single dirty disk in an N-disk cluster made
admin StorageInfo report only the scanned subset's bytes (a large
transient undercount for that request and same-cycle joiners).
`CachedDiskCapacity` also dropped each disk's `file_count`/`is_estimated`,
so subset refreshes could not recompute a correct cluster file count and
would launder an estimated per-disk value into an exact cluster total.
Fix:
- `update_capacity` now reconciles `total_used`, `file_count`, and
`is_estimated` from the full per-disk cache and returns the corrected
`CapacityUpdate`.
- `CachedDiskCapacity` stores `file_count`/`is_estimated` per disk;
`file_count` sums the cache and `is_estimated` is the OR across disks.
- `refresh_or_join` and `spawn_refresh_if_needed` rebind their result to
the reconciled update so the leader return value and the value
published to joiners both carry cluster totals.
Tests: extend the subset-refresh test to assert the returned update's
reconciled `total_used`/`file_count`/`is_estimated`, and add a
`refresh_or_join` dirty-subset test asserting the leader returns the
merged cluster total (not the subset sum) and matches the cache.
Refs: https://github.com/rustfs/backlog/issues/1011
fix(object-capacity): use u128 intermediate to stop sampling estimate overflow (backlog#1012)
The three sampling extrapolation sites in scan.rs computed
`overflow_sampled_bytes.saturating_mul(overflow_count) / sampled_count`,
multiplying in u64 before dividing. On very large disks the product exceeds
u64::MAX and saturates to a constant, so dividing by a sampled_count that grows
with disk size yields an estimate that monotonically shrinks — bigger disks
report less capacity. It only sets is_estimated, with no alert.
Extract a shared `estimate_overflow_bytes` helper that performs the
multiplication in u128, divides, then clamps to u64::MAX, so the intermediate
product can never overflow. It also guards division by zero with `.max(1)` even
though callers only enter the branch when sampled_count > 0.
Adds unit tests covering the realistic ~105 TB regression scenario (asserting
the inputs actually overflow u64 and the fixed estimate dwarfs the saturated
value), monotonic scaling across disk sizes, extreme-value clamping, small
non-overflowing inputs (unchanged), and the zero-sampled-count guard, all
compared against a u128 reference implementation.
Refs: https://github.com/rustfs/backlog/issues/1012
* fix(object-capacity): stop cancelled/remote-disk refreshes from corrupting capacity
Two independent capacity-refresh bugs (backlog rustfs/backlog#805):
- refresh_or_join set the singleflight `running` flag then awaited the
refresh future with no drop guard. When the admin request that became
leader was cancelled (client disconnect) mid-await, `running` stayed true
forever: joiners blocked indefinitely and the 120s scheduled refresh could
never start again. catch_unwind covered panics but not cancellation. A
RefreshLeaderGuard now resets the state and publishes an error on drop.
- capacity_disk_refs mapped the cluster-wide storage_info disk list without
filtering non-local disks, so admin-triggered refreshes ran a local WalkDir
over remote disks' drive_path. On multi-node clusters this double-counted
local bytes (shared mount layout) or hit NotFound (per-node layouts), and
poisoned the per-disk cache the scheduled local-only refresh depends on,
making the cached total oscillate. Filter to local disks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(rio): harden internode HTTP client build, cache, and PUT body integrity
Follow-ups from the internode HTTP review (backlog rustfs/backlog#805):
- handle_put_file accepted a truncated body as success: a HttpWriter dropped
mid-stream closes the chunked body cleanly, indistinguishable from EOF, and
the server never compared bytes copied against the declared size. Reject
size mismatches on the create path (append/unknown-size writes send size<=0
and are exempt).
- build_http_client used `.expect()` on ClientBuilder::build(), which runs
lazily on the first request and on every TLS generation bump (cert
rotation), so a build failure panicked a serving task. It now returns an
io::Error; get_http_client falls back to the previous TLS generation when a
rebuild fails instead of failing the request.
- CLIENT_CACHE was a tokio::Mutex taken on every stream open (data_shards
times per GET). Replaced with arc_swap::ArcSwapOption for lock-free reads on
the hot path; the generation-monotonic replacement guard is preserved via rcu.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Remove or consolidate 57 test cases that cannot catch regressions
(literal-constant asserts, construct-then-assert, derived-serde
round-trips, near-duplicate env/getter matrices) in common, config,
iam, madmin, and object-capacity, keeping all wire-format and
error-path guards. Add 13 tests for previously uncovered high-risk
behavior: filemeta version-sort determinism and merge resilience to
garbage headers, zip extraction path-traversal rejection and exact
limit boundaries, JWT tampered-signature rejection, and the bytes
variant of dual-key (rustfs/minio) metadata fallback and precedence.
Test-only change; no production code touched.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* perf(put): add eager path metrics and isolation tooling
* fix(decommission): persist progress adaptively (#3497)
Persist decommission progress after either the existing time interval or a migrated-item threshold, and flush progress baselines after bucket and terminal-state saves.
Also stabilize the OIDC discovery mock used by the pre-commit gate.
* refactor: move bucket operations contract (#3507)
* fix(s3): handle multipart flexible checksums (#3508)
* fix(io-core): avoid blocking on pooled buffer return
* perf(put): add slow inflight diagnostics
* perf(put): fix 16KiB regression with threshold and pool bypass
- Lower SMALL_EAGER_PUT_MAX_SIZE from 256KB to 8KB so objects >8KiB
use the streaming BufReader path (matches baseline behavior)
- Add POOL_BYPASE_MAX_SIZE (16KiB) to bypass BytesPool for very small
objects, avoiding Small-tier Mutex contention under high concurrency
- Add read_small_put_body_exact_direct() for direct Vec<u8> allocation
- Fix stale test assertions to match new 8KB threshold
Root cause analysis: the 16KiB regression was primarily caused by
instrumentation overhead in set_disk.rs (4x Instant::now() + metrics
per PUT), not BytesPool contention. Lowering the threshold eliminates
the eager-path overhead for 16KiB+ objects.
* perf(put): gate stage metrics behind observability flag
Add put_stage_metrics_enabled() AtomicBool switch in io-metrics crate.
When disabled (default), record_put_object_path() and
record_put_object_stage_duration() are no-ops, avoiding unnecessary
histogram/counter macro overhead in the PUT hot path.
The flag is set to true during startup when OTEL metric export is
enabled (rustfs_obs::observability_metric_enabled() == true).
This eliminates the per-request metrics overhead that contributed
to the 16KiB PUT regression when metrics collection is not active.
* perf(put): comprehensive optimization - restore eager path, cache env, remove UUID
Change 1: Restore SMALL_EAGER_PUT_MAX_SIZE from 8KB to 1MB
- The try_lock() fix (d13a189e3) eliminates the blocking that caused
service health timeouts under 512KiB c64 load
- Eager path with BytesPool is now safe for objects up to 1MB
- Recovers the eager path benefit for 32KiB-256KiB objects
Change 2: Adjust POOL_BYPASE_MAX_SIZE from 16KB to 4KB
- With eager path restored to 1MB, objects 4KB-1MB benefit from pool reuse
- Only ≤4KB objects bypass the pool (allocation cost negligible)
Change 3: Cache RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES via OnceLock
- Eliminates per-encode std::env::var() syscall
- Env var still works (read once at first use)
Change 4: Replace Uuid::new_v4() with Uuid::nil() in Erasure construction
- _id field is unused in hot paths (documented in code)
- Eliminates CSPRNG syscall per PUT request
Change 5: Add concurrency-aware buffer sizing to PUT path
- Reuses get_concurrency_aware_buffer_size() from GET path
- Reduces buffer size under high concurrency (0.4x at >8 concurrent)
- Lowers memory pressure for >1MB streaming PUTs
* chore: add pyroscope feature flag and clean up imports
- Add pyroscope feature flag forwarding to rustfs-obs
- Remove unused allow(non_upper_case_globals) in globals.rs
- Sort imports and fix Cargo.toml formatting consistency
* style: fix import ordering and code formatting
- Sort imports alphabetically in globals.rs, encode.rs
- Fix indentation in erasure_coding encode/erasure
- Clean up HashReader formatting in object_usecase.rs
* fix(test): use tokio::test for request_logging_layer tests
The tests call tokio::spawn via RequestContextLayer, which requires a
Tokio runtime. Changed from #[test] + futures::executor::block_on to
#[tokio::test] + .await, and replaced tracing::subscriber::with_default
with tracing::subscriber::set_default to support async.
* fix(bench): normalize no-space throughput/latency parsing in to_bps/to_ms
When a benchmark tool prints throughput without a separator (e.g. 123MiB/s),
awk '{print $2}' returns empty because the whole string is one field,
causing to_bps to return N/A and losing valid measurements in CSV output.
Insert a space between number and unit via sed before awk field splitting.
Same fix applied to to_ms for latency values like '50ms'.
Also add TODO comment on PUT path noting that get_concurrency_aware_buffer_size
reads ACTIVE_GET_REQUESTS instead of PUT concurrency (PR #3514 review).
Refs: PR #3514 review comments by chatgpt-codex-connector
* fix(metrics): correct POOL_BYPASS comments and separate PUT vs generic stage metrics
- Fix 3 comment-code mismatches: POOL_BYPASS_MAX_SIZE is 4KiB, not 16KiB
- Add generic record_stage_duration() with separate histogram
(rustfs_internal_stage_duration_ms) for non-PUT paths
- Replace record_put_object_stage_duration with record_stage_duration in
metacache_set, store_list_objects, and bucket_lifecycle_ops to avoid
polluting PUT-specific dashboards with listing/lifecycle timings
- Fix flaky test: serialize tests mutating PUT_STAGE_METRICS_ENABLED with
METRICS_FLAG_LOCK mutex and explicitly set desired state at test start
Refs: PR #3514 review comments by chatgpt-codex-connector
* style: apply cargo fmt to metacache_set.rs
---------
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Treat empty ping bodies as liveness probes, add a startup cleanup barrier for early walk_dir calls, and delay immediate background cleanup/capacity timers to reduce transient restart-time VolumeNotFound noise.
Also downgrade expected missing-path producer results during startup from generic errors to warnings while preserving existing storage semantics.
Zhengchao An