* feat(get): SF01 - bucket validation cache
Add 5s TTL cache for bucket validation to avoid repeated stat_volume()
calls on every GET request.
Changes:
- Add BUCKET_VALIDATED_CACHE (OnceLock + RwLock + HashMap)
- Add invalidate_bucket_validation_cache() for cache invalidation
- Add invalidate_all_bucket_validation_cache() for bulk invalidation
- Update get_validated_store() to use cache
- Add cache invalidation in execute_delete_bucket()
Expected impact: 3-5x improvement for small file GET latency.
Closesrustfs/backlog#766
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(get): SF03 - metadata cache TTL increase
Increase metadata cache TTL from 250ms to 2s and capacity from 1024
to 4096 entries.
Changes:
- GET_OBJECT_METADATA_CACHE_TTL: 250ms -> 2s
- GET_OBJECT_METADATA_CACHE_MAX_ENTRIES: 1024 -> 4096
All mutation paths already call invalidate_get_object_metadata_cache,
so the longer TTL is safe.
Expected impact: 10-50x improvement for hot objects.
Closesrustfs/backlog#768
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(get): SF04 - remove unnecessary tokio::spawn in metadata fanout
Replace tokio::spawn with direct async future in read_all_fileinfo_full_wait.
join_all already provides concurrency, so tokio::spawn adds unnecessary
task creation and scheduling overhead.
Changes:
- Remove tokio::spawn from metadata fanout futures
- Update result handling for direct future results
Expected impact: 16-32us reduction per GET request.
Closesrustfs/backlog#769
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(get): SF06 - conditional lifecycle check
Only call resolve_put_object_expiration when the object has an
x-amz-expiration metadata marker. This avoids unnecessary lifecycle
configuration reads on every GET request.
Expected impact: 50-100us reduction per GET request.
Closesrustfs/backlog#771
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(get): SF07 - conditional metrics recording
Gate hot path metrics behind get_stage_metrics_enabled() to reduce
overhead when metrics are not needed.
Changes:
- Conditional record_zero_copy_read
- Conditional manager.record_disk_operation
- Conditional manager.record_access
- Conditional manager.record_transfer
Expected impact: 20-50us reduction per GET request.
Closesrustfs/backlog#772
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(get): SF01 - use moka instead of dashmap for bucket cache
Replace OnceLock + RwLock + HashMap with moka::sync::Cache for bucket
validation cache. moka provides built-in TTL support and is already
available in the workspace.
Changes:
- Add moka dependency to rustfs crate
- Replace manual TTL management with moka's time_to_live
- Simplify cache operations
Closesrustfs/backlog#766
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(get): SF02 - inline data fast path
Add fast path for small inline objects that bypasses duplex pipe,
tokio::spawn, and bitrot reader creation when data is already in memory.
Changes:
- Add inline data detection before codec streaming gate
- Direct in-memory erasure decode for inline objects <= 128KB
- Add GET_OBJECT_PATH_INLINE_DIRECT metric path
- Skip duplex pipe and background task for inline data
Conditions for fast path:
- Single part object
- Inline data available
- Size <= 128KB
- Not encrypted/compressed/remote
- No range request
Expected impact: 2-3x improvement for small file GET latency.
Closesrustfs/backlog#767
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor: translate Chinese comments to English
Translate all Chinese comments to English in modified files:
- rustfs/src/storage/ecfs_extend.rs
- rustfs/src/app/bucket_usecase.rs
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix
* add
* fmt and improve import
* fmt
* feat(get): SF05 skip IO planning + refactor inline detection + adaptive bucket cache
SF05: Skip disk I/O semaphore for inline data fast path
- Reorder prepare_get_object_read_execution: read first, then decide semaphore
- Inline objects skip acquire_disk_read_permit() entirely (saves 100-200us)
- Add is_inline_fast_path field to GetObjectReadSetup
Refactor: Unify inline detection logic
- Add ObjectInfo::is_inline_fast_path_eligible() as single source of truth
- Version-aware thresholds: non-versioned 128KB, versioned 16KB (matches PUT)
- Eliminates divergent conditions between set_disk/mod.rs and object_usecase.rs
Refactor: Restore fault tolerance in metadata fanout
- Restore tokio::spawn + JoinError handling in read_all_fileinfo_full_wait
- Prevents single disk read panic from unwinding the entire operation
Refactor: Restore lifecycle check correctness
- Remove incorrect SF06 conditional that skipped lifecycle for most objects
- Always call resolve_put_object_expiration (original behavior)
Fix: make_bucket cache invalidation
- Invalidate bucket validation cache on create_bucket
Fix: erasure decode written validation
- Check decode() return value; error if 0 bytes written for non-empty object
Adaptive bucket cache
- Default: RwLock<HashMap> for < 100 buckets (low overhead)
- Opt-in: starshard::ShardedHashMap via RUSTFS_BUCKET_CACHE_STARSHARD=1
- 5s TTL with manual timestamp checking
Benchmark results (warp get, concurrency 32, 10s, 3 rounds):
- 10KiB: 25.10 MiB/s (+28.2% vs SF01-07)
- 100KiB: 221.81 MiB/s
- 1MiB: 1972.78 MiB/s
- vs main: -10% to -12% (inline path not triggered by warp)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(versioning): use read lock for versioning config query + five-expert analysis
P0 fix: BucketVersioningSys::get() was using write lock on
GLOBAL_BucketMetadataSys for a pure read operation. This serialized
all concurrent GET requests (3 write-lock acquisitions per request).
Changed to read lock — get_versioning_config() handles its own
internal locking via metadata_map RwLock.
Five-expert analysis identified top bottlenecks:
1. Versioning write lock (P0, fixed)
2. Inline fast path not triggered (P0, needs verification)
3. Metadata fanout no early-stop (P1, early-stop has bug, reverted)
4. Request-level versioning cache (P1, pending)
5. Duplex pipe for small objects (P2, pending)
Benchmark (read-lock fix, warp concurrency 32):
- 1KiB: 2.29 MiB/s (vs 2.53 before, within variance)
- 10KiB: 25.00 MiB/s (same as before)
- 100KiB: 246.72 MiB/s (+11% vs 221.81)
- 1MiB: 2039.95 MiB/s (+3% vs 1972.78)
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore: remove benchmark results from git, keep locally only
Remove docs/benchmark/*.md from version control.
Files remain on disk but are no longer tracked by git.
Added docs/benchmark/*.md to .gitignore.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(get): decode inline fast path through bitrot readers
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* perf(put): add eager path metrics and isolation tooling
* fix(decommission): persist progress adaptively (#3497)
Persist decommission progress after either the existing time interval or a migrated-item threshold, and flush progress baselines after bucket and terminal-state saves.
Also stabilize the OIDC discovery mock used by the pre-commit gate.
* refactor: move bucket operations contract (#3507)
* fix(s3): handle multipart flexible checksums (#3508)
* fix(io-core): avoid blocking on pooled buffer return
* perf(put): add slow inflight diagnostics
* perf(put): fix 16KiB regression with threshold and pool bypass
- Lower SMALL_EAGER_PUT_MAX_SIZE from 256KB to 8KB so objects >8KiB
use the streaming BufReader path (matches baseline behavior)
- Add POOL_BYPASE_MAX_SIZE (16KiB) to bypass BytesPool for very small
objects, avoiding Small-tier Mutex contention under high concurrency
- Add read_small_put_body_exact_direct() for direct Vec<u8> allocation
- Fix stale test assertions to match new 8KB threshold
Root cause analysis: the 16KiB regression was primarily caused by
instrumentation overhead in set_disk.rs (4x Instant::now() + metrics
per PUT), not BytesPool contention. Lowering the threshold eliminates
the eager-path overhead for 16KiB+ objects.
* perf(put): gate stage metrics behind observability flag
Add put_stage_metrics_enabled() AtomicBool switch in io-metrics crate.
When disabled (default), record_put_object_path() and
record_put_object_stage_duration() are no-ops, avoiding unnecessary
histogram/counter macro overhead in the PUT hot path.
The flag is set to true during startup when OTEL metric export is
enabled (rustfs_obs::observability_metric_enabled() == true).
This eliminates the per-request metrics overhead that contributed
to the 16KiB PUT regression when metrics collection is not active.
* perf(put): comprehensive optimization - restore eager path, cache env, remove UUID
Change 1: Restore SMALL_EAGER_PUT_MAX_SIZE from 8KB to 1MB
- The try_lock() fix (d13a189e3) eliminates the blocking that caused
service health timeouts under 512KiB c64 load
- Eager path with BytesPool is now safe for objects up to 1MB
- Recovers the eager path benefit for 32KiB-256KiB objects
Change 2: Adjust POOL_BYPASE_MAX_SIZE from 16KB to 4KB
- With eager path restored to 1MB, objects 4KB-1MB benefit from pool reuse
- Only ≤4KB objects bypass the pool (allocation cost negligible)
Change 3: Cache RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES via OnceLock
- Eliminates per-encode std::env::var() syscall
- Env var still works (read once at first use)
Change 4: Replace Uuid::new_v4() with Uuid::nil() in Erasure construction
- _id field is unused in hot paths (documented in code)
- Eliminates CSPRNG syscall per PUT request
Change 5: Add concurrency-aware buffer sizing to PUT path
- Reuses get_concurrency_aware_buffer_size() from GET path
- Reduces buffer size under high concurrency (0.4x at >8 concurrent)
- Lowers memory pressure for >1MB streaming PUTs
* chore: add pyroscope feature flag and clean up imports
- Add pyroscope feature flag forwarding to rustfs-obs
- Remove unused allow(non_upper_case_globals) in globals.rs
- Sort imports and fix Cargo.toml formatting consistency
* style: fix import ordering and code formatting
- Sort imports alphabetically in globals.rs, encode.rs
- Fix indentation in erasure_coding encode/erasure
- Clean up HashReader formatting in object_usecase.rs
* fix(test): use tokio::test for request_logging_layer tests
The tests call tokio::spawn via RequestContextLayer, which requires a
Tokio runtime. Changed from #[test] + futures::executor::block_on to
#[tokio::test] + .await, and replaced tracing::subscriber::with_default
with tracing::subscriber::set_default to support async.
* fix(bench): normalize no-space throughput/latency parsing in to_bps/to_ms
When a benchmark tool prints throughput without a separator (e.g. 123MiB/s),
awk '{print $2}' returns empty because the whole string is one field,
causing to_bps to return N/A and losing valid measurements in CSV output.
Insert a space between number and unit via sed before awk field splitting.
Same fix applied to to_ms for latency values like '50ms'.
Also add TODO comment on PUT path noting that get_concurrency_aware_buffer_size
reads ACTIVE_GET_REQUESTS instead of PUT concurrency (PR #3514 review).
Refs: PR #3514 review comments by chatgpt-codex-connector
* fix(metrics): correct POOL_BYPASS comments and separate PUT vs generic stage metrics
- Fix 3 comment-code mismatches: POOL_BYPASS_MAX_SIZE is 4KiB, not 16KiB
- Add generic record_stage_duration() with separate histogram
(rustfs_internal_stage_duration_ms) for non-PUT paths
- Replace record_put_object_stage_duration with record_stage_duration in
metacache_set, store_list_objects, and bucket_lifecycle_ops to avoid
polluting PUT-specific dashboards with listing/lifecycle timings
- Fix flaky test: serialize tests mutating PUT_STAGE_METRICS_ENABLED with
METRICS_FLAG_LOCK mutex and explicitly set desired state at test start
Refs: PR #3514 review comments by chatgpt-codex-connector
* style: apply cargo fmt to metacache_set.rs
---------
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
The session watchdog now selects its detection method per
platform. It previously probed kernel TCP state through a
Linux-only procfs path, so on macOS every healthy idle session
was killed within a minute, and on Windows the watchdog never
spawned at all, leaving wedged sessions with no cleanup. Linux
keeps its fast-kill watchdog unchanged. Other platforms get a
silence-only backstop that kills a session only at the
documented 30-minute idle ceiling.
The host-key loader now has a Windows arm. It loads OpenSSH
format host keys from the configured directory and logs a
one-time warning to restrict NTFS ACLs on the key directory,
the same operator-managed approach FTPS, WebDAV, KMS, and IAM
already use on Windows. Startup previously aborted with
UnsupportedPlatform because the Unix mode-bit permission check
has no Windows equivalent. tokio's io-uring feature is now
enabled only in Linux builds. io-uring is a Linux kernel
interface and enabling it unconditionally broke the Windows
build.
Co-authored-by: houseme <housemecn@gmail.com>
* fix(allocator): restore validated jemalloc target gating
Restrict the global allocator and jemalloc profiling paths to linux-gnu-x86_64 so Linux ARM64 builds fall back to mimalloc again.
Update the runtime profiling, allocator reclaim, and admin profiling handlers to use the same target gating and avoid exposing jemalloc-only code on unsupported targets.
Verification:
- cargo fmt --all
- cargo fmt --all --check
- cargo check -p rustfs
- make pre-commit
* fix(profiling): restore non-jemalloc platform behavior
Restore CPU profiling support on macOS while keeping jemalloc-backed memory profiling restricted to validated linux-gnu-x86_64 targets.
Also restore Unix log directory permission hardening so the allocator regression fix does not roll back unrelated observability behavior on other supported platforms.
Verification in progress:
- cargo fmt --all
- cargo check -p rustfs
- make pre-commit (running)
* fix(profiling): qualify periodic memory log macros
Use explicit tracing macro paths in periodic jemalloc memory profiling logging so builds do not fail when the local target configuration excludes those branches from the current import set.
Verification:
- cargo fmt --all
- cargo check -p rustfs
* fix(profiling): gate unsupported memory helper
Restrict the unsupported memory profiling helper to non-linux-gnu-x86_64 targets so Linux builds do not emit dead_code warnings for an unreachable fallback.
Verification:
- cargo fmt --all
- cargo check -p rustfs
* chore(deps): refresh workspace deps and linux fs_type gating
- refresh workspace dependency pins and lockfile updates
- remove now-unused crate dependency entries in multiple Cargo.toml files
- enable profiling export defaults in config and scripts/run.sh
- gate os::fs_type module/function/tests to Linux to avoid non-Linux dead_code warnings
* fix(utils): simplify fs_type linux gating
- keep fs_type module-level linux cfg in os::mod
- remove redundant linux cfg on get_fs_type and test module
* chore(deps): bump s3s git revision
- update workspace s3s dependency to rev 507e1312b211c3ddc214b03875d6fabd15d22ed5
- refresh Cargo.lock source entry for s3s
* chore(dev): allow mysql_async git source and env overrides
- allow mysql_async git source in deny.toml allow-git list
- make scripts/run.sh core env vars overrideable via existing shell env
* fix(utils): import get_fs_type in fs_type tests
- add explicit super::get_fs_type import in fs_type test module
- fix Linux E0425 unresolved function errors in unit tests
* chore(dev): tune run script observability defaults
- make profiling export env overrideable in scripts/run.sh
- set RUSTFS_OBS_SAMPLE_RATIO default from 2.0 to 1.0
- update allow-git review window comments in deny.toml
* test(obs): stabilize profiling env alias tests
Zhengchao An