* feat(get): SF01 - bucket validation cache
Add 5s TTL cache for bucket validation to avoid repeated stat_volume()
calls on every GET request.
Changes:
- Add BUCKET_VALIDATED_CACHE (OnceLock + RwLock + HashMap)
- Add invalidate_bucket_validation_cache() for cache invalidation
- Add invalidate_all_bucket_validation_cache() for bulk invalidation
- Update get_validated_store() to use cache
- Add cache invalidation in execute_delete_bucket()
Expected impact: 3-5x improvement for small file GET latency.
Closesrustfs/backlog#766
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(get): SF03 - metadata cache TTL increase
Increase metadata cache TTL from 250ms to 2s and capacity from 1024
to 4096 entries.
Changes:
- GET_OBJECT_METADATA_CACHE_TTL: 250ms -> 2s
- GET_OBJECT_METADATA_CACHE_MAX_ENTRIES: 1024 -> 4096
All mutation paths already call invalidate_get_object_metadata_cache,
so the longer TTL is safe.
Expected impact: 10-50x improvement for hot objects.
Closesrustfs/backlog#768
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(get): SF04 - remove unnecessary tokio::spawn in metadata fanout
Replace tokio::spawn with direct async future in read_all_fileinfo_full_wait.
join_all already provides concurrency, so tokio::spawn adds unnecessary
task creation and scheduling overhead.
Changes:
- Remove tokio::spawn from metadata fanout futures
- Update result handling for direct future results
Expected impact: 16-32us reduction per GET request.
Closesrustfs/backlog#769
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(get): SF06 - conditional lifecycle check
Only call resolve_put_object_expiration when the object has an
x-amz-expiration metadata marker. This avoids unnecessary lifecycle
configuration reads on every GET request.
Expected impact: 50-100us reduction per GET request.
Closesrustfs/backlog#771
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(get): SF07 - conditional metrics recording
Gate hot path metrics behind get_stage_metrics_enabled() to reduce
overhead when metrics are not needed.
Changes:
- Conditional record_zero_copy_read
- Conditional manager.record_disk_operation
- Conditional manager.record_access
- Conditional manager.record_transfer
Expected impact: 20-50us reduction per GET request.
Closesrustfs/backlog#772
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(get): SF01 - use moka instead of dashmap for bucket cache
Replace OnceLock + RwLock + HashMap with moka::sync::Cache for bucket
validation cache. moka provides built-in TTL support and is already
available in the workspace.
Changes:
- Add moka dependency to rustfs crate
- Replace manual TTL management with moka's time_to_live
- Simplify cache operations
Closesrustfs/backlog#766
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(get): SF02 - inline data fast path
Add fast path for small inline objects that bypasses duplex pipe,
tokio::spawn, and bitrot reader creation when data is already in memory.
Changes:
- Add inline data detection before codec streaming gate
- Direct in-memory erasure decode for inline objects <= 128KB
- Add GET_OBJECT_PATH_INLINE_DIRECT metric path
- Skip duplex pipe and background task for inline data
Conditions for fast path:
- Single part object
- Inline data available
- Size <= 128KB
- Not encrypted/compressed/remote
- No range request
Expected impact: 2-3x improvement for small file GET latency.
Closesrustfs/backlog#767
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor: translate Chinese comments to English
Translate all Chinese comments to English in modified files:
- rustfs/src/storage/ecfs_extend.rs
- rustfs/src/app/bucket_usecase.rs
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix
* add
* fmt and improve import
* fmt
* feat(get): SF05 skip IO planning + refactor inline detection + adaptive bucket cache
SF05: Skip disk I/O semaphore for inline data fast path
- Reorder prepare_get_object_read_execution: read first, then decide semaphore
- Inline objects skip acquire_disk_read_permit() entirely (saves 100-200us)
- Add is_inline_fast_path field to GetObjectReadSetup
Refactor: Unify inline detection logic
- Add ObjectInfo::is_inline_fast_path_eligible() as single source of truth
- Version-aware thresholds: non-versioned 128KB, versioned 16KB (matches PUT)
- Eliminates divergent conditions between set_disk/mod.rs and object_usecase.rs
Refactor: Restore fault tolerance in metadata fanout
- Restore tokio::spawn + JoinError handling in read_all_fileinfo_full_wait
- Prevents single disk read panic from unwinding the entire operation
Refactor: Restore lifecycle check correctness
- Remove incorrect SF06 conditional that skipped lifecycle for most objects
- Always call resolve_put_object_expiration (original behavior)
Fix: make_bucket cache invalidation
- Invalidate bucket validation cache on create_bucket
Fix: erasure decode written validation
- Check decode() return value; error if 0 bytes written for non-empty object
Adaptive bucket cache
- Default: RwLock<HashMap> for < 100 buckets (low overhead)
- Opt-in: starshard::ShardedHashMap via RUSTFS_BUCKET_CACHE_STARSHARD=1
- 5s TTL with manual timestamp checking
Benchmark results (warp get, concurrency 32, 10s, 3 rounds):
- 10KiB: 25.10 MiB/s (+28.2% vs SF01-07)
- 100KiB: 221.81 MiB/s
- 1MiB: 1972.78 MiB/s
- vs main: -10% to -12% (inline path not triggered by warp)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(versioning): use read lock for versioning config query + five-expert analysis
P0 fix: BucketVersioningSys::get() was using write lock on
GLOBAL_BucketMetadataSys for a pure read operation. This serialized
all concurrent GET requests (3 write-lock acquisitions per request).
Changed to read lock — get_versioning_config() handles its own
internal locking via metadata_map RwLock.
Five-expert analysis identified top bottlenecks:
1. Versioning write lock (P0, fixed)
2. Inline fast path not triggered (P0, needs verification)
3. Metadata fanout no early-stop (P1, early-stop has bug, reverted)
4. Request-level versioning cache (P1, pending)
5. Duplex pipe for small objects (P2, pending)
Benchmark (read-lock fix, warp concurrency 32):
- 1KiB: 2.29 MiB/s (vs 2.53 before, within variance)
- 10KiB: 25.00 MiB/s (same as before)
- 100KiB: 246.72 MiB/s (+11% vs 221.81)
- 1MiB: 2039.95 MiB/s (+3% vs 1972.78)
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore: remove benchmark results from git, keep locally only
Remove docs/benchmark/*.md from version control.
Files remain on disk but are no longer tracked by git.
Added docs/benchmark/*.md to .gitignore.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(get): decode inline fast path through bitrot readers
---------
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): improve expect() messages in replication_resyncer
Replace unwrap() with expect() for better error diagnostics in
replication_resyncer.rs:
- HashMap get_mut calls now have descriptive expect messages
- format() calls now use unwrap_or_else for error handling
Refs https://github.com/rustfs/backlog/issues/729
fix(ecstore): replace k.unwrap() with safe pattern in bucket_target_sys
Replace unsafe k.unwrap().as_str() with if let Some(key_str) pattern
in 5 locations where HeaderMap iterator yields (Option<HeaderName>, Value).
This prevents potential panics if header names are invalid.
Refs https://github.com/rustfs/backlog/issues/729
* feat(replication): support insecure https bucket targets
* feat(replication): support custom ca bucket targets
* test(replication): satisfy e2e clippy for TLS helpers
* fix(replication): avoid native root panic for custom trust stores
* test(replication): decouple private IP target test from TLS roots
* test(replication): use target TLS client in private IP unit test
* refactor: move delete object contracts to storage api
* refactor: narrow store api compatibility exports
* refactor: route table catalog test through storage compat
* perf(put): add eager path metrics and isolation tooling
* fix(decommission): persist progress adaptively (#3497)
Persist decommission progress after either the existing time interval or a migrated-item threshold, and flush progress baselines after bucket and terminal-state saves.
Also stabilize the OIDC discovery mock used by the pre-commit gate.
* refactor: move bucket operations contract (#3507)
* fix(s3): handle multipart flexible checksums (#3508)
* fix(io-core): avoid blocking on pooled buffer return
* perf(put): add slow inflight diagnostics
* perf(put): fix 16KiB regression with threshold and pool bypass
- Lower SMALL_EAGER_PUT_MAX_SIZE from 256KB to 8KB so objects >8KiB
use the streaming BufReader path (matches baseline behavior)
- Add POOL_BYPASE_MAX_SIZE (16KiB) to bypass BytesPool for very small
objects, avoiding Small-tier Mutex contention under high concurrency
- Add read_small_put_body_exact_direct() for direct Vec<u8> allocation
- Fix stale test assertions to match new 8KB threshold
Root cause analysis: the 16KiB regression was primarily caused by
instrumentation overhead in set_disk.rs (4x Instant::now() + metrics
per PUT), not BytesPool contention. Lowering the threshold eliminates
the eager-path overhead for 16KiB+ objects.
* perf(put): gate stage metrics behind observability flag
Add put_stage_metrics_enabled() AtomicBool switch in io-metrics crate.
When disabled (default), record_put_object_path() and
record_put_object_stage_duration() are no-ops, avoiding unnecessary
histogram/counter macro overhead in the PUT hot path.
The flag is set to true during startup when OTEL metric export is
enabled (rustfs_obs::observability_metric_enabled() == true).
This eliminates the per-request metrics overhead that contributed
to the 16KiB PUT regression when metrics collection is not active.
* perf(put): comprehensive optimization - restore eager path, cache env, remove UUID
Change 1: Restore SMALL_EAGER_PUT_MAX_SIZE from 8KB to 1MB
- The try_lock() fix (d13a189e3) eliminates the blocking that caused
service health timeouts under 512KiB c64 load
- Eager path with BytesPool is now safe for objects up to 1MB
- Recovers the eager path benefit for 32KiB-256KiB objects
Change 2: Adjust POOL_BYPASE_MAX_SIZE from 16KB to 4KB
- With eager path restored to 1MB, objects 4KB-1MB benefit from pool reuse
- Only ≤4KB objects bypass the pool (allocation cost negligible)
Change 3: Cache RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES via OnceLock
- Eliminates per-encode std::env::var() syscall
- Env var still works (read once at first use)
Change 4: Replace Uuid::new_v4() with Uuid::nil() in Erasure construction
- _id field is unused in hot paths (documented in code)
- Eliminates CSPRNG syscall per PUT request
Change 5: Add concurrency-aware buffer sizing to PUT path
- Reuses get_concurrency_aware_buffer_size() from GET path
- Reduces buffer size under high concurrency (0.4x at >8 concurrent)
- Lowers memory pressure for >1MB streaming PUTs
* chore: add pyroscope feature flag and clean up imports
- Add pyroscope feature flag forwarding to rustfs-obs
- Remove unused allow(non_upper_case_globals) in globals.rs
- Sort imports and fix Cargo.toml formatting consistency
* style: fix import ordering and code formatting
- Sort imports alphabetically in globals.rs, encode.rs
- Fix indentation in erasure_coding encode/erasure
- Clean up HashReader formatting in object_usecase.rs
* fix(test): use tokio::test for request_logging_layer tests
The tests call tokio::spawn via RequestContextLayer, which requires a
Tokio runtime. Changed from #[test] + futures::executor::block_on to
#[tokio::test] + .await, and replaced tracing::subscriber::with_default
with tracing::subscriber::set_default to support async.
* fix(bench): normalize no-space throughput/latency parsing in to_bps/to_ms
When a benchmark tool prints throughput without a separator (e.g. 123MiB/s),
awk '{print $2}' returns empty because the whole string is one field,
causing to_bps to return N/A and losing valid measurements in CSV output.
Insert a space between number and unit via sed before awk field splitting.
Same fix applied to to_ms for latency values like '50ms'.
Also add TODO comment on PUT path noting that get_concurrency_aware_buffer_size
reads ACTIVE_GET_REQUESTS instead of PUT concurrency (PR #3514 review).
Refs: PR #3514 review comments by chatgpt-codex-connector
* fix(metrics): correct POOL_BYPASS comments and separate PUT vs generic stage metrics
- Fix 3 comment-code mismatches: POOL_BYPASS_MAX_SIZE is 4KiB, not 16KiB
- Add generic record_stage_duration() with separate histogram
(rustfs_internal_stage_duration_ms) for non-PUT paths
- Replace record_put_object_stage_duration with record_stage_duration in
metacache_set, store_list_objects, and bucket_lifecycle_ops to avoid
polluting PUT-specific dashboards with listing/lifecycle timings
- Fix flaky test: serialize tests mutating PUT_STAGE_METRICS_ENABLED with
METRICS_FLAG_LOCK mutex and explicitly set desired state at test start
Refs: PR #3514 review comments by chatgpt-codex-connector
* style: apply cargo fmt to metacache_set.rs
---------
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
* fix(bucket-repl): persist MRF retry queue to disk and reload on startup
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(bucket-repl): address three blocking MRF issues from review
1. MRF replay loses delete operations — add `MrfOpKind` discriminator to
`MrfReplicateEntry` (Object | Delete, default=Object for backward
compat). `DeletedObjectReplicationInfo::to_mrf_entry` now persists
`op=Delete`, `version_id`, `delete_marker_version_id`, and
`delete_marker`. `start_mrf_processor` branches on `op`: delete
entries skip `get_object_info` and replay via
`schedule_replication_delete` with `ReplicationType::Heal`; object
entries follow the existing heal path.
2. `flush_mrf_to_disk` cleared the in-memory batch even on encode/write
failure — changed return type to `bool` and callers now only
`pending.clear()` on `true`, so a transient storage error retries
on the next tick instead of silently dropping the batch.
3. Add focused tests: encode/decode roundtrips for object, delete-marker,
versioned-delete, and mixed-batch entries; a routing test confirming
op-kind propagates correctly and that the default is Object for
legacy files; a legacy-compat test verifying old entries round-trip
cleanly through the new format.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: fix clippy redundant-clone in MRF tests
Replace &[entry.clone()] with std::slice::from_ref(&entry) in two
encode_mrf_file call sites flagged by clippy's redundant_clone lint
under --all-targets --features rio-v2.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test(bucket-repl): strengthen legacy MRF compat test with hand-built msgpack
The previous mrf_legacy_file_without_op_field_decoded_as_object test
round-tripped through encode_mrf_file, so it exercised the new format
and never touched a truly-legacy payload.
Replace it with a hand-built msgpack payload that genuinely omits the
"op", "deleteMarker", and "deleteMarkerVersionID" keys — exactly what
the old binary would have written before MrfOpKind existed. The test
now fails if #[serde(default)] is removed from the op field, which
proves real backward compatibility rather than round-trip stability.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
Zhengchao An