The GET body cache exports 11 `rustfs_object_data_cache_*` metrics but no
bundled dashboard visualized them. This adds one so operators can see the
cache's behaviour without hand-writing PromQL.
New `grafana-object-data-cache.json` (auto-provisioned from the dashboards
directory, `${DS_PROMETHEUS}`, schema 38) with 19 panels covering every metric:
hit ratio and lookup outcomes, plan decisions and cacheable ratio, fill
outcomes and fill-duration quantiles, hit-vs-fill byte throughput, entries and
weighted bytes vs capacity, in-flight fills, memory-pressure skips,
invalidations by reason/outcome, and size-class breakdowns. PromQL matches each
metric's type — rate() for counters, histogram_quantile() for the fill-duration
histogram, direct reads for gauges.
The observability README (EN + ZH) dashboard table gains a matching row.
Refs: backlog#1107
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(internode): P0 gRPC transport tuning, message limits, payload metrics
Land the P0 subtask from docs/grpc-optimization: close the client-vs-server
transport gaps and add instrumentation to size which unary RPCs need channel
isolation in P1.
Transport tuning (G3): the client `Endpoint` now disables Nagle and raises the
HTTP/2 stream/connection flow-control windows to mirror the server socket, so
small lock/health RPCs are not batched and larger metadata responses are not
throttled by the 64KiB default window. All env-overridable, 0 opts out.
Message-size limits (G1): both `NodeServiceClient` and `NodeServiceServer` set
max decode/encode size (default 100MiB) instead of tonic's silent 4MiB cap, so a
large multi-version xl.meta or aggregated ReadMultiple no longer fails
out_of_range. The server limit is set on `NodeServiceServer` before wrapping in
the auth `InterceptedService` (the interceptor type does not expose it).
Payload instrumentation (P1 prep): ReadAll/ReadMultiple record a payload-size
histogram plus a large-payload counter when a response crosses the configured
threshold (default 8MiB), feeding alerting on paths that contend with
latency-sensitive control-plane traffic on the shared channel. Threshold-only
counter, no per-call hot-path log.
Verification: cargo check/test on config, io-metrics, ecstore, rustfs; clippy
clean on touched files; make pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): P1 control/bulk gRPC channel isolation (opt-in)
Land the P1 subtask from docs/grpc-optimization: physically separate large
bytes-carrying unary RPCs from latency-sensitive control-plane RPCs so a big
transfer can no longer head-of-line block a lock/health RPC on the shared
HTTP/2 connection (G2/G5).
Introduce ChannelClass { Control, Bulk } and get_channel_for_class in protos.
Control RPCs keep the per-peer connection keyed by the bare address; Bulk RPCs
(ReadAll/WriteAll/ReadMultiple/BatchReadVersion, via a new get_bulk_client) are
round-robined across a small per-peer bulk pool.
Rather than restructuring the global GLOBAL_CONN_MAP (and every consumer), bulk
channels are cached under a composite key (addr\0bulk\0idx). The NUL separator
cannot appear in a URL, so bulk keys never collide with the control key. This
keeps the blast radius small on a consistency-sensitive path. create_new_channel
is refactored into build_channel(dial_addr, cache_key) so several physically
distinct channels to one peer cache independently while dialing/TLS still use
the real address.
Gated by RUSTFS_INTERNODE_CHANNEL_ISOLATION (default OFF) so the default build
is byte-for-byte the pre-P1 behavior: bulk resolves to the control channel and
the switch is a single-env rollback. RUSTFS_INTERNODE_BULK_CHANNELS (default 2,
clamped >=1) sizes the pool. On failure, evict_failed_connection drops the whole
bulk pool for the peer (round-robin hides which index was used), avoiding
half-dead cached channels.
Lock RPCs (remote_locker) already use the default Control path, so lock
semantics and retry behavior are unchanged.
Verification: cargo check/test on config, protos, ecstore, rustfs; new protos
tests for bulk key routing and isolation-off passthrough; clippy clean on
touched files; make pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): P2 msgpack/JSON codec observability + encode buffer presizing
Land the safe, wire-compatible slice of P2 from docs/grpc-optimization: the
observability prerequisite for retiring the redundant JSON fields, plus a codec
micro-optimization. No proto/wire-format change; JSON is still dual-written.
Internode RPCs today dual-encode each metadata value as both msgpack (`*_bin`)
and a JSON compatibility string, and decoders prefer `_bin` with a JSON
fallback. Before the JSON fields can ever be dropped (a cross-version change),
that fallback must be proven unused in production.
Add rustfs_system_network_internode_msgpack_json_fallback_total{direction,
message}: incremented whenever a decode falls back to the JSON field because the
msgpack payload was absent. Wired into both directions — the client decoding
peer responses (remote_disk.rs, incl. the list-level read_multiple/batch
fallbacks) and the server decoding peer requests (node_service/disk.rs). This
counter must read zero across a release window before send paths stop writing
JSON and the proto text fields are reserved/removed (the deferred P2-1 steps).
Also pre-size the msgpack encode buffers (Vec::with_capacity(512)) on both
sides, eliminating the repeated growth reallocations for typical FileInfo
payloads with zero added copy. Full thread_local buffer pooling is deferred: it
needs either an extra copy (unclear net win) or a send-path buffer-return
lifecycle, to be justified by a codec microbenchmark first.
Verification: cargo check/test on io-metrics, ecstore, rustfs; new fallback
counter smoke test; existing codec decode tests green; clippy clean on touched
files; make pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(internode): add msgpack/JSON convergence observation runbook
Runbook driving the observation-gated retirement of the redundant JSON
compatibility fields on internode gRPC metadata RPCs (grpc-optimization P2-1).
Documents the shipped fallback counter
(rustfs_system_network_internode_msgpack_json_fallback_total{direction,message}),
the PromQL to confirm it reads zero across a release window, a standing alert,
and the staged flip/rollback procedure (env-gated msgpack-only send, then proto
field removal in N+1).
Includes the verified field -> peer-decoder audit: only fields whose peer
decodes _bin first may be converged. Notes DeleteVersion.opts (DeleteOptions) is
NOT convergence-ready — its server handler is not _bin-first and must gain a
decode_msgpack_or_json path first. This gates the send-side change so it cannot
empty a JSON field an old peer still needs.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): env-gated msgpack-only send + DeleteVersion _bin support (P2-1)
Implements the send-side lever for retiring the redundant JSON compatibility
fields on internode gRPC metadata RPCs, plus the missing `_bin` support on the
delete path that it depends on (grpc-optimization P2-1). Default-off: the base
build is byte-for-byte the prior dual-write behavior.
Gated msgpack-only send (RUSTFS_INTERNODE_RPC_MSGPACK_ONLY, default false):
- New rustfs_protos::internode_rpc_msgpack_only() reads the flag.
- Client (remote_disk.rs) compat_json() and server (node_service/disk.rs)
compat_response_json() emit an empty JSON string when the flag is on, so only
the msgpack _bin payload is sent. The _bin field is always sent; decoders keep
the JSON read fallback. Applied only to fields with a confirmed _bin-first peer
decoder (WriteMetadata/UpdateMetadata/RenameData file_info, UpdateMetadata opts,
ReadOptions, ReadMultipleReq, BatchReadVersionReq; ReadVersion/ReadXL/RenameData
responses and the ReadMultiple/BatchReadVersion response lists).
- Only enable after the P2 fallback counter has read zero across a release window
(see docs/operations/internode-msgpack-json-convergence-runbook.md). Single-env
rollback; no wire-format break.
DeleteVersion(s) _bin support (prerequisite):
- The DeleteVersion/DeleteVersions protos had NO _bin fields. Add additive
(backward-compatible) bytes file_info_bin/opts_bin (DeleteVersion) and repeated
bytes versions_bin + bytes opts_bin (DeleteVersions); regenerate the checked-in
prost struct.
- Client dual-writes them; server decodes them _bin-first with JSON fallback.
- These delete fields are kept OUT of the msgpack-only set (always dual-write)
until their own fallback counter reads zero across a window with the new
decoders fully deployed. DeleteVersion.raw_file_info stays JSON-only (no _bin
field yet).
Verification: cargo check/test on protos, config, ecstore, rustfs (incl. the six
delete request handler tests and a compat_json default-path test); clippy clean
on touched files; make pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): P3 cluster peer online/offline health metric
Land the safe observability core of P3 (grpc-optimization G6/G8): track each
internode peer's reachability and expose the offline count, for parity with
MinIO's minio_cluster_servers_offline_total. Pure instrumentation — peer
selection and quorum are unchanged.
- io-metrics: per-peer PeerHealthState { online, consecutive_failures } registry
plus record_peer_reachable/record_peer_unreachable. A peer flips offline after
N consecutive failures (dial failures or RPC-triggered evictions) and back
online on the next successful dial; the count of offline peers is published to
the rustfs_cluster_servers_offline_total gauge.
- config: RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD (default 3, clamped >= 1).
- protos: build_channel marks the peer reachable on a successful dial and
unreachable on a dial failure; evict_failed_connection feeds the failure signal
too. Keyed by the real peer address, so control and bulk channels to one peer
share health state.
Deferred (documented in docs/grpc-optimization P3): startup prewarm (no clean
topology-ready hook yet), the offline fast-bypass in peer routing (consistency-
sensitive; must not change quorum), and idempotent-read-only retry. This commit
is observability only.
Verification: cargo check/test on io-metrics, config, protos (new peer-health
state-machine and threshold-clamp tests); clippy clean on touched files; make
pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): P3 control-channel prewarm + self-healing offline bypass
Add the remaining P3 connection-lifecycle levers (grpc-optimization G6/G8), both
env-gated and default-off so the base build is unchanged.
Prewarm (RUSTFS_INTERNODE_PREWARM, default off): RemoteDisk::new spawns a
best-effort background dial of the peer's control channel, deduped per peer
address, moving the connect cost off the first RPC. Failures fall through to the
existing lazy connect + recovery monitor.
Offline bypass (RUSTFS_INTERNODE_OFFLINE_BYPASS, default off): remote_disk
get_client/get_bulk_client fast-fail a peer already marked offline instead of
paying the connect timeout, so the erasure layer proceeds on quorum sooner. This
does NOT change quorum. It is self-healing: cluster_peer_should_bypass lets one
request per RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS (default 5s) through to recover
the peer even with no background monitor, and the recovery monitor's own probe
path calls the client directly so it is never bypassed.
io-metrics gains cluster_peer_is_offline / cluster_peer_should_bypass (with a
per-peer re-probe timestamp). Scope: data path only — remote_locker (lock RPCs,
most consistency-sensitive) is left dual-writing/unbypassed as a follow-up.
Verification: cargo check/test on io-metrics, config, ecstore (new self-healing
bypass tests; all 105 rpc tests green); clippy clean on touched files; make
pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(internode): add A/B benchmark runbook for gRPC optimization stages
Reproducible before/after collection procedure for grpc-optimization P0–P3.
Since every stage is env-gated, before/after is the same binary with different
env — no rebuild. Documents, per stage: the exact env toggles (baseline vs
enabled column), which existing bench script to run
(run_internode_transport_baseline.sh / run_four_node_cluster_failover_bench.sh),
the Prometheus metrics to capture, and the acceptance gates from the design docs
(e.g. lock p99 down >= 20% for P1, msgpack fallback counter = 0 before enabling
P2, correct rustfs_cluster_servers_offline_total for P3).
Live runs require a multi-node cluster + load tool + Prometheus scrape and cannot
be produced in a single-process sandbox; artifacts land under target/bench
(gitignored) and attach to the PR.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): P3-2 lock-path offline bypass + P3-3 idempotent read retry
Extend the offline bypass to the lock path and add opt-in retries for idempotent
reads (grpc-optimization P3-2/P3-3). Both env-gated and default-off/zero.
Offline bypass (lock path): factor the bypass decision into a shared pub(crate)
internode_offline_bypass_reason(addr) and call it from remote_locker::get_client
too, so lock RPCs to an offline peer fast-fail (letting dsync reach quorum
sooner) instead of paying the connect timeout. Does not change quorum; the
self-healing re-probe keeps peers recoverable. Gated by
RUSTFS_INTERNODE_OFFLINE_BYPASS (default off).
Idempotent read retry (P3-3): add execute_read_with_retry — a bounded,
exponential-backoff retry for read-only/reentrant RPCs on transient network
errors — and route disk_info through it. RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES
defaults to 0 (disabled). Write/lock RPCs are never retried (quorum/idempotency
safety, per CLAUDE.md); the wrapper requires an Fn closure so only reads that
rebuild their request from borrowed inputs qualify.
Deferred: grpc.health.v1 (optional ecosystem-compat only; needs a new
tonic-health dep and 3-way hybrid-service wiring — internal needs are met by the
existing Ping RPC).
Verification: cargo check/test on config, ecstore (105 rpc tests green incl.
disk_info now via the retry wrapper); clippy clean on touched files; make
pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(scripts): one-click internode gRPC A/B benchmark driver
Wrap the per-stage env matrix from the benchmark runbook into
scripts/run_internode_grpc_ab_bench.sh: given --stage <p0|p1|p2|p3> and --phase
<before|after>, it emits the stage/phase RUSTFS_INTERNODE_* server env to
<out-dir>/server-env.sh and runs the right underlying bench
(run_internode_transport_baseline.sh for p0/p1/p2, run_four_node_cluster_failover_bench.sh
for p3) into a labeled target/bench/internode-transport/<stage>-<phase>/.
Passthrough args after `--` reach the underlying bench; --dry-run previews the
env + command. The script is explicit that RUSTFS_INTERNODE_* are server env, so
for the load-driven stages the operator must restart rustfs with the emitted env
before the run; the docker four-node (p3) path exports them for a forwarding
compose. shellcheck-clean.
Runbook updated with a "One-click driver" section.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(compose): forward RUSTFS_INTERNODE_* into the four-node cluster
The four-node local-build compose only forwarded a fixed whitelist of env, so
the internode gRPC knobs (grpc-optimization P0-P3) never reached the containers
and the A/B bench driver's "after" phase was a no-op. Forward the full
RUSTFS_INTERNODE_* set with defaults matching the binary defaults, so leaving
them unset is a no-op and the A/B driver can toggle a stage per phase.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(internode): address Copilot review — retry health action + poison-safe peer health
Two review nits on #4337:
- P3-3 idempotent read retry (remote_disk.rs): execute_read_with_retry ran every
attempt through execute_with_timeout_for_op, which hardcodes
FailureHealthAction::MarkFailure. So the first transient error could flip the
disk faulty and short-circuit the remaining retries, and each attempt
over-counted the failure. Route all but the final attempt through
execute_with_timeout_for_op_and_health_action with IgnoreFailure; only the last
attempt marks faulty/evicts. No default impact (retries default 0).
- Peer-health helpers (io-metrics): record_peer_reachable/record_peer_unreachable,
cluster_peer_is_offline and cluster_peer_should_bypass early-returned on a
poisoned mutex, permanently stalling the offline gauge and bypass state after a
single panic. Recover via PoisonError::into_inner().
Verification: cargo check/test on io-metrics + ecstore (105 rpc tests green);
clippy clean on touched files; make pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(site-replication): Add Docker Compose setup for site replication testing
- Fixed the site replication flag issue and resolved the replication storm.
- Introduced a new directory for site replication tests with Docker Compose.
- Created `docker-compose.yml` to define three RustFS sites and a setup container.
- Added `README.md` to document the purpose, usage, and test flow of the replication setup.
- Implemented `run-object-flow-check.sh` script to verify object replication across sites.
- Configured health checks and volume permissions for the RustFS containers.
- Enabled customization of access keys, bucket names, and other parameters via environment variables.
* fix
* fix
* feat(ecstore): add object lock diagnostics
Add configurable namespace lock diagnostics for object operations so production contention can be traced by operation, owner, and key.
Wrap object read/write lock acquisition in diagnostic guards across get, head, put, delete, copy, and multipart flows, and log slow acquisition and long hold durations behind new RUSTFS_OBJECT_LOCK_DIAG_* settings.
Verification:
- make pre-commit
* feat(obs): expose object lock diagnostics metrics
Add Prometheus metrics and Grafana panels for object namespace lock diagnostics, covering slow acquire counts, slow hold counts, acquire duration, hold duration, and the diagnostics-enabled state.
Adopt PR review feedback by keeping diagnostic guards alive through the guarded operation so long-hold warnings and metrics are emitted, reusing shared env helpers, and reducing default-path overhead when diagnostics are disabled.
Verification:
- cargo check -p rustfs-ecstore -p rustfs-io-metrics
- cargo test -p rustfs-ecstore store::object -- --nocapture
- make pre-commit
* perf(obs): reduce object lock diag overhead
Avoid repeated environment parsing on hot object-lock paths by caching the diagnostics-enabled flag, and stop allocating label strings for object lock metrics by recording static labels directly.
Strengthen io-metrics tests by using a local recorder and asserting that the expected object lock diagnostic counters, gauges, and histograms are emitted.
Verification:
- cargo check -p rustfs-ecstore -p rustfs-io-metrics
- cargo test -p rustfs-io-metrics -- --nocapture
- make pre-commit
* fix(ecstore): harden issue3031 multipart validation path
- clear stale multipart part destinations before rename fan-out
- add repeated part overwrite regression coverage
- reduce remote disk startup false-fault escalation to suspect-first
- refine remote locker diagnostics and lower scanner leader-lock log noise
- add a dedicated 4-node issue3031 docker validation script
* refactor(admin): inline console version json macro
- drop the unused serde_json::json import in admin console
- call serde_json::json! inline in version_handler
- keep the console version response behavior unchanged
* fix(remote-disk): recover suspect health on probe success
- record probe success during remote disk health checks so suspect drives recover
- use async_with_vars for the remote disk health probe test
- make the missing-listener test assert the state transition more robustly
* Initial plan
* feat: add concurrency-aware buffer sizing and hot object caching for GetObject
- Implement adaptive buffer sizing based on concurrent request load
- Add per-request tracking with automatic cleanup using RAII guards
- Implement hot object cache (LRU) for frequently accessed small files (<= 10MB)
- Add disk I/O semaphore to prevent saturation under extreme load
- Integrate concurrency module into GetObject implementation
- Buffer sizes now adapt: low concurrency uses large buffers for throughput,
high concurrency uses smaller buffers for fairness and memory efficiency
- Add comprehensive metrics collection for monitoring performance
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* docs: add comprehensive documentation and tests for concurrent GetObject optimization
- Add detailed technical documentation explaining the solution
- Document root cause analysis and solution architecture
- Include performance expectations and testing recommendations
- Add integration tests for concurrency tracking and buffer sizing
- Add cache behavior tests
- Include benchmark tests for concurrent request handling
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix: address code review issues in concurrency module
- Fix race condition in cache size tracking by using consistent atomic operations within lock
- Correct buffer sizing logic: 1-2 requests use 100%, 3-4 use 75%, 5-8 use 50%, >8 use 40%
- Improve error message for semaphore acquire failure
- Document limitation of streaming cache implementation (not yet implemented)
- Add TODO for proper streaming cache with suggested approaches
- Update tests to match corrected buffer sizing thresholds
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* docs: add comprehensive implementation summary for concurrent GetObject optimization
- Executive summary of problem and solution
- Detailed architecture documentation for each optimization
- Integration points and code examples
- Complete testing strategy and performance validation steps
- Monitoring and observability guidelines with Prometheus queries
- Deployment considerations and rollout strategy
- Future enhancement roadmap
- Success criteria and quantitative metrics
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix
* fix
* refactor: optimize cache with lru 0.16.2 read-first pattern and add advanced features
- Implement optimized read-first cache access using peek() to reduce write lock contention
- Add batch cache operations: get_cached_batch() for efficient multi-object retrieval
- Add cache utility methods: is_cached(), remove_cached(), get_hot_keys()
- Implement warm_cache() for pre-populating cache on startup
- Add get_advanced_buffer_size() with file size and sequential read optimization
- Enhance test suite with 8 new comprehensive tests covering:
- Batch operations and cache warming
- Hot keys tracking and analysis
- Cache removal and LRU behavior verification
- Concurrent cache access performance
- Advanced buffer sizing strategies
- Improve documentation and code comments in English throughout
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* docs: add final comprehensive optimization summary
- Complete technical documentation of all optimizations
- Detailed performance analysis and metrics
- Production deployment guide with examples
- Comprehensive API reference and usage patterns
- Migration guide and future enhancement roadmap
- All documentation in professional English
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix
* fix
* add moka crate for cache
* feat: replace LRU with Moka cache and add comprehensive metrics
- Replace lru crate with moka 0.12.11 for superior concurrent performance
- Implement lock-free cache with automatic TTL/TTI expiration
- Add size-based eviction using Moka's weigher function
- Integrate comprehensive metrics collection throughout GetObject flow:
* Cache hit/miss tracking with per-key access counts
* Request concurrency gauges
* Disk permit wait time histograms
* Total request duration tracking
* Response size and buffer size histograms
- Deep integration with ecfs.rs GetObject operation
- Add hit rate calculation method
- Enhanced CacheStats with hit/miss counters
- Lock-free concurrent reads for better scalability
Moka advantages over LRU:
- True lock-free concurrent access
- Built-in TTL and TTI support
- Automatic size-based eviction
- Better performance under high concurrency
- Native async support
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* docs: add comprehensive Moka cache migration and metrics documentation
- Complete technical documentation of LRU to Moka migration
- Detailed performance comparison and benchmarks
- Comprehensive metrics catalog with 15+ Prometheus metrics
- Prometheus query examples for monitoring
- Dashboard and alerting recommendations
- Migration guide with code examples
- Troubleshooting guide for common issues
- Future enhancement roadmap
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix
* fix
* refactor: update tests for Moka cache implementation
- Completely refactor test suite to align with Moka-based concurrency.rs
- Add Clone derive to ConcurrencyManager for test convenience
- Update all tests to handle Moka's async behavior with proper delays
- Add new tests:
* test_cache_hit_rate - validate hit rate calculation
* test_ttl_expiration - verify TTL configuration
* test_is_cached_no_side_effects - ensure contains doesn't affect LRU
* bench_concurrent_cache_performance - benchmark concurrent access
- Updated existing tests:
* test_moka_cache_operations - renamed and updated for Moka API
* test_moka_cache_eviction - validate automatic eviction
* test_hot_keys_tracking - improved assertions for sorted results
* test_concurrent_cache_access - validate lock-free performance
- All tests now include appropriate sleep delays for Moka's async processing
- Enhanced documentation and assertions for better test clarity
- Total: 18 comprehensive integration tests
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* docs: add comprehensive Moka test suite documentation
- Complete test suite documentation for all 18 tests
- Detailed test patterns and best practices for Moka
- Debugging guide for common test failures
- Performance expectations and CI integration
- 100% API coverage validation
- Future enhancement roadmap
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix
* fmt
* docs: enhance code comments and test documentation throughout
- Significantly improve module-level documentation in concurrency.rs
- Add detailed explanations of performance characteristics and expected improvements
- Document the solution to issue #911 with before/after metrics
- Enhance constant documentation with purpose and impact explanations
- Add comprehensive RAII guard documentation with thread safety notes
- Improve test documentation with detailed test strategy and rationale
- Add Moka-specific test patterns and timing guidance
- Include performance expectations for each test category
- Document why each test matters for solving the original issue
- All documentation written in professional English
- Follow Rust documentation best practices with examples
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* remove lru crate
* upgrade version
* fix: resolve test errors by correcting module structure and test assertions
- Fix test import paths to use crate:: instead of rustfs:: (binary-only crate)
- Keep test file in src/storage/ instead of tests/ (no lib.rs exists)
- Add #[cfg(test)] guard to mod declaration in storage/mod.rs
- Fix Arc type annotations for Moka's ConcurrencyManager in concurrent tests
- Correct test_buffer_size_bounds assertions to match actual implementation:
* Minimum buffer is 32KB for files <100KB, 64KB otherwise
* Maximum buffer respects base_buffer_size when concurrency is low
* Buffer sizing doesn't cap at file size, only at min/max constraints
- All 17 integration tests now pass successfully
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix: modify `TimeoutLayer::new` to `TimeoutLayer::with_status_code` and improve docker health check
* fix
* feat: implement cache writeback for small objects in GetObject
- Add cache writeback logic for objects meeting caching criteria:
* No range/part request (full object retrieval)
* Object size known and <= 10MB (max_object_size threshold)
* Not encrypted (SSE-C or managed encryption)
- Read eligible objects into memory and cache via background task
- Serve response from in-memory data for immediate client response
- Add metrics counter for cache writeback operations
- Add 3 new tests for cache writeback functionality:
* test_cache_writeback_flow - validates round-trip caching
* test_cache_writeback_size_limit - ensures large objects aren't cached
* test_cache_writeback_concurrent - validates thread-safe concurrent writes
- Update test suite documentation (now 20 comprehensive tests)
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* improve code for const
* cargo clippy
* feat: add cache enable/disable configuration via environment variable
- Add is_cache_enabled() method to ConcurrencyManager
- Read RUSTFS_OBJECT_CACHE_ENABLE env var (default: false) at startup
- Update ecfs.rs to check is_cache_enabled() before cache lookup and writeback
- Cache lookup and writeback now respect the enable flag
- Add test_cache_enable_configuration test
- Constants already exist in rustfs_config:
* ENV_OBJECT_CACHE_ENABLE = "RUSTFS_OBJECT_CACHE_ENABLE"
* DEFAULT_OBJECT_CACHE_ENABLE = false
- Total: 21 comprehensive tests passing
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix
* fmt
* fix
* fix
* feat: implement comprehensive CachedGetObject response cache with metadata
- Add CachedGetObject struct with full response metadata fields:
* body, content_length, content_type, e_tag, last_modified
* expires, cache_control, content_disposition, content_encoding
* storage_class, version_id, delete_marker, tag_count, etc.
- Add dual cache architecture in HotObjectCache:
* Legacy simple byte cache for backward compatibility
* New response cache for complete GetObject responses
- Add ConcurrencyManager methods for response caching:
* get_cached_object() - retrieve cached response with metadata
* put_cached_object() - store complete response
* invalidate_cache() - invalidate on write operations
* invalidate_cache_versioned() - invalidate both version and latest
* make_cache_key() - generate cache keys with version support
* max_object_size() - get cache threshold
- Add builder pattern for CachedGetObject construction
- Add 6 new tests for response cache functionality (27 total):
* test_cached_get_object_basic - basic operations
* test_cached_get_object_versioned - version key handling
* test_cache_invalidation - write operation invalidation
* test_cache_invalidation_versioned - versioned invalidation
* test_cached_get_object_size_limit - size enforcement
* test_max_object_size - threshold accessor
All 27 tests pass successfully.
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* feat: integrate CachedGetObject cache in ecfs.rs with full metadata and cache invalidation
Integration of CachedGetObject response cache in ecfs.rs:
1. get_object: Cache lookup uses get_cached_object() with full metadata
- Returns complete response with e_tag, last_modified, content_type, etc.
- Parses last_modified from RFC3339 string
- Supports versioned cache keys via make_cache_key()
2. get_object: Cache writeback uses put_cached_object() with metadata
- Stores content_type, e_tag, last_modified in CachedGetObject
- Background writeback via tokio::spawn()
3. Cache invalidation added to write operations:
- put_object: invalidate_cache_versioned() after store.put_object()
- put_object_extract: invalidate_cache_versioned() after each file extraction
- copy_object: invalidate_cache_versioned() after store.copy_object()
- delete_object: invalidate_cache_versioned() after store.delete_object()
- delete_objects: invalidate_cache_versioned() for each deleted object
- complete_multipart_upload: invalidate_cache_versioned() after completion
4. Fixed test_adaptive_buffer_sizing to be more robust with parallel tests
All 27 tests pass.
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix: add error logging for time parsing in cache lookup and writeback
- Add warning log when RFC3339 parsing fails in cache lookup
- Add warning log when time formatting fails in cache writeback
- Improves debugging for cache-related issues
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix
* fix
* upgrade version
* fmt
* add http: 4138 and add max_object_size key
* fix
* fix
* fix
* modify metrics key
* add
* upgrade crates version and improve docker observability
* feat: implement adaptive I/O strategy based on disk permit wait time
Implements adaptive I/O strategy for GetObject operations based on disk permit
wait times, as requested in issue #911.
Key changes:
1. New types in concurrency.rs:
- IoLoadLevel enum: Low (<10ms), Medium (10-50ms), High (50-200ms), Critical (>200ms)
- IoStrategy struct: contains optimized I/O parameters based on load level
- IoLoadMetrics: rolling window metrics for load tracking
2. New ConcurrencyManager methods:
- calculate_io_strategy(): calculates adaptive strategy from permit wait duration
- record_permit_wait(): records observation for load tracking
- smoothed_load_level(): returns averaged load level for stability
- io_load_stats(): returns (avg_wait, p95_wait, max_wait, count) for monitoring
- adaptive_buffer_size(): convenience method for buffer sizing
3. Integration in ecfs.rs get_object:
- Calculate IoStrategy after acquiring disk permit
- Use strategy buffer_size instead of static sizing
- Consider strategy.cache_writeback_enabled in cache decision
- Record new metrics: io.load.level gauge, io.buffer.multiplier gauge,
io.strategy.selected counter by load level
4. New tests (4 added, 31 total):
- test_io_load_level_classification: validates load level thresholds
- test_io_strategy_buffer_sizing: validates buffer multipliers
- test_calculate_io_strategy: validates manager strategy calculation
- test_io_load_stats: validates stats tracking
All 31 tests pass. Clippy clean. Formatted.
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix
* fix
* docs: add comprehensive architecture design and fix cache hit event notification
1. Added CONCURRENCY_ARCHITECTURE.md with complete design documentation:
- Executive summary and problem statement
- Architecture overview with request flow diagram
- Detailed module analysis for concurrency.rs and ecfs.rs
- Critical analysis of helper.complete() for cache hits
- Adaptive I/O strategy design with algorithm
- Cache architecture with CachedGetObject structure
- Metrics and monitoring with Prometheus queries
- Performance characteristics and future enhancements
2. Fixed critical issue: Cache hit path now calls helper.complete()
- S3 bucket notifications (s3:GetObject events) now trigger for cache hits
- Event-driven workflows (Lambda, SNS) work correctly for all object access
- Maintains audit trail for both cache hits and misses
All 31 tests pass.
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix: set object info and version_id on helper before complete() for cache hits
When serving from cache, properly configure the OperationHelper before
calling complete() to ensure S3 bucket notifications include complete
object metadata:
1. Build ObjectInfo from cached metadata:
- bucket, name, size, actual_size
- etag, mod_time, version_id, delete_marker
- storage_class, content_type, content_encoding
- user_metadata (user_defined)
2. Set helper.object(event_info).version_id(version_id_str) before complete()
3. Updated CONCURRENCY_ARCHITECTURE.md with:
- Complete code example for cache hit event notification
- Explanation of why ObjectInfo is required
- Documentation of version_id handling
This ensures:
- Lambda triggers receive proper object metadata for cache hits
- SNS/SQS notifications include complete information
- Audit logs contain accurate object details
- Version-specific event routing works correctly
All 31 tests pass.
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix
* improve code
* fmt
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
* improve code for metrics
* improve code for metrics
* fix
* fix
* Refactor telemetry initialization and environment functions ordering
- Reorder functions in envs.rs by type size (8-bit to 64-bit, signed before unsigned) and add missing variants like get_env_opt_u16.
- Optimize init_telemetry to support three modes: stdout logging (default error level with span tracing), file rolling logs (size-based with retention), and HTTP-based observability with sub-endpoints (trace, metric, log) falling back to unified endpoint.
- Fix stdout logging issue by retaining WorkerGuard in OtelGuard to prevent premature release of async writer threads.
- Enhance observability mode with HTTP protocol, compression, and proper resource management.
- Update OtelGuard to include tracing_guard for stdout and flexi_logger_handles for file logging.
- Improve error handling and configuration extraction in OtelConfig.
* fix
* up
* fix
* fix
* improve code for obs
* fix
* fix
houseme