Add a Grafana dashboard for the four KMS backend operation metrics
emitted at the policy choke point, Prometheus alert rules with
conservative default thresholds (pending staging baseline calibration),
and an operations runbook documenting the metric contract and the
response procedure for each alert. Register the new alert rules file in
the observability READMEs.
Refs rustfs/backlog#1584 (part of rustfs/backlog#1562)
The GET body cache exports 11 `rustfs_object_data_cache_*` metrics but no
bundled dashboard visualized them. This adds one so operators can see the
cache's behaviour without hand-writing PromQL.
New `grafana-object-data-cache.json` (auto-provisioned from the dashboards
directory, `${DS_PROMETHEUS}`, schema 38) with 19 panels covering every metric:
hit ratio and lookup outcomes, plan decisions and cacheable ratio, fill
outcomes and fill-duration quantiles, hit-vs-fill byte throughput, entries and
weighted bytes vs capacity, in-flight fills, memory-pressure skips,
invalidations by reason/outcome, and size-class breakdowns. PromQL matches each
metric's type — rate() for counters, histogram_quantile() for the fill-duration
histogram, direct reads for gauges.
The observability README (EN + ZH) dashboard table gains a matching row.
Refs: backlog#1107
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(internode): P0 gRPC transport tuning, message limits, payload metrics
Land the P0 subtask from docs/grpc-optimization: close the client-vs-server
transport gaps and add instrumentation to size which unary RPCs need channel
isolation in P1.
Transport tuning (G3): the client `Endpoint` now disables Nagle and raises the
HTTP/2 stream/connection flow-control windows to mirror the server socket, so
small lock/health RPCs are not batched and larger metadata responses are not
throttled by the 64KiB default window. All env-overridable, 0 opts out.
Message-size limits (G1): both `NodeServiceClient` and `NodeServiceServer` set
max decode/encode size (default 100MiB) instead of tonic's silent 4MiB cap, so a
large multi-version xl.meta or aggregated ReadMultiple no longer fails
out_of_range. The server limit is set on `NodeServiceServer` before wrapping in
the auth `InterceptedService` (the interceptor type does not expose it).
Payload instrumentation (P1 prep): ReadAll/ReadMultiple record a payload-size
histogram plus a large-payload counter when a response crosses the configured
threshold (default 8MiB), feeding alerting on paths that contend with
latency-sensitive control-plane traffic on the shared channel. Threshold-only
counter, no per-call hot-path log.
Verification: cargo check/test on config, io-metrics, ecstore, rustfs; clippy
clean on touched files; make pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): P1 control/bulk gRPC channel isolation (opt-in)
Land the P1 subtask from docs/grpc-optimization: physically separate large
bytes-carrying unary RPCs from latency-sensitive control-plane RPCs so a big
transfer can no longer head-of-line block a lock/health RPC on the shared
HTTP/2 connection (G2/G5).
Introduce ChannelClass { Control, Bulk } and get_channel_for_class in protos.
Control RPCs keep the per-peer connection keyed by the bare address; Bulk RPCs
(ReadAll/WriteAll/ReadMultiple/BatchReadVersion, via a new get_bulk_client) are
round-robined across a small per-peer bulk pool.
Rather than restructuring the global GLOBAL_CONN_MAP (and every consumer), bulk
channels are cached under a composite key (addr\0bulk\0idx). The NUL separator
cannot appear in a URL, so bulk keys never collide with the control key. This
keeps the blast radius small on a consistency-sensitive path. create_new_channel
is refactored into build_channel(dial_addr, cache_key) so several physically
distinct channels to one peer cache independently while dialing/TLS still use
the real address.
Gated by RUSTFS_INTERNODE_CHANNEL_ISOLATION (default OFF) so the default build
is byte-for-byte the pre-P1 behavior: bulk resolves to the control channel and
the switch is a single-env rollback. RUSTFS_INTERNODE_BULK_CHANNELS (default 2,
clamped >=1) sizes the pool. On failure, evict_failed_connection drops the whole
bulk pool for the peer (round-robin hides which index was used), avoiding
half-dead cached channels.
Lock RPCs (remote_locker) already use the default Control path, so lock
semantics and retry behavior are unchanged.
Verification: cargo check/test on config, protos, ecstore, rustfs; new protos
tests for bulk key routing and isolation-off passthrough; clippy clean on
touched files; make pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): P2 msgpack/JSON codec observability + encode buffer presizing
Land the safe, wire-compatible slice of P2 from docs/grpc-optimization: the
observability prerequisite for retiring the redundant JSON fields, plus a codec
micro-optimization. No proto/wire-format change; JSON is still dual-written.
Internode RPCs today dual-encode each metadata value as both msgpack (`*_bin`)
and a JSON compatibility string, and decoders prefer `_bin` with a JSON
fallback. Before the JSON fields can ever be dropped (a cross-version change),
that fallback must be proven unused in production.
Add rustfs_system_network_internode_msgpack_json_fallback_total{direction,
message}: incremented whenever a decode falls back to the JSON field because the
msgpack payload was absent. Wired into both directions — the client decoding
peer responses (remote_disk.rs, incl. the list-level read_multiple/batch
fallbacks) and the server decoding peer requests (node_service/disk.rs). This
counter must read zero across a release window before send paths stop writing
JSON and the proto text fields are reserved/removed (the deferred P2-1 steps).
Also pre-size the msgpack encode buffers (Vec::with_capacity(512)) on both
sides, eliminating the repeated growth reallocations for typical FileInfo
payloads with zero added copy. Full thread_local buffer pooling is deferred: it
needs either an extra copy (unclear net win) or a send-path buffer-return
lifecycle, to be justified by a codec microbenchmark first.
Verification: cargo check/test on io-metrics, ecstore, rustfs; new fallback
counter smoke test; existing codec decode tests green; clippy clean on touched
files; make pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(internode): add msgpack/JSON convergence observation runbook
Runbook driving the observation-gated retirement of the redundant JSON
compatibility fields on internode gRPC metadata RPCs (grpc-optimization P2-1).
Documents the shipped fallback counter
(rustfs_system_network_internode_msgpack_json_fallback_total{direction,message}),
the PromQL to confirm it reads zero across a release window, a standing alert,
and the staged flip/rollback procedure (env-gated msgpack-only send, then proto
field removal in N+1).
Includes the verified field -> peer-decoder audit: only fields whose peer
decodes _bin first may be converged. Notes DeleteVersion.opts (DeleteOptions) is
NOT convergence-ready — its server handler is not _bin-first and must gain a
decode_msgpack_or_json path first. This gates the send-side change so it cannot
empty a JSON field an old peer still needs.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): env-gated msgpack-only send + DeleteVersion _bin support (P2-1)
Implements the send-side lever for retiring the redundant JSON compatibility
fields on internode gRPC metadata RPCs, plus the missing `_bin` support on the
delete path that it depends on (grpc-optimization P2-1). Default-off: the base
build is byte-for-byte the prior dual-write behavior.
Gated msgpack-only send (RUSTFS_INTERNODE_RPC_MSGPACK_ONLY, default false):
- New rustfs_protos::internode_rpc_msgpack_only() reads the flag.
- Client (remote_disk.rs) compat_json() and server (node_service/disk.rs)
compat_response_json() emit an empty JSON string when the flag is on, so only
the msgpack _bin payload is sent. The _bin field is always sent; decoders keep
the JSON read fallback. Applied only to fields with a confirmed _bin-first peer
decoder (WriteMetadata/UpdateMetadata/RenameData file_info, UpdateMetadata opts,
ReadOptions, ReadMultipleReq, BatchReadVersionReq; ReadVersion/ReadXL/RenameData
responses and the ReadMultiple/BatchReadVersion response lists).
- Only enable after the P2 fallback counter has read zero across a release window
(see docs/operations/internode-msgpack-json-convergence-runbook.md). Single-env
rollback; no wire-format break.
DeleteVersion(s) _bin support (prerequisite):
- The DeleteVersion/DeleteVersions protos had NO _bin fields. Add additive
(backward-compatible) bytes file_info_bin/opts_bin (DeleteVersion) and repeated
bytes versions_bin + bytes opts_bin (DeleteVersions); regenerate the checked-in
prost struct.
- Client dual-writes them; server decodes them _bin-first with JSON fallback.
- These delete fields are kept OUT of the msgpack-only set (always dual-write)
until their own fallback counter reads zero across a window with the new
decoders fully deployed. DeleteVersion.raw_file_info stays JSON-only (no _bin
field yet).
Verification: cargo check/test on protos, config, ecstore, rustfs (incl. the six
delete request handler tests and a compat_json default-path test); clippy clean
on touched files; make pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): P3 cluster peer online/offline health metric
Land the safe observability core of P3 (grpc-optimization G6/G8): track each
internode peer's reachability and expose the offline count, for parity with
MinIO's minio_cluster_servers_offline_total. Pure instrumentation — peer
selection and quorum are unchanged.
- io-metrics: per-peer PeerHealthState { online, consecutive_failures } registry
plus record_peer_reachable/record_peer_unreachable. A peer flips offline after
N consecutive failures (dial failures or RPC-triggered evictions) and back
online on the next successful dial; the count of offline peers is published to
the rustfs_cluster_servers_offline_total gauge.
- config: RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD (default 3, clamped >= 1).
- protos: build_channel marks the peer reachable on a successful dial and
unreachable on a dial failure; evict_failed_connection feeds the failure signal
too. Keyed by the real peer address, so control and bulk channels to one peer
share health state.
Deferred (documented in docs/grpc-optimization P3): startup prewarm (no clean
topology-ready hook yet), the offline fast-bypass in peer routing (consistency-
sensitive; must not change quorum), and idempotent-read-only retry. This commit
is observability only.
Verification: cargo check/test on io-metrics, config, protos (new peer-health
state-machine and threshold-clamp tests); clippy clean on touched files; make
pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): P3 control-channel prewarm + self-healing offline bypass
Add the remaining P3 connection-lifecycle levers (grpc-optimization G6/G8), both
env-gated and default-off so the base build is unchanged.
Prewarm (RUSTFS_INTERNODE_PREWARM, default off): RemoteDisk::new spawns a
best-effort background dial of the peer's control channel, deduped per peer
address, moving the connect cost off the first RPC. Failures fall through to the
existing lazy connect + recovery monitor.
Offline bypass (RUSTFS_INTERNODE_OFFLINE_BYPASS, default off): remote_disk
get_client/get_bulk_client fast-fail a peer already marked offline instead of
paying the connect timeout, so the erasure layer proceeds on quorum sooner. This
does NOT change quorum. It is self-healing: cluster_peer_should_bypass lets one
request per RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS (default 5s) through to recover
the peer even with no background monitor, and the recovery monitor's own probe
path calls the client directly so it is never bypassed.
io-metrics gains cluster_peer_is_offline / cluster_peer_should_bypass (with a
per-peer re-probe timestamp). Scope: data path only — remote_locker (lock RPCs,
most consistency-sensitive) is left dual-writing/unbypassed as a follow-up.
Verification: cargo check/test on io-metrics, config, ecstore (new self-healing
bypass tests; all 105 rpc tests green); clippy clean on touched files; make
pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(internode): add A/B benchmark runbook for gRPC optimization stages
Reproducible before/after collection procedure for grpc-optimization P0–P3.
Since every stage is env-gated, before/after is the same binary with different
env — no rebuild. Documents, per stage: the exact env toggles (baseline vs
enabled column), which existing bench script to run
(run_internode_transport_baseline.sh / run_four_node_cluster_failover_bench.sh),
the Prometheus metrics to capture, and the acceptance gates from the design docs
(e.g. lock p99 down >= 20% for P1, msgpack fallback counter = 0 before enabling
P2, correct rustfs_cluster_servers_offline_total for P3).
Live runs require a multi-node cluster + load tool + Prometheus scrape and cannot
be produced in a single-process sandbox; artifacts land under target/bench
(gitignored) and attach to the PR.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): P3-2 lock-path offline bypass + P3-3 idempotent read retry
Extend the offline bypass to the lock path and add opt-in retries for idempotent
reads (grpc-optimization P3-2/P3-3). Both env-gated and default-off/zero.
Offline bypass (lock path): factor the bypass decision into a shared pub(crate)
internode_offline_bypass_reason(addr) and call it from remote_locker::get_client
too, so lock RPCs to an offline peer fast-fail (letting dsync reach quorum
sooner) instead of paying the connect timeout. Does not change quorum; the
self-healing re-probe keeps peers recoverable. Gated by
RUSTFS_INTERNODE_OFFLINE_BYPASS (default off).
Idempotent read retry (P3-3): add execute_read_with_retry — a bounded,
exponential-backoff retry for read-only/reentrant RPCs on transient network
errors — and route disk_info through it. RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES
defaults to 0 (disabled). Write/lock RPCs are never retried (quorum/idempotency
safety, per CLAUDE.md); the wrapper requires an Fn closure so only reads that
rebuild their request from borrowed inputs qualify.
Deferred: grpc.health.v1 (optional ecosystem-compat only; needs a new
tonic-health dep and 3-way hybrid-service wiring — internal needs are met by the
existing Ping RPC).
Verification: cargo check/test on config, ecstore (105 rpc tests green incl.
disk_info now via the retry wrapper); clippy clean on touched files; make
pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(scripts): one-click internode gRPC A/B benchmark driver
Wrap the per-stage env matrix from the benchmark runbook into
scripts/run_internode_grpc_ab_bench.sh: given --stage <p0|p1|p2|p3> and --phase
<before|after>, it emits the stage/phase RUSTFS_INTERNODE_* server env to
<out-dir>/server-env.sh and runs the right underlying bench
(run_internode_transport_baseline.sh for p0/p1/p2, run_four_node_cluster_failover_bench.sh
for p3) into a labeled target/bench/internode-transport/<stage>-<phase>/.
Passthrough args after `--` reach the underlying bench; --dry-run previews the
env + command. The script is explicit that RUSTFS_INTERNODE_* are server env, so
for the load-driven stages the operator must restart rustfs with the emitted env
before the run; the docker four-node (p3) path exports them for a forwarding
compose. shellcheck-clean.
Runbook updated with a "One-click driver" section.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(compose): forward RUSTFS_INTERNODE_* into the four-node cluster
The four-node local-build compose only forwarded a fixed whitelist of env, so
the internode gRPC knobs (grpc-optimization P0-P3) never reached the containers
and the A/B bench driver's "after" phase was a no-op. Forward the full
RUSTFS_INTERNODE_* set with defaults matching the binary defaults, so leaving
them unset is a no-op and the A/B driver can toggle a stage per phase.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(internode): address Copilot review — retry health action + poison-safe peer health
Two review nits on #4337:
- P3-3 idempotent read retry (remote_disk.rs): execute_read_with_retry ran every
attempt through execute_with_timeout_for_op, which hardcodes
FailureHealthAction::MarkFailure. So the first transient error could flip the
disk faulty and short-circuit the remaining retries, and each attempt
over-counted the failure. Route all but the final attempt through
execute_with_timeout_for_op_and_health_action with IgnoreFailure; only the last
attempt marks faulty/evicts. No default impact (retries default 0).
- Peer-health helpers (io-metrics): record_peer_reachable/record_peer_unreachable,
cluster_peer_is_offline and cluster_peer_should_bypass early-returned on a
poisoned mutex, permanently stalling the offline gauge and bypass state after a
single panic. Recover via PoisonError::into_inner().
Verification: cargo check/test on io-metrics + ecstore (105 rpc tests green);
clippy clean on touched files; make pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(site-replication): Add Docker Compose setup for site replication testing
- Fixed the site replication flag issue and resolved the replication storm.
- Introduced a new directory for site replication tests with Docker Compose.
- Created `docker-compose.yml` to define three RustFS sites and a setup container.
- Added `README.md` to document the purpose, usage, and test flow of the replication setup.
- Implemented `run-object-flow-check.sh` script to verify object replication across sites.
- Configured health checks and volume permissions for the RustFS containers.
- Enabled customization of access keys, bucket names, and other parameters via environment variables.
* fix
* fix
* feat(ecstore): add object lock diagnostics
Add configurable namespace lock diagnostics for object operations so production contention can be traced by operation, owner, and key.
Wrap object read/write lock acquisition in diagnostic guards across get, head, put, delete, copy, and multipart flows, and log slow acquisition and long hold durations behind new RUSTFS_OBJECT_LOCK_DIAG_* settings.
Verification:
- make pre-commit
* feat(obs): expose object lock diagnostics metrics
Add Prometheus metrics and Grafana panels for object namespace lock diagnostics, covering slow acquire counts, slow hold counts, acquire duration, hold duration, and the diagnostics-enabled state.
Adopt PR review feedback by keeping diagnostic guards alive through the guarded operation so long-hold warnings and metrics are emitted, reusing shared env helpers, and reducing default-path overhead when diagnostics are disabled.
Verification:
- cargo check -p rustfs-ecstore -p rustfs-io-metrics
- cargo test -p rustfs-ecstore store::object -- --nocapture
- make pre-commit
* perf(obs): reduce object lock diag overhead
Avoid repeated environment parsing on hot object-lock paths by caching the diagnostics-enabled flag, and stop allocating label strings for object lock metrics by recording static labels directly.
Strengthen io-metrics tests by using a local recorder and asserting that the expected object lock diagnostic counters, gauges, and histograms are emitted.
Verification:
- cargo check -p rustfs-ecstore -p rustfs-io-metrics
- cargo test -p rustfs-io-metrics -- --nocapture
- make pre-commit
* fix(ecstore): harden issue3031 multipart validation path
- clear stale multipart part destinations before rename fan-out
- add repeated part overwrite regression coverage
- reduce remote disk startup false-fault escalation to suspect-first
- refine remote locker diagnostics and lower scanner leader-lock log noise
- add a dedicated 4-node issue3031 docker validation script
* refactor(admin): inline console version json macro
- drop the unused serde_json::json import in admin console
- call serde_json::json! inline in version_handler
- keep the console version response behavior unchanged
* fix(remote-disk): recover suspect health on probe success
- record probe success during remote disk health checks so suspect drives recover
- use async_with_vars for the remote disk health probe test
- make the missing-listener test assert the state transition more robustly
Zhengchao An