Eight tests in this crate called a recorder and asserted nothing. Five of them were worse than that: every `record_*` in `list_objects_metrics` returns early unless `get_stage_metrics_enabled()` is true, and that flag defaults to false, so those tests only ever exercised the early return — never the code their names describe.
They now run against a local `DebuggingRecorder` with the flag on, and each asserts the boundary it is named for: an empty page reports the scan count as its amplification instead of dividing by zero, a zero read quorum is recorded rather than skipped, index serving divides verification attempts by returned objects, and the `-1` whole-directory sentinel reaches the limit histogram unclamped.
`msgpack_json_fallback_counter_records_without_panicking` has no in-struct total to check, so it now asserts the emission: two direction/message pairs must land in two separate series, which a dropped label would collapse into one.
The two process-sampler tests discarded their snapshots. They now assert what cannot differ between callers — a process has one start time and one descriptor limit regardless of which entry point or which sampler observed it, and the status enum must match its numeric projection.
This clears io-metrics from the census (`scripts/find_assertless_tests.py`), taking the tree from 61 candidates to 53.
Refs backlog#1836
Track accepted replay cache records by gRPC operation and split Lock/Unlock and ReadVersion methods out of grpc_other so hotpath validation can attribute nonce pressure without changing replay protection semantics.
Co-authored-by: heihutu <heihutu@gmail.com>
Add stable server labels to node-local Prometheus metrics and OTLP resource attributes so dashboards can distinguish per-node CPU, memory, host network, and internode traffic series.
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(rpc): bind canonical body digest into internode mutating disk RPC signatures
Binds a domain-separated, length-prefixed canonical request-body digest into the
v2 HMAC signature scope for every mutating NodeService disk RPC, so an on-path
attacker on the default-plaintext internode channel can no longer tamper with a
mutation payload (or strip the msgpack `_bin` field to force the JSON fallback
decode) without invalidating the signature.
Covers 13 mutating disk RPCs: RenameData, DeleteVersion, DeleteVersions,
WriteMetadata, UpdateMetadata, WriteAll, Delete, DeletePaths, RenameFile,
RenamePart, DeleteVolume, MakeVolume, MakeVolumes. The digest covers both the
msgpack `_bin` payloads and their JSON compatibility copies. Gated fail-open by
default (RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT) with a convergence counter, so
rolling upgrades are byte-for-byte unaffected; the replay-cache capacity is now
configurable and overflow fails closed with a metric.
Refs https://github.com/rustfs/backlog/issues/1327
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(rpc): satisfy architecture-migration compat-marker guard
Put the removal condition on the RUSTFS_COMPAT_TODO marker line itself, and
stop backticking env-var/metric names in the cleanup-register entry so the
guard's id extractor only sees the task-id.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add decode-error metrics for internode msgpack/json compatibility paths, extend the mixed fallback e2e assertions for transitioned multipart partNumber reads, and cover manual transition async status polling plus inactive-owner status behavior.
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(rpc): gate internode legacy signature fallback behind a convergence counter and strict env
Add the backlog#1327 Plan-A rollout infrastructure on top of the already-merged v2 target-bound internode gRPC authentication: a rustfs_system_network_internode_signature_v1_fallback_total counter that increments only when a request without any v2 auth headers is accepted through the legacy constant-target signature, and a RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT env (default false, compile-time asserted) that, when enabled later, closes the legacy fallback path. Default behavior is fail-open and byte-identical for legacy-only peers; requests carrying v2 headers are verified as v2 with no downgrade exactly as before.
Refs https://github.com/rustfs/backlog/issues/1327
* ci: fix internode auth test lint failures
* perf(ecstore): cache internode sig strict env
---------
Co-authored-by: houseme <housemecn@gmail.com>
`cluster_peer_should_bypass` is called before every internode RPC when the
offline-bypass feature is enabled (remote disk `get_client`, remote locker), and
it took a single process-global `Mutex<HashMap>` even for the overwhelmingly
common case of an online or unknown peer, which is read-only. That serialized
all concurrent internode client acquisitions on one lock.
Switch `CLUSTER_PEER_HEALTH` to a `RwLock`:
- the hot check takes a shared read lock and returns immediately for unknown or
online peers, so concurrent RPCs no longer serialize;
- only an offline peer (rare) drops to the write lock to record a re-probe, with
a re-fetch/re-check because the state can flip back online between releasing the
read lock and taking the write lock;
- the write paths (dial reachable/unreachable) and the read-only
`cluster_peer_is_offline` move to `write()`/`read()` accordingly.
Behavior is unchanged on every path (online/unknown -> not bypassed; offline ->
bypass with one re-probe per interval); `Instant::now()` now runs only on the
offline path. Poison recovery is preserved. All internode unit tests pass.
Addresses rustfs/backlog#1185 (P2).
Co-authored-by: heihutu <heihutu@gmail.com>
fix(metrics): normalize peer-health key so servers_offline_total counts each peer once
rustfs_cluster_servers_offline_total is the count of distinct keys in the
addr-keyed CLUSTER_PEER_HEALTH map. The data path keys a peer by
endpoint.grid_host() (scheme://host:port) while the lock path keys it by
url::Url::to_string() (scheme://host:port/, trailing slash), so one physical
peer becomes two health entries and a single downed node reports 2 (2N for N).
Normalize the address (trim trailing slashes) at the map boundary in
record_peer_reachable/record_peer_unreachable/cluster_peer_is_offline/
cluster_peer_should_bypass so both forms collapse to one canonical key.
Pure observability fix; peer selection and quorum are unchanged.
Fixesrustfs/backlog#1043
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(internode): P0 gRPC transport tuning, message limits, payload metrics
Land the P0 subtask from docs/grpc-optimization: close the client-vs-server
transport gaps and add instrumentation to size which unary RPCs need channel
isolation in P1.
Transport tuning (G3): the client `Endpoint` now disables Nagle and raises the
HTTP/2 stream/connection flow-control windows to mirror the server socket, so
small lock/health RPCs are not batched and larger metadata responses are not
throttled by the 64KiB default window. All env-overridable, 0 opts out.
Message-size limits (G1): both `NodeServiceClient` and `NodeServiceServer` set
max decode/encode size (default 100MiB) instead of tonic's silent 4MiB cap, so a
large multi-version xl.meta or aggregated ReadMultiple no longer fails
out_of_range. The server limit is set on `NodeServiceServer` before wrapping in
the auth `InterceptedService` (the interceptor type does not expose it).
Payload instrumentation (P1 prep): ReadAll/ReadMultiple record a payload-size
histogram plus a large-payload counter when a response crosses the configured
threshold (default 8MiB), feeding alerting on paths that contend with
latency-sensitive control-plane traffic on the shared channel. Threshold-only
counter, no per-call hot-path log.
Verification: cargo check/test on config, io-metrics, ecstore, rustfs; clippy
clean on touched files; make pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): P1 control/bulk gRPC channel isolation (opt-in)
Land the P1 subtask from docs/grpc-optimization: physically separate large
bytes-carrying unary RPCs from latency-sensitive control-plane RPCs so a big
transfer can no longer head-of-line block a lock/health RPC on the shared
HTTP/2 connection (G2/G5).
Introduce ChannelClass { Control, Bulk } and get_channel_for_class in protos.
Control RPCs keep the per-peer connection keyed by the bare address; Bulk RPCs
(ReadAll/WriteAll/ReadMultiple/BatchReadVersion, via a new get_bulk_client) are
round-robined across a small per-peer bulk pool.
Rather than restructuring the global GLOBAL_CONN_MAP (and every consumer), bulk
channels are cached under a composite key (addr\0bulk\0idx). The NUL separator
cannot appear in a URL, so bulk keys never collide with the control key. This
keeps the blast radius small on a consistency-sensitive path. create_new_channel
is refactored into build_channel(dial_addr, cache_key) so several physically
distinct channels to one peer cache independently while dialing/TLS still use
the real address.
Gated by RUSTFS_INTERNODE_CHANNEL_ISOLATION (default OFF) so the default build
is byte-for-byte the pre-P1 behavior: bulk resolves to the control channel and
the switch is a single-env rollback. RUSTFS_INTERNODE_BULK_CHANNELS (default 2,
clamped >=1) sizes the pool. On failure, evict_failed_connection drops the whole
bulk pool for the peer (round-robin hides which index was used), avoiding
half-dead cached channels.
Lock RPCs (remote_locker) already use the default Control path, so lock
semantics and retry behavior are unchanged.
Verification: cargo check/test on config, protos, ecstore, rustfs; new protos
tests for bulk key routing and isolation-off passthrough; clippy clean on
touched files; make pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): P2 msgpack/JSON codec observability + encode buffer presizing
Land the safe, wire-compatible slice of P2 from docs/grpc-optimization: the
observability prerequisite for retiring the redundant JSON fields, plus a codec
micro-optimization. No proto/wire-format change; JSON is still dual-written.
Internode RPCs today dual-encode each metadata value as both msgpack (`*_bin`)
and a JSON compatibility string, and decoders prefer `_bin` with a JSON
fallback. Before the JSON fields can ever be dropped (a cross-version change),
that fallback must be proven unused in production.
Add rustfs_system_network_internode_msgpack_json_fallback_total{direction,
message}: incremented whenever a decode falls back to the JSON field because the
msgpack payload was absent. Wired into both directions — the client decoding
peer responses (remote_disk.rs, incl. the list-level read_multiple/batch
fallbacks) and the server decoding peer requests (node_service/disk.rs). This
counter must read zero across a release window before send paths stop writing
JSON and the proto text fields are reserved/removed (the deferred P2-1 steps).
Also pre-size the msgpack encode buffers (Vec::with_capacity(512)) on both
sides, eliminating the repeated growth reallocations for typical FileInfo
payloads with zero added copy. Full thread_local buffer pooling is deferred: it
needs either an extra copy (unclear net win) or a send-path buffer-return
lifecycle, to be justified by a codec microbenchmark first.
Verification: cargo check/test on io-metrics, ecstore, rustfs; new fallback
counter smoke test; existing codec decode tests green; clippy clean on touched
files; make pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(internode): add msgpack/JSON convergence observation runbook
Runbook driving the observation-gated retirement of the redundant JSON
compatibility fields on internode gRPC metadata RPCs (grpc-optimization P2-1).
Documents the shipped fallback counter
(rustfs_system_network_internode_msgpack_json_fallback_total{direction,message}),
the PromQL to confirm it reads zero across a release window, a standing alert,
and the staged flip/rollback procedure (env-gated msgpack-only send, then proto
field removal in N+1).
Includes the verified field -> peer-decoder audit: only fields whose peer
decodes _bin first may be converged. Notes DeleteVersion.opts (DeleteOptions) is
NOT convergence-ready — its server handler is not _bin-first and must gain a
decode_msgpack_or_json path first. This gates the send-side change so it cannot
empty a JSON field an old peer still needs.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): env-gated msgpack-only send + DeleteVersion _bin support (P2-1)
Implements the send-side lever for retiring the redundant JSON compatibility
fields on internode gRPC metadata RPCs, plus the missing `_bin` support on the
delete path that it depends on (grpc-optimization P2-1). Default-off: the base
build is byte-for-byte the prior dual-write behavior.
Gated msgpack-only send (RUSTFS_INTERNODE_RPC_MSGPACK_ONLY, default false):
- New rustfs_protos::internode_rpc_msgpack_only() reads the flag.
- Client (remote_disk.rs) compat_json() and server (node_service/disk.rs)
compat_response_json() emit an empty JSON string when the flag is on, so only
the msgpack _bin payload is sent. The _bin field is always sent; decoders keep
the JSON read fallback. Applied only to fields with a confirmed _bin-first peer
decoder (WriteMetadata/UpdateMetadata/RenameData file_info, UpdateMetadata opts,
ReadOptions, ReadMultipleReq, BatchReadVersionReq; ReadVersion/ReadXL/RenameData
responses and the ReadMultiple/BatchReadVersion response lists).
- Only enable after the P2 fallback counter has read zero across a release window
(see docs/operations/internode-msgpack-json-convergence-runbook.md). Single-env
rollback; no wire-format break.
DeleteVersion(s) _bin support (prerequisite):
- The DeleteVersion/DeleteVersions protos had NO _bin fields. Add additive
(backward-compatible) bytes file_info_bin/opts_bin (DeleteVersion) and repeated
bytes versions_bin + bytes opts_bin (DeleteVersions); regenerate the checked-in
prost struct.
- Client dual-writes them; server decodes them _bin-first with JSON fallback.
- These delete fields are kept OUT of the msgpack-only set (always dual-write)
until their own fallback counter reads zero across a window with the new
decoders fully deployed. DeleteVersion.raw_file_info stays JSON-only (no _bin
field yet).
Verification: cargo check/test on protos, config, ecstore, rustfs (incl. the six
delete request handler tests and a compat_json default-path test); clippy clean
on touched files; make pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): P3 cluster peer online/offline health metric
Land the safe observability core of P3 (grpc-optimization G6/G8): track each
internode peer's reachability and expose the offline count, for parity with
MinIO's minio_cluster_servers_offline_total. Pure instrumentation — peer
selection and quorum are unchanged.
- io-metrics: per-peer PeerHealthState { online, consecutive_failures } registry
plus record_peer_reachable/record_peer_unreachable. A peer flips offline after
N consecutive failures (dial failures or RPC-triggered evictions) and back
online on the next successful dial; the count of offline peers is published to
the rustfs_cluster_servers_offline_total gauge.
- config: RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD (default 3, clamped >= 1).
- protos: build_channel marks the peer reachable on a successful dial and
unreachable on a dial failure; evict_failed_connection feeds the failure signal
too. Keyed by the real peer address, so control and bulk channels to one peer
share health state.
Deferred (documented in docs/grpc-optimization P3): startup prewarm (no clean
topology-ready hook yet), the offline fast-bypass in peer routing (consistency-
sensitive; must not change quorum), and idempotent-read-only retry. This commit
is observability only.
Verification: cargo check/test on io-metrics, config, protos (new peer-health
state-machine and threshold-clamp tests); clippy clean on touched files; make
pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): P3 control-channel prewarm + self-healing offline bypass
Add the remaining P3 connection-lifecycle levers (grpc-optimization G6/G8), both
env-gated and default-off so the base build is unchanged.
Prewarm (RUSTFS_INTERNODE_PREWARM, default off): RemoteDisk::new spawns a
best-effort background dial of the peer's control channel, deduped per peer
address, moving the connect cost off the first RPC. Failures fall through to the
existing lazy connect + recovery monitor.
Offline bypass (RUSTFS_INTERNODE_OFFLINE_BYPASS, default off): remote_disk
get_client/get_bulk_client fast-fail a peer already marked offline instead of
paying the connect timeout, so the erasure layer proceeds on quorum sooner. This
does NOT change quorum. It is self-healing: cluster_peer_should_bypass lets one
request per RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS (default 5s) through to recover
the peer even with no background monitor, and the recovery monitor's own probe
path calls the client directly so it is never bypassed.
io-metrics gains cluster_peer_is_offline / cluster_peer_should_bypass (with a
per-peer re-probe timestamp). Scope: data path only — remote_locker (lock RPCs,
most consistency-sensitive) is left dual-writing/unbypassed as a follow-up.
Verification: cargo check/test on io-metrics, config, ecstore (new self-healing
bypass tests; all 105 rpc tests green); clippy clean on touched files; make
pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(internode): add A/B benchmark runbook for gRPC optimization stages
Reproducible before/after collection procedure for grpc-optimization P0–P3.
Since every stage is env-gated, before/after is the same binary with different
env — no rebuild. Documents, per stage: the exact env toggles (baseline vs
enabled column), which existing bench script to run
(run_internode_transport_baseline.sh / run_four_node_cluster_failover_bench.sh),
the Prometheus metrics to capture, and the acceptance gates from the design docs
(e.g. lock p99 down >= 20% for P1, msgpack fallback counter = 0 before enabling
P2, correct rustfs_cluster_servers_offline_total for P3).
Live runs require a multi-node cluster + load tool + Prometheus scrape and cannot
be produced in a single-process sandbox; artifacts land under target/bench
(gitignored) and attach to the PR.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(internode): P3-2 lock-path offline bypass + P3-3 idempotent read retry
Extend the offline bypass to the lock path and add opt-in retries for idempotent
reads (grpc-optimization P3-2/P3-3). Both env-gated and default-off/zero.
Offline bypass (lock path): factor the bypass decision into a shared pub(crate)
internode_offline_bypass_reason(addr) and call it from remote_locker::get_client
too, so lock RPCs to an offline peer fast-fail (letting dsync reach quorum
sooner) instead of paying the connect timeout. Does not change quorum; the
self-healing re-probe keeps peers recoverable. Gated by
RUSTFS_INTERNODE_OFFLINE_BYPASS (default off).
Idempotent read retry (P3-3): add execute_read_with_retry — a bounded,
exponential-backoff retry for read-only/reentrant RPCs on transient network
errors — and route disk_info through it. RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES
defaults to 0 (disabled). Write/lock RPCs are never retried (quorum/idempotency
safety, per CLAUDE.md); the wrapper requires an Fn closure so only reads that
rebuild their request from borrowed inputs qualify.
Deferred: grpc.health.v1 (optional ecosystem-compat only; needs a new
tonic-health dep and 3-way hybrid-service wiring — internal needs are met by the
existing Ping RPC).
Verification: cargo check/test on config, ecstore (105 rpc tests green incl.
disk_info now via the retry wrapper); clippy clean on touched files; make
pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(scripts): one-click internode gRPC A/B benchmark driver
Wrap the per-stage env matrix from the benchmark runbook into
scripts/run_internode_grpc_ab_bench.sh: given --stage <p0|p1|p2|p3> and --phase
<before|after>, it emits the stage/phase RUSTFS_INTERNODE_* server env to
<out-dir>/server-env.sh and runs the right underlying bench
(run_internode_transport_baseline.sh for p0/p1/p2, run_four_node_cluster_failover_bench.sh
for p3) into a labeled target/bench/internode-transport/<stage>-<phase>/.
Passthrough args after `--` reach the underlying bench; --dry-run previews the
env + command. The script is explicit that RUSTFS_INTERNODE_* are server env, so
for the load-driven stages the operator must restart rustfs with the emitted env
before the run; the docker four-node (p3) path exports them for a forwarding
compose. shellcheck-clean.
Runbook updated with a "One-click driver" section.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(compose): forward RUSTFS_INTERNODE_* into the four-node cluster
The four-node local-build compose only forwarded a fixed whitelist of env, so
the internode gRPC knobs (grpc-optimization P0-P3) never reached the containers
and the A/B bench driver's "after" phase was a no-op. Forward the full
RUSTFS_INTERNODE_* set with defaults matching the binary defaults, so leaving
them unset is a no-op and the A/B driver can toggle a stage per phase.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(internode): address Copilot review — retry health action + poison-safe peer health
Two review nits on #4337:
- P3-3 idempotent read retry (remote_disk.rs): execute_read_with_retry ran every
attempt through execute_with_timeout_for_op, which hardcodes
FailureHealthAction::MarkFailure. So the first transient error could flip the
disk faulty and short-circuit the remaining retries, and each attempt
over-counted the failure. Route all but the final attempt through
execute_with_timeout_for_op_and_health_action with IgnoreFailure; only the last
attempt marks faulty/evicts. No default impact (retries default 0).
- Peer-health helpers (io-metrics): record_peer_reachable/record_peer_unreachable,
cluster_peer_is_offline and cluster_peer_should_bypass early-returned on a
poisoned mutex, permanently stalling the offline gauge and bypass state after a
single panic. Recover via PoisonError::into_inner().
Verification: cargo check/test on io-metrics + ecstore (105 rpc tests green);
clippy clean on touched files; make pre-commit green.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(rio): propagate http writer shutdown errors
* fix(ecstore): unify remote lock rpc deadlines
* fix(storage): reject corrupt read multiple payloads
* feat(rio): add internode http tuning profiles
* feat(metrics): add internode baseline signals
* feat(ecstore): observe shard locality topology
* feat(ecstore): gate shard locality scheduling
* feat(ecstore): gate batch read version rpc
* feat(ecstore): observe batch processor adaptation
* feat(ecstore): gate batch processor observation
* docs: add get benchmark regression analysis
* docs: add issue 797 execution plan status
* fix(ecstore): require explicit batch rpc support
* fix(ecstore): honor documented batch read gate
* fix(ecstore): keep batch read gate stable per call
* chore: update workspace dependencies
* feat(ecstore): log batch read gate decisions
* feat(ecstore): count batch read gate decisions
* test(issue-797): add local internode A/B runner
* test(rio): fix tuning profile spelling fixture
* fix(protocols): adapt sftp channel open callbacks
* fix(metrics): wrap batch processor observation args
* chore(docs): keep issue notes local only
* fix(storage): address internode review feedback
* fix(storage): address internode data-path review findings
- Run the BatchReadVersion auto-mode unary fallback outside the batch
RPC deadline so each read_version keeps its own per-op timeout and
health accounting instead of racing the whole batch against one
drive timeout.
- Cap adaptive batch-processor concurrency growth at a hard multiple
of the configured baseline so sustained fast batches cannot ratchet
past the configured limit.
- Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE,
and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading
the environment on hot paths.
- Skip shard read-cost collection in observe mode when stage metrics
are disabled, and cache the local endpoint host list instead of
rebuilding it on every read.
- Allow --warp-extra-args values starting with -- and drop the unused
warp_hosts_csv helper in the issue-797 A/B runner.
* fix(storage): address internode data-path review findings
- Run the BatchReadVersion auto-mode unary fallback outside the batch
RPC deadline so each read_version keeps its own per-op timeout and
health accounting instead of racing the whole batch against one
drive timeout.
- Cap adaptive batch-processor concurrency growth at a hard multiple
of the configured baseline so sustained fast batches cannot ratchet
past the configured limit.
- Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE,
and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading
the environment on hot paths.
- Skip shard read-cost collection in observe mode when stage metrics
are disabled, and cache the local endpoint host list instead of
rebuilding it on every read.
- Allow --warp-extra-args values starting with -- and drop the unused
warp_hosts_csv helper in the issue-797 A/B runner.
Co-Authored-By: heihutu<heihutu@gmail.com>
* fix(storage): align buffer clamp test with media cap
* fix(ecstore): release optimized read locks before streaming
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
houseme