gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-30 08:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(internode): relax keepalive/RPC timeouts to fix large-object GET EOF (#4284)

Browse Source 
fix(internode): relax aggressive HTTP/2 keepalive & RPC timeouts to stop large-object GET truncation

The internode gRPC channel used a 3s HTTP/2 keepalive timeout and a 10s
overall RPC timeout. Under high-concurrency large-object reads a saturated
peer's PING ACK is legitimately delayed past 3s, so the whole channel (and
every RPC/stream on it) is torn down as a 'dead peer'. In-flight peer shard
reads then fail mid-object and large GETs truncate after headers+Content-Length
are already sent, surfacing to clients as 'download error: unexpected EOF'.

Reproduced on a 4-node erasure cluster with pure GET-only warp (no concurrent
writes) at 64 concurrency: 10MiB GET ~31 unexpected-EOF / 3min. Raising the
internode keepalive timeout (3s->30s via env) alone cut that to ~7; also raising
the RPC timeout cut it to ~3.

- Raise DEFAULT_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS 3 -> 20
- Raise DEFAULT_INTERNODE_RPC_TIMEOUT_SECS 10 -> 30
- Wire the data-plane rio HttpReader keepalive timeout (was hardcoded 3s) to the
  same env/default so control- and data-plane stay consistent.

Refs backlog#832.
This commit is contained in:
Zhengchao An
2026-07-05 19:27:23 +08:00
committed by GitHub
parent 9cf211930d
commit 9959c828a1
2 changed files with 18 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/config/src/constants/internode.rs
+11 -4
View File
@@ -25,12 +25,19 @@ pub const ENV_INTERNODE_HTTP2_KEEPALIVE_INTERVAL_SECS: &str = "RUSTFS_INTERNODE_
pub const DEFAULT_INTERNODE_HTTP2_KEEPALIVE_INTERVAL_SECS: u64 = 5;
/// HTTP/2 keepalive timeout for internode gRPC channels.
///
/// This is the time a peer has to ACK a keepalive PING before the whole channel
/// (and every RPC/stream multiplexed on it) is torn down. A very aggressive value
/// misfires under load: on a saturated node the PING ACK is legitimately delayed,
/// the channel is wrongly declared dead, in-flight peer reads fail, and large-object
/// GETs truncate mid-stream (client "unexpected EOF"). See backlog#832. Keep this
/// generous enough to tolerate transient load while still detecting truly dead peers.
pub const ENV_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS: &str = "RUSTFS_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS";
pub const DEFAULT_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS: u64 = 3;
pub const DEFAULT_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS: u64 = 20;
/// Overall timeout for a single internode gRPC request.
pub const ENV_INTERNODE_RPC_TIMEOUT_SECS: &str = "RUSTFS_INTERNODE_RPC_TIMEOUT_SECS";
pub const DEFAULT_INTERNODE_RPC_TIMEOUT_SECS: u64 = 10;
pub const DEFAULT_INTERNODE_RPC_TIMEOUT_SECS: u64 = 30;
/// Profile selector for conservative internode HTTP data-plane client tuning.
pub const ENV_INTERNODE_HTTP_TUNING_PROFILE: &str = "RUSTFS_INTERNODE_HTTP_TUNING_PROFILE";
@@ -73,8 +80,8 @@ mod tests {
assert_eq!(DEFAULT_INTERNODE_CONNECT_TIMEOUT_SECS, 3);
assert_eq!(DEFAULT_INTERNODE_TCP_KEEPALIVE_SECS, 10);
assert_eq!(DEFAULT_INTERNODE_HTTP2_KEEPALIVE_INTERVAL_SECS, 5);
assert_eq!(DEFAULT_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS, 3);
assert_eq!(DEFAULT_INTERNODE_RPC_TIMEOUT_SECS, 10);
assert_eq!(DEFAULT_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS, 20);
assert_eq!(DEFAULT_INTERNODE_RPC_TIMEOUT_SECS, 30);
assert_eq!(DEFAULT_INTERNODE_HTTP_TUNING_PROFILE, "legacy");
}