gsadmin/garage
1
0
Fork 0
mirror of https://github.com/deuxfleurs-org/garage.git synced 2026-07-26 07:58:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix: add TCP connect timeout

Browse Source 
known_addrs in PeerInfoInternal is append-only — addresses accumulate
via add_addr() and PeerList gossip but are never removed. In dynamic
environments (k8s pod restarts, DHCP, NAT traversal), this list grows
unboundedly with stale addresses.

Combined with sequential iteration in try_connect() and no TCP connect
timeout in netapp.rs, each unreachable address blocks reconnection for
the kernel's TCP SYN timeout (75-130s on Linux). With 10+ stale
addresses, worst-case reconnection exceeds 750s — a full outage for
replication_factor=3 clusters.

This patches includes a first change to fix this issue:

1. TCP connect timeout (netapp.rs): Wrap TcpStream::connect() in
   tokio::time::timeout(10s). Caps per-address attempt from 75-130s
   to 10s, reducing worst-case 10-addr reconnection from ~750s to ~100s.
This commit is contained in:
Raj Singh
2026-02-15 14:07:12 -06:00
committed by Alex
parent 0b5e82a18b
commit 3a355b1617
1 changed files with 11 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/net/netapp.rs
+11 -2
View File
@@ -45,6 +45,11 @@ const TCP_KEEPALIVE_TIME: Duration = Duration::from_secs(30);
/// Interval between keepalive probes after the first. /// Interval between keepalive probes after the first.
const TCP_KEEPALIVE_INTERVAL: Duration = Duration::from_secs(10); const TCP_KEEPALIVE_INTERVAL: Duration = Duration::from_secs(10);
/// Timeout for outgoing TCP connection attempts.
/// Caps per-address connection time instead of relying on the kernel's
/// TCP SYN timeout (75-130s on Linux, ~20s on macOS).
const CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
fn set_keepalive(stream: &TcpStream) -> Result<(), std::io::Error> { fn set_keepalive(stream: &TcpStream) -> Result<(), std::io::Error> {
let sock_ref = socket2::SockRef::from(stream); let sock_ref = socket2::SockRef::from(stream);
let keepalive = socket2::TcpKeepalive::new() let keepalive = socket2::TcpKeepalive::new()
@@ -331,9 +336,13 @@ impl NetApp {
TcpSocket::new_v6()? TcpSocket::new_v6()?
}; };
socket.bind(SocketAddr::new(addr, 0))?; socket.bind(SocketAddr::new(addr, 0))?;
socket.connect(ip).await? tokio::time::timeout(CONNECT_TIMEOUT, socket.connect(ip))
.await
.map_err(|_| Error::Message(format!("connect to {} timed out", ip)))??
} }
None => TcpStream::connect(ip).await?, None => tokio::time::timeout(CONNECT_TIMEOUT, TcpStream::connect(ip))
.await
.map_err(|_| Error::Message(format!("connect to {} timed out", ip)))??,
}; };
if let Err(e) = set_keepalive(&stream) { if let Err(e) = set_keepalive(&stream) {
warn!("Failed to set keepalive on connection to {}: {}", ip, e); warn!("Failed to set keepalive on connection to {}: {}", ip, e);