mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-08 04:58:12 +00:00
fix(storage): bound multipart admission wait below SDK write timeouts (#7408)
A multipart UploadPart queued for a foreground write permit is not read while it waits, so the client's socket write stalls for the whole wait and the client's own write timeout decides the outcome; botocore reports that as ConnectionClosedError. Lower the default queue wait from 30 s to 10 s so the part receives SlowDown before mainstream SDK timeouts, and stop forcing a 4 MiB SO_RCVBUF on the API listener (new RUSTFS_HTTP_SOCKET_RECV_BUFFER_BYTES restores a fixed size) so a queued connection no longer lets up to 8 MiB of unread body accumulate in kernel memory. Fixes #7385. Co-authored-by: houseme <housemecn@gmail.com>
This commit is contained in:
Zhengchao An
@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
## [Unreleased]
|
||||
|
||||
### Fixed
|
||||
- **Multipart admission queue**: an `UploadPart` waiting for a foreground write permit now waits at most 10 s by default (`RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS`, previously 30 s), so a queued part returns S3 `SlowDown` before the client's socket write timeout drops the connection. Separately, the API listener no longer forces a 4 MiB `SO_RCVBUF` on every accepted socket (kernel autotuning applies; `RUSTFS_HTTP_SOCKET_RECV_BUFFER_BYTES` restores a fixed size), so a queued part no longer lets up to 8 MiB of unread body accumulate in kernel memory per connection, which is what throttled whole nodes under SDK-default multipart concurrency. Fixes #7385.
|
||||
- **Helm Ingress**: `customAnnotations` are now merged with class-specific annotations (nginx/traefik) instead of being ignored when `ingress.className` is set.
|
||||
- **Per-pool erasure parity**: Erasure parity (STANDARD and reduced-redundancy) is now resolved independently for every pool instead of reusing the first pool's value. A heterogeneous topology — for example a 4-drive pool plus a 2-drive pool created during expansion — previously inherited the first pool's parity and could resolve to zero data shards in the smaller pool, panicking Reed-Solomon construction on write. Automatic parity now resolves per pool (for example `2+2` in the 4-drive pool and `1+1` in the 2-drive pool). Fixes #4801.
|
||||
|
||||
|
||||
+11
-1
@@ -153,14 +153,24 @@ concurrently. Small direct PUTs stay on the legacy path.
|
||||
- default is `0`.
|
||||
- `RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS`
|
||||
- how long an `UploadPart` waits in the bounded queue for a permit before returning S3 `SlowDown`; `0` rejects immediately when the pool is full.
|
||||
- default is `30000`. Parts wait before body ingest, so SDK-default clients that send every part of an upload concurrently drain through the pool instead of failing.
|
||||
- default is `10000`. Parts wait before body ingest, so SDK-default clients that send every part of an upload concurrently drain through the pool instead of failing on a full pool.
|
||||
- RustFS does not read the request body while a part is queued, so the client's socket write stalls for the whole wait and whatever timeout the client or an intermediary has configured competes with this value. Keep it with margin below the shortest such timeout in use (botocore applies its 60 s `connect_timeout` to the body write; the AWS SDK for Java v2 has a 30 s socket write timeout; reverse proxies add their own body timeouts); a wait that outlives the client timeout surfaces as a dropped connection instead of `SlowDown`.
|
||||
- `RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_MAX_PENDING`
|
||||
- maximum `UploadPart` requests waiting for a permit at once; parts beyond it return `SlowDown` without waiting.
|
||||
- default is `0`, which derives 16 times the permit limit (512 at stock settings).
|
||||
- each queued HTTP/1 part holds whatever unread body the client already pushed into the connection's kernel receive buffer (an HTTP/2 part holds up to its flow-control window in process memory), so this depth also bounds that memory. RustFS leaves the receive buffer to kernel autotuning (see `RUSTFS_HTTP_SOCKET_RECV_BUFFER_BYTES` below), which keeps an unread connection at the kernel's initial size (128 KiB on current Linux).
|
||||
- `RUSTFS_PUT_FOREGROUND_ADMISSION_ENABLE`, `RUSTFS_PUT_FOREGROUND_ADMISSION_LIMIT`, `RUSTFS_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS`
|
||||
- experimental strict gate that applies to every foreground write regardless of size and replaces the pool above when enabled.
|
||||
- default is disabled; enabling it with limit `0` disables foreground write admission entirely.
|
||||
|
||||
## HTTP listener socket environment variables
|
||||
|
||||
- `RUSTFS_HTTP_SOCKET_RECV_BUFFER_BYTES`
|
||||
- fixed `SO_RCVBUF` for the API listener, inherited by every accepted socket; `0` leaves the receive buffer to kernel autotuning.
|
||||
- default is `0`. Earlier releases hard-coded 4 MiB, which Linux doubles to 8 MiB and which disables autotuning, so every connection whose body was not being read yet (a multipart part queued for a foreground write permit) could accumulate up to 8 MiB of unread body in kernel memory; at SDK-default multipart concurrency that was enough to push a node into TCP memory pressure.
|
||||
- with autotuning the per-connection receive ceiling is the kernel's (`net.ipv4.tcp_rmem` max, 6 MiB on stock Linux) instead of the former fixed 8 MiB, so a single very high-bandwidth-delay connection may see a somewhat lower ceiling; raise `net.ipv4.tcp_rmem` first, and set this variable only on kernels without receive-buffer autotuning (illumos/Solaris) or where the sysctl cannot be changed.
|
||||
- the send buffer stays fixed at 4 MiB because the stock Linux send autotuning ceiling (`net.ipv4.tcp_wmem` max, 4 MiB) is lower than a GB-level response stream needs.
|
||||
|
||||
## Remote tier timeout environment variables
|
||||
|
||||
- `RUSTFS_TIER_REMOTE_CONNECT_TIMEOUT_SECS`
|
||||
|
||||
@@ -376,21 +376,39 @@ pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 250;
|
||||
///
|
||||
/// SDK-default multipart clients send every part of an upload concurrently, so
|
||||
/// a single node routinely sees several times more parts in flight than the
|
||||
/// permit pool allows. Those parts have not ingested a body yet, so queueing
|
||||
/// them costs a connection rather than memory or internode streams; the pool
|
||||
/// still bounds the number of parts being written. The wait is long enough for
|
||||
/// an ordinary queue to drain on modest hardware, and a part that cannot get a
|
||||
/// permit within it fails with S3 `SlowDown`/503 for the client to retry.
|
||||
/// `0` rejects immediately when the pool is full.
|
||||
/// permit pool allows. A queued part waits before body ingest, so the pool
|
||||
/// still bounds the number of parts being written, but the wait is not free:
|
||||
/// RustFS does not read the request body while the part is queued (hyper only
|
||||
/// sends `100 Continue` once the body is first polled, and the AWS SDKs send
|
||||
/// the body after a 1-3 s `Expect: 100-continue` grace anyway), so the
|
||||
/// client's socket write stalls once the kernel buffers fill, and whatever
|
||||
/// timeout the client or an intermediary has configured decides the outcome.
|
||||
/// botocore applies its `connect_timeout` (60 s) to the body write, the AWS
|
||||
/// SDK for Java v2 has a 30 s socket write timeout, and MinIO bounds the same
|
||||
/// wait with a 10 s request deadline. The wait must leave margin under the
|
||||
/// shortest of those, not merely fall below an SDK default, so the part
|
||||
/// receives S3 `SlowDown`/503 for the client to retry instead of losing its
|
||||
/// connection (issue #7385). `0` rejects immediately when the pool is full.
|
||||
pub const ENV_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: &str =
|
||||
"RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS";
|
||||
pub const DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 30_000;
|
||||
pub const DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 10_000;
|
||||
|
||||
// A queued part holds the client's body write open for the whole wait. The
|
||||
// shortest write timeout among mainstream S3 SDKs is the AWS SDK for Java v2's
|
||||
// 30 s socket write timeout; keep the compiled default at no more than a third
|
||||
// of it. This locks only the default; the environment variable may still raise
|
||||
// the wait past any client timeout.
|
||||
const _: () = assert!(DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS * 3 <= 30_000);
|
||||
|
||||
/// Maximum multipart UploadPart requests waiting for a foreground write permit per process.
|
||||
///
|
||||
/// Parts beyond this queue depth are rejected with S3 `SlowDown`/503 without
|
||||
/// waiting, so a genuinely saturated node still fails fast instead of holding
|
||||
/// an unbounded set of connections open for the whole wait timeout.
|
||||
/// an unbounded set of connections open for the whole wait timeout. Each
|
||||
/// queued HTTP/1 part also holds whatever unread body the client already
|
||||
/// pushed into that connection's kernel receive buffer, and a queued HTTP/2
|
||||
/// part holds up to its flow-control window in process memory, so the depth
|
||||
/// bounds socket and window memory as well as connections.
|
||||
/// `0` derives the depth from the permit limit.
|
||||
pub const ENV_PUT_MULTIPART_FOREGROUND_ADMISSION_MAX_PENDING: &str = "RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_MAX_PENDING";
|
||||
pub const DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_MAX_PENDING: usize = 0;
|
||||
|
||||
@@ -159,6 +159,24 @@ pub const DEFAULT_HTTP1_HEADER_READ_TIMEOUT: u64 = 75;
|
||||
pub const ENV_HTTP1_MAX_BUF_SIZE: &str = "RUSTFS_HTTP1_MAX_BUF_SIZE";
|
||||
pub const DEFAULT_HTTP1_MAX_BUF_SIZE: usize = 64 * 1024; // 64 KB
|
||||
|
||||
/// Environment variable for a fixed kernel receive buffer (`SO_RCVBUF`, bytes)
|
||||
/// on the API listener. Default: 0, which leaves the buffer to kernel
|
||||
/// autotuning.
|
||||
///
|
||||
/// A fixed `SO_RCVBUF` is inherited by every accepted socket and disables
|
||||
/// receive-buffer autotuning, so a connection whose request body is not being
|
||||
/// read yet (a multipart part queued for a foreground write permit) lets up to
|
||||
/// the fixed size of unread body accumulate in kernel memory — Linux doubles
|
||||
/// the requested value, so the former hard-coded 4 MiB held up to 8 MiB per
|
||||
/// queued connection (issue #7385). Autotuning keeps an unread connection at
|
||||
/// the kernel's initial size and grows only connections that are being
|
||||
/// drained. Set this only on kernels without receive-buffer autotuning
|
||||
/// (illumos/Solaris) or on very high-bandwidth-delay links where the kernel's
|
||||
/// autotuning ceiling (`net.ipv4.tcp_rmem` on Linux) is too low and cannot be
|
||||
/// raised.
|
||||
pub const ENV_HTTP_SOCKET_RECV_BUFFER_BYTES: &str = "RUSTFS_HTTP_SOCKET_RECV_BUFFER_BYTES";
|
||||
pub const DEFAULT_HTTP_SOCKET_RECV_BUFFER_BYTES: usize = 0;
|
||||
|
||||
/// Environment variable for the S3 request-body inter-chunk read timeout
|
||||
/// (seconds). Default: 300. Set to 0 to disable.
|
||||
///
|
||||
|
||||
@@ -1014,6 +1014,10 @@ pub async fn start_http_server(
|
||||
// Common setup for both IPv4 and successful dual-stack IPv6
|
||||
let backlog = get_listen_backlog();
|
||||
let keepalive = get_default_tcp_keepalive();
|
||||
let recv_buffer_bytes = rustfs_utils::get_env_usize(
|
||||
rustfs_config::ENV_HTTP_SOCKET_RECV_BUFFER_BYTES,
|
||||
rustfs_config::DEFAULT_HTTP_SOCKET_RECV_BUFFER_BYTES,
|
||||
);
|
||||
|
||||
// Helper to configure socket with optimized parameters
|
||||
let configure_socket = |socket: &socket2::Socket| -> Result<()> {
|
||||
@@ -1068,10 +1072,25 @@ pub async fn start_http_server(
|
||||
);
|
||||
}
|
||||
|
||||
// 4. Increase receive/send buffer to support BDP at GB-level throughput.
|
||||
// 4. Socket buffers. The receive buffer is left to kernel autotuning
|
||||
// unless RUSTFS_HTTP_SOCKET_RECV_BUFFER_BYTES is set: a fixed SO_RCVBUF
|
||||
// is inherited by every accepted socket and disables autotuning, so a
|
||||
// request whose body is not being read yet (a multipart part queued
|
||||
// for a foreground write permit) lets up to the fixed size of unread
|
||||
// body accumulate in kernel memory — the former hard-coded 4 MiB held
|
||||
// up to 8 MiB per queued connection on Linux, which doubles the
|
||||
// requested size. Autotuning keeps an unread connection at the
|
||||
// kernel's initial size and grows only connections that are actually
|
||||
// being drained (issue #7385). The send buffer stays fixed at 4 MiB
|
||||
// because the stock Linux send autotuning ceiling (`tcp_wmem` max,
|
||||
// 4 MiB) is below what a GB-level response stream needs, whereas the
|
||||
// receive ceiling (`tcp_rmem` max, 6 MiB) already exceeds the old
|
||||
// fixed request.
|
||||
// Some constrained local environments reject these socket options with
|
||||
// EPERM/ENOPROTOOPT-style failures; log and continue in that case.
|
||||
if let Err(e) = socket.set_recv_buffer_size(4 * rustfs_config::MI_B) {
|
||||
if recv_buffer_bytes > 0
|
||||
&& let Err(e) = socket.set_recv_buffer_size(recv_buffer_bytes)
|
||||
{
|
||||
debug!(
|
||||
event = "socket_option_unavailable",
|
||||
component = LOG_COMPONENT_SERVER,
|
||||
@@ -1566,9 +1585,11 @@ pub async fn start_http_server(
|
||||
let socket_ref = SockRef::from(&socket);
|
||||
|
||||
// ── POST-ACCEPT SOCKET SYSCALLS ──
|
||||
// The listening socket already sets TCP_NODELAY, TCP_KEEPALIVE,
|
||||
// SO_RCVBUF, and SO_SNDBUF. On Linux/BSD, these are inherited by
|
||||
// accepted sockets, so we skip redundant re-application here.
|
||||
// The listening socket already sets TCP_NODELAY, TCP_KEEPALIVE, and
|
||||
// SO_SNDBUF (SO_RCVBUF stays kernel-autotuned unless
|
||||
// RUSTFS_HTTP_SOCKET_RECV_BUFFER_BYTES is set, see the listener
|
||||
// setup). On Linux/BSD, these are inherited by accepted sockets, so
|
||||
// we skip redundant re-application here.
|
||||
//
|
||||
// Only TCP_QUICKACK (Linux) is kept — it is inherently per-connection
|
||||
// and NOT inherited from the listening socket.
|
||||
|
||||
@@ -36,9 +36,12 @@ use tokio::sync::{OwnedSemaphorePermit, Semaphore};
|
||||
use tracing::debug;
|
||||
|
||||
const DERIVED_LARGE_PUT_ADMISSION_LIMIT_MAX: usize = 32;
|
||||
// A queued multipart part holds a connection but no body, so the queue can be
|
||||
// several times deeper than the permit pool. Sixteen uploads sending sixteen
|
||||
// parts each through one node fits inside the derived depth of 32 * 16.
|
||||
// A queued multipart part holds a connection but no user-space body buffer on
|
||||
// HTTP/1 (only whatever unread body the client already pushed into the kernel
|
||||
// receive buffer); on HTTP/2 it holds up to the per-stream flow-control window
|
||||
// in process memory. Either way the queue can be several times deeper than the
|
||||
// permit pool. Sixteen uploads sending sixteen parts each through one node
|
||||
// fits inside the derived depth of 32 * 16.
|
||||
const DERIVED_MULTIPART_ADMISSION_MAX_PENDING_FACTOR: usize = 16;
|
||||
// Framed S2 alone can retain one encoded and one decoded block of roughly
|
||||
// 4 MiB each, while other codecs have their own larger windows. Four keeps
|
||||
@@ -332,6 +335,16 @@ impl ForegroundWriteAdmissionPolicy {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
fn multipart_wait_timeout_for_test(&self) -> Option<Duration> {
|
||||
match self {
|
||||
Self::Large {
|
||||
multipart_wait_timeout, ..
|
||||
} => Some(*multipart_wait_timeout),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
fn strict_for_test(enabled: bool, limit: usize, wait_timeout: Duration) -> Self {
|
||||
if enabled {
|
||||
@@ -1228,8 +1241,9 @@ mod integration_tests {
|
||||
use super::super::io_schedule::{IoLoadLevel, IoPriority};
|
||||
use super::super::request_guard::GetObjectGuard;
|
||||
use super::{
|
||||
ConcurrencyManager, ForegroundWriteAdmission, SNOWBALL_ARCHIVE_DECODER_LIMIT, SNOWBALL_MEMBER_COMMIT_LIMIT,
|
||||
SNOWBALL_STAGING_BYTES_LIMIT, derive_large_put_admission_limit, derive_multipart_admission_max_pending,
|
||||
ConcurrencyManager, ForegroundWriteAdmission, ForegroundWriteAdmissionPolicy, SNOWBALL_ARCHIVE_DECODER_LIMIT,
|
||||
SNOWBALL_MEMBER_COMMIT_LIMIT, SNOWBALL_STAGING_BYTES_LIMIT, derive_large_put_admission_limit,
|
||||
derive_multipart_admission_max_pending,
|
||||
};
|
||||
use crate::storage::storage_api::concurrency_consumer::PutObjectGuard;
|
||||
use rustfs_concurrency::{AdmissionState, WorkloadAdmissionSnapshotProvider, WorkloadClass};
|
||||
@@ -1742,6 +1756,51 @@ mod integration_tests {
|
||||
drop(held);
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
#[serial]
|
||||
async fn test_concurrency_manager_multipart_part_zero_wait_rejects_without_queueing() {
|
||||
let manager = ConcurrencyManager::with_multipart_admission_queue_for_test(1, Duration::ZERO, 4);
|
||||
let held = manager
|
||||
.admit_multipart_part(1024)
|
||||
.await
|
||||
.expect("first multipart part admission should acquire");
|
||||
|
||||
let rejected = manager
|
||||
.admit_multipart_part(1024)
|
||||
.await
|
||||
.expect("zero multipart wait should reject, not close");
|
||||
assert!(matches!(rejected, ForegroundWriteAdmission::Rejected));
|
||||
assert_eq!(manager.put_object_admission_snapshot().queued, Some(0));
|
||||
drop(held);
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn test_concurrency_manager_multipart_wait_default_stays_below_sdk_write_timeouts() {
|
||||
let unset = [
|
||||
(rustfs_config::ENV_PUT_FOREGROUND_ADMISSION_ENABLE, None::<&str>),
|
||||
(rustfs_config::ENV_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE, None),
|
||||
(rustfs_config::ENV_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS, None),
|
||||
];
|
||||
temp_env::with_vars(unset, || {
|
||||
let policy = ForegroundWriteAdmissionPolicy::from_env(64);
|
||||
// A queued part stalls the client's socket write for the whole wait, so the
|
||||
// default must answer with `SlowDown` before mainstream SDK write timeouts.
|
||||
assert_eq!(policy.multipart_wait_timeout_for_test(), Some(Duration::from_secs(10)));
|
||||
});
|
||||
|
||||
let overridden = [
|
||||
(rustfs_config::ENV_PUT_FOREGROUND_ADMISSION_ENABLE, None::<&str>),
|
||||
(rustfs_config::ENV_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE, None),
|
||||
(rustfs_config::ENV_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS, Some("60000")),
|
||||
];
|
||||
temp_env::with_vars(overridden, || {
|
||||
let policy = ForegroundWriteAdmissionPolicy::from_env(64);
|
||||
// The bound applies to the default only; operators may still raise the wait.
|
||||
assert_eq!(policy.multipart_wait_timeout_for_test(), Some(Duration::from_secs(60)));
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_concurrency_manager_derives_multipart_admission_max_pending_from_limit() {
|
||||
assert_eq!(derive_multipart_admission_max_pending(7, 32), 7);
|
||||
|
||||
Reference in New Issue
Block a user