fix(get): bound GET disk-read admission with a hard cap instead of unbounded permit bypass (#4935)

Refs https://github.com/rustfs/backlog/issues/1317

Previously, when the primary disk-read permit pool stayed saturated past `RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT` (default 5s) — which a handful of slow clients can cause because a permit is held for the whole body transfer — the GET path set `disk_permit = None` and continued reading with no admission token at all. That made the disk-read concurrency limit unbounded under exactly the overload it is meant to protect against: any number of GETs could pile onto the disks simultaneously.

This replaces the permit-less bypass with a bounded overflow lane and a hard cap:

- A new bounded degraded semaphore (`RUSTFS_OBJECT_DISK_DEGRADED_READ_CAP`, default mirrors the primary cap) is consulted only after the primary wait times out. A GET takes one degraded permit without blocking, or is rejected with `SlowDown`/503 once that lane is also full. The total number of GETs performing disk-active reads is therefore hard-capped at `primary_cap + degraded_cap`, and no GET ever reads without holding an admission token.
- Admission is centralized in `ConcurrencyManager::admit_disk_read`, returning `Primary`/`Degraded`/`Unbounded`/`Rejected`. `Degraded` and `Rejected` are counted in metrics (`rustfs.get_object.disk_permit.degraded.total`, `rustfs.get_object.disk_permit.hard_reject.total`), replacing the removed `rustfs.get_object.disk_permit.bypass.total`.
- A primary cap of `0` (disk-read throttling disabled) is preserved as the only intentional permit-less path via `Unbounded`, so that degenerate-but-served configuration is not turned into all-503. The `primary_wait == 0` wait-forever opt-out is likewise unchanged.

Healthy GETs are unaffected: concurrency at or below the primary cap is admitted immediately from the primary pool exactly as before, with no new latency or rejection. The rejection is surfaced before response headers are constructed, so it is a clean pre-header 503, never a post-header 504 masquerade. Degraded-lane GETs use the identical streaming reader and forward-progress stall timeout as primary GETs, so a slow but progressing large download is not killed. Owned permits (primary or degraded) are held by `DiskReadPermitReader` and released on body EOF or client drop/cancel, so tokens are always returned.

Body stall timeout already resets on forward progress and post-header failures already surface as body errors, so no timeout-split changes were needed here.

Scope: this PR implements the minimal safe correctness core (eliminate unbounded bypass + hard cap + SlowDown). The two-level weighted-fair + aging scheduler (small setup cap feeding a size-aware data-producer fair queue) and the strictly-bounded producer/client buffer with cancellation propagation from the issue plan remain follow-ups. The black-box slow-client soak is deferred to the unbuilt facility in https://github.com/rustfs/backlog/issues/1325 rather than faked.

Tests (white-box, virtual clock via `start_paused`): degrade-then-hard-reject with token release; 100 concurrent GETs against primary=1/degraded=1 never exceed 2 simultaneous admissions with every request either admitted or explicitly rejected; disabled cap serves unbounded; zero-wait blocks on the primary lane. Reverting the hard cap makes the concurrency invariant fail.
This commit is contained in:
Zhengchao An
2026-07-17 08:30:39 +08:00
committed by GitHub
parent a2a336aec3
commit 78469aa63b
6 changed files with 318 additions and 34 deletions
+25 -4
View File
@@ -73,15 +73,36 @@ pub const DEFAULT_OBJECT_MAX_CONCURRENT_DISK_READS: usize = 64;
/// - Example: `export RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT=5`
pub const ENV_OBJECT_DISK_PERMIT_WAIT_TIMEOUT: &str = "RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT";
/// Maximum time a GET request waits for a disk read permit (seconds).
/// Maximum time a GET request waits for a primary disk read permit (seconds).
///
/// Permits are held for the whole response body transfer, so slow clients can
/// occupy all of them while the disks sit idle. Instead of stalling until the
/// request-level timeout fires, a GET that waits longer than this proceeds
/// without a permit (degraded pass-through) and the bypass is counted in
/// metrics/logs. Set to 0 to wait indefinitely (previous behavior).
/// request-level timeout fires, a GET that waits longer than this falls through
/// to a bounded degraded admission lane; if that lane is also full the request
/// is rejected with `SlowDown`/503 rather than proceeding without any permit.
/// Set to 0 to wait on the primary lane indefinitely (never degrade or reject).
pub const DEFAULT_OBJECT_DISK_PERMIT_WAIT_TIMEOUT: u64 = 5;
/// Environment variable for the bounded degraded disk-read admission lane size.
/// - Purpose: Cap how many GETs may proceed after the primary disk-read permit
/// pool is saturated, giving a hard upper bound on concurrent disk-active
/// reads (primary cap + degraded cap) instead of an unbounded pass-through.
/// - Unit: request count (usize). `0` means "mirror the primary cap", so the
/// absolute hard cap defaults to twice the primary disk-read cap.
/// - Example: `export RUSTFS_OBJECT_DISK_DEGRADED_READ_CAP=16`
pub const ENV_OBJECT_DISK_DEGRADED_READ_CAP: &str = "RUSTFS_OBJECT_DISK_DEGRADED_READ_CAP";
/// Size of the bounded degraded disk-read admission lane.
///
/// When the primary disk-read permit pool is saturated and a GET exceeds
/// [`DEFAULT_OBJECT_DISK_PERMIT_WAIT_TIMEOUT`], it may take one permit from this
/// bounded overflow lane instead of reading without any admission token. The
/// total number of GETs performing disk-active reads is therefore hard-capped at
/// `primary_cap + degraded_cap`; beyond that a GET is rejected with `SlowDown`.
/// The default `0` mirrors the primary cap, so the hard cap is twice the primary
/// disk-read concurrency.
pub const DEFAULT_OBJECT_DISK_DEGRADED_READ_CAP: usize = 0;
/// Skip bitrot hash verification on GetObject reads.
///
/// When enabled, GetObject reads skip the per-shard hash