mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-22 04:16:38 +00:00
perf(get,heal): fix GET hot-path overhead and heal checkpoint scaling (#4237)
perf(get,heal): land verified fixes for backlog #800-#804 Five fixes from the GET-path performance audit and scanner/heal completeness audit (rustfs/backlog#800..#804), each verified locally: - backlog#800 (heal checkpoint O(N^2)): ResumeCheckpoint object sets are now HashSet (Vec::contains was O(n) per healed object; 1.5ms at n=1M vs 11ns measured), and per-object checkpoint/resume-state persistence is batched (1000 mutations / 5s) instead of rewriting the whole file per object. complete_page() prunes the sets at page boundaries so memory stays bounded; positions still persist unconditionally and legacy Vec-format checkpoints still deserialize. - backlog#801 (DiskInfo.healing never set): erasure-set heal now writes a healing marker (.rustfs.sys/healing.bin) on the disks it rebuilds (endpoints plumbed via HealRequest/HealTask.heal_endpoints from the auto disk scanner) and clears it on success. LocalDisk::disk_info surfaces the marker, so scanner heal coordination, lock selection and admin/metrics healing counts see the rebuild. - backlog#802 (cache probe after data read): new GetObjectBodyCacheHook in ecstore lets the app-layer object data cache serve the body inside get_object_reader, after metadata quorum resolution (etag known) but before the erasure shard read/decode. Previously a hit still paid the full disk read. Hook is None/no-op when the cache is disabled. - backlog#803 (GET hot-path redundant work): ObjectInfo is cloned for event notification only when an event will actually be built (GET and HEAD paths; events are currently suppressed so the clone was pure waste); get_opts/put_opts/del_opts resolve bucket versioning with one metadata-sys lookup instead of two; skip_verify_bitrot and get_lock_acquire_timeout env reads are cached via OnceLock; the io-priority metric is no longer double-counted; GetObject input fields are cloned selectively instead of cloning the whole input. - backlog#804 (disk permit starvation): the disk-read permit wait is now bounded (RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT, default 5s, 0 = previous unbounded behavior); on timeout the GET proceeds without a permit and the bypass is counted. DiskReadPermitReader also releases the permit at body EOF instead of holding it until the client drops the stream. Verification: make pre-commit; cargo clippy -D warnings on the four changed crates; full rustfs lib suite (2096 tests) green; rustfs-heal lib suite green with new unit tests for checkpoint pruning/legacy format/throttle, permit EOF release, and the cache hook (hit + SSE skip). The heal_integration_test and one set_disk listing test fail identically on unmodified main (pre-existing global-state ordering flakes, verified via git stash A/B). Co-authored-by: houseme <housemecn@gmail.com>
This commit is contained in:
@@ -67,6 +67,21 @@ pub const DEFAULT_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD: usize = 4;
|
||||
/// Default is set to 64 concurrent reads.
|
||||
pub const DEFAULT_OBJECT_MAX_CONCURRENT_DISK_READS: usize = 64;
|
||||
|
||||
/// Environment variable for the disk read permit wait timeout (seconds).
|
||||
/// - Purpose: Bound how long a GET waits for a disk read permit before proceeding without one.
|
||||
/// - Unit: seconds (u64). `0` waits indefinitely.
|
||||
/// - Example: `export RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT=5`
|
||||
pub const ENV_OBJECT_DISK_PERMIT_WAIT_TIMEOUT: &str = "RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT";
|
||||
|
||||
/// Maximum time a GET request waits for a disk read permit (seconds).
|
||||
///
|
||||
/// Permits are held for the whole response body transfer, so slow clients can
|
||||
/// occupy all of them while the disks sit idle. Instead of stalling until the
|
||||
/// request-level timeout fires, a GET that waits longer than this proceeds
|
||||
/// without a permit (degraded pass-through) and the bypass is counted in
|
||||
/// metrics/logs. Set to 0 to wait indefinitely (previous behavior).
|
||||
pub const DEFAULT_OBJECT_DISK_PERMIT_WAIT_TIMEOUT: u64 = 5;
|
||||
|
||||
/// Skip bitrot hash verification on GetObject reads.
|
||||
///
|
||||
/// When enabled, GetObject reads skip the per-shard hash
|
||||
|
||||
Reference in New Issue
Block a user