test(ecstore): fit concurrent-resend guard inside lock acquire ceiling (#4384)

Under a full nextest run on loaded machines the legitimately serialized
cross-disk commits in concurrent_resend_same_part_commits_one_generation
exceed the acquire deadline by themselves: observed at the 5s default,
at the 30s production default (#4370), and on CI even at 60s — which is
a hard ceiling, because fast_lock clamps every requested timeout to
MAX_ACQUIRE_TIMEOUT (60s) while the Timeout error still reports the
requested value, so raising the env override higher is a no-op.

Request the full 60s ceiling explicitly and cap the queue depth at three
concurrent resends, bounding the last waiter to two serialized commits
(~12s each on the slowest observed CI runner). Three resends still race
the streaming phase and contend on the commit lock, which is all the
generation-mixing regression (backlog#853) needs; every correctness
assertion is unchanged.

Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
2026-07-08 04:01:27 +08:00
committed by GitHub
parent 86940a9452
commit 9ae4ca5f99
@@ -4710,7 +4710,7 @@ mod tests {
// Distinct payloads with distinct sizes: a mixed-generation reassembly // Distinct payloads with distinct sizes: a mixed-generation reassembly
// would produce bytes matching none of them (or fail the read outright). // would produce bytes matching none of them (or fail the read outright).
let candidates: Vec<Vec<u8>> = (0..6) let candidates: Vec<Vec<u8>> = (0..3)
.map(|g| { .map(|g| {
let len = 4096 + g * 512; let len = 4096 + g * 512;
vec![b'a' + g as u8; len] vec![b'a' + g as u8; len]
@@ -4722,14 +4722,22 @@ mod tests {
// 1. A lost/stolen fast-lock wakeup could strand a waiter until the // 1. A lost/stolen fast-lock wakeup could strand a waiter until the
// deadline — fixed for real in fast_lock::shard by bounding each // deadline — fixed for real in fast_lock::shard by bounding each
// notification wait (NOTIFY_WAIT_CAP re-polling). // notification wait (NOTIFY_WAIT_CAP re-polling).
// 2. Under the full nextest suite on loaded CI disks, the six // 2. Under the full nextest suite on loaded CI disks, the
// *legitimately serialized* cross-disk commits can exceed the small // *legitimately serialized* cross-disk commits can exceed the
// default acquire timeout (5s) all by themselves — observed on CI // acquire deadline all by themselves — observed on CI at the 5s
// even with fix 1 in place. Raise the timeout to the production // default and the 30s production default with six resends, and
// default (30s) for the concurrent section so the guard asserts the // again at 60s, which is a hard ceiling: fast_lock clamps every
// correctness property (exactly one intact generation), not disk // requested timeout to MAX_ACQUIRE_TIMEOUT (60s), so raising the
// latency. `#[serial]` keeps the process-wide env override isolated. // env override higher is a no-op (the Timeout error still reports
let results = temp_env::async_with_vars([(rustfs_config::ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT, Some("30"))], async { // the requested value). Keep the guard about the correctness
// property, not disk latency: request the full 60s ceiling and cap
// the queue depth at three resends, so the last waiter sits behind
// at most two serialized commits (~12s each on the slowest observed
// CI runner, comfortably inside the deadline). Three concurrent
// resends still race the streaming phase and contend on the commit
// lock, which is all the generation-mixing regression needs.
// `#[serial]` keeps the process-wide env override isolated.
let results = temp_env::async_with_vars([(rustfs_config::ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT, Some("60"))], async {
let mut tasks = tokio::task::JoinSet::new(); let mut tasks = tokio::task::JoinSet::new();
for payload in candidates.iter().cloned() { for payload in candidates.iter().cloned() {
let store = ecstore.clone(); let store = ecstore.clone();