From 59f69fbfb5073a61862d567528d799d1855541f1 Mon Sep 17 00:00:00 2001 From: Zhengchao An Date: Sat, 1 Aug 2026 20:00:31 +0800 Subject: [PATCH] ci: raise CARGO_BUILD_JOBS to 3 on main pushes as a measured experiment (#5571) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The #5394 mitigation set it to 2 on the belief that three concurrent workspace test links saturate the runner's overlay I/O. The sampler's cgroup v2 readings show the pod has 14 CPUs and 28GB with a 2.1GB peak, so 2 throttles compilation to a seventh of what is available and memory was never the constraint. The label name sm-standard-4 had led everyone, including that mitigation and every description written during this series, to assume four cores. Raised to 3 for push events only. PRs keep 2, so the merge path is untouched while the experiment runs. Baseline over 17 non-cancelled samples at 2: median nextest/clippy step ratio 1.95, spread 1.85-2.06. Judging by that ratio rather than absolute wall clock is what makes the signal usable — clippy runs on the same node at the same time and is check-only for workspace members, so it never links the ~100 test binaries this limit throttles, making it a control arm rather than a second measurement. The criterion is the ratio dropping at least 10%, below about 1.76, with no 75m timeout and no run showing three consecutive samples of rustc, collect2 or rust-lld in D state. If it does not drop, the conclusion is that this limit is not the bottleneck: fix it back at 2 and record the experiment. That is a result, not a failure. Kept at step level deliberately. rust-cache hashes CARGO/CC/CFLAGS/CXX/CMAKE/RUST prefixed variables from process.env into its key, so promoting this to job level would rotate every cache key on this lane. Refs: rustfs/backlog#1598, rustfs/backlog#1601 --- .github/workflows/ci.yml | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8eb28c886..4358ff5a8 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -216,9 +216,29 @@ jobs: - name: Run nextest tests env: - # Three concurrent workspace test links saturate the self-hosted - # runner's overlay I/O and can wedge Cargo until the 75m timeout. - CARGO_BUILD_JOBS: "2" + # #5394 mitigation, now under a measured experiment (backlog#1601). + # + # 2 was chosen when three concurrent workspace test links were believed + # to saturate the runner's overlay I/O and wedge Cargo until the 75m + # timeout. cgroup v2 readings from the sampler show the pod actually + # has 14 CPUs and 28GB (peak use 2.1GB), so 2 throttles compilation to + # a seventh of what is available and memory was never the constraint — + # the label name "sm-standard-4" had led everyone, including the + # original mitigation, to assume 4 cores. + # + # Raised to 3 on main pushes only; PRs keep 2 so the merge path is + # untouched while the experiment runs. Baseline over 17 samples at 2: + # median nextest/clippy step ratio 1.95, spread 1.85-2.06. The gate-2 + # criterion is that ratio dropping at least 10% (below ~1.76) with no + # 75m timeout and no run showing three consecutive samples of + # rustc/collect2/rust-lld in D state. If it does not, the conclusion is + # "this limit is not the bottleneck" — fix it back at 2 and record the + # experiment, which is a result, not a failure. + # + # Must stay step-level: rust-cache hashes CARGO/CC/CFLAGS/CXX/CMAKE/RUST + # prefixed variables from process.env into the cache key, so promoting + # this to job level would rotate every key on this lane. + CARGO_BUILD_JOBS: ${{ github.event_name == 'push' && '3' || '2' }} run: | mkdir -p artifacts/test-and-lint ./scripts/ci/resource_sampler.sh start nextest