mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-02 03:19:19 +00:00
ci: narrow the io_uring lane and make the sampler attributable (#5540)
Two independent changes to the Test and Lint area. The io_uring lane compiled 7 integration binaries to run none of their tests. Job log 91309868055 shows the lib target reporting "18 passed; 3453 filtered out" while every binary under crates/ecstore/tests/ reported "running 0 tests". Adding --lib drops them from the build without changing the selected set. The `uring_` filter itself must not be touched. libtest matches on substring, so it also selects names containing `during_` — 6 of the 18 selected tests are such incidental matches, and narrowing the filter to `io_uring` would silently drop them. scripts/check_uring_lane_lib_only.sh asserts the precondition --lib depends on: no test function whose name contains `uring_` may live under crates/ecstore/tests/. A count floor would not do, because the dangerous case — someone adding a matching test there — leaves the lib count unchanged and CI green. The resource sampler was measuring the wrong machine. The runners are ARC pods, so /proc/loadavg, /proc/pressure/*, `free` and `df` are node-level and include every other runner pod on the same Kubernetes node: one sample reported loadavg 6.67 with 832 threads node-wide while `ps` inside the pod showed about 10 processes. Judging a CARGO_BUILD_JOBS change on those numbers cannot work. The sampler moves to scripts/ci/resource_sampler.sh and now records /sys/fs/cgroup cpu.max, memory.max, memory.peak and the cpu/io/memory pressure files, which are this pod's own. The node-level readings stay — co-tenancy is a real cause of stalls, and a 9m57s plain `git checkout` was traced to it — but are labelled NODE-LEVEL so nobody reads them as this job's load. Phase markers are written on start, and clippy is now sampled too: it is the natural control arm for a CARGO_BUILD_JOBS experiment, since --all-targets is check-only for workspace members and never links the ~100 test binaries the limit throttles. No numbers are changed in this commit. CARGO_BUILD_JOBS stays at 2 until there is attributable data to change it on. Refs: rustfs/backlog#1598, rustfs/backlog#1601
This commit is contained in:
Zhengchao An
Executable
+41
@@ -0,0 +1,41 @@
|
||||
#!/usr/bin/env bash
|
||||
# The io_uring CI lane runs `cargo test -p rustfs-ecstore --lib uring_`.
|
||||
#
|
||||
# `--lib` is there to avoid compiling and linking the 7 integration binaries
|
||||
# under crates/ecstore/tests/, which selected zero tests for this filter. That is
|
||||
# only safe while it stays true: add a matching test under tests/ and `--lib`
|
||||
# would skip it silently, with CI staying green — the failure mode is invisible,
|
||||
# so it is asserted here instead.
|
||||
#
|
||||
# The pattern must use the same `uring_` substring semantics as libtest, which
|
||||
# also matches names containing `during_`. Narrowing it to `io_uring` would make
|
||||
# this guard disagree with the filter it is protecting.
|
||||
#
|
||||
# Usage: scripts/check_uring_lane_lib_only.sh
|
||||
set -euo pipefail
|
||||
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
TESTS_DIR="crates/ecstore/tests"
|
||||
|
||||
if [ ! -d "$TESTS_DIR" ]; then
|
||||
echo "OK: $TESTS_DIR does not exist; nothing for --lib to miss"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
hits="$(grep -rEn '^[[:space:]]*(pub[[:space:]]+)?(async[[:space:]]+)?fn[[:space:]]+[A-Za-z0-9_]*uring_' \
|
||||
"$TESTS_DIR" || true)"
|
||||
|
||||
if [ -n "$hits" ]; then
|
||||
echo "ERROR: test functions matching the 'uring_' substring found under $TESTS_DIR:" >&2
|
||||
printf '%s\n' "$hits" | sed 's/^/ /' >&2
|
||||
echo "" >&2
|
||||
echo "The io_uring lane in .github/workflows/ci.yml uses --lib, so these would" >&2
|
||||
echo "never run and CI would stay green. Pick one:" >&2
|
||||
echo " - move them into a #[cfg(test)] mod under crates/ecstore/src, or" >&2
|
||||
echo " - drop --lib from that step (and pay the link cost for all 7 binaries), or" >&2
|
||||
echo " - add an explicit --test <name> for the binary and update this script." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "OK: no 'uring_'-matching test functions under $TESTS_DIR; --lib is safe"
|
||||
Executable
+107
@@ -0,0 +1,107 @@
|
||||
#!/usr/bin/env bash
|
||||
# Background resource sampler for the Test and Lint lane (issue #5394).
|
||||
#
|
||||
# The post-mortem `pgrep` in ci.yml runs only after `timeout` has already TERM'd
|
||||
# the whole cargo process group, so it can never name a wedged process. This
|
||||
# samples every 60s instead; the last samples before a timeout show what was
|
||||
# stuck — rustc, the linker, a build script, memory pressure.
|
||||
#
|
||||
# WHY THE CGROUP READINGS MATTER
|
||||
#
|
||||
# The runners are ARC (Actions Runner Controller) pods, so /proc/loadavg,
|
||||
# /proc/pressure/*, `free` and `df` are all *node*-level: they include every
|
||||
# other runner pod scheduled on the same Kubernetes node. A sample showing high
|
||||
# I/O pressure says nothing about whether *this* job caused it. Measured
|
||||
# evidence: one sample reported loadavg 6.67 with 832 threads node-wide while
|
||||
# `ps` inside the pod showed about 10 processes.
|
||||
#
|
||||
# Only the /sys/fs/cgroup/* values and `ps` are attributable to this job, so
|
||||
# those are what any CARGO_BUILD_JOBS experiment must be judged on. The
|
||||
# node-level readings are kept — co-tenancy is itself a real cause of stalls,
|
||||
# and a 9m57s `git checkout` was traced to it — but labelled NODE-LEVEL so
|
||||
# nobody reads them as this pod's own load.
|
||||
#
|
||||
# Usage:
|
||||
# scripts/ci/resource_sampler.sh start <phase> # e.g. clippy, nextest, doctest
|
||||
# scripts/ci/resource_sampler.sh stop
|
||||
#
|
||||
# Safe to call `stop` when nothing is running, and safe to `start` a new phase
|
||||
# while another is sampling — the previous one is stopped first. Never fails the
|
||||
# calling step: this is diagnostics, not a gate.
|
||||
set -uo pipefail
|
||||
|
||||
# GNU date on the runners; the fallback keeps local smoke tests on macOS quiet.
|
||||
now() { date --utc --iso-8601=seconds 2>/dev/null || date -u +%Y-%m-%dT%H:%M:%S+00:00; }
|
||||
|
||||
LOG_DIR="artifacts/test-and-lint"
|
||||
LOG_FILE="${LOG_DIR}/sampler.log"
|
||||
PID_FILE="${LOG_DIR}/.sampler.pid"
|
||||
INTERVAL="${SAMPLER_INTERVAL_SECS:-60}"
|
||||
|
||||
stop_sampler() {
|
||||
[ -f "$PID_FILE" ] || return 0
|
||||
local pid
|
||||
pid="$(cat "$PID_FILE" 2>/dev/null || true)"
|
||||
if [ -n "${pid:-}" ]; then
|
||||
kill "$pid" 2>/dev/null || true
|
||||
fi
|
||||
rm -f "$PID_FILE"
|
||||
}
|
||||
|
||||
sample_once() {
|
||||
echo "=== $(now)"
|
||||
|
||||
# Attributable to this pod: cgroup v2 accounting and our own process table.
|
||||
echo "--- POD cpu/mem limits"
|
||||
grep -H . /sys/fs/cgroup/cpu.max /sys/fs/cgroup/memory.max \
|
||||
/sys/fs/cgroup/memory.peak /sys/fs/cgroup/memory.current 2>/dev/null || true
|
||||
echo "--- POD pressure (cgroup v2, this pod only)"
|
||||
grep -H . /sys/fs/cgroup/cpu.pressure /sys/fs/cgroup/io.pressure \
|
||||
/sys/fs/cgroup/memory.pressure 2>/dev/null || true
|
||||
echo "--- POD nproc"; nproc 2>/dev/null || true
|
||||
|
||||
# NODE-LEVEL: shared with every other runner pod on this Kubernetes node.
|
||||
# Useful for spotting co-tenancy, useless for attributing load to this job.
|
||||
echo "--- NODE-LEVEL load (shared with co-tenant runners)"; cat /proc/loadavg 2>/dev/null || true
|
||||
echo "--- NODE-LEVEL psi (shared with co-tenant runners)"
|
||||
grep -H . /proc/pressure/* 2>/dev/null || true
|
||||
echo "--- NODE-LEVEL mem (shared with co-tenant runners)"; free -m 2>/dev/null || true
|
||||
echo "--- NODE-LEVEL disk (shared with co-tenant runners)"
|
||||
df -h / /home/runner 2>/dev/null || df -h / 2>/dev/null || true
|
||||
|
||||
echo "--- top-rss"
|
||||
ps -eo pid,ppid,stat,etime,rss,pcpu,args --sort=-rss 2>/dev/null | head -15 || true
|
||||
echo "--- build/test processes"
|
||||
ps -eo pid,ppid,stat,etime,rss,pcpu,args 2>/dev/null \
|
||||
| grep -E '[c]argo|[r]ustc|[n]extest|[c]ollect2|rust-ll[d]|[b]uild-script|deps[/]' || true
|
||||
echo "--- d-state (uninterruptible IO)"
|
||||
ps -eo pid,stat,etime,args 2>/dev/null | awk 'NR > 1 && $2 ~ /D/' || true
|
||||
echo
|
||||
}
|
||||
|
||||
case "${1:-}" in
|
||||
start)
|
||||
phase="${2:-unknown}"
|
||||
mkdir -p "$LOG_DIR"
|
||||
stop_sampler
|
||||
{
|
||||
echo "--- phase=${phase} started_at=$(now) runner=${RUNNER_NAME:-unknown}"
|
||||
} >> "$LOG_FILE" 2>&1 || true
|
||||
(
|
||||
while true; do
|
||||
sample_once >> "$LOG_FILE" 2>&1 || true
|
||||
sleep "$INTERVAL"
|
||||
done
|
||||
) &
|
||||
echo $! > "$PID_FILE"
|
||||
;;
|
||||
stop)
|
||||
stop_sampler
|
||||
;;
|
||||
*)
|
||||
echo "usage: $0 start <phase> | stop" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
|
||||
exit 0
|
||||
Reference in New Issue
Block a user