mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-02 11:29:17 +00:00
Zhengchao An
b7805caa58
The four consolidated ci keys plus the build keys still do not fit the repository's fixed 10GB Actions cache quota, so LRU keeps evicting them: measured demand is ci-dev 2331MB + ci-feat-proto 2310MB + ci-feat-rio 1951MB + ci-uring 1317MB + two build legs at ~1429MB + cargo-deny 844MB, and main pushes add two more build legs. That is roughly 14.4GB against 10.24GB. The symptom is misleading: Cache Warm reports every job successful and the restores log "full match: true", yet ci-feat-rio and ci-uring disappear from the cache list between runs. cache-all-crates was the wrong default for this repository. With it set to true, rust-cache's cleanup returns before pruning ~/.cargo/registry/src and its config archives the whole registry, so every cache carried the unpacked source tree of every dependency — not, as the name suggests, just a few extra crates. Setting it to false is rust-cache's own default and loses no coverage: the package set comes from `cargo metadata --all-features`, a strict superset of any single lane's feature closure; -sys crates are explicitly exempt from pruning, since their source timestamps would otherwise trigger rebuilds; and everything pruned is re-unpacked from the .crate files still in registry/cache, whose mtimes crates.io normalises, so cargo fingerprints stay valid. Applied to the setup composite and to audit.yml's own rust-cache. Cache Warm now also reports the sizes of registry/src, registry/cache, registry/index, ~/.cargo/git and target/ to the step summary, immediately before rust-cache's post step archives them, so the size of the effect is measured rather than assumed. The new sizes only appear once the cache key next rotates, since rust-cache skips the save entirely on an exact key hit. Deliberately not forcing that by bumping prefix-key: it would invalidate every family at once and produce a repository-wide cold build. Refs: rustfs/backlog#1598, rustfs/backlog#1600
78 lines
3.5 KiB
Bash
Executable File
78 lines
3.5 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Assert that self-hosted runners are ephemeral: one job per runner, ever.
|
|
#
|
|
# WHY THIS IS CHECKED CONTINUOUSLY RATHER THAN ONCE
|
|
#
|
|
# rustfs/rustfs is public, and pull_request jobs run on self-hosted runners —
|
|
# they execute the PR's own build.rs, proc-macros and tests, which is arbitrary
|
|
# code. What keeps that from reaching the release build is that each runner is a
|
|
# fresh ARC pod that handles exactly one job and is destroyed. Nothing in this
|
|
# repository enforces it: it is a property of the ARC scale-set configuration,
|
|
# which lives outside the repo and can be changed without any PR. So it is
|
|
# asserted from the outside, on real data.
|
|
#
|
|
# A repeated runner_name means a runner served two jobs, i.e. state survived
|
|
# between them, i.e. a poisoned PR job could leave something behind for whatever
|
|
# runs next — including, while the release build shares the sm-standard-2 pool,
|
|
# a release build.
|
|
#
|
|
# Usage: scripts/ci/check_runner_ephemerality.sh [run-count]
|
|
# run-count defaults to 40. MIN_SAMPLE (default 10) is the number of observed
|
|
# assignments below which the window is reported INCONCLUSIVE rather than OK.
|
|
#
|
|
# Exit codes: 0 ephemeral, 1 a runner served two jobs, 2 inconclusive or broken.
|
|
set -euo pipefail
|
|
|
|
REPO="${GITHUB_REPOSITORY:-rustfs/rustfs}"
|
|
LIMIT="${1:-40}"
|
|
# Below this many observed assignments the window says nothing useful: with the
|
|
# pool saturated, most sm-* jobs sit queued with runner_name still null, and a
|
|
# handful of samples would pass by coincidence rather than by evidence.
|
|
MIN_SAMPLE="${MIN_SAMPLE:-10}"
|
|
|
|
command -v gh >/dev/null || { echo "ERROR: gh CLI not found" >&2; exit 2; }
|
|
|
|
runs="$(gh run list --repo "$REPO" --limit "$LIMIT" --json databaseId --jq '.[].databaseId')"
|
|
[ -n "$runs" ] || { echo "ERROR: no runs returned for $REPO" >&2; exit 2; }
|
|
|
|
names="$(
|
|
for run in $runs; do
|
|
gh api "repos/${REPO}/actions/runs/${run}/jobs" --paginate \
|
|
--jq '.jobs[] | select(.runner_name != null) | select(.runner_name | startswith("sm-")) | .runner_name' 2>/dev/null || true
|
|
done
|
|
)"
|
|
|
|
total="$(printf '%s\n' "$names" | grep -c . || true)"
|
|
if [ "${total:-0}" -eq 0 ]; then
|
|
echo "ERROR: no self-hosted (sm-*) runner names found across $LIMIT runs." >&2
|
|
echo " Either the label scheme changed or the API shape did — this check" >&2
|
|
echo " must not silently pass by finding nothing." >&2
|
|
exit 2
|
|
fi
|
|
|
|
if [ "$total" -lt "$MIN_SAMPLE" ]; then
|
|
echo "INCONCLUSIVE: only ${total} self-hosted job assignments across ${LIMIT} runs" >&2
|
|
echo " (need ${MIN_SAMPLE}). Most sm-* jobs are probably still queued, so" >&2
|
|
echo " runner_name is null. Re-run with a larger window when the pool drains." >&2
|
|
exit 2
|
|
fi
|
|
|
|
dupes="$(printf '%s\n' "$names" | sort | uniq -d)"
|
|
|
|
if [ -n "$dupes" ]; then
|
|
echo "ERROR: self-hosted runners served more than one job — not ephemeral:" >&2
|
|
printf '%s\n' "$dupes" | while read -r name; do
|
|
count="$(printf '%s\n' "$names" | grep -c "^${name}$")"
|
|
echo " ${name}: ${count} jobs" >&2
|
|
done
|
|
echo "" >&2
|
|
echo "State can survive between jobs on those runners. Because this is a" >&2
|
|
echo "public repository whose pull_request jobs run PR-authored code, and" >&2
|
|
echo "because build.yml's release legs share the sm-standard-2 pool, a" >&2
|
|
echo "poisoned PR job could reach a release build. Check the ARC scale-set" >&2
|
|
echo "configuration (rustfs/backlog#1602)." >&2
|
|
exit 1
|
|
fi
|
|
|
|
echo "OK: ${total} self-hosted job assignments across ${LIMIT} runs, all on distinct runners"
|