mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-02 03:19:19 +00:00
ci: bound every job's runtime and stop pasting inputs into shell (#5537)
Three hardening changes with no effect on what any workflow produces. Declare timeout-minutes on the 25 jobs that lacked it. GitHub's default is 360 minutes, and this repository has a history of runners stalling intermittently (#5394) plus a measured 9m57s plain `git checkout` under node-level I/O contention, so one wedged job could hold a runner for six hours out of a pool of roughly 15-21. Budgets follow what the jobs actually do: 10 minutes for echo-only and guard-script jobs, 30 for anything calling the GitHub API, uploading release assets or pushing over the network. scripts/security/check_job_timeouts.sh keeps it that way, checking only jobs that declare runs-on so reusable-workflow callers are not flagged. Pass workflow inputs and workflow_run fields through env instead of `${{ }}` interpolation in run blocks. A git ref name may contain `$(...)` — any string without a space is a legal tag — and interpolation pastes it into the script where bash evaluates it. The worst instance was helm-package's final commit message: it is built from the triggering tag name inside the job that holds the cross-repository push token with rustfs/helm already checked out. Also converted in build.yml, docker.yml and performance-ab.yml; the last is currently disabled, but a disabled workflow can be re-enabled. Not touched: helm-package's `contains(head_branch, '.')` tag test, since GitHub expressions have no regex and this repository's tags carry no `v` prefix, so rewriting the condition would change which builds publish a chart. Give audit.yml a scheduled-failure alert and run it daily. A scheduled cargo-deny failure usually means the dependency tree just matched a newly published RustSec advisory — the most important signal this workflow produces, and until now it was visible only to whoever happened to open the Actions tab. coverage.yml and e2e-replication-nightly.yml already use this ci-8 mechanism. The cron moves from weekly to daily so a new advisory against an unchanged tree surfaces within a day instead of seven; the check list is untouched, since splitting it into a light daily run and a weekly full run would create runs where sources, bans and licenses go unverified. Refs: rustfs/backlog#1598, rustfs/backlog#1602
This commit is contained in:
Executable
+65
@@ -0,0 +1,65 @@
|
||||
#!/usr/bin/env bash
|
||||
# Every job that occupies a runner must declare timeout-minutes.
|
||||
#
|
||||
# GitHub's default is 360 minutes. This repository has a history of runners
|
||||
# stalling intermittently (#5394) and of a plain `git checkout` taking 9m57s
|
||||
# under node-level I/O contention, so an undeclared timeout means one wedged job
|
||||
# can hold a runner for six hours out of a pool of roughly 15-21.
|
||||
#
|
||||
# Only jobs with `runs-on` are checked: a job that calls a reusable workflow has
|
||||
# no runner of its own and cannot declare a timeout.
|
||||
#
|
||||
# Usage: scripts/security/check_job_timeouts.sh
|
||||
set -euo pipefail
|
||||
|
||||
cd "$(dirname "$0")/../.."
|
||||
|
||||
status=0
|
||||
|
||||
for file in .github/workflows/*.yml .github/workflows/*.yaml; do
|
||||
[ -e "$file" ] || continue
|
||||
|
||||
awk -v file="$file" '
|
||||
function flush() {
|
||||
if (job != "" && has_runs_on && !has_timeout) {
|
||||
printf "%s:%d: job `%s` has runs-on but no timeout-minutes\n", file, job_line, job > "/dev/stderr"
|
||||
bad++
|
||||
}
|
||||
job = ""; has_runs_on = 0; has_timeout = 0
|
||||
}
|
||||
/^jobs:[[:space:]]*$/ { in_jobs = 1; next }
|
||||
{
|
||||
line = $0
|
||||
sub(/[[:space:]]+$/, "", line)
|
||||
if (line ~ /^[[:space:]]*#/ || line ~ /^[[:space:]]*$/) next
|
||||
|
||||
# A non-indented key ends the jobs mapping.
|
||||
if (in_jobs && line !~ /^[[:space:]]/) { flush(); in_jobs = 0 }
|
||||
if (!in_jobs) next
|
||||
|
||||
if (line ~ /^ [a-zA-Z0-9_-]+:[[:space:]]*$/) {
|
||||
flush()
|
||||
job = line
|
||||
sub(/^[[:space:]]*/, "", job)
|
||||
sub(/:.*$/, "", job)
|
||||
job_line = NR
|
||||
next
|
||||
}
|
||||
if (job == "") next
|
||||
if (line ~ /^ runs-on:/) has_runs_on = 1
|
||||
if (line ~ /^ timeout-minutes:/) has_timeout = 1
|
||||
}
|
||||
END { flush(); exit (bad > 0) }
|
||||
' "$file" || status=1
|
||||
done
|
||||
|
||||
if [ "$status" -ne 0 ]; then
|
||||
echo "" >&2
|
||||
echo "Add timeout-minutes to each job above. Rough budgets used in this repo:" >&2
|
||||
echo " 10 echo-only and guard-script jobs" >&2
|
||||
echo " 30 jobs that call the GitHub API, upload assets, or push over the network" >&2
|
||||
echo " 90+ anything using ./.github/actions/setup (cold cache restore alone is 11-21 min)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "OK: every job with runs-on declares timeout-minutes"
|
||||
Reference in New Issue
Block a user