fix(ci): take the host benchmark lock before killing predastore nodes (#1065)

cell-24 clears leftover predastore nodes with `pkill -KILL -f '/bin/s3d
-config'`, which matches every predastore node on the runner and not only the
ones this run left behind. The predastore repository runs the same harnesses on
the same machine, so "still running" does not mean "stale" — on 2026-09-16 this
step killed a predastore benchmark 32 seconds into its workload, and the run
failed with 61 connection-refused errors that look nothing like their cause.

Both kills now run under the host benchmark lock the harnesses themselves take,
so anything alive while it is held really is a leftover. The second one needs it
as much as the first: by the time cleanup runs the harness has released the
lock, so another repository's benchmark can already be up.
This commit is contained in:
Ben Duncan
2026-09-17 10:59:39 -05:00
committed by GitHub
parent 451df1a8cd
commit 2039c05c17
+14 -2
View File
@@ -1493,8 +1493,16 @@ jobs:
# Leftovers from a run killed before its cleanup trap: a surviving node
# holds its ports and fails this run somewhere unrelated.
#
# Under the host benchmark lock, because this pattern matches every
# predastore node on the runner and the predastore repository runs the
# same harnesses on the same machine. Holding the lock is what makes
# "still running" mean "stale" rather than "somebody else's benchmark":
# on 2026-09-16 this step killed a predastore run 32s into its workload.
- name: Clear stale nodes
run: pkill -KILL -f '/bin/s3d -config' || true
run: |
flock -w 3600 "${PREDA_BENCH_LOCK:-/var/lock/predastore-bench.lock}" \
pkill -KILL -f '/bin/s3d -config' || true
- name: Run fault injection
working-directory: predastore
@@ -1514,10 +1522,14 @@ jobs:
if-no-files-found: warn
retention-days: 90
# Under the lock for the same reason as "Clear stale nodes": by the time
# this runs the harness has released it, so another repository's benchmark
# can already have a cluster of its own up.
- name: Clean up
if: always()
run: |
pkill -KILL -f '/bin/s3d -config' || true
flock -w 3600 "${PREDA_BENCH_LOCK:-/var/lock/predastore-bench.lock}" \
pkill -KILL -f '/bin/s3d -config' || true
rm -rf "$STRESS_RESULTS_ROOT"
# Reading a nightly meant opening every cell job, and the Actions UI does not