mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-21 11:46:37 +00:00
ci(e2e-go): propagate cancel to remote .test procs + nuke VM before destroy
Two related leaks made cancelled runs strand QEMU work on the libvirt host: 1. The long `ssh tf-user@VM "for SUITE in …; do ./$SUITE.test"` ran without -tt, so when the runner cancelled the step the local ssh died but sshd never sent SIGHUP through (no PTY). The remote `.test` binary orphaned and kept running until its own 30m `-test.timeout`, holding tmp dirs, NATS connections, and qcow2 file handles. 2. Teardown went straight to `tofu destroy` without first killing any stragglers on the VM, so destroy raced the .test's open file handles on the root disk — sometimes successfully, sometimes leaving the libvirt host with a half-released qcow2. Fix: - Allocate a PTY on the run-suites ssh (`-tt`) so sshd HUPs children when the local ssh dies. - Add a trap (EXIT INT TERM) in the run-suites step that ssh's back to the VM and pkill's `/spinifex-e2e-bin/*.test` belt-and-braces. - Mirror the same pkill at the top of Teardown so destroy runs against a quiesced VM. Both cleanups are scoped to the per-run wan_ip — safe to land while multiple ci-single runner agents run concurrent jobs against different tofu envs on the same host (we deliberately keep that parallelism). Refs: runner host wedged 2026-05-19 (post-cancel orphans suspected; no journal access yet to confirm).
This commit is contained in:
@@ -176,8 +176,25 @@ jobs:
|
||||
# suite gets its own ::group:: + cyan banner inside so the Actions
|
||||
# UI collapses suite output by default; expand only what failed.
|
||||
SUITES_RUN=$(echo "${SUITES}" | sed 's/e2e-//g')
|
||||
ssh -o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}" \
|
||||
tf-user@${{ steps.vm.outputs.wan_ip }} \
|
||||
VM_HOST="tf-user@${{ steps.vm.outputs.wan_ip }}"
|
||||
SSH_KEY="${{ steps.vm.outputs.ssh_key }}"
|
||||
# On cancel / runner SIGTERM / step timeout, force-kill any test
|
||||
# binaries left running on the VM. Without this, the remote `.test`
|
||||
# process keeps holding the suite's NATS connections + tmp dirs and
|
||||
# the subsequent Teardown's tofu-destroy races with its file
|
||||
# handles. -o ConnectTimeout=5 so a wedged VM doesn't block cleanup.
|
||||
cleanup_remote() {
|
||||
ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 -i "$SSH_KEY" "$VM_HOST" \
|
||||
'pkill -u tf-user -TERM -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; true' \
|
||||
2>/dev/null || true
|
||||
}
|
||||
trap cleanup_remote EXIT INT TERM
|
||||
# `ssh -tt` forces PTY allocation. When the runner cancels this
|
||||
# step, the local ssh dies and the remote sshd sends SIGHUP through
|
||||
# the PTY to its children — without -tt the remote `.test` would
|
||||
# orphan and keep running until its own 30m timeout.
|
||||
ssh -tt -o StrictHostKeyChecking=no -i "$SSH_KEY" \
|
||||
"$VM_HOST" \
|
||||
"set -u; mkdir -p '${ARTIFACT_DIR}'; \
|
||||
cd \$HOME/spinifex-e2e-bin; \
|
||||
R=\$'\033[0m' B=\$'\033[1m' C=\$'\033[36m' G=\$'\033[32m' RED=\$'\033[31m'; \
|
||||
@@ -263,6 +280,18 @@ jobs:
|
||||
set +e
|
||||
# shellcheck source=/dev/null
|
||||
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
|
||||
# Kill any test binaries still running on the VM (cancelled mid-suite
|
||||
# paths) BEFORE tofu destroy yanks the qcow2 from underneath them.
|
||||
# Scoped to this run's VM by wan_ip — safe with sibling agents on
|
||||
# the same host. ConnectTimeout=5 so an already-wedged VM doesn't
|
||||
# block teardown.
|
||||
if [ -n "${{ steps.vm.outputs.wan_ip }}" ] && [ -n "${{ steps.vm.outputs.ssh_key }}" ]; then
|
||||
ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
|
||||
-i "${{ steps.vm.outputs.ssh_key }}" \
|
||||
tf-user@${{ steps.vm.outputs.wan_ip }} \
|
||||
'pkill -u tf-user -TERM -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; sleep 1; pkill -u tf-user -KILL -f "/spinifex-e2e-bin/.*\.test( |$)" 2>/dev/null; true' \
|
||||
2>/dev/null || true
|
||||
fi
|
||||
# Cancelled jobs from this point on can orphan an in-flight
|
||||
# tofu/libvirt SSH; clear them before destroy so the next run's
|
||||
# pre-clean isn't doing it.
|
||||
|
||||
Reference in New Issue
Block a user