mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-22 04:07:04 +00:00
joshsiv-mulga
660fe5227e
Three minimal edits to close the orphan-OVS-port failure mode that survives PR1a's systemd ordering fix (host crash, OOM-kill of ovsdb, operator manually stopping OVS on a live host) and the terminate-during-pending race that leaks taps when a user aborts a slow launch. 1. SetupTapDevice (network.go): unconditional `ovs-vsctl --if-exists del-port br-int <tap>` + separately-gated kernel `ip tuntap del`. The previous kernel-side-gated cleanup missed orphan OVS ports because OVS conf.db survives reboot while kernel taps don't. Rejected `--may-exist add-port`: it would silently keep stale external_ids:iface-id/attached-mac from a prior launch. 2. stopInstance mgmt-tap branch (daemon.go) + CleanupMgmtTapDevice (network.go): derive the mgmt tap name from instance.ID rather than reading instance.MgmtTap, since that field is set only after SetupMgmtTapDevice returns and a terminate racing mid-launch would skip cleanup. CleanupMgmtTapDevice gains a kernel-presence short-circuit so unconditional invocation doesn't log spurious "Device does not exist" errors for instances that never reached SetupMgmtTapDevice. 3. LaunchInstance (daemon.go): `launchStillValid` helper re-checks instance status before StartInstance and TransitionState, in addition to the entry guard. On detection of shutting-down / terminated, log Info and return nil — the concurrent terminate goroutine owns cleanup; this is an expected race outcome, not an error. Removes the spurious "Failed to transition instance to running" ERROR that surfaced on every legitimate terminate-during-pending. Plus the entry path now returns nil instead of an error for the same reason. Also lands `scripts/start-then-terminate-pending.sh` (added during investigation) which reproduces the runtime race for manual verification. Bead: mulga-siv-20.2 Plan: docs/development/bugs/orphan-tap-on-launch-race.md (PR1b)
58 lines
2.1 KiB
Bash
Executable File
58 lines
2.1 KiB
Bash
Executable File
#!/bin/sh
|
|
# Launch instance then terminate it while still in `pending` state.
|
|
|
|
set -e
|
|
|
|
export AWS_PROFILE=spinifex
|
|
|
|
ARCH=$(uname -m)
|
|
case "$ARCH" in
|
|
x86_64) IMAGE_NAME="ami-ubuntu-24.04-x86_64" ;;
|
|
aarch64|arm64) IMAGE_NAME="ami-ubuntu-24.04-arm64" ;;
|
|
*) echo "Unknown arch $ARCH, default x86_64"; IMAGE_NAME="ami-ubuntu-24.04-x86_64" ;;
|
|
esac
|
|
|
|
INSTANCE_TYPE=$(aws ec2 describe-instance-types \
|
|
--query "InstanceTypes[?contains(InstanceType, '.micro')].InstanceType | [0]" \
|
|
--output text)
|
|
[ -z "$INSTANCE_TYPE" ] || [ "$INSTANCE_TYPE" = "None" ] && { echo "no micro type"; exit 1; }
|
|
|
|
AMI=$(aws ec2 describe-images --query "Images[?Name=='$IMAGE_NAME'].ImageId" --output text)
|
|
[ -z "$AMI" ] && { echo "no AMI $IMAGE_NAME"; exit 1; }
|
|
|
|
SUBNET=$(aws ec2 describe-subnets --query 'Subnets[0].SubnetId' --output text)
|
|
SG=$(aws ec2 describe-security-groups --query 'SecurityGroups[0].GroupId' --output text)
|
|
[ -z "$SUBNET" ] || [ "$SUBNET" = "None" ] && { echo "no subnet"; exit 1; }
|
|
[ -z "$SG" ] || [ "$SG" = "None" ] && { echo "no SG"; exit 1; }
|
|
|
|
echo "Launch: ami=$AMI type=$INSTANCE_TYPE subnet=$SUBNET sg=$SG"
|
|
|
|
INSTANCE_ID=$(aws ec2 run-instances \
|
|
--image-id "$AMI" \
|
|
--instance-type "$INSTANCE_TYPE" \
|
|
--key-name spinifex-key \
|
|
--security-group-ids "$SG" \
|
|
--subnet-id "$SUBNET" \
|
|
--count 1 \
|
|
--query 'Instances[0].InstanceId' \
|
|
--output text)
|
|
|
|
echo "Launched: $INSTANCE_ID — waiting for QEMU"
|
|
|
|
# Wait until QEMU process exists (LaunchInstance.StartInstance fired) but
|
|
# BEFORE TransitionState moves to running. Poll tight; terminate on first hit.
|
|
TIMEOUT=60
|
|
DEADLINE=$(( $(date +%s) + TIMEOUT ))
|
|
QEMU_PID=""
|
|
while [ "$(date +%s)" -lt "$DEADLINE" ]; do
|
|
QEMU_PID=$(pgrep -f "qemu.*$INSTANCE_ID" | head -1 || true)
|
|
[ -n "$QEMU_PID" ] && break
|
|
done
|
|
|
|
[ -z "$QEMU_PID" ] && { echo "ERR: QEMU never appeared"; exit 1; }
|
|
|
|
echo "QEMU pid=$QEMU_PID — terminate now"
|
|
aws ec2 terminate-instances --instance-ids "$INSTANCE_ID" \
|
|
--query 'TerminatingInstances[0].[InstanceId,PreviousState.Name,CurrentState.Name]' \
|
|
--output text
|