Files
Clint Branham 9817d30a11 feat(ci): multi-node PVE provisioning and Docker-based shared storage
Provision two PVE nodes per version (a/b) for future cluster testing,
plus Docker-based iSCSI target and NFS server for shared storage tests.

Multi-node changes:
- Each PVE version gets two nodes: 9a/9b and 8a/8b (4 VMs total)
- Parameterized answer.toml FQDN for unique hostnames per node
- Per-node auto-install ISOs with unique answer files
- Node name discovered from FQDN and included in test config
- API token creation handles pre-existing tokens (delete + recreate)
- New test env vars: PVETEST_HOST_B, PVETEST_APITOKEN_B
- Removed preflight cleanup from provision (use explicit cleanup instead)

Docker storage services:
- New docker-compose.storage.yml with iSCSI (tgt) and NFS containers
- Host networking so PVE nodes can reach storage services
- Docker socket mounted into dev-infra container for host Docker access
- Docker CLI added to dev-infra Dockerfile stage
- New test env vars: PVETEST_STORAGE_VM_IP, PVETEST_ISCSI_IQN, PVETEST_NFS_EXPORT

Other fixes:
- first-boot.sh installs open-iscsi on PVE nodes
- preflight-cleanup.sh handles empty ISO filename gracefully
- TMPDIR set to work dir to avoid /tmp overflow during ISO uploads

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 13:34:15 -05:00

94 lines
3.5 KiB
Bash
Executable File

#!/usr/bin/env bash
# Pre-flight cleanup for CI runs.
# Ensures no leftover resources from a previous failed run before starting fresh.
#
# Usage: preflight-cleanup.sh <pve-endpoint> <api-token> <vm-id> <iso-filename> <terraform-dir>
set -uo pipefail
# Note: not using -e — we want to attempt all cleanup steps even if some fail
PVE_ENDPOINT="${1%/}"
API_TOKEN="$2"
VM_ID="$3"
ISO_FILENAME="$4"
TF_DIR="$5"
API_BASE="${PVE_ENDPOINT}/api2/json"
# Discover node name
NODES_JSON=$(curl -sk -H "Authorization: PVEAPIToken=${API_TOKEN}" "${API_BASE}/nodes" 2>/dev/null)
NODE=$(echo "$NODES_JSON" | python3 -c "import json,sys; print(json.load(sys.stdin)['data'][0]['node'])" 2>/dev/null || echo "pve")
echo "=== Pre-flight cleanup (node: ${NODE}, vmid: ${VM_ID}) ==="
# --- Clean up orphaned VM ---
VM_STATUS=$(curl -sk -H "Authorization: PVEAPIToken=${API_TOKEN}" \
"${API_BASE}/nodes/${NODE}/qemu/${VM_ID}/status/current" 2>/dev/null \
| python3 -c "import json,sys; print(json.load(sys.stdin).get('data',{}).get('status',''))" 2>/dev/null || true)
if [ -n "$VM_STATUS" ]; then
echo "Found orphaned VM ${VM_ID} (status: ${VM_STATUS})"
if [ "$VM_STATUS" = "running" ]; then
echo " Stopping VM..."
curl -sk -X POST -H "Authorization: PVEAPIToken=${API_TOKEN}" \
"${API_BASE}/nodes/${NODE}/qemu/${VM_ID}/status/stop" >/dev/null 2>&1
# Wait for stop
for i in $(seq 1 12); do
sleep 5
S=$(curl -sk -H "Authorization: PVEAPIToken=${API_TOKEN}" \
"${API_BASE}/nodes/${NODE}/qemu/${VM_ID}/status/current" 2>/dev/null \
| python3 -c "import json,sys; print(json.load(sys.stdin).get('data',{}).get('status',''))" 2>/dev/null || true)
if [ "$S" = "stopped" ]; then break; fi
done
fi
echo " Deleting VM..."
curl -sk -X DELETE -H "Authorization: PVEAPIToken=${API_TOKEN}" \
"${API_BASE}/nodes/${NODE}/qemu/${VM_ID}?destroy-unreferenced-disks=1&purge=1" >/dev/null 2>&1
sleep 3
echo " VM cleanup done"
else
echo "No orphaned VM ${VM_ID} found"
fi
# --- Clean up orphaned ISO ---
if [ -z "$ISO_FILENAME" ]; then
echo "No ISO filename specified, skipping ISO cleanup"
else
ISO_EXISTS=$(curl -sk -H "Authorization: PVEAPIToken=${API_TOKEN}" \
"${API_BASE}/nodes/${NODE}/storage/local/content" 2>/dev/null \
| python3 -c "
import json, sys
data = json.load(sys.stdin).get('data', [])
for item in data:
if item.get('volid', '').endswith('/${ISO_FILENAME}'):
print(item['volid'])
break
" 2>/dev/null || true)
if [ -n "$ISO_EXISTS" ]; then
echo "Found orphaned ISO: ${ISO_EXISTS}"
echo " Deleting..."
ENCODED=$(python3 -c "import urllib.parse; print(urllib.parse.quote('${ISO_EXISTS}', safe=''))")
curl -sk -X DELETE -H "Authorization: PVEAPIToken=${API_TOKEN}" \
"${API_BASE}/nodes/${NODE}/storage/local/content/${ENCODED}" >/dev/null 2>&1
sleep 2
echo " ISO cleanup done"
else
echo "No orphaned ISO found"
fi
fi # end ISO_FILENAME check
# --- Clean up stale Terraform state ---
if [ -d "$TF_DIR" ]; then
if [ -f "${TF_DIR}/.terraform.tfstate.lock.info" ]; then
echo "Found stale Terraform lock, removing..."
rm -f "${TF_DIR}/.terraform.tfstate.lock.info"
fi
if [ -f "${TF_DIR}/terraform.tfstate" ]; then
echo "Found stale Terraform state, removing..."
rm -f "${TF_DIR}/terraform.tfstate" "${TF_DIR}/terraform.tfstate.backup"
fi
else
echo "Terraform dir not found (clean checkout)"
fi
echo "=== Pre-flight cleanup complete ==="