mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-23 12:47:48 +00:00
feat: VFIO GPU passthrough (#188)
* feat: add GPU discovery package (mulga-992) Adds spinifex/gpu package with sysfs-based GPU enumeration, IOMMU group member lookup, and a PCI device ID model table. Vendor-specific enrichment via nvidia-smi is attempted opportunistically and falls back to the built-in table when unavailable. All discovery and IOMMU logic is covered by unit tests using mock sysfs trees. * feat: add VFIO bind/unbind and GPU pool manager (mulga-993) Adds gpu.BindVFIO/UnbindVFIO using sysfs driver_override for per-device driver switching, and gpu.Manager which owns the claim/release lifecycle for the GPU pool including full IOMMU group binding, rollback on partial failure, and marking entries unavailable when rebind fails after VM termination. * feat: add G5 instance type generator with GPU model support (mulga-994) - Add GPUModel type, NVIDIAa10g constant, and GPUModelForVendorDevice lookup - Add generateGPUTypes to emit g5.* types with GpuInfo populated - Add IsGPUType helper for consumers filtering GPU-capable types - Update DetectAndGenerate to accept gpuModels []GPUModel parameter * feat: integrate GPU passthrough into daemon lifecycle (mulga-995) - Add gpu_passthrough config flag; discover GPUs and build manager at startup - Gate g5 instance types on GPU pool availability via canAllocateCount - Claim GPU (VFIO bind) in LaunchInstance; inject vfio-pci QEMU device - Release GPU (VFIO unbind, rebind to host driver) in stopInstance - Re-claim GPU on daemon restart when QEMU process is still alive - Add UseUEFI / OVMF support for x86_64 VMs (optional, falls back to SeaBIOS) - Fix Claim to use stored OriginalDriver when device is already vfio-pci bound * feat: add GPU fields to NodeStatus and spx top nodes (mulga-996) - Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes) - Populate from gpuManager in handleNodeStatus - Show GPU (used/total) column in spx top nodes; "-" when no GPUs present * feat: Ubuntu 24.04 GPU AMI build pipeline (mulga-997) Extends build-system-image.sh to support Ubuntu alongside Alpine. Key differences handled: partition table (parted resizepart p1), bind mounts for DKMS (/proc /sys /dev), apt vs apk, systemctl vs rc-update. DISTRO defaults to "alpine" for backward compatibility. Adds ubuntu-gpu.conf manifest and ubuntu-gpu-setup.sh chroot script that installs nvidia-headless-550-server with DKMS pre-built against a pinned linux-image-generic kernel, blacklists nouveau, and prevents unattended-upgrades from invalidating the pre-built module. * Fixed GPU passthrough and added test script * feat: always-probe GPU at startup, surface capability in NodeStatusResponse - Add gpuProbeResult struct and probeGPU() which runs at daemon startup regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU groups, and vfio_pci module presence. - Store gpuProbe on Daemon struct; split NewDaemon into probe (always) and activate (opt-in) phases. - Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse so operators can see hardware state before enabling passthrough. - Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but passthrough not active, 'used/total' = passthrough active. * feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001) - Export GenerateGPUTypes for use by the reload path - Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place (preserves references held by instanceService) and updates NATS subscriptions without full daemon restart - Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP in a context-aware loop; setupShutdown no longer intercepts SIGHUP - Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal, handles false→true (re-probe + activate) and true→false (refuse if instances running, otherwise deactivate) transitions * feat: spx admin gpu status/enable/disable commands (mulga-1000) - admin.SetGPUPassthrough: Go implementation of the idempotent TOML write (replaces the inline Python heredoc in spx-test-gpu.sh) - spx admin gpu status: shows GPU hardware, IOMMU/vfio state, passthrough activation, and advertised g5 instance types via NATS NodeStatusResponse - spx admin gpu enable: checks prerequisites, writes TOML, sends SIGHUP, polls until daemon confirms GPUPassthrough=true - spx admin gpu disable: refuses while GPU instances are running, otherwise writes TOML, sends SIGHUP, polls until confirmed - spx-test-gpu.sh: replace Python TOML heredoc with spx admin gpu enable * fixed admin * fix: filter Intel iGPU from passthrough pool, use lspci for consumer GPU names Discover() now skips non-NVIDIA/AMD vendors so Intel iGPUs never enter the passthrough pool. enrichFromLspci() uses the system PCI ID database via lspci -vmm to populate the model name for any card not in the hardcoded table, without needing nvidia-smi or driver tools. * fix: grant CAP_SYS_ADMIN and vfio device access to spinifex-daemon GPU passthrough requires writing to /sys/bus/pci/drivers/vfio-pci/unbind on instance stop. The kernel gates this on CAP_SYS_ADMIN regardless of file permissions. QEMU subprocesses also need char-vfio rw access to open /dev/vfio/N for the passed-through device. * fixing gpu bining permissions * working state of gpu passthrough on phyiscal box * installer grub image fixes * made gpu-passthrough more user facing * fixed banner * fixed banner * banner fix * gpu passthrough docs * preflight tweak * vpcd fixes to set br-wan to promisc for public IP DHCP processes * fixed vpcd br-wan promisc and veth pair * ssh keygen fix * fixes after running /review-branch * feat: re-wire GPU claim and device injection after vm.Manager refactor GPU passthrough claim (VFIO bind) and QEMU device injection were lost when dev's vm.Manager refactor (#185/#186) moved StartInstance/ CreateQMPClient into vm/lifecycle.go. Re-add both in their correct new homes: - daemon_handlers_instance.go: claim GPU before vmMgr.Run() in both RunInstances and StartStoppedInstance handlers; release on failure - vm/lifecycle.go startQEMU(): inject vfio-pci device with x-vga flag derived from instance.GPUXVGAEnabled when GPUPCIAddress is set * fix: Create spinifex user ~/.ssh and keys in firstboot --------- Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com> Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
This commit is contained in:
tomnewton-mulga
+144
-45
@@ -1,10 +1,11 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
# build-system-image.sh — Build a minimal Alpine AMI from a manifest
|
||||
# build-system-image.sh — Build a minimal system AMI from a manifest
|
||||
#
|
||||
# Creates a pre-baked Alpine Linux image with custom packages, binaries,
|
||||
# and setup scripts installed, ready for import as a Spinifex AMI.
|
||||
# Supports Alpine Linux and Ubuntu 24.04 as base distros.
|
||||
# Creates a pre-baked image with custom packages, binaries, and setup scripts
|
||||
# installed, ready for import as a Spinifex AMI.
|
||||
#
|
||||
# Requirements: qemu-nbd, qemu-img, sudo (for mount/chroot), curl
|
||||
# Usage: ./scripts/build-system-image.sh <manifest.conf> [--import]
|
||||
@@ -46,8 +47,11 @@ done
|
||||
# shellcheck source=/dev/null
|
||||
source "$MANIFEST"
|
||||
|
||||
# Default distro is alpine for backward compatibility
|
||||
DISTRO="${DISTRO:-alpine}"
|
||||
|
||||
# Validate required manifest fields
|
||||
for field in IMAGE_NAME ALPINE_VERSION IMAGE_SIZE; do
|
||||
for field in IMAGE_NAME IMAGE_SIZE; do
|
||||
if [[ -z "${!field:-}" ]]; then
|
||||
echo "ERROR: Manifest missing required field: $field"
|
||||
exit 1
|
||||
@@ -56,16 +60,48 @@ done
|
||||
|
||||
# Derived paths
|
||||
BUILD_DIR="/tmp/${IMAGE_NAME}-image-build"
|
||||
ALPINE_IMAGE="generic_alpine-${ALPINE_VERSION}-x86_64-bios-cloudinit-r0.qcow2"
|
||||
ALPINE_URL="https://dl-cdn.alpinelinux.org/alpine/v${ALPINE_VERSION%.*}/releases/cloud/${ALPINE_IMAGE}"
|
||||
OUTPUT_IMAGE="${BUILD_DIR}/${IMAGE_NAME}-alpine.qcow2"
|
||||
OUTPUT_RAW="${BUILD_DIR}/${IMAGE_NAME}-alpine.raw"
|
||||
NBD_DEV="/dev/nbd0"
|
||||
MOUNT_DIR="${BUILD_DIR}/mnt"
|
||||
|
||||
case "$DISTRO" in
|
||||
alpine)
|
||||
if [[ -z "${ALPINE_VERSION:-}" ]]; then
|
||||
echo "ERROR: Manifest missing required field: ALPINE_VERSION"
|
||||
exit 1
|
||||
fi
|
||||
SOURCE_IMAGE="generic_alpine-${ALPINE_VERSION}-x86_64-bios-cloudinit-r0.qcow2"
|
||||
SOURCE_URL="https://dl-cdn.alpinelinux.org/alpine/v${ALPINE_VERSION%.*}/releases/cloud/${SOURCE_IMAGE}"
|
||||
OUTPUT_IMAGE="${BUILD_DIR}/${IMAGE_NAME}-alpine.qcow2"
|
||||
OUTPUT_RAW="${BUILD_DIR}/${IMAGE_NAME}-alpine.raw"
|
||||
DISTRO_VERSION="${ALPINE_VERSION}"
|
||||
ROOT_PART="${NBD_DEV}"
|
||||
;;
|
||||
ubuntu)
|
||||
if [[ -z "${UBUNTU_VERSION:-}" ]]; then
|
||||
echo "ERROR: Manifest missing required field: UBUNTU_VERSION"
|
||||
exit 1
|
||||
fi
|
||||
SOURCE_IMAGE="ubuntu-${UBUNTU_VERSION}-minimal-cloudimg-amd64.img"
|
||||
SOURCE_URL="https://cloud-images.ubuntu.com/minimal/releases/noble/release/${SOURCE_IMAGE}"
|
||||
OUTPUT_IMAGE="${BUILD_DIR}/${IMAGE_NAME}-ubuntu.qcow2"
|
||||
OUTPUT_RAW="${BUILD_DIR}/${IMAGE_NAME}-ubuntu.raw"
|
||||
DISTRO_VERSION="${UBUNTU_VERSION}"
|
||||
ROOT_PART="${NBD_DEV}p1"
|
||||
;;
|
||||
*)
|
||||
echo "ERROR: Unknown DISTRO: $DISTRO (supported: alpine, ubuntu)"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
cleanup() {
|
||||
echo "Cleaning up..."
|
||||
sudo umount "${MOUNT_DIR}" 2>/dev/null || true
|
||||
# Ubuntu chroot has bind mounts that must be released first
|
||||
sudo umount "${MOUNT_DIR}/dev/pts" 2>/dev/null || true
|
||||
sudo umount "${MOUNT_DIR}/dev" 2>/dev/null || true
|
||||
sudo umount "${MOUNT_DIR}/sys" 2>/dev/null || true
|
||||
sudo umount "${MOUNT_DIR}/proc" 2>/dev/null || true
|
||||
sudo umount "${MOUNT_DIR}" 2>/dev/null || true
|
||||
sudo qemu-nbd --disconnect "${NBD_DEV}" 2>/dev/null || true
|
||||
exec 9>&- 2>/dev/null || true # release nbd lock if held
|
||||
echo "Done."
|
||||
@@ -80,7 +116,7 @@ fi
|
||||
|
||||
echo "=== System Image Builder ==="
|
||||
echo "Image: ${IMAGE_NAME} — ${IMAGE_DESCRIPTION:-}"
|
||||
echo "Alpine: ${ALPINE_VERSION}"
|
||||
echo "Distro: ${DISTRO} ${DISTRO_VERSION}"
|
||||
echo "Size: ${IMAGE_SIZE}"
|
||||
echo "Build: ${BUILD_DIR}"
|
||||
echo ""
|
||||
@@ -96,6 +132,11 @@ if ! command -v qemu-img &>/dev/null; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "$DISTRO" == "ubuntu" ]] && ! command -v parted &>/dev/null; then
|
||||
echo "ERROR: parted not found. Install parted."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Build binaries if BUILD_COMMANDS is set
|
||||
if [[ -n "${BUILD_COMMANDS:-}" ]]; then
|
||||
echo "Building binaries: ${BUILD_COMMANDS}"
|
||||
@@ -105,7 +146,7 @@ if [[ -n "${BUILD_COMMANDS:-}" ]]; then
|
||||
fi
|
||||
fi
|
||||
|
||||
# Verify binaries exist and are statically linked
|
||||
# Verify binaries exist; Alpine also requires static linking (uses musl)
|
||||
if [[ -n "${INSTALL_BINARIES:-}" ]]; then
|
||||
IFS=' ' read -ra BINARY_PAIRS <<< "$INSTALL_BINARIES"
|
||||
for pair in "${BINARY_PAIRS[@]}"; do
|
||||
@@ -115,7 +156,7 @@ if [[ -n "${INSTALL_BINARIES:-}" ]]; then
|
||||
echo "ERROR: Binary not found: $src_path"
|
||||
exit 1
|
||||
fi
|
||||
if ! file "$src_path" | grep -q "statically linked"; then
|
||||
if [[ "$DISTRO" == "alpine" ]] && ! file "$src_path" | grep -q "statically linked"; then
|
||||
echo "ERROR: $src_path is not statically linked (Alpine uses musl — dynamic glibc binaries will fail)"
|
||||
echo " Rebuild with: CGO_ENABLED=0 go build ..."
|
||||
exit 1
|
||||
@@ -147,7 +188,7 @@ if [[ -f "$OUTPUT_RAW" ]] && [[ $(( $(date +%s) - $(stat -c %Y "$OUTPUT_RAW") ))
|
||||
|
||||
if [[ "$DO_IMPORT" == true ]]; then
|
||||
echo "Importing as AMI..."
|
||||
IMPORT_ARGS=(--file "$OUTPUT_RAW" --distro alpine --version "${ALPINE_VERSION}" --arch x86_64)
|
||||
IMPORT_ARGS=(--file "$OUTPUT_RAW" --distro "${DISTRO}" --version "${DISTRO_VERSION}" --arch x86_64)
|
||||
if [[ -n "${SYSTEM_TAG:-}" ]]; then
|
||||
IMPORT_ARGS+=(--tag "$SYSTEM_TAG")
|
||||
fi
|
||||
@@ -158,20 +199,20 @@ fi
|
||||
|
||||
mkdir -p "$BUILD_DIR" "$MOUNT_DIR"
|
||||
|
||||
# Step 1: Download Alpine cloud image
|
||||
if [[ -f "${BUILD_DIR}/${ALPINE_IMAGE}" ]]; then
|
||||
echo "Alpine image already downloaded."
|
||||
# Step 1: Download base cloud image
|
||||
if [[ -f "${BUILD_DIR}/${SOURCE_IMAGE}" ]]; then
|
||||
echo "Base image already downloaded."
|
||||
else
|
||||
echo "Downloading Alpine ${ALPINE_VERSION} cloud image..."
|
||||
if ! curl --fail -L -o "${BUILD_DIR}/${ALPINE_IMAGE}" "$ALPINE_URL"; then
|
||||
rm -f "${BUILD_DIR}/${ALPINE_IMAGE}"
|
||||
echo "ERROR: Failed to download Alpine image from $ALPINE_URL"
|
||||
echo "Downloading ${DISTRO} ${DISTRO_VERSION} cloud image..."
|
||||
if ! curl --fail -L -o "${BUILD_DIR}/${SOURCE_IMAGE}" "$SOURCE_URL"; then
|
||||
rm -f "${BUILD_DIR}/${SOURCE_IMAGE}"
|
||||
echo "ERROR: Failed to download image from $SOURCE_URL"
|
||||
exit 1
|
||||
fi
|
||||
# Verify the download is a valid qcow2 image
|
||||
if ! qemu-img info "${BUILD_DIR}/${ALPINE_IMAGE}" &>/dev/null; then
|
||||
rm -f "${BUILD_DIR}/${ALPINE_IMAGE}"
|
||||
echo "ERROR: Downloaded file is not a valid qcow2 image"
|
||||
# Verify the download is a valid qcow2/img
|
||||
if ! qemu-img info "${BUILD_DIR}/${SOURCE_IMAGE}" &>/dev/null; then
|
||||
rm -f "${BUILD_DIR}/${SOURCE_IMAGE}"
|
||||
echo "ERROR: Downloaded file is not a valid disk image"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
@@ -179,9 +220,9 @@ fi
|
||||
# Step 2: Copy image for customization
|
||||
rm -f "$OUTPUT_IMAGE"
|
||||
echo "Copying image for customization..."
|
||||
cp "${BUILD_DIR}/${ALPINE_IMAGE}" "$OUTPUT_IMAGE"
|
||||
cp "${BUILD_DIR}/${SOURCE_IMAGE}" "$OUTPUT_IMAGE"
|
||||
|
||||
# Resize the image (Alpine cloud images are ~200MB, need room for packages)
|
||||
# Resize the image to provide room for packages
|
||||
qemu-img resize "$OUTPUT_IMAGE" "$IMAGE_SIZE"
|
||||
|
||||
# Step 3: Connect via qemu-nbd
|
||||
@@ -194,7 +235,7 @@ fi
|
||||
sudo qemu-nbd --disconnect "${NBD_DEV}" 2>/dev/null || true
|
||||
sudo qemu-nbd --connect="${NBD_DEV}" "$OUTPUT_IMAGE"
|
||||
|
||||
# Wait for the nbd device to be ready (kernel needs time to scan the block device)
|
||||
# Wait for the nbd device to be ready
|
||||
for i in $(seq 1 10); do
|
||||
if sudo blockdev --getsize64 "${NBD_DEV}" &>/dev/null; then
|
||||
break
|
||||
@@ -203,32 +244,57 @@ for i in $(seq 1 10); do
|
||||
done
|
||||
sleep 1
|
||||
|
||||
# Ubuntu images have a partition table; wait for partition devices then resize
|
||||
# the partition to fill the expanded image before resizing the filesystem.
|
||||
if [[ "$DISTRO" == "ubuntu" ]]; then
|
||||
for i in $(seq 1 10); do
|
||||
if [[ -e "${ROOT_PART}" ]]; then break; fi
|
||||
sleep 1
|
||||
done
|
||||
echo "Resizing partition..."
|
||||
# After qemu-img resize the GPT backup header sits at the old end of disk.
|
||||
# Move it to the new end before parted tries to resize, otherwise parted
|
||||
# refuses with "Unable to satisfy all constraints on the partition."
|
||||
sudo sgdisk --move-second-header "${NBD_DEV}" 2>/dev/null || true
|
||||
sudo parted --script "${NBD_DEV}" resizepart 1 100%
|
||||
fi
|
||||
|
||||
# Alpine cloud images have ext4 directly on the block device (no partition table).
|
||||
# Resize the filesystem to fill the resized image.
|
||||
echo "Checking filesystem..."
|
||||
sudo e2fsck -f -y "${NBD_DEV}" || {
|
||||
sudo e2fsck -f -y "${ROOT_PART}" || {
|
||||
ec=$?
|
||||
if [[ $ec -gt 1 ]]; then
|
||||
echo "ERROR: e2fsck failed with exit code $ec on ${NBD_DEV}"
|
||||
echo "ERROR: e2fsck failed with exit code $ec on ${ROOT_PART}"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
echo "Resizing filesystem..."
|
||||
if ! sudo resize2fs "${NBD_DEV}"; then
|
||||
echo "ERROR: resize2fs failed on ${NBD_DEV}"
|
||||
if ! sudo resize2fs "${ROOT_PART}"; then
|
||||
echo "ERROR: resize2fs failed on ${ROOT_PART}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Step 4: Mount and customize
|
||||
echo "Mounting root filesystem..."
|
||||
sudo mount "${NBD_DEV}" "$MOUNT_DIR"
|
||||
sudo mount "${ROOT_PART}" "$MOUNT_DIR"
|
||||
|
||||
# Set up resolv.conf for DNS inside chroot
|
||||
# Set up resolv.conf for DNS inside chroot.
|
||||
# Ubuntu cloud images symlink /etc/resolv.conf → /run/systemd/resolve/stub-resolv.conf
|
||||
# which doesn't exist yet. Remove the symlink and write a real file.
|
||||
sudo rm -f "${MOUNT_DIR}/etc/resolv.conf"
|
||||
sudo cp /etc/resolv.conf "${MOUNT_DIR}/etc/resolv.conf"
|
||||
|
||||
# Ubuntu chroot requires /proc /sys /dev bind mounts for systemd and DKMS
|
||||
if [[ "$DISTRO" == "ubuntu" ]]; then
|
||||
sudo mount --bind /proc "${MOUNT_DIR}/proc"
|
||||
sudo mount --bind /sys "${MOUNT_DIR}/sys"
|
||||
sudo mount --bind /dev "${MOUNT_DIR}/dev"
|
||||
sudo mount --bind /dev/pts "${MOUNT_DIR}/dev/pts"
|
||||
fi
|
||||
|
||||
# Install packages
|
||||
if [[ -n "${APK_PACKAGES:-}" ]]; then
|
||||
if [[ "$DISTRO" == "alpine" ]] && [[ -n "${APK_PACKAGES:-}" ]]; then
|
||||
echo "Installing packages: ${APK_PACKAGES}..."
|
||||
sudo chroot "$MOUNT_DIR" /bin/sh -c "
|
||||
set -e
|
||||
@@ -244,16 +310,30 @@ fi
|
||||
apk update
|
||||
apk add ${APK_PACKAGES}
|
||||
"
|
||||
elif [[ "$DISTRO" == "ubuntu" ]] && [[ -n "${APT_PACKAGES:-}" ]]; then
|
||||
echo "Installing packages: ${APT_PACKAGES}..."
|
||||
sudo chroot "$MOUNT_DIR" /bin/bash -c "
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get update
|
||||
apt-get install -y --no-install-recommends ${APT_PACKAGES}
|
||||
"
|
||||
fi
|
||||
|
||||
# Enable OpenRC services
|
||||
# Enable services
|
||||
if [[ -n "${ENABLE_SERVICES:-}" ]]; then
|
||||
echo "Enabling services: ${ENABLE_SERVICES}..."
|
||||
IFS=' ' read -ra SERVICES <<< "$ENABLE_SERVICES"
|
||||
for svc in "${SERVICES[@]}"; do
|
||||
if ! sudo chroot "$MOUNT_DIR" /bin/sh -c "rc-update add ${svc} default"; then
|
||||
echo "ERROR: Failed to enable service '${svc}' — does it exist in the image?"
|
||||
exit 1
|
||||
if [[ "$DISTRO" == "alpine" ]]; then
|
||||
if ! sudo chroot "$MOUNT_DIR" /bin/sh -c "rc-update add ${svc} default"; then
|
||||
echo "ERROR: Failed to enable service '${svc}' — does it exist in the image?"
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
if ! sudo chroot "$MOUNT_DIR" /bin/bash -c "systemctl enable ${svc}"; then
|
||||
echo "ERROR: Failed to enable service '${svc}'"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
done
|
||||
fi
|
||||
@@ -282,21 +362,40 @@ if [[ -n "${SETUP_SCRIPT:-}" ]]; then
|
||||
echo "Running setup script: ${SETUP_SCRIPT}..."
|
||||
sudo cp "$setup_path" "${MOUNT_DIR}/tmp/setup.sh"
|
||||
sudo chmod 755 "${MOUNT_DIR}/tmp/setup.sh"
|
||||
sudo chroot "$MOUNT_DIR" /tmp/setup.sh
|
||||
if [[ "$DISTRO" == "ubuntu" ]]; then
|
||||
sudo chroot "$MOUNT_DIR" /bin/bash /tmp/setup.sh
|
||||
else
|
||||
sudo chroot "$MOUNT_DIR" /tmp/setup.sh
|
||||
fi
|
||||
sudo rm -f "${MOUNT_DIR}/tmp/setup.sh"
|
||||
fi
|
||||
|
||||
# Step 6: Clean up and unmount
|
||||
echo "Cleaning up image..."
|
||||
sudo chroot "$MOUNT_DIR" /bin/sh -c '
|
||||
if [[ "$DISTRO" == "alpine" ]]; then
|
||||
sudo chroot "$MOUNT_DIR" /bin/sh -c '
|
||||
apk cache clean 2>/dev/null || true
|
||||
rm -rf /var/cache/apk/* /tmp/*
|
||||
'
|
||||
else
|
||||
sudo chroot "$MOUNT_DIR" /bin/bash -c '
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get clean
|
||||
rm -rf /var/lib/apt/lists/* /tmp/*
|
||||
'
|
||||
fi
|
||||
|
||||
# Restore original resolv.conf (cloud-init will set it on boot)
|
||||
# Restore the systemd-resolved symlink (Ubuntu default); cloud-init sets DNS on boot.
|
||||
sudo rm -f "${MOUNT_DIR}/etc/resolv.conf"
|
||||
sudo ln -sf /run/systemd/resolve/stub-resolv.conf "${MOUNT_DIR}/etc/resolv.conf"
|
||||
|
||||
echo "Unmounting..."
|
||||
if [[ "$DISTRO" == "ubuntu" ]]; then
|
||||
sudo umount "${MOUNT_DIR}/dev/pts"
|
||||
sudo umount "${MOUNT_DIR}/dev"
|
||||
sudo umount "${MOUNT_DIR}/sys"
|
||||
sudo umount "${MOUNT_DIR}/proc"
|
||||
fi
|
||||
sudo umount "$MOUNT_DIR"
|
||||
sudo qemu-nbd --disconnect "${NBD_DEV}"
|
||||
|
||||
@@ -318,7 +417,7 @@ echo ""
|
||||
|
||||
if [[ "$DO_IMPORT" == true ]]; then
|
||||
echo "Importing as AMI..."
|
||||
IMPORT_ARGS=(--file "$OUTPUT_RAW" --distro alpine --version "${ALPINE_VERSION}" --arch x86_64)
|
||||
IMPORT_ARGS=(--file "$OUTPUT_RAW" --distro "${DISTRO}" --version "${DISTRO_VERSION}" --arch x86_64)
|
||||
if [[ -n "${SYSTEM_TAG:-}" ]]; then
|
||||
IMPORT_ARGS+=(--tag "$SYSTEM_TAG")
|
||||
fi
|
||||
@@ -328,9 +427,9 @@ else
|
||||
echo " cd $PROJECT_DIR && ./bin/spx admin images import \\"
|
||||
echo " --file $OUTPUT_RAW \\"
|
||||
if [[ -n "${SYSTEM_TAG:-}" ]]; then
|
||||
echo " --distro alpine --version ${ALPINE_VERSION} --arch x86_64 \\"
|
||||
echo " --distro ${DISTRO} --version ${DISTRO_VERSION} --arch x86_64 \\"
|
||||
echo " --tag ${SYSTEM_TAG}"
|
||||
else
|
||||
echo " --distro alpine --version ${ALPINE_VERSION} --arch x86_64"
|
||||
echo " --distro ${DISTRO} --version ${DISTRO_VERSION} --arch x86_64"
|
||||
fi
|
||||
fi
|
||||
|
||||
@@ -33,6 +33,8 @@ mkdir -p "$STAGING/systemd"
|
||||
cp "$PROJECT_ROOT/build/systemd/"* "$STAGING/systemd/"
|
||||
mkdir -p "$STAGING/scripts"
|
||||
cp "$PROJECT_ROOT/build/scripts/"* "$STAGING/scripts/"
|
||||
mkdir -p "$STAGING/udev"
|
||||
cp "$PROJECT_ROOT/build/udev/"* "$STAGING/udev/"
|
||||
cp "$PROJECT_ROOT/build/logrotate/spinifex" "$STAGING/logrotate-spinifex"
|
||||
tar czf /tmp/spinifex-local.tar.gz -C "$STAGING" .
|
||||
|
||||
|
||||
Executable
+82
@@ -0,0 +1,82 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
# ubuntu-gpu-setup.sh — Chroot setup for Ubuntu 24.04 GPU AMI
|
||||
#
|
||||
# Installs NVIDIA headless server driver with DKMS module pre-built against
|
||||
# a pinned kernel. Blacklists nouveau. Prevents unattended-upgrades from
|
||||
# invalidating the pre-built module.
|
||||
#
|
||||
# Must run inside an Ubuntu 24.04 chroot with /proc /sys /dev bind-mounted
|
||||
# (required by DKMS during the nvidia package post-install hook).
|
||||
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
apt-get update
|
||||
|
||||
# Install a pinned kernel and its headers so DKMS has a stable build target.
|
||||
# linux-image-generic and linux-headers-generic pull in the current HWE stack.
|
||||
apt-get install -y --no-install-recommends \
|
||||
linux-image-generic \
|
||||
linux-headers-generic
|
||||
|
||||
# Capture the installed kernel version before adding the driver.
|
||||
KVER=$(ls /boot/vmlinuz-* 2>/dev/null | sort -V | tail -1 | sed 's|/boot/vmlinuz-||')
|
||||
if [[ -z "${KVER}" ]]; then
|
||||
echo "ERROR: No kernel found under /boot after linux-image-generic install"
|
||||
exit 1
|
||||
fi
|
||||
echo "Target kernel: ${KVER}"
|
||||
|
||||
# Install the NVIDIA headless server driver.
|
||||
# The nvidia-dkms-550-server post-install hook runs `dkms autoinstall`, which
|
||||
# builds the module against every kernel found under /lib/modules/ — including
|
||||
# ${KVER} whose headers we just installed above.
|
||||
apt-get install -y --no-install-recommends \
|
||||
nvidia-headless-550-server \
|
||||
nvidia-utils-550-server
|
||||
|
||||
# Verify the DKMS module was registered and built.
|
||||
if dkms status 2>/dev/null | grep -q "nvidia"; then
|
||||
echo "NVIDIA DKMS module status: $(dkms status 2>/dev/null | grep nvidia | head -1)"
|
||||
else
|
||||
echo "WARNING: nvidia DKMS module not found in dkms status — manual verification required"
|
||||
fi
|
||||
|
||||
# Pin the kernel so unattended-upgrades cannot replace it and invalidate
|
||||
# the pre-built DKMS module.
|
||||
apt-mark hold \
|
||||
linux-image-generic \
|
||||
linux-headers-generic \
|
||||
"linux-image-${KVER}" \
|
||||
"linux-headers-${KVER}"
|
||||
|
||||
# Pin all installed NVIDIA packages for the same reason.
|
||||
NVIDIA_PKGS=$(dpkg --get-selections 'nvidia-*' 2>/dev/null | awk '/install$/{print $1}' || true)
|
||||
LIBNVIDIA_PKGS=$(dpkg --get-selections 'libnvidia-*' 2>/dev/null | awk '/install$/{print $1}' || true)
|
||||
# shellcheck disable=SC2086
|
||||
[[ -n "${NVIDIA_PKGS}" ]] && apt-mark hold ${NVIDIA_PKGS}
|
||||
# shellcheck disable=SC2086
|
||||
[[ -n "${LIBNVIDIA_PKGS}" ]] && apt-mark hold ${LIBNVIDIA_PKGS}
|
||||
|
||||
# Blacklist nouveau so it does not conflict with the NVIDIA driver on boot.
|
||||
cat > /etc/modprobe.d/blacklist-nouveau.conf << 'EOF'
|
||||
blacklist nouveau
|
||||
options nouveau modeset=0
|
||||
EOF
|
||||
|
||||
# Prevent unattended-upgrades from updating kernel or NVIDIA packages,
|
||||
# which would break the pre-built DKMS module.
|
||||
mkdir -p /etc/apt/apt.conf.d
|
||||
cat > /etc/apt/apt.conf.d/99-gpu-ami << 'EOF'
|
||||
Unattended-Upgrade::Package-Blacklist {
|
||||
"linux-";
|
||||
"nvidia-";
|
||||
"libnvidia-";
|
||||
};
|
||||
EOF
|
||||
|
||||
# Rebuild initramfs with nouveau blacklisted and NVIDIA module included.
|
||||
update-initramfs -u -k "${KVER}"
|
||||
|
||||
echo "GPU AMI setup complete: kernel=${KVER}, NVIDIA 550 driver installed, DKMS module pre-built"
|
||||
@@ -0,0 +1,12 @@
|
||||
IMAGE_NAME="ubuntu-gpu"
|
||||
IMAGE_DESCRIPTION="Ubuntu 24.04 GPU-ready AMI (NVIDIA A10G / G5 family)"
|
||||
DISTRO="ubuntu"
|
||||
UBUNTU_VERSION="24.04"
|
||||
|
||||
# 12G provides headroom for kernel headers, NVIDIA driver, and DKMS build artefacts
|
||||
IMAGE_SIZE="12G"
|
||||
|
||||
# NVIDIA driver install and DKMS pre-build are handled in the setup script
|
||||
SETUP_SCRIPT="scripts/images/ubuntu-gpu-setup.sh"
|
||||
|
||||
SYSTEM_TAG="gpu"
|
||||
+26
-4
@@ -15,7 +15,7 @@
|
||||
# VERBOSE Set to 1 to echo "[setup] <stage>" before each top-level step.
|
||||
# SETUP_STAGES Comma-separated subset of stages to run:
|
||||
# deps, aws, users, sudoers, files, directories,
|
||||
# env, systemd, logrotate, fixown, migrations
|
||||
# env, systemd, logrotate, udev, fixown, migrations
|
||||
# Unset = run every stage appropriate for the current mode.
|
||||
|
||||
set -e
|
||||
@@ -192,6 +192,7 @@ spinifex-vpcd ALL=(root) NOPASSWD: /usr/sbin/dhcpcd
|
||||
spinifex-vpcd ALL=(root) NOPASSWD: /usr/bin/ovs-vsctl, /usr/bin/ovs-appctl
|
||||
spinifex-vpcd ALL=(root) NOPASSWD: /usr/bin/ovn-nbctl, /usr/bin/ovn-sbctl
|
||||
spinifex-vpcd ALL=(root) NOPASSWD: /usr/bin/systemctl is-active --quiet ovn-controller
|
||||
spinifex-vpcd ALL=(root) NOPASSWD: /sbin/ip, /usr/sbin/ip
|
||||
SUDOERS
|
||||
$SUDO chmod 0440 /etc/sudoers.d/spinifex-network
|
||||
$SUDO visudo -cf /etc/sudoers.d/spinifex-network || fatal "Invalid sudoers syntax in spinifex-network"
|
||||
@@ -213,8 +214,9 @@ install_apt_deps() {
|
||||
|
||||
DEBIAN_FRONTEND=noninteractive $SUDO apt-get install -y -qq \
|
||||
nbdkit \
|
||||
$QEMU_PACKAGES qemu-utils qemu-kvm less \
|
||||
$QEMU_PACKAGES qemu-utils qemu-kvm ovmf less \
|
||||
libvirt-daemon-system libvirt-clients \
|
||||
pciutils \
|
||||
jq curl iproute2 netcat-openbsd wget unzip xz-utils file \
|
||||
ovn-central ovn-host openvswitch-switch dhcpcd-base \
|
||||
> /dev/null
|
||||
@@ -608,6 +610,24 @@ install_logrotate() {
|
||||
info "Logrotate config installed"
|
||||
}
|
||||
|
||||
# --- Install udev rules ---
|
||||
install_udev() {
|
||||
stage "installing udev rules"
|
||||
if [ ! -d "$EXTRACT_DIR/udev" ]; then
|
||||
return
|
||||
fi
|
||||
$SUDO install -d /etc/udev/rules.d
|
||||
for rule in "$EXTRACT_DIR"/udev/*; do
|
||||
$SUDO install -m 0644 "$rule" "/etc/udev/rules.d/$(basename "$rule")"
|
||||
info " /etc/udev/rules.d/$(basename "$rule")"
|
||||
done
|
||||
if [ "${ISO_BUILD:-0}" != "1" ]; then
|
||||
$SUDO udevadm control --reload-rules
|
||||
$SUDO udevadm trigger --subsystem-match=vfio 2>/dev/null || true
|
||||
fi
|
||||
info "udev rules installed"
|
||||
}
|
||||
|
||||
# --- Upgrade handling ---
|
||||
handle_upgrade() {
|
||||
if $SUDO systemctl is-active --quiet spinifex.target 2>/dev/null; then
|
||||
@@ -688,10 +708,11 @@ main() {
|
||||
handle_upgrade
|
||||
fi
|
||||
|
||||
# Stages that need EXTRACT_DIR: files, directories, systemd, logrotate.
|
||||
# Stages that need EXTRACT_DIR: files, directories, systemd, logrotate, udev.
|
||||
# Only download/extract when at least one such stage is enabled.
|
||||
if stage_enabled files || stage_enabled directories \
|
||||
|| stage_enabled systemd || stage_enabled logrotate; then
|
||||
|| stage_enabled systemd || stage_enabled logrotate \
|
||||
|| stage_enabled udev; then
|
||||
download_spinifex
|
||||
fi
|
||||
|
||||
@@ -705,6 +726,7 @@ main() {
|
||||
stage_enabled fixown && fix_file_ownership
|
||||
stage_enabled systemd && install_systemd
|
||||
stage_enabled logrotate && install_logrotate
|
||||
stage_enabled udev && install_udev
|
||||
|
||||
# Migrations are only safe on a live system (need a running NATS and a
|
||||
# persisted config file). Skip under ISO_BUILD and under any explicit
|
||||
|
||||
+77
-2
@@ -6,8 +6,23 @@
|
||||
#
|
||||
# Assumes services are running and AWS_PROFILE=spinifex is configured
|
||||
# by a prior 'spx admin init'.
|
||||
#
|
||||
# GPU passthrough test (opt-in):
|
||||
# TEST_GPU=1 scripts/smoke-test.sh
|
||||
# TEST_GPU=1 GPU_VENDOR_ID=10de GPU_DEVICE_ID=2236 scripts/smoke-test.sh
|
||||
#
|
||||
# GPU env vars:
|
||||
# TEST_GPU=1 enable GPU passthrough checks
|
||||
# GPU_VENDOR_ID PCI vendor ID (default: auto-detect first NVIDIA VGA/3D device)
|
||||
# GPU_DEVICE_ID PCI device ID (default: auto-detect)
|
||||
# GPU_FAMILY instance family to test (default: g5)
|
||||
set -euo pipefail
|
||||
|
||||
TEST_GPU="${TEST_GPU:-0}"
|
||||
GPU_VENDOR_ID="${GPU_VENDOR_ID:-}"
|
||||
GPU_DEVICE_ID="${GPU_DEVICE_ID:-}"
|
||||
GPU_FAMILY="${GPU_FAMILY:-g5}"
|
||||
|
||||
export AWS_PROFILE=spinifex
|
||||
|
||||
# --- Wait for EC2 daemon to subscribe to NATS ---
|
||||
@@ -42,6 +57,41 @@ echo "==> Importing SSH key"
|
||||
--public-key-material "fileb://$SSH_KEY.pub"
|
||||
aws ec2 describe-key-pairs
|
||||
|
||||
# --- GPU pre-flight (TEST_GPU=1 only) ---
|
||||
if [[ "$TEST_GPU" == "1" ]]; then
|
||||
echo "==> [GPU] Checking IOMMU"
|
||||
if ! dmesg | grep -qi "DMAR.*IOMMU enabled\|iommu.*Translated\|Adding to iommu group\|AMD-Vi.*enabled"; then
|
||||
echo "❌ IOMMU not detected — add intel_iommu=on (or amd_iommu=on) and iommu=pt to kernel cmdline, then reboot"
|
||||
exit 1
|
||||
fi
|
||||
echo " IOMMU active"
|
||||
|
||||
echo "==> [GPU] Detecting NVIDIA GPU"
|
||||
if [[ -z "$GPU_VENDOR_ID" || -z "$GPU_DEVICE_ID" ]]; then
|
||||
GPU_LINE=$(lspci -nn | grep -i nvidia | grep -iE "VGA|3D" | head -1)
|
||||
if [[ -z "$GPU_LINE" ]]; then
|
||||
echo "❌ No NVIDIA GPU found via lspci — is the card seated and detectable?"
|
||||
lspci | grep -i nvidia >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
echo " Detected: $GPU_LINE"
|
||||
PCI_IDS=$(echo "$GPU_LINE" | grep -oP '\[\K[0-9a-f]{4}:[0-9a-f]{4}(?=\])')
|
||||
GPU_VENDOR_ID=$(echo "$PCI_IDS" | cut -d: -f1)
|
||||
GPU_DEVICE_ID=$(echo "$PCI_IDS" | cut -d: -f2)
|
||||
fi
|
||||
echo " GPU PCI ID: ${GPU_VENDOR_ID}:${GPU_DEVICE_ID}"
|
||||
|
||||
echo "==> [GPU] Verifying ${GPU_FAMILY} instance types advertised"
|
||||
G5_TYPES=$(aws ec2 describe-instance-types \
|
||||
--query "InstanceTypes[?starts_with(InstanceType, \`${GPU_FAMILY}\`)].InstanceType" \
|
||||
--output text 2>/dev/null || true)
|
||||
if [[ -z "$G5_TYPES" ]]; then
|
||||
echo "❌ No ${GPU_FAMILY}.* instance types returned by describe-instance-types"
|
||||
exit 1
|
||||
fi
|
||||
echo " ${GPU_FAMILY} types available: $G5_TYPES"
|
||||
fi
|
||||
|
||||
# --- Import AMI ---
|
||||
# --- Import AMI ---
|
||||
echo "==> Importing AMI"
|
||||
@@ -79,7 +129,9 @@ fi
|
||||
|
||||
# --- Launch smoke-test instance ---
|
||||
echo "==> Launching smoke-test instance"
|
||||
if grep -q 'AuthenticAMD' /proc/cpuinfo; then
|
||||
if [[ "$TEST_GPU" == "1" ]]; then
|
||||
INSTANCE_TYPE="${GPU_FAMILY}.xlarge"
|
||||
elif grep -q 'AuthenticAMD' /proc/cpuinfo; then
|
||||
INSTANCE_TYPE="t3a.small"
|
||||
else
|
||||
INSTANCE_TYPE="t3.small"
|
||||
@@ -195,4 +247,27 @@ if ! echo "$SSH_OUT" | grep -q "ec2-user"; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "✅ Smoke test passed — instance $INSTANCE_ID launched, running, and SSH-verified"
|
||||
# --- GPU guest verification (TEST_GPU=1 only) ---
|
||||
if [[ "$TEST_GPU" == "1" ]]; then
|
||||
echo "==> [GPU] Verifying GPU visible inside guest"
|
||||
GPU_VISIBLE=0
|
||||
for _i in $(seq 1 12); do
|
||||
if ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
|
||||
-o ConnectTimeout=5 -o BatchMode=yes \
|
||||
-p "$SSH_INST_PORT" -i "$SSH_KEY" \
|
||||
ec2-user@"$SSH_INST_HOST" \
|
||||
'lspci 2>/dev/null | grep -i nvidia' 2>/dev/null; then
|
||||
GPU_VISIBLE=1
|
||||
break
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
if [[ $GPU_VISIBLE -eq 0 ]]; then
|
||||
echo "❌ NVIDIA GPU not visible in guest via lspci"
|
||||
exit 1
|
||||
fi
|
||||
echo " GPU visible in guest"
|
||||
echo "✅ Smoke test passed (GPU passthrough) — instance $INSTANCE_ID launched with GPU and verified"
|
||||
else
|
||||
echo "✅ Smoke test passed — instance $INSTANCE_ID launched, running, and SSH-verified"
|
||||
fi
|
||||
|
||||
Reference in New Issue
Block a user