mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
tomnewton-mulga
28ca689130
* feat: add GPU discovery package (mulga-992) Adds spinifex/gpu package with sysfs-based GPU enumeration, IOMMU group member lookup, and a PCI device ID model table. Vendor-specific enrichment via nvidia-smi is attempted opportunistically and falls back to the built-in table when unavailable. All discovery and IOMMU logic is covered by unit tests using mock sysfs trees. * feat: add VFIO bind/unbind and GPU pool manager (mulga-993) Adds gpu.BindVFIO/UnbindVFIO using sysfs driver_override for per-device driver switching, and gpu.Manager which owns the claim/release lifecycle for the GPU pool including full IOMMU group binding, rollback on partial failure, and marking entries unavailable when rebind fails after VM termination. * feat: add G5 instance type generator with GPU model support (mulga-994) - Add GPUModel type, NVIDIAa10g constant, and GPUModelForVendorDevice lookup - Add generateGPUTypes to emit g5.* types with GpuInfo populated - Add IsGPUType helper for consumers filtering GPU-capable types - Update DetectAndGenerate to accept gpuModels []GPUModel parameter * feat: integrate GPU passthrough into daemon lifecycle (mulga-995) - Add gpu_passthrough config flag; discover GPUs and build manager at startup - Gate g5 instance types on GPU pool availability via canAllocateCount - Claim GPU (VFIO bind) in LaunchInstance; inject vfio-pci QEMU device - Release GPU (VFIO unbind, rebind to host driver) in stopInstance - Re-claim GPU on daemon restart when QEMU process is still alive - Add UseUEFI / OVMF support for x86_64 VMs (optional, falls back to SeaBIOS) - Fix Claim to use stored OriginalDriver when device is already vfio-pci bound * feat: add GPU fields to NodeStatus and spx top nodes (mulga-996) - Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes) - Populate from gpuManager in handleNodeStatus - Show GPU (used/total) column in spx top nodes; "-" when no GPUs present * feat: Ubuntu 24.04 GPU AMI build pipeline (mulga-997) Extends build-system-image.sh to support Ubuntu alongside Alpine. Key differences handled: partition table (parted resizepart p1), bind mounts for DKMS (/proc /sys /dev), apt vs apk, systemctl vs rc-update. DISTRO defaults to "alpine" for backward compatibility. Adds ubuntu-gpu.conf manifest and ubuntu-gpu-setup.sh chroot script that installs nvidia-headless-550-server with DKMS pre-built against a pinned linux-image-generic kernel, blacklists nouveau, and prevents unattended-upgrades from invalidating the pre-built module. * Fixed GPU passthrough and added test script * feat: always-probe GPU at startup, surface capability in NodeStatusResponse - Add gpuProbeResult struct and probeGPU() which runs at daemon startup regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU groups, and vfio_pci module presence. - Store gpuProbe on Daemon struct; split NewDaemon into probe (always) and activate (opt-in) phases. - Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse so operators can see hardware state before enabling passthrough. - Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but passthrough not active, 'used/total' = passthrough active. * feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001) - Export GenerateGPUTypes for use by the reload path - Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place (preserves references held by instanceService) and updates NATS subscriptions without full daemon restart - Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP in a context-aware loop; setupShutdown no longer intercepts SIGHUP - Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal, handles false→true (re-probe + activate) and true→false (refuse if instances running, otherwise deactivate) transitions * feat: spx admin gpu status/enable/disable commands (mulga-1000) - admin.SetGPUPassthrough: Go implementation of the idempotent TOML write (replaces the inline Python heredoc in spx-test-gpu.sh) - spx admin gpu status: shows GPU hardware, IOMMU/vfio state, passthrough activation, and advertised g5 instance types via NATS NodeStatusResponse - spx admin gpu enable: checks prerequisites, writes TOML, sends SIGHUP, polls until daemon confirms GPUPassthrough=true - spx admin gpu disable: refuses while GPU instances are running, otherwise writes TOML, sends SIGHUP, polls until confirmed - spx-test-gpu.sh: replace Python TOML heredoc with spx admin gpu enable * fixed admin * fix: filter Intel iGPU from passthrough pool, use lspci for consumer GPU names Discover() now skips non-NVIDIA/AMD vendors so Intel iGPUs never enter the passthrough pool. enrichFromLspci() uses the system PCI ID database via lspci -vmm to populate the model name for any card not in the hardcoded table, without needing nvidia-smi or driver tools. * fix: grant CAP_SYS_ADMIN and vfio device access to spinifex-daemon GPU passthrough requires writing to /sys/bus/pci/drivers/vfio-pci/unbind on instance stop. The kernel gates this on CAP_SYS_ADMIN regardless of file permissions. QEMU subprocesses also need char-vfio rw access to open /dev/vfio/N for the passed-through device. * fixing gpu bining permissions * working state of gpu passthrough on phyiscal box * installer grub image fixes * made gpu-passthrough more user facing * fixed banner * fixed banner * banner fix * gpu passthrough docs * preflight tweak * vpcd fixes to set br-wan to promisc for public IP DHCP processes * fixed vpcd br-wan promisc and veth pair * ssh keygen fix * fixes after running /review-branch * feat: re-wire GPU claim and device injection after vm.Manager refactor GPU passthrough claim (VFIO bind) and QEMU device injection were lost when dev's vm.Manager refactor (#185/#186) moved StartInstance/ CreateQMPClient into vm/lifecycle.go. Re-add both in their correct new homes: - daemon_handlers_instance.go: claim GPU before vmMgr.Run() in both RunInstances and StartStoppedInstance handlers; release on failure - vm/lifecycle.go startQEMU(): inject vfio-pci device with x-vga flag derived from instance.GPUXVGAEnabled when GPUPCIAddress is set * fix: Create spinifex user ~/.ssh and keys in firstboot --------- Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com> Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
103 lines
4.2 KiB
Bash
Executable File
103 lines
4.2 KiB
Bash
Executable File
#!/bin/bash
|
|
# dev-install.sh — Full local development setup via production installer.
|
|
# Builds from source, assembles a tarball, runs setup.sh for scaffolding
|
|
# (user creation, directories, systemd units), initializes the cluster,
|
|
# and starts services via systemd.
|
|
#
|
|
# Environment variables:
|
|
# SPINIFEX_REGION Region for spx admin init (default: ap-southeast-2)
|
|
# SPINIFEX_AZ AZ for spx admin init (default: ${SPINIFEX_REGION}a)
|
|
# DEV_INSTALL_SKIP_INIT Internal flag used by reset-dev-env.sh. When 1,
|
|
# stops after setup.sh and skips setup-ovn, init,
|
|
# CA trust, service start, and LB image import —
|
|
# caller owns those steps. Do not set manually.
|
|
#
|
|
# Usage: ./scripts/dev-install.sh
|
|
set -euo pipefail
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
|
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
|
MULGA_ROOT="$(cd "$PROJECT_ROOT/.." && pwd)"
|
|
|
|
echo "=== Building ==="
|
|
cd "$PROJECT_ROOT" && make build
|
|
cd "$MULGA_ROOT/viperblock" && make go_build_nbd
|
|
|
|
echo "=== Assembling tarball for setup.sh ==="
|
|
STAGING=$(mktemp -d)
|
|
trap 'rm -rf "$STAGING"' EXIT
|
|
cp "$PROJECT_ROOT/bin/spx" "$STAGING/spx"
|
|
cp "$MULGA_ROOT/viperblock/lib/nbdkit-viperblock-plugin.so" "$STAGING/"
|
|
cp "$PROJECT_ROOT/scripts/setup-ovn.sh" "$STAGING/"
|
|
mkdir -p "$STAGING/systemd"
|
|
cp "$PROJECT_ROOT/build/systemd/"* "$STAGING/systemd/"
|
|
mkdir -p "$STAGING/scripts"
|
|
cp "$PROJECT_ROOT/build/scripts/"* "$STAGING/scripts/"
|
|
mkdir -p "$STAGING/udev"
|
|
cp "$PROJECT_ROOT/build/udev/"* "$STAGING/udev/"
|
|
cp "$PROJECT_ROOT/build/logrotate/spinifex" "$STAGING/logrotate-spinifex"
|
|
tar czf /tmp/spinifex-local.tar.gz -C "$STAGING" .
|
|
|
|
echo "=== Cleaning stale state from previous installs ==="
|
|
# Stop any running services before modifying files
|
|
sudo systemctl stop spinifex.target 2>/dev/null || true
|
|
sudo systemctl reset-failed 'spinifex-*' 2>/dev/null || true
|
|
|
|
# Remove stale files owned by the dev user in production paths.
|
|
# Previous dev-mode installs (admin init without sudo, start-dev.sh) leave
|
|
# files owned by tf-user that service users (spinifex-nats, etc.) can't read
|
|
# under systemd's ProtectSystem=strict sandboxing.
|
|
for dir in /var/lib/spinifex /var/log/spinifex /etc/spinifex; do
|
|
if [ -d "$dir" ]; then
|
|
# Remove PID files, stale logs, and the legacy ~/spinifex/config symlink
|
|
sudo find "$dir" -name '*.pid' -delete 2>/dev/null || true
|
|
sudo find "$dir" -name '*.log' -delete 2>/dev/null || true
|
|
sudo find "$dir" -name '*.log.*' -delete 2>/dev/null || true
|
|
fi
|
|
done
|
|
# Remove legacy data dir contents that conflict with production layout
|
|
if [ -d /var/lib/spinifex/config ]; then
|
|
sudo rm -rf /var/lib/spinifex/config
|
|
fi
|
|
|
|
echo "=== Running setup.sh (creates users, dirs, systemd units) ==="
|
|
sudo INSTALL_SPINIFEX_TARBALL=/tmp/spinifex-local.tar.gz INSTALL_SPINIFEX_SKIP_NEWGRP=1 bash "$PROJECT_ROOT/scripts/setup.sh"
|
|
rm -f /tmp/spinifex-local.tar.gz
|
|
|
|
if [ "${DEV_INSTALL_SKIP_INIT:-0}" = "1" ]; then
|
|
echo "=== DEV_INSTALL_SKIP_INIT=1 — stopping after setup.sh ==="
|
|
echo "Caller owns: setup-ovn, spx admin init, CA trust, service start, LB image."
|
|
exit 0
|
|
fi
|
|
|
|
echo "=== Setting up OVN ==="
|
|
sudo /usr/local/share/spinifex/setup-ovn.sh --management
|
|
|
|
SPINIFEX_REGION="${SPINIFEX_REGION:-ap-southeast-2}"
|
|
SPINIFEX_AZ="${SPINIFEX_AZ:-${SPINIFEX_REGION}a}"
|
|
|
|
echo "=== Initializing (region=$SPINIFEX_REGION az=$SPINIFEX_AZ) ==="
|
|
sudo spx admin init --force --region "$SPINIFEX_REGION" --az "$SPINIFEX_AZ" --node node1 --nodes 1
|
|
|
|
echo "=== Installing CA certificate ==="
|
|
sudo cp /etc/spinifex/ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt
|
|
sudo update-ca-certificates
|
|
|
|
echo "=== Starting services ==="
|
|
sudo systemctl start spinifex.target
|
|
|
|
# Check if user needs to re-login for spinifex group membership
|
|
if ! id -Gn 2>/dev/null | grep -qw spinifex; then
|
|
newgrp spinifex
|
|
fi
|
|
|
|
echo "=== Building and importing LB image ==="
|
|
cd "$PROJECT_ROOT" && make build-lb-agent
|
|
"$PROJECT_ROOT/scripts/build-system-image.sh" "$PROJECT_ROOT/scripts/images/lb.conf" --import --quiet
|
|
|
|
echo "=== Done ==="
|
|
echo "Services: sudo systemctl status spinifex.target"
|
|
echo "Logs: journalctl -u 'spinifex-*' -f"
|
|
echo "Test: AWS_PROFILE=spinifex aws ec2 describe-instances"
|
|
echo "Iterate: make deploy (rebuild + restart all)"
|