tomnewton-mulga 28ca689130 feat: VFIO GPU passthrough (#188) ...

* feat: add GPU discovery package (mulga-992)

Adds spinifex/gpu package with sysfs-based GPU enumeration, IOMMU group
member lookup, and a PCI device ID model table. Vendor-specific enrichment
via nvidia-smi is attempted opportunistically and falls back to the built-in
table when unavailable. All discovery and IOMMU logic is covered by unit tests
using mock sysfs trees.

* feat: add VFIO bind/unbind and GPU pool manager (mulga-993)

Adds gpu.BindVFIO/UnbindVFIO using sysfs driver_override for per-device
driver switching, and gpu.Manager which owns the claim/release lifecycle
for the GPU pool including full IOMMU group binding, rollback on partial
failure, and marking entries unavailable when rebind fails after VM
termination.

* feat: add G5 instance type generator with GPU model support (mulga-994)

- Add GPUModel type, NVIDIAa10g constant, and GPUModelForVendorDevice lookup
- Add generateGPUTypes to emit g5.* types with GpuInfo populated
- Add IsGPUType helper for consumers filtering GPU-capable types
- Update DetectAndGenerate to accept gpuModels []GPUModel parameter

* feat: integrate GPU passthrough into daemon lifecycle (mulga-995)

- Add gpu_passthrough config flag; discover GPUs and build manager at startup
- Gate g5 instance types on GPU pool availability via canAllocateCount
- Claim GPU (VFIO bind) in LaunchInstance; inject vfio-pci QEMU device
- Release GPU (VFIO unbind, rebind to host driver) in stopInstance
- Re-claim GPU on daemon restart when QEMU process is still alive
- Add UseUEFI / OVMF support for x86_64 VMs (optional, falls back to SeaBIOS)
- Fix Claim to use stored OriginalDriver when device is already vfio-pci bound

* feat: add GPU fields to NodeStatus and spx top nodes (mulga-996)

- Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes)
- Populate from gpuManager in handleNodeStatus
- Show GPU (used/total) column in spx top nodes; "-" when no GPUs present

* feat: Ubuntu 24.04 GPU AMI build pipeline (mulga-997)

Extends build-system-image.sh to support Ubuntu alongside Alpine.
Key differences handled: partition table (parted resizepart p1),
bind mounts for DKMS (/proc /sys /dev), apt vs apk, systemctl vs
rc-update. DISTRO defaults to "alpine" for backward compatibility.

Adds ubuntu-gpu.conf manifest and ubuntu-gpu-setup.sh chroot script
that installs nvidia-headless-550-server with DKMS pre-built against
a pinned linux-image-generic kernel, blacklists nouveau, and prevents
unattended-upgrades from invalidating the pre-built module.

* Fixed GPU passthrough and added test script

* feat: always-probe GPU at startup, surface capability in NodeStatusResponse

- Add gpuProbeResult struct and probeGPU() which runs at daemon startup
  regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU
  groups, and vfio_pci module presence.
- Store gpuProbe on Daemon struct; split NewDaemon into probe (always)
  and activate (opt-in) phases.
- Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse
  so operators can see hardware state before enabling passthrough.
- Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but
  passthrough not active, 'used/total' = passthrough active.

* feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001)

- Export GenerateGPUTypes for use by the reload path
- Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place
  (preserves references held by instanceService) and updates NATS
  subscriptions without full daemon restart
- Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP
  in a context-aware loop; setupShutdown no longer intercepts SIGHUP
- Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal,
  handles false→true (re-probe + activate) and true→false (refuse if
  instances running, otherwise deactivate) transitions

* feat: spx admin gpu status/enable/disable commands (mulga-1000)

- admin.SetGPUPassthrough: Go implementation of the idempotent TOML write
  (replaces the inline Python heredoc in spx-test-gpu.sh)
- spx admin gpu status: shows GPU hardware, IOMMU/vfio state, passthrough
  activation, and advertised g5 instance types via NATS NodeStatusResponse
- spx admin gpu enable: checks prerequisites, writes TOML, sends SIGHUP,
  polls until daemon confirms GPUPassthrough=true
- spx admin gpu disable: refuses while GPU instances are running, otherwise
  writes TOML, sends SIGHUP, polls until confirmed
- spx-test-gpu.sh: replace Python TOML heredoc with spx admin gpu enable

* fixed admin

* fix: filter Intel iGPU from passthrough pool, use lspci for consumer GPU names

Discover() now skips non-NVIDIA/AMD vendors so Intel iGPUs never enter
the passthrough pool. enrichFromLspci() uses the system PCI ID database
via lspci -vmm to populate the model name for any card not in the
hardcoded table, without needing nvidia-smi or driver tools.

* fix: grant CAP_SYS_ADMIN and vfio device access to spinifex-daemon

GPU passthrough requires writing to /sys/bus/pci/drivers/vfio-pci/unbind
on instance stop. The kernel gates this on CAP_SYS_ADMIN regardless of
file permissions. QEMU subprocesses also need char-vfio rw access to
open /dev/vfio/N for the passed-through device.

* fixing gpu bining permissions

* working state of gpu passthrough on phyiscal box

* installer grub image fixes

* made gpu-passthrough more user facing

* fixed banner

* fixed banner

* banner fix

* gpu passthrough docs

* preflight tweak

* vpcd fixes to set br-wan to promisc for public IP DHCP processes

* fixed vpcd br-wan promisc and veth pair

* ssh keygen fix

* fixes after running /review-branch

* feat: re-wire GPU claim and device injection after vm.Manager refactor

GPU passthrough claim (VFIO bind) and QEMU device injection were lost
when dev's vm.Manager refactor (#185/#186) moved StartInstance/
CreateQMPClient into vm/lifecycle.go. Re-add both in their correct
new homes:

- daemon_handlers_instance.go: claim GPU before vmMgr.Run() in both
  RunInstances and StartStoppedInstance handlers; release on failure
- vm/lifecycle.go startQEMU(): inject vfio-pci device with x-vga flag
  derived from instance.GPUXVGAEnabled when GPUPCIAddress is set

* fix: Create spinifex user ~/.ssh and keys in firstboot

---------

Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>

2026-05-06 09:52:23 +10:00

..

main.go

feat: VFIO GPU passthrough (#188)

2026-05-06 09:52:23 +10:00