Add SkipFormation bool to firstboot.Config. When set, the cluster
formation block (spx admin init/join) is replaced with a no-op echo,
leaving formation to the provisioning controller.
Wired through: autoinstall reads SPINIFEX_SKIP_FORMATION=1 from env,
install.go passes it to firstboot.Config. spinifex-init.sh parses the
kernel cmdline param and exports it. boot-handler.go hardcodes
SPINIFEX_SKIP_FORMATION=1 so all bare-metal CI nodes skip formation —
bm-bootstrap.sh always runs spx admin init/join with the correct
--nodes N from nodes.json, enabling proper multi-node formation.
In the PXE path SPINIFEX_INSTALL_CALLBACK arrives on the iPXE kernel
cmdline but was never threaded into the firstboot script, so firstboot
never curled the controller's /done endpoint. The flag stayed on disk,
the node re-entered the install loop on the next PXE boot.
Fix: read SPINIFEX_INSTALL_CALLBACK from the live-environment env in
toFirstbootConfig and embed a best-effort curl into the generated
firstboot script, fired once after the success marker is written.
Interactive and USB-stick installs are unaffected (env var absent
-> callbackBlock is empty, no curl injected).
* feat: add GPU discovery package (mulga-992)
Adds spinifex/gpu package with sysfs-based GPU enumeration, IOMMU group
member lookup, and a PCI device ID model table. Vendor-specific enrichment
via nvidia-smi is attempted opportunistically and falls back to the built-in
table when unavailable. All discovery and IOMMU logic is covered by unit tests
using mock sysfs trees.
* feat: add VFIO bind/unbind and GPU pool manager (mulga-993)
Adds gpu.BindVFIO/UnbindVFIO using sysfs driver_override for per-device
driver switching, and gpu.Manager which owns the claim/release lifecycle
for the GPU pool including full IOMMU group binding, rollback on partial
failure, and marking entries unavailable when rebind fails after VM
termination.
* feat: add G5 instance type generator with GPU model support (mulga-994)
- Add GPUModel type, NVIDIAa10g constant, and GPUModelForVendorDevice lookup
- Add generateGPUTypes to emit g5.* types with GpuInfo populated
- Add IsGPUType helper for consumers filtering GPU-capable types
- Update DetectAndGenerate to accept gpuModels []GPUModel parameter
* feat: integrate GPU passthrough into daemon lifecycle (mulga-995)
- Add gpu_passthrough config flag; discover GPUs and build manager at startup
- Gate g5 instance types on GPU pool availability via canAllocateCount
- Claim GPU (VFIO bind) in LaunchInstance; inject vfio-pci QEMU device
- Release GPU (VFIO unbind, rebind to host driver) in stopInstance
- Re-claim GPU on daemon restart when QEMU process is still alive
- Add UseUEFI / OVMF support for x86_64 VMs (optional, falls back to SeaBIOS)
- Fix Claim to use stored OriginalDriver when device is already vfio-pci bound
* feat: add GPU fields to NodeStatus and spx top nodes (mulga-996)
- Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes)
- Populate from gpuManager in handleNodeStatus
- Show GPU (used/total) column in spx top nodes; "-" when no GPUs present
* feat: Ubuntu 24.04 GPU AMI build pipeline (mulga-997)
Extends build-system-image.sh to support Ubuntu alongside Alpine.
Key differences handled: partition table (parted resizepart p1),
bind mounts for DKMS (/proc /sys /dev), apt vs apk, systemctl vs
rc-update. DISTRO defaults to "alpine" for backward compatibility.
Adds ubuntu-gpu.conf manifest and ubuntu-gpu-setup.sh chroot script
that installs nvidia-headless-550-server with DKMS pre-built against
a pinned linux-image-generic kernel, blacklists nouveau, and prevents
unattended-upgrades from invalidating the pre-built module.
* Fixed GPU passthrough and added test script
* feat: always-probe GPU at startup, surface capability in NodeStatusResponse
- Add gpuProbeResult struct and probeGPU() which runs at daemon startup
regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU
groups, and vfio_pci module presence.
- Store gpuProbe on Daemon struct; split NewDaemon into probe (always)
and activate (opt-in) phases.
- Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse
so operators can see hardware state before enabling passthrough.
- Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but
passthrough not active, 'used/total' = passthrough active.
* feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001)
- Export GenerateGPUTypes for use by the reload path
- Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place
(preserves references held by instanceService) and updates NATS
subscriptions without full daemon restart
- Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP
in a context-aware loop; setupShutdown no longer intercepts SIGHUP
- Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal,
handles false→true (re-probe + activate) and true→false (refuse if
instances running, otherwise deactivate) transitions
* feat: spx admin gpu status/enable/disable commands (mulga-1000)
- admin.SetGPUPassthrough: Go implementation of the idempotent TOML write
(replaces the inline Python heredoc in spx-test-gpu.sh)
- spx admin gpu status: shows GPU hardware, IOMMU/vfio state, passthrough
activation, and advertised g5 instance types via NATS NodeStatusResponse
- spx admin gpu enable: checks prerequisites, writes TOML, sends SIGHUP,
polls until daemon confirms GPUPassthrough=true
- spx admin gpu disable: refuses while GPU instances are running, otherwise
writes TOML, sends SIGHUP, polls until confirmed
- spx-test-gpu.sh: replace Python TOML heredoc with spx admin gpu enable
* fixed admin
* fix: filter Intel iGPU from passthrough pool, use lspci for consumer GPU names
Discover() now skips non-NVIDIA/AMD vendors so Intel iGPUs never enter
the passthrough pool. enrichFromLspci() uses the system PCI ID database
via lspci -vmm to populate the model name for any card not in the
hardcoded table, without needing nvidia-smi or driver tools.
* fix: grant CAP_SYS_ADMIN and vfio device access to spinifex-daemon
GPU passthrough requires writing to /sys/bus/pci/drivers/vfio-pci/unbind
on instance stop. The kernel gates this on CAP_SYS_ADMIN regardless of
file permissions. QEMU subprocesses also need char-vfio rw access to
open /dev/vfio/N for the passed-through device.
* fixing gpu bining permissions
* working state of gpu passthrough on phyiscal box
* installer grub image fixes
* made gpu-passthrough more user facing
* fixed banner
* fixed banner
* banner fix
* gpu passthrough docs
* preflight tweak
* vpcd fixes to set br-wan to promisc for public IP DHCP processes
* fixed vpcd br-wan promisc and veth pair
* ssh keygen fix
* fixes after running /review-branch
* feat: re-wire GPU claim and device injection after vm.Manager refactor
GPU passthrough claim (VFIO bind) and QEMU device injection were lost
when dev's vm.Manager refactor (#185/#186) moved StartInstance/
CreateQMPClient into vm/lifecycle.go. Re-add both in their correct
new homes:
- daemon_handlers_instance.go: claim GPU before vmMgr.Run() in both
RunInstances and StartStoppedInstance handlers; release on failure
- vm/lifecycle.go startQEMU(): inject vfio-pci device with x-vga flag
derived from instance.GPUXVGAEnabled when GPUPCIAddress is set
* fix: Create spinifex user ~/.ssh and keys in firstboot
---------
Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
- partUUID: error on empty blkid output instead of writing UUID= to fstab
- installBootloader: fail if both EFI and BIOS targets fail; track efiErr
- machine-id fallback: write at mode 0600 so systemd can commit on first boot
- Run: defer unmount so partitions are always cleaned up on failure
- copyRootfs: verify critical paths exist after rsync before continuing
- resolveNIC: skip virtual/bridge interfaces (docker0, virbr0, ovs-system)
- EjectAndReboot: use syscall.Reboot instead of exec(reboot) which needs D-Bus
- ui.Run: return error on TTY open failure instead of silent fallback
- firstboot script: trap EXIT to always self-disable, preventing retry loops
- enableUnit/enableBannerUnit: deduplicate into enableSystemdUnit; check Remove
- runAdminBanner: promote service restart failure to slog.Error
- parseNodeConf: log non-ENOENT read failures
- autoinstall: replace file scanner with env var reader (SPINIFEX_AUTO,
SPINIFEX_HOSTNAME, etc.) set by spinifex-init.sh from kernel cmdline
- autoinstall: disk selection modes — auto (single or fail), largest,
smallest, nvme, or exact path; clear error listing available disks
when auto/nvme finds multiple candidates
- admin banner: add Login row showing spinifex account name
- admin banner: write to /etc/issue (pre-login console) as well as
/etc/motd (post-login SSH); fix SSH line root@ → spinifex@
Scans the USB EFI partition at boot for autoinstall.toml. When
enabled = true, the installer runs fully unattended using values from
the config file then ejects the USB and reboots — no user interaction
needed.
- New autoinstall package: TOML config struct, boot-media scanner,
install.Config converter, EjectAndReboot
- Scanner skips iso9660 mounts so the bundled reference copy of the
template never accidentally triggers headless mode
- "auto" disk target selects the largest non-removable disk
- "auto" NIC target picks the first physical interface
- Validates all required fields before touching any disk
- main.go: check for autoinstall config before launching TUI
Julian Sommer