mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-20 03:11:46 +00:00
28ca689130
* feat: add GPU discovery package (mulga-992) Adds spinifex/gpu package with sysfs-based GPU enumeration, IOMMU group member lookup, and a PCI device ID model table. Vendor-specific enrichment via nvidia-smi is attempted opportunistically and falls back to the built-in table when unavailable. All discovery and IOMMU logic is covered by unit tests using mock sysfs trees. * feat: add VFIO bind/unbind and GPU pool manager (mulga-993) Adds gpu.BindVFIO/UnbindVFIO using sysfs driver_override for per-device driver switching, and gpu.Manager which owns the claim/release lifecycle for the GPU pool including full IOMMU group binding, rollback on partial failure, and marking entries unavailable when rebind fails after VM termination. * feat: add G5 instance type generator with GPU model support (mulga-994) - Add GPUModel type, NVIDIAa10g constant, and GPUModelForVendorDevice lookup - Add generateGPUTypes to emit g5.* types with GpuInfo populated - Add IsGPUType helper for consumers filtering GPU-capable types - Update DetectAndGenerate to accept gpuModels []GPUModel parameter * feat: integrate GPU passthrough into daemon lifecycle (mulga-995) - Add gpu_passthrough config flag; discover GPUs and build manager at startup - Gate g5 instance types on GPU pool availability via canAllocateCount - Claim GPU (VFIO bind) in LaunchInstance; inject vfio-pci QEMU device - Release GPU (VFIO unbind, rebind to host driver) in stopInstance - Re-claim GPU on daemon restart when QEMU process is still alive - Add UseUEFI / OVMF support for x86_64 VMs (optional, falls back to SeaBIOS) - Fix Claim to use stored OriginalDriver when device is already vfio-pci bound * feat: add GPU fields to NodeStatus and spx top nodes (mulga-996) - Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes) - Populate from gpuManager in handleNodeStatus - Show GPU (used/total) column in spx top nodes; "-" when no GPUs present * feat: Ubuntu 24.04 GPU AMI build pipeline (mulga-997) Extends build-system-image.sh to support Ubuntu alongside Alpine. Key differences handled: partition table (parted resizepart p1), bind mounts for DKMS (/proc /sys /dev), apt vs apk, systemctl vs rc-update. DISTRO defaults to "alpine" for backward compatibility. Adds ubuntu-gpu.conf manifest and ubuntu-gpu-setup.sh chroot script that installs nvidia-headless-550-server with DKMS pre-built against a pinned linux-image-generic kernel, blacklists nouveau, and prevents unattended-upgrades from invalidating the pre-built module. * Fixed GPU passthrough and added test script * feat: always-probe GPU at startup, surface capability in NodeStatusResponse - Add gpuProbeResult struct and probeGPU() which runs at daemon startup regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU groups, and vfio_pci module presence. - Store gpuProbe on Daemon struct; split NewDaemon into probe (always) and activate (opt-in) phases. - Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse so operators can see hardware state before enabling passthrough. - Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but passthrough not active, 'used/total' = passthrough active. * feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001) - Export GenerateGPUTypes for use by the reload path - Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place (preserves references held by instanceService) and updates NATS subscriptions without full daemon restart - Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP in a context-aware loop; setupShutdown no longer intercepts SIGHUP - Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal, handles false→true (re-probe + activate) and true→false (refuse if instances running, otherwise deactivate) transitions * feat: spx admin gpu status/enable/disable commands (mulga-1000) - admin.SetGPUPassthrough: Go implementation of the idempotent TOML write (replaces the inline Python heredoc in spx-test-gpu.sh) - spx admin gpu status: shows GPU hardware, IOMMU/vfio state, passthrough activation, and advertised g5 instance types via NATS NodeStatusResponse - spx admin gpu enable: checks prerequisites, writes TOML, sends SIGHUP, polls until daemon confirms GPUPassthrough=true - spx admin gpu disable: refuses while GPU instances are running, otherwise writes TOML, sends SIGHUP, polls until confirmed - spx-test-gpu.sh: replace Python TOML heredoc with spx admin gpu enable * fixed admin * fix: filter Intel iGPU from passthrough pool, use lspci for consumer GPU names Discover() now skips non-NVIDIA/AMD vendors so Intel iGPUs never enter the passthrough pool. enrichFromLspci() uses the system PCI ID database via lspci -vmm to populate the model name for any card not in the hardcoded table, without needing nvidia-smi or driver tools. * fix: grant CAP_SYS_ADMIN and vfio device access to spinifex-daemon GPU passthrough requires writing to /sys/bus/pci/drivers/vfio-pci/unbind on instance stop. The kernel gates this on CAP_SYS_ADMIN regardless of file permissions. QEMU subprocesses also need char-vfio rw access to open /dev/vfio/N for the passed-through device. * fixing gpu bining permissions * working state of gpu passthrough on phyiscal box * installer grub image fixes * made gpu-passthrough more user facing * fixed banner * fixed banner * banner fix * gpu passthrough docs * preflight tweak * vpcd fixes to set br-wan to promisc for public IP DHCP processes * fixed vpcd br-wan promisc and veth pair * ssh keygen fix * fixes after running /review-branch * feat: re-wire GPU claim and device injection after vm.Manager refactor GPU passthrough claim (VFIO bind) and QEMU device injection were lost when dev's vm.Manager refactor (#185/#186) moved StartInstance/ CreateQMPClient into vm/lifecycle.go. Re-add both in their correct new homes: - daemon_handlers_instance.go: claim GPU before vmMgr.Run() in both RunInstances and StartStoppedInstance handlers; release on failure - vm/lifecycle.go startQEMU(): inject vfio-pci device with x-vga flag derived from instance.GPUXVGAEnabled when GPUPCIAddress is set * fix: Create spinifex user ~/.ssh and keys in firstboot --------- Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com> Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
351 lines
10 KiB
Go
351 lines
10 KiB
Go
/*
|
|
Copyright © 2026 Mulga Defense Corporation
|
|
|
|
This program is free software: you can redistribute it and/or modify
|
|
it under the terms of the GNU Affero General Public License as published by
|
|
the Free Software Foundation, either version 3 of the License, or
|
|
(at your option) any later version.
|
|
|
|
This program is distributed in the hope that it will be useful,
|
|
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
GNU Affero General Public License for more details.
|
|
|
|
You should have received a copy of the GNU Affero General Public License
|
|
along with this program. If not, see <https://www.gnu.org/licenses/>.
|
|
*/
|
|
|
|
// Package autoinstall reads headless install parameters from environment
|
|
// variables exported by spinifex-init.sh from the kernel cmdline.
|
|
// Parameters are set in the GRUB "Headless" menu entry in grub.cfg.
|
|
package autoinstall
|
|
|
|
import (
|
|
"fmt"
|
|
"log/slog"
|
|
"net"
|
|
"os"
|
|
"os/exec"
|
|
"sort"
|
|
"strconv"
|
|
"strings"
|
|
"syscall"
|
|
"time"
|
|
|
|
"github.com/mulgadc/spinifex/cmd/installer/install"
|
|
)
|
|
|
|
// Load returns an install.Config built from SPINIFEX_* environment variables,
|
|
// or nil if SPINIFEX_AUTO is not set to "1" (interactive mode).
|
|
func Load() (*install.Config, error) {
|
|
if os.Getenv("SPINIFEX_AUTO") != "1" {
|
|
return nil, nil
|
|
}
|
|
|
|
slog.Info("autoinstall: headless mode enabled via kernel cmdline")
|
|
|
|
cfg, err := buildConfig()
|
|
if err != nil {
|
|
return nil, fmt.Errorf("autoinstall config: %w", err)
|
|
}
|
|
return cfg, nil
|
|
}
|
|
|
|
// EjectAndReboot ejects the boot device (best-effort) then reboots.
|
|
func EjectAndReboot() {
|
|
srcDev, _ := os.ReadFile("/run/iso-dev")
|
|
if dev := strings.TrimSpace(string(srcDev)); dev != "" {
|
|
slog.Info("autoinstall: ejecting boot device", "device", dev)
|
|
_ = exec.Command("eject", dev).Run()
|
|
}
|
|
|
|
fmt.Println()
|
|
fmt.Println("Installation complete.")
|
|
fmt.Println("Remove the USB drive now if it was not ejected automatically.")
|
|
fmt.Println("Rebooting in 10 seconds...")
|
|
time.Sleep(10 * time.Second)
|
|
// Use the kernel syscall directly — the live environment runs spinifex-init
|
|
// as PID 1 (not systemd), so exec("reboot") fails trying to reach D-Bus.
|
|
if err := syscall.Reboot(syscall.LINUX_REBOOT_CMD_RESTART); err != nil {
|
|
slog.Error("autoinstall: reboot syscall failed", "err", err)
|
|
}
|
|
}
|
|
|
|
func buildConfig() (*install.Config, error) {
|
|
password := os.Getenv("SPINIFEX_PASSWORD")
|
|
if password == "" {
|
|
return nil, fmt.Errorf("SPINIFEX_PASSWORD is required")
|
|
}
|
|
|
|
hostname := os.Getenv("SPINIFEX_HOSTNAME")
|
|
if hostname == "" {
|
|
hostname = "spinifex-node"
|
|
}
|
|
|
|
disk, err := resolveDisk(os.Getenv("SPINIFEX_DISK"))
|
|
if err != nil {
|
|
return nil, fmt.Errorf("disk: %w", err)
|
|
}
|
|
|
|
wanIface, err := resolveNIC(os.Getenv("SPINIFEX_WAN_IFACE"), "")
|
|
if err != nil {
|
|
return nil, fmt.Errorf("WAN NIC: %w", err)
|
|
}
|
|
|
|
cfg := &install.Config{
|
|
Disk: disk,
|
|
Hostname: hostname,
|
|
RootPassword: password,
|
|
// Optional on the headless path — empty means no --email passed to
|
|
// spx admin init, which omits the operator identity from telemetry.
|
|
Email: strings.TrimSpace(os.Getenv("SPINIFEX_EMAIL")),
|
|
WANInterface: wanIface,
|
|
WANDHCPMode: strings.ToLower(os.Getenv("SPINIFEX_WAN_MODE")) != "static",
|
|
}
|
|
|
|
if !cfg.WANDHCPMode {
|
|
ip := os.Getenv("SPINIFEX_WAN_IP")
|
|
mask := os.Getenv("SPINIFEX_WAN_MASK")
|
|
gw := os.Getenv("SPINIFEX_WAN_GW")
|
|
if ip == "" || mask == "" || gw == "" {
|
|
return nil, fmt.Errorf("SPINIFEX_WAN_IP, SPINIFEX_WAN_MASK, SPINIFEX_WAN_GW required for static mode")
|
|
}
|
|
cfg.WANAddress = ip
|
|
cfg.WANMask = mask
|
|
cfg.WANGateway = gw
|
|
if dns := os.Getenv("SPINIFEX_WAN_DNS"); dns != "" {
|
|
cfg.WANDNS = strings.Split(dns, ",")
|
|
}
|
|
}
|
|
|
|
if lanIface := os.Getenv("SPINIFEX_LAN_IFACE"); lanIface != "" {
|
|
lan, err := resolveNIC(lanIface, wanIface)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("LAN NIC: %w", err)
|
|
}
|
|
cfg.LANInterface = lan
|
|
cfg.LANDHCPMode = strings.ToLower(os.Getenv("SPINIFEX_LAN_MODE")) != "static"
|
|
if !cfg.LANDHCPMode {
|
|
cfg.LANAddress = os.Getenv("SPINIFEX_LAN_IP")
|
|
cfg.LANMask = os.Getenv("SPINIFEX_LAN_MASK")
|
|
if dns := os.Getenv("SPINIFEX_LAN_DNS"); dns != "" {
|
|
cfg.LANDNS = strings.Split(dns, ",")
|
|
}
|
|
}
|
|
}
|
|
|
|
if os.Getenv("SPINIFEX_GPU_PASSTHROUGH") == "1" {
|
|
cfg.GPUPassthrough = true
|
|
}
|
|
|
|
role := strings.ToLower(os.Getenv("SPINIFEX_ROLE"))
|
|
if role == "" {
|
|
role = "init"
|
|
}
|
|
cfg.ClusterRole = role
|
|
if role == "join" {
|
|
joinAddr := os.Getenv("SPINIFEX_JOIN_ADDR")
|
|
if joinAddr == "" {
|
|
return nil, fmt.Errorf("SPINIFEX_JOIN_ADDR required when SPINIFEX_ROLE=join")
|
|
}
|
|
cfg.JoinAddr = joinAddr
|
|
}
|
|
|
|
return cfg, nil
|
|
}
|
|
|
|
// resolveDisk maps the spx_disk value from grub.cfg to a block device path.
|
|
//
|
|
// Supported values:
|
|
//
|
|
// auto — use the only non-removable disk; fail if multiple found
|
|
// largest — largest non-removable disk (explicit opt-in)
|
|
// smallest — smallest non-removable disk (typical OS-on-SSD pattern)
|
|
// nvme — the only NVMe disk; fail if multiple found
|
|
// /dev/sda (etc.) — exact device path; fail if not present
|
|
func resolveDisk(target string) (string, error) {
|
|
switch strings.ToLower(target) {
|
|
case "", "auto":
|
|
return singleDisk()
|
|
case "largest":
|
|
return diskBySize(true)
|
|
case "smallest":
|
|
return diskBySize(false)
|
|
case "nvme":
|
|
return nvmeDisk()
|
|
default:
|
|
if _, err := os.Stat(target); err != nil {
|
|
return "", fmt.Errorf("%q not found", target)
|
|
}
|
|
return target, nil
|
|
}
|
|
}
|
|
|
|
type diskCandidate struct {
|
|
dev string
|
|
bytes int64
|
|
}
|
|
|
|
// nonRemovableDisks returns all non-removable, non-virtual block devices.
|
|
func nonRemovableDisks() ([]diskCandidate, error) {
|
|
entries, err := os.ReadDir("/sys/block")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
var out []diskCandidate
|
|
for _, e := range entries {
|
|
dev := e.Name()
|
|
switch {
|
|
case strings.HasPrefix(dev, "loop"),
|
|
strings.HasPrefix(dev, "ram"),
|
|
strings.HasPrefix(dev, "dm-"),
|
|
strings.HasPrefix(dev, "sr"):
|
|
continue
|
|
}
|
|
removable, _ := os.ReadFile("/sys/block/" + dev + "/removable")
|
|
if strings.TrimSpace(string(removable)) == "1" {
|
|
continue
|
|
}
|
|
sizeRaw, _ := os.ReadFile("/sys/block/" + dev + "/size")
|
|
sectors, parseErr := strconv.ParseInt(strings.TrimSpace(string(sizeRaw)), 10, 64)
|
|
if parseErr != nil || sectors == 0 {
|
|
continue
|
|
}
|
|
out = append(out, diskCandidate{dev: dev, bytes: sectors * 512})
|
|
}
|
|
return out, nil
|
|
}
|
|
|
|
// diskList returns a human-readable list of candidates for error messages.
|
|
func diskList(disks []diskCandidate) string {
|
|
lines := make([]string, len(disks))
|
|
for i, d := range disks {
|
|
lines[i] = fmt.Sprintf(" /dev/%s (%dG)", d.dev, d.bytes>>30)
|
|
}
|
|
return strings.Join(lines, "\n")
|
|
}
|
|
|
|
// singleDisk selects the only non-removable disk, or fails listing all found.
|
|
func singleDisk() (string, error) {
|
|
disks, err := nonRemovableDisks()
|
|
if err != nil {
|
|
return "", err
|
|
}
|
|
switch len(disks) {
|
|
case 0:
|
|
return "", fmt.Errorf("no non-removable disks found")
|
|
case 1:
|
|
slog.Info("autoinstall: single disk selected", "disk", disks[0].dev, "size_gb", disks[0].bytes>>30)
|
|
return "/dev/" + disks[0].dev, nil
|
|
default:
|
|
return "", fmt.Errorf(
|
|
"multiple disks found — set SPINIFEX_DISK to one of:\n%s\n"+
|
|
" largest (largest disk)\n"+
|
|
" smallest (smallest disk)\n"+
|
|
" nvme (NVMe only)\n"+
|
|
" /dev/sdX (exact path)",
|
|
diskList(disks),
|
|
)
|
|
}
|
|
}
|
|
|
|
// diskBySize selects the largest (largest=true) or smallest non-removable disk.
|
|
func diskBySize(largest bool) (string, error) {
|
|
disks, err := nonRemovableDisks()
|
|
if err != nil {
|
|
return "", err
|
|
}
|
|
if len(disks) == 0 {
|
|
return "", fmt.Errorf("no non-removable disks found")
|
|
}
|
|
sort.Slice(disks, func(i, j int) bool {
|
|
if largest {
|
|
return disks[i].bytes > disks[j].bytes
|
|
}
|
|
return disks[i].bytes < disks[j].bytes
|
|
})
|
|
label := "smallest"
|
|
if largest {
|
|
label = "largest"
|
|
}
|
|
slog.Info("autoinstall: disk selected by size", "mode", label, "disk", disks[0].dev, "size_gb", disks[0].bytes>>30)
|
|
return "/dev/" + disks[0].dev, nil
|
|
}
|
|
|
|
// nvmeDisk selects the only NVMe disk, or fails listing all NVMe disks found.
|
|
func nvmeDisk() (string, error) {
|
|
disks, err := nonRemovableDisks()
|
|
if err != nil {
|
|
return "", err
|
|
}
|
|
var nvmes []diskCandidate
|
|
for _, d := range disks {
|
|
if strings.HasPrefix(d.dev, "nvme") {
|
|
nvmes = append(nvmes, d)
|
|
}
|
|
}
|
|
switch len(nvmes) {
|
|
case 0:
|
|
return "", fmt.Errorf("no NVMe disks found")
|
|
case 1:
|
|
slog.Info("autoinstall: NVMe disk selected", "disk", nvmes[0].dev, "size_gb", nvmes[0].bytes>>30)
|
|
return "/dev/" + nvmes[0].dev, nil
|
|
default:
|
|
return "", fmt.Errorf(
|
|
"multiple NVMe disks found — set SPINIFEX_DISK to one of:\n%s",
|
|
diskList(nvmes),
|
|
)
|
|
}
|
|
}
|
|
|
|
// virtualNICPrefixes lists interface name prefixes that identify non-physical
|
|
// interfaces (bridges, tunnels, container/OVS veth pairs, etc.). These are
|
|
// skipped when auto-selecting a NIC so we don't configure docker0 or
|
|
// ovs-system as the management interface on a machine that previously ran
|
|
// Docker or OVN.
|
|
var virtualNICPrefixes = []string{
|
|
"docker", "veth", "virbr", "br-", "ovs-", "vxlan", "genev", "tun", "tap",
|
|
}
|
|
|
|
// resolveNIC returns the interface name to use. "auto" or empty picks the
|
|
// first physical (non-loopback, non-virtual) interface that is not exclude.
|
|
func resolveNIC(name, exclude string) (string, error) {
|
|
if name != "" && name != "auto" {
|
|
return name, nil
|
|
}
|
|
ifaces, err := net.Interfaces()
|
|
if err != nil {
|
|
return "", err
|
|
}
|
|
for _, iface := range ifaces {
|
|
if iface.Flags&net.FlagLoopback != 0 {
|
|
continue
|
|
}
|
|
// Require broadcast capability — filters out point-to-point tunnels.
|
|
if iface.Flags&net.FlagBroadcast == 0 {
|
|
continue
|
|
}
|
|
// Require a MAC address — virtual/tunnel interfaces have none.
|
|
if len(iface.HardwareAddr) == 0 {
|
|
continue
|
|
}
|
|
if iface.Name == exclude {
|
|
continue
|
|
}
|
|
// Skip known virtual interface prefixes.
|
|
virtual := false
|
|
for _, pfx := range virtualNICPrefixes {
|
|
if strings.HasPrefix(iface.Name, pfx) {
|
|
virtual = true
|
|
break
|
|
}
|
|
}
|
|
if virtual {
|
|
continue
|
|
}
|
|
slog.Info("autoinstall: NIC resolved", "mode", "auto", "selected", iface.Name)
|
|
return iface.Name, nil
|
|
}
|
|
return "", fmt.Errorf("no suitable physical NIC found (non-loopback, broadcast-capable, with MAC)")
|
|
}
|