mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-20 03:11:46 +00:00
28ca689130
* feat: add GPU discovery package (mulga-992) Adds spinifex/gpu package with sysfs-based GPU enumeration, IOMMU group member lookup, and a PCI device ID model table. Vendor-specific enrichment via nvidia-smi is attempted opportunistically and falls back to the built-in table when unavailable. All discovery and IOMMU logic is covered by unit tests using mock sysfs trees. * feat: add VFIO bind/unbind and GPU pool manager (mulga-993) Adds gpu.BindVFIO/UnbindVFIO using sysfs driver_override for per-device driver switching, and gpu.Manager which owns the claim/release lifecycle for the GPU pool including full IOMMU group binding, rollback on partial failure, and marking entries unavailable when rebind fails after VM termination. * feat: add G5 instance type generator with GPU model support (mulga-994) - Add GPUModel type, NVIDIAa10g constant, and GPUModelForVendorDevice lookup - Add generateGPUTypes to emit g5.* types with GpuInfo populated - Add IsGPUType helper for consumers filtering GPU-capable types - Update DetectAndGenerate to accept gpuModels []GPUModel parameter * feat: integrate GPU passthrough into daemon lifecycle (mulga-995) - Add gpu_passthrough config flag; discover GPUs and build manager at startup - Gate g5 instance types on GPU pool availability via canAllocateCount - Claim GPU (VFIO bind) in LaunchInstance; inject vfio-pci QEMU device - Release GPU (VFIO unbind, rebind to host driver) in stopInstance - Re-claim GPU on daemon restart when QEMU process is still alive - Add UseUEFI / OVMF support for x86_64 VMs (optional, falls back to SeaBIOS) - Fix Claim to use stored OriginalDriver when device is already vfio-pci bound * feat: add GPU fields to NodeStatus and spx top nodes (mulga-996) - Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes) - Populate from gpuManager in handleNodeStatus - Show GPU (used/total) column in spx top nodes; "-" when no GPUs present * feat: Ubuntu 24.04 GPU AMI build pipeline (mulga-997) Extends build-system-image.sh to support Ubuntu alongside Alpine. Key differences handled: partition table (parted resizepart p1), bind mounts for DKMS (/proc /sys /dev), apt vs apk, systemctl vs rc-update. DISTRO defaults to "alpine" for backward compatibility. Adds ubuntu-gpu.conf manifest and ubuntu-gpu-setup.sh chroot script that installs nvidia-headless-550-server with DKMS pre-built against a pinned linux-image-generic kernel, blacklists nouveau, and prevents unattended-upgrades from invalidating the pre-built module. * Fixed GPU passthrough and added test script * feat: always-probe GPU at startup, surface capability in NodeStatusResponse - Add gpuProbeResult struct and probeGPU() which runs at daemon startup regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU groups, and vfio_pci module presence. - Store gpuProbe on Daemon struct; split NewDaemon into probe (always) and activate (opt-in) phases. - Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse so operators can see hardware state before enabling passthrough. - Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but passthrough not active, 'used/total' = passthrough active. * feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001) - Export GenerateGPUTypes for use by the reload path - Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place (preserves references held by instanceService) and updates NATS subscriptions without full daemon restart - Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP in a context-aware loop; setupShutdown no longer intercepts SIGHUP - Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal, handles false→true (re-probe + activate) and true→false (refuse if instances running, otherwise deactivate) transitions * feat: spx admin gpu status/enable/disable commands (mulga-1000) - admin.SetGPUPassthrough: Go implementation of the idempotent TOML write (replaces the inline Python heredoc in spx-test-gpu.sh) - spx admin gpu status: shows GPU hardware, IOMMU/vfio state, passthrough activation, and advertised g5 instance types via NATS NodeStatusResponse - spx admin gpu enable: checks prerequisites, writes TOML, sends SIGHUP, polls until daemon confirms GPUPassthrough=true - spx admin gpu disable: refuses while GPU instances are running, otherwise writes TOML, sends SIGHUP, polls until confirmed - spx-test-gpu.sh: replace Python TOML heredoc with spx admin gpu enable * fixed admin * fix: filter Intel iGPU from passthrough pool, use lspci for consumer GPU names Discover() now skips non-NVIDIA/AMD vendors so Intel iGPUs never enter the passthrough pool. enrichFromLspci() uses the system PCI ID database via lspci -vmm to populate the model name for any card not in the hardcoded table, without needing nvidia-smi or driver tools. * fix: grant CAP_SYS_ADMIN and vfio device access to spinifex-daemon GPU passthrough requires writing to /sys/bus/pci/drivers/vfio-pci/unbind on instance stop. The kernel gates this on CAP_SYS_ADMIN regardless of file permissions. QEMU subprocesses also need char-vfio rw access to open /dev/vfio/N for the passed-through device. * fixing gpu bining permissions * working state of gpu passthrough on phyiscal box * installer grub image fixes * made gpu-passthrough more user facing * fixed banner * fixed banner * banner fix * gpu passthrough docs * preflight tweak * vpcd fixes to set br-wan to promisc for public IP DHCP processes * fixed vpcd br-wan promisc and veth pair * ssh keygen fix * fixes after running /review-branch * feat: re-wire GPU claim and device injection after vm.Manager refactor GPU passthrough claim (VFIO bind) and QEMU device injection were lost when dev's vm.Manager refactor (#185/#186) moved StartInstance/ CreateQMPClient into vm/lifecycle.go. Re-add both in their correct new homes: - daemon_handlers_instance.go: claim GPU before vmMgr.Run() in both RunInstances and StartStoppedInstance handlers; release on failure - vm/lifecycle.go startQEMU(): inject vfio-pci device with x-vga flag derived from instance.GPUXVGAEnabled when GPUPCIAddress is set * fix: Create spinifex user ~/.ssh and keys in firstboot --------- Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com> Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
173 lines
5.5 KiB
Go
173 lines
5.5 KiB
Go
// dhcptest is a diagnostic tool that runs a DHCP DORA handshake using the
|
|
// same nclient4 library and options that spinifex-vpcd uses for gateway pool
|
|
// leases. Run it on a node to verify DHCP works at the library level before
|
|
// debugging the full spinifex stack.
|
|
//
|
|
// Usage (must run as root — needs AF_PACKET):
|
|
//
|
|
// sudo dhcptest --iface=br-wan
|
|
// sudo dhcptest --iface=br-wan --mac=02:00:00:7c:b7:b9 --promisc
|
|
// sudo dhcptest --iface=br-wan --spinifex-id=gateway-wan1
|
|
package main
|
|
|
|
import (
|
|
"context"
|
|
"flag"
|
|
"fmt"
|
|
"log"
|
|
"net"
|
|
"os"
|
|
"os/exec"
|
|
"strings"
|
|
"time"
|
|
|
|
"github.com/insomniacslk/dhcp/dhcpv4"
|
|
"github.com/insomniacslk/dhcp/dhcpv4/nclient4"
|
|
)
|
|
|
|
func main() {
|
|
iface := flag.String("iface", "br-wan", "Interface/bridge to bind the DHCP client to")
|
|
macStr := flag.String("mac", "", "Hardware address for DORA (chaddr + AF_PACKET filter). If empty, uses the real interface MAC.")
|
|
spinifexID := flag.String("spinifex-id", "", "Generate a spinifex virtual MAC from this resource ID (e.g. 'gateway-wan1'). Overrides --mac.")
|
|
clientID := flag.String("client-id", "", "DHCP option 61 client identifier. Defaults to --spinifex-id or --mac value.")
|
|
hostname := flag.String("hostname", "", "DHCP option 12 hostname")
|
|
vendorClass := flag.String("vendor-class", "mulga-spinifex-gw", "DHCP option 60 vendor class identifier")
|
|
promisc := flag.Bool("promisc", false, "Enable promiscuous mode on the interface before the test (mirrors vpcd's setBridgePromisc)")
|
|
timeout := flag.Duration("timeout", 10*time.Second, "Per-attempt DHCP timeout")
|
|
retries := flag.Int("retries", 3, "Number of DORA attempts before giving up")
|
|
verbose := flag.Bool("verbose", false, "Print full DHCP packet contents")
|
|
flag.Parse()
|
|
|
|
// Resolve the hardware address to use.
|
|
var hwAddr net.HardwareAddr
|
|
var err error
|
|
|
|
switch {
|
|
case *spinifexID != "":
|
|
hwAddr, err = net.ParseMAC(generateMAC(*spinifexID))
|
|
if err != nil {
|
|
log.Fatalf("generateMAC(%q): %v", *spinifexID, err)
|
|
}
|
|
fmt.Printf("spinifex virtual MAC for %q: %s\n", *spinifexID, hwAddr)
|
|
case *macStr != "":
|
|
hwAddr, err = net.ParseMAC(*macStr)
|
|
if err != nil {
|
|
log.Fatalf("parse --mac %q: %v", *macStr, err)
|
|
}
|
|
default:
|
|
hwAddr, err = ifaceMAC(*iface)
|
|
if err != nil {
|
|
log.Fatalf("get MAC for %s: %v", *iface, err)
|
|
}
|
|
fmt.Printf("using real interface MAC: %s\n", hwAddr)
|
|
}
|
|
|
|
// Resolve client-id: default to spinifex-id, then MAC string.
|
|
cid := *clientID
|
|
if cid == "" {
|
|
if *spinifexID != "" {
|
|
cid = *spinifexID
|
|
} else {
|
|
cid = hwAddr.String()
|
|
}
|
|
}
|
|
|
|
fmt.Printf("iface: %s\n", *iface)
|
|
fmt.Printf("hw-addr: %s\n", hwAddr)
|
|
fmt.Printf("client-id: %s\n", cid)
|
|
fmt.Printf("hostname: %s\n", *hostname)
|
|
fmt.Printf("vendor-class: %s\n", *vendorClass)
|
|
fmt.Printf("promisc: %v\n", *promisc)
|
|
fmt.Printf("timeout: %s retries: %d\n", *timeout, *retries)
|
|
fmt.Println()
|
|
|
|
if *promisc {
|
|
if err := setPromisc(*iface, true); err != nil {
|
|
log.Printf("WARNING: failed to set promisc on %s: %v", *iface, err)
|
|
} else {
|
|
fmt.Printf("set %s promisc ON\n\n", *iface)
|
|
}
|
|
}
|
|
|
|
// Build nclient4 options — identical to spinifex/services/vpcd/dhcp/nclient4.go.
|
|
opts := []nclient4.ClientOpt{
|
|
nclient4.WithHWAddr(hwAddr),
|
|
nclient4.WithTimeout(*timeout),
|
|
nclient4.WithRetry(*retries),
|
|
}
|
|
if *verbose {
|
|
opts = append(opts, nclient4.WithLogger(nclient4.ShortSummaryLogger{Printfer: log.New(os.Stderr, "[nclient4] ", 0)}))
|
|
}
|
|
|
|
client, err := nclient4.New(*iface, opts...)
|
|
if err != nil {
|
|
log.Fatalf("nclient4.New(%s): %v", *iface, err)
|
|
}
|
|
defer client.Close()
|
|
|
|
// Identity modifiers — identical to identityModifiers() in vpcd dhcp package.
|
|
var mods []dhcpv4.Modifier
|
|
if cid != "" {
|
|
mods = append(mods, dhcpv4.WithOption(dhcpv4.OptClientIdentifier([]byte(cid))))
|
|
}
|
|
if *hostname != "" {
|
|
mods = append(mods, dhcpv4.WithOption(dhcpv4.OptHostName(*hostname)))
|
|
}
|
|
if *vendorClass != "" {
|
|
mods = append(mods, dhcpv4.WithOption(dhcpv4.OptClassIdentifier(*vendorClass)))
|
|
}
|
|
|
|
fmt.Println("Sending DHCP Discover...")
|
|
start := time.Now()
|
|
ctx, cancel := context.WithTimeout(context.Background(), *timeout*time.Duration(*retries+1))
|
|
defer cancel()
|
|
|
|
lease, err := client.Request(ctx, mods...)
|
|
if err != nil {
|
|
fmt.Printf("\nFAILED after %s: %v\n", time.Since(start).Round(time.Millisecond), err)
|
|
os.Exit(1)
|
|
}
|
|
|
|
fmt.Printf("SUCCESS in %s\n\n", time.Since(start).Round(time.Millisecond))
|
|
ack := lease.ACK
|
|
fmt.Printf(" Your IP: %s\n", ack.YourIPAddr)
|
|
fmt.Printf(" Subnet: %s\n", ack.SubnetMask())
|
|
fmt.Printf(" Gateway: %v\n", ack.Router())
|
|
fmt.Printf(" DNS: %v\n", ack.DNS())
|
|
fmt.Printf(" Server ID: %s\n", ack.ServerIdentifier())
|
|
fmt.Printf(" Lease time: %s\n", ack.IPAddressLeaseTime(24*time.Hour))
|
|
if *verbose {
|
|
fmt.Printf("\n--- Full ACK ---\n%s\n", ack.Summary())
|
|
}
|
|
}
|
|
|
|
// generateMAC is the same deterministic hash used by vpcd's topology.go.
|
|
// It produces a locally-administered MAC (02:00:00:xx:xx:xx) from a resource ID.
|
|
func generateMAC(resourceID string) string {
|
|
h := uint32(0)
|
|
for _, c := range resourceID {
|
|
h = h*31 + uint32(c)
|
|
}
|
|
return fmt.Sprintf("02:00:00:%02x:%02x:%02x", (h>>16)&0xff, (h>>8)&0xff, h&0xff)
|
|
}
|
|
|
|
func ifaceMAC(name string) (net.HardwareAddr, error) {
|
|
ifc, err := net.InterfaceByName(name)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return ifc.HardwareAddr, nil
|
|
}
|
|
|
|
func setPromisc(iface string, on bool) error {
|
|
state := "on"
|
|
if !on {
|
|
state = "off"
|
|
}
|
|
out, err := exec.Command("ip", "link", "set", iface, "promisc", state).CombinedOutput()
|
|
if err != nil {
|
|
return fmt.Errorf("%w: %s", err, strings.TrimSpace(string(out)))
|
|
}
|
|
return nil
|
|
}
|