mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-21 03:36:35 +00:00
tomnewton-mulga
28ca689130
* feat: add GPU discovery package (mulga-992) Adds spinifex/gpu package with sysfs-based GPU enumeration, IOMMU group member lookup, and a PCI device ID model table. Vendor-specific enrichment via nvidia-smi is attempted opportunistically and falls back to the built-in table when unavailable. All discovery and IOMMU logic is covered by unit tests using mock sysfs trees. * feat: add VFIO bind/unbind and GPU pool manager (mulga-993) Adds gpu.BindVFIO/UnbindVFIO using sysfs driver_override for per-device driver switching, and gpu.Manager which owns the claim/release lifecycle for the GPU pool including full IOMMU group binding, rollback on partial failure, and marking entries unavailable when rebind fails after VM termination. * feat: add G5 instance type generator with GPU model support (mulga-994) - Add GPUModel type, NVIDIAa10g constant, and GPUModelForVendorDevice lookup - Add generateGPUTypes to emit g5.* types with GpuInfo populated - Add IsGPUType helper for consumers filtering GPU-capable types - Update DetectAndGenerate to accept gpuModels []GPUModel parameter * feat: integrate GPU passthrough into daemon lifecycle (mulga-995) - Add gpu_passthrough config flag; discover GPUs and build manager at startup - Gate g5 instance types on GPU pool availability via canAllocateCount - Claim GPU (VFIO bind) in LaunchInstance; inject vfio-pci QEMU device - Release GPU (VFIO unbind, rebind to host driver) in stopInstance - Re-claim GPU on daemon restart when QEMU process is still alive - Add UseUEFI / OVMF support for x86_64 VMs (optional, falls back to SeaBIOS) - Fix Claim to use stored OriginalDriver when device is already vfio-pci bound * feat: add GPU fields to NodeStatus and spx top nodes (mulga-996) - Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes) - Populate from gpuManager in handleNodeStatus - Show GPU (used/total) column in spx top nodes; "-" when no GPUs present * feat: Ubuntu 24.04 GPU AMI build pipeline (mulga-997) Extends build-system-image.sh to support Ubuntu alongside Alpine. Key differences handled: partition table (parted resizepart p1), bind mounts for DKMS (/proc /sys /dev), apt vs apk, systemctl vs rc-update. DISTRO defaults to "alpine" for backward compatibility. Adds ubuntu-gpu.conf manifest and ubuntu-gpu-setup.sh chroot script that installs nvidia-headless-550-server with DKMS pre-built against a pinned linux-image-generic kernel, blacklists nouveau, and prevents unattended-upgrades from invalidating the pre-built module. * Fixed GPU passthrough and added test script * feat: always-probe GPU at startup, surface capability in NodeStatusResponse - Add gpuProbeResult struct and probeGPU() which runs at daemon startup regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU groups, and vfio_pci module presence. - Store gpuProbe on Daemon struct; split NewDaemon into probe (always) and activate (opt-in) phases. - Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse so operators can see hardware state before enabling passthrough. - Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but passthrough not active, 'used/total' = passthrough active. * feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001) - Export GenerateGPUTypes for use by the reload path - Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place (preserves references held by instanceService) and updates NATS subscriptions without full daemon restart - Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP in a context-aware loop; setupShutdown no longer intercepts SIGHUP - Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal, handles false→true (re-probe + activate) and true→false (refuse if instances running, otherwise deactivate) transitions * feat: spx admin gpu status/enable/disable commands (mulga-1000) - admin.SetGPUPassthrough: Go implementation of the idempotent TOML write (replaces the inline Python heredoc in spx-test-gpu.sh) - spx admin gpu status: shows GPU hardware, IOMMU/vfio state, passthrough activation, and advertised g5 instance types via NATS NodeStatusResponse - spx admin gpu enable: checks prerequisites, writes TOML, sends SIGHUP, polls until daemon confirms GPUPassthrough=true - spx admin gpu disable: refuses while GPU instances are running, otherwise writes TOML, sends SIGHUP, polls until confirmed - spx-test-gpu.sh: replace Python TOML heredoc with spx admin gpu enable * fixed admin * fix: filter Intel iGPU from passthrough pool, use lspci for consumer GPU names Discover() now skips non-NVIDIA/AMD vendors so Intel iGPUs never enter the passthrough pool. enrichFromLspci() uses the system PCI ID database via lspci -vmm to populate the model name for any card not in the hardcoded table, without needing nvidia-smi or driver tools. * fix: grant CAP_SYS_ADMIN and vfio device access to spinifex-daemon GPU passthrough requires writing to /sys/bus/pci/drivers/vfio-pci/unbind on instance stop. The kernel gates this on CAP_SYS_ADMIN regardless of file permissions. QEMU subprocesses also need char-vfio rw access to open /dev/vfio/N for the passed-through device. * fixing gpu bining permissions * working state of gpu passthrough on phyiscal box * installer grub image fixes * made gpu-passthrough more user facing * fixed banner * fixed banner * banner fix * gpu passthrough docs * preflight tweak * vpcd fixes to set br-wan to promisc for public IP DHCP processes * fixed vpcd br-wan promisc and veth pair * ssh keygen fix * fixes after running /review-branch * feat: re-wire GPU claim and device injection after vm.Manager refactor GPU passthrough claim (VFIO bind) and QEMU device injection were lost when dev's vm.Manager refactor (#185/#186) moved StartInstance/ CreateQMPClient into vm/lifecycle.go. Re-add both in their correct new homes: - daemon_handlers_instance.go: claim GPU before vmMgr.Run() in both RunInstances and StartStoppedInstance handlers; release on failure - vm/lifecycle.go startQEMU(): inject vfio-pci device with x-vga flag derived from instance.GPUXVGAEnabled when GPUPCIAddress is set * fix: Create spinifex user ~/.ssh and keys in firstboot --------- Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com> Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
65 lines
2.2 KiB
Desktop File
65 lines
2.2 KiB
Desktop File
[Unit]
|
|
Description=Spinifex Daemon (VM Orchestration)
|
|
PartOf=spinifex.target
|
|
After=spinifex-nats.service spinifex-viperblock.service openvswitch-switch.service ovn-controller.service
|
|
Wants=spinifex-nats.service
|
|
|
|
[Service]
|
|
Type=simple
|
|
User=spinifex-daemon
|
|
Group=spinifex
|
|
ExecStartPre=/var/lib/spinifex/wait-for-nats.sh
|
|
ExecStart=/usr/local/bin/spx service spinifex start
|
|
Restart=on-failure
|
|
RestartSec=5
|
|
OOMScoreAdjust=-500
|
|
Environment=SPINIFEX_CONFIG_PATH=/etc/spinifex/spinifex.toml
|
|
Environment=SPINIFEX_BASE_DIR=/var/lib/spinifex/spinifex/
|
|
Environment=SPINIFEX_WAL_DIR=/var/lib/spinifex/spinifex/
|
|
|
|
# Filesystem access — NoNewPrivileges omitted (daemon uses sudo for ip, ovs-vsctl, dhcpcd)
|
|
# CAP_SYS_ADMIN: required for GPU passthrough sysfs operations (vfio group management).
|
|
# CAP_DAC_OVERRIDE: sysfs driver bind/unbind paths (/sys/bus/pci/drivers/vfio-pci/bind,
|
|
# unbind) are root:root 0200; non-root processes need CAP_DAC_OVERRIDE to open them.
|
|
AmbientCapabilities=CAP_SYS_ADMIN CAP_DAC_OVERRIDE
|
|
ProtectSystem=full
|
|
PrivateDevices=no
|
|
DevicePolicy=closed
|
|
DeviceAllow=/dev/kvm rw
|
|
DeviceAllow=/dev/net/tun rw
|
|
DeviceAllow=char-vfio rw
|
|
DeviceAllow=/dev/vfio/vfio rw
|
|
ReadOnlyPaths=/etc/spinifex/spinifex.toml /etc/spinifex/server.pem /etc/spinifex/server.key /etc/spinifex/ca.pem
|
|
ReadWritePaths=/var/lib/spinifex/spinifex /var/log/spinifex /run/spinifex /run/spinifex/nbd
|
|
|
|
# Hardening
|
|
ProtectHome=yes
|
|
PrivateTmp=yes
|
|
RestrictSUIDSGID=yes
|
|
LockPersonality=yes
|
|
RestrictRealtime=yes
|
|
RestrictNamespaces=yes
|
|
ProtectClock=yes
|
|
ProtectKernelTunables=yes
|
|
ProtectKernelModules=yes
|
|
ProtectKernelLogs=yes
|
|
ProtectControlGroups=yes
|
|
ProtectProc=invisible
|
|
SystemCallArchitectures=native
|
|
MemoryDenyWriteExecute=yes
|
|
# DHCP lease acquisition moved to spinifex-vpcd (see dhcp-allocate-error.md);
|
|
# the daemon no longer needs AF_PACKET or the softened kernel protections.
|
|
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK
|
|
ProtectHostname=yes
|
|
KeyringMode=private
|
|
RemoveIPC=yes
|
|
LimitCORE=0
|
|
LimitNOFILE=65536
|
|
LimitNPROC=4096
|
|
# VFIO GPU passthrough requires locking all guest RAM into physical memory.
|
|
# Without this, QEMU fails with RLIMIT_MEMLOCK exceeded when pinning guest pages.
|
|
LimitMEMLOCK=infinity
|
|
|
|
[Install]
|
|
WantedBy=spinifex.target
|