* feat(gpu): add NVIDIA MIG multi-instance GPU support
Partitions MIG-capable GPUs (A100, A30, H100) into isolated slices exposed
as mdev devices instead of whole-GPU VFIO passthrough. VMs attach via
-device vfio-pci,sysfsdev=<mdev-path> using the GPUAttachment discriminated
union (PCIAddress for whole-GPU, MdevPath for MIG).
Key changes:
- gpu: new MIGProfile/MIGInstance types, IsMIGCapable model map, mig.go ops
(EnableMIGMode, DisableMIGMode, CreateInstances, ListInstances,
DestroyAllInstances, enrichMdevPaths)
- gpu.Manager: Claim() returns (*GPUDevice, *MIGInstance, error); AddMIGInstances,
ReclaimByMdev for restart recovery; MarkMIGFailed per-slice
- daemon: buildGPUPool helper partitions devices into whole-GPU vs MIG pools;
MIGProfile config field selects profile per node or per GPU override
- vm.VM: GPUAttachments []gpu.GPUAttachment replaces GPUPCIAddresses []string
- admin CLI: spx admin gpu mig status/enable/disable subcommands with
admin.SetMIGProfile for spinifex.toml writes
- docs: COMMANDS.md updated with three new MIG management command rows
* test(gpu): unit tests for MIG parsers and pool manager
Cover all pure parsing functions (parseMIGMemory, isCapacityExhaustedError,
extractProfileFields/GIFields, parseMIGProfiles, parseMIGInstances,
parseCreatedGIID/CIID) with realistic nvidia-smi fixture output.
Manager MIG tests cover AddMIGInstances, Claim/Release on mdev-backed entries,
MarkMIGFailed, ReclaimByMdev, and interaction with whole-GPU MarkFailed — all
using fake mdev directories in t.TempDir() so no GPU hardware is required.
* feat(gpu): dynamic MIG profile-per-request with on-demand GI/CI lifecycle
Replaces static config-driven MIG pre-carving with a dynamic model where
the MIG profile is specified per RunInstances call via mig.<profile>
instance type names (e.g. mig.1g.10gb) that mirror nvidia-smi nomenclature
directly.
- Manager.Claim(instanceID, profileName): fast path claims a pre-carved
slice; slow path dequeues a free MIG GPU, releases the lock, calls
nvidia-smi mig -cgi/-cci to carve all slices, then reclaims the lock to
commit and return the first slice
- Manager.Release: last release on a dynamically-carved GPU calls
DestroyAllInstances and returns the GPU to freeMIGGPUs, allowing a
different profile on the next claim
- AddMIGGPU(dev): registers a fresh MIG GPU for dynamic allocation;
AddMIGInstances (restart recovery) keeps its no-destroy semantics
- GenerateMIGTypes: produces mig.<profile> InstanceTypeInfo entries from
gpu.ListProfiles at daemon startup; IsMIGType/MIGProfileFromType let
callers inspect these names
- buildGPUPool: calls ListInstances for recovery, AddMIGGPU for fresh
GPUs, ListProfiles to collect MIG profile specs for type generation;
removes setupMIGInstances and migProfileForDevice
- GPUClaimer.Claim gains profileName param; daemonGPUClaimer, service_impl
call sites, and fakeGPUClaimer all updated
* fix(gpu): correct mdev parent detection and make --profile optional for mig enable
enrichMdevPaths was using os.Readlink on "parent_bus_0" (a non-existent
sysfs attribute) and then on ".." (a directory, not a symlink) — both
would fail on real NVIDIA hardware, breaking CreateInstances and
ListInstances on daemon restart recovery. Fix: read the mdev UUID's own
symlink target, which always encodes the parent GPU's PCI address in its
path (e.g. .../0000:01:00.0/<uuid>).
Make basePath injectable in enrichMdevPaths so the fix is covered by a
new set of unit tests using temp-dir symlinks (TestEnrichMdevPaths_*).
Remove the MarkFlagRequired("profile") from "spx admin gpu mig enable":
Phase 2 is fully dynamic — the daemon carves slices on demand at
RunInstances time. Without --profile, the command just enables MIG mode
and destroys any pre-carved slices so the daemon starts with a free GPU.
With --profile, behaviour is unchanged (pre-carve a static pool).
* chore: writeups for smci and arcee
* test: add coverage for MIG exec, manager slow-path, instancetypes and admin
- Add mig_exec_test.go with fake nvidia-smi harness covering Enable/Disable MIGMode,
IsMIGModeEnabled, DestroyAllInstances, ListProfiles, CreateInstances, ListInstances
- Extend manager_mig_test.go with slow-path Claim/Release tests using fake nvidia-smi
- Add generate_test.go coverage for IsMIGType, MIGProfileFromType, GenerateMIGTypes
- Add admin_test.go coverage for SetMIGProfile (create, update, idempotent, error paths)
- Make mdevBasePath a var to allow sysfs injection in tests
- Add json tags to GPUAttachment fields to satisfy musttag linter
Diff coverage: 75.6% (threshold: 60%).
* fixed gpu passthrough doc
* Revert "fixed gpu passthrough doc"
This reverts commit 1e20f5ab13.
* feat: Added RTX 6000 GPU MIG support and fixed docs
* feat: expose GPU/MIG inventory in GetNodes and GetVMs admin API
Add Manager.Snapshot() for lock-safe pool inspection. Extend
NodeStatusResponse with GPUs []GPUInfo (per-physical-GPU inventory
with per-slice detail for MIG) and VMInfo with GPU *VMGPUInfo
(model, VRAM, profile, mdev path for the attached GPU). Populate
both in handleNodeStatus and handleNodeVMs via buildGPUInventory,
buildPoolLookup, and resolveVMGPU helpers.
* feat: surface GPU info on EC2 instance detail page (mulga-new-03)
Admin users see a GPU card (model, VRAM, attachment type, MIG profile/mdev
path or PCI address) populated from the GetVMs admin API. Adds formatVRAMMiB
helper and VMGPUInfo/GPUInfo/GPUSliceInfo types to the frontend.
* feat: GPU columns + inventory card on Nodes page (mulga-new-04)
Adds GPU column to VMsTable (model/VRAM or MIG profile) and ResourceTable
(alloc/total GPUs). Section 4 GPU Inventory card shows per-node GPU state with
per-slice MIG drill-down. Also adds total_gpus/alloc_gpus to NodeInfo type.
* feat: grouped GPU-aware instance type picker in Run Instances (mulga-new-05)
Splits the flat instance type dropdown into Standard and GPU-Accelerated groups.
GPU items show total VRAM alongside the type name. Selecting a GPU type surfaces
an info line with GPU name and VRAM.
* testing image builder scripts
* fixed nvidia build script
* more descriptive errors for image build script
* more fixes
* more fixes
* fix: surface PCI address in whole-GPU VMGPUInfo and rebuild frontend dist
Add PCIAddress to VMGPUInfo, populate it in resolveVMGPU for whole-GPU
passthrough, and fix the instance detail page which was incorrectly
showing instance type in the PCI address field. Rebuilt frontend dist
to include GPU UI changes (inventory card, resource columns, GPU-aware
instance type picker).
* fix: improve kernel version detection in nvidia image build script
Fall back to /lib/modules when /boot/vmlinuz-* is absent, and add
diagnostic output to help debug kernel detection failures.
* fixing nvidia image build script
* fix ami name issue
* correct kernel versions
* another fix
* image build works, added docker and nvidia toolkit for QoL
* fix: add gpg
* fixing amd image builder
* add universe
* another fix
* nvm
* fix docker user group issues
* make rocm land where it is supposed to
* fix docker groups
* feat: add Ubuntu 26.04 GPU AMI catalog entries and fix build script parted detection
Add ubuntu-26.04-nvidia-gpu-x86_64 and ubuntu-26.04-amd-gpu-x86_64 to AvailableImages,
hosted on iso.mulgadc.com/system-ami/ with SHA256 checksums. Fix build-system-image.sh
to detect parted in /usr/sbin when not on PATH. Suppress gosec false positives in DHCP
manager (cancel stored in leaseLoop) and promisc ioctl (standard syscall uintptr pattern).
* Updated passthrough doc regarding GPU images
* fix: remove orphaned nolint directives (golangci-lint v2.12.2 no longer flags G115/G118)
* chore: remove stale spinifex@192.168.3.107 directory
Scripts were accidentally committed with a node hostname as the path
prefix during initial testing. Correct copies live in scripts/images/.
* feat: add GPU fields to NodeStatus and spx top nodes (mulga-996)
- Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes)
- Populate from gpuManager in handleNodeStatus
- Show GPU (used/total) column in spx top nodes; "-" when no GPUs present
* Fixed GPU passthrough and added test script
* feat: always-probe GPU at startup, surface capability in NodeStatusResponse
- Add gpuProbeResult struct and probeGPU() which runs at daemon startup
regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU
groups, and vfio_pci module presence.
- Store gpuProbe on Daemon struct; split NewDaemon into probe (always)
and activate (opt-in) phases.
- Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse
so operators can see hardware state before enabling passthrough.
- Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but
passthrough not active, 'used/total' = passthrough active.
* feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001)
- Export GenerateGPUTypes for use by the reload path
- Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place
(preserves references held by instanceService) and updates NATS
subscriptions without full daemon restart
- Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP
in a context-aware loop; setupShutdown no longer intercepts SIGHUP
- Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal,
handles false→true (re-probe + activate) and true→false (refuse if
instances running, otherwise deactivate) transitions
* installer grub image fixes
* made gpu-passthrough more user facing
* vpcd fixes to set br-wan to promisc for public IP DHCP processes
* fixed vpcd br-wan promisc and veth pair
* ssh keygen fix
* Fix: AMD GPU edge cases
* script for running 8x instances on SMCI computer
* fix: increase disk space on instances
* feat: multi-GPU passthrough support for g7e.12xlarge (2× MI350X)
- vm.VM: GPUPCIAddress string → GPUPCIAddresses []string for multi-GPU
- instancetypes: add g7e.12xlarge (2 GPUs), GPUCountForType(), AMDmi350x model
- instancetypes/generate: use GPUCountForType for GpuInfo.Count and TotalGpuMemoryInMiB
- gpu/manager: Release() loops all pool entries for instanceID (not just first)
- vm/lifecycle: loop over GPUPCIAddresses, emit gpu0/gpu1/... QEMU devices
- daemon/handlers: claim N GPUs in loop for both RunInstances and StartStoppedInstance
- daemon/vm_adapters: ReclaimByAddress loops GPUPCIAddresses; ReleaseGPU checks len > 0
- scripts/launch-gpu-fleet: fixed fleet — 6× g7e.4xlarge + 1× g7e.12xlarge
* feat: per-instance-type disk size in GPU fleet (300 GB single, 600 GB dual)
* fix: multi-gpu instance types
* fix: kill QEMU synchronously when QMP handshake fails to free VFIO device
When startQEMU succeeds but newQMPClientWithHandshake times out, the
instance has not yet been inserted into the manager map so vmMgr.Delete
in the rollback path is a no-op. QEMU keeps running and holds the VFIO
group open. The GPU pool entry gets freed, the next Claim returns the
same PCI address, and every subsequent QEMU fails with 'device or
resource busy'.
Fix: SIGKILL QEMU and wait for it to exit before returning the QMP
error, ensuring the VFIO device is released before the caller recycles
the GPU pool slot.
* fix: quarantine GPU after QEMU launch failure to prevent re-assignment cascade
When QEMU crashes during startup (e.g. pci_irq_handler assertion on a
specific GPU), the pool Release frees that address and the next NATS
request immediately re-claims the same broken device. Add MarkFailed()
to gpu.Manager which sets Available=false for a PCI address after a
launch failure, preventing re-assignment until the daemon restarts.
Call MarkFailed for each GPU address in both RunInstances and
StartStoppedInstance rollback paths after the Release succeeds.
* fixed instance fleet launch script
* remove known hosts and stagger provisioning
* improvements to launch script
* more upgrades
* single instance gpu instance provisioning
* fixed single instance provision
* smci computer demo dashboard
* fixes to launch script
* updated smci dashboard_server
* fixed demo dashboard
* updated dashboard.py
* feat: GPU Passthrough fixes to passthrough process - builds proper AMD or NVIDA ready images, generalised launch script
* feat: gpu-passthrough updating image builder to include gpu-ready ubuntu images
* fixing image builder process
* fixed image builder to remove /tmp/ file before import to free space
* fix: Updated COMMANDS.md to reflect new GPU management admin commands
* Update docs, update internal launch script to work with new SGs, test tweaks
* Updated Docs, fixed internal launch-gpu.sh script, test tweaks
* fxing CI race due to GPU load
Julian Sommer