mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
tomnewton-mulga
9d8597317a
* feat(gpu): add NVIDIA MIG multi-instance GPU support
Partitions MIG-capable GPUs (A100, A30, H100) into isolated slices exposed
as mdev devices instead of whole-GPU VFIO passthrough. VMs attach via
-device vfio-pci,sysfsdev=<mdev-path> using the GPUAttachment discriminated
union (PCIAddress for whole-GPU, MdevPath for MIG).
Key changes:
- gpu: new MIGProfile/MIGInstance types, IsMIGCapable model map, mig.go ops
(EnableMIGMode, DisableMIGMode, CreateInstances, ListInstances,
DestroyAllInstances, enrichMdevPaths)
- gpu.Manager: Claim() returns (*GPUDevice, *MIGInstance, error); AddMIGInstances,
ReclaimByMdev for restart recovery; MarkMIGFailed per-slice
- daemon: buildGPUPool helper partitions devices into whole-GPU vs MIG pools;
MIGProfile config field selects profile per node or per GPU override
- vm.VM: GPUAttachments []gpu.GPUAttachment replaces GPUPCIAddresses []string
- admin CLI: spx admin gpu mig status/enable/disable subcommands with
admin.SetMIGProfile for spinifex.toml writes
- docs: COMMANDS.md updated with three new MIG management command rows
* test(gpu): unit tests for MIG parsers and pool manager
Cover all pure parsing functions (parseMIGMemory, isCapacityExhaustedError,
extractProfileFields/GIFields, parseMIGProfiles, parseMIGInstances,
parseCreatedGIID/CIID) with realistic nvidia-smi fixture output.
Manager MIG tests cover AddMIGInstances, Claim/Release on mdev-backed entries,
MarkMIGFailed, ReclaimByMdev, and interaction with whole-GPU MarkFailed — all
using fake mdev directories in t.TempDir() so no GPU hardware is required.
* feat(gpu): dynamic MIG profile-per-request with on-demand GI/CI lifecycle
Replaces static config-driven MIG pre-carving with a dynamic model where
the MIG profile is specified per RunInstances call via mig.<profile>
instance type names (e.g. mig.1g.10gb) that mirror nvidia-smi nomenclature
directly.
- Manager.Claim(instanceID, profileName): fast path claims a pre-carved
slice; slow path dequeues a free MIG GPU, releases the lock, calls
nvidia-smi mig -cgi/-cci to carve all slices, then reclaims the lock to
commit and return the first slice
- Manager.Release: last release on a dynamically-carved GPU calls
DestroyAllInstances and returns the GPU to freeMIGGPUs, allowing a
different profile on the next claim
- AddMIGGPU(dev): registers a fresh MIG GPU for dynamic allocation;
AddMIGInstances (restart recovery) keeps its no-destroy semantics
- GenerateMIGTypes: produces mig.<profile> InstanceTypeInfo entries from
gpu.ListProfiles at daemon startup; IsMIGType/MIGProfileFromType let
callers inspect these names
- buildGPUPool: calls ListInstances for recovery, AddMIGGPU for fresh
GPUs, ListProfiles to collect MIG profile specs for type generation;
removes setupMIGInstances and migProfileForDevice
- GPUClaimer.Claim gains profileName param; daemonGPUClaimer, service_impl
call sites, and fakeGPUClaimer all updated
* fix(gpu): correct mdev parent detection and make --profile optional for mig enable
enrichMdevPaths was using os.Readlink on "parent_bus_0" (a non-existent
sysfs attribute) and then on ".." (a directory, not a symlink) — both
would fail on real NVIDIA hardware, breaking CreateInstances and
ListInstances on daemon restart recovery. Fix: read the mdev UUID's own
symlink target, which always encodes the parent GPU's PCI address in its
path (e.g. .../0000:01:00.0/<uuid>).
Make basePath injectable in enrichMdevPaths so the fix is covered by a
new set of unit tests using temp-dir symlinks (TestEnrichMdevPaths_*).
Remove the MarkFlagRequired("profile") from "spx admin gpu mig enable":
Phase 2 is fully dynamic — the daemon carves slices on demand at
RunInstances time. Without --profile, the command just enables MIG mode
and destroys any pre-carved slices so the daemon starts with a free GPU.
With --profile, behaviour is unchanged (pre-carve a static pool).
* chore: writeups for smci and arcee
* test: add coverage for MIG exec, manager slow-path, instancetypes and admin
- Add mig_exec_test.go with fake nvidia-smi harness covering Enable/Disable MIGMode,
IsMIGModeEnabled, DestroyAllInstances, ListProfiles, CreateInstances, ListInstances
- Extend manager_mig_test.go with slow-path Claim/Release tests using fake nvidia-smi
- Add generate_test.go coverage for IsMIGType, MIGProfileFromType, GenerateMIGTypes
- Add admin_test.go coverage for SetMIGProfile (create, update, idempotent, error paths)
- Make mdevBasePath a var to allow sysfs injection in tests
- Add json tags to GPUAttachment fields to satisfy musttag linter
Diff coverage: 75.6% (threshold: 60%).
* fixed gpu passthrough doc
* Revert "fixed gpu passthrough doc"
This reverts commit 1e20f5ab13.
* feat: Added RTX 6000 GPU MIG support and fixed docs
* feat: expose GPU/MIG inventory in GetNodes and GetVMs admin API
Add Manager.Snapshot() for lock-safe pool inspection. Extend
NodeStatusResponse with GPUs []GPUInfo (per-physical-GPU inventory
with per-slice detail for MIG) and VMInfo with GPU *VMGPUInfo
(model, VRAM, profile, mdev path for the attached GPU). Populate
both in handleNodeStatus and handleNodeVMs via buildGPUInventory,
buildPoolLookup, and resolveVMGPU helpers.
* feat: surface GPU info on EC2 instance detail page (mulga-new-03)
Admin users see a GPU card (model, VRAM, attachment type, MIG profile/mdev
path or PCI address) populated from the GetVMs admin API. Adds formatVRAMMiB
helper and VMGPUInfo/GPUInfo/GPUSliceInfo types to the frontend.
* feat: GPU columns + inventory card on Nodes page (mulga-new-04)
Adds GPU column to VMsTable (model/VRAM or MIG profile) and ResourceTable
(alloc/total GPUs). Section 4 GPU Inventory card shows per-node GPU state with
per-slice MIG drill-down. Also adds total_gpus/alloc_gpus to NodeInfo type.
* feat: grouped GPU-aware instance type picker in Run Instances (mulga-new-05)
Splits the flat instance type dropdown into Standard and GPU-Accelerated groups.
GPU items show total VRAM alongside the type name. Selecting a GPU type surfaces
an info line with GPU name and VRAM.
* testing image builder scripts
* fixed nvidia build script
* more descriptive errors for image build script
* more fixes
* more fixes
* fix: surface PCI address in whole-GPU VMGPUInfo and rebuild frontend dist
Add PCIAddress to VMGPUInfo, populate it in resolveVMGPU for whole-GPU
passthrough, and fix the instance detail page which was incorrectly
showing instance type in the PCI address field. Rebuilt frontend dist
to include GPU UI changes (inventory card, resource columns, GPU-aware
instance type picker).
* fix: improve kernel version detection in nvidia image build script
Fall back to /lib/modules when /boot/vmlinuz-* is absent, and add
diagnostic output to help debug kernel detection failures.
* fixing nvidia image build script
* fix ami name issue
* correct kernel versions
* another fix
* image build works, added docker and nvidia toolkit for QoL
* fix: add gpg
* fixing amd image builder
* add universe
* another fix
* nvm
* fix docker user group issues
* make rocm land where it is supposed to
* fix docker groups
* feat: add Ubuntu 26.04 GPU AMI catalog entries and fix build script parted detection
Add ubuntu-26.04-nvidia-gpu-x86_64 and ubuntu-26.04-amd-gpu-x86_64 to AvailableImages,
hosted on iso.mulgadc.com/system-ami/ with SHA256 checksums. Fix build-system-image.sh
to detect parted in /usr/sbin when not on PATH. Suppress gosec false positives in DHCP
manager (cancel stored in leaseLoop) and promisc ioctl (standard syscall uintptr pattern).
* Updated passthrough doc regarding GPU images
* fix: remove orphaned nolint directives (golangci-lint v2.12.2 no longer flags G115/G118)
* chore: remove stale spinifex@192.168.3.107 directory
Scripts were accidentally committed with a node hostname as the path
prefix during initial testing. Correct copies live in scripts/images/.
15 lines
413 B
Plaintext
15 lines
413 B
Plaintext
IMAGE_NAME="ubuntu-amd-gpu"
|
|
IMAGE_DESCRIPTION="Ubuntu 26.04 AMD GPU guest image — linux-firmware, ROCm CLI, Python toolchain"
|
|
DISTRO="ubuntu"
|
|
UBUNTU_VERSION="26.04"
|
|
UBUNTU_CODENAME="resolute"
|
|
|
|
# 16G provides headroom for kernel headers, linux-firmware, ROCm packages, and Docker CE.
|
|
IMAGE_SIZE="16G"
|
|
|
|
SETUP_SCRIPT="scripts/images/ubuntu-gpu-amd-setup.sh"
|
|
|
|
AMI_NAME="ubuntu-amd-gpu"
|
|
|
|
SYSTEM_TAG="gpu-vendor=amd"
|