Align the top of the README with mulgadc.com: new headline and intro,
rewritten "What is Spinifex?", a new "Three ways to deploy" section
(Neocloud / on-premise / edge) and a new "AWS compatibility" service
matrix (EC2, EBS, S3, VPC, IAM, ALB/NLB, EKS, ECR, ECS available; RDS
Q3 2026) linking the GitHub releases. Refresh the architecture prose and
Key Features, add Deploy and AWS-services nav links, and swap banner.svg
for the ember-on-ink mini-stack design matching platform.svg.
Rebuild the message-driven architecture diagram in the ember-on-ink
theme to match platform.svg: ember gradient NATS bus, ember-tinted
gateway and daemon cards, neutral clients and backend cards. Content
unchanged (clients → gateway → NATS → ec2/ebs/s3/vpc/iam daemons →
local backends). De-slop the copy and refresh the img alt text.
Replace the platform diagram with a static SVG matching the mulgadc.com
front-page stack diagram: ember accent, ink panel, the portable-stack
box (Neocloud / on-prem / edge) wrapping the Spinifex AWS-compatible
service layer, over standard Linux on CPU/GPU hardware. Self-contained
and GitHub-safe (presentation attributes only). Refresh the img alt text.
* docs: add ECS (Elastic Container Service) guide + register ecs-quickstart workbook
Add a Container Services -> ECS doc mirroring the EKS/ECR shape: concepts
(cluster, task definition, task, service, container instances via the
ECS-AMI EC2 launch type), an aws ecs / console / Terraform quickstart,
task IAM roles via the credential endpoint, awsvpc networking, and the
honest v1 limitations (no service discovery, awslogs discarded, no rolling
deployments, no capacity providers, no distinct execution role).
Convert the ecs-quickstart workbook README to the doc-page format
(frontmatter + overview/prerequisites/instructions/troubleshooting) so it
renders as a Terraform Workbooks entry alongside the EKS workbooks.
* docs: add node-image prerequisite banner + infra diagram to EKS and ECS pages
Add a GitHub-style [!IMPORTANT] banner to the EKS and ECS prerequisites
flagging the required node-image import (eks-node / spinifex-ecs-node),
mirroring the install page's WAN-bridge banner, each with a describe-images
verification command.
Add a basic Spinifex topology diagram (SVG) to each Overview, in the
tf-nginx-alb visual style: the EKS managed control plane + node-group
workers, and the ECS ALB + container instances with the gateway/agent
and task-credential path.
* feat(gpu): add NVIDIA MIG multi-instance GPU support
Partitions MIG-capable GPUs (A100, A30, H100) into isolated slices exposed
as mdev devices instead of whole-GPU VFIO passthrough. VMs attach via
-device vfio-pci,sysfsdev=<mdev-path> using the GPUAttachment discriminated
union (PCIAddress for whole-GPU, MdevPath for MIG).
Key changes:
- gpu: new MIGProfile/MIGInstance types, IsMIGCapable model map, mig.go ops
(EnableMIGMode, DisableMIGMode, CreateInstances, ListInstances,
DestroyAllInstances, enrichMdevPaths)
- gpu.Manager: Claim() returns (*GPUDevice, *MIGInstance, error); AddMIGInstances,
ReclaimByMdev for restart recovery; MarkMIGFailed per-slice
- daemon: buildGPUPool helper partitions devices into whole-GPU vs MIG pools;
MIGProfile config field selects profile per node or per GPU override
- vm.VM: GPUAttachments []gpu.GPUAttachment replaces GPUPCIAddresses []string
- admin CLI: spx admin gpu mig status/enable/disable subcommands with
admin.SetMIGProfile for spinifex.toml writes
- docs: COMMANDS.md updated with three new MIG management command rows
* test(gpu): unit tests for MIG parsers and pool manager
Cover all pure parsing functions (parseMIGMemory, isCapacityExhaustedError,
extractProfileFields/GIFields, parseMIGProfiles, parseMIGInstances,
parseCreatedGIID/CIID) with realistic nvidia-smi fixture output.
Manager MIG tests cover AddMIGInstances, Claim/Release on mdev-backed entries,
MarkMIGFailed, ReclaimByMdev, and interaction with whole-GPU MarkFailed — all
using fake mdev directories in t.TempDir() so no GPU hardware is required.
* feat(gpu): dynamic MIG profile-per-request with on-demand GI/CI lifecycle
Replaces static config-driven MIG pre-carving with a dynamic model where
the MIG profile is specified per RunInstances call via mig.<profile>
instance type names (e.g. mig.1g.10gb) that mirror nvidia-smi nomenclature
directly.
- Manager.Claim(instanceID, profileName): fast path claims a pre-carved
slice; slow path dequeues a free MIG GPU, releases the lock, calls
nvidia-smi mig -cgi/-cci to carve all slices, then reclaims the lock to
commit and return the first slice
- Manager.Release: last release on a dynamically-carved GPU calls
DestroyAllInstances and returns the GPU to freeMIGGPUs, allowing a
different profile on the next claim
- AddMIGGPU(dev): registers a fresh MIG GPU for dynamic allocation;
AddMIGInstances (restart recovery) keeps its no-destroy semantics
- GenerateMIGTypes: produces mig.<profile> InstanceTypeInfo entries from
gpu.ListProfiles at daemon startup; IsMIGType/MIGProfileFromType let
callers inspect these names
- buildGPUPool: calls ListInstances for recovery, AddMIGGPU for fresh
GPUs, ListProfiles to collect MIG profile specs for type generation;
removes setupMIGInstances and migProfileForDevice
- GPUClaimer.Claim gains profileName param; daemonGPUClaimer, service_impl
call sites, and fakeGPUClaimer all updated
* fix(gpu): correct mdev parent detection and make --profile optional for mig enable
enrichMdevPaths was using os.Readlink on "parent_bus_0" (a non-existent
sysfs attribute) and then on ".." (a directory, not a symlink) — both
would fail on real NVIDIA hardware, breaking CreateInstances and
ListInstances on daemon restart recovery. Fix: read the mdev UUID's own
symlink target, which always encodes the parent GPU's PCI address in its
path (e.g. .../0000:01:00.0/<uuid>).
Make basePath injectable in enrichMdevPaths so the fix is covered by a
new set of unit tests using temp-dir symlinks (TestEnrichMdevPaths_*).
Remove the MarkFlagRequired("profile") from "spx admin gpu mig enable":
Phase 2 is fully dynamic — the daemon carves slices on demand at
RunInstances time. Without --profile, the command just enables MIG mode
and destroys any pre-carved slices so the daemon starts with a free GPU.
With --profile, behaviour is unchanged (pre-carve a static pool).
* chore: writeups for smci and arcee
* test: add coverage for MIG exec, manager slow-path, instancetypes and admin
- Add mig_exec_test.go with fake nvidia-smi harness covering Enable/Disable MIGMode,
IsMIGModeEnabled, DestroyAllInstances, ListProfiles, CreateInstances, ListInstances
- Extend manager_mig_test.go with slow-path Claim/Release tests using fake nvidia-smi
- Add generate_test.go coverage for IsMIGType, MIGProfileFromType, GenerateMIGTypes
- Add admin_test.go coverage for SetMIGProfile (create, update, idempotent, error paths)
- Make mdevBasePath a var to allow sysfs injection in tests
- Add json tags to GPUAttachment fields to satisfy musttag linter
Diff coverage: 75.6% (threshold: 60%).
* fixed gpu passthrough doc
* Revert "fixed gpu passthrough doc"
This reverts commit 1e20f5ab13.
* feat: add GPU discovery package (mulga-992)
Adds spinifex/gpu package with sysfs-based GPU enumeration, IOMMU group
member lookup, and a PCI device ID model table. Vendor-specific enrichment
via nvidia-smi is attempted opportunistically and falls back to the built-in
table when unavailable. All discovery and IOMMU logic is covered by unit tests
using mock sysfs trees.
* feat: add VFIO bind/unbind and GPU pool manager (mulga-993)
Adds gpu.BindVFIO/UnbindVFIO using sysfs driver_override for per-device
driver switching, and gpu.Manager which owns the claim/release lifecycle
for the GPU pool including full IOMMU group binding, rollback on partial
failure, and marking entries unavailable when rebind fails after VM
termination.
* feat: add G5 instance type generator with GPU model support (mulga-994)
- Add GPUModel type, NVIDIAa10g constant, and GPUModelForVendorDevice lookup
- Add generateGPUTypes to emit g5.* types with GpuInfo populated
- Add IsGPUType helper for consumers filtering GPU-capable types
- Update DetectAndGenerate to accept gpuModels []GPUModel parameter
* feat: integrate GPU passthrough into daemon lifecycle (mulga-995)
- Add gpu_passthrough config flag; discover GPUs and build manager at startup
- Gate g5 instance types on GPU pool availability via canAllocateCount
- Claim GPU (VFIO bind) in LaunchInstance; inject vfio-pci QEMU device
- Release GPU (VFIO unbind, rebind to host driver) in stopInstance
- Re-claim GPU on daemon restart when QEMU process is still alive
- Add UseUEFI / OVMF support for x86_64 VMs (optional, falls back to SeaBIOS)
- Fix Claim to use stored OriginalDriver when device is already vfio-pci bound
* feat: add GPU fields to NodeStatus and spx top nodes (mulga-996)
- Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes)
- Populate from gpuManager in handleNodeStatus
- Show GPU (used/total) column in spx top nodes; "-" when no GPUs present
* feat: Ubuntu 24.04 GPU AMI build pipeline (mulga-997)
Extends build-system-image.sh to support Ubuntu alongside Alpine.
Key differences handled: partition table (parted resizepart p1),
bind mounts for DKMS (/proc /sys /dev), apt vs apk, systemctl vs
rc-update. DISTRO defaults to "alpine" for backward compatibility.
Adds ubuntu-gpu.conf manifest and ubuntu-gpu-setup.sh chroot script
that installs nvidia-headless-550-server with DKMS pre-built against
a pinned linux-image-generic kernel, blacklists nouveau, and prevents
unattended-upgrades from invalidating the pre-built module.
* Fixed GPU passthrough and added test script
* feat: always-probe GPU at startup, surface capability in NodeStatusResponse
- Add gpuProbeResult struct and probeGPU() which runs at daemon startup
regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU
groups, and vfio_pci module presence.
- Store gpuProbe on Daemon struct; split NewDaemon into probe (always)
and activate (opt-in) phases.
- Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse
so operators can see hardware state before enabling passthrough.
- Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but
passthrough not active, 'used/total' = passthrough active.
* feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001)
- Export GenerateGPUTypes for use by the reload path
- Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place
(preserves references held by instanceService) and updates NATS
subscriptions without full daemon restart
- Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP
in a context-aware loop; setupShutdown no longer intercepts SIGHUP
- Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal,
handles false→true (re-probe + activate) and true→false (refuse if
instances running, otherwise deactivate) transitions
* feat: spx admin gpu status/enable/disable commands (mulga-1000)
- admin.SetGPUPassthrough: Go implementation of the idempotent TOML write
(replaces the inline Python heredoc in spx-test-gpu.sh)
- spx admin gpu status: shows GPU hardware, IOMMU/vfio state, passthrough
activation, and advertised g5 instance types via NATS NodeStatusResponse
- spx admin gpu enable: checks prerequisites, writes TOML, sends SIGHUP,
polls until daemon confirms GPUPassthrough=true
- spx admin gpu disable: refuses while GPU instances are running, otherwise
writes TOML, sends SIGHUP, polls until confirmed
- spx-test-gpu.sh: replace Python TOML heredoc with spx admin gpu enable
* fixed admin
* fix: filter Intel iGPU from passthrough pool, use lspci for consumer GPU names
Discover() now skips non-NVIDIA/AMD vendors so Intel iGPUs never enter
the passthrough pool. enrichFromLspci() uses the system PCI ID database
via lspci -vmm to populate the model name for any card not in the
hardcoded table, without needing nvidia-smi or driver tools.
* fix: grant CAP_SYS_ADMIN and vfio device access to spinifex-daemon
GPU passthrough requires writing to /sys/bus/pci/drivers/vfio-pci/unbind
on instance stop. The kernel gates this on CAP_SYS_ADMIN regardless of
file permissions. QEMU subprocesses also need char-vfio rw access to
open /dev/vfio/N for the passed-through device.
* fixing gpu bining permissions
* working state of gpu passthrough on phyiscal box
* installer grub image fixes
* made gpu-passthrough more user facing
* fixed banner
* fixed banner
* banner fix
* gpu passthrough docs
* preflight tweak
* vpcd fixes to set br-wan to promisc for public IP DHCP processes
* fixed vpcd br-wan promisc and veth pair
* ssh keygen fix
* fixes after running /review-branch
* feat: re-wire GPU claim and device injection after vm.Manager refactor
GPU passthrough claim (VFIO bind) and QEMU device injection were lost
when dev's vm.Manager refactor (#185/#186) moved StartInstance/
CreateQMPClient into vm/lifecycle.go. Re-add both in their correct
new homes:
- daemon_handlers_instance.go: claim GPU before vmMgr.Run() in both
RunInstances and StartStoppedInstance handlers; release on failure
- vm/lifecycle.go startQEMU(): inject vfio-pci device with x-vga flag
derived from instance.GPUXVGAEnabled when GPUPCIAddress is set
* fix: Create spinifex user ~/.ssh and keys in firstboot
---------
Co-authored-by: Ben Duncan <5178243+benduncan@users.noreply.github.com>
Co-authored-by: Julian Sommer <115516918+juliansommer@users.noreply.github.com>
tomnewton-mulga