Three related nat-mode fixes:
- ura01: nat-transit pool now emits gw_lrp_range_start/end. admin init
sets 100.127.0.16-.254 (239 VPC gateways) instead of the auto-derived
top-16 (~15). Adds GwLrpRange* to PoolData + the spinifex.toml template
so operators can widen any pool.
- 724mm: detectPoolMode returns true only for external_mode=pool. nat and
disabled clusters now skip Phase 8b/8d, which otherwise failed on
AllocateAddress UnsupportedOperation.
- pnz2i: document that system instances (ECS/EKS/LB agents) are
unsupported in nat v1 (require pool mode) in the vpc-networking README
and COMMANDS.md; document the gw_lrp_range_* fields.
* feat(network): add routed uplink mode, Routed wiring, and NAT egress rule helper
* feat(external): route gateway LRP over transit pool and install VPC SNAT in routed NAT mode
* feat(vpcd): support bridge_mode=nat (routed uplink, transit pool, egress rules)
* feat(daemon): register EIP handlers unconditionally with disabled-mode stub; seed default subnet per external mode
* feat(admin): support --external-mode=nat and suggest it for non-bridgeable uplinks
* feat(scripts): add --nat-uplink routed mode to setup-ovn.sh
* test(e2e): add routed-NAT single-node natuplink suite
Covers the external_mode=nat lane end to end on a nat-initialized node:
host transit wiring (veth, br-ext port, ip_forward, iptables egress
rules), spinifex.toml contents, disabled EIP surface (empty
describe-addresses, UnsupportedOperation on allocate), default subnet
MapPublicIpOnLaunch=false, OVN gateway LRP + VPC SNAT on the transit
/24, guest outbound egress proven via a serial-console probe (nat mode
has no inbound path), and unique transit gateway IPs across VPCs.
Suite self-gates on external_mode=nat; registered as e2e-natuplink and
in SUITES_BUILD so it compiles in CI. Manifest entry stays commented
until CI can provision a nat-mode environment.
* feat(ovn): Address_Set CRUD and NAT exempted_ext_ips column
Adds the exempted_ext_ips column (optional strong Address_Set ref) to the
nbdb NAT model, Address_Set ensure/get with wait-op create semantics, and an
in-place SetNATExemptedExtIPs update on the client interface, live client,
and mock. Test NB schema gains the column and covers insert-resolves-UUID,
converge, clear, and missing-NAT paths.
* feat(policy): exempt address set on routed SNAT/EIP rows
Adds WithSNATExemptSet so routed mode stamps exempted_ext_ips (the
spinifex_nat_exempt Address_Set: transit /24 plus operator extras) on IGW
snat and EIP dnat_and_snat rows, keeping VM replies to host-initiated flows
un-NATted. AddSNAT gains an idempotency guard keyed on (router, VPC CIDR)
with stale-IP replacement, and both AddSNAT and the AddEIP idempotent-skip
path patch the ref in place on pre-existing rows. The set is a singleton
strong ref and is never deleted. Non-routed modes are unchanged.
* feat(host): VPC ingress route primitives for routed-NAT mode
EnsureVPCIngressRoute / RemoveVPCIngressRoute manage the host route that
sends VPC-CIDR traffic into OVN via the gateway LRP transit IP on the
spx-nat veth. Replace/del semantics keep both idempotent; a missing route
on removal is not an error.
* feat(external): routed ingress leg on IGW attach/detach, reconcile-safe
AttachIGW's routed leg now also installs the host VPC ingress route via
RoutedIngressHooks, and re-runs on the already-attached path so the
reconcile loop restores host routes (and patches legacy SNAT exempt refs)
after a reboot — OVN state survives restarts but host routes do not.
DetachIGW removes the route alongside the SNAT. vpcd wires the hooks and
the spinifex_nat_exempt set (transit /24 + [network] nat_exempt_cidrs,
validated nat-mode-only) exclusively in routed-NAT mode; pool/veth/direct
pass nil hooks and are unchanged.
* test(natuplink): Tier 1 host-ingress phase; docs for jumpbox pattern
New Phase 7 asserts the automatic ingress tier: host route to the VPC CIDR
via the gateway LRP transit IP, exempted_ext_ips ref on the SNAT row
pointing at spinifex_nat_exempt (containing the transit /24), and — after
opening the SG from the transit net (default-closed, AWS parity) — ping and
an sshd TCP handshake against the instance private IP from the host, which
proves the un-SNATted return path. Phase 8 additionally waits for the
second VPC's ingress route to appear on IGW attach. vpc-networking README
documents the jumpbox pattern, SG note, and nat_exempt_cidrs.
* feat(config,admin): public wan pool alongside nat-transit
Restructure admin config generation from single-pool scalars to a Pools
slice so nat mode can emit both the transit pool and an optional public
wan pool. nat mode now accepts --external-pool/--external-source (plus
--external-bind-bridge/--gateway-ip when a public pool is requested),
validated by the shared resolvePublicPoolFlags helper also used by pool
mode. Transit pool always renders first so the IGW allocator picks it.
Formation wire format unchanged (nat mode is single-node only).
* feat(daemon): enable EIP service and auto-assign in nat mode with public pool
External IPAM (and with it the real EIP service, RunInstances public IP
auto-assign, and MapPublicIpOnLaunch on default subnets) now activates
whenever public pools exist: pool mode as before, plus nat mode when a
wan pool rides alongside the transit segment. The transit pool is
filtered out of IPAM — its addresses are gateway-LRP plumbing, never
allocatable EIPs. Transit-only nat configs keep the disabled EIP stub.
* feat(host): per-EIP ingress primitives for routed-NAT mode
DetectUplinkFor parses ip route get to find the interface and source IP
facing the pool gateway (re-run per ensure: WiFi/cellular uplinks move).
EnsureEIPIngress steers an EIP into OVN with a /32 route via the VPC
gateway LRP transit IP (src-hinted so host-local traffic still DNATs),
answers ARP for the EIP with a proxy neighbor entry (L3 only, same MAC,
WiFi-safe), sends gratuitous ARP, and adds per-EIP FORWARD accepts for
DROP-policy hosts. RemoveEIPIngress mirrors; missing state tolerated.
* feat(policy,vpcd): host EIP binder option on NATManager
WithHostEIPBinder injects routed-mode host plumbing hooks. AddEIP binds
after priming on both the fresh-add and idempotent-skip paths (host
routes and proxy-ARP are volatile even when the OVN row survives, so
reconcile re-ensures them after reboot); DeleteEIP unbinds best-effort.
The next hop comes from the new gatewayPortIP helper reading the gateway
LRP's stamped spinifex:gateway_ip (Networks fallback), with the ext-ID
key now an exported policy const shared with the external package.
vpcd selects the IGW pool by name instead of index — with two pools in
nat mode the gateway allocator must stay on the transit pool — and wires
the binder over host.EnsureEIPIngress/RemoveEIPIngress when a public
pool is present. EnsureEIPIngress gains an uplink hint so dhcp-sourced
pools (gateway unknown until lease) proxy-ARP on their bind bridge.
* feat(dhcp): interface-MAC lease variant with MAC-keyed-router detection
dhcp_mac = "derived" (default) | "interface" on external pools. The
interface variant leases with the bind interface's own MAC — WiFi/WWAN
uplinks drop frames with foreign source MACs, so derived chaddr MACs
never complete DORA there. Option 61 stays the lease discriminator.
UseIfaceMAC threads AllocateRequest → NATS wire → nclient4 → lease and
persists through KV so restart re-acquires repeat it.
Routers that key leases on MAC and ignore client-id would hand every
lease the same IP; the manager detects an ACK matching the interface's
own address or another client-id's live lease, releases the colliding
lease, and hard-errors advising a static range instead.
admin init auto-writes dhcp_mac="interface" for nat-mode public dhcp
pools bound to non-bridgeable uplinks (wlan/wwan/ppp).
* test(natuplink)+docs: Tier 2 EIP ingress e2e phases and public-pool docs
The natuplink suite now detects a public (non-transit) pool in spinifex.toml
and switches lanes: Tier-1-only configs keep the EIP-disabled and
no-public-IP assertions; public-pool configs assert the EIP surface is
enabled, the default subnet has MapPublicIpOnLaunch=true, instances
auto-assign public IPs, and a new Phase 9 exercises the full EIP ingress
lifecycle — host /32 route via the gateway LRP, proxy-ARP on the uplink,
per-EIP FORWARD accepts, exempt-stamped dnat_and_snat, TCP handshake to the
EIP, reconcile replay across a vpcd restart, and teardown on disassociate.
Docs: vpc-networking README gains a public-pool section for routed-NAT nodes
(init flags, dhcp_mac=interface behaviour, host-to-EIP caveat) and an updated
mode comparison; COMMANDS.md notes nat-mode EIP support behind a public pool.
* fix(setup): whitelist iptables, neigh sysctl, and arping for spinifex-vpcd
Routed-NAT mode shells out to iptables for the transit masquerade and
per-EIP FORWARD accepts, tunes proxy_delay, and sends gratuitous ARP for
host-delivered EIPs. Without these sudoers entries vpcd crash-loops with
"command not allowed" on nat-mode nodes.
* fix(ovn): scope DeleteStaticRoute prefix match to the target router
Every VPC router carries a 0.0.0.0/0 static route, so an unscoped cache
match could grab another router's row and fail with a referential
integrity violation, wedging IGW attach on multi-VPC nodes. Match only
rows referenced by the target router, mirroring FindStaticRoute. Mock
updated to the same semantics, with a regression test covering drift
replacement across two routers.
* fix(reconcile): always replay AttachIGW so host ingress state is re-ensured
applyIGWs skipped AttachIGW whenever the gateway switch port existed in
actual state, so the already-attached re-ensure path never ran and
routed-NAT host ingress routes lost on reboot were never reinstalled.
AttachIGW is idempotent; run it unconditionally.
* fix(vpcd): resolve duplicate-CIDR host ingress route collisions, bootstrap VPC wins
Every account's default VPC shares 172.31.0.0/16, so Tier-1 host routes
collide on the single host route table and the last IGW attach silently
stole the route (route replace). Pass vpcID and the gateway LRP IP
through RoutedIngressHooks; the bootstrap VPC always installs its route,
other VPCs only install when the CIDR is free and only remove a route
their own gateway IP holds. Adds host.VPCIngressRouteVia to read the
current route holder.
* ci(e2e): add nightly cell 19 for routed-NAT single-node suite
Cell 19 bootstraps with --external-mode=nat plus the CI static pool as
the public wan pool and runs the natuplink suite, covering the Tier-1
transit datapath and Tier-2 EIP ingress. The run step exits 0 when
natuplink.test is absent from the tests tar so scheduled runs against
branches predating the suite stay green.
* fix(admin): nat-mode init summary reflects configured public pool
The summary printed 'outbound-only — no public IPs/EIPs' even when
--external-pool/--external-source configured a public wan pool. Print
the pool range (or dhcp bind) and note EIPs are enabled.
* fix(eip,vpc): thread ctx through disabled EIP stub and NAT call sites
Reconcile the routed-NAT branch with dev's ctx-first service APIs after
rebase: DisabledEIPService and its tests take ctx to satisfy the EIPService
interface, the vm_adapters public-IP resolver asserts the ctx-signed
AssociatedPublicIPForInstance, and the default-subnet mapping test threads
ctx into DescribeSubnets.
* feat: add OVSDB RAFT cluster primitives to setup-ovn.sh
Add --db-cluster-local-addr / --db-cluster-remote-addr flags so a
management node can run the OVN NB/SB databases as a native OVSDB RAFT
cluster instead of a single non-clustered ovn-central. When a local addr
is given, write a shared OVN_CTL_OPTS to /etc/default/ovn-central (NB
RAFT 6643 / SB RAFT 6644) and restart the per-DB units directly, since
the ovn-central aggregator is ExecStart=/bin/true and won't restart its
children. Empty remote addr creates the cluster (init node); a set remote
addr joins it. NB/SB client set-connection runs only on the create node
(RAFT replicates it cluster-wide).
The single-node --management path is unchanged when no cluster addrs are
supplied, so dev boxes and existing single-node clusters keep working.
--ovn-remote already passes through verbatim, so a comma-separated SB
endpoint list works for compute nodes; documented in usage.
* feat: OVN client multi-endpoint failover for RAFT clusters
Accept a comma-separated OVN NB/SB endpoint list so vpcd can target a
clustered DB. config.ParseEndpoints splits the list (trimming whitespace,
dropping empties); live.go Connect passes one libovsdb WithEndpoint per
endpoint, so the client tries them in order, fails over on disconnect, and
follows RAFT leader redirects. Single-endpoint strings still yield one
endpoint, so single-node dev is unchanged.
ovn-sbctl --db= already accepts the raw comma list, so the SB CLI
consumers (discoverChassis, gateway_claim) need no change.
* feat: write clustered OVN NB/SB endpoints across the RAFT quorum
spx admin init/join now emit comma-separated OVN NB/SB endpoint lists for
the first three quorum nodes (formation.BuildOVNDBAddrs) instead of loopback
or a single leader address, so vpcd fails over across the cluster.
firstboot defers OVN bring-up to the provisioning controller when it owns
formation (SkipFormation): a standalone single-node .db would otherwise block
the controller's clustered create/join.
* test: add multinode OVN RAFT cluster + failover e2e; document clustered topology
ovn_raft_test.go asserts NB/SB cluster/status reports a 3-server quorum with
one leader on every DB node, then stops the NB leader's ovn-central and proves
an NB write still lands via a surviving endpoint.
Docs: vpc-networking gains the multi-endpoint vpcd config and a RAFT
cluster-status debugging section; network-connections lists the 6643/6644 RAFT
ports.
* feat(gpu): add NVIDIA MIG multi-instance GPU support
Partitions MIG-capable GPUs (A100, A30, H100) into isolated slices exposed
as mdev devices instead of whole-GPU VFIO passthrough. VMs attach via
-device vfio-pci,sysfsdev=<mdev-path> using the GPUAttachment discriminated
union (PCIAddress for whole-GPU, MdevPath for MIG).
Key changes:
- gpu: new MIGProfile/MIGInstance types, IsMIGCapable model map, mig.go ops
(EnableMIGMode, DisableMIGMode, CreateInstances, ListInstances,
DestroyAllInstances, enrichMdevPaths)
- gpu.Manager: Claim() returns (*GPUDevice, *MIGInstance, error); AddMIGInstances,
ReclaimByMdev for restart recovery; MarkMIGFailed per-slice
- daemon: buildGPUPool helper partitions devices into whole-GPU vs MIG pools;
MIGProfile config field selects profile per node or per GPU override
- vm.VM: GPUAttachments []gpu.GPUAttachment replaces GPUPCIAddresses []string
- admin CLI: spx admin gpu mig status/enable/disable subcommands with
admin.SetMIGProfile for spinifex.toml writes
- docs: COMMANDS.md updated with three new MIG management command rows
* test(gpu): unit tests for MIG parsers and pool manager
Cover all pure parsing functions (parseMIGMemory, isCapacityExhaustedError,
extractProfileFields/GIFields, parseMIGProfiles, parseMIGInstances,
parseCreatedGIID/CIID) with realistic nvidia-smi fixture output.
Manager MIG tests cover AddMIGInstances, Claim/Release on mdev-backed entries,
MarkMIGFailed, ReclaimByMdev, and interaction with whole-GPU MarkFailed — all
using fake mdev directories in t.TempDir() so no GPU hardware is required.
* feat(gpu): dynamic MIG profile-per-request with on-demand GI/CI lifecycle
Replaces static config-driven MIG pre-carving with a dynamic model where
the MIG profile is specified per RunInstances call via mig.<profile>
instance type names (e.g. mig.1g.10gb) that mirror nvidia-smi nomenclature
directly.
- Manager.Claim(instanceID, profileName): fast path claims a pre-carved
slice; slow path dequeues a free MIG GPU, releases the lock, calls
nvidia-smi mig -cgi/-cci to carve all slices, then reclaims the lock to
commit and return the first slice
- Manager.Release: last release on a dynamically-carved GPU calls
DestroyAllInstances and returns the GPU to freeMIGGPUs, allowing a
different profile on the next claim
- AddMIGGPU(dev): registers a fresh MIG GPU for dynamic allocation;
AddMIGInstances (restart recovery) keeps its no-destroy semantics
- GenerateMIGTypes: produces mig.<profile> InstanceTypeInfo entries from
gpu.ListProfiles at daemon startup; IsMIGType/MIGProfileFromType let
callers inspect these names
- buildGPUPool: calls ListInstances for recovery, AddMIGGPU for fresh
GPUs, ListProfiles to collect MIG profile specs for type generation;
removes setupMIGInstances and migProfileForDevice
- GPUClaimer.Claim gains profileName param; daemonGPUClaimer, service_impl
call sites, and fakeGPUClaimer all updated
* fix(gpu): correct mdev parent detection and make --profile optional for mig enable
enrichMdevPaths was using os.Readlink on "parent_bus_0" (a non-existent
sysfs attribute) and then on ".." (a directory, not a symlink) — both
would fail on real NVIDIA hardware, breaking CreateInstances and
ListInstances on daemon restart recovery. Fix: read the mdev UUID's own
symlink target, which always encodes the parent GPU's PCI address in its
path (e.g. .../0000:01:00.0/<uuid>).
Make basePath injectable in enrichMdevPaths so the fix is covered by a
new set of unit tests using temp-dir symlinks (TestEnrichMdevPaths_*).
Remove the MarkFlagRequired("profile") from "spx admin gpu mig enable":
Phase 2 is fully dynamic — the daemon carves slices on demand at
RunInstances time. Without --profile, the command just enables MIG mode
and destroys any pre-carved slices so the daemon starts with a free GPU.
With --profile, behaviour is unchanged (pre-carve a static pool).
* chore: writeups for smci and arcee
* test: add coverage for MIG exec, manager slow-path, instancetypes and admin
- Add mig_exec_test.go with fake nvidia-smi harness covering Enable/Disable MIGMode,
IsMIGModeEnabled, DestroyAllInstances, ListProfiles, CreateInstances, ListInstances
- Extend manager_mig_test.go with slow-path Claim/Release tests using fake nvidia-smi
- Add generate_test.go coverage for IsMIGType, MIGProfileFromType, GenerateMIGTypes
- Add admin_test.go coverage for SetMIGProfile (create, update, idempotent, error paths)
- Make mdevBasePath a var to allow sysfs injection in tests
- Add json tags to GPUAttachment fields to satisfy musttag linter
Diff coverage: 75.6% (threshold: 60%).
* fixed gpu passthrough doc
* Revert "fixed gpu passthrough doc"
This reverts commit 1e20f5ab13.
* feat: Added RTX 6000 GPU MIG support and fixed docs
* feat: expose GPU/MIG inventory in GetNodes and GetVMs admin API
Add Manager.Snapshot() for lock-safe pool inspection. Extend
NodeStatusResponse with GPUs []GPUInfo (per-physical-GPU inventory
with per-slice detail for MIG) and VMInfo with GPU *VMGPUInfo
(model, VRAM, profile, mdev path for the attached GPU). Populate
both in handleNodeStatus and handleNodeVMs via buildGPUInventory,
buildPoolLookup, and resolveVMGPU helpers.
* feat: surface GPU info on EC2 instance detail page (mulga-new-03)
Admin users see a GPU card (model, VRAM, attachment type, MIG profile/mdev
path or PCI address) populated from the GetVMs admin API. Adds formatVRAMMiB
helper and VMGPUInfo/GPUInfo/GPUSliceInfo types to the frontend.
* feat: GPU columns + inventory card on Nodes page (mulga-new-04)
Adds GPU column to VMsTable (model/VRAM or MIG profile) and ResourceTable
(alloc/total GPUs). Section 4 GPU Inventory card shows per-node GPU state with
per-slice MIG drill-down. Also adds total_gpus/alloc_gpus to NodeInfo type.
* feat: grouped GPU-aware instance type picker in Run Instances (mulga-new-05)
Splits the flat instance type dropdown into Standard and GPU-Accelerated groups.
GPU items show total VRAM alongside the type name. Selecting a GPU type surfaces
an info line with GPU name and VRAM.
* testing image builder scripts
* fixed nvidia build script
* more descriptive errors for image build script
* more fixes
* more fixes
* fix: surface PCI address in whole-GPU VMGPUInfo and rebuild frontend dist
Add PCIAddress to VMGPUInfo, populate it in resolveVMGPU for whole-GPU
passthrough, and fix the instance detail page which was incorrectly
showing instance type in the PCI address field. Rebuilt frontend dist
to include GPU UI changes (inventory card, resource columns, GPU-aware
instance type picker).
* fix: improve kernel version detection in nvidia image build script
Fall back to /lib/modules when /boot/vmlinuz-* is absent, and add
diagnostic output to help debug kernel detection failures.
* fixing nvidia image build script
* fix ami name issue
* correct kernel versions
* another fix
* image build works, added docker and nvidia toolkit for QoL
* fix: add gpg
* fixing amd image builder
* add universe
* another fix
* nvm
* fix docker user group issues
* make rocm land where it is supposed to
* fix docker groups
* feat: add Ubuntu 26.04 GPU AMI catalog entries and fix build script parted detection
Add ubuntu-26.04-nvidia-gpu-x86_64 and ubuntu-26.04-amd-gpu-x86_64 to AvailableImages,
hosted on iso.mulgadc.com/system-ami/ with SHA256 checksums. Fix build-system-image.sh
to detect parted in /usr/sbin when not on PATH. Suppress gosec false positives in DHCP
manager (cancel stored in leaseLoop) and promisc ioctl (standard syscall uintptr pattern).
* Updated passthrough doc regarding GPU images
* fix: remove orphaned nolint directives (golangci-lint v2.12.2 no longer flags G115/G118)
* chore: remove stale spinifex@192.168.3.107 directory
Scripts were accidentally committed with a node hostname as the path
prefix during initial testing. Correct copies live in scripts/images/.
* feat: add GPU fields to NodeStatus and spx top nodes (mulga-996)
- Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes)
- Populate from gpuManager in handleNodeStatus
- Show GPU (used/total) column in spx top nodes; "-" when no GPUs present
* Fixed GPU passthrough and added test script
* feat: always-probe GPU at startup, surface capability in NodeStatusResponse
- Add gpuProbeResult struct and probeGPU() which runs at daemon startup
regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU
groups, and vfio_pci module presence.
- Store gpuProbe on Daemon struct; split NewDaemon into probe (always)
and activate (opt-in) phases.
- Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse
so operators can see hardware state before enabling passthrough.
- Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but
passthrough not active, 'used/total' = passthrough active.
* feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001)
- Export GenerateGPUTypes for use by the reload path
- Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place
(preserves references held by instanceService) and updates NATS
subscriptions without full daemon restart
- Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP
in a context-aware loop; setupShutdown no longer intercepts SIGHUP
- Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal,
handles false→true (re-probe + activate) and true→false (refuse if
instances running, otherwise deactivate) transitions
* installer grub image fixes
* made gpu-passthrough more user facing
* vpcd fixes to set br-wan to promisc for public IP DHCP processes
* fixed vpcd br-wan promisc and veth pair
* ssh keygen fix
* Fix: AMD GPU edge cases
* script for running 8x instances on SMCI computer
* fix: increase disk space on instances
* feat: multi-GPU passthrough support for g7e.12xlarge (2× MI350X)
- vm.VM: GPUPCIAddress string → GPUPCIAddresses []string for multi-GPU
- instancetypes: add g7e.12xlarge (2 GPUs), GPUCountForType(), AMDmi350x model
- instancetypes/generate: use GPUCountForType for GpuInfo.Count and TotalGpuMemoryInMiB
- gpu/manager: Release() loops all pool entries for instanceID (not just first)
- vm/lifecycle: loop over GPUPCIAddresses, emit gpu0/gpu1/... QEMU devices
- daemon/handlers: claim N GPUs in loop for both RunInstances and StartStoppedInstance
- daemon/vm_adapters: ReclaimByAddress loops GPUPCIAddresses; ReleaseGPU checks len > 0
- scripts/launch-gpu-fleet: fixed fleet — 6× g7e.4xlarge + 1× g7e.12xlarge
* feat: per-instance-type disk size in GPU fleet (300 GB single, 600 GB dual)
* fix: multi-gpu instance types
* fix: kill QEMU synchronously when QMP handshake fails to free VFIO device
When startQEMU succeeds but newQMPClientWithHandshake times out, the
instance has not yet been inserted into the manager map so vmMgr.Delete
in the rollback path is a no-op. QEMU keeps running and holds the VFIO
group open. The GPU pool entry gets freed, the next Claim returns the
same PCI address, and every subsequent QEMU fails with 'device or
resource busy'.
Fix: SIGKILL QEMU and wait for it to exit before returning the QMP
error, ensuring the VFIO device is released before the caller recycles
the GPU pool slot.
* fix: quarantine GPU after QEMU launch failure to prevent re-assignment cascade
When QEMU crashes during startup (e.g. pci_irq_handler assertion on a
specific GPU), the pool Release frees that address and the next NATS
request immediately re-claims the same broken device. Add MarkFailed()
to gpu.Manager which sets Available=false for a PCI address after a
launch failure, preventing re-assignment until the daemon restarts.
Call MarkFailed for each GPU address in both RunInstances and
StartStoppedInstance rollback paths after the Release succeeds.
* fixed instance fleet launch script
* remove known hosts and stagger provisioning
* improvements to launch script
* more upgrades
* single instance gpu instance provisioning
* fixed single instance provision
* smci computer demo dashboard
* fixes to launch script
* updated smci dashboard_server
* fixed demo dashboard
* updated dashboard.py
* feat: GPU Passthrough fixes to passthrough process - builds proper AMD or NVIDA ready images, generalised launch script
* feat: gpu-passthrough updating image builder to include gpu-ready ubuntu images
* fixing image builder process
* fixed image builder to remove /tmp/ file before import to free space
* fix: Updated COMMANDS.md to reflect new GPU management admin commands
* Update docs, update internal launch script to work with new SGs, test tweaks
* Updated Docs, fixed internal launch-gpu.sh script, test tweaks
* fxing CI race due to GPU load
Josh Siviour