3 Commits

Author	SHA1	Message	Date
Josh Siviour	1e1d2f3c6d	ci: GPU e2e workflow on ci-gpu self-hosted runner (#518) ... * ci: GPU e2e workflow on ci-gpu self-hosted runner Runs tests/e2e/gpu (EC2/ECS/EKS GPU passthrough) against the persistent, hand-bootstrapped wattle shared host, driven over SSH by a banksia runner labelled ci-gpu (matches the ci-single/ci-multi convention) — no runner installed on the shared box. - runs-on [self-hosted, ci-gpu]; concurrency ci-gpu, cancel-in-progress false (single A1000 = single-flight, queue not stomp) - workflow_dispatch + GPU-path filter (not every push) - manual-work mutex gate (~/.ci-gpu-paused / CI_GPU_PAUSED) so hands-on GPU testing blocks CI - if: always() cleanup terminates launched GPU instances; never resets the box (no setup-ovn.sh/dev-reset/cluster-reset) - drives the control plane with spx + the already-configured aws spinifex profile directly (no wrapper scripts, nothing to install on wattle) * ci(gpu-e2e): advisory AMI preflight, fail only if zero GPU AMIs The suite is self-guarding — each test skips when its AMI is absent. The hard-fail on the base ubuntu GPU AMI blocked the whole run even when the ECS GPU AMI (the only one imported) could execute. Inventory the AMIs and fail only when none are present. * ci(gpu-e2e): self-provision base GPU AMI, hard-require it in preflight The GPU E2E workflow was green-because-skipped: with no GPU AMIs imported the self-guarding suite skipped every test, passing as a false green. Make the raw-EC2 GPU passthrough test the always-on baseline. - Add an import-if-absent setup step that pulls the prebuilt base GPU image from the catalog by name (ubuntu-26.04-nvidia-gpu-x86_64), naming the AMI exactly what tests/e2e/gpu filters for. Idempotent; caches on wattle's persistent loop-fs. Runs spx under sg spinifex (local storage write). - Preflight now hard-requires the base AMI (fail, not skip) so a silent import failure can't pass as green; ECS/EKS node AMIs stay opt-in warn. - Reconcile the local build path: ubuntu-gpu-nvidia.conf AMI_NAME and launch-gpu.sh's nvidia default now use the same canonical name, so a locally built+imported image also satisfies the suite. * ci(gpu-e2e): pass --ami-name on base GPU import so the AMI name matches spx admin images import --name selects the catalog entry but names the resulting AMI generically (ami-<distro>-<version>-<arch>), so the import landed as ami-ubuntu-26.04-x86_64 and the tests/e2e/gpu name filter never matched. Pass --ami-name ubuntu-26.04-nvidia-gpu-x86_64 explicitly (same flag build-system-image.sh uses) and poll briefly for availability. * fix(network,vm): reconcile auto-assigned public-IP NAT + scale VFIO QMP timeout Auto-assigned public IPs were never carried through the network reconciler. applyEIPs ran AddEIP + ensureGuestPortDatapath for user EIPs only; auto-assigned IPs (intent.Ports[].PublicIP) got egress exemption but no NAT re-assert and no gatewayLRP->guest convergence. Their dnat_and_snat was created once at RunInstances and never touched, so a vpcd/OVN rebuild dropped it and the guest-port hop never converged: inbound DNATs at the gateway but never reaches the guest (TCP/SSH dark) while ICMP is answered by the gateway LR (ping OK). floatingIPSpecs now folds every auto-assigned/ELB public IP into the same AddEIP + guest-port convergence path as user EIPs (deduped by private IP; a user EIP wins), making the datapath self-healing across reconcile ticks and deploys. qmpGreetingTimeout for VFIO guests now scales with guest RAM (base 30s + 8s/GiB, floored at 180s, capped at 600s) instead of a flat 180s, so a large guest is not SIGKILLed mid-pin while the whole guest RAM is IOMMU-locked before QMP answers. * fix(vm): scale QMP handshake command deadline for VFIO launches The greeting timeout scaling alone did not fix the VFIO launch SIGKILL: a GPU guest sends its QMP greeting before the RAM pin completes, then cannot service qmp_capabilities until the pin finishes. sendQMPCommand hardcoded a 30s response deadline, so the handshake decode timed out (~43s for a 16 GiB g5.xlarge) and the launch cleanup SIGKILLed a QEMU that would have come up. Parameterize the command deadline (sendQMPCommandWithTimeout); the handshake now uses the RAM-scaled qmpGreetingTimeout (base 30s + 8s/GiB, floor 180s, cap 600s) for qmp_capabilities. Other callers keep the 30s default. * test(gpu-e2e): share one GPU instance across passthrough subtests to avoid mid-suite IP recycle	2026-07-13 21:28:48 +10:00
Julian Sommer	8a94886b13	feat: cloud-init EC2 datasource (#411)	2026-06-25 13:48:08 +10:00
tomnewton-mulga	43fa6eae59	feat: Updated GPU-Passthrough - AMD support, GPU images, /demo folder (#217) ... * feat: add GPU fields to NodeStatus and spx top nodes (mulga-996) - Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes) - Populate from gpuManager in handleNodeStatus - Show GPU (used/total) column in spx top nodes; "-" when no GPUs present * Fixed GPU passthrough and added test script * feat: always-probe GPU at startup, surface capability in NodeStatusResponse - Add gpuProbeResult struct and probeGPU() which runs at daemon startup regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU groups, and vfio_pci module presence. - Store gpuProbe on Daemon struct; split NewDaemon into probe (always) and activate (opt-in) phases. - Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse so operators can see hardware state before enabling passthrough. - Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but passthrough not active, 'used/total' = passthrough active. * feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001) - Export GenerateGPUTypes for use by the reload path - Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place (preserves references held by instanceService) and updates NATS subscriptions without full daemon restart - Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP in a context-aware loop; setupShutdown no longer intercepts SIGHUP - Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal, handles false→true (re-probe + activate) and true→false (refuse if instances running, otherwise deactivate) transitions * installer grub image fixes * made gpu-passthrough more user facing * vpcd fixes to set br-wan to promisc for public IP DHCP processes * fixed vpcd br-wan promisc and veth pair * ssh keygen fix * Fix: AMD GPU edge cases * script for running 8x instances on SMCI computer * fix: increase disk space on instances * feat: multi-GPU passthrough support for g7e.12xlarge (2× MI350X) - vm.VM: GPUPCIAddress string → GPUPCIAddresses []string for multi-GPU - instancetypes: add g7e.12xlarge (2 GPUs), GPUCountForType(), AMDmi350x model - instancetypes/generate: use GPUCountForType for GpuInfo.Count and TotalGpuMemoryInMiB - gpu/manager: Release() loops all pool entries for instanceID (not just first) - vm/lifecycle: loop over GPUPCIAddresses, emit gpu0/gpu1/... QEMU devices - daemon/handlers: claim N GPUs in loop for both RunInstances and StartStoppedInstance - daemon/vm_adapters: ReclaimByAddress loops GPUPCIAddresses; ReleaseGPU checks len > 0 - scripts/launch-gpu-fleet: fixed fleet — 6× g7e.4xlarge + 1× g7e.12xlarge * feat: per-instance-type disk size in GPU fleet (300 GB single, 600 GB dual) * fix: multi-gpu instance types * fix: kill QEMU synchronously when QMP handshake fails to free VFIO device When startQEMU succeeds but newQMPClientWithHandshake times out, the instance has not yet been inserted into the manager map so vmMgr.Delete in the rollback path is a no-op. QEMU keeps running and holds the VFIO group open. The GPU pool entry gets freed, the next Claim returns the same PCI address, and every subsequent QEMU fails with 'device or resource busy'. Fix: SIGKILL QEMU and wait for it to exit before returning the QMP error, ensuring the VFIO device is released before the caller recycles the GPU pool slot. * fix: quarantine GPU after QEMU launch failure to prevent re-assignment cascade When QEMU crashes during startup (e.g. pci_irq_handler assertion on a specific GPU), the pool Release frees that address and the next NATS request immediately re-claims the same broken device. Add MarkFailed() to gpu.Manager which sets Available=false for a PCI address after a launch failure, preventing re-assignment until the daemon restarts. Call MarkFailed for each GPU address in both RunInstances and StartStoppedInstance rollback paths after the Release succeeds. * fixed instance fleet launch script * remove known hosts and stagger provisioning * improvements to launch script * more upgrades * single instance gpu instance provisioning * fixed single instance provision * smci computer demo dashboard * fixes to launch script * updated smci dashboard_server * fixed demo dashboard * updated dashboard.py * feat: GPU Passthrough fixes to passthrough process - builds proper AMD or NVIDA ready images, generalised launch script * feat: gpu-passthrough updating image builder to include gpu-ready ubuntu images * fixing image builder process * fixed image builder to remove /tmp/ file before import to free space * fix: Updated COMMANDS.md to reflect new GPU management admin commands * Update docs, update internal launch script to work with new SGs, test tweaks * Updated Docs, fixed internal launch-gpu.sh script, test tweaks * fxing CI race due to GPU load	2026-05-13 11:07:13 +10:00