mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
feat: Updated GPU-Passthrough - AMD support, GPU images, /demo folder (#217)
* feat: add GPU fields to NodeStatus and spx top nodes (mulga-996) - Add TotalGPUs/AllocGPUs to NodeStatusResponse (zero on non-GPU nodes) - Populate from gpuManager in handleNodeStatus - Show GPU (used/total) column in spx top nodes; "-" when no GPUs present * Fixed GPU passthrough and added test script * feat: always-probe GPU at startup, surface capability in NodeStatusResponse - Add gpuProbeResult struct and probeGPU() which runs at daemon startup regardless of gpu_passthrough config. Checks sysfs for GPUs, IOMMU groups, and vfio_pci module presence. - Store gpuProbe on Daemon struct; split NewDaemon into probe (always) and activate (opt-in) phases. - Populate GPUCapable, GPUPassthrough, GPUModels on NodeStatusResponse so operators can see hardware state before enabling passthrough. - Update 'spx top' GPU column: '-' = no hardware, '0/N*' = capable but passthrough not active, 'used/total' = passthrough active. * feat: SIGHUP hot-reload for GPU passthrough config (mulga-1001) - Export GenerateGPUTypes for use by the reload path - Add ResourceManager.reloadGPUTypes: mutates instanceTypes map in-place (preserves references held by instanceService) and updates NATS subscriptions without full daemon restart - Separate SIGHUP from SIGTERM/SIGINT: setupReload() listens for SIGHUP in a context-aware loop; setupShutdown no longer intercepts SIGHUP - Add reloadConfig() + applyGPUConfig(): re-reads spinifex.toml on signal, handles false→true (re-probe + activate) and true→false (refuse if instances running, otherwise deactivate) transitions * installer grub image fixes * made gpu-passthrough more user facing * vpcd fixes to set br-wan to promisc for public IP DHCP processes * fixed vpcd br-wan promisc and veth pair * ssh keygen fix * Fix: AMD GPU edge cases * script for running 8x instances on SMCI computer * fix: increase disk space on instances * feat: multi-GPU passthrough support for g7e.12xlarge (2× MI350X) - vm.VM: GPUPCIAddress string → GPUPCIAddresses []string for multi-GPU - instancetypes: add g7e.12xlarge (2 GPUs), GPUCountForType(), AMDmi350x model - instancetypes/generate: use GPUCountForType for GpuInfo.Count and TotalGpuMemoryInMiB - gpu/manager: Release() loops all pool entries for instanceID (not just first) - vm/lifecycle: loop over GPUPCIAddresses, emit gpu0/gpu1/... QEMU devices - daemon/handlers: claim N GPUs in loop for both RunInstances and StartStoppedInstance - daemon/vm_adapters: ReclaimByAddress loops GPUPCIAddresses; ReleaseGPU checks len > 0 - scripts/launch-gpu-fleet: fixed fleet — 6× g7e.4xlarge + 1× g7e.12xlarge * feat: per-instance-type disk size in GPU fleet (300 GB single, 600 GB dual) * fix: multi-gpu instance types * fix: kill QEMU synchronously when QMP handshake fails to free VFIO device When startQEMU succeeds but newQMPClientWithHandshake times out, the instance has not yet been inserted into the manager map so vmMgr.Delete in the rollback path is a no-op. QEMU keeps running and holds the VFIO group open. The GPU pool entry gets freed, the next Claim returns the same PCI address, and every subsequent QEMU fails with 'device or resource busy'. Fix: SIGKILL QEMU and wait for it to exit before returning the QMP error, ensuring the VFIO device is released before the caller recycles the GPU pool slot. * fix: quarantine GPU after QEMU launch failure to prevent re-assignment cascade When QEMU crashes during startup (e.g. pci_irq_handler assertion on a specific GPU), the pool Release frees that address and the next NATS request immediately re-claims the same broken device. Add MarkFailed() to gpu.Manager which sets Available=false for a PCI address after a launch failure, preventing re-assignment until the daemon restarts. Call MarkFailed for each GPU address in both RunInstances and StartStoppedInstance rollback paths after the Release succeeds. * fixed instance fleet launch script * remove known hosts and stagger provisioning * improvements to launch script * more upgrades * single instance gpu instance provisioning * fixed single instance provision * smci computer demo dashboard * fixes to launch script * updated smci dashboard_server * fixed demo dashboard * updated dashboard.py * feat: GPU Passthrough fixes to passthrough process - builds proper AMD or NVIDA ready images, generalised launch script * feat: gpu-passthrough updating image builder to include gpu-ready ubuntu images * fixing image builder process * fixed image builder to remove /tmp/ file before import to free space * fix: Updated COMMANDS.md to reflect new GPU management admin commands * Update docs, update internal launch script to work with new SGs, test tweaks * Updated Docs, fixed internal launch-gpu.sh script, test tweaks * fxing CI race due to GPU load
This commit is contained in:
tomnewton-mulga
@@ -172,6 +172,7 @@ func copyRootfs() error {
|
||||
"--exclude=/etc/openvswitch/",
|
||||
"--exclude=/var/lib/openvswitch/",
|
||||
"--exclude=/var/lib/dhcpcd/",
|
||||
"--exclude=/etc/ssh/ssh_host_*",
|
||||
"--exclude=/lost+found",
|
||||
"--exclude=/boot/efi",
|
||||
"/", mountRoot+"/",
|
||||
@@ -555,6 +556,11 @@ func installBootloader(disk string) error {
|
||||
}
|
||||
return biosErr
|
||||
}
|
||||
// Copy splash image and unicode font from the ISO (mounted at /cdrom) so the
|
||||
// installed GRUB shows the same branded background as the installer GRUB.
|
||||
// The font must be at /boot/grub/fonts/unicode.pf2 so update-grub finds it
|
||||
// there and emits the same loadfont path as the ISO's grub.cfg — GRUB 2.12
|
||||
// (trixie) needs the font in the boot partition, not just /usr/share/grub/.
|
||||
copySplashImage(mountRoot)
|
||||
copyGrubFont(mountRoot)
|
||||
|
||||
@@ -713,9 +719,46 @@ func partitionPaths(disk string) (efi, root string) {
|
||||
return disk + "2", disk + "3"
|
||||
}
|
||||
|
||||
// copySplashImage copies the GRUB splash (embedded in the squashfs at build time by
|
||||
// inject-bins.sh) into the installed system so the post-install GRUB shows the same
|
||||
// branded background as the installer GRUB. Non-fatal — missing source is logged and skipped.
|
||||
// copyGrubFont copies the unicode.pf2 GRUB font into the installed system's
|
||||
// /boot/grub/fonts/ directory. This ensures update-grub finds the font at
|
||||
// /boot/grub/fonts/unicode.pf2 — the same path the ISO's grub.cfg uses —
|
||||
// so the generated grub.cfg enables gfxterm and the background image.
|
||||
// Without this, grub-mkconfig falls back to /usr/share/grub/unicode.pf2,
|
||||
// a path that GRUB 2.12 (trixie) may fail to resolve at boot time.
|
||||
// Non-fatal — a missing source is logged and skipped.
|
||||
func copyGrubFont(root string) {
|
||||
candidates := []string{
|
||||
"/cdrom/boot/grub/fonts/unicode.pf2", // ISO tree (preferred)
|
||||
"/usr/share/grub/unicode.pf2", // live system grub-common fallback
|
||||
}
|
||||
for _, src := range candidates {
|
||||
in, err := os.Open(src)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
defer in.Close()
|
||||
dstDir := filepath.Join(root, "boot/grub/fonts")
|
||||
if err := os.MkdirAll(dstDir, 0o755); err != nil {
|
||||
slog.Warn("copyGrubFont: cannot create fonts dir", "err", err)
|
||||
return
|
||||
}
|
||||
out, err := os.OpenFile(filepath.Join(dstDir, "unicode.pf2"), os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644)
|
||||
if err != nil {
|
||||
slog.Warn("copyGrubFont: cannot open destination", "err", err)
|
||||
return
|
||||
}
|
||||
defer out.Close()
|
||||
if _, err := io.Copy(out, in); err != nil {
|
||||
slog.Warn("copyGrubFont: copy failed", "err", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
slog.Warn("copyGrubFont: no unicode.pf2 found, splash may not display")
|
||||
}
|
||||
|
||||
// copySplashImage copies the GRUB splash from the live ISO (/cdrom/boot/grub/splash.png)
|
||||
// into the installed system so the post-install GRUB shows the same branded background
|
||||
// as the installer. Non-fatal — a missing or unreadable source is logged and skipped.
|
||||
func copySplashImage(root string) {
|
||||
const src = "/usr/share/spinifex/grub-splash.png"
|
||||
in, err := os.Open(src)
|
||||
@@ -741,34 +784,6 @@ func copySplashImage(root string) {
|
||||
}
|
||||
}
|
||||
|
||||
// copyGrubFont copies the unicode font into the installed system's
|
||||
// /boot/grub/fonts/ so the loadfont path in 05_spinifex resolves at boot.
|
||||
// grub-install does not copy fonts; we mirror what build-iso.sh does.
|
||||
func copyGrubFont(root string) {
|
||||
const src = "/usr/share/grub/unicode.pf2"
|
||||
in, err := os.Open(src)
|
||||
if err != nil {
|
||||
slog.Warn("copyGrubFont: font not found, graphical GRUB may not work", "path", src)
|
||||
return
|
||||
}
|
||||
defer in.Close()
|
||||
|
||||
dstDir := filepath.Join(root, "boot/grub/fonts")
|
||||
if err := os.MkdirAll(dstDir, 0o755); err != nil {
|
||||
slog.Warn("copyGrubFont: cannot create fonts dir", "err", err)
|
||||
return
|
||||
}
|
||||
out, err := os.OpenFile(filepath.Join(dstDir, "unicode.pf2"), os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644)
|
||||
if err != nil {
|
||||
slog.Warn("copyGrubFont: cannot open destination", "err", err)
|
||||
return
|
||||
}
|
||||
defer out.Close()
|
||||
if _, err := io.Copy(out, in); err != nil {
|
||||
slog.Warn("copyGrubFont: copy failed", "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// maskSystemdUnit creates a symlink to /dev/null for the given unit, which is
|
||||
// the standard way to permanently disable a unit so systemd will never start it.
|
||||
func maskSystemdUnit(root, unit string) error {
|
||||
|
||||
@@ -1113,6 +1113,20 @@ func runAdminInit(cmd *cobra.Command, args []string) {
|
||||
|
||||
finalizeNodeSetup(spxRoot, certPath, bootstrapResult.AdminAccessKey, bootstrapResult.AdminSecretKey, region, bindIP, advertiseIP)
|
||||
|
||||
// Write node.conf so spx admin banner works on source installs (not just ISO).
|
||||
nodeHostname, _ := os.Hostname()
|
||||
if nodeHostname == "" {
|
||||
nodeHostname = node
|
||||
}
|
||||
nodeConfPath := filepath.Join(configDir, "node.conf")
|
||||
if err := writeNodeConf(nodeConfPath, map[string]string{
|
||||
"MANAGEMENT_IP": advertiseIP,
|
||||
"MANAGEMENT_IFACE": "br-wan",
|
||||
"NODE_HOSTNAME": nodeHostname,
|
||||
}); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "⚠️ Warning: could not write %s: %v\n", nodeConfPath, err)
|
||||
}
|
||||
|
||||
// Print success message
|
||||
fmt.Println("\n🎉 Spinifex initialization complete!")
|
||||
fmt.Println()
|
||||
@@ -2361,6 +2375,12 @@ func runAdminBanner(cmd *cobra.Command, _ []string) {
|
||||
}
|
||||
|
||||
// Resolve current IP from the management interface at runtime.
|
||||
// If node.conf is absent or has no MANAGEMENT_IFACE (source installs before
|
||||
// the first spx admin init writes node.conf), fall back to br-wan which
|
||||
// setup-ovn.sh always creates as the management bridge.
|
||||
if iface == "" {
|
||||
iface = "br-wan"
|
||||
}
|
||||
currentIP := resolveIfaceIP(iface)
|
||||
if currentIP == "" {
|
||||
currentIP = recordedIP // fall back to value recorded at install time
|
||||
|
||||
@@ -120,18 +120,16 @@ func runAdminGpuStatus(cmd *cobra.Command, _ []string) {
|
||||
fmt.Printf("Passthrough: enabled\n")
|
||||
fmt.Printf("GPU pool: %d/%d allocated\n", resp.AllocGPUs, resp.TotalGPUs)
|
||||
|
||||
var g5Types []string
|
||||
var gpuTypes []string
|
||||
for _, cap := range resp.InstanceTypes {
|
||||
if strings.HasPrefix(cap.Name, "g5.") {
|
||||
g5Types = append(g5Types, cap.Name)
|
||||
}
|
||||
gpuTypes = append(gpuTypes, cap.Name)
|
||||
}
|
||||
if len(g5Types) > 0 {
|
||||
fmt.Printf("Instance types: %s\n", strings.Join(g5Types, " "))
|
||||
if len(gpuTypes) > 0 {
|
||||
fmt.Printf("Instance types: %s\n", strings.Join(gpuTypes, " "))
|
||||
}
|
||||
} else if resp.GPUCapable {
|
||||
fmt.Printf("Passthrough: disabled\n")
|
||||
fmt.Printf("Instance types: (none — run 'spx admin gpu enable' to activate g5.*)\n")
|
||||
fmt.Printf("Instance types: (none — run 'spx admin gpu enable' to activate)\n")
|
||||
} else {
|
||||
fmt.Printf("Passthrough: disabled\n")
|
||||
fmt.Printf("Instance types: (prerequisites not met)\n")
|
||||
@@ -209,6 +207,9 @@ func gpuToggle(_ *cobra.Command, enable bool) {
|
||||
}
|
||||
if r.GPUPassthrough == enable {
|
||||
fmt.Println(" done.")
|
||||
// Refresh the MOTD banner immediately so the login prompt
|
||||
// reflects the new GPU passthrough state without requiring a reboot.
|
||||
_ = exec.Command("spx", "admin", "banner", "--boot-check").Run()
|
||||
runAdminGpuStatus(adminGpuStatusCmd, nil)
|
||||
return
|
||||
}
|
||||
@@ -306,6 +307,30 @@ func runAdminGpuSetup(_ *cobra.Command, _ []string) {
|
||||
fmt.Println(" Already blacklisted")
|
||||
}
|
||||
|
||||
fmt.Println("==> Configuring amdgpu blacklist")
|
||||
const amdgpuConf = "/etc/modprobe.d/blacklist-amdgpu.conf"
|
||||
hasAMD := false
|
||||
for _, d := range devices {
|
||||
if d.Vendor == gpu.VendorAMD {
|
||||
hasAMD = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if hasAMD {
|
||||
if _, err := os.Stat(amdgpuConf); os.IsNotExist(err) {
|
||||
if err := os.WriteFile(amdgpuConf, []byte("blacklist amdgpu\n"), 0o644); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Failed to write amdgpu blacklist: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
fmt.Println(" amdgpu blacklisted")
|
||||
rebootNeeded = true
|
||||
} else {
|
||||
fmt.Println(" Already blacklisted")
|
||||
}
|
||||
} else {
|
||||
fmt.Println(" No AMD GPUs — skipping")
|
||||
}
|
||||
|
||||
fmt.Println("==> Configuring vfio-pci early binding")
|
||||
const vfioPCIConf = "/etc/modprobe.d/vfio-pci.conf"
|
||||
existing, _ := os.ReadFile(vfioPCIConf)
|
||||
@@ -350,7 +375,7 @@ func runAdminGpuSetup(_ *cobra.Command, _ []string) {
|
||||
}
|
||||
fmt.Println("\nSetup complete — reboot required.")
|
||||
fmt.Println(" sudo reboot")
|
||||
fmt.Println(" Then re-run: sudo spx admin gpu setup")
|
||||
fmt.Println(" Then run: sudo spx admin gpu enable")
|
||||
return
|
||||
}
|
||||
|
||||
@@ -363,15 +388,29 @@ func runAdminGpuSetup(_ *cobra.Command, _ []string) {
|
||||
|
||||
fmt.Println("==> Verifying GPU driver binding")
|
||||
for _, d := range devices {
|
||||
driverLink := "/sys/bus/pci/devices/" + d.PCIAddress + "/driver"
|
||||
target, err := os.Readlink(driverLink)
|
||||
driver := filepath.Base(target)
|
||||
if err != nil || driver != "vfio-pci" {
|
||||
fmt.Fprintf(os.Stderr, "%s is bound to %q, not vfio-pci\n", d.PCIAddress, driver)
|
||||
fmt.Fprintf(os.Stderr, " Check: ls -la %s\n", driverLink)
|
||||
current, err := gpu.CurrentDriver(d.PCIAddress)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Cannot read driver for %s: %v\n", d.PCIAddress, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
fmt.Printf(" %s - vfio-pci\n", d.PCIAddress)
|
||||
if current == "vfio-pci" {
|
||||
fmt.Printf(" %s → vfio-pci\n", d.PCIAddress)
|
||||
continue
|
||||
}
|
||||
if current != "" {
|
||||
// Bound to native driver — blacklist/initramfs changes may not have taken effect.
|
||||
fmt.Fprintf(os.Stderr, "%s is bound to %q, not vfio-pci — reboot may be required\n", d.PCIAddress, current)
|
||||
fmt.Fprintf(os.Stderr, " Check: ls -la /sys/bus/pci/devices/%s/driver\n", d.PCIAddress)
|
||||
os.Exit(1)
|
||||
}
|
||||
// Device is unbound (ids= did not auto-claim it) — bind explicitly via driver_override.
|
||||
fmt.Printf(" %s unbound — binding to vfio-pci\n", d.PCIAddress)
|
||||
if _, err := gpu.BindVFIO(d.PCIAddress); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Failed to bind %s to vfio-pci: %v\n", d.PCIAddress, err)
|
||||
fmt.Fprintf(os.Stderr, " Check: ls -la /sys/bus/pci/devices/%s/driver\n", d.PCIAddress)
|
||||
os.Exit(1)
|
||||
}
|
||||
fmt.Printf(" %s → vfio-pci\n", d.PCIAddress)
|
||||
}
|
||||
|
||||
fmt.Println("==> Enabling GPU passthrough")
|
||||
|
||||
+10
-1
@@ -45,7 +45,7 @@ Operational commands for inspecting cluster state. These fan out NATS requests t
|
||||
|
||||
| Command | Flags | Prerequisites | Basic Logic | Output | Status |
|
||||
|---------|-------|---------------|-------------|--------|--------|
|
||||
| `spx top nodes` | `--timeout` (default: 3s) | Cluster must be running (NATS) | Publishes to `spinifex.node.status` fan-out topic → collects CPU/memory usage per node → aggregates instance type capacity across all nodes → prints two tables: per-node resource usage and cluster-wide instance type availability | Table 1: NAME, CPU (used/total), MEM (used/total), VMs. Table 2: INSTANCE TYPE, AVAILABLE, VCPU, MEMORY | **DONE** |
|
||||
| `spx top nodes` | `--timeout` (default: 3s) | Cluster must be running (NATS) | Publishes to `spinifex.node.status` fan-out topic → collects CPU/memory usage per node → aggregates instance type capacity across all nodes → prints two tables: per-node resource usage and cluster-wide instance type availability | Table 1: NAME, CPU (used/total), MEM (used/total), GPU (used/total), VMs — GPU column shows `allocated/total` when passthrough is enabled, `0/count*` when the node is GPU-capable but passthrough is disabled, or `-` when no GPU is present. Table 2: INSTANCE TYPE, AVAILABLE, VCPU, MEMORY | **DONE** |
|
||||
|
||||
### Cluster Initialization
|
||||
|
||||
@@ -123,6 +123,15 @@ served. A mirror compromise that swaps both image and sums file is not
|
||||
detected; closing that gap requires GPG signature verification of the sums
|
||||
file, deferred to a later phase.
|
||||
|
||||
### GPU Management
|
||||
|
||||
| Command | Flags | Prerequisites | Basic Logic | Test Cases | Status |
|
||||
|---------|-------|---------------|-------------|------------|--------|
|
||||
| `spx admin gpu status` | `--node` (default: local node) | Cluster must be running (NATS) | Queries `spinifex.node.status` fan-out → finds the target node response → prints Node, GPU hardware (model list or "none detected"), IOMMU state, vfio-pci state, passthrough enabled/disabled, and GPU pool allocation (`allocated/total`). Also lists GPU-capable instance types when passthrough is active. | 1. Local node with passthrough enabled (shows pool and instance types)<br>2. Local node capable but passthrough disabled (shows "run enable")<br>3. Node without GPU hardware (shows "none detected")<br>4. `--node` targets a remote node<br>5. Node not found or not responding (error) | **DONE** |
|
||||
| `spx admin gpu enable` | — | Cluster must be running (NATS); node must be GPU-capable (IOMMU + vfio-pci configured) | Checks current passthrough state via NATS → errors if already enabled or prerequisites not met (directs to `setup`) → writes `gpu_passthrough = true` to `spinifex.toml` via `admin.SetGPUPassthrough` → sends SIGHUP to `spinifex-daemon` via `systemctl kill -s HUP` → polls node status for up to 30 s until daemon confirms new state → prints final `gpu status` output. Must be run directly on the target host. | 1. Enable on capable node<br>2. Already enabled (no-op)<br>3. Prerequisites not met (error, suggests `setup`)<br>4. Daemon does not confirm within 30 s (timeout error) | **DONE** |
|
||||
| `spx admin gpu disable` | — | Cluster must be running (NATS); no GPU instances may be running | Checks current passthrough state via NATS → errors if already disabled or if `AllocGPUs > 0` (must terminate GPU instances first) → writes `gpu_passthrough = false` → sends SIGHUP to `spinifex-daemon` → polls for up to 30 s → prints final `gpu status` output. Must be run directly on the target host. | 1. Disable with no GPU instances running<br>2. Already disabled (no-op)<br>3. GPU instances running (error with count)<br>4. Daemon does not confirm within 30 s (timeout error) | **DONE** |
|
||||
| `spx admin gpu setup` | — | Must be run as root; NVIDIA or AMD GPU must be present | Idempotent host configuration for GPU passthrough. Steps (skipped if already applied): detect GPUs via `gpu.Discover` → collect PCI IDs for all IOMMU-group siblings → check/enable IOMMU in GRUB (`intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt`) → write vfio udev rule (`/etc/udev/rules.d/99-spinifex-vfio.rules`) → blacklist nouveau (`/etc/modprobe.d/blacklist-nouveau.conf`) → blacklist amdgpu if AMD GPU present (`/etc/modprobe.d/blacklist-amdgpu.conf`) → write vfio-pci early binding config (`/etc/modprobe.d/vfio-pci.conf`) → add vfio modules to initramfs. If any change requires a reboot: runs `update-initramfs -u` and exits with reboot instructions. After reboot: verifies `vfio_pci` module is loaded → verifies each GPU is bound to `vfio-pci` (binds explicitly via `driver_override` if unbound) → calls `gpu enable` to activate passthrough. | 1. First run on unconfigured host (GRUB updated, reboot required)<br>2. Re-run after reboot (verifies bindings, enables passthrough)<br>3. Already fully configured (all steps skipped, enables passthrough)<br>4. AMD GPU present (blacklists amdgpu in addition to nouveau)<br>5. Not run as root (error)<br>6. No GPU hardware detected (error) | **DONE** |
|
||||
|
||||
## AWS Commands
|
||||
|
||||
### EC2 - Instance Management
|
||||
|
||||
@@ -0,0 +1,223 @@
|
||||
---
|
||||
title: "GPU Passthrough"
|
||||
description: "Configure VFIO GPU passthrough on a Spinifex node to expose NVIDIA or AMD GPUs to EC2 instances."
|
||||
category: "Compute"
|
||||
tags:
|
||||
- gpu
|
||||
- passthrough
|
||||
- vfio
|
||||
- ec2
|
||||
resources:
|
||||
- title: "Spinifex Repository"
|
||||
url: "https://github.com/mulgadc/spinifex"
|
||||
- title: "AWS EC2 Accelerated Computing Instance Types"
|
||||
url: "https://docs.aws.amazon.com/ec2/latest/instancetypes/ac.html"
|
||||
---
|
||||
|
||||
# GPU Passthrough
|
||||
|
||||
> Configure VFIO GPU passthrough on a Spinifex node to expose NVIDIA or AMD GPUs to EC2 instances.
|
||||
|
||||
> **Note:** GPU passthrough is supported on x86_64 hosts only.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Host Setup](#host-setup)
|
||||
- [Enable Passthrough](#enable-passthrough)
|
||||
- [Status and Monitoring](#status-and-monitoring)
|
||||
- [GPU Instance Types](#gpu-instance-types)
|
||||
- [Launching GPU Instances](#launching-gpu-instances)
|
||||
- [Disable Passthrough](#disable-passthrough)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Spinifex supports VFIO-based GPU passthrough, binding NVIDIA and AMD GPUs directly to guest VMs via the `vfio-pci` kernel driver. Once configured, the node exposes GPU-enabled EC2 instance types and the GPU is allocated exclusively to individual instances.
|
||||
|
||||
At startup, Spinifex always probes for GPU hardware and surfaces the result in the node banner and in `spx top nodes` — regardless of whether passthrough is enabled. This lets operators verify hardware before activating the feature.
|
||||
|
||||
The passthrough state is reflected in the login banner:
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- IOMMU must be supported and enabled in the host BIOS/UEFI (`Intel VT-d` or `AMD-Vi`).
|
||||
- An NVIDIA or AMD GPU must be physically installed.
|
||||
- The Spinifex node must be running on bare metal (not inside a VM).
|
||||
|
||||
---
|
||||
|
||||
## Host Setup
|
||||
|
||||
Run the one-time host configuration as root. This command is idempotent — safe to re-run after a reboot.
|
||||
|
||||
```bash
|
||||
sudo spx admin gpu setup
|
||||
```
|
||||
|
||||
What it does:
|
||||
|
||||
1. Detects installed GPUs and their PCI IDs.
|
||||
2. Enables IOMMU in GRUB (`intel_iommu=on iommu=pt` or `amd_iommu=on iommu=pt`).
|
||||
3. Writes the vfio udev rule (`/etc/udev/rules.d/99-spinifex-vfio.rules`).
|
||||
4. Blacklists `nouveau` (NVIDIA) and `amdgpu` (AMD) so the host kernel does not claim the device.
|
||||
5. Configures `vfio-pci` early binding in `/etc/modprobe.d/vfio-pci.conf`.
|
||||
6. Adds vfio modules to initramfs.
|
||||
|
||||
If any change requires a reboot, the command prints instructions and exits:
|
||||
|
||||
```
|
||||
Setup complete — reboot required.
|
||||
sudo reboot
|
||||
Then run: sudo spx admin gpu enable
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Enable Passthrough
|
||||
|
||||
After setup and reboot, enable GPU passthrough:
|
||||
|
||||
```bash
|
||||
sudo spx admin gpu enable
|
||||
```
|
||||
|
||||
The command writes the configuration, signals the daemon (`SIGHUP`), and waits up to 30 seconds for the daemon to confirm the new state. On success it prints the current GPU status.
|
||||
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Status and Monitoring
|
||||
|
||||
### Per-node status
|
||||
|
||||
```bash
|
||||
spx admin gpu status
|
||||
# or for a specific node in the cluster:
|
||||
spx admin gpu status --node <node-name>
|
||||
```
|
||||
|
||||
Output includes hardware detected, IOMMU state, vfio-pci binding, passthrough enabled/disabled, GPU pool allocation (`allocated/total`), and the GPU-capable EC2 instance types available on that node.
|
||||
|
||||
### Cluster view
|
||||
|
||||
`spx top nodes` includes a `GPU (used/total)` column:
|
||||
|
||||
| Value | Meaning |
|
||||
|-------|---------|
|
||||
| `1/2` | Passthrough active — 1 of 2 GPUs allocated |
|
||||
| `0/1*` | Node has a GPU but passthrough is not enabled |
|
||||
| `-` | No GPU hardware detected |
|
||||
|
||||
```bash
|
||||
spx top nodes
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## GPU Instance Types
|
||||
|
||||
GPU instance types are derived from detected hardware.
|
||||
|
||||
To list available GPU instance types on the current node:
|
||||
|
||||
```bash
|
||||
aws ec2 describe-instance-types \
|
||||
--filters Name=instance-type,Values=g*
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Launching GPU Instances
|
||||
|
||||
Import the GPU-enabled AMI before launching:
|
||||
|
||||
```bash
|
||||
spx admin images import --name ubuntu-gpu-nvidia # NVIDIA hosts
|
||||
spx admin images import --name ubuntu-gpu-amd # AMD hosts
|
||||
```
|
||||
|
||||
Launch a GPU instance:
|
||||
|
||||
```bash
|
||||
aws ec2 run-instances \
|
||||
--image-id $GPU_AMI \
|
||||
--instance-type $INSTANCE_TYPE \
|
||||
--key-name spinifex-key
|
||||
```
|
||||
|
||||
To verify the GPU is visible from inside the instance, SSH in and run:
|
||||
|
||||
```bash
|
||||
# NVIDIA
|
||||
nvidia-smi
|
||||
|
||||
# AMD
|
||||
rocm-smi
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Disable Passthrough
|
||||
|
||||
Passthrough cannot be disabled while GPU instances are running. Terminate all GPU instances first, then:
|
||||
|
||||
```bash
|
||||
sudo spx admin gpu disable
|
||||
```
|
||||
|
||||
The command signals the daemon and waits for confirmation. The GPU is released back to the host kernel on the next reboot (the vfio-pci binding persists until `setup` is re-run without the blacklists).
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### IOMMU not active after reboot
|
||||
|
||||
Verify IOMMU is enabled in BIOS/UEFI (`Intel VT-d` or `AMD-Vi`). Then check GRUB picked up the parameters:
|
||||
|
||||
```bash
|
||||
cat /proc/cmdline | grep iommu
|
||||
ls /sys/kernel/iommu_groups/
|
||||
```
|
||||
|
||||
If `/sys/kernel/iommu_groups/` is empty, IOMMU is not active. Re-run `sudo spx admin gpu setup` after enabling it in firmware.
|
||||
|
||||
### GPU bound to native driver instead of vfio-pci
|
||||
|
||||
```bash
|
||||
lspci -k | grep -A3 -i "vga\|3d\|display"
|
||||
```
|
||||
|
||||
If the driver is `amdgpu` or `nvidia` rather than `vfio-pci`, the blacklist or early binding did not take effect. Re-run setup and reboot:
|
||||
|
||||
```bash
|
||||
sudo spx admin gpu setup
|
||||
sudo reboot
|
||||
```
|
||||
|
||||
### `spx admin gpu enable` fails with "prerequisites not met"
|
||||
|
||||
Run `setup` first to configure the host, then retry `enable`:
|
||||
|
||||
```bash
|
||||
sudo spx admin gpu setup
|
||||
sudo spx admin gpu enable
|
||||
```
|
||||
|
||||
### Daemon does not confirm within 30 seconds
|
||||
|
||||
Check the daemon log for errors:
|
||||
|
||||
```bash
|
||||
journalctl -u spinifex-daemon -n 50
|
||||
```
|
||||
@@ -1,47 +0,0 @@
|
||||
# Configuring GPU Passthrough for Spinifex Nodes
|
||||
|
||||
|
||||
Spinifex is designed for bare-metal hardware, edge nodes and data-centre use. Follow this guide to configure your Spinifex node to enable GPU Passthrough to its VM instances if hardware is available.
|
||||
|
||||
>**Note:** this tutorial is for x86 architecture.
|
||||
|
||||
|
||||
|
||||
### Getting Started - GPU Setup
|
||||
By default, Spinifex will probe for the presence of a GPU on its machine. After booting, the presence and availability of GPU hardware is printed in the info banner:
|
||||
|
||||
|
||||
To enable GPU passthrough, you may first need to run:
|
||||
```bash
|
||||
sudo spx admin gpu setup
|
||||
```
|
||||
Note that this command will blacklist the GPU's drivers from the host machine such that they can be used by VM instances Therefore the GPU will **NOT** be available to the host machine until after the VM instance It will also require rebooting the host machine before the changes take effect.
|
||||
|
||||
### Enable Passthrough
|
||||
After reboot, the banner message should reflect that GPU passthrough is **ready** but not **enabled**.
|
||||
|
||||
|
||||
|
||||
This will require you to run:
|
||||
```bash
|
||||
sudo spx admin gpu enable
|
||||
```
|
||||
After running this command, you should see:
|
||||
|
||||
|
||||
|
||||
The host machine is now configured to provide GPU passthrough to EC2 instances on the host machine.
|
||||
|
||||
Lastly, import the Spinifex GPU enabled AMI: [link]
|
||||
|
||||
As a final check, spin up an EC2 instance, SSH into it, and check that the host machine's GPU is visible from the EC2 instance.
|
||||
|
||||
### Multiple GPUs
|
||||
Spinifex is able to detect multiple GPUs. After running the setup commands, the total number of allocatable GPUs is printed.
|
||||
|
||||
### Mapping GPUs to EC2 instances
|
||||
By defualt, any consumer grade GPUs are mapped to the g5 EC2 instance class, which is the most basic "Accelerated Computing" Ec2 instance provided by AWS. If using a consumer GPU, running:
|
||||
```bash
|
||||
aws ec2 describe-instance-types
|
||||
```
|
||||
will return the various sizes of g5 class. All other AWS "Accelerated Computing" EC2 instance classes map directly to their associated GPU, so the Spinifex host machine will need to have the specified GPU if a specific EC2 instance class is desired.
|
||||
@@ -82,7 +82,7 @@ case "$DISTRO" in
|
||||
exit 1
|
||||
fi
|
||||
SOURCE_IMAGE="ubuntu-${UBUNTU_VERSION}-minimal-cloudimg-amd64.img"
|
||||
SOURCE_URL="https://cloud-images.ubuntu.com/minimal/releases/noble/release/${SOURCE_IMAGE}"
|
||||
SOURCE_URL="https://cloud-images.ubuntu.com/minimal/releases/${UBUNTU_CODENAME:-noble}/release/${SOURCE_IMAGE}"
|
||||
OUTPUT_IMAGE="${BUILD_DIR}/${IMAGE_NAME}-ubuntu.qcow2"
|
||||
OUTPUT_RAW="${BUILD_DIR}/${IMAGE_NAME}-ubuntu.raw"
|
||||
DISTRO_VERSION="${UBUNTU_VERSION}"
|
||||
@@ -188,11 +188,12 @@ if [[ -f "$OUTPUT_RAW" ]] && [[ $(( $(date +%s) - $(stat -c %Y "$OUTPUT_RAW") ))
|
||||
|
||||
if [[ "$DO_IMPORT" == true ]]; then
|
||||
echo "Importing as AMI..."
|
||||
rm -f "$OUTPUT_IMAGE"
|
||||
IMPORT_ARGS=(--file "$OUTPUT_RAW" --distro "${DISTRO}" --version "${DISTRO_VERSION}" --arch x86_64)
|
||||
if [[ -n "${SYSTEM_TAG:-}" ]]; then
|
||||
IMPORT_ARGS+=(--tag "$SYSTEM_TAG")
|
||||
fi
|
||||
(cd "$PROJECT_DIR" && ./bin/spx admin images import "${IMPORT_ARGS[@]}")
|
||||
spx admin images import "${IMPORT_ARGS[@]}"
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
@@ -253,9 +254,13 @@ if [[ "$DISTRO" == "ubuntu" ]]; then
|
||||
done
|
||||
echo "Resizing partition..."
|
||||
# After qemu-img resize the GPT backup header sits at the old end of disk.
|
||||
# Move it to the new end before parted tries to resize, otherwise parted
|
||||
# refuses with "Unable to satisfy all constraints on the partition."
|
||||
sudo sgdisk --move-second-header "${NBD_DEV}" 2>/dev/null || true
|
||||
# Relocate it to the new end (sfdisk --relocate is util-linux built-in,
|
||||
# no extra packages needed), then re-read the partition table before parted
|
||||
# runs — without this, parted sees stale GPT geometry and fails with
|
||||
# "Unable to satisfy all constraints on the partition."
|
||||
sudo sgdisk --move-second-header "${NBD_DEV}"
|
||||
sudo partprobe "${NBD_DEV}" 2>/dev/null || true
|
||||
sleep 1
|
||||
sudo parted --script "${NBD_DEV}" resizepart 1 100%
|
||||
fi
|
||||
|
||||
@@ -417,14 +422,15 @@ echo ""
|
||||
|
||||
if [[ "$DO_IMPORT" == true ]]; then
|
||||
echo "Importing as AMI..."
|
||||
rm -f "$OUTPUT_IMAGE"
|
||||
IMPORT_ARGS=(--file "$OUTPUT_RAW" --distro "${DISTRO}" --version "${DISTRO_VERSION}" --arch x86_64)
|
||||
if [[ -n "${SYSTEM_TAG:-}" ]]; then
|
||||
IMPORT_ARGS+=(--tag "$SYSTEM_TAG")
|
||||
fi
|
||||
(cd "$PROJECT_DIR" && sudo -u spinifex-storage ./bin/spx admin images import "${IMPORT_ARGS[@]}")
|
||||
spx admin images import "${IMPORT_ARGS[@]}"
|
||||
else
|
||||
echo "To import as AMI, run:"
|
||||
echo " cd $PROJECT_DIR && ./bin/spx admin images import \\"
|
||||
echo " spx admin images import \\"
|
||||
echo " --file $OUTPUT_RAW \\"
|
||||
if [[ -n "${SYSTEM_TAG:-}" ]]; then
|
||||
echo " --distro ${DISTRO} --version ${DISTRO_VERSION} --arch x86_64 \\"
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Dashboard server for the Spinifex GPU demo.
|
||||
|
||||
Proxies streams from the GPU instances and serves the web UI.
|
||||
|
||||
Environment variables:
|
||||
YOLO_HOST Base URL of the YOLO instance, e.g. http://10.0.0.3:8080
|
||||
OLLAMA_HOST Base URL of the Ollama instance, e.g. http://10.0.0.2:11434
|
||||
OLLAMA_MODEL Model name to use for chat (default: llama3.1:70b)
|
||||
PORT Server port (default: 8000)
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
|
||||
import httpx
|
||||
import uvicorn
|
||||
from fastapi import FastAPI, Request
|
||||
from fastapi.responses import HTMLResponse, StreamingResponse
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
|
||||
YOLO_HOST = os.environ.get("YOLO_HOST", "http://localhost:8080")
|
||||
OLLAMA_HOST = os.environ.get("OLLAMA_HOST", "http://localhost:11434")
|
||||
OLLAMA_MODEL = os.environ.get("OLLAMA_MODEL", "llama3.1:70b")
|
||||
PORT = int(os.environ.get("PORT", "8000"))
|
||||
|
||||
app = FastAPI()
|
||||
app.mount("/static", StaticFiles(directory="static"), name="static")
|
||||
|
||||
|
||||
@app.get("/", response_class=HTMLResponse)
|
||||
async def index():
|
||||
with open("static/index.html") as f:
|
||||
return f.read()
|
||||
|
||||
|
||||
@app.get("/proxy/video")
|
||||
async def proxy_video():
|
||||
async def stream():
|
||||
while True:
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=None) as client:
|
||||
async with client.stream("GET", f"{YOLO_HOST}/video") as resp:
|
||||
async for chunk in resp.aiter_bytes(8192):
|
||||
yield chunk
|
||||
except Exception:
|
||||
await asyncio.sleep(5)
|
||||
|
||||
return StreamingResponse(stream(), media_type="multipart/x-mixed-replace; boundary=frame")
|
||||
|
||||
|
||||
@app.get("/proxy/descriptions")
|
||||
async def proxy_descriptions():
|
||||
async def stream():
|
||||
while True:
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(connect=10.0, read=None, write=None, pool=None)) as client:
|
||||
async with client.stream("GET", f"{YOLO_HOST}/descriptions") as resp:
|
||||
async for line in resp.aiter_lines():
|
||||
yield line + "\n"
|
||||
except Exception:
|
||||
yield 'data: {"type":"error","text":"YOLO stream offline, retrying..."}\n\n'
|
||||
await asyncio.sleep(5)
|
||||
|
||||
return StreamingResponse(stream(), media_type="text/event-stream")
|
||||
|
||||
|
||||
@app.post("/chat")
|
||||
async def chat(request: Request):
|
||||
body = await request.json()
|
||||
prompt = body.get("prompt", "").strip()
|
||||
if not prompt:
|
||||
return {"error": "empty prompt"}
|
||||
|
||||
async def stream():
|
||||
async with httpx.AsyncClient(timeout=120) as client:
|
||||
async with client.stream(
|
||||
"POST",
|
||||
f"{OLLAMA_HOST}/v1/chat/completions",
|
||||
json={
|
||||
"model": OLLAMA_MODEL,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"stream": True,
|
||||
},
|
||||
) as resp:
|
||||
async for line in resp.aiter_lines():
|
||||
if not line.startswith("data: "):
|
||||
continue
|
||||
data_str = line[6:]
|
||||
if data_str.strip() == "[DONE]":
|
||||
yield f"data: {json.dumps({'token': '', 'done': True})}\n\n"
|
||||
break
|
||||
try:
|
||||
data = json.loads(data_str)
|
||||
token = data["choices"][0]["delta"].get("content", "")
|
||||
yield f"data: {json.dumps({'token': token, 'done': False})}\n\n"
|
||||
except (json.JSONDecodeError, KeyError, IndexError):
|
||||
pass
|
||||
|
||||
return StreamingResponse(stream(), media_type="text/event-stream")
|
||||
|
||||
|
||||
@app.get("/status")
|
||||
async def status():
|
||||
results: dict[str, str] = {}
|
||||
async with httpx.AsyncClient(timeout=3) as client:
|
||||
for name, url in [
|
||||
("yolo", f"{YOLO_HOST}/health"),
|
||||
("ollama", f"{OLLAMA_HOST}/v1/models"),
|
||||
]:
|
||||
try:
|
||||
r = await client.get(url)
|
||||
results[name] = "ok" if r.status_code < 500 else "error"
|
||||
except Exception:
|
||||
results[name] = "offline"
|
||||
return results
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
uvicorn.run(app, host="0.0.0.0", port=PORT)
|
||||
Executable
+116
@@ -0,0 +1,116 @@
|
||||
#!/bin/bash
|
||||
# deploy.sh — Deploy and start the Spinifex GPU demo across four instances.
|
||||
#
|
||||
# Usage:
|
||||
# DASHBOARD_IP=<ip> YOLO_IP=<ip> OLLAMA_IP=<ip> QWEN_IP=<ip> \
|
||||
# VIDEO_PATH=/path/to/video.mp4 \
|
||||
# scripts/demo/deploy.sh
|
||||
#
|
||||
# Required env vars:
|
||||
# DASHBOARD_IP 4x-1: serves the web dashboard
|
||||
# YOLO_IP 4x-3: runs YOLO inference
|
||||
# OLLAMA_IP 4x-2: runs Ollama + llama3.1:70b
|
||||
# QWEN_IP 12x: runs Ollama + qwen3-vl:235b
|
||||
# VIDEO_PATH Local path to the .mp4 file to run YOLO on
|
||||
#
|
||||
# Optional:
|
||||
# SSH_KEY (default: ~/.ssh/spinifex-key)
|
||||
# SSH_USER (default: ec2-user)
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
DASHBOARD_IP="${DASHBOARD_IP:?set DASHBOARD_IP}"
|
||||
YOLO_IP="${YOLO_IP:?set YOLO_IP}"
|
||||
OLLAMA_IP="${OLLAMA_IP:?set OLLAMA_IP}"
|
||||
QWEN_IP="${QWEN_IP:?set QWEN_IP}"
|
||||
VIDEO_PATH="${VIDEO_PATH:?set VIDEO_PATH}"
|
||||
|
||||
SSH_KEY="${SSH_KEY:-$HOME/.ssh/spinifex-key}"
|
||||
SSH_USER="${SSH_USER:-ec2-user}"
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
_ssh() {
|
||||
local ip="$1"; shift
|
||||
ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
|
||||
-o ConnectTimeout=10 -o BatchMode=yes \
|
||||
-i "$SSH_KEY" "${SSH_USER}@${ip}" "$@"
|
||||
}
|
||||
|
||||
_scp() {
|
||||
scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
|
||||
-i "$SSH_KEY" -r "$@"
|
||||
}
|
||||
|
||||
# ── YOLO instance (4x-3) ────────────────────────────────────────────────────
|
||||
echo "==> [YOLO] Deploying to $YOLO_IP..."
|
||||
_scp "$SCRIPT_DIR/yolo_stream.py" "$SCRIPT_DIR/requirements-yolo.txt" \
|
||||
"${SSH_USER}@${YOLO_IP}:~/"
|
||||
_scp "$VIDEO_PATH" "${SSH_USER}@${YOLO_IP}:~/demo_video.mp4"
|
||||
|
||||
echo " Installing Python deps..."
|
||||
_ssh "$YOLO_IP" '
|
||||
. ~/yolo-rocm/bin/activate
|
||||
pip install --quiet -r ~/requirements-yolo.txt
|
||||
'
|
||||
|
||||
echo " Starting yolo_stream.py..."
|
||||
_ssh "$YOLO_IP" "
|
||||
pkill -f yolo_stream.py || true
|
||||
sleep 1
|
||||
. ~/yolo-rocm/bin/activate
|
||||
nohup env \
|
||||
VIDEO_PATH=~/demo_video.mp4 \
|
||||
QWEN_HOST=http://${QWEN_IP}:11434 \
|
||||
YOLO_DEVICE=0 \
|
||||
PORT=8080 \
|
||||
python ~/yolo_stream.py > ~/yolo_stream.log 2>&1 &
|
||||
echo \" PID \$!\"
|
||||
"
|
||||
echo " YOLO stream: http://${YOLO_IP}:8080/video"
|
||||
|
||||
# ── Dashboard instance (4x-1) ───────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "==> [Dashboard] Deploying to $DASHBOARD_IP..."
|
||||
_ssh "$DASHBOARD_IP" 'mkdir -p ~/demo/static'
|
||||
_scp "$SCRIPT_DIR/dashboard_server.py" "$SCRIPT_DIR/requirements-dashboard.txt" \
|
||||
"${SSH_USER}@${DASHBOARD_IP}:~/demo/"
|
||||
_scp "$SCRIPT_DIR/static/index.html" \
|
||||
"${SSH_USER}@${DASHBOARD_IP}:~/demo/static/"
|
||||
|
||||
echo " Installing Python deps..."
|
||||
_ssh "$DASHBOARD_IP" '
|
||||
python3 -m venv ~/demo-venv 2>/dev/null || true
|
||||
. ~/demo-venv/bin/activate
|
||||
pip install --quiet -r ~/demo/requirements-dashboard.txt
|
||||
'
|
||||
|
||||
echo " Starting dashboard_server.py..."
|
||||
_ssh "$DASHBOARD_IP" "
|
||||
pkill -f dashboard_server.py || true
|
||||
sleep 1
|
||||
. ~/demo-venv/bin/activate
|
||||
cd ~/demo
|
||||
nohup env \
|
||||
YOLO_HOST=http://${YOLO_IP}:8080 \
|
||||
OLLAMA_HOST=http://${OLLAMA_IP}:11434 \
|
||||
OLLAMA_MODEL=llama3.1:70b \
|
||||
PORT=8000 \
|
||||
python dashboard_server.py > ~/dashboard.log 2>&1 &
|
||||
echo \" PID \$!\"
|
||||
"
|
||||
|
||||
echo ""
|
||||
echo "✅ Demo deployed"
|
||||
echo ""
|
||||
echo " Dashboard: http://${DASHBOARD_IP}:8000"
|
||||
echo " YOLO feed: http://${YOLO_IP}:8080/video"
|
||||
echo " Ollama: http://${OLLAMA_IP}:11434"
|
||||
echo " Qwen3-VL: http://${QWEN_IP}:11434"
|
||||
echo ""
|
||||
echo " Logs:"
|
||||
echo " ssh -i ${SSH_KEY} ${SSH_USER}@${YOLO_IP} tail -f ~/yolo_stream.log"
|
||||
echo " ssh -i ${SSH_KEY} ${SSH_USER}@${DASHBOARD_IP} tail -f ~/dashboard.log"
|
||||
echo ""
|
||||
echo " NOTE: Ollama and qwen3-vl must already be running on their instances."
|
||||
echo " If not: _ssh \$IP 'sudo systemctl start ollama && ollama run <model>'"
|
||||
@@ -0,0 +1,4 @@
|
||||
fastapi
|
||||
uvicorn[standard]
|
||||
httpx
|
||||
python-multipart
|
||||
@@ -0,0 +1,5 @@
|
||||
fastapi
|
||||
uvicorn[standard]
|
||||
httpx
|
||||
opencv-python-headless
|
||||
ultralytics
|
||||
@@ -0,0 +1,498 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Spinifex — GPU Compute Demo</title>
|
||||
<style>
|
||||
* { margin: 0; padding: 0; box-sizing: border-box; }
|
||||
|
||||
:root {
|
||||
--bg: #07070d;
|
||||
--surface: #0e0e18;
|
||||
--border: #1a1a2e;
|
||||
--accent: #6366f1;
|
||||
--accent2: #22d3ee;
|
||||
--text: #d4d4d8;
|
||||
--muted: #52525b;
|
||||
--green: #22c55e;
|
||||
--red: #ef4444;
|
||||
}
|
||||
|
||||
body {
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||
height: 100vh;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
/* ── Header ── */
|
||||
header {
|
||||
background: var(--surface);
|
||||
border-bottom: 1px solid var(--border);
|
||||
padding: 0 24px;
|
||||
height: 52px;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 20px;
|
||||
flex-shrink: 0;
|
||||
z-index: 10;
|
||||
}
|
||||
|
||||
.logo {
|
||||
font-size: 17px;
|
||||
font-weight: 700;
|
||||
letter-spacing: -0.3px;
|
||||
color: #fff;
|
||||
white-space: nowrap;
|
||||
}
|
||||
.logo em { color: var(--accent); font-style: normal; }
|
||||
|
||||
.divider { width: 1px; height: 24px; background: var(--border); }
|
||||
|
||||
.subtitle {
|
||||
font-size: 12px;
|
||||
color: var(--muted);
|
||||
letter-spacing: 0.3px;
|
||||
}
|
||||
|
||||
.instances {
|
||||
margin-left: auto;
|
||||
display: flex;
|
||||
gap: 20px;
|
||||
}
|
||||
|
||||
.inst {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
font-size: 11px;
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
.inst-dot {
|
||||
width: 7px; height: 7px;
|
||||
border-radius: 50%;
|
||||
background: var(--green);
|
||||
}
|
||||
.inst-dot.offline { background: var(--red); }
|
||||
.inst-dot.live {
|
||||
background: var(--green);
|
||||
box-shadow: 0 0 0 0 var(--green);
|
||||
animation: ring 2s infinite;
|
||||
}
|
||||
|
||||
@keyframes ring {
|
||||
0% { box-shadow: 0 0 0 0 rgba(34,197,94,.6); }
|
||||
70% { box-shadow: 0 0 0 6px rgba(34,197,94,0); }
|
||||
100% { box-shadow: 0 0 0 0 rgba(34,197,94,0); }
|
||||
}
|
||||
|
||||
/* ── Layout ── */
|
||||
main {
|
||||
flex: 1;
|
||||
display: grid;
|
||||
grid-template-columns: 1fr 400px;
|
||||
grid-template-rows: 1fr 1fr;
|
||||
gap: 10px;
|
||||
padding: 10px;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
/* ── Panel ── */
|
||||
.panel {
|
||||
background: var(--surface);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 10px;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
overflow: hidden;
|
||||
min-height: 0;
|
||||
}
|
||||
|
||||
.panel-head {
|
||||
padding: 9px 14px;
|
||||
background: rgba(0,0,0,.3);
|
||||
border-bottom: 1px solid var(--border);
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.panel-head h2 {
|
||||
font-size: 11px;
|
||||
font-weight: 600;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 1px;
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
.chip {
|
||||
margin-left: auto;
|
||||
font-size: 10px;
|
||||
font-weight: 500;
|
||||
padding: 2px 8px;
|
||||
border-radius: 99px;
|
||||
border: 1px solid var(--border);
|
||||
color: var(--muted);
|
||||
white-space: nowrap;
|
||||
}
|
||||
.chip.accent { border-color: rgba(99,102,241,.4); color: #818cf8; }
|
||||
.chip.cyan { border-color: rgba(34,211,238,.3); color: var(--accent2); }
|
||||
|
||||
/* ── Video panel ── */
|
||||
.video-panel { grid-row: 1 / 3; position: relative; }
|
||||
|
||||
.video-wrap {
|
||||
flex: 1;
|
||||
background: #000;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
overflow: hidden;
|
||||
min-height: 0;
|
||||
}
|
||||
|
||||
.video-wrap img {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
object-fit: contain;
|
||||
display: block;
|
||||
}
|
||||
|
||||
.video-footer {
|
||||
padding: 8px 14px;
|
||||
font-size: 10px;
|
||||
color: var(--muted);
|
||||
border-top: 1px solid var(--border);
|
||||
flex-shrink: 0;
|
||||
display: flex;
|
||||
gap: 16px;
|
||||
}
|
||||
|
||||
/* ── Description panel ── */
|
||||
.desc-content {
|
||||
flex: 1;
|
||||
overflow-y: auto;
|
||||
padding: 14px;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 14px;
|
||||
scroll-behavior: smooth;
|
||||
min-height: 0;
|
||||
}
|
||||
|
||||
.desc-entry {
|
||||
padding-bottom: 14px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
}
|
||||
.desc-entry:last-child { border-bottom: none; padding-bottom: 0; }
|
||||
|
||||
.desc-time {
|
||||
font-size: 10px;
|
||||
color: var(--muted);
|
||||
margin-bottom: 5px;
|
||||
font-variant-numeric: tabular-nums;
|
||||
}
|
||||
|
||||
.desc-text {
|
||||
font-size: 13px;
|
||||
line-height: 1.65;
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
.cursor {
|
||||
display: inline-block;
|
||||
width: 2px; height: 13px;
|
||||
background: var(--accent);
|
||||
vertical-align: text-bottom;
|
||||
margin-left: 1px;
|
||||
animation: blink 1s step-end infinite;
|
||||
}
|
||||
@keyframes blink { 0%,100%{opacity:1} 50%{opacity:0} }
|
||||
|
||||
.desc-placeholder {
|
||||
font-size: 12px;
|
||||
color: var(--muted);
|
||||
font-style: italic;
|
||||
}
|
||||
|
||||
/* ── Chat panel ── */
|
||||
.chat-messages {
|
||||
flex: 1;
|
||||
overflow-y: auto;
|
||||
padding: 14px;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 10px;
|
||||
min-height: 0;
|
||||
}
|
||||
|
||||
.msg {
|
||||
max-width: 88%;
|
||||
padding: 9px 13px;
|
||||
border-radius: 10px;
|
||||
font-size: 13px;
|
||||
line-height: 1.55;
|
||||
word-break: break-word;
|
||||
}
|
||||
.msg.user {
|
||||
background: #1e1e42;
|
||||
align-self: flex-end;
|
||||
color: #c7d2fe;
|
||||
}
|
||||
.msg.assistant {
|
||||
background: #111120;
|
||||
align-self: flex-start;
|
||||
color: var(--text);
|
||||
border: 1px solid var(--border);
|
||||
}
|
||||
|
||||
.chat-foot {
|
||||
border-top: 1px solid var(--border);
|
||||
padding: 10px;
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.chat-foot input {
|
||||
flex: 1;
|
||||
background: var(--bg);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 7px;
|
||||
padding: 8px 12px;
|
||||
color: var(--text);
|
||||
font-size: 13px;
|
||||
outline: none;
|
||||
transition: border-color .15s;
|
||||
}
|
||||
.chat-foot input:focus { border-color: var(--accent); }
|
||||
.chat-foot input::placeholder { color: var(--muted); }
|
||||
|
||||
.send-btn {
|
||||
background: var(--accent);
|
||||
color: #fff;
|
||||
border: none;
|
||||
border-radius: 7px;
|
||||
padding: 8px 16px;
|
||||
font-size: 13px;
|
||||
font-weight: 600;
|
||||
cursor: pointer;
|
||||
transition: background .15s, opacity .15s;
|
||||
white-space: nowrap;
|
||||
}
|
||||
.send-btn:hover { background: #4f52d1; }
|
||||
.send-btn:disabled { opacity: .4; cursor: not-allowed; }
|
||||
|
||||
/* scrollbar */
|
||||
::-webkit-scrollbar { width: 4px; }
|
||||
::-webkit-scrollbar-track { background: transparent; }
|
||||
::-webkit-scrollbar-thumb { background: var(--border); border-radius: 2px; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
|
||||
<header>
|
||||
<div class="logo">Spini<em>fex</em></div>
|
||||
<div class="divider"></div>
|
||||
<div class="subtitle">GPU Compute Platform — Live Demo</div>
|
||||
<div class="instances">
|
||||
<div class="inst">
|
||||
<div class="inst-dot live" id="dot-yolo"></div>
|
||||
<span>YOLO · g7e.4xlarge</span>
|
||||
</div>
|
||||
<div class="inst">
|
||||
<div class="inst-dot live" id="dot-qwen"></div>
|
||||
<span>Qwen3-VL 235B · g7e.12xlarge</span>
|
||||
</div>
|
||||
<div class="inst">
|
||||
<div class="inst-dot live" id="dot-llama"></div>
|
||||
<span>Llama 3.1 70B · g7e.4xlarge</span>
|
||||
</div>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<main>
|
||||
|
||||
<!-- Left: YOLO video, full height -->
|
||||
<div class="panel video-panel">
|
||||
<div class="panel-head">
|
||||
<h2>Live Object Detection</h2>
|
||||
<span class="chip accent">YOLO11 · AMD MI350X</span>
|
||||
</div>
|
||||
<div class="video-wrap">
|
||||
<img id="video-feed" src="/proxy/video" alt="YOLO stream loading...">
|
||||
</div>
|
||||
<div class="video-footer">
|
||||
<span>Real-time inference — AMD MI350X GPU</span>
|
||||
<span id="fps-counter"></span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Top right: qwen3-vl scene analysis -->
|
||||
<div class="panel">
|
||||
<div class="panel-head">
|
||||
<h2>Scene Analysis</h2>
|
||||
<span class="chip cyan">Qwen3-VL 235B · MI350X</span>
|
||||
</div>
|
||||
<div class="desc-content" id="desc-content">
|
||||
<p class="desc-placeholder">Waiting for first frame analysis…</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Bottom right: Llama3.1 chat -->
|
||||
<div class="panel">
|
||||
<div class="panel-head">
|
||||
<h2>Chat</h2>
|
||||
<span class="chip accent">Llama 3.1 70B · MI350X</span>
|
||||
</div>
|
||||
<div class="chat-messages" id="chat-messages">
|
||||
<div class="msg assistant">
|
||||
Hello! I’m Llama 3.1 70B running on an AMD MI350X GPU, orchestrated by Spinifex. Ask me anything.
|
||||
</div>
|
||||
</div>
|
||||
<div class="chat-foot">
|
||||
<input type="text" id="chat-input" placeholder="Ask Llama 3.1…" autocomplete="off">
|
||||
<button class="send-btn" id="send-btn" onclick="sendChat()">Send</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</main>
|
||||
|
||||
<script>
|
||||
// ── Scene description stream ─────────────────────────────────────────────────
|
||||
const descContent = document.getElementById('desc-content');
|
||||
let activeEntry = null;
|
||||
let activeText = '';
|
||||
|
||||
function initDescriptions() {
|
||||
const es = new EventSource('/proxy/descriptions');
|
||||
|
||||
es.onmessage = (e) => {
|
||||
let data;
|
||||
try { data = JSON.parse(e.data); } catch { return; }
|
||||
|
||||
if (data.type === 'token' && data.text) {
|
||||
if (!activeEntry) {
|
||||
// Remove placeholder if present
|
||||
descContent.querySelector('.desc-placeholder')?.remove();
|
||||
|
||||
activeEntry = document.createElement('div');
|
||||
activeEntry.className = 'desc-entry';
|
||||
activeEntry.innerHTML = `<div class="desc-time">${new Date().toLocaleTimeString()}</div><div class="desc-text"></div>`;
|
||||
descContent.appendChild(activeEntry);
|
||||
activeText = '';
|
||||
}
|
||||
activeText += data.text;
|
||||
const textEl = activeEntry.querySelector('.desc-text');
|
||||
textEl.innerHTML = escHtml(activeText) + '<span class="cursor"></span>';
|
||||
descContent.scrollTop = descContent.scrollHeight;
|
||||
}
|
||||
|
||||
if (data.type === 'end' && activeEntry) {
|
||||
activeEntry.querySelector('.desc-text').textContent = activeText;
|
||||
activeEntry = null;
|
||||
activeText = '';
|
||||
// Keep only the last 4 entries
|
||||
const entries = descContent.querySelectorAll('.desc-entry');
|
||||
entries.forEach((el, i) => { if (i < entries.length - 4) el.remove(); });
|
||||
}
|
||||
};
|
||||
|
||||
es.onerror = () => { setTimeout(initDescriptions, 3000); };
|
||||
}
|
||||
|
||||
// ── Chat ─────────────────────────────────────────────────────────────────────
|
||||
const chatMessages = document.getElementById('chat-messages');
|
||||
const chatInput = document.getElementById('chat-input');
|
||||
const sendBtn = document.getElementById('send-btn');
|
||||
|
||||
async function sendChat() {
|
||||
const prompt = chatInput.value.trim();
|
||||
if (!prompt || sendBtn.disabled) return;
|
||||
|
||||
chatInput.value = '';
|
||||
sendBtn.disabled = true;
|
||||
|
||||
appendMsg('user', prompt);
|
||||
|
||||
const assistantEl = appendMsg('assistant', '');
|
||||
let text = '';
|
||||
|
||||
try {
|
||||
const resp = await fetch('/chat', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ prompt }),
|
||||
});
|
||||
|
||||
const reader = resp.body.getReader();
|
||||
const decoder = new TextDecoder();
|
||||
let buf = '';
|
||||
|
||||
while (true) {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
buf += decoder.decode(value, { stream: true });
|
||||
const lines = buf.split('\n');
|
||||
buf = lines.pop();
|
||||
for (const line of lines) {
|
||||
if (!line.startsWith('data: ')) continue;
|
||||
let d;
|
||||
try { d = JSON.parse(line.slice(6)); } catch { continue; }
|
||||
text += d.token;
|
||||
assistantEl.innerHTML = escHtml(text) + (d.done ? '' : '<span class="cursor"></span>');
|
||||
chatMessages.scrollTop = chatMessages.scrollHeight;
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
assistantEl.textContent = '(connection error)';
|
||||
}
|
||||
|
||||
sendBtn.disabled = false;
|
||||
chatInput.focus();
|
||||
}
|
||||
|
||||
function appendMsg(role, text) {
|
||||
const el = document.createElement('div');
|
||||
el.className = `msg ${role}`;
|
||||
el.textContent = text;
|
||||
chatMessages.appendChild(el);
|
||||
chatMessages.scrollTop = chatMessages.scrollHeight;
|
||||
return el;
|
||||
}
|
||||
|
||||
chatInput.addEventListener('keydown', (e) => { if (e.key === 'Enter') sendChat(); });
|
||||
|
||||
// ── Status polling ────────────────────────────────────────────────────────────
|
||||
async function pollStatus() {
|
||||
try {
|
||||
const data = await fetch('/status').then(r => r.json());
|
||||
setDot('dot-yolo', data.yolo === 'ok');
|
||||
setDot('dot-llama', data.ollama === 'ok');
|
||||
} catch {}
|
||||
setTimeout(pollStatus, 5000);
|
||||
}
|
||||
|
||||
function setDot(id, ok) {
|
||||
const el = document.getElementById(id);
|
||||
if (!el) return;
|
||||
el.className = 'inst-dot' + (ok ? ' live' : ' offline');
|
||||
}
|
||||
|
||||
// ── Helpers ───────────────────────────────────────────────────────────────────
|
||||
function escHtml(str) {
|
||||
return str.replace(/&/g,'&').replace(/</g,'<').replace(/>/g,'>');
|
||||
}
|
||||
|
||||
// ── Boot ──────────────────────────────────────────────────────────────────────
|
||||
initDescriptions();
|
||||
pollStatus();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,159 @@
|
||||
#!/usr/bin/env python3
|
||||
"""YOLO inference server for the Spinifex demo.
|
||||
|
||||
Reads a video file, runs YOLO detection, and serves:
|
||||
GET /video — MJPEG stream of annotated frames
|
||||
GET /descriptions — SSE stream of qwen3-vl scene descriptions
|
||||
GET /health — {"ok": true}
|
||||
|
||||
Environment variables:
|
||||
VIDEO_PATH Path to input .mp4 (required)
|
||||
QWEN_HOST Base URL of the qwen3-vl Ollama instance, e.g. http://10.0.0.5:11434
|
||||
YOLO_MODEL Model name (default: yolo11n.pt)
|
||||
YOLO_DEVICE GPU device index (default: 0)
|
||||
DESCRIBE_INTERVAL Seconds between qwen3-vl calls (default: 5.0)
|
||||
PORT Server port (default: 8080)
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import queue
|
||||
import threading
|
||||
import time
|
||||
|
||||
import cv2
|
||||
import httpx
|
||||
import uvicorn
|
||||
from fastapi import FastAPI
|
||||
from fastapi.responses import StreamingResponse
|
||||
from ultralytics import YOLO
|
||||
|
||||
VIDEO_PATH = os.environ["VIDEO_PATH"]
|
||||
QWEN_HOST = os.environ.get("QWEN_HOST", "")
|
||||
YOLO_MODEL = os.environ.get("YOLO_MODEL", "yolo11x.pt")
|
||||
YOLO_DEVICE = os.environ.get("YOLO_DEVICE", "0")
|
||||
DESCRIBE_INTERVAL = float(os.environ.get("DESCRIBE_INTERVAL", "5.0"))
|
||||
PORT = int(os.environ.get("PORT", "8080"))
|
||||
|
||||
app = FastAPI()
|
||||
|
||||
_frame_lock = threading.Lock()
|
||||
_latest_frame: bytes | None = None
|
||||
_desc_queue: queue.Queue = queue.Queue(maxsize=100)
|
||||
|
||||
|
||||
def _encode(frame, quality: int = 80) -> bytes:
|
||||
_, buf = cv2.imencode(".jpg", frame, [cv2.IMWRITE_JPEG_QUALITY, quality])
|
||||
return buf.tobytes()
|
||||
|
||||
|
||||
def _describe(img_b64: str) -> None:
|
||||
if not QWEN_HOST:
|
||||
return
|
||||
try:
|
||||
with httpx.Client(timeout=180) as client:
|
||||
with client.stream(
|
||||
"POST",
|
||||
f"{QWEN_HOST}/api/generate",
|
||||
json={
|
||||
"model": "qwen3-vl:235b",
|
||||
"prompt": "Briefly describe what is happening in this scene. Focus on the objects detected and any notable activity.",
|
||||
"images": [img_b64],
|
||||
"stream": True,
|
||||
},
|
||||
) as resp:
|
||||
for line in resp.iter_lines():
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
data = json.loads(line)
|
||||
token = data.get("response", "")
|
||||
if token and not _desc_queue.full():
|
||||
_desc_queue.put(("token", token))
|
||||
if data.get("done"):
|
||||
_desc_queue.put(("end", ""))
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
except Exception as exc:
|
||||
_desc_queue.put(("error", str(exc)))
|
||||
|
||||
|
||||
def inference_loop() -> None:
|
||||
model = YOLO(YOLO_MODEL)
|
||||
cap = cv2.VideoCapture(VIDEO_PATH)
|
||||
if not cap.isOpened():
|
||||
raise RuntimeError(f"Cannot open video: {VIDEO_PATH}")
|
||||
|
||||
fps = cap.get(cv2.CAP_PROP_FPS) or 25.0
|
||||
frame_delay = 1.0 / fps
|
||||
last_describe = 0.0
|
||||
|
||||
while True:
|
||||
ret, frame = cap.read()
|
||||
if not ret:
|
||||
cap.set(cv2.CAP_PROP_POS_FRAMES, 0)
|
||||
continue
|
||||
|
||||
results = model(frame, device=YOLO_DEVICE, verbose=False)
|
||||
annotated = results[0].plot()
|
||||
|
||||
with _frame_lock:
|
||||
global _latest_frame
|
||||
_latest_frame = _encode(annotated)
|
||||
|
||||
now = time.monotonic()
|
||||
if QWEN_HOST and now - last_describe >= DESCRIBE_INTERVAL:
|
||||
last_describe = now
|
||||
img_b64 = base64.b64encode(_encode(annotated, quality=60)).decode()
|
||||
threading.Thread(target=_describe, args=(img_b64,), daemon=True).start()
|
||||
|
||||
time.sleep(frame_delay)
|
||||
|
||||
|
||||
@app.on_event("startup")
|
||||
async def startup() -> None:
|
||||
threading.Thread(target=inference_loop, daemon=True).start()
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
async def health():
|
||||
return {"ok": True}
|
||||
|
||||
|
||||
@app.get("/video")
|
||||
async def video():
|
||||
async def generate():
|
||||
while True:
|
||||
with _frame_lock:
|
||||
frame = _latest_frame
|
||||
if frame:
|
||||
yield b"--frame\r\nContent-Type: image/jpeg\r\n\r\n" + frame + b"\r\n"
|
||||
await asyncio.sleep(0.04)
|
||||
|
||||
return StreamingResponse(
|
||||
generate(),
|
||||
media_type="multipart/x-mixed-replace; boundary=frame",
|
||||
)
|
||||
|
||||
|
||||
@app.get("/descriptions")
|
||||
async def descriptions():
|
||||
async def generate():
|
||||
yield 'data: {"type":"ready"}\n\n'
|
||||
loop = asyncio.get_event_loop()
|
||||
while True:
|
||||
try:
|
||||
kind, text = await loop.run_in_executor(
|
||||
None, lambda: _desc_queue.get(timeout=30)
|
||||
)
|
||||
yield f"data: {json.dumps({'type': kind, 'text': text})}\n\n"
|
||||
except queue.Empty:
|
||||
yield 'data: {"type":"ping"}\n\n'
|
||||
|
||||
return StreamingResponse(generate(), media_type="text/event-stream")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
uvicorn.run(app, host="0.0.0.0", port=PORT)
|
||||
Executable
+62
@@ -0,0 +1,62 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
# ubuntu-gpu-amd-setup.sh — Chroot setup for an AMD GPU-capable guest image.
|
||||
#
|
||||
# Installs linux-firmware (AMD firmware blobs required by the amdgpu kernel
|
||||
# module), ROCm CLI tools, Python toolchain, and common utilities. Rebuilds
|
||||
# initramfs so AMD firmware is available at early boot inside the guest.
|
||||
#
|
||||
# Host-side VFIO setup (driver blacklisting, IOMMU, initramfs) is handled
|
||||
# by `spx admin gpu setup` — this script is for the guest VM image only.
|
||||
#
|
||||
# Must run inside an Ubuntu 26.04 chroot with /proc /sys /dev bind-mounted.
|
||||
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
apt-get update -qq
|
||||
apt-get install -y --no-install-recommends curl gnupg ca-certificates
|
||||
|
||||
# AMD ROCm apt repository — rocminfo and rocm-smi-lib are not in the Ubuntu
|
||||
# default repos, so we add AMD's signed repo before the main package install.
|
||||
UBUNTU_CODENAME=$(. /etc/os-release && echo "${UBUNTU_CODENAME:-${VERSION_CODENAME}}")
|
||||
mkdir -p /etc/apt/keyrings
|
||||
curl -fsSL https://repo.radeon.com/rocm/rocm.gpg.key \
|
||||
| gpg --dearmor -o /etc/apt/keyrings/rocm.gpg
|
||||
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/6.3 ${UBUNTU_CODENAME} main" \
|
||||
> /etc/apt/sources.list.d/rocm.list
|
||||
apt-get update -qq
|
||||
|
||||
# linux-firmware carries the AMD GPU firmware blobs loaded by the amdgpu
|
||||
# kernel module at boot. Without them the GPU is invisible to guest userland.
|
||||
apt-get install -y -o Acquire::Retries=3 --no-install-recommends \
|
||||
linux-image-generic \
|
||||
linux-headers-generic \
|
||||
linux-firmware \
|
||||
initramfs-tools \
|
||||
pciutils \
|
||||
rocminfo rocm-smi-lib \
|
||||
python3 python3-venv python3-pip \
|
||||
git curl wget htop tmux \
|
||||
ffmpeg libgl1 libglib2.0-0
|
||||
|
||||
KVER=$(ls /boot/vmlinuz-* 2>/dev/null | sort -V | tail -1 | sed 's|/boot/vmlinuz-||')
|
||||
if [ -z "${KVER}" ]; then
|
||||
echo "ERROR: No kernel found under /boot"
|
||||
exit 1
|
||||
fi
|
||||
echo "Rebuilding initramfs for kernel: ${KVER}"
|
||||
update-initramfs -u -k "${KVER}"
|
||||
|
||||
# Prevent unattended-upgrades from replacing linux-firmware mid-lifecycle;
|
||||
# a firmware change without a matching initramfs rebuild can make the GPU invisible.
|
||||
mkdir -p /etc/apt/apt.conf.d
|
||||
cat > /etc/apt/apt.conf.d/99-gpu-ami <<'EOF'
|
||||
Unattended-Upgrade::Package-Blacklist {
|
||||
"linux-firmware";
|
||||
"linux-image-";
|
||||
"linux-headers-";
|
||||
};
|
||||
EOF
|
||||
|
||||
echo "AMD GPU image setup complete: kernel=${KVER}, linux-firmware and ROCm userland installed"
|
||||
@@ -0,0 +1,12 @@
|
||||
IMAGE_NAME="ubuntu-amd-gpu"
|
||||
IMAGE_DESCRIPTION="Ubuntu 26.04 AMD GPU guest image — linux-firmware, ROCm CLI, Python toolchain"
|
||||
DISTRO="ubuntu"
|
||||
UBUNTU_VERSION="26.04"
|
||||
UBUNTU_CODENAME="resolute"
|
||||
|
||||
# 12G provides headroom for kernel headers, linux-firmware, and ROCm packages
|
||||
IMAGE_SIZE="12G"
|
||||
|
||||
SETUP_SCRIPT="scripts/images/ubuntu-gpu-amd-setup.sh"
|
||||
|
||||
SYSTEM_TAG="gpu-vendor=amd"
|
||||
Executable
+107
@@ -0,0 +1,107 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
# ubuntu-gpu-nvidia-setup.sh — Chroot setup for an NVIDIA GPU-capable guest image.
|
||||
#
|
||||
# Pre-builds the NVIDIA headless server driver via DKMS against a pinned kernel
|
||||
# so guests can use the GPU immediately after launch. Also installs Python
|
||||
# toolchain and common utilities.
|
||||
#
|
||||
# nouveau is blacklisted inside the guest image so the NVIDIA kernel module can
|
||||
# bind — this is distinct from the host-side blacklist that `spx admin gpu setup`
|
||||
# configures for VFIO passthrough.
|
||||
#
|
||||
# Host-side VFIO setup (IOMMU, vfio-pci binding, host initramfs) is handled
|
||||
# by `spx admin gpu setup` — this script is for the guest VM image only.
|
||||
#
|
||||
# Must run inside an Ubuntu 26.04 chroot with /proc /sys /dev bind-mounted
|
||||
# (required by DKMS during the nvidia package post-install hook).
|
||||
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
apt-get update -qq
|
||||
|
||||
# Detect the cloud image's existing kernel BEFORE installing anything.
|
||||
# Installing linux-image-generic would upgrade to a newer kernel version;
|
||||
# DKMS would then build for that version but the VM boots the original
|
||||
# cloud image kernel, causing a version mismatch at modprobe time.
|
||||
KVER=$(ls /boot/vmlinuz-* 2>/dev/null | sort -V | tail -1 | sed 's|/boot/vmlinuz-||')
|
||||
if [[ -z "${KVER}" ]]; then
|
||||
echo "ERROR: No kernel found under /boot"
|
||||
exit 1
|
||||
fi
|
||||
echo "Target kernel: ${KVER}"
|
||||
|
||||
# Install headers for the exact kernel already present, plus initramfs-tools.
|
||||
apt-get install -y --no-install-recommends \
|
||||
"linux-headers-${KVER}" \
|
||||
initramfs-tools
|
||||
|
||||
# Install the NVIDIA server DKMS driver. Use the distro default version (no
|
||||
# hardcoded suffix) so Ubuntu 26.04's 580 driver is used instead of 550.
|
||||
# nvidia-dkms-server is a meta package that resolves to the current recommended
|
||||
# version; nvidia-utils-server brings nvidia-smi and related tools.
|
||||
apt-get install -y --no-install-recommends \
|
||||
nvidia-dkms-server \
|
||||
nvidia-utils-server
|
||||
|
||||
# Detect the installed NVIDIA DKMS module name + version for explicit build.
|
||||
NVIDIA_DKMS_NAME=$(dkms status 2>/dev/null | awk -F'[,/ ]+' '/nvidia/{print $1; exit}')
|
||||
NVIDIA_DKMS_VER=$(dkms status 2>/dev/null | awk -F'[,/ ]+' '/nvidia/{print $2; exit}')
|
||||
|
||||
if [[ -n "${NVIDIA_DKMS_NAME}" && -n "${NVIDIA_DKMS_VER}" ]]; then
|
||||
echo "Building NVIDIA DKMS module ${NVIDIA_DKMS_NAME}/${NVIDIA_DKMS_VER} for kernel ${KVER}..."
|
||||
dkms install -m "${NVIDIA_DKMS_NAME}" -v "${NVIDIA_DKMS_VER}" --kernelver "${KVER}" --force 2>&1
|
||||
else
|
||||
echo "WARNING: nvidia DKMS module not found in dkms status — driver install may have failed"
|
||||
fi
|
||||
|
||||
# Verify the module was actually built.
|
||||
NVIDIA_KO=$(find "/lib/modules/${KVER}/updates" -name "nvidia.ko*" 2>/dev/null | head -1)
|
||||
if [[ -z "${NVIDIA_KO}" ]]; then
|
||||
echo "ERROR: nvidia.ko not found under /lib/modules/${KVER}/updates after DKMS build"
|
||||
exit 1
|
||||
fi
|
||||
echo "NVIDIA kernel module: ${NVIDIA_KO}"
|
||||
|
||||
# Pin the exact kernel + headers so unattended-upgrades cannot replace the
|
||||
# kernel the DKMS module was built for.
|
||||
apt-mark hold \
|
||||
"linux-image-${KVER}" \
|
||||
"linux-headers-${KVER}"
|
||||
|
||||
NVIDIA_PKGS=$(dpkg --get-selections 'nvidia-*' 2>/dev/null | awk '/install$/{print $1}' || true)
|
||||
LIBNVIDIA_PKGS=$(dpkg --get-selections 'libnvidia-*' 2>/dev/null | awk '/install$/{print $1}' || true)
|
||||
# shellcheck disable=SC2086
|
||||
[[ -n "${NVIDIA_PKGS}" ]] && apt-mark hold ${NVIDIA_PKGS}
|
||||
# shellcheck disable=SC2086
|
||||
[[ -n "${LIBNVIDIA_PKGS}" ]] && apt-mark hold ${LIBNVIDIA_PKGS}
|
||||
|
||||
# Blacklist nouveau inside the guest — nvidia.ko and nouveau conflict, and
|
||||
# without this blacklist the NVIDIA driver will not bind after boot.
|
||||
cat > /etc/modprobe.d/blacklist-nouveau.conf <<'EOF'
|
||||
blacklist nouveau
|
||||
options nouveau modeset=0
|
||||
EOF
|
||||
|
||||
# Common tooling matching the AMD image.
|
||||
apt-get install -y -o Acquire::Retries=3 \
|
||||
pciutils \
|
||||
python3 python3-venv python3-pip \
|
||||
git curl wget htop tmux \
|
||||
ffmpeg libgl1 libglib2.0-0
|
||||
|
||||
mkdir -p /etc/apt/apt.conf.d
|
||||
cat > /etc/apt/apt.conf.d/99-gpu-ami <<'EOF'
|
||||
Unattended-Upgrade::Package-Blacklist {
|
||||
"linux-";
|
||||
"nvidia-";
|
||||
"libnvidia-";
|
||||
};
|
||||
EOF
|
||||
|
||||
# Rebuild initramfs with nouveau blacklisted and NVIDIA module included.
|
||||
update-initramfs -u -k "${KVER}"
|
||||
|
||||
|
||||
echo "NVIDIA GPU image setup complete: kernel=${KVER}, NVIDIA driver + DKMS pre-built"
|
||||
@@ -0,0 +1,12 @@
|
||||
IMAGE_NAME="ubuntu-nvidia-gpu"
|
||||
IMAGE_DESCRIPTION="Ubuntu 26.04 NVIDIA GPU guest image — NVIDIA server driver (DKMS pre-built), Python toolchain"
|
||||
DISTRO="ubuntu"
|
||||
UBUNTU_VERSION="26.04"
|
||||
UBUNTU_CODENAME="resolute"
|
||||
|
||||
# 12G provides headroom for kernel headers, NVIDIA driver, and DKMS build artefacts
|
||||
IMAGE_SIZE="12G"
|
||||
|
||||
SETUP_SCRIPT="scripts/images/ubuntu-gpu-nvidia-setup.sh"
|
||||
|
||||
SYSTEM_TAG="gpu-vendor=nvidia"
|
||||
@@ -1,82 +0,0 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
# ubuntu-gpu-setup.sh — Chroot setup for Ubuntu 24.04 GPU AMI
|
||||
#
|
||||
# Installs NVIDIA headless server driver with DKMS module pre-built against
|
||||
# a pinned kernel. Blacklists nouveau. Prevents unattended-upgrades from
|
||||
# invalidating the pre-built module.
|
||||
#
|
||||
# Must run inside an Ubuntu 24.04 chroot with /proc /sys /dev bind-mounted
|
||||
# (required by DKMS during the nvidia package post-install hook).
|
||||
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
apt-get update
|
||||
|
||||
# Install a pinned kernel and its headers so DKMS has a stable build target.
|
||||
# linux-image-generic and linux-headers-generic pull in the current HWE stack.
|
||||
apt-get install -y --no-install-recommends \
|
||||
linux-image-generic \
|
||||
linux-headers-generic
|
||||
|
||||
# Capture the installed kernel version before adding the driver.
|
||||
KVER=$(ls /boot/vmlinuz-* 2>/dev/null | sort -V | tail -1 | sed 's|/boot/vmlinuz-||')
|
||||
if [[ -z "${KVER}" ]]; then
|
||||
echo "ERROR: No kernel found under /boot after linux-image-generic install"
|
||||
exit 1
|
||||
fi
|
||||
echo "Target kernel: ${KVER}"
|
||||
|
||||
# Install the NVIDIA headless server driver.
|
||||
# The nvidia-dkms-550-server post-install hook runs `dkms autoinstall`, which
|
||||
# builds the module against every kernel found under /lib/modules/ — including
|
||||
# ${KVER} whose headers we just installed above.
|
||||
apt-get install -y --no-install-recommends \
|
||||
nvidia-headless-550-server \
|
||||
nvidia-utils-550-server
|
||||
|
||||
# Verify the DKMS module was registered and built.
|
||||
if dkms status 2>/dev/null | grep -q "nvidia"; then
|
||||
echo "NVIDIA DKMS module status: $(dkms status 2>/dev/null | grep nvidia | head -1)"
|
||||
else
|
||||
echo "WARNING: nvidia DKMS module not found in dkms status — manual verification required"
|
||||
fi
|
||||
|
||||
# Pin the kernel so unattended-upgrades cannot replace it and invalidate
|
||||
# the pre-built DKMS module.
|
||||
apt-mark hold \
|
||||
linux-image-generic \
|
||||
linux-headers-generic \
|
||||
"linux-image-${KVER}" \
|
||||
"linux-headers-${KVER}"
|
||||
|
||||
# Pin all installed NVIDIA packages for the same reason.
|
||||
NVIDIA_PKGS=$(dpkg --get-selections 'nvidia-*' 2>/dev/null | awk '/install$/{print $1}' || true)
|
||||
LIBNVIDIA_PKGS=$(dpkg --get-selections 'libnvidia-*' 2>/dev/null | awk '/install$/{print $1}' || true)
|
||||
# shellcheck disable=SC2086
|
||||
[[ -n "${NVIDIA_PKGS}" ]] && apt-mark hold ${NVIDIA_PKGS}
|
||||
# shellcheck disable=SC2086
|
||||
[[ -n "${LIBNVIDIA_PKGS}" ]] && apt-mark hold ${LIBNVIDIA_PKGS}
|
||||
|
||||
# Blacklist nouveau so it does not conflict with the NVIDIA driver on boot.
|
||||
cat > /etc/modprobe.d/blacklist-nouveau.conf << 'EOF'
|
||||
blacklist nouveau
|
||||
options nouveau modeset=0
|
||||
EOF
|
||||
|
||||
# Prevent unattended-upgrades from updating kernel or NVIDIA packages,
|
||||
# which would break the pre-built DKMS module.
|
||||
mkdir -p /etc/apt/apt.conf.d
|
||||
cat > /etc/apt/apt.conf.d/99-gpu-ami << 'EOF'
|
||||
Unattended-Upgrade::Package-Blacklist {
|
||||
"linux-";
|
||||
"nvidia-";
|
||||
"libnvidia-";
|
||||
};
|
||||
EOF
|
||||
|
||||
# Rebuild initramfs with nouveau blacklisted and NVIDIA module included.
|
||||
update-initramfs -u -k "${KVER}"
|
||||
|
||||
echo "GPU AMI setup complete: kernel=${KVER}, NVIDIA 550 driver installed, DKMS module pre-built"
|
||||
@@ -1,12 +0,0 @@
|
||||
IMAGE_NAME="ubuntu-gpu"
|
||||
IMAGE_DESCRIPTION="Ubuntu 24.04 GPU-ready AMI (NVIDIA A10G / G5 family)"
|
||||
DISTRO="ubuntu"
|
||||
UBUNTU_VERSION="24.04"
|
||||
|
||||
# 12G provides headroom for kernel headers, NVIDIA driver, and DKMS build artefacts
|
||||
IMAGE_SIZE="12G"
|
||||
|
||||
# NVIDIA driver install and DKMS pre-build are handled in the setup script
|
||||
SETUP_SCRIPT="scripts/images/ubuntu-gpu-setup.sh"
|
||||
|
||||
SYSTEM_TAG="gpu"
|
||||
Executable
+248
@@ -0,0 +1,248 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
export AWS_PROFILE=spinifex
|
||||
|
||||
# Resolve the actual user's home — $HOME is /root when run under sudo.
|
||||
_USER_HOME=$(getent passwd "${SUDO_USER:-$(id -un)}" | cut -d: -f6)
|
||||
SSH_KEY="${SSH_KEY:-${_USER_HOME}/.ssh/spinifex-key}"
|
||||
SSH_USER="${SSH_USER:-ec2-user}"
|
||||
SSH_TIMEOUT="${SSH_TIMEOUT:-300}"
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
launch-gpu.sh — Launch a GPU instance from the Spinifex AMD or NVIDIA GPU AMI.
|
||||
|
||||
Detects the GPU brand from the specified instance type and selects the matching
|
||||
pre-built AMI automatically. Use --amd or --nvidia to override.
|
||||
|
||||
Usage:
|
||||
scripts/launch-gpu.sh --type <instance-type> [--disk GB] [--amd|--nvidia]
|
||||
|
||||
Options:
|
||||
--type TYPE EC2 instance type to launch (required)
|
||||
--disk GB Root disk size in GB (default: 50)
|
||||
--ami NAME Override the AMI name to use (default: ubuntu-amd-gpu or ubuntu-nvidia-gpu)
|
||||
--amd Force AMD GPU AMI (ubuntu-amd-gpu)
|
||||
--nvidia Force NVIDIA GPU AMI (ubuntu-nvidia-gpu)
|
||||
|
||||
Env overrides:
|
||||
SSH_KEY Path to SSH private key (default: ~/.ssh/spinifex-key)
|
||||
SSH_USER SSH user inside guest (default: ec2-user)
|
||||
SSH_TIMEOUT Seconds to wait for initial SSH (default: 300)
|
||||
EOF
|
||||
}
|
||||
|
||||
INSTANCE_TYPE=""
|
||||
DISK_GB=50
|
||||
GPU_BRAND=""
|
||||
AMI_OVERRIDE=""
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--type) INSTANCE_TYPE="$2"; shift 2 ;;
|
||||
--disk) DISK_GB="$2"; shift 2 ;;
|
||||
--ami) AMI_OVERRIDE="$2"; shift 2 ;;
|
||||
--amd) GPU_BRAND="amd"; shift ;;
|
||||
--nvidia) GPU_BRAND="nvidia"; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) echo "Unknown option: $1"; usage; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
list_gpu_types() {
|
||||
aws ec2 describe-instance-types \
|
||||
--query 'InstanceTypes[?GpuInfo].[InstanceType,GpuInfo.Gpus[0].Count,GpuInfo.Gpus[0].Manufacturer,GpuInfo.Gpus[0].Name]' \
|
||||
--output table 2>/dev/null || true
|
||||
}
|
||||
|
||||
if [ -z "$INSTANCE_TYPE" ]; then
|
||||
echo "Error: --type is required."
|
||||
echo ""
|
||||
echo "Available GPU instance types on this node:"
|
||||
list_gpu_types
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --- Validate instance type and fetch GPU info in one call ---
|
||||
echo "==> Checking instance type: ${INSTANCE_TYPE}"
|
||||
INSTANCE_INFO=$(aws ec2 describe-instance-types \
|
||||
--query "InstanceTypes[?InstanceType=='${INSTANCE_TYPE}' && GpuInfo].[GpuInfo.Gpus[0].Count,GpuInfo.Gpus[0].Manufacturer,GpuInfo.Gpus[0].Name]" \
|
||||
--output text 2>/dev/null || true)
|
||||
|
||||
if [ -z "$INSTANCE_INFO" ] || [ "$INSTANCE_INFO" = "None" ]; then
|
||||
echo "'${INSTANCE_TYPE}' is not available or is not a GPU type on this node."
|
||||
echo ""
|
||||
echo "Available GPU instance types:"
|
||||
list_gpu_types
|
||||
exit 1
|
||||
fi
|
||||
|
||||
IFS=$'\t' read -r GPU_COUNT GPU_MANUFACTURER GPU_NAME <<< "$INSTANCE_INFO"
|
||||
echo " ${GPU_COUNT}× ${GPU_NAME} (${GPU_MANUFACTURER})"
|
||||
|
||||
# --- Resolve GPU brand ---
|
||||
if [ -z "$GPU_BRAND" ]; then
|
||||
case "$GPU_MANUFACTURER" in
|
||||
AMD|amd) GPU_BRAND="amd" ;;
|
||||
NVIDIA|nvidia) GPU_BRAND="nvidia" ;;
|
||||
*)
|
||||
echo "Cannot auto-detect GPU brand from manufacturer '${GPU_MANUFACTURER}'."
|
||||
echo "Use --amd or --nvidia to specify explicitly."
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
fi
|
||||
|
||||
case "$GPU_BRAND" in
|
||||
amd) AMI_NAME="ubuntu-amd-gpu" ;;
|
||||
nvidia) AMI_NAME="ubuntu-nvidia-gpu" ;;
|
||||
esac
|
||||
|
||||
# --- Resolve AMI ---
|
||||
AMI_LOOKUP="${AMI_OVERRIDE:-${AMI_NAME}}"
|
||||
echo "==> Resolving AMI: ${AMI_LOOKUP}"
|
||||
AMI_ID=$(aws ec2 describe-images \
|
||||
--query "Images[?Name=='${AMI_LOOKUP}'].ImageId | [0]" --output text 2>/dev/null || true)
|
||||
if [ -z "$AMI_ID" ] || [ "$AMI_ID" = "None" ]; then
|
||||
echo "No AMI found with name '${AMI_LOOKUP}'."
|
||||
echo "Build and import it first, or specify the AMI name with --ami <name>"
|
||||
echo " scripts/build-system-image.sh scripts/images/ubuntu-gpu-${GPU_BRAND}.conf --import"
|
||||
exit 1
|
||||
fi
|
||||
echo " $AMI_ID"
|
||||
|
||||
# --- Resolve subnet ---
|
||||
echo "==> Resolving subnet"
|
||||
SUBNET_ID=$(aws ec2 describe-subnets \
|
||||
--query 'Subnets[?MapPublicIpOnLaunch==`true`].SubnetId | [0]' --output text 2>/dev/null || true)
|
||||
if [ -z "$SUBNET_ID" ] || [ "$SUBNET_ID" = "None" ]; then
|
||||
echo "No public subnet found"
|
||||
exit 1
|
||||
fi
|
||||
echo " $SUBNET_ID"
|
||||
|
||||
VPC_ID=$(aws ec2 describe-subnets --subnet-ids "$SUBNET_ID" \
|
||||
--query 'Subnets[0].VpcId' --output text 2>/dev/null || true)
|
||||
if [ -z "$VPC_ID" ] || [ "$VPC_ID" = "None" ]; then
|
||||
echo "Could not resolve VPC for subnet $SUBNET_ID"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --- Ensure SSH security group ---
|
||||
# The default VPC SG only allows intra-SG traffic; we need a group that
|
||||
# permits port 22 from anywhere so the SSH wait below can connect.
|
||||
SSH_SG_NAME="spinifex-gpu-ssh"
|
||||
echo "==> Ensuring SSH security group (${SSH_SG_NAME})"
|
||||
SG_ID=$(aws ec2 describe-security-groups \
|
||||
--filters "Name=group-name,Values=${SSH_SG_NAME}" "Name=vpc-id,Values=${VPC_ID}" \
|
||||
--query 'SecurityGroups[0].GroupId' --output text 2>/dev/null || true)
|
||||
if [ -z "$SG_ID" ] || [ "$SG_ID" = "None" ]; then
|
||||
SG_ID=$(aws ec2 create-security-group \
|
||||
--group-name "$SSH_SG_NAME" \
|
||||
--description "Allow SSH access to GPU instances" \
|
||||
--vpc-id "$VPC_ID" \
|
||||
--query 'GroupId' --output text)
|
||||
aws ec2 authorize-security-group-ingress \
|
||||
--group-id "$SG_ID" \
|
||||
--protocol tcp \
|
||||
--port 22 \
|
||||
--cidr 0.0.0.0/0 >/dev/null
|
||||
echo " Created $SG_ID"
|
||||
else
|
||||
echo " Reusing $SG_ID"
|
||||
fi
|
||||
|
||||
# --- Ensure SSH key ---
|
||||
if [ ! -f "$SSH_KEY" ]; then
|
||||
echo "==> Generating SSH key"
|
||||
sudo -u "${SUDO_USER:-$(id -un)}" mkdir -p "$(dirname "$SSH_KEY")"
|
||||
sudo -u "${SUDO_USER:-$(id -un)}" ssh-keygen -t ed25519 -f "$SSH_KEY" -N ""
|
||||
fi
|
||||
# Delete before import — stale key pair material causes import to fail
|
||||
# silently, leaving the instance unreachable via SSH.
|
||||
echo "==> Importing SSH key"
|
||||
aws ec2 delete-key-pair --key-name spinifex-key >/dev/null 2>&1 || true
|
||||
aws ec2 import-key-pair --key-name spinifex-key \
|
||||
--public-key-material "fileb://${SSH_KEY}.pub" >/dev/null 2>&1 || true
|
||||
|
||||
# --- Launch ---
|
||||
echo "==> Launching ${INSTANCE_TYPE} (${DISK_GB} GB)..."
|
||||
ID=$(aws ec2 run-instances \
|
||||
--image-id "$AMI_ID" \
|
||||
--instance-type "$INSTANCE_TYPE" \
|
||||
--key-name spinifex-key \
|
||||
--subnet-id "$SUBNET_ID" \
|
||||
--security-group-ids "$SG_ID" \
|
||||
--count 1 \
|
||||
--block-device-mappings "[{\"DeviceName\":\"/dev/sda1\",\"Ebs\":{\"VolumeSize\":${DISK_GB},\"VolumeType\":\"gp3\",\"DeleteOnTermination\":true}}]" \
|
||||
--query 'Instances[0].InstanceId' --output text)
|
||||
if [ -z "$ID" ] || [ "$ID" = "None" ] || [ "$ID" = "null" ]; then
|
||||
echo "run-instances returned no InstanceId"
|
||||
exit 1
|
||||
fi
|
||||
echo " $ID"
|
||||
|
||||
# --- Wait for running ---
|
||||
echo "==> Waiting for running state..."
|
||||
COUNT=0; STATE="unknown"
|
||||
while [ $COUNT -lt 120 ]; do
|
||||
STATE=$(aws ec2 describe-instances --instance-ids "$ID" \
|
||||
--query "Reservations[0].Instances[0].State.Name" --output text 2>/dev/null || echo "unknown")
|
||||
[ "$STATE" = "running" ] && break
|
||||
[ "$STATE" = "terminated" ] && { echo "Instance terminated unexpectedly"; exit 1; }
|
||||
sleep 2; COUNT=$((COUNT + 2))
|
||||
done
|
||||
[ "$STATE" = "running" ] || { echo "Instance did not reach running state (last: $STATE)"; exit 1; }
|
||||
|
||||
# --- Wait for public IP ---
|
||||
echo "==> Waiting for public IP..."
|
||||
IP=""
|
||||
for _i in $(seq 1 120); do
|
||||
IP=$(aws ec2 describe-instances --instance-ids "$ID" \
|
||||
--query 'Reservations[0].Instances[0].PublicIpAddress' --output text 2>/dev/null || true)
|
||||
[ -n "$IP" ] && [ "$IP" != "None" ] && [ "$IP" != "null" ] && break
|
||||
sleep 1
|
||||
done
|
||||
[ -n "$IP" ] && [ "$IP" != "None" ] || { echo "No public IP assigned after 120s"; exit 1; }
|
||||
echo " $IP"
|
||||
|
||||
# Clear stale known_hosts entry — instances reuse IPs across runs.
|
||||
KNOWN_HOSTS="$HOME/.ssh/known_hosts"
|
||||
[ -f "$KNOWN_HOSTS" ] && ssh-keygen -f "$KNOWN_HOSTS" -R "$IP" >/dev/null 2>&1 || true
|
||||
|
||||
# --- Wait for SSH ---
|
||||
echo "==> Waiting for SSH..."
|
||||
_ssh() {
|
||||
ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
|
||||
-o ConnectTimeout=5 -o BatchMode=yes \
|
||||
-i "$SSH_KEY" "${SSH_USER}@${IP}" "$@"
|
||||
}
|
||||
|
||||
elapsed=0
|
||||
while [ "$elapsed" -lt "$SSH_TIMEOUT" ]; do
|
||||
_ssh 'echo ready' >/dev/null 2>&1 && break
|
||||
sleep 5; elapsed=$((elapsed + 5))
|
||||
done
|
||||
[ "$elapsed" -lt "$SSH_TIMEOUT" ] || { echo "SSH timeout after ${SSH_TIMEOUT}s"; exit 1; }
|
||||
|
||||
# --- GPU visibility check ---
|
||||
echo "==> Checking GPU visibility..."
|
||||
lspci_out=$(_ssh 'lspci -nn 2>/dev/null' || true)
|
||||
if printf '%s\n' "$lspci_out" | grep -qiE "1002:|10de:|Instinct|Aqua|Processing accelerator|NVIDIA"; then
|
||||
printf '%s\n' "$lspci_out" \
|
||||
| grep -iE "1002:|10de:|Instinct|Aqua|Processing accelerator|NVIDIA" \
|
||||
| sed 's/^/ ✓ /'
|
||||
else
|
||||
echo " WARNING: GPU not visible in lspci — check host-side passthrough assignment"
|
||||
printf '%s\n' "$lspci_out" | head -20 | sed 's/^/ /'
|
||||
fi
|
||||
|
||||
# --- Done ---
|
||||
echo ""
|
||||
echo "✅ ${INSTANCE_TYPE} ready — ${ID} (${IP})"
|
||||
echo ""
|
||||
echo " ssh -i ${SSH_KEY} -o StrictHostKeyChecking=no ${SSH_USER}@${IP}"
|
||||
echo ""
|
||||
echo " Terminate when done:"
|
||||
echo " AWS_PROFILE=spinifex aws ec2 terminate-instances --instance-ids ${ID}"
|
||||
+19
-13
@@ -25,6 +25,12 @@ GPU_FAMILY="${GPU_FAMILY:-g5}"
|
||||
|
||||
export AWS_PROFILE=spinifex
|
||||
|
||||
# AWS commands run as the invoking user so their ~/.aws/config (spinifex profile
|
||||
# + CA bundle) is used when the script is called via sudo.
|
||||
INVOKING_USER="${SUDO_USER:-$(id -un)}"
|
||||
INVOKING_HOME=$(getent passwd "$INVOKING_USER" | cut -d: -f6)
|
||||
aws_as_user() { sudo -u "$INVOKING_USER" env HOME="$INVOKING_HOME" AWS_PROFILE=spinifex aws "$@"; }
|
||||
|
||||
# --- Wait for EC2 daemon to subscribe to NATS ---
|
||||
# Port 3000 (UI) becomes ready before the daemon finishes initialising.
|
||||
# Poll describe-key-pairs until it doesn't return InternalError.
|
||||
@@ -32,7 +38,7 @@ echo "==> Waiting for EC2 daemon to be ready"
|
||||
DAEMON_TIMEOUT=60
|
||||
DAEMON_ELAPSED=0
|
||||
while [ $DAEMON_ELAPSED -lt $DAEMON_TIMEOUT ]; do
|
||||
if aws ec2 describe-key-pairs --output text >/dev/null 2>&1; then
|
||||
if aws_as_user ec2 describe-key-pairs --output text >/dev/null 2>&1; then
|
||||
break
|
||||
fi
|
||||
sleep 2
|
||||
@@ -53,9 +59,9 @@ if [ ! -f "$SSH_KEY.pub" ]; then
|
||||
fi
|
||||
|
||||
echo "==> Importing SSH key"
|
||||
! aws ec2 import-key-pair --key-name spinifex-key \
|
||||
--public-key-material "fileb://$SSH_KEY.pub"
|
||||
aws ec2 describe-key-pairs
|
||||
aws_as_user ec2 import-key-pair --key-name spinifex-key \
|
||||
--public-key-material "fileb://$SSH_KEY.pub" || true
|
||||
aws_as_user ec2 describe-key-pairs
|
||||
|
||||
# --- GPU pre-flight (TEST_GPU=1 only) ---
|
||||
if [[ "$TEST_GPU" == "1" ]]; then
|
||||
@@ -82,7 +88,7 @@ if [[ "$TEST_GPU" == "1" ]]; then
|
||||
echo " GPU PCI ID: ${GPU_VENDOR_ID}:${GPU_DEVICE_ID}"
|
||||
|
||||
echo "==> [GPU] Verifying ${GPU_FAMILY} instance types advertised"
|
||||
G5_TYPES=$(aws ec2 describe-instance-types \
|
||||
G5_TYPES=$(aws_as_user ec2 describe-instance-types \
|
||||
--query "InstanceTypes[?starts_with(InstanceType, \`${GPU_FAMILY}\`)].InstanceType" \
|
||||
--output text 2>/dev/null || true)
|
||||
if [[ -z "$G5_TYPES" ]]; then
|
||||
@@ -110,7 +116,7 @@ esac
|
||||
|
||||
AMI_NAME="ami-${IMAGE_NAME}"
|
||||
|
||||
EXISTING_AMI_ID=$(aws ec2 describe-images \
|
||||
EXISTING_AMI_ID=$(aws_as_user ec2 describe-images \
|
||||
--query "Images[?Name=='${AMI_NAME}'] | [0].ImageId" \
|
||||
--output text)
|
||||
|
||||
@@ -137,7 +143,7 @@ else
|
||||
INSTANCE_TYPE="t3.small"
|
||||
fi
|
||||
|
||||
AMI_ID=$(aws ec2 describe-images \
|
||||
AMI_ID=$(aws_as_user ec2 describe-images \
|
||||
--query "Images[?Name=='${AMI_NAME}'] | [0].ImageId" \
|
||||
--output text)
|
||||
|
||||
@@ -146,7 +152,7 @@ if [ -z "$AMI_ID" ] || [ "$AMI_ID" = "None" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
SUBNET_ID=$(aws ec2 describe-subnets --query 'Subnets[?MapPublicIpOnLaunch==`true`].SubnetId | [0]' --output text)
|
||||
SUBNET_ID=$(aws_as_user ec2 describe-subnets --query 'Subnets[?MapPublicIpOnLaunch==`true`].SubnetId | [0]' --output text)
|
||||
if [ -z "$SUBNET_ID" ] || [ "$SUBNET_ID" = "None" ]; then
|
||||
echo "❌ No subnet found"
|
||||
exit 1
|
||||
@@ -154,7 +160,7 @@ fi
|
||||
|
||||
echo " AMI: $AMI_ID type: $INSTANCE_TYPE subnet: $SUBNET_ID"
|
||||
|
||||
INSTANCE_ID=$(aws ec2 run-instances \
|
||||
INSTANCE_ID=$(aws_as_user ec2 run-instances \
|
||||
--image-id "$AMI_ID" \
|
||||
--instance-type "$INSTANCE_TYPE" \
|
||||
--key-name spinifex-key \
|
||||
@@ -173,7 +179,7 @@ echo "==> Waiting for instance to reach running state"
|
||||
COUNT=0
|
||||
STATE="unknown"
|
||||
while [ $COUNT -lt 60 ]; do
|
||||
DESCRIBE=$(aws ec2 describe-instances --instance-ids "$INSTANCE_ID" 2>/dev/null) || {
|
||||
DESCRIBE=$(aws_as_user ec2 describe-instances --instance-ids "$INSTANCE_ID" 2>/dev/null) || {
|
||||
sleep 2; COUNT=$((COUNT + 1)); continue
|
||||
}
|
||||
STATE=$(echo "$DESCRIBE" | jq -r '.Reservations[0].Instances[0].State.Name // "not-found"')
|
||||
@@ -196,7 +202,7 @@ echo "==> Waiting for public IP assignment (up to 300s)"
|
||||
SSH_INST_PORT=22
|
||||
SSH_INST_HOST=""
|
||||
for _i in $(seq 1 300); do
|
||||
_ip=$(aws ec2 describe-instances --instance-ids "$INSTANCE_ID" \
|
||||
_ip=$(aws_as_user ec2 describe-instances --instance-ids "$INSTANCE_ID" \
|
||||
--query 'Reservations[0].Instances[0].PublicIpAddress' --output text 2>/dev/null || true)
|
||||
if [ -n "$_ip" ] && [ "$_ip" != "None" ] && [ "$_ip" != "null" ]; then
|
||||
SSH_INST_HOST="$_ip"
|
||||
@@ -256,14 +262,14 @@ if [[ "$TEST_GPU" == "1" ]]; then
|
||||
-o ConnectTimeout=5 -o BatchMode=yes \
|
||||
-p "$SSH_INST_PORT" -i "$SSH_KEY" \
|
||||
ec2-user@"$SSH_INST_HOST" \
|
||||
'lspci 2>/dev/null | grep -i nvidia' 2>/dev/null; then
|
||||
'lspci 2>/dev/null | grep -iE "nvidia|amdgpu|instinct"' 2>/dev/null; then
|
||||
GPU_VISIBLE=1
|
||||
break
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
if [[ $GPU_VISIBLE -eq 0 ]]; then
|
||||
echo "❌ NVIDIA GPU not visible in guest via lspci"
|
||||
echo "❌ GPU not visible in guest via lspci"
|
||||
exit 1
|
||||
fi
|
||||
echo " GPU visible in guest"
|
||||
|
||||
+29
-10
@@ -489,17 +489,34 @@ func (rm *ResourceManager) GetAvailableInstanceTypeInfos(showCapacity bool) []*e
|
||||
continue
|
||||
}
|
||||
|
||||
requiresGPU := instancetypes.IsGPUType(it)
|
||||
availGPU := 0
|
||||
if rm.gpuManager != nil && requiresGPU {
|
||||
availGPU = rm.gpuManager.Available()
|
||||
// GPU types are capacity-gated by GPU pool size, not host CPU/memory.
|
||||
// The GPU is the scarce resource; CPU/memory on GPU-class hardware is abundant.
|
||||
if instancetypes.IsGPUType(it) {
|
||||
availGPU := 0
|
||||
if rm.gpuManager != nil {
|
||||
availGPU = rm.gpuManager.Available()
|
||||
}
|
||||
gpusNeeded := instancetypes.GPUCountForType(name)
|
||||
count := 0
|
||||
if gpusNeeded > 0 {
|
||||
count = availGPU / gpusNeeded
|
||||
}
|
||||
if showCapacity {
|
||||
for range count {
|
||||
infos = append(infos, it)
|
||||
}
|
||||
} else if count > 0 {
|
||||
infos = append(infos, it)
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
count := canAllocateCount(
|
||||
rm.hostVCPU-rm.reservedVCPU, rm.allocatedVCPU,
|
||||
rm.hostMemGB-rm.reservedMem, rm.allocatedMem,
|
||||
vCPUs, memMiB,
|
||||
1<<30, // effectively unlimited — let resources be the constraint
|
||||
availGPU, requiresGPU,
|
||||
0, false,
|
||||
)
|
||||
|
||||
if showCapacity {
|
||||
@@ -1732,6 +1749,7 @@ func (d *Daemon) applyGPUConfig(enabled bool) {
|
||||
mgr := gpu.NewManager(probe.Devices)
|
||||
d.gpuManager = mgr
|
||||
d.resourceMgr.reloadGPUTypes(models, mgr)
|
||||
d.instanceService.SetGPUClaimer(&daemonGPUClaimer{d: d})
|
||||
slog.Info("GPU passthrough enabled via config reload", "gpus", len(probe.Devices))
|
||||
return
|
||||
}
|
||||
@@ -1743,6 +1761,7 @@ func (d *Daemon) applyGPUConfig(enabled bool) {
|
||||
return
|
||||
}
|
||||
d.gpuManager = nil
|
||||
d.instanceService.SetGPUClaimer(nil)
|
||||
d.resourceMgr.reloadGPUTypes(nil, nil)
|
||||
slog.Info("GPU passthrough disabled via config reload")
|
||||
}
|
||||
@@ -1848,10 +1867,10 @@ func (rm *ResourceManager) canAllocate(instanceType *ec2.InstanceTypeInfo, count
|
||||
rm.mu.RLock()
|
||||
defer rm.mu.RUnlock()
|
||||
|
||||
requiresGPU := instancetypes.IsGPUType(instanceType)
|
||||
availGPU := 0
|
||||
if rm.gpuManager != nil && requiresGPU {
|
||||
availGPU = rm.gpuManager.Available()
|
||||
// GPU capacity is managed exclusively by gpuManager.Claim; don't double-gate
|
||||
// on host CPU/memory which is always abundant on GPU-class hardware.
|
||||
if instancetypes.IsGPUType(instanceType) {
|
||||
return count
|
||||
}
|
||||
|
||||
return canAllocateCount(
|
||||
@@ -1860,7 +1879,7 @@ func (rm *ResourceManager) canAllocate(instanceType *ec2.InstanceTypeInfo, count
|
||||
instanceTypeVCPUs(instanceType),
|
||||
instanceTypeMemoryMiB(instanceType),
|
||||
count,
|
||||
availGPU, requiresGPU,
|
||||
0, false,
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
@@ -48,10 +48,13 @@ import (
|
||||
func drainAndClose(t *testing.T, nc *nats.Conn) {
|
||||
t.Helper()
|
||||
done := make(chan struct{})
|
||||
nc.SetClosedHandler(func(*nats.Conn) { close(done) })
|
||||
var once sync.Once
|
||||
nc.SetClosedHandler(func(*nats.Conn) { once.Do(func() { close(done) }) })
|
||||
if err := nc.Drain(); err != nil {
|
||||
nc.Close()
|
||||
return
|
||||
// Do not return early: the handler callback may still be running
|
||||
// (e.g. writing state to the temp dir). Fall through to the wait so
|
||||
// we give it time to finish before t.TempDir() RemoveAll runs.
|
||||
}
|
||||
select {
|
||||
case <-done:
|
||||
@@ -3137,6 +3140,117 @@ func TestGetAvailableInstanceTypeInfos_ShowCapacity(t *testing.T) {
|
||||
assert.Len(t, infos, 1)
|
||||
}
|
||||
|
||||
// --- GPU-gated GetAvailableInstanceTypeInfos ---
|
||||
|
||||
func makeGPUInstanceType(name string, vcpus int64, memMiB int64) *ec2.InstanceTypeInfo {
|
||||
return &ec2.InstanceTypeInfo{
|
||||
InstanceType: aws.String(name),
|
||||
VCpuInfo: &ec2.VCpuInfo{DefaultVCpus: aws.Int64(vcpus)},
|
||||
MemoryInfo: &ec2.MemoryInfo{SizeInMiB: aws.Int64(memMiB)},
|
||||
GpuInfo: &ec2.GpuInfo{
|
||||
Gpus: []*ec2.GpuDeviceInfo{{
|
||||
Count: aws.Int64(int64(instancetypes.GPUCountForType(name))),
|
||||
Manufacturer: aws.String("AMD"),
|
||||
Name: aws.String("Instinct MI350X"),
|
||||
MemoryInfo: &ec2.GpuDeviceMemoryInfo{SizeInMiB: aws.Int64(294896)},
|
||||
}},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func makeGPUDevice() gpu.GPUDevice {
|
||||
return gpu.GPUDevice{VendorID: "1002", DeviceID: "75a0", PCIAddress: "0000:03:00.0", Vendor: gpu.VendorAMD}
|
||||
}
|
||||
|
||||
// GPU types with no gpuManager are hidden — there is no GPU hardware on this node.
|
||||
func TestGetAvailableInstanceTypeInfos_GPUType_NoManager_Hidden(t *testing.T) {
|
||||
rm := &ResourceManager{
|
||||
hostVCPU: 128,
|
||||
hostMemGB: 1024.0,
|
||||
instanceTypes: map[string]*ec2.InstanceTypeInfo{
|
||||
"g7e.4xlarge": makeGPUInstanceType("g7e.4xlarge", 16, 128*1024),
|
||||
},
|
||||
}
|
||||
infos := rm.GetAvailableInstanceTypeInfos(false)
|
||||
assert.Empty(t, infos, "GPU type must not appear when gpuManager is nil")
|
||||
}
|
||||
|
||||
// GPU types gate on pool availability, not host CPU/memory — even a tiny host
|
||||
// with a GPU pool exposes the type.
|
||||
func TestGetAvailableInstanceTypeInfos_GPUType_GatedByPool(t *testing.T) {
|
||||
mgr := gpu.NewManager([]gpu.GPUDevice{makeGPUDevice(), makeGPUDevice()})
|
||||
rm := &ResourceManager{
|
||||
hostVCPU: 4, // would normally block a 16-vCPU instance
|
||||
hostMemGB: 8.0, // would normally block a 128 GB instance
|
||||
instanceTypes: map[string]*ec2.InstanceTypeInfo{
|
||||
"g7e.4xlarge": makeGPUInstanceType("g7e.4xlarge", 16, 128*1024),
|
||||
},
|
||||
gpuManager: mgr,
|
||||
}
|
||||
infos := rm.GetAvailableInstanceTypeInfos(false)
|
||||
assert.Len(t, infos, 1, "GPU type must appear when pool has free GPUs regardless of CPU/memory")
|
||||
}
|
||||
|
||||
// Multi-GPU type (g7e.12xlarge, needs 2) appears only when pool has ≥ 2 free GPUs.
|
||||
func TestGetAvailableInstanceTypeInfos_MultiGPUType_RequiresTwoGPUs(t *testing.T) {
|
||||
oneGPU := gpu.NewManager([]gpu.GPUDevice{makeGPUDevice()})
|
||||
rm := &ResourceManager{
|
||||
hostVCPU: 128,
|
||||
hostMemGB: 1024.0,
|
||||
instanceTypes: map[string]*ec2.InstanceTypeInfo{
|
||||
"g7e.12xlarge": makeGPUInstanceType("g7e.12xlarge", 48, 512*1024),
|
||||
},
|
||||
gpuManager: oneGPU,
|
||||
}
|
||||
infos := rm.GetAvailableInstanceTypeInfos(false)
|
||||
assert.Empty(t, infos, "g7e.12xlarge must not appear with only 1 free GPU")
|
||||
|
||||
twoGPUs := gpu.NewManager([]gpu.GPUDevice{makeGPUDevice(), makeGPUDevice()})
|
||||
rm.gpuManager = twoGPUs
|
||||
infos = rm.GetAvailableInstanceTypeInfos(false)
|
||||
assert.Len(t, infos, 1, "g7e.12xlarge must appear when pool has ≥ 2 free GPUs")
|
||||
}
|
||||
|
||||
// showCapacity=true for GPU types emits pool/gpuCount slots.
|
||||
func TestGetAvailableInstanceTypeInfos_GPUType_ShowCapacity(t *testing.T) {
|
||||
mgr := gpu.NewManager([]gpu.GPUDevice{makeGPUDevice(), makeGPUDevice(), makeGPUDevice(), makeGPUDevice()})
|
||||
rm := &ResourceManager{
|
||||
hostVCPU: 128,
|
||||
hostMemGB: 1024.0,
|
||||
instanceTypes: map[string]*ec2.InstanceTypeInfo{
|
||||
"g7e.4xlarge": makeGPUInstanceType("g7e.4xlarge", 16, 128*1024),
|
||||
"g7e.12xlarge": makeGPUInstanceType("g7e.12xlarge", 48, 512*1024),
|
||||
},
|
||||
gpuManager: mgr,
|
||||
}
|
||||
// 4 GPUs → 4× g7e.4xlarge (1 GPU each) or 2× g7e.12xlarge (2 GPUs each)
|
||||
infos := rm.GetAvailableInstanceTypeInfos(true)
|
||||
count4xl := 0
|
||||
count12xl := 0
|
||||
for _, it := range infos {
|
||||
switch *it.InstanceType {
|
||||
case "g7e.4xlarge":
|
||||
count4xl++
|
||||
case "g7e.12xlarge":
|
||||
count12xl++
|
||||
}
|
||||
}
|
||||
assert.Equal(t, 4, count4xl, "4 GPUs → 4 slots for g7e.4xlarge")
|
||||
assert.Equal(t, 2, count12xl, "4 GPUs → 2 slots for g7e.12xlarge")
|
||||
}
|
||||
|
||||
// canAllocate for GPU types returns count without CPU/memory gating.
|
||||
func TestCanAllocate_GPUType_BypassesCPUMemory(t *testing.T) {
|
||||
rm := &ResourceManager{
|
||||
hostVCPU: 4, // deliberately tiny
|
||||
hostMemGB: 8.0, // deliberately tiny
|
||||
instanceTypes: map[string]*ec2.InstanceTypeInfo{},
|
||||
}
|
||||
gpuType := makeGPUInstanceType("g7e.4xlarge", 16, 128*1024)
|
||||
assert.Equal(t, 1, rm.canAllocate(gpuType, 1),
|
||||
"GPU type must be allocatable even when CPU/memory would normally block it")
|
||||
}
|
||||
|
||||
// --- NewDaemon ---
|
||||
|
||||
func TestNewDaemon_WalDirDefaultsToBaseDir(t *testing.T) {
|
||||
|
||||
@@ -56,6 +56,10 @@ func createDaemonWithJetStream(t *testing.T) *Daemon {
|
||||
ConsumeCleanShutdownMarker: daemon.consumeCleanShutdownMarker(),
|
||||
})
|
||||
|
||||
// Registered after t.TempDir() so it runs first (LIFO): goroutines
|
||||
// spawned by MarkFailed/MarkRecoveryFailed finish before RemoveAll runs.
|
||||
t.Cleanup(daemon.vmMgr.WaitForBackgroundWork)
|
||||
|
||||
return daemon
|
||||
}
|
||||
|
||||
|
||||
@@ -398,16 +398,18 @@ func (d *Daemon) onInstanceUpHook() func(*vm.VM) error {
|
||||
}
|
||||
d.natsSubscriptions[consoleSubKey] = consoleSub
|
||||
|
||||
// Re-claim GPU after a daemon restart with a still-running QEMU
|
||||
// Re-claim GPU(s) after a daemon restart with a still-running QEMU
|
||||
// process: the manager's reconnect path fires OnInstanceUp without
|
||||
// going through the handler-side Claim, so the GPU pool would
|
||||
// otherwise treat the slot as free. ReclaimByAddress is a no-op
|
||||
// when the same instance already owns the slot, so the launch and
|
||||
// start-stopped paths (which Claim before Run) are unaffected.
|
||||
if d.gpuManager != nil && instance.GPUPCIAddress != "" {
|
||||
if err := d.gpuManager.ReclaimByAddress(instance.GPUPCIAddress, instance.ID); err != nil {
|
||||
slog.Warn("Failed to re-claim GPU on instance up",
|
||||
"gpu", instance.GPUPCIAddress, "instanceId", instance.ID, "err", err)
|
||||
if d.gpuManager != nil {
|
||||
for _, addr := range instance.GPUPCIAddresses {
|
||||
if err := d.gpuManager.ReclaimByAddress(addr, instance.ID); err != nil {
|
||||
slog.Warn("Failed to re-claim GPU on instance up",
|
||||
"gpu", addr, "instanceId", instance.ID, "err", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
@@ -630,13 +632,13 @@ func (a *instanceCleanerAdapter) RemoveFromPlacementGroup(instance *vm.VM) {
|
||||
// original host driver. No-op for instances without a GPU allocation or
|
||||
// when GPU passthrough is disabled.
|
||||
func (a *instanceCleanerAdapter) ReleaseGPU(instance *vm.VM) {
|
||||
if a.d.gpuManager == nil || instance.GPUPCIAddress == "" {
|
||||
if a.d.gpuManager == nil || len(instance.GPUPCIAddresses) == 0 {
|
||||
return
|
||||
}
|
||||
if err := a.d.gpuManager.Release(instance.ID); err != nil {
|
||||
slog.Error("Failed to release GPU on stop, device may need manual rebind",
|
||||
"gpu", instance.GPUPCIAddress, "instanceId", instance.ID, "err", err)
|
||||
"gpus", instance.GPUPCIAddresses, "instanceId", instance.ID, "err", err)
|
||||
return
|
||||
}
|
||||
slog.Info("GPU released", "gpu", instance.GPUPCIAddress, "instanceId", instance.ID)
|
||||
slog.Info("GPU released", "gpus", instance.GPUPCIAddresses, "instanceId", instance.ID)
|
||||
}
|
||||
|
||||
@@ -97,10 +97,10 @@ func TestOnInstanceDownHook_NoOpWhenAbsent(t *testing.T) {
|
||||
}
|
||||
|
||||
// When the daemon's gpuManager is unset, the hook must still register NATS
|
||||
// subscriptions and ignore the GPUPCIAddress on the VM.
|
||||
// subscriptions and ignore the GPUPCIAddresses on the VM.
|
||||
func TestOnInstanceUpHook_NoGPUManager_SkipsReclaim(t *testing.T) {
|
||||
d, _ := newHookTestDaemon(t)
|
||||
instance := &vm.VM{ID: "i-up-nogpu", GPUPCIAddress: "0000:01:00.0"}
|
||||
instance := &vm.VM{ID: "i-up-nogpu", GPUPCIAddresses: []string{"0000:01:00.0"}}
|
||||
|
||||
require.NoError(t, d.onInstanceUpHook()(instance))
|
||||
require.Contains(t, d.natsSubscriptions, instance.ID)
|
||||
@@ -116,17 +116,17 @@ func TestOnInstanceUpHook_NoGPUAddress_SkipsReclaim(t *testing.T) {
|
||||
|
||||
require.NoError(t, d.onInstanceUpHook()(instance))
|
||||
assert.Equal(t, 0, d.gpuManager.AllocatedCount(),
|
||||
"hook must not call Reclaim for instances without a GPUPCIAddress")
|
||||
"hook must not call Reclaim for instances without GPUPCIAddresses")
|
||||
}
|
||||
|
||||
// With a gpuManager that has no entries, calling Reclaim for an instance
|
||||
// with a GPUPCIAddress will fail inside the manager. The hook logs a warning
|
||||
// with a GPUPCIAddresses entry will fail inside the manager. The hook logs a warning
|
||||
// and returns nil — the NATS subscriptions must still register so the
|
||||
// reconnect path doesn't roll back.
|
||||
func TestOnInstanceUpHook_GPUReclaimError_DoesNotPropagate(t *testing.T) {
|
||||
d, _ := newHookTestDaemon(t)
|
||||
d.gpuManager = gpu.NewManager(nil)
|
||||
instance := &vm.VM{ID: "i-up-gpu-missing", GPUPCIAddress: "0000:99:00.0"}
|
||||
instance := &vm.VM{ID: "i-up-gpu-missing", GPUPCIAddresses: []string{"0000:99:00.0"}}
|
||||
|
||||
require.NoError(t, d.onInstanceUpHook()(instance))
|
||||
require.Contains(t, d.natsSubscriptions, instance.ID)
|
||||
@@ -149,3 +149,35 @@ func TestOnInstanceDownHook_OnlyRemovesTargetedInstance(t *testing.T) {
|
||||
_, dropPresent := d.natsSubscriptions[drop.ID]
|
||||
assert.False(t, dropPresent)
|
||||
}
|
||||
|
||||
// --- ReleaseGPU ---
|
||||
|
||||
// ReleaseGPU is a no-op when the daemon has no GPU manager.
|
||||
func TestReleaseGPU_NoManager_NoOp(t *testing.T) {
|
||||
d := &Daemon{}
|
||||
a := newInstanceCleanerAdapter(d)
|
||||
instance := &vm.VM{ID: "i-nogpu", GPUPCIAddresses: []string{"0000:03:00.0"}}
|
||||
// Must not panic.
|
||||
a.ReleaseGPU(instance)
|
||||
}
|
||||
|
||||
// ReleaseGPU is a no-op for instances without a GPU allocation.
|
||||
func TestReleaseGPU_NoAddresses_NoOp(t *testing.T) {
|
||||
mgr := gpu.NewManager(nil)
|
||||
d := &Daemon{gpuManager: mgr}
|
||||
a := newInstanceCleanerAdapter(d)
|
||||
instance := &vm.VM{ID: "i-nogpu"}
|
||||
// Must not panic or error.
|
||||
a.ReleaseGPU(instance)
|
||||
}
|
||||
|
||||
// ReleaseGPU logs a warning when the manager returns an error (instance not claimed).
|
||||
func TestReleaseGPU_ManagerError_LogsWarning(t *testing.T) {
|
||||
mgr := gpu.NewManager(nil)
|
||||
d := &Daemon{gpuManager: mgr}
|
||||
a := newInstanceCleanerAdapter(d)
|
||||
// GPU address set but no claim registered — Release returns an error.
|
||||
instance := &vm.VM{ID: "i-unclaimed", GPUPCIAddresses: []string{"0000:03:00.0"}}
|
||||
// Must not panic; the error is logged as a warning.
|
||||
a.ReleaseGPU(instance)
|
||||
}
|
||||
|
||||
@@ -29,7 +29,7 @@ func discover(sysfsRoot string) ([]GPUDevice, error) {
|
||||
devPath := filepath.Join(devicesPath, entry.Name())
|
||||
|
||||
class, err := readSysfsString(filepath.Join(devPath, "class"))
|
||||
if err != nil || !isDisplayClass(class) {
|
||||
if err != nil || !isPassthroughClass(class) {
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -144,12 +144,13 @@ func enrichNVIDIA(gpu *GPUDevice) {
|
||||
gpu.MemoryMiB = memMiB
|
||||
}
|
||||
|
||||
// isDisplayClass reports whether a PCI class string (e.g. "0x030200") is a
|
||||
// display controller (class byte 0x03).
|
||||
func isDisplayClass(class string) bool {
|
||||
// isPassthroughClass reports whether a PCI class string identifies a GPU
|
||||
// eligible for VFIO passthrough: display controllers (0x03xx) or processing
|
||||
// accelerators (0x12xx) such as AMD Instinct compute GPUs.
|
||||
func isPassthroughClass(class string) bool {
|
||||
s := strings.ToLower(strings.TrimSpace(class))
|
||||
s = strings.TrimPrefix(s, "0x")
|
||||
return len(s) >= 2 && s[:2] == "03"
|
||||
return len(s) >= 2 && (s[:2] == "03" || s[:2] == "12")
|
||||
}
|
||||
|
||||
// readIOMMUGroup reads the IOMMU group number from the iommu_group symlink.
|
||||
|
||||
@@ -209,7 +209,7 @@ func TestDiscover_SkipsIntelGPU(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsDisplayClass(t *testing.T) {
|
||||
func TestIsPassthroughClass(t *testing.T) {
|
||||
cases := []struct {
|
||||
class string
|
||||
want bool
|
||||
@@ -217,13 +217,14 @@ func TestIsDisplayClass(t *testing.T) {
|
||||
{"0x030200", true}, // 3D controller
|
||||
{"0x030000", true}, // VGA
|
||||
{"0x038000", true}, // display controller
|
||||
{"0x120000", true}, // processing accelerator (AMD Instinct)
|
||||
{"0x010802", false}, // NVMe
|
||||
{"0x0c0330", false}, // USB
|
||||
{"0x020000", false}, // Ethernet
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := isDisplayClass(c.class); got != c.want {
|
||||
t.Errorf("isDisplayClass(%q) = %v, want %v", c.class, got, c.want)
|
||||
if got := isPassthroughClass(c.class); got != c.want {
|
||||
t.Errorf("isPassthroughClass(%q) = %v, want %v", c.class, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+53
-36
@@ -108,52 +108,69 @@ func (m *Manager) Release(instanceID string) error {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
idx := -1
|
||||
for i := range m.pool {
|
||||
if m.pool[i].InstanceID == instanceID {
|
||||
idx = i
|
||||
break
|
||||
}
|
||||
}
|
||||
if idx == -1 {
|
||||
return fmt.Errorf("no GPU claimed by instance %s", instanceID)
|
||||
}
|
||||
|
||||
entry := &m.pool[idx]
|
||||
var firstErr error
|
||||
released := false
|
||||
|
||||
for _, member := range entry.groupMembers {
|
||||
orig := entry.memberDrivers[member.PCIAddress]
|
||||
if orig == "vfio-pci" {
|
||||
// GPU was already bound to vfio-pci before this instance (e.g. by
|
||||
// spx-test-gpu at node setup). QEMU released the group fd on exit;
|
||||
// the device stays vfio-pci bound and is immediately available for
|
||||
// the next claim. Skipping unbind avoids a needless round-trip and
|
||||
// eliminates a brief window where the device is unbound.
|
||||
for i := range m.pool {
|
||||
if m.pool[i].InstanceID != instanceID {
|
||||
continue
|
||||
}
|
||||
if err := unbindVFIO(m.sysfsRoot, member.PCIAddress, orig); err != nil {
|
||||
slog.Error("GPU release failed for IOMMU group member",
|
||||
"instance", instanceID, "pci", member.PCIAddress, "err", err)
|
||||
if firstErr == nil {
|
||||
firstErr = err
|
||||
released = true
|
||||
entry := &m.pool[i]
|
||||
|
||||
for _, member := range entry.groupMembers {
|
||||
orig := entry.memberDrivers[member.PCIAddress]
|
||||
if orig == "vfio-pci" {
|
||||
// GPU was already bound to vfio-pci before this instance (e.g. by
|
||||
// spx-test-gpu at node setup). QEMU released the group fd on exit;
|
||||
// the device stays vfio-pci bound and is immediately available for
|
||||
// the next claim. Skipping unbind avoids a needless round-trip and
|
||||
// eliminates a brief window where the device is unbound.
|
||||
continue
|
||||
}
|
||||
if err := unbindVFIO(m.sysfsRoot, member.PCIAddress, orig); err != nil {
|
||||
slog.Error("GPU release failed for IOMMU group member",
|
||||
"instance", instanceID, "pci", member.PCIAddress, "err", err)
|
||||
if firstErr == nil {
|
||||
firstErr = err
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
entry.InstanceID = ""
|
||||
entry.groupMembers = nil
|
||||
entry.memberDrivers = nil
|
||||
|
||||
if firstErr != nil {
|
||||
entry.Available = false
|
||||
slog.Error("GPU marked unavailable after failed release — operator action required",
|
||||
"gpu", entry.Device.PCIAddress)
|
||||
} else {
|
||||
slog.Info("GPU released", "instance", instanceID, "gpu", entry.Device.PCIAddress)
|
||||
}
|
||||
}
|
||||
|
||||
entry.InstanceID = ""
|
||||
entry.groupMembers = nil
|
||||
entry.memberDrivers = nil
|
||||
|
||||
if firstErr != nil {
|
||||
entry.Available = false
|
||||
slog.Error("GPU marked unavailable after failed release — operator action required",
|
||||
"gpu", entry.Device.PCIAddress)
|
||||
return firstErr
|
||||
if !released {
|
||||
return fmt.Errorf("no GPU claimed by instance %s", instanceID)
|
||||
}
|
||||
return firstErr
|
||||
}
|
||||
|
||||
slog.Info("GPU released", "instance", instanceID, "gpu", entry.Device.PCIAddress)
|
||||
return nil
|
||||
// MarkFailed marks a GPU as permanently unavailable after a VFIO/QEMU launch
|
||||
// error. The entry is skipped by Claim until the daemon restarts. Call this
|
||||
// after releasing a GPU whose QEMU crashed during startup so the same broken
|
||||
// device is not immediately re-assigned to the next instance.
|
||||
func (m *Manager) MarkFailed(pciAddress string) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
for i := range m.pool {
|
||||
if m.pool[i].Device.PCIAddress == pciAddress && m.pool[i].InstanceID == "" {
|
||||
m.pool[i].Available = false
|
||||
slog.Warn("GPU marked unavailable after launch failure — restart daemon to retry",
|
||||
"pci", pciAddress)
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Available returns the count of GPUs that can be claimed right now.
|
||||
|
||||
@@ -292,6 +292,38 @@ func TestManagerCounts_MultiGPU(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagerMarkFailed_MakesUnavailable(t *testing.T) {
|
||||
root, gpu := buildManagerSysfs(t)
|
||||
m := newManagerForTest([]GPUDevice{gpu}, root)
|
||||
|
||||
if m.Available() != 1 {
|
||||
t.Fatalf("want Available=1 before MarkFailed, got %d", m.Available())
|
||||
}
|
||||
m.MarkFailed(gpu.PCIAddress)
|
||||
if m.Available() != 0 {
|
||||
t.Errorf("want Available=0 after MarkFailed, got %d", m.Available())
|
||||
}
|
||||
if m.TotalCount() != 1 {
|
||||
t.Errorf("want TotalCount=1 after MarkFailed (entry retained), got %d", m.TotalCount())
|
||||
}
|
||||
// MarkFailed on an unknown address must be a no-op (not panic).
|
||||
m.MarkFailed("ffff:ff:ff.f")
|
||||
}
|
||||
|
||||
func TestManagerMarkFailed_ClaimedEntryIgnored(t *testing.T) {
|
||||
root, gpu := buildManagerSysfs(t)
|
||||
m := newManagerForTest([]GPUDevice{gpu}, root)
|
||||
|
||||
if _, err := m.Claim("i-running"); err != nil {
|
||||
t.Fatalf("Claim: %v", err)
|
||||
}
|
||||
// MarkFailed on a claimed (non-empty InstanceID) entry must not mark it unavailable.
|
||||
m.MarkFailed(gpu.PCIAddress)
|
||||
if m.AllocatedCount() != 1 {
|
||||
t.Errorf("claimed GPU must still be allocated after MarkFailed, AllocatedCount=%d", m.AllocatedCount())
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagerReclaim_Success(t *testing.T) {
|
||||
root, gpu := buildManagerSysfs(t)
|
||||
m := newManagerForTest([]GPUDevice{gpu}, root)
|
||||
|
||||
@@ -25,6 +25,7 @@ var knownModels = map[string]modelInfo{
|
||||
"10de:2782": {"NVIDIA GeForce RTX 4070", 12288},
|
||||
"10de:2204": {"NVIDIA GeForce RTX 3090", 24576},
|
||||
// AMD datacenter
|
||||
"1002:75a0": {"AMD Instinct MI350X", 294896},
|
||||
"1002:7448": {"AMD Instinct MI300X", 196608},
|
||||
"1002:740c": {"AMD Instinct MI250X", 65536},
|
||||
// AMD consumer
|
||||
@@ -49,6 +50,7 @@ var computeModels = map[string]bool{
|
||||
"10de:26b5": true, // L40
|
||||
"10de:26b9": true, // L40S
|
||||
// AMD datacenter
|
||||
"1002:75a0": true, // MI350X
|
||||
"1002:7448": true, // MI300X
|
||||
"1002:740c": true, // MI250X
|
||||
}
|
||||
|
||||
@@ -609,14 +609,14 @@ func (s *InstanceServiceImpl) LaunchRunInstances(instances []*vm.VM, input *ec2.
|
||||
s.vmMgr.MarkFailed(instance, "gpu_claim_failed")
|
||||
continue
|
||||
}
|
||||
instance.GPUPCIAddress = pciAddr
|
||||
instance.GPUPCIAddresses = []string{pciAddr}
|
||||
instance.GPUXVGAEnabled = xvga
|
||||
slog.Info("LaunchRunInstances: GPU claimed for instance", "instanceId", instance.ID, "gpu", pciAddr, "xvga", xvga)
|
||||
}
|
||||
|
||||
if err := s.vmMgr.Run(instance); err != nil {
|
||||
slog.Error("LaunchRunInstances: vmMgr.Run failed", "instanceId", instance.ID, "err", err)
|
||||
if instance.GPUPCIAddress != "" && s.gpuClaimer != nil {
|
||||
if len(instance.GPUPCIAddresses) > 0 && s.gpuClaimer != nil {
|
||||
if releaseErr := s.gpuClaimer.Release(instance.ID); releaseErr != nil {
|
||||
slog.Error("LaunchRunInstances: GPU release failed after launch failure",
|
||||
"instanceId", instance.ID, "err", releaseErr)
|
||||
@@ -1776,7 +1776,7 @@ func (s *InstanceServiceImpl) StartStoppedInstance(input *StartStoppedInstanceIn
|
||||
s.vmMgr.Delete(instance.ID)
|
||||
return nil, errors.New(awserrors.ErrorInsufficientInstanceCapacity)
|
||||
}
|
||||
instance.GPUPCIAddress = pciAddr
|
||||
instance.GPUPCIAddresses = []string{pciAddr}
|
||||
instance.GPUXVGAEnabled = xvga
|
||||
gpuClaimed = true
|
||||
slog.Info("GPU claimed for instance", "instanceId", input.InstanceID, "gpu", pciAddr, "xvga", xvga)
|
||||
|
||||
@@ -29,6 +29,7 @@ var (
|
||||
|
||||
// AMD GPU models
|
||||
AMDradeonV520 = GPUModel{"1002", "7362", "g4ad", "AMD", "Radeon Pro V520", 8192}
|
||||
AMDmi350x = GPUModel{"1002", "75a0", "g7e", "AMD", "Instinct MI350X", 294896}
|
||||
)
|
||||
|
||||
var knownGPUModels = []GPUModel{
|
||||
@@ -45,6 +46,7 @@ var knownGPUModels = []GPUModel{
|
||||
NVIDIAh100pcie,
|
||||
NVIDIAh200sxm,
|
||||
AMDradeonV520,
|
||||
AMDmi350x,
|
||||
}
|
||||
|
||||
// GPUModelForVendorDevice returns the GPUModel for a PCI vendor/device ID pair,
|
||||
@@ -197,6 +199,29 @@ var gr6Sizes = []instanceSize{
|
||||
{"8xlarge", 32, 256},
|
||||
}
|
||||
|
||||
// g7eSizes are the G7e instance sizes (AMD Instinct MI350X).
|
||||
// 12xlarge carries 2 GPUs; sizes above that are excluded (require 4+ GPUs).
|
||||
var g7eSizes = []instanceSize{
|
||||
{"2xlarge", 8, 64},
|
||||
{"4xlarge", 16, 128},
|
||||
{"8xlarge", 32, 256},
|
||||
{"12xlarge", 48, 512},
|
||||
}
|
||||
|
||||
// gpuCountPerType overrides the default of 1 GPU for multi-GPU instance sizes.
|
||||
var gpuCountPerType = map[string]int{
|
||||
"g7e.12xlarge": 2,
|
||||
}
|
||||
|
||||
// GPUCountForType returns the number of GPUs required by the given instance type.
|
||||
// Returns 1 for all single-GPU and non-GPU types.
|
||||
func GPUCountForType(instanceType string) int {
|
||||
if n, ok := gpuCountPerType[instanceType]; ok {
|
||||
return n
|
||||
}
|
||||
return 1
|
||||
}
|
||||
|
||||
// g6eSizes are the single-GPU G6e instance sizes (1x NVIDIA L40S each).
|
||||
var g6eSizes = []instanceSize{
|
||||
{"xlarge", 4, 32},
|
||||
@@ -313,6 +338,7 @@ var instanceFamilyDefs = []instanceFamilyDef{
|
||||
{name: "g6", sizes: g6Sizes, currentGen: true},
|
||||
{name: "gr6", sizes: gr6Sizes, currentGen: true}, // same L4 GPU as g6; requires operator config (PCI ID indistinguishable)
|
||||
{name: "g6e", sizes: g6eSizes, currentGen: true},
|
||||
{name: "g7e", sizes: g7eSizes, currentGen: true},
|
||||
{name: "p3", sizes: p3Sizes, currentGen: false},
|
||||
{name: "p3dn", sizes: p3dnSizes, currentGen: false},
|
||||
{name: "p4d", sizes: p4dSizes, currentGen: true},
|
||||
|
||||
@@ -120,6 +120,7 @@ func generateGPUTypes(models []GPUModel, arch string) map[string]*ec2.InstanceTy
|
||||
}
|
||||
for _, size := range def.sizes {
|
||||
name := fmt.Sprintf("%s.%s", def.name, size.suffix)
|
||||
gpuCount := int64(GPUCountForType(name))
|
||||
types[name] = &ec2.InstanceTypeInfo{
|
||||
InstanceType: aws.String(name),
|
||||
VCpuInfo: &ec2.VCpuInfo{
|
||||
@@ -133,14 +134,14 @@ func generateGPUTypes(models []GPUModel, arch string) map[string]*ec2.InstanceTy
|
||||
},
|
||||
GpuInfo: &ec2.GpuInfo{
|
||||
Gpus: []*ec2.GpuDeviceInfo{{
|
||||
Count: aws.Int64(1),
|
||||
Count: aws.Int64(gpuCount),
|
||||
Manufacturer: aws.String(model.Manufacturer),
|
||||
Name: aws.String(model.Name),
|
||||
MemoryInfo: &ec2.GpuDeviceMemoryInfo{
|
||||
SizeInMiB: aws.Int64(model.MemoryMiB),
|
||||
},
|
||||
}},
|
||||
TotalGpuMemoryInMiB: aws.Int64(model.MemoryMiB),
|
||||
TotalGpuMemoryInMiB: aws.Int64(model.MemoryMiB * gpuCount),
|
||||
},
|
||||
CurrentGeneration: aws.Bool(def.currentGen),
|
||||
BurstablePerformanceSupported: aws.Bool(false),
|
||||
|
||||
@@ -658,7 +658,7 @@ func verifyBridgeMode(mode, externalIface, dhcpBindBridge string) error {
|
||||
}
|
||||
br, err := portToBr("veth-wan-ovs")
|
||||
if err != nil {
|
||||
return fmt.Errorf("vpcd: veth bridge mode: veth-wan-ovs not on OVS — is setup-ovn.sh's veth branch installed and systemd-networkd up? %w", err)
|
||||
return fmt.Errorf("vpcd: veth bridge mode: veth-wan-ovs not on OVS — is spinifex-veth-wan.service running? %w", err)
|
||||
}
|
||||
if br != OvnExternalBridge {
|
||||
return fmt.Errorf("vpcd: veth bridge mode: veth-wan-ovs is on OVS bridge %q, expected %q",
|
||||
@@ -666,7 +666,7 @@ func verifyBridgeMode(mode, externalIface, dhcpBindBridge string) error {
|
||||
}
|
||||
master, err := readLinkMaster("veth-wan-br")
|
||||
if err != nil {
|
||||
return fmt.Errorf("vpcd: veth bridge mode: veth-wan-br missing or has no master — systemd-networkd drop-in not applied? %w", err)
|
||||
return fmt.Errorf("vpcd: veth bridge mode: veth-wan-br missing or has no master — is spinifex-veth-wan.service running? %w", err)
|
||||
}
|
||||
if master != dhcpBindBridge {
|
||||
return fmt.Errorf("vpcd: veth bridge mode: veth-wan-br master is %q, expected %q (dhcp_bind_bridge)",
|
||||
|
||||
@@ -111,6 +111,16 @@ func (m *Manager) launch(instance *VM) error {
|
||||
qmpClient, err := newQMPClientWithHandshake(instance)
|
||||
if err != nil {
|
||||
slog.Error("Failed to create QMP client", "err", err)
|
||||
// QEMU started but QMP handshake failed. Kill it synchronously so the
|
||||
// VFIO device is released before the caller frees the GPU pool entry;
|
||||
// otherwise the next Claim gets the same PCI address while QEMU still
|
||||
// holds /dev/vfio/<group>, causing "device or resource busy".
|
||||
if pid, pidErr := utils.ReadPidFile(instance.ID); pidErr == nil && pid > 0 {
|
||||
if proc, procErr := os.FindProcess(pid); procErr == nil {
|
||||
_ = proc.Signal(syscall.SIGKILL)
|
||||
}
|
||||
_ = utils.WaitForProcessExit(pid, 10*time.Second)
|
||||
}
|
||||
return err
|
||||
}
|
||||
instance.QMPClient = qmpClient
|
||||
@@ -253,16 +263,16 @@ func (m *Manager) startQEMU(instance *VM) error {
|
||||
|
||||
instance.Config.Devices = append(instance.Config.Devices, Device{Value: "virtio-rng-pci"})
|
||||
|
||||
if instance.GPUPCIAddress != "" {
|
||||
for i, addr := range instance.GPUPCIAddresses {
|
||||
xvga := "off"
|
||||
if instance.GPUXVGAEnabled {
|
||||
xvga = "on"
|
||||
}
|
||||
instance.Config.Devices = append(instance.Config.Devices, Device{
|
||||
Value: fmt.Sprintf("vfio-pci,host=%s,id=gpu0,x-vga=%s", instance.GPUPCIAddress, xvga),
|
||||
Value: fmt.Sprintf("vfio-pci,host=%s,id=gpu%d,x-vga=%s", addr, i, xvga),
|
||||
})
|
||||
slog.Info("GPU passthrough device configured",
|
||||
"pci", instance.GPUPCIAddress, "instanceId", instance.ID, "xvga", xvga)
|
||||
"pci", addr, "index", i, "instanceId", instance.ID, "xvga", xvga)
|
||||
}
|
||||
|
||||
qmpSocket, err := utils.GenerateSocketFile(fmt.Sprintf("qmp-%s", instance.ID))
|
||||
|
||||
+11
-3
@@ -76,9 +76,10 @@ type Deps struct {
|
||||
// Manager owns the in-memory map of running VMs on this node and every
|
||||
// lifecycle transition that mutates that map.
|
||||
type Manager struct {
|
||||
mu sync.Mutex
|
||||
vms map[string]*VM
|
||||
deps Deps
|
||||
mu sync.Mutex
|
||||
vms map[string]*VM
|
||||
deps Deps
|
||||
goroutineWg sync.WaitGroup
|
||||
}
|
||||
|
||||
// NewManager returns a Manager with no collaborators wired. Production code
|
||||
@@ -259,3 +260,10 @@ func (m *Manager) Replace(vms map[string]*VM) {
|
||||
m.vms = make(map[string]*VM, len(vms))
|
||||
maps.Copy(m.vms, vms)
|
||||
}
|
||||
|
||||
// WaitForBackgroundWork blocks until all background goroutines started by
|
||||
// MarkFailed and MarkRecoveryFailed have completed. Intended for tests that
|
||||
// need the temp directory to be quiescent before cleanup runs.
|
||||
func (m *Manager) WaitForBackgroundWork() {
|
||||
m.goroutineWg.Wait()
|
||||
}
|
||||
|
||||
@@ -165,12 +165,12 @@ func (m *Manager) MarkFailed(instance *VM, reason string) {
|
||||
}
|
||||
slog.Info("Instance marked as failed", "instanceId", instance.ID, "reason", reason)
|
||||
|
||||
go func() {
|
||||
m.goroutineWg.Go(func() {
|
||||
m.terminateCleanup(instance)
|
||||
if err := m.finalizeTerminated(instance); err != nil {
|
||||
slog.Error("MarkFailed finalize failed", "instanceId", instance.ID, "err", err)
|
||||
}
|
||||
}()
|
||||
})
|
||||
}
|
||||
|
||||
// MarkRecoveryFailed handles the case where daemon-restart recovery cannot
|
||||
@@ -217,14 +217,14 @@ func (m *Manager) MarkRecoveryFailed(instance *VM, reason string) {
|
||||
slog.Error("Instance marked recovery_failed; volumes and ENIs preserved for operator action",
|
||||
"instanceId", instance.ID, "reason", reason)
|
||||
|
||||
go func() {
|
||||
m.goroutineWg.Go(func() {
|
||||
m.stopCleanup(instance)
|
||||
m.Inspect(instance, func(v *VM) { v.LastNode = m.deps.NodeID })
|
||||
if err := m.writeRunningState(); err != nil {
|
||||
slog.Error("Failed to persist state after recovery failure",
|
||||
"instanceId", instance.ID, "err", err)
|
||||
}
|
||||
}()
|
||||
})
|
||||
}
|
||||
|
||||
// finalizeTerminated transitions instance to terminated, writes the
|
||||
|
||||
@@ -416,7 +416,7 @@ func TestStopCleanup_InvokesReleaseGPU(t *testing.T) {
|
||||
t.Setenv("XDG_RUNTIME_DIR", t.TempDir())
|
||||
cleaner := &recordingInstanceCleaner{}
|
||||
m := NewManagerWithDeps(Deps{InstanceCleaner: cleaner})
|
||||
instance := &VM{ID: "i-stop", GPUPCIAddress: "0000:01:00.0"}
|
||||
instance := &VM{ID: "i-stop", GPUPCIAddresses: []string{"0000:01:00.0"}}
|
||||
|
||||
m.stopCleanup(instance)
|
||||
|
||||
@@ -433,7 +433,7 @@ func TestTerminateCleanup_InvokesReleaseGPU(t *testing.T) {
|
||||
t.Setenv("XDG_RUNTIME_DIR", t.TempDir())
|
||||
cleaner := &recordingInstanceCleaner{}
|
||||
m := NewManagerWithDeps(Deps{InstanceCleaner: cleaner})
|
||||
instance := &VM{ID: "i-term", GPUPCIAddress: "0000:01:00.0"}
|
||||
instance := &VM{ID: "i-term", GPUPCIAddresses: []string{"0000:01:00.0"}}
|
||||
|
||||
m.terminateCleanup(instance)
|
||||
|
||||
|
||||
+4
-3
@@ -118,9 +118,10 @@ type VM struct {
|
||||
// filters out tagged VMs from customer-facing listings.
|
||||
ManagedBy string `json:"managed_by,omitempty"`
|
||||
|
||||
// GPUPCIAddress is the PCI address of the GPU bound to this instance via VFIO
|
||||
// (e.g. "0000:03:00.0"). Empty for non-GPU instances.
|
||||
GPUPCIAddress string `json:"gpu_pci_address,omitempty"`
|
||||
// GPUPCIAddresses holds the PCI addresses of all GPUs bound to this instance via
|
||||
// VFIO (e.g. ["0000:03:00.0"]). Empty for non-GPU instances. Single-GPU instances
|
||||
// have one entry; multi-GPU instances have one per assigned GPU.
|
||||
GPUPCIAddresses []string `json:"gpu_pci_addresses,omitempty"`
|
||||
// GPUXVGAEnabled controls whether x-vga=on is passed to QEMU for this instance's
|
||||
// GPU device. True for consumer GPUs; false for headless datacenter cards.
|
||||
GPUXVGAEnabled bool `json:"gpu_xvga_enabled,omitempty"`
|
||||
|
||||
Reference in New Issue
Block a user