gsadmin/spinifex
1
0
Fork 0
mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-15 09:01:31 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix: links

Browse Source
This commit is contained in:
Julian Sommer
2026-09-14 12:42:58 +10:00
parent ba98509c5b
commit 5b70435bb4
9 changed files with 37 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/containers/eks/README.md
+1 -1
View File
@@ -272,4 +272,4 @@ The workbooks form a ladder. **eks-quickstart** is a NodePort demo; **eks-https-
A complete end-to-end example of Spinifex EKS in production: a GPU-accelerated AI inference platform running on a Supermicro X14 2U with two NVIDIA RTX Pro 6000 Blackwell GPUs. The guide provisions an EKS cluster with two GPU worker nodes via VFIO PCIe passthrough, deploys an OpenAI-compatible LLM API (llama.cpp / Llama 3.2 3B) and a real-time YOLO object-detection stream on separate GPU nodes, and wires them together behind an ALB using IAM, ECR, ACM, and EBS — all with standard AWS tooling and a single `AWS_PROFILE` swap.
→ [EKS AI Platform Reference Architecture](/docs/RTX-Pro-6000-EKS)
→ [EKS AI Platform Reference Architecture](/hardware/supermicro/rtx-pro-6000)
docs/hardware/cisco/llm-serving/README.md
+5 -5
View File
@@ -19,9 +19,9 @@ resources:
- title: "GPU Passthrough"
url: "/docs/gpu-passthrough"
- title: "Cisco UCS: AWS-compatible cloud at the edge (companion document)"
url: "/docs/cisco-ucs-platform-benchmark"
url: "/hardware/cisco/platform-benchmark"
- title: "Vision pipeline (companion document)"
url: "/docs/cisco-ucs-vision-pipeline"
url: "/hardware/cisco/vision-pipeline"
- title: "Benchmark and workload materials (CISCO-refarch)"
url: "https://github.com/tomnewton-mulga/CISCO-refarch"
- title: "vLLM"
@@ -45,7 +45,7 @@ cluster, comparing Intel AMX-accelerated CPU serving on an `m8i.2xlarge`-class i
against NVIDIA L4 GPU serving on a `g6.2xlarge`-class instance — a real,
production-representative LLM-serving stack, with the two accelerators measured independently at matched model, version, and serving configuration.
**Companion architectures:** [Cisco UCS: AWS-compatible cloud at the edge](/docs/cisco-ucs-platform-benchmark) · [Spinifex Vision Pipeline on Cisco UCS](/docs/cisco-ucs-vision-pipeline)
**Companion architectures:** [Cisco UCS: AWS-compatible cloud at the edge](/hardware/cisco/platform-benchmark) · [Spinifex Vision Pipeline on Cisco UCS](/hardware/cisco/vision-pipeline)
### Platform
@@ -170,9 +170,9 @@ across the full matrix. Figures below are the mean of each cell's 3 repeats (med
metric per run). Raw per-run JSON results are available in the
[LLM serving repository](https://github.com/mulgadc/cisco-ucs-llm-serving).
<img src="../../../.github/assets/images/cisco-ucs-llm-serving/tpot-vs-concurrency.png" alt="Median time per output token versus concurrency, CPU AMX versus GPU, by input length">
<img src="../../../../.github/assets/images/cisco-ucs-llm-serving/tpot-vs-concurrency.png" alt="Median time per output token versus concurrency, CPU AMX versus GPU, by input length">
<img src="../../../.github/assets/images/cisco-ucs-llm-serving/throughput-vs-concurrency.png" alt="Total token throughput versus concurrency, CPU AMX versus GPU, by input length">
<img src="../../../../.github/assets/images/cisco-ucs-llm-serving/throughput-vs-concurrency.png" alt="Total token throughput versus concurrency, CPU AMX versus GPU, by input length">
| Engine | Input tok | Concurrency | TTFT p50 (ms) | TPOT p50 (ms) | E2E p50 (ms) | Output tok/s | Total tok/s |
|---|---:|---:|---:|---:|---:|---:|---:|
docs/hardware/cisco/platform-benchmark/README.md
+4 -4
View File
@@ -39,7 +39,7 @@ EBS, S3 and EKS-compatible APIs on the cluster, so the operational model is
familiar: use the AWS CLI, SDKs, Terraform or OpenTofu, Kubernetes manifests and
CI/CD systems already used for AWS. The change is the endpoint, not the workflow.
**Companion architectures:** [Vision Pipeline on Cisco UCS](/docs/cisco-ucs-vision-pipeline) shows a local AI pipeline that streams inputs from S3; [vLLM Serving on Cisco UCS](/docs/cisco-ucs-llm-serving) shows CPU and GPU model serving on the same cluster.
**Companion architectures:** [Vision Pipeline on Cisco UCS](/hardware/cisco/vision-pipeline) shows a local AI pipeline that streams inputs from S3; [vLLM Serving on Cisco UCS](/hardware/cisco/llm-serving) shows CPU and GPU model serving on the same cluster.
### Platform
@@ -63,7 +63,7 @@ launched an instance.
### Hardware Chassis
<img src="../../../.github/assets/images/cisco-ucs-platform-benchmark/reference-architectures-cisco-ucs-platform-hardware1.png" alt="Cisco UCS x Spinifex">
<img src="../../../../.github/assets/images/cisco-ucs-platform-benchmark/reference-architectures-cisco-ucs-platform-hardware1.png" alt="Cisco UCS x Spinifex">
## Architecture
@@ -219,8 +219,8 @@ kubectl get pods -n default
```
Kubernetes pod rescheduling is the platform's primary answer to node failure for
scheduled workloads. The companion [vision pipeline](/docs/cisco-ucs-vision-pipeline)
and [vLLM serving](/docs/cisco-ucs-llm-serving) architectures use raw EC2 instances;
scheduled workloads. The companion [vision pipeline](/hardware/cisco/vision-pipeline)
and [vLLM serving](/hardware/cisco/llm-serving) architectures use raw EC2 instances;
EKS is the pattern to reach for when automatic rescheduling matters over manual placement.
### 5. Storage benchmark
docs/hardware/cisco/vision-pipeline/README.md
+5 -5
View File
@@ -25,7 +25,7 @@ resources:
- title: "GPU Passthrough"
url: "/docs/gpu-passthrough"
- title: "Cisco UCS: AWS-compatible cloud at the edge (companion document)"
url: "/docs/cisco-ucs-platform-benchmark"
url: "/hardware/cisco/platform-benchmark"
- title: "Ultralytics YOLO"
url: "https://docs.ultralytics.com"
- title: "AWS: Accelerate CPU-based AI inference with Intel AMX on EC2"
@@ -48,7 +48,7 @@ cluster running Spinifex: real-time object detection plus a vision-language mode
producing plain-English scene descriptions, both streaming from the same shared Predastore
(S3-compatible) bucket as two entirely independent EC2-compatible instances.
**Companion architectures:** [Cisco UCS: AWS-compatible cloud at the edge](/docs/cisco-ucs-platform-benchmark) · [vLLM Serving on Cisco UCS: Intel AMX vs NVIDIA L4](/docs/cisco-ucs-llm-serving)
**Companion architectures:** [Cisco UCS: AWS-compatible cloud at the edge](/hardware/cisco/platform-benchmark) · [vLLM Serving on Cisco UCS: Intel AMX vs NVIDIA L4](/hardware/cisco/llm-serving)
### Platform
@@ -164,9 +164,9 @@ measures what that kernel difference is worth on YOLO11m:
| GPU, TensorRT FP16 | 1 | 62.60 | 0.5067 |
| GPU, TensorRT FP16 | 8 | 118.90 | — |
<img src="../../../.github/assets/images/cisco-ucs-vision-pipeline/precision-throughput.png" alt="YOLO11m throughput by precision and engine — CPU FP32/BF16/INT8 versus GPU FP16, batch 1 and 8">
<img src="../../../../.github/assets/images/cisco-ucs-vision-pipeline/precision-throughput.png" alt="YOLO11m throughput by precision and engine — CPU FP32/BF16/INT8 versus GPU FP16, batch 1 and 8">
<img src="../../../.github/assets/images/cisco-ucs-vision-pipeline/precision-map.png" alt="COCO mAP50-95 by precision, axis zoomed to 0.49-0.502 to show the near-zero accuracy cost of BF16">
<img src="../../../../.github/assets/images/cisco-ucs-vision-pipeline/precision-map.png" alt="COCO mAP50-95 by precision, axis zoomed to 0.49-0.502 to show the near-zero accuracy cost of BF16">
**BF16/AMX gives a real 2.4–3.1x throughput gain over true FP32 on this hardware, at
essentially zero accuracy cost** (Δ mAP50-95 = −0.0001). INT8 costs a small but real
@@ -189,7 +189,7 @@ the instance's own local Viperblock volume:
| cpu1 (OpenVINO BF16) | 33.02 | 13.44 | −59% |
| cpu2 (OpenVINO default) | ~34.79 | 13.44 | −61% |
<img src="../../../.github/assets/images/cisco-ucs-vision-pipeline/local-vs-s3.png" alt="Local Viperblock volume versus Predastore-streamed throughput, by worker">
<img src="../../../../.github/assets/images/cisco-ucs-vision-pipeline/local-vs-s3.png" alt="Local Viperblock volume versus Predastore-streamed throughput, by worker">
A solo cpu1 S3 run (no concurrent workers) scored 12.53 img/s — essentially identical to
its 3-worker-concurrent number (13.44). **Concurrency from the other two workers cost
docs/hardware/onlogic/hx401/README.md
+1 -1
View File
@@ -33,7 +33,7 @@ The [OnLogic HX401](https://www.onlogic.com/hx401/) is a fanless, passive-cooled
Single-node removes the distributed-storage guarantees of a multi-node cluster — Predastore runs RS(1,0) on one disk, so there is no erasure-coded redundancy. For edge deployments where data is generated locally, processed on-node, and shipped upstream on a schedule, or where the box provides compute capacity next to a sensor or control network rather than acting as a primary data store, this is the expected and appropriate configuration.
<p align="center"><img src="../../../.github/assets/images/onlogic/HX401.png" alt="OnLogic HX401 fanless edge node"></p>
<p align="center"><img src="../../../../.github/assets/images/onlogic/HX401.png" alt="OnLogic HX401 fanless edge node"></p>
### Platform
docs/hardware/supermicro/arcee-trinity-mi350x/README.md
+7 -7
View File
@@ -147,7 +147,7 @@ ssh -i ~/.ssh/spinifex-key ubuntu@$INSTANCE_IP 'lspci | grep -i amd'
Or install and run `amd-smi` on the instance:
<img src="../../../.github/assets/images/2gpus.png" alt="amd-smi inside the guest VM confirming two MI350Xs are directly attached, each with a unique UUID">
<img src="../../../../.github/assets/images/2gpus.png" alt="amd-smi inside the guest VM confirming two MI350Xs are directly attached, each with a unique UUID">
Two MI350X entries with distinct UUIDs confirm direct PCIe passthrough.
@@ -267,37 +267,37 @@ The lower avg utilisation in the chat session reflects idle gaps between convers
### Generation throughput
<img src="../../../.github/assets/images/trinity/trinity-throughput.png" alt="Trinity generation throughput — scripted benchmark">
<img src="../../../../.github/assets/images/trinity/trinity-throughput.png" alt="Trinity generation throughput — scripted benchmark">
Generation and prompt-ingest tok/s over the 5-minute benchmark. The model sustains ~53 tok/s average at continuous load, peaking at 67 tok/s.
### Per-GPU compute utilisation
<img src="../../../.github/assets/images/trinity/trinity-per-gpu-util.png" alt="Trinity per-GPU utilisation — scripted benchmark">
<img src="../../../../.github/assets/images/trinity/trinity-per-gpu-util.png" alt="Trinity per-GPU utilisation — scripted benchmark">
GPU 0 and GPU 1 track each other closely throughout the benchmark — TP=2 distributes attention and FFN layers symmetrically across both MI350Xs.
### Per-GPU VRAM
<img src="../../../.github/assets/images/trinity/trinity-per-gpu-vram.png" alt="Trinity per-GPU VRAM — scripted benchmark">
<img src="../../../../.github/assets/images/trinity/trinity-per-gpu-vram.png" alt="Trinity per-GPU VRAM — scripted benchmark">
Both GPUs hold 259.6 GB — model weights sharded evenly, plus KV cache. 90.2% of HBM3e is occupied at rest, leaving ~26 GB headroom per GPU for active KV cache during generation.
### Per-GPU power
<img src="../../../.github/assets/images/trinity/trinity-per-gpu-power.png" alt="Trinity per-GPU power — scripted benchmark">
<img src="../../../../.github/assets/images/trinity/trinity-per-gpu-power.png" alt="Trinity per-GPU power — scripted benchmark">
Power draw during active generation peaks at ~420 W per GPU, for a combined chassis draw well under the MI350X's 750 W TDP per card. Idle between prompts returns to ~300 W.
### Latency
<img src="../../../.github/assets/images/trinity/trinity-latency.png" alt="Trinity latency — scripted benchmark">
<img src="../../../../.github/assets/images/trinity/trinity-latency.png" alt="Trinity latency — scripted benchmark">
Cumulative average TTFT and E2E latency over the benchmark. TTFT settles to ~758 ms avg once the KV cache is warm; E2E tracks request length as expected.
### KV cache utilisation
<img src="../../../.github/assets/images/trinity/trinity-kv-cache.png" alt="Trinity KV cache utilisation — scripted benchmark">
<img src="../../../../.github/assets/images/trinity/trinity-kv-cache.png" alt="Trinity KV cache utilisation — scripted benchmark">
KV cache occupancy rises as requests pile up during the sustained load phase. The queue depth stays near zero — the model keeps pace with the 1.5 s prompt cadence.
docs/hardware/supermicro/rtx-pro-6000/README.md
+2 -2
View File
@@ -39,7 +39,7 @@ This guide walks through the use of Spinifex to deploy a self-contained AI infer
Specifically, we create an EKS cluster with two worker nodes, each consisting of a g7e.2xlarge EC2 instance with an attached GPU via VFIO passthrough, an ALB to route traffic to each node, ECR for storing and managing our workload images, and all of the associated security and certificate management requirements (IAM, ACM) you would expect from real AWS.
<p align="center"><img src="../../../.github/assets/images/rtx6000/platform.png" alt="AI Platform request flow: HTTPS ingress → ALB routing → EKS ai-platform GPU workers"></p>
<p align="center"><img src="../../../../.github/assets/images/rtx6000/platform.png" alt="AI Platform request flow: HTTPS ingress → ALB routing → EKS ai-platform GPU workers"></p>
### Platform
@@ -70,7 +70,7 @@ In this case, the two workloads demonstrated could both run comfortably on a sin
## Architecture
<p align="center"><img src="../../../.github/assets/images/rtx6000/ai-platform-request-flow-2.png" alt="AI Platform request flow: HTTPS ingress → ALB routing → EKS ai-platform GPU workers"></p>
<p align="center"><img src="../../../../.github/assets/images/rtx6000/ai-platform-request-flow-2.png" alt="AI Platform request flow: HTTPS ingress → ALB routing → EKS ai-platform GPU workers"></p>
### AWS services exercised
docs/hardware/supermicro/smci-h14/README.md
+8 -8
View File
@@ -76,7 +76,7 @@ sudo spx admin gpu enable
`spx admin gpu setup` blacklists the AMD driver on the host and binds each GPU to `vfio-pci`. After reboot, `spx admin gpu enable` verifies the binding and makes the GPU pool available to `RunInstances`.
<img src="../../../.github/assets/images/gpu-enabled.png" alt="Spinifex console confirming PCIe passthrough is active for each GPU">
<img src="../../../../.github/assets/images/gpu-enabled.png" alt="Spinifex console confirming PCIe passthrough is active for each GPU">
To check available GPU instance types:
@@ -177,7 +177,7 @@ ssh -i ~/.ssh/spinifex-key ubuntu@$CHAT_IP \
amd-smi list'
```
<img src="../../../.github/assets/images/2gpus.png" alt="amd-smi inside vm-chat confirming two MI350Xs are directly attached, each with a unique UUID">
<img src="../../../../.github/assets/images/2gpus.png" alt="amd-smi inside vm-chat confirming two MI350Xs are directly attached, each with a unique UUID">
Two MI350X entries with distinct UUIDs confirm direct PCIe passthrough is working.
@@ -216,11 +216,11 @@ The only difference between the two runs was tensor parallelism:
The charts below show the three independent workloads spiking activity on their associated GPUs, measured in GFX utilisation and power draw. YOLO average FPS stays at ~17 despite utilisation in the YOLO VM never exceeding 20%, indicating a bottleneck in networking or encoding rather than inference. Each chart highlights workload independence — pulses of GPU activity share no correlation across VMs.
<img src="../../../.github/assets/images/smci/smci-demo1-gpu-utilisation.png" alt="GPU utilisation across all three VMs — Demo 1 TP=1">
<img src="../../../../.github/assets/images/smci/smci-demo1-gpu-utilisation.png" alt="GPU utilisation across all three VMs — Demo 1 TP=1">
GPU utilisation per VM across the full Demo 1 capture.
<img src="../../../.github/assets/images/smci/smci-demo2-gpu-power.png" alt="GPU power draw per workload — Demo 2 TP=2">
<img src="../../../../.github/assets/images/smci/smci-demo2-gpu-power.png" alt="GPU power draw per workload — Demo 2 TP=2">
Power draw per VM during Demo 2.
@@ -228,11 +228,11 @@ Power draw per VM during Demo 2.
When TP=2 is enabled, each VM shards its model symmetrically across both GPUs. Inside the chat VM, both GPUs pulse together on every request, each holding exactly **248 GB of VRAM** (weights split + KV cache shards). With TP=1, GPU 1 sat at 0% for the entire run.
<img src="../../../.github/assets/images/smci/smci-demo2-per-gpu-chat.png" alt="Llama 3.3 70B per-GPU activity — TP=2">
<img src="../../../../.github/assets/images/smci/smci-demo2-per-gpu-chat.png" alt="Llama 3.3 70B per-GPU activity — TP=2">
Per-GPU utilisation inside `vm-chat` with TP=2.
<img src="../../../.github/assets/images/smci/smci-demo1-per-gpu-chat.png" alt="Llama 3.3 70B per-GPU activity — TP=1">
<img src="../../../../.github/assets/images/smci/smci-demo1-per-gpu-chat.png" alt="Llama 3.3 70B per-GPU activity — TP=1">
Same VM with TP=1.
@@ -240,11 +240,11 @@ Same VM with TP=1.
With TP=1, Qwen peaks at ~60 tok/s and Llama caps around 35 tok/s — both compute-bound on a single MI350X. Enabling TP=2 lifts Llama's peak to ~50 tok/s (+47%) as generation is distributed symmetrically across both GPUs. Qwen's throughput is slightly lower under TP=2 due to collective overhead on the shared-memory transport.
<img src="../../../.github/assets/images/smci/smci-demo1-vllm-throughput.png" alt="vLLM generation throughput — TP=1">
<img src="../../../../.github/assets/images/smci/smci-demo1-vllm-throughput.png" alt="vLLM generation throughput — TP=1">
Generation throughput during Demo 1 (TP=1).
<img src="../../../.github/assets/images/smci/smci-demo2-vllm-throughput.png" alt="vLLM generation throughput — TP=2">
<img src="../../../../.github/assets/images/smci/smci-demo2-vllm-throughput.png" alt="vLLM generation throughput — TP=2">
Generation throughput during Demo 2 (TP=2).
docs/hardware/supermicro/smci-mig/README.md
+4 -4
View File
@@ -30,7 +30,7 @@ Spinifex is an open-source infrastructure platform that brings core AWS services
This guide documents a full bare-metal AI deployment on a single [Supermicro X13](https://www.supermicro.com/en/products/x13) chassis using NVIDIA H200 SXM GPUs. These GPUs include the [NVIDIA Multi-Instance GPU](https://www.nvidia.com/en-au/technologies/multi-instance-gpu/) (MIG) capability, which allows a single GPU to be "sliced" into up to seven independent GPU partitions that are hardware isolated. Each partition is capable of running its own workloads with reserved resources from the "host" GPU.
<p align="center"><img src="../../../.github/assets/images/h200/X13.png" alt="Supermicro X13 8U GPU System"></p>
<p align="center"><img src="../../../../.github/assets/images/h200/X13.png" alt="Supermicro X13 8U GPU System"></p>
In this setup, each EC2 instance receives an entire H200 via PCIe passthrough and manages its own MIG partitions. This gives each tenant full control over how they slice their GPU — including the ability to run heterogeneous workloads at different partition sizes on the same physical card.
@@ -72,7 +72,7 @@ Twelve concurrent inference endpoints in total: 7 fast 3B slots, 2 mid-tier 32B
### 1. Verify GPU visibility on host
Before making any changes, verify that the host can see its GPUs:
<img src="../../../.github/assets/images/h200/nvidia-smi.png" alt="NVIDIA SMI">
<img src="../../../../.github/assets/images/h200/nvidia-smi.png" alt="NVIDIA SMI">
## Instructions
@@ -183,7 +183,7 @@ aws s3 ls --endpoint-url https://localhost:8443
### 5. Enable GPU Passthrough
Spinifex allows GPUs to be utilised by guest VMS via PCIe-passthrough. This can be enabled via `sudo spx admin gpu setup`. Then, after a reboot, run `sudo spx admin gpu enable`. The Spinifex banner should update to reflect GPU passthrough state after every step:
<p align="center"><img src="../../../.github/assets/images/h200/spinifex-banner1.png" alt="GPU Enabled"></p>
<p align="center"><img src="../../../../.github/assets/images/h200/spinifex-banner1.png" alt="GPU Enabled"></p>
### 6. Import the GPU AMI
@@ -281,7 +281,7 @@ nvidia-smi | grep "MIG M."
With MIG enabled, we are now able to partition each VM's assigned H200 GPU into several separate GPU instances. This process assigns each GPU slice its own UUID, so each VM goes from seeing one whole GPU to seeing a number of "MIG devices":
<p align="center"><img src="../../../.github/assets/images/h200/mig-activated.png" alt="NVIDIA SMI"></p>
<p align="center"><img src="../../../../.github/assets/images/h200/mig-activated.png" alt="NVIDIA SMI"></p>
This allows us to run several separate workloads, each assigned to its own GPU instance, on the same physical GPU, and thus utilise more of the overall GPU's resources.