diff --git a/docs/containers/eks/README.md b/docs/containers/eks/README.md index e15d94664..d8bbb06b8 100644 --- a/docs/containers/eks/README.md +++ b/docs/containers/eks/README.md @@ -272,4 +272,4 @@ The workbooks form a ladder. **eks-quickstart** is a NodePort demo; **eks-https- A complete end-to-end example of Spinifex EKS in production: a GPU-accelerated AI inference platform running on a Supermicro X14 2U with two NVIDIA RTX Pro 6000 Blackwell GPUs. The guide provisions an EKS cluster with two GPU worker nodes via VFIO PCIe passthrough, deploys an OpenAI-compatible LLM API (llama.cpp / Llama 3.2 3B) and a real-time YOLO object-detection stream on separate GPU nodes, and wires them together behind an ALB using IAM, ECR, ACM, and EBS — all with standard AWS tooling and a single `AWS_PROFILE` swap. -→ [EKS AI Platform Reference Architecture](/docs/RTX-Pro-6000-EKS) +→ [EKS AI Platform Reference Architecture](/hardware/supermicro/rtx-pro-6000) diff --git a/docs/hardware/cisco/llm-serving/README.md b/docs/hardware/cisco/llm-serving/README.md index 34d860956..257b013f5 100644 --- a/docs/hardware/cisco/llm-serving/README.md +++ b/docs/hardware/cisco/llm-serving/README.md @@ -19,9 +19,9 @@ resources: - title: "GPU Passthrough" url: "/docs/gpu-passthrough" - title: "Cisco UCS: AWS-compatible cloud at the edge (companion document)" - url: "/docs/cisco-ucs-platform-benchmark" + url: "/hardware/cisco/platform-benchmark" - title: "Vision pipeline (companion document)" - url: "/docs/cisco-ucs-vision-pipeline" + url: "/hardware/cisco/vision-pipeline" - title: "Benchmark and workload materials (CISCO-refarch)" url: "https://github.com/tomnewton-mulga/CISCO-refarch" - title: "vLLM" @@ -45,7 +45,7 @@ cluster, comparing Intel AMX-accelerated CPU serving on an `m8i.2xlarge`-class i against NVIDIA L4 GPU serving on a `g6.2xlarge`-class instance — a real, production-representative LLM-serving stack, with the two accelerators measured independently at matched model, version, and serving configuration. -**Companion architectures:** [Cisco UCS: AWS-compatible cloud at the edge](/docs/cisco-ucs-platform-benchmark) · [Spinifex Vision Pipeline on Cisco UCS](/docs/cisco-ucs-vision-pipeline) +**Companion architectures:** [Cisco UCS: AWS-compatible cloud at the edge](/hardware/cisco/platform-benchmark) · [Spinifex Vision Pipeline on Cisco UCS](/hardware/cisco/vision-pipeline) ### Platform @@ -170,9 +170,9 @@ across the full matrix. Figures below are the mean of each cell's 3 repeats (med metric per run). Raw per-run JSON results are available in the [LLM serving repository](https://github.com/mulgadc/cisco-ucs-llm-serving). -Median time per output token versus concurrency, CPU AMX versus GPU, by input length +Median time per output token versus concurrency, CPU AMX versus GPU, by input length -Total token throughput versus concurrency, CPU AMX versus GPU, by input length +Total token throughput versus concurrency, CPU AMX versus GPU, by input length | Engine | Input tok | Concurrency | TTFT p50 (ms) | TPOT p50 (ms) | E2E p50 (ms) | Output tok/s | Total tok/s | |---|---:|---:|---:|---:|---:|---:|---:| diff --git a/docs/hardware/cisco/platform-benchmark/README.md b/docs/hardware/cisco/platform-benchmark/README.md index 77aa3cfdd..e8c0ced38 100644 --- a/docs/hardware/cisco/platform-benchmark/README.md +++ b/docs/hardware/cisco/platform-benchmark/README.md @@ -39,7 +39,7 @@ EBS, S3 and EKS-compatible APIs on the cluster, so the operational model is familiar: use the AWS CLI, SDKs, Terraform or OpenTofu, Kubernetes manifests and CI/CD systems already used for AWS. The change is the endpoint, not the workflow. -**Companion architectures:** [Vision Pipeline on Cisco UCS](/docs/cisco-ucs-vision-pipeline) shows a local AI pipeline that streams inputs from S3; [vLLM Serving on Cisco UCS](/docs/cisco-ucs-llm-serving) shows CPU and GPU model serving on the same cluster. +**Companion architectures:** [Vision Pipeline on Cisco UCS](/hardware/cisco/vision-pipeline) shows a local AI pipeline that streams inputs from S3; [vLLM Serving on Cisco UCS](/hardware/cisco/llm-serving) shows CPU and GPU model serving on the same cluster. ### Platform @@ -63,7 +63,7 @@ launched an instance. ### Hardware Chassis -Cisco UCS x Spinifex +Cisco UCS x Spinifex ## Architecture @@ -219,8 +219,8 @@ kubectl get pods -n default ``` Kubernetes pod rescheduling is the platform's primary answer to node failure for -scheduled workloads. The companion [vision pipeline](/docs/cisco-ucs-vision-pipeline) -and [vLLM serving](/docs/cisco-ucs-llm-serving) architectures use raw EC2 instances; +scheduled workloads. The companion [vision pipeline](/hardware/cisco/vision-pipeline) +and [vLLM serving](/hardware/cisco/llm-serving) architectures use raw EC2 instances; EKS is the pattern to reach for when automatic rescheduling matters over manual placement. ### 5. Storage benchmark diff --git a/docs/hardware/cisco/vision-pipeline/README.md b/docs/hardware/cisco/vision-pipeline/README.md index ac231a2d7..edb5a92b8 100644 --- a/docs/hardware/cisco/vision-pipeline/README.md +++ b/docs/hardware/cisco/vision-pipeline/README.md @@ -25,7 +25,7 @@ resources: - title: "GPU Passthrough" url: "/docs/gpu-passthrough" - title: "Cisco UCS: AWS-compatible cloud at the edge (companion document)" - url: "/docs/cisco-ucs-platform-benchmark" + url: "/hardware/cisco/platform-benchmark" - title: "Ultralytics YOLO" url: "https://docs.ultralytics.com" - title: "AWS: Accelerate CPU-based AI inference with Intel AMX on EC2" @@ -48,7 +48,7 @@ cluster running Spinifex: real-time object detection plus a vision-language mode producing plain-English scene descriptions, both streaming from the same shared Predastore (S3-compatible) bucket as two entirely independent EC2-compatible instances. -**Companion architectures:** [Cisco UCS: AWS-compatible cloud at the edge](/docs/cisco-ucs-platform-benchmark) · [vLLM Serving on Cisco UCS: Intel AMX vs NVIDIA L4](/docs/cisco-ucs-llm-serving) +**Companion architectures:** [Cisco UCS: AWS-compatible cloud at the edge](/hardware/cisco/platform-benchmark) · [vLLM Serving on Cisco UCS: Intel AMX vs NVIDIA L4](/hardware/cisco/llm-serving) ### Platform @@ -164,9 +164,9 @@ measures what that kernel difference is worth on YOLO11m: | GPU, TensorRT FP16 | 1 | 62.60 | 0.5067 | | GPU, TensorRT FP16 | 8 | 118.90 | — | -YOLO11m throughput by precision and engine — CPU FP32/BF16/INT8 versus GPU FP16, batch 1 and 8 +YOLO11m throughput by precision and engine — CPU FP32/BF16/INT8 versus GPU FP16, batch 1 and 8 -COCO mAP50-95 by precision, axis zoomed to 0.49-0.502 to show the near-zero accuracy cost of BF16 +COCO mAP50-95 by precision, axis zoomed to 0.49-0.502 to show the near-zero accuracy cost of BF16 **BF16/AMX gives a real 2.4–3.1x throughput gain over true FP32 on this hardware, at essentially zero accuracy cost** (Δ mAP50-95 = −0.0001). INT8 costs a small but real @@ -189,7 +189,7 @@ the instance's own local Viperblock volume: | cpu1 (OpenVINO BF16) | 33.02 | 13.44 | −59% | | cpu2 (OpenVINO default) | ~34.79 | 13.44 | −61% | -Local Viperblock volume versus Predastore-streamed throughput, by worker +Local Viperblock volume versus Predastore-streamed throughput, by worker A solo cpu1 S3 run (no concurrent workers) scored 12.53 img/s — essentially identical to its 3-worker-concurrent number (13.44). **Concurrency from the other two workers cost diff --git a/docs/hardware/onlogic/hx401/README.md b/docs/hardware/onlogic/hx401/README.md index e0ef0f12f..e9c7ec8f7 100644 --- a/docs/hardware/onlogic/hx401/README.md +++ b/docs/hardware/onlogic/hx401/README.md @@ -33,7 +33,7 @@ The [OnLogic HX401](https://www.onlogic.com/hx401/) is a fanless, passive-cooled Single-node removes the distributed-storage guarantees of a multi-node cluster — Predastore runs RS(1,0) on one disk, so there is no erasure-coded redundancy. For edge deployments where data is generated locally, processed on-node, and shipped upstream on a schedule, or where the box provides compute capacity next to a sensor or control network rather than acting as a primary data store, this is the expected and appropriate configuration. -

OnLogic HX401 fanless edge node

+

OnLogic HX401 fanless edge node

### Platform diff --git a/docs/hardware/supermicro/arcee-trinity-mi350x/README.md b/docs/hardware/supermicro/arcee-trinity-mi350x/README.md index 27c7498f2..c9829cf3d 100644 --- a/docs/hardware/supermicro/arcee-trinity-mi350x/README.md +++ b/docs/hardware/supermicro/arcee-trinity-mi350x/README.md @@ -147,7 +147,7 @@ ssh -i ~/.ssh/spinifex-key ubuntu@$INSTANCE_IP 'lspci | grep -i amd' Or install and run `amd-smi` on the instance: -amd-smi inside the guest VM confirming two MI350Xs are directly attached, each with a unique UUID +amd-smi inside the guest VM confirming two MI350Xs are directly attached, each with a unique UUID Two MI350X entries with distinct UUIDs confirm direct PCIe passthrough. @@ -267,37 +267,37 @@ The lower avg utilisation in the chat session reflects idle gaps between convers ### Generation throughput -Trinity generation throughput — scripted benchmark +Trinity generation throughput — scripted benchmark Generation and prompt-ingest tok/s over the 5-minute benchmark. The model sustains ~53 tok/s average at continuous load, peaking at 67 tok/s. ### Per-GPU compute utilisation -Trinity per-GPU utilisation — scripted benchmark +Trinity per-GPU utilisation — scripted benchmark GPU 0 and GPU 1 track each other closely throughout the benchmark — TP=2 distributes attention and FFN layers symmetrically across both MI350Xs. ### Per-GPU VRAM -Trinity per-GPU VRAM — scripted benchmark +Trinity per-GPU VRAM — scripted benchmark Both GPUs hold 259.6 GB — model weights sharded evenly, plus KV cache. 90.2% of HBM3e is occupied at rest, leaving ~26 GB headroom per GPU for active KV cache during generation. ### Per-GPU power -Trinity per-GPU power — scripted benchmark +Trinity per-GPU power — scripted benchmark Power draw during active generation peaks at ~420 W per GPU, for a combined chassis draw well under the MI350X's 750 W TDP per card. Idle between prompts returns to ~300 W. ### Latency -Trinity latency — scripted benchmark +Trinity latency — scripted benchmark Cumulative average TTFT and E2E latency over the benchmark. TTFT settles to ~758 ms avg once the KV cache is warm; E2E tracks request length as expected. ### KV cache utilisation -Trinity KV cache utilisation — scripted benchmark +Trinity KV cache utilisation — scripted benchmark KV cache occupancy rises as requests pile up during the sustained load phase. The queue depth stays near zero — the model keeps pace with the 1.5 s prompt cadence. diff --git a/docs/hardware/supermicro/rtx-pro-6000/README.md b/docs/hardware/supermicro/rtx-pro-6000/README.md index 50c1b8627..39b351bbe 100644 --- a/docs/hardware/supermicro/rtx-pro-6000/README.md +++ b/docs/hardware/supermicro/rtx-pro-6000/README.md @@ -39,7 +39,7 @@ This guide walks through the use of Spinifex to deploy a self-contained AI infer Specifically, we create an EKS cluster with two worker nodes, each consisting of a g7e.2xlarge EC2 instance with an attached GPU via VFIO passthrough, an ALB to route traffic to each node, ECR for storing and managing our workload images, and all of the associated security and certificate management requirements (IAM, ACM) you would expect from real AWS. -

AI Platform request flow: HTTPS ingress → ALB routing → EKS ai-platform GPU workers

+

AI Platform request flow: HTTPS ingress → ALB routing → EKS ai-platform GPU workers

### Platform @@ -70,7 +70,7 @@ In this case, the two workloads demonstrated could both run comfortably on a sin ## Architecture -

AI Platform request flow: HTTPS ingress → ALB routing → EKS ai-platform GPU workers

+

AI Platform request flow: HTTPS ingress → ALB routing → EKS ai-platform GPU workers

### AWS services exercised diff --git a/docs/hardware/supermicro/smci-h14/README.md b/docs/hardware/supermicro/smci-h14/README.md index e91781a49..21a023a71 100644 --- a/docs/hardware/supermicro/smci-h14/README.md +++ b/docs/hardware/supermicro/smci-h14/README.md @@ -76,7 +76,7 @@ sudo spx admin gpu enable `spx admin gpu setup` blacklists the AMD driver on the host and binds each GPU to `vfio-pci`. After reboot, `spx admin gpu enable` verifies the binding and makes the GPU pool available to `RunInstances`. -Spinifex console confirming PCIe passthrough is active for each GPU +Spinifex console confirming PCIe passthrough is active for each GPU To check available GPU instance types: @@ -177,7 +177,7 @@ ssh -i ~/.ssh/spinifex-key ubuntu@$CHAT_IP \ amd-smi list' ``` -amd-smi inside vm-chat confirming two MI350Xs are directly attached, each with a unique UUID +amd-smi inside vm-chat confirming two MI350Xs are directly attached, each with a unique UUID Two MI350X entries with distinct UUIDs confirm direct PCIe passthrough is working. @@ -216,11 +216,11 @@ The only difference between the two runs was tensor parallelism: The charts below show the three independent workloads spiking activity on their associated GPUs, measured in GFX utilisation and power draw. YOLO average FPS stays at ~17 despite utilisation in the YOLO VM never exceeding 20%, indicating a bottleneck in networking or encoding rather than inference. Each chart highlights workload independence — pulses of GPU activity share no correlation across VMs. -GPU utilisation across all three VMs — Demo 1 TP=1 +GPU utilisation across all three VMs — Demo 1 TP=1 GPU utilisation per VM across the full Demo 1 capture. -GPU power draw per workload — Demo 2 TP=2 +GPU power draw per workload — Demo 2 TP=2 Power draw per VM during Demo 2. @@ -228,11 +228,11 @@ Power draw per VM during Demo 2. When TP=2 is enabled, each VM shards its model symmetrically across both GPUs. Inside the chat VM, both GPUs pulse together on every request, each holding exactly **248 GB of VRAM** (weights split + KV cache shards). With TP=1, GPU 1 sat at 0% for the entire run. -Llama 3.3 70B per-GPU activity — TP=2 +Llama 3.3 70B per-GPU activity — TP=2 Per-GPU utilisation inside `vm-chat` with TP=2. -Llama 3.3 70B per-GPU activity — TP=1 +Llama 3.3 70B per-GPU activity — TP=1 Same VM with TP=1. @@ -240,11 +240,11 @@ Same VM with TP=1. With TP=1, Qwen peaks at ~60 tok/s and Llama caps around 35 tok/s — both compute-bound on a single MI350X. Enabling TP=2 lifts Llama's peak to ~50 tok/s (+47%) as generation is distributed symmetrically across both GPUs. Qwen's throughput is slightly lower under TP=2 due to collective overhead on the shared-memory transport. -vLLM generation throughput — TP=1 +vLLM generation throughput — TP=1 Generation throughput during Demo 1 (TP=1). -vLLM generation throughput — TP=2 +vLLM generation throughput — TP=2 Generation throughput during Demo 2 (TP=2). diff --git a/docs/hardware/supermicro/smci-mig/README.md b/docs/hardware/supermicro/smci-mig/README.md index 15ab0a00f..0af8279d9 100644 --- a/docs/hardware/supermicro/smci-mig/README.md +++ b/docs/hardware/supermicro/smci-mig/README.md @@ -30,7 +30,7 @@ Spinifex is an open-source infrastructure platform that brings core AWS services This guide documents a full bare-metal AI deployment on a single [Supermicro X13](https://www.supermicro.com/en/products/x13) chassis using NVIDIA H200 SXM GPUs. These GPUs include the [NVIDIA Multi-Instance GPU](https://www.nvidia.com/en-au/technologies/multi-instance-gpu/) (MIG) capability, which allows a single GPU to be "sliced" into up to seven independent GPU partitions that are hardware isolated. Each partition is capable of running its own workloads with reserved resources from the "host" GPU. -

Supermicro X13 8U GPU System

+

Supermicro X13 8U GPU System

In this setup, each EC2 instance receives an entire H200 via PCIe passthrough and manages its own MIG partitions. This gives each tenant full control over how they slice their GPU — including the ability to run heterogeneous workloads at different partition sizes on the same physical card. @@ -72,7 +72,7 @@ Twelve concurrent inference endpoints in total: 7 fast 3B slots, 2 mid-tier 32B ### 1. Verify GPU visibility on host Before making any changes, verify that the host can see its GPUs: -NVIDIA SMI +NVIDIA SMI ## Instructions @@ -183,7 +183,7 @@ aws s3 ls --endpoint-url https://localhost:8443 ### 5. Enable GPU Passthrough Spinifex allows GPUs to be utilised by guest VMS via PCIe-passthrough. This can be enabled via `sudo spx admin gpu setup`. Then, after a reboot, run `sudo spx admin gpu enable`. The Spinifex banner should update to reflect GPU passthrough state after every step: -

GPU Enabled

+

GPU Enabled

### 6. Import the GPU AMI @@ -281,7 +281,7 @@ nvidia-smi | grep "MIG M." With MIG enabled, we are now able to partition each VM's assigned H200 GPU into several separate GPU instances. This process assigns each GPU slice its own UUID, so each VM goes from seeing one whole GPU to seeing a number of "MIG devices": -

NVIDIA SMI

+

NVIDIA SMI

This allows us to run several separate workloads, each assigned to its own GPU instance, on the same physical GPU, and thus utilise more of the overall GPU's resources.