mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-16 01:22:00 +00:00
fix: Rewrite Cisco and OnLogic docs
This commit is contained in:
tomnewton-mulga
@@ -1,23 +1,26 @@
|
||||
---
|
||||
title: "Single-Node Spinifex on OnLogic HX401"
|
||||
seoTitle: "Single-Node Spinifex on OnLogic HX401 — Spinifex Docs"
|
||||
description: "Run a complete EC2, EBS, and S3 stack on one fanless OnLogic HX401 edge node, and measure what the platform layer costs compared with bare-metal baselines."
|
||||
description: "EC2, EBS and S3 on a single fanless OnLogic HX401 — the full Spinifex service set on a passive-cooled industrial node for factory and remote edge deployments."
|
||||
category: "Reference Architectures"
|
||||
tags:
|
||||
- onlogic
|
||||
- hx401
|
||||
- edge
|
||||
- single-node
|
||||
- benchmarks
|
||||
- viperblock
|
||||
- predastore
|
||||
resources:
|
||||
- title: "Installation"
|
||||
- title: "Single-Node Install"
|
||||
url: "/docs/install"
|
||||
- title: "VPC Networking"
|
||||
url: "/docs/vpc-networking"
|
||||
- title: "Launching Instances"
|
||||
url: "/docs/launching-instances"
|
||||
- title: "OnLogic HX401"
|
||||
url: "https://www.onlogic.com/hx401/"
|
||||
- title: "Benchmark data and platform notes (OnLogic-refarch)"
|
||||
url: "https://github.com/tomnewton-mulga/OnLogic-refarch"
|
||||
sections:
|
||||
- overview
|
||||
- prerequisites
|
||||
@@ -26,11 +29,9 @@ sections:
|
||||
|
||||
## Overview
|
||||
|
||||
Spinifex is an open-source infrastructure platform that brings core AWS services — EC2, EBS and S3 — to bare-metal, edge and on-prem environments. It exposes an EC2-compatible API, so tooling that works against AWS (the `aws` CLI, Terraform, SDKs) works against a Spinifex node unchanged, with a single profile swap.
|
||||
The [OnLogic HX401](https://www.onlogic.com/hx401/) is a fanless, passive-cooled industrial edge computer that runs the complete Spinifex service set — EC2, EBS, S3 and VPC networking — in a form factor small enough to panel-mount inside a factory cabinet, attach to a DIN rail in a control room, or deploy unattended at a remote site without active cooling or rack infrastructure. Spinifex exposes EC2, EBS and S3-compatible APIs on the node, so the operational model is unchanged from AWS: use the `aws` CLI, SDKs, Terraform or OpenTofu, and CI/CD pipelines already used for cloud workloads. The change is the endpoint, not the workflow.
|
||||
|
||||
This document covers a node deployment of Spinifex on the OnLogic HX401. The OnLogic HX401 is a fanless industrial box with 12 physical cores and 31 GiB of RAM — enough to run the full service set and still host a meaningful guest.
|
||||
|
||||
The goal of this deployment is to measure what the platform costs against bare metal — CPU overhead from virtualisation, guest block storage throughput, and S3 data-path performance. The same benchmark suite was run against both Spinifex v1.15.0 and v1.16.0 on the same hardware, to help the team quantify the efficacy of the new releases's fixes and improvements.
|
||||
Single-node removes the distributed-storage guarantees of a multi-node cluster — Predastore runs RS(1,0) on one disk, so there is no erasure-coded redundancy. For edge deployments where data is generated locally, processed on-node, and shipped upstream on a schedule, or where the box provides compute capacity next to a sensor or control network rather than acting as a primary data store, this is the expected and appropriate configuration.
|
||||
|
||||
<p align="center"><img src="../../../.github/assets/images/onlogic/HX401.png" alt="OnLogic HX401 fanless edge node"></p>
|
||||
|
||||
@@ -38,25 +39,33 @@ The goal of this deployment is to measure what the platform costs against bare m
|
||||
|
||||
| Component | Specification |
|
||||
|---|---|
|
||||
| **Bare-metal host** | OnLogic HX401 |
|
||||
| **Bare-metal host** | [OnLogic HX401](https://www.onlogic.com/hx401/) — fanless, passive-cooled industrial edge computer |
|
||||
| **CPU** | Intel i5-1250PE — 12 physical cores / 16 threads |
|
||||
| **Memory** | 31 GiB |
|
||||
| **Storage** | Transcend TS256GMTE652T2, 256 GB NVMe (PCIe Gen3 x4) |
|
||||
| **Networking** | 2× Intel GbE (I210-IT + I219-LM) |
|
||||
| **Host OS** | Debian 13 |
|
||||
| **Orchestration** | Spinifex v1.16.0 — EC2-compatible bare-metal API |
|
||||
| **Guest OS** | Ubuntu 26.04 LTS |
|
||||
| **Instance type** | `c6i.2xlarge` — 8 vCPU, 16 GiB |
|
||||
| **Block storage** | Viperblock — EBS-compatible, local NVMe-backed |
|
||||
| **Object storage** | Predastore — S3-compatible, RS(1,0) on a single node |
|
||||
| **Networking** | OVN, standalone NB/SB on the node itself |
|
||||
| **Form factor** | Fanless, passive-cooled; DIN rail, wall and VESA mountable |
|
||||
|
||||
## Architecture
|
||||
|
||||
### AWS services exercised
|
||||
|
||||
| Service | Role |
|
||||
|---|---|
|
||||
| **EC2** | Guest instances on the host NVMe; no spread placement groups on a single node |
|
||||
| **EBS (Viperblock)** | Local NVMe-backed block volumes per instance, surviving termination independently when `delete_on_termination = false` |
|
||||
| **S3 (Predastore)** | RS(1,0) single-node object storage — no erasure-coding overhead, no distributed redundancy |
|
||||
| **VPC (OVN)** | Standalone OVN northbound/southbound databases on the node itself |
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- OnLogic HX401 (or comparable x86 node) running **Debian 13** or **Ubuntu 26.04 LTS**
|
||||
- The WAN interface enslaved to a Linux bridge named `br-wan` — the host IP, default route and DHCP must live on the bridge, not the bare NIC
|
||||
- A reserved range of addresses on your LAN for guest instances (this deployment uses `192.168.157.201-250`)
|
||||
- AWS CLI installed
|
||||
- A reserved range of addresses on your LAN for guest instances — this deployment uses `192.168.157.201-250`
|
||||
- AWS CLI configured with `AWS_PROFILE=spinifex` pointing at the node's EC2-compatible endpoint (`https://<host>:9999`)
|
||||
|
||||
Verify the bridge before installing:
|
||||
|
||||
@@ -67,11 +76,11 @@ ip route
|
||||
|
||||
## Instructions
|
||||
|
||||
### 1. Install Spinifex and initialise the node
|
||||
### 1. Install Spinifex and verify the node
|
||||
|
||||
Follow the [Single Node Install](/docs/install) guide, using `--nodes 1` to select the single-node templates — RS(1,0) storage and a one-member NATS.
|
||||
Follow the [Single-Node Install](/docs/install) guide, using `--nodes 1` to select the single-node templates — RS(1,0) storage and a single-member NATS cluster.
|
||||
|
||||
**Set a static external pool.** `init` auto-detects the external network and will select `source = "dhcp"` when the host itself holds a DHCP lease. That does not work for guests on every network: the upstream server answers the host on `br-wan` but does not necessarily offer to guest ENI MAC addresses, and instance launches then fail with
|
||||
**Set a static external pool.** `init` auto-detects the external network and selects `source = "dhcp"` when the host holds a DHCP lease. That does not work for guests on every network: the upstream DHCP server answers the host on `br-wan` but does not necessarily offer addresses to guest ENI MAC addresses, causing instance launches to fail with:
|
||||
|
||||
```text
|
||||
PrepareRunInstances: public IP allocation failed — aborting launch
|
||||
@@ -91,7 +100,7 @@ prefix_len = 24
|
||||
dns_servers = ["1.1.1.1", "8.8.8.8"]
|
||||
```
|
||||
|
||||
See the [VPC Networking](/docs/vpc-networking) guide for full configuration options. Then start the platform and verify:
|
||||
See the [VPC Networking](/docs/vpc-networking) guide for full configuration options. Then start the platform and confirm the node is ready:
|
||||
|
||||
```bash
|
||||
sudo systemctl start spinifex.target
|
||||
@@ -103,187 +112,156 @@ NAME | STATUS | ROLES | IP | REGION | AZ
|
||||
node1 | Ready | nats:leader | 192.168.157.134 | ap-southeast-2 | ap-southeast-2a | 0 | nats,predastore,viperblock,daemon,awsgw,vpcd,ui
|
||||
```
|
||||
|
||||
### 2. Establish the bare-metal baseline
|
||||
All services — Predastore, Viperblock, and the OVN databases — run on the single node. There is no Raft group to converge and no leader election to wait for.
|
||||
|
||||
Benchmark the host before launching anything, with the platform idle. Without this, a slow guest result cannot be attributed to the platform rather than the hardware.
|
||||
### 2. Launch instances through the EC2-compatible API
|
||||
|
||||
This uses `spx-bench.sh` — the same script is run on the host and inside the guest so the two results diff directly. It measures CPU (sysbench), memory, and disk (fio: `randrw` 70/30 4k, `randread` 4k, `seqread` 1M, `seqwrite` 1M, all at 4 jobs / iodepth 32 / `direct=1`).
|
||||
Point an existing AWS profile at the Spinifex endpoint and use the standard EC2 instance lifecycle:
|
||||
|
||||
```bash
|
||||
export AWS_PROFILE=spinifex
|
||||
|
||||
aws ec2 run-instances \
|
||||
--image-id "$AMI" --instance-type c6i.2xlarge \
|
||||
--count 1 \
|
||||
--subnet-id "$SUBNET" --security-group-ids "$SECURITY_GROUP" \
|
||||
--key-name "$KEY_NAME"
|
||||
```
|
||||
|
||||
`c6i.2xlarge` provisions 8 vCPU and 16 GiB from the host's 12 cores and 31 GiB, leaving headroom for the platform services. On a single node there are no spread placement groups — all instances land on the same physical host, which is expected.
|
||||
|
||||
### 3. Attach Viperblock volumes and access Predastore object storage
|
||||
|
||||
Create and attach a dedicated data volume. The default root volume on the stock Ubuntu AMI is 4 GiB — too small for most workloads. Attach a separate volume and benchmark or write data there:
|
||||
|
||||
```bash
|
||||
VOLUME_ID=$(aws ec2 create-volume \
|
||||
--availability-zone ap-southeast-2a \
|
||||
--size 30 --volume-type gp2 \
|
||||
--query VolumeId --output text)
|
||||
|
||||
aws ec2 attach-volume \
|
||||
--volume-id "$VOLUME_ID" \
|
||||
--instance-id "$INSTANCE_ID" \
|
||||
--device /dev/sdf
|
||||
```
|
||||
|
||||
Inside the guest, format and mount:
|
||||
|
||||
```bash
|
||||
sudo mkfs.ext4 -q -L data /dev/vdb
|
||||
sudo mkdir -p /mnt/data && sudo mount /dev/vdb /mnt/data
|
||||
```
|
||||
|
||||
Setting `delete_on_termination = false` in Terraform keeps the volume alive across instance replacement — relaunching a terminated instance re-attaches the same volume rather than starting from empty storage:
|
||||
|
||||
```hcl
|
||||
resource "aws_ebs_volume" "data" {
|
||||
availability_zone = var.az
|
||||
size = 30
|
||||
type = "gp2"
|
||||
}
|
||||
|
||||
resource "aws_volume_attachment" "data" {
|
||||
device_name = "/dev/sdf"
|
||||
volume_id = aws_ebs_volume.data.id
|
||||
instance_id = aws_instance.worker.id
|
||||
delete_on_termination = false
|
||||
}
|
||||
```
|
||||
|
||||
Predastore presents an S3-compatible endpoint — standard S3 tooling works without modification:
|
||||
|
||||
```bash
|
||||
aws s3 mb s3://my-bucket
|
||||
aws s3 cp ./data.tar.gz s3://my-bucket/
|
||||
aws s3 sync ./results/ s3://my-bucket/results/
|
||||
```
|
||||
|
||||
### 4. Storage benchmark
|
||||
|
||||
Run this after setup or hardware changes to confirm both storage services are performing as expected. The methodology: fio against an ext4-formatted Viperblock data volume (four jobs, iodepth 32, direct I/O, 30-second run), with a bare-metal host baseline run first so the two results diff directly using the same `spx-bench.sh` script.
|
||||
|
||||
**Host bare-metal baseline** (platform idle, no guests):
|
||||
|
||||
```bash
|
||||
./spx-bench.sh --tag host-baremetal
|
||||
```
|
||||
|
||||
```text
|
||||
--- 2. CPU (sysbench, events/sec — higher is better) ------------
|
||||
--- CPU (sysbench, events/sec) ---------------------------------
|
||||
threads=1 1456.50 events/s 95th 0.70 ms
|
||||
threads=16 12822.85 events/s 95th 2.00 ms
|
||||
|
||||
--- 3. Memory --------------------------------------------------
|
||||
random write, 16 threads: 1555.84 MiB/sec
|
||||
|
||||
--- 4. Disk (file-backed, NVMe root) ---------------------------
|
||||
randrw 70/30 4k read 66,169 IOPS 258 MiB/s p99 10.42 ms
|
||||
randrw 70/30 4k write 28,426 IOPS 111 MiB/s p99 0.72 ms
|
||||
randread 4k read 171,326 IOPS 669 MiB/s p99 2.06 ms
|
||||
seqread 1M read 1,818 IOPS 1,818 MiB/s p99 128.45 ms
|
||||
seqwrite 1M write 308 IOPS 308 MiB/s p99 1115.68 ms
|
||||
--- Disk (fio, NVMe root) --------------------------------------
|
||||
randrw 70/30 4k read 66,169 IOPS 258 MiB/s
|
||||
randrw 70/30 4k write 28,426 IOPS 111 MiB/s
|
||||
randread 4k read 171,326 IOPS 669 MiB/s
|
||||
seqread 1M read 1,818 MiB/s
|
||||
seqwrite 1M write 308 MiB/s
|
||||
```
|
||||
|
||||
### 3. Launch a guest instance
|
||||
|
||||
```bash
|
||||
aws ec2 run-instances \
|
||||
--image-id "$AMI" --instance-type c6i.2xlarge \
|
||||
--key-name onlogic --security-group-ids "$SG" --subnet-id "$SUBNET"
|
||||
```
|
||||
|
||||
**Attach a data volume for the benchmark.** The stock `ubuntu-26.04-x86_64` AMI declares a 4 GiB root which fills just installing benchmark tools, and `run-instances` does not grow it without explicit `--block-device-mappings`. Every fio job fails with `ENOSPC` if you benchmark the root. Attach a gp3 volume instead, which is also the more representative target:
|
||||
|
||||
```bash
|
||||
VOL=$(aws ec2 create-volume --availability-zone ap-southeast-2a \
|
||||
--size 30 --volume-type gp3 --query VolumeId --output text)
|
||||
aws ec2 attach-volume --volume-id "$VOL" --instance-id "$ID" --device /dev/vdb
|
||||
```
|
||||
|
||||
```bash
|
||||
# in the guest
|
||||
sudo mkfs.ext4 -q -L bench /dev/vdb
|
||||
sudo mkdir -p /mnt/bench && sudo mount /dev/vdb /mnt/bench
|
||||
```
|
||||
|
||||
### 4. Measure the guest
|
||||
**Guest** (`c6i.2xlarge`, 8 vCPU, attached gp2 volume):
|
||||
|
||||
```bash
|
||||
BENCHDIR=/mnt/bench ./spx-bench.sh --tag guest --require-mount
|
||||
```
|
||||
|
||||
```text
|
||||
--- 2. CPU (sysbench, events/sec — higher is better) ------------
|
||||
threads=1 1357.98 events/s 95th 0.75 ms
|
||||
threads=8 8608.31 events/s 95th 1.06 ms
|
||||
|
||||
--- 3. Memory --------------------------------------------------
|
||||
random write, 8 threads: 1133.25 MiB/sec
|
||||
```
|
||||
|
||||
### 5. Measure the S3 data path
|
||||
|
||||
Predastore's S3 API listens on port `8443`. This is **not** the general AWS gateway endpoint on `9999`.
|
||||
|
||||
```bash
|
||||
./s3-bench.sh 256 # 256 MB object, write + read + checksum verify
|
||||
```
|
||||
|
||||
## Results
|
||||
|
||||
### CPU and memory
|
||||
|
||||
| | host | guest | guest/host |
|
||||
|---|---|---|---|
|
||||
| sysbench, 1 thread | 1,456 events/s | 1,358 events/s | **93.2%** |
|
||||
| sysbench, all threads | 12,823 (16t) | 8,608 (8t) | not comparable |
|
||||
| Memory random write | 1,556 MiB/s (16t) | 1,133 MiB/s (8t) | not comparable |
|
||||
|
||||
Single thread is the only directly comparable row, and **93.2% of bare metal** is the headline CPU number — roughly 7% single-thread virtualisation overhead. The multi-thread and memory rows use different thread counts (16 on the host, 8 in the guest) and are listed for completeness, not as ratios.
|
||||
|
||||
<p align="center"><img src="images/cpu-comparison.png" alt="sysbench single-thread: host 1,456 events/s vs guest 1,358 events/s (93.2%)"></p>
|
||||
|
||||
### Storage
|
||||
|
||||
Median of three runs. Guest measured on the attached gp3 volume.
|
||||
|
||||
| Test | host | guest | guest/host |
|
||||
|---|---|---|---|
|
||||
| randrw 70/30 4k read | 66,169 IOPS | 6,860 IOPS | 10.4% |
|
||||
| randrw 70/30 4k write | 28,426 IOPS | 2,951 IOPS | 10.4% |
|
||||
| Test | Host | Guest | Guest/Host |
|
||||
|---|---:|---:|---:|
|
||||
| sysbench single-thread | 1,456 events/s | 1,358 events/s | **93.2%** |
|
||||
| randrw 4k read | 66,169 IOPS | 6,860 IOPS | 10.4% |
|
||||
| randrw 4k write | 28,426 IOPS | 2,951 IOPS | 10.4% |
|
||||
| randread 4k | 171,326 IOPS | 9,263 IOPS | 5.4% |
|
||||
| seqread 1M | 1,818 MiB/s | 484 MiB/s | 26.6% |
|
||||
| seqwrite 1M | 308 MiB/s | 73 MiB/s | 23.7% |
|
||||
|
||||
The block path is where virtualisation costs real throughput. Sequential work retains roughly a quarter of bare metal; small random I/O retains between a twentieth and a tenth.
|
||||
|
||||
<p align="center">
|
||||
<img src="images/storage-iops-comparison.png" alt="Random I/O IOPS: host vs guest">
|
||||
<img src="images/storage-seq-comparison.png" alt="Sequential throughput MiB/s: host vs guest">
|
||||
<img src="images/cpu-comparison.png" alt="sysbench single-thread: host 1,456 events/s vs guest 1,358 events/s (93.2%)">
|
||||
</p>
|
||||
|
||||
**Read the guest storage figures with their variance, not as point values.** Across three runs:
|
||||
Guest CPU lands at **93.2% of bare metal** — roughly 7% single-thread virtualisation overhead, a result that reproduced on separate hardware. Block storage reflects the cost of the current single-queue virtio-blk path and unthreaded NBD backend sharing a single 256 GB NVMe between the host OS, platform services, and guest volumes; these are the same implementation constraints as on the Cisco cluster, expressed more sharply on constrained hardware.
|
||||
|
||||
| Test | run 1 | run 2 | run 3 | spread |
|
||||
|---|---|---|---|---|
|
||||
Variance across three runs is notable for two metrics:
|
||||
|
||||
| Test | Run 1 | Run 2 | Run 3 | Spread |
|
||||
|---|---:|---:|---:|---:|
|
||||
| randrw 4k read | 7,321 | 6,424 | 6,860 | 14% |
|
||||
| randread 4k | 9,959 | **3,813** | 9,263 | **2.6×** |
|
||||
| seqread 1M | 518 | 484 | 453 | 14% |
|
||||
| seqwrite 1M | 83 | 73 | **31** | **2.7×** |
|
||||
| seqwrite p99 | 7.95 s | 7.28 s | **15.90 s** | 2.2× |
|
||||
|
||||
<p align="center"><img src="images/storage-variance.png" alt="Guest storage variance across 3 runs: randread and seqwrite show bimodal behaviour"></p>
|
||||
<p align="center">
|
||||
<img src="images/storage-iops-comparison.png" alt="Random I/O IOPS: host vs guest">
|
||||
<img src="images/storage-seq-comparison.png" alt="Sequential throughput MiB/s: host vs guest">
|
||||
<img src="images/storage-variance.png" alt="Guest storage variance across 3 runs: randread and seqwrite show bimodal behaviour">
|
||||
</p>
|
||||
|
||||
`randread` and `seqwrite` are bimodal rather than noisy around a mean — one run in three falls to roughly a third of its neighbours. The likely mechanism is a viperblock WAL flush or a Predastore compaction cycle landing inside some runs and not others. Anything below about a 3× change in those two metrics is not resolvable at this sample size.
|
||||
`randread` and `seqwrite` are bimodal rather than noisy — one run in three falls to roughly a third of its neighbours, likely a Viperblock WAL flush or Predastore compaction cycle intersecting some runs but not others. Treat those two as directionally correct rather than stable point values at this sample size. The `seqwrite` p99 reaching 7–16 seconds is worth noting for write-heavy workloads.
|
||||
|
||||
`seqwrite` p99 reaching 7-16 seconds is worth attention before committing this platform to write-heavy edge workloads.
|
||||
**Predastore S3 validation** (256 MiB objects, three runs, write + read + checksum verify):
|
||||
|
||||
## Release comparison: v1.15.0 vs v1.16.0
|
||||
|
||||
Both releases were built from their release tags and installed on the same machine, each with a full re-initialisation, an identical `c6i.2xlarge` guest, and an identical 30 GiB gp3 volume. Three runs each.
|
||||
|
||||
### S3 / Predastore
|
||||
|
||||
| | v1.15.0 | v1.16.0 | change |
|
||||
|---|---|---|---|
|
||||
| write | 77.7 / 75.5 / 72.0 → **75.5 MiB/s** | 146.6 / 142.0 / 148.2 → **146.6 MiB/s** | **1.94× faster** |
|
||||
| read | 249.9 / 234.9 / 243.8 → **243.8 MiB/s** | 298.1 / 301.8 / 290.9 → **298.1 MiB/s** | **1.22× faster** |
|
||||
|
||||
Checksums matched on every run. Run-to-run spread is about 4%, so both differences are far outside the noise.
|
||||
|
||||
**The mechanism is a changed default, not a faster code path.** On a single node:
|
||||
|
||||
```text
|
||||
v1.15.0 [rs] data = 2, parity = 1 6 host entries
|
||||
v1.16.0 [rs] data = 1, parity = 0 1 host entry
|
||||
```bash
|
||||
./s3-bench.sh 256
|
||||
```
|
||||
|
||||
v1.15.0 splits every object into two data shards plus a parity shard and writes all three to the same disk — 1.5× write amplification plus the erasure-coding cost, buying no redundancy, because there is only one machine to lose. v1.16.0 ships a single-node storage template that does none of that.
|
||||
|
||||
The write path gains most (1.94×), which is what removing amplification and parity computation predicts. Reads gain less (1.22×) because the read path never had to reconstruct anything, only reassemble shards.
|
||||
|
||||
Anyone still running v1.15.0 on a single node should note that this is a configuration change as much as a software one.
|
||||
| | v1.15.0 | v1.16.0 | Change |
|
||||
|---|---:|---:|---:|
|
||||
| Write | 75.5 MiB/s | 146.6 MiB/s | **1.94×** |
|
||||
| Read | 243.8 MiB/s | 298.1 MiB/s | **1.22×** |
|
||||
|
||||
<p align="center"><img src="images/s3-version-comparison.png" alt="S3 throughput: v1.15.0 vs v1.16.0 — 1.94× write, 1.22× read"></p>
|
||||
|
||||
### Guest block path
|
||||
The v1.16.0 write improvement comes from the single-node storage template switching to RS(1,0): v1.15.0 split every object into two data shards plus a parity shard and wrote all three to the same disk, paying 1.5× write amplification for redundancy a single-node cluster cannot deliver. The read gain is smaller because the read path never had to reconstruct anything. All checksum validations passed on every run.
|
||||
|
||||
Medians of three runs, with the full range in brackets.
|
||||
Raw fio output, S3 results, and the v1.15 / v1.16 annotated comparisons are in the
|
||||
[benchmark repository](https://github.com/tomnewton-mulga/OnLogic-refarch).
|
||||
|
||||
| Test | v1.15.0 | v1.16.0 | v1.16 gain |
|
||||
|---|---|---|---|
|
||||
| randrw 70/30 4k read | **5,075** IOPS [4,104-6,236] | **6,860** IOPS [6,424-7,321] | 1.35× |
|
||||
| randrw 70/30 4k write | **2,191** IOPS [1,771-2,684] | **2,951** IOPS [2,764-3,147] | 1.35× |
|
||||
| randread 4k | **6,658** IOPS [4,881-6,813] | **9,263** IOPS [3,813-9,959] | 1.39× |
|
||||
| seqread 1M | **387** MiB/s [370-402] | **484** MiB/s [453-518] | 1.25× |
|
||||
| seqwrite 1M | **52** MiB/s [40-58] | **73** MiB/s [31-83] | 1.40× |
|
||||
| seqwrite p99 | **13.89 s** [13.76-17.11] | **7.95 s** [7.28-15.90] | 1.75× lower |
|
||||
## What this architecture unlocks
|
||||
|
||||
<p align="center"><img src="images/block-path-version-comparison.png" alt="Guest block path: v1.15.0 vs v1.16.0 with full run ranges"></p>
|
||||
|
||||
v1.16.0 is ahead on every median, by 1.25× to 1.40×. For `randrw` read, `randrw` write and `seqread` the two ranges **do not overlap at all**, so those differences are credible despite only three runs each. For `randread` and `seqwrite` the ranges do overlap — entirely because of v1.16.0's bimodal low outliers — so treat those two as directionally consistent rather than demonstrated.
|
||||
|
||||
This is the indirect effect of the storage template change. Viperblock's own code is unchanged between the releases, but it writes through Predastore, so removing the 1.5× write amplification and parity computation underneath reduces the work behind every guest block flush as well.
|
||||
|
||||
## Teardown
|
||||
|
||||
```bash
|
||||
aws ec2 terminate-instances --instance-ids "$ID"
|
||||
aws ec2 delete-volume --volume-id "$VOL" # after the instance is terminated
|
||||
aws ec2 delete-security-group --group-id "$SG"
|
||||
```
|
||||
|
||||
Terminate instances before deleting attached volumes — the volume detaches with the instance, not before it.
|
||||
|
||||
## Conclusion
|
||||
|
||||
A single OnLogic HX401 runs the complete Spinifex service set — EC2, EBS, S3, VPC networking and DNS — and hosts an 8 vCPU guest with room to spare. Guest CPU performance lands at **93.2% of bare metal**, a figure that reproduced on separate hardware.
|
||||
|
||||
The release comparison is the most actionable result here. Moving from v1.15.0 to v1.16.0 on identical hardware gave **1.94× S3 write throughput** and **1.25-1.40× on every guest block metric** — not because the data paths were rewritten, but because the single-node storage template stopped erasure-coding data across a machine that has only one disk. v1.15.0 wrote two data shards and a parity shard to the same NVMe, paying 1.5× write amplification and the encode cost for redundancy it could never deliver.
|
||||
|
||||
Every operation in this document used standard `aws ec2` and `aws s3api` calls against Spinifex's EC2-compatible endpoint. Teams already operating AWS tooling can point it at an edge node with a profile change.
|
||||
- **Cloud tooling at the edge:** run EC2, EBS and S3 workflows on hardware that fits inside a panel, on a DIN rail, or in a small enclosure at a factory, retail site, branch or remote location — without a rack, a UPS, or active cooling.
|
||||
- **No-moving-parts reliability:** the fanless passive-cooled design tolerates dusty, vibration-prone, or thermally variable environments where server-class hardware is impractical to operate or maintain.
|
||||
- **Familiar operations:** keep AWS CLI profiles, Terraform or OpenTofu modules, SDKs and deployment pipelines rather than adopting a separate edge-only platform.
|
||||
- **Local processing with S3 access:** workloads running in EC2-compatible instances read from and write to Predastore over the standard S3 API — sensor data, inference outputs and logs remain on-node until explicitly synced upstream.
|
||||
- **A path back to cloud:** the same AWS API surface at the edge and on AWS makes workload movement, cloud burst, and consistent application packaging straightforward — the profile swap that points tooling at the HX401 is the same one that points it back at AWS.
|
||||
|
||||
@@ -18,12 +18,12 @@ resources:
|
||||
url: "/docs/launching-instances"
|
||||
- title: "GPU Passthrough"
|
||||
url: "/docs/gpu-passthrough"
|
||||
- title: "Platform benchmark (companion document)"
|
||||
- title: "Cisco UCS: AWS-compatible cloud at the edge (companion document)"
|
||||
url: "../cisco-ucs-platform-benchmark/README.md"
|
||||
- title: "Vision pipeline (companion document)"
|
||||
url: "../cisco-ucs-vision-pipeline/README.md"
|
||||
- title: "cisco-ucs-llm-serving (benchmark scripts and results)"
|
||||
url: "https://github.com/mulgadc/cisco-ucs-llm-serving"
|
||||
- title: "Benchmark and workload materials (CISCO-refarch)"
|
||||
url: "https://github.com/tomnewton-mulga/CISCO-refarch"
|
||||
- title: "vLLM"
|
||||
url: "https://docs.vllm.ai"
|
||||
- title: "Qwen2.5-7B-Instruct"
|
||||
@@ -45,7 +45,7 @@ cluster, comparing Intel AMX-accelerated CPU serving on an `m8i.2xlarge`-class i
|
||||
against NVIDIA L4 GPU serving on a `g6.2xlarge`-class instance — a real,
|
||||
production-representative LLM-serving stack, with the two accelerators measured independently at matched model, version, and serving configuration.
|
||||
|
||||
**Companion documents:** [Spinifex Platform Benchmark on Cisco UCS](../cisco-ucs-platform-benchmark/README.md) · [Spinifex Vision Pipeline on Cisco UCS](../cisco-ucs-vision-pipeline/README.md)
|
||||
**Companion architectures:** [Cisco UCS: AWS-compatible cloud at the edge](../cisco-ucs-platform-benchmark/README.md) · [Spinifex Vision Pipeline on Cisco UCS](../cisco-ucs-vision-pipeline/README.md)
|
||||
|
||||
### Platform
|
||||
|
||||
|
||||
@@ -1,14 +1,15 @@
|
||||
---
|
||||
title: "Spinifex Platform Benchmark on Cisco UCS (3-Node)"
|
||||
seoTitle: "Spinifex Benchmark on 3-Node Cisco UCS — Spinifex Docs"
|
||||
description: "Measured CPU, disk, network, and S3 performance of Spinifex on a 3-node Cisco Unified Edge cluster, comparing host against guest with Intel AMX confirmed."
|
||||
title: "Cisco UCS: AWS-compatible cloud at the edge"
|
||||
seoTitle: "Cisco UCS Edge Cloud with Spinifex — Spinifex Docs"
|
||||
description: "Place EC2 instances, EBS volumes, S3 object storage and Kubernetes workloads on a resilient three-node Cisco UCS cluster, using familiar AWS APIs and tooling."
|
||||
category: "Reference Architectures"
|
||||
tags:
|
||||
- cisco
|
||||
- ucs
|
||||
- edge
|
||||
- aws-compatible
|
||||
- xeon-6
|
||||
- intel-amx
|
||||
- benchmark
|
||||
- predastore
|
||||
- viperblock
|
||||
- raft
|
||||
@@ -21,12 +22,8 @@ resources:
|
||||
url: "/docs/launching-instances"
|
||||
- title: "GPU Passthrough"
|
||||
url: "/docs/gpu-passthrough"
|
||||
- title: "AWS: Accelerate CPU-based AI inference with Intel AMX on EC2"
|
||||
url: "https://aws.amazon.com/blogs/compute/accelerate-cpu-based-ai-inference-workloads-using-intel-amx-on-amazon-ec2/"
|
||||
- title: "cisco-ucs-platform-benchmark (benchmark scripts and results)"
|
||||
url: "https://github.com/mulgadc/cisco-ucs-platform-benchmark"
|
||||
- title: "Dell EMC PowerEdge R7525 Review (inspiration for methodology)"
|
||||
url: "https://www.servethehome.com/dell-emc-poweredge-r7525-review-flagship-dell-dual-socket-server-amd-epyc/4/"
|
||||
- title: "cisco-ucs-platform-benchmark (scripts and raw results)"
|
||||
url: "https://github.com/tomnewton-mulga/CISCO-refarch"
|
||||
sections:
|
||||
- overview
|
||||
- prerequisites
|
||||
@@ -35,16 +32,14 @@ sections:
|
||||
|
||||
## Overview
|
||||
|
||||
Spinifex is an open-source infrastructure platform that brings core AWS services — EC2,
|
||||
S3, EBS and EKS — to bare-metal, edge, and on-prem deployments, exposing a fully
|
||||
AWS-compatible API.
|
||||
This reference architecture turns three Cisco Unified Edge servers into a small,
|
||||
resilient cloud that can run where data is produced: a factory, retail estate,
|
||||
branch, lab, sovereign environment, or disconnected site. Spinifex exposes EC2,
|
||||
EBS, S3 and EKS-compatible APIs on the cluster, so the operational model is
|
||||
familiar: use the AWS CLI, SDKs, Terraform or OpenTofu, Kubernetes manifests and
|
||||
CI/CD systems already used for AWS. The change is the endpoint, not the workflow.
|
||||
|
||||
This document characterizes the platform itself on a 3-node Cisco Unified Edge cluster,
|
||||
independent of any specific application: CPU, disk, network, and S3 (Predastore)
|
||||
throughput measured identically on bare metal and inside guest instances, and how many
|
||||
concurrent worker processes each instance can sustain before latency degrades.
|
||||
|
||||
**Companion documents:** [Spinifex Vision Pipeline on Cisco UCS](../cisco-ucs-vision-pipeline/README.md) · [vLLM Serving on Cisco UCS: Intel AMX vs NVIDIA L4](../cisco-ucs-llm-serving/README.md)
|
||||
**Companion architectures:** [Vision Pipeline on Cisco UCS](../cisco-ucs-vision-pipeline/README.md) shows a local AI pipeline that streams inputs from S3; [vLLM Serving on Cisco UCS](../cisco-ucs-llm-serving/README.md) shows CPU and GPU model serving on the same cluster.
|
||||
|
||||
### Platform
|
||||
|
||||
@@ -56,185 +51,243 @@ concurrent worker processes each instance can sustain before latency degrades.
|
||||
| **ISA** | AVX-512 (F/DQ/BW/VL/VNNI/BF16/FP16), **Intel AMX** (tile/BF16/INT8), VT-x, VT-d |
|
||||
| **Memory** | 499 GiB usable per node (≈1.5 TiB aggregate) |
|
||||
| **GPU** | 1× NVIDIA L4 (23,034 MiB) via VFIO PCIe passthrough on one node; the other two nodes are CPU-only |
|
||||
| **Storage** | 4× KIOXIA CD8P NVMe (1.92 TB each) per node, raw — Predastore/Viperblock claim these directly |
|
||||
| **Storage** | 4× KIOXIA CD8P NVMe (1.92 TB each) per node, raw — Predastore and Viperblock claim these directly |
|
||||
| **Boot** | Cisco SATA RAID VD (Marvell 88SE9230 controller) |
|
||||
| **Network** | 2× Intel E825-C 25GbE ports per node, each its own bond: one for management/WAN (VLAN 1337, 1 Gbps upstream uplink), one dedicated to storage/cluster traffic (VLAN 1336, full 25GbE) |
|
||||
| **Network** | 2× Intel E825-C 25GbE ports per node — one bond for management/WAN (VLAN 1337), one dedicated to storage/cluster traffic (VLAN 1336, full 25GbE) |
|
||||
| **Aggregate** | 96 cores / 192 threads, ~1.5 TiB RAM, ~23 TiB raw NVMe, 1× L4 GPU |
|
||||
|
||||
Single-NUMA-per-node is a meaningful simplification versus a typical dual-socket
|
||||
reference design — no vCPU/memory locality tuning required for VM placement. The GPU is
|
||||
present on one node only — GPU-accelerated instances are bound to that node, while
|
||||
CPU-only instances can schedule across all three.
|
||||
Single-NUMA-per-node simplifies placement — no vCPU/memory locality tuning required.
|
||||
GPU workloads schedule to the L4 node; CPU instances can use all three. Storage and
|
||||
object data are distributed across the cluster rather than tied to the node that
|
||||
launched an instance.
|
||||
|
||||
## Architecture
|
||||
|
||||
### AWS services exercised
|
||||
|
||||
| Service | Role |
|
||||
|---|---|
|
||||
| **EC2** | Guest instances across all three nodes; spread placement groups distribute across physical nodes |
|
||||
| **EBS (Viperblock)** | Root and data volumes per instance, surviving termination independently when `delete_on_termination = false` |
|
||||
| **S3 (Predastore)** | Cluster-wide object storage for datasets, model weights, artifacts and backups |
|
||||
| **VPC (OVN)** | Overlay networking between instances |
|
||||
| **EKS** | (Optional) Kubernetes control plane with pod rescheduling on node failure |
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- 3-node Spinifex cluster installed and services healthy on all nodes — follow the [Multi-Node Install](/docs/install-multi-node) guide.
|
||||
- VPC, network pool, SSH key pair, and security group configured — see [VPC Networking](/docs/vpc-networking) and [Launching Instances](/docs/launching-instances)
|
||||
- GPU passthrough configured on the one node with the NVIDIA L4 — see [GPU Passthrough](/docs/gpu-passthrough):
|
||||
- VPC, subnet, network address pool, SSH key pair, and security group configured — see [VPC Networking](/docs/vpc-networking) and [Launching Instances](/docs/launching-instances).
|
||||
- GPU passthrough enabled on the L4 node when GPU instances are required — see [GPU Passthrough](/docs/gpu-passthrough):
|
||||
|
||||
```bash
|
||||
sudo spx admin gpu setup # reboot required after this step
|
||||
sudo spx admin gpu enable
|
||||
```
|
||||
|
||||
- AWS CLI configured with `AWS_PROFILE=spinifex` pointing at the cluster's EC2-compatible endpoint
|
||||
- AWS CLI configured with `AWS_PROFILE=spinifex` pointing at the cluster's EC2-compatible endpoint (`https://<host>:9999`).
|
||||
|
||||
## Instructions
|
||||
|
||||
Benchmark scripts and raw results for all sections below are available in the
|
||||
[platform benchmark repository](https://github.com/mulgadc/cisco-ucs-platform-benchmark).
|
||||
### 1. Verify the cluster and its control plane
|
||||
|
||||
### 1. Matched-width CPU
|
||||
|
||||
sysbench, identical thread counts, run on bare metal and inside a guest of the same vCPU
|
||||
width:
|
||||
|
||||
| Scope | Target | Median events/s |
|
||||
|---|---|---:|
|
||||
| host | mulga-01 | 8,205.4 |
|
||||
| host | mulga-02 | 8,209.3 |
|
||||
| host | mulga-03 | 8,210.4 |
|
||||
| guest | cpu1 | 8,168.4 |
|
||||
| guest | cpu2 | 8,174.0 |
|
||||
| guest | gpu | 8,160.8 |
|
||||
|
||||
Guests land within ~0.5% of host — virtualization overhead on this platform is
|
||||
negligible for CPU-bound work.
|
||||
|
||||
### 2. Confirm Intel AMX is actually executing
|
||||
|
||||
Intel AMX (Advanced Matrix Extensions) is a relatively new instruction set available on
|
||||
the Xeon 6543P-B, designed specifically to accelerate the matrix operations that dominate
|
||||
AI inference workloads. CPUID flags alone don't prove it's actually in use — the proof
|
||||
is oneDNN's own kernel-selection trace during a real inference run at each precision:
|
||||
Before placing workloads, confirm all three nodes are ready and that the storage and
|
||||
control-plane services are running on each:
|
||||
|
||||
```bash
|
||||
ONEDNN_VERBOSE=1 python3 <any BF16-hinted inference workload> 2> onednn-verbose.log
|
||||
grep -o 'avx10_1_512_amx[a-z_]*\|avx512_core[a-z_]*' onednn-verbose.log | sort -u
|
||||
spx get nodes
|
||||
systemctl is-active spinifex-predastore spinifex-viperblock spinifex-daemon
|
||||
```
|
||||
|
||||
| Precision | Kernel selected |
|
||||
|---|---|
|
||||
| FP32 (forced, negative control) | `avx512_core` only — **zero** AMX kernel selections |
|
||||
| BF16 | `avx10_1_512_amx` |
|
||||
| INT8 | `avx10_1_512_amx` (+ one `avx512_core` fallback for an unquantized matmul) |
|
||||
Predastore's Raft group and OVN's northbound/southbound databases are both distributed
|
||||
across all three nodes — losing one node triggers a clean leader re-election among the
|
||||
remaining two, and object storage and networking continue serving. Check that all three
|
||||
nodes are contributing to the Raft group before exposing the cluster to workload traffic.
|
||||
|
||||
`avx10_1_512_amx` is the current kernel-naming scheme on this hardware generation — not
|
||||
the older `brgemm_avx512_amx*` string some documentation still references. Confirmed by
|
||||
direct trace inspection, not inferred from CPUID or throughput alone.
|
||||
### 2. Launch instances through the EC2-compatible API
|
||||
|
||||
### 3. Disk — host versus guest
|
||||
Point an existing AWS profile at the Spinifex endpoint and use the standard EC2
|
||||
instance lifecycle. Terraform is the recommended path — Spinifex's spread placement
|
||||
group implementation holds node reservations that are not always released on destroy,
|
||||
so a fixed group name can cause a subsequent apply to hang. Using a per-deploy unique
|
||||
name (via `random_id` or equivalent) avoids this:
|
||||
|
||||
fio, file-backed (non-destructive), identical parameters on bare metal and inside a
|
||||
guest:
|
||||
```hcl
|
||||
resource "random_id" "suffix" {
|
||||
byte_length = 4
|
||||
}
|
||||
|
||||
| Metric | Host (mulga-01) | Guest (cpu1, idle) | Ratio |
|
||||
|---|---:|---:|---:|
|
||||
| Random 4K read | 4,460.8 MiB/s | 11.6 MiB/s | ~1/385 |
|
||||
| Sequential 1M read | 7,122.6 MiB/s | 375.5 MiB/s | ~1/19 |
|
||||
resource "aws_placement_group" "spread" {
|
||||
name = "edge-spread-${random_id.suffix.hex}"
|
||||
strategy = "spread"
|
||||
}
|
||||
|
||||
<img src="../../../.github/assets/images/cisco-ucs-platform-benchmark/disk-host-vs-guest.png" alt="Host versus guest disk throughput, random 4K and sequential 1M read, log scale">
|
||||
resource "aws_instance" "worker" {
|
||||
count = 2
|
||||
ami = var.ami_id
|
||||
instance_type = "m8i.2xlarge"
|
||||
subnet_id = var.subnet_id
|
||||
key_name = var.key_name
|
||||
placement_group = aws_placement_group.spread.name
|
||||
vpc_security_group_ids = [var.security_group_id]
|
||||
}
|
||||
```
|
||||
|
||||
Guest I/O traverses Viperblock's NBD path; the random 4K gap is far larger than the sequential 1M gap because small requests pay the full per-request serialization overhead without amortizing it across a large transfer. This is a known area for improvement in upcoming Spinifex releases.
|
||||
For a one-off launch via the CLI, use a unique group name each time for the same reason:
|
||||
|
||||
### 4. Network — physical fabric versus guest overlay
|
||||
```bash
|
||||
export AWS_PROFILE=spinifex
|
||||
GROUP="edge-spread-$(openssl rand -hex 4)"
|
||||
|
||||
iperf3, matched methodology, on the physical storage fabric (bare metal) and between
|
||||
guest instances over the OVN/Geneve overlay:
|
||||
aws ec2 create-placement-group \
|
||||
--group-name "$GROUP" \
|
||||
--strategy spread
|
||||
|
||||
| Path | Streams | Median Gbit/s |
|
||||
aws ec2 run-instances \
|
||||
--image-id "$AMI" --instance-type m8i.2xlarge \
|
||||
--count 2 \
|
||||
--subnet-id "$SUBNET" --security-group-ids "$SECURITY_GROUP" \
|
||||
--key-name "$KEY_NAME" \
|
||||
--placement "{\"GroupName\":\"$GROUP\"}"
|
||||
```
|
||||
|
||||
A `g6.2xlarge` instance type routes to the node with GPU passthrough configured;
|
||||
`m8i` types schedule across all three nodes.
|
||||
|
||||
### 3. Attach Viperblock volumes and access Predastore object storage
|
||||
|
||||
Create and attach an EBS-compatible data volume to a running instance:
|
||||
|
||||
```bash
|
||||
VOLUME_ID=$(aws ec2 create-volume \
|
||||
--availability-zone "$AZ" \
|
||||
--size 100 --volume-type gp2 \
|
||||
--query VolumeId --output text)
|
||||
|
||||
aws ec2 attach-volume \
|
||||
--volume-id "$VOLUME_ID" \
|
||||
--instance-id "$INSTANCE_ID" \
|
||||
--device /dev/sdf
|
||||
```
|
||||
|
||||
Setting `delete_on_termination = false` in Terraform keeps the volume alive across
|
||||
instance replacement — relaunching a terminated instance re-attaches the same volume
|
||||
rather than starting from empty storage:
|
||||
|
||||
```hcl
|
||||
resource "aws_ebs_volume" "data" {
|
||||
availability_zone = var.az
|
||||
size = 100
|
||||
type = "gp2"
|
||||
}
|
||||
|
||||
resource "aws_volume_attachment" "data" {
|
||||
device_name = "/dev/sdf"
|
||||
volume_id = aws_ebs_volume.data.id
|
||||
instance_id = aws_instance.worker[0].id
|
||||
delete_on_termination = false
|
||||
}
|
||||
```
|
||||
|
||||
Predastore presents an S3-compatible endpoint — standard S3 tooling works without
|
||||
modification:
|
||||
|
||||
```bash
|
||||
aws s3 mb s3://my-bucket
|
||||
aws s3 cp ./dataset.tar.gz s3://my-bucket/
|
||||
aws s3 sync ./results/ s3://my-bucket/results/
|
||||
```
|
||||
|
||||
All three nodes share the same bucket over the cluster's internal storage fabric, so
|
||||
datasets, model weights and pipeline outputs are accessible to any instance without
|
||||
mounting a shared filesystem.
|
||||
|
||||
### 4. Optional: deploy Kubernetes workloads through the EKS-compatible API
|
||||
|
||||
Spinifex also exposes an EKS-compatible control plane — this was not part of the Cisco
|
||||
exercise documented here, which used EC2, EBS and S3 directly. For workloads where
|
||||
automatic pod rescheduling matters, Spinifex supports the same `kubectl` workflows as
|
||||
a standard EKS cluster:
|
||||
|
||||
```bash
|
||||
kubectl get nodes
|
||||
kubectl apply -f deployment.yaml
|
||||
kubectl get pods -n default
|
||||
```
|
||||
|
||||
Kubernetes pod rescheduling is the platform's primary answer to node failure for
|
||||
scheduled workloads. The companion [vision pipeline](../cisco-ucs-vision-pipeline/README.md)
|
||||
and [vLLM serving](../cisco-ucs-llm-serving/README.md) architectures use raw EC2 instances;
|
||||
EKS is the pattern to reach for when automatic rescheduling matters over manual placement.
|
||||
|
||||
### 5. Storage benchmark
|
||||
|
||||
Predastore and Viperblock — Spinifex's S3 and EBS implementations — represent the
|
||||
most substantial engineering work in the platform. Implementing reliable erasure-coded
|
||||
object storage and a distributed block device stack on commodity NVMe, while exposing
|
||||
the AWS API surface that most workloads depend on, is where most of the hard problems
|
||||
live. Both subsystems are under active development, and a production-grade environment
|
||||
like this cluster is where integration-level issues surface before they reach users. Run this check after cluster setup or hardware
|
||||
changes to confirm both services are performing as expected.
|
||||
|
||||
The methodology: one guest per physical node, fio against an ext4 Viperblock data
|
||||
volume (four jobs, iodepth 32, direct I/O, 512 MiB file, 30-second run), three
|
||||
sequential repetitions per guest.
|
||||
|
||||
| Guest | 16K mixed read/write | 16K random read | 128K mixed read/write | 128K random read |
|
||||
|---|---:|---:|---:|---:|
|
||||
| GPU | 75 / 32 MiB/s | 75 MiB/s | 210 / 91 MiB/s | 332 MiB/s |
|
||||
| CPU1 | 74 / 32 MiB/s | 76 MiB/s | 218 / 94 MiB/s | 333 MiB/s |
|
||||
| CPU2 | 69 / 30 MiB/s | 107 MiB/s | 226 / 98 MiB/s | 405 MiB/s |
|
||||
|
||||
The 128K guest random-read numbers (332–405 MiB/s) sit between the [AWS EBS gp3](https://docs.aws.amazon.com/ebs/latest/userguide/general-purpose.html)
|
||||
baseline (125 MiB/s) and its provisioned maximum (1,000 MiB/s) — a meaningful result
|
||||
given that gp3 baseline is what most AWS operators treat as the default floor for
|
||||
general-purpose block storage. The 16K numbers (75–107 MiB/s) land near that baseline.
|
||||
The gap to the raw NVMe host performance (the [KIOXIA CD8P](https://americas.kioxia.com/en-us/business/ssd/data-center-ssd/cd8p-r.html) delivers over 4,000 MiB/s
|
||||
host-side at comparable queue depths) is not architectural — it is the current
|
||||
single-queue virtio-blk attach path and unthreaded NBD backend, both of which have
|
||||
a clear source-level fix. These numbers are the baseline to improve against in future
|
||||
releases, not the ceiling.
|
||||
|
||||
128K mixed I/O improves throughput but carries a high p99 latency envelope (roughly
|
||||
0.47–0.75 s); size queues and write patterns accordingly for latency-sensitive
|
||||
applications.
|
||||
|
||||
Predastore S3 validation used 1 GiB objects with three sequential and three
|
||||
distributed-concurrent repetitions per host:
|
||||
|
||||
| Workload | Write, median | Read, median |
|
||||
|---|---:|---:|
|
||||
| Physical storage fabric (bare metal) | 1 | 23.5 (≈94% of 25GbE line rate) |
|
||||
| Guest↔guest | 1 | 13.4–20.0 |
|
||||
| Guest↔guest | 2–4 | 14.5–19.0 |
|
||||
| Guest↔guest | 16 | 10.3–13.3 |
|
||||
| One host client | 121.9 MiB/s | 213.2 MiB/s |
|
||||
| Three-host aggregate | 184.2 MiB/s | 381.7 MiB/s |
|
||||
|
||||
<img src="../../../.github/assets/images/cisco-ucs-platform-benchmark/network-guest-vs-fabric.png" alt="Guest-to-guest overlay throughput versus the physical storage fabric, by concurrent TCP stream count">
|
||||
The single-host sequential read (213 MiB/s) is a credible result for a single-node
|
||||
S3-compatible store at low-to-moderate concurrency with 1 GiB objects — throughput at
|
||||
this scale is typically network- and per-request-latency-bound rather than disk-bound,
|
||||
and 213 MiB/s uses roughly 17% of the 25GbE storage fabric. The three-node distributed read (381.7 MiB/s, 1.79× single-host) confirms
|
||||
Predastore is distributing reads across the cluster. The write scaling ratio (184.2 vs
|
||||
121.9 MiB/s, 1.51×) is narrower than read, so worth tracking as a leading indicator of
|
||||
backend contention as the cluster grows and write load increases.
|
||||
|
||||
Guest-to-guest throughput lands 20–45% below the physical fabric depending on stream
|
||||
count and pairing.
|
||||
All sequential S3 checksum validations passed. Full methodology, raw fio JSON and S3
|
||||
metrics are in the [benchmark repository](https://github.com/tomnewton-mulga/CISCO-refarch).
|
||||
|
||||
The gap is primarily single-queue virtio-net — `ethtool -l` on all three guest instances
|
||||
reports `Combined: 1` with no guest-configurable multiqueue support, concentrating all
|
||||
network work on a single core regardless of vCPU count. This explains why 16 concurrent
|
||||
streams perform *worse* than 1: they compete for the same queue rather than
|
||||
parallelizing. Geneve encapsulation reduces tenant MTU to 1408 (TCP MSS 1356), adding a
|
||||
smaller compounding overhead on top.
|
||||
## What this architecture unlocks
|
||||
|
||||
### 5. Predastore (S3) throughput
|
||||
- **Cloud-to-edge placement:** run the same EC2, volume, S3 and Kubernetes patterns
|
||||
at the site where latency, privacy, bandwidth or sovereignty requires it.
|
||||
- **Familiar operations:** keep AWS CLI profiles, Terraform/OpenTofu modules, SDKs,
|
||||
CI/CD pipelines and deployment tooling rather than introducing a separate
|
||||
edge-only platform.
|
||||
- **Local AI with shared data:** place GPU and AMX-capable CPU inference next to the
|
||||
data source while keeping datasets and artifacts available to all instances through S3.
|
||||
- **A resilient foundation:** distribute control and storage services across three
|
||||
physical Cisco nodes instead of concentrating the site on one server. Kubernetes
|
||||
rescheduling and Raft-based storage both tolerate a single node loss without
|
||||
operator intervention.
|
||||
- **A path back to cloud:** the same APIs at the edge and on AWS make workload
|
||||
movement, burst strategies and consistent application packaging straightforward.
|
||||
|
||||
Three simultaneous clients per side, one per physical node, over HTTPS to the cluster's
|
||||
S3 endpoint:
|
||||
|
||||
| Direction | Guest-aggregate | Host-aggregate |
|
||||
|---|---:|---:|
|
||||
| Read | 175.1 MiB/s | 299.0 MiB/s |
|
||||
| Write | 293.1 MiB/s | 124.1 MiB/s |
|
||||
|
||||
Reads run 1.71x faster on the host aggregate than guest. Write reverses — guests
|
||||
outperform the host (293.1 vs 124.1 MiB/s).
|
||||
|
||||
### 6. Worker concurrency per instance — how many processes can each node sustain
|
||||
|
||||
With one EC2 instance per physical node (3 instances fixed), we
|
||||
scale the number of concurrent worker processes within each instance from 1 to N, until
|
||||
p95 request latency exceeds 2x the single-worker baseline or aggregate throughput stops
|
||||
improving. Load generator: OpenVINO/YOLO11m inference against Predastore (CPU workers) and
|
||||
TensorRT/YOLO11m (GPU worker).
|
||||
|
||||
| Workers/instance | Total workers | Aggregate img/s | cpu1 p95 (ms) | cpu2 p95 (ms) | gpu p95 (ms) |
|
||||
|---:|---:|---:|---:|---:|---:|
|
||||
| 1 | 3 | 49.0 | 90.9 | 88.1 | 51.0 |
|
||||
| 2 | 6 | 82.6 | 140.9 | 141.7 | 49.8 |
|
||||
| 3 | 9 | 103.3 | 239.0 | 232.2 | 53.0 |
|
||||
| 4 | 12 | 121.1 | 319.3 | 318.4 | 53.0 |
|
||||
| 5 | 15 | 133.1 | 402.1 | 413.6 | 57.4 |
|
||||
| 6 | 18 | 142.5 | 490.4 | 493.6 | 62.2 |
|
||||
|
||||
|
||||
<img src="../../../.github/assets/images/cisco-ucs-platform-benchmark/consolidation-throughput.png" alt="Aggregate throughput versus concurrent workers per instance, by worker type and cluster total">
|
||||
|
||||
<img src="../../../.github/assets/images/cisco-ucs-platform-benchmark/consolidation-p95-latency.png" alt="p95 request latency versus concurrent workers per instance, by worker type">
|
||||
|
||||
Both CPU workers track each other closely across the entire sweep (e.g. 18.05 vs.
|
||||
18.25 img/s at N=2, 490.4 vs. 493.6 ms p95 at N=6), consistent with identical hardware
|
||||
on identical nodes.
|
||||
|
||||
**CPU workers' aggregate throughput peaks at 2 workers/instance (~18.1 img/s each) and
|
||||
declines steadily from there** (~16.6 at N=4, ~14.8 at N=6), while p95 latency climbs
|
||||
almost linearly with N (91–88 ms → ~490 ms, a ~5.4–5.6x increase over the sweep) — **the
|
||||
2x-baseline SLA is breached at 3 workers/instance** (239.0 ms cpu1 / 232.2 ms cpu2 vs. a
|
||||
181.8 ms / 176.2 ms threshold respectively) and never recovers. Both signals agree: **the
|
||||
practical ceiling for this CPU-bound workload is 2 workers/instance (6/cluster)** if
|
||||
throughput is the priority, or 2 if latency matters at all — by 3 it's already costing
|
||||
more than it gains.
|
||||
|
||||
**The GPU worker shows no strain through the full sweep** — p95 stays essentially flat
|
||||
(51.0–62.2 ms, well under its own 102.0 ms breach threshold) and aggregate throughput
|
||||
keeps climbing almost linearly through 6 workers/instance (22.85 → 112.82 img/s, a ~4.9x
|
||||
increase for 6x the load) — consistent with the L4 sitting at only 22–35% utilisation under a single worker. The sweep was capped at 6
|
||||
workers/instance by design; the GPU's actual ceiling is higher and wasn't reached here.
|
||||
|
||||
**Predastore itself is not the limiting subsystem for this result.** mulga-01's
|
||||
Predastore CPU% is already elevated (85.4%) at N=1, briefly exceeds 100% at N=2
|
||||
(119.0%, multi-threaded process accounting), and settles at 93.2% by N=6 — it fluctuates
|
||||
around a high baseline rather than climbing *with* N the way the observed CPU-worker
|
||||
degradation does, and the other two nodes' Predastore CPU% (27–50%) never comes close.
|
||||
|
||||
The cause is CPU-compute contention rather than storage or network: inference latency
|
||||
grows ~11x across the sweep while S3 read/write latency grows only ~24–40%, and each
|
||||
OpenVINO worker consumes roughly 3.5 cores unpinned — oversubscribing the 8-vCPU guest
|
||||
by N=3, which lines up exactly with the observed SLA breach.
|
||||
|
||||
### 7. Conclusion
|
||||
|
||||
On matched-width CPU work, guests cost virtually nothing versus bare metal (~0.5%), and
|
||||
Intel AMX is confirmed executing at the kernel-selection level, not just present in
|
||||
CPUID. Network and disk both show real overhead versus bare metal — guest network at 20–45% below the physical fabric, guest disk far more so,
|
||||
currently the platform's clearest area for near-term improvement. The practical ceiling
|
||||
for CPU-bound concurrent work is 2 worker processes per 8-vCPU instance before latency
|
||||
degrades — confirmed as compute contention rather than a storage or network ceiling —
|
||||
while the GPU instance shows no strain through the full sweep, consistent with the L4
|
||||
sitting at 22–35% utilisation under a single worker.
|
||||
|
||||
All benchmark instances were provisioned through Spinifex's EC2-compatible endpoint using standard `aws ec2` CLI calls — the same instance types, placement groups, and security groups that work on AWS work here unchanged. Teams already operating AWS infrastructure can point their existing tooling at a Spinifex node with a single profile swap.
|
||||
The raw measurements are evidence that the platform is functioning as designed; the
|
||||
larger outcome is a portable, resilient operating model for workloads that need cloud
|
||||
interfaces outside a public-cloud region.
|
||||
|
||||
@@ -14,8 +14,8 @@ tags:
|
||||
- predastore
|
||||
- terraform
|
||||
resources:
|
||||
- title: "cisco-ucs-vision-pipeline (Terraform workbook, scripts)"
|
||||
url: "https://github.com/mulgadc/cisco-ucs-vision-pipeline"
|
||||
- title: "Benchmark and workload materials (CISCO-refarch)"
|
||||
url: "https://github.com/tomnewton-mulga/CISCO-refarch"
|
||||
- title: "Multi-Node Install"
|
||||
url: "/docs/install-multi-node"
|
||||
- title: "VPC Networking"
|
||||
@@ -24,7 +24,7 @@ resources:
|
||||
url: "/docs/launching-instances"
|
||||
- title: "GPU Passthrough"
|
||||
url: "/docs/gpu-passthrough"
|
||||
- title: "Platform benchmark (companion document)"
|
||||
- title: "Cisco UCS: AWS-compatible cloud at the edge (companion document)"
|
||||
url: "../cisco-ucs-platform-benchmark/README.md"
|
||||
- title: "Ultralytics YOLO"
|
||||
url: "https://docs.ultralytics.com"
|
||||
@@ -48,7 +48,7 @@ cluster running Spinifex: real-time object detection plus a vision-language mode
|
||||
producing plain-English scene descriptions, both streaming from the same shared Predastore
|
||||
(S3-compatible) bucket as two entirely independent EC2-compatible instances.
|
||||
|
||||
**Companion documents:** [Spinifex Platform Benchmark on Cisco UCS](../cisco-ucs-platform-benchmark/README.md) · [vLLM Serving on Cisco UCS: Intel AMX vs NVIDIA L4](../cisco-ucs-llm-serving/README.md)
|
||||
**Companion architectures:** [Cisco UCS: AWS-compatible cloud at the edge](../cisco-ucs-platform-benchmark/README.md) · [vLLM Serving on Cisco UCS: Intel AMX vs NVIDIA L4](../cisco-ucs-llm-serving/README.md)
|
||||
|
||||
### Platform
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user