gsadmin/spinifex
1
0
Fork 0
mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-15 09:01:31 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore: update cisco docs

Browse Source
This commit is contained in:
Tom Newton
2026-09-17 13:27:58 +10:00
parent ef9bfb4c1d
commit c4b380b2ed
6 changed files with 149 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/assets/images/cisco-ucs-platform-benchmark/guest-storage-throughput.svg
+44
View File
@@ -0,0 +1,44 @@
<svg xmlns="http://www.w3.org/2000/svg" width="760" height="470" viewBox="0 0 760 470" font-family="-apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif">
<rect width="760" height="470" fill="#ffffff"/>
<text x="74" y="30" font-size="19" font-weight="700" fill="#1f2937">Guest block storage throughput (Viperblock)</text>
<text x="74" y="49" font-size="12.5" fill="#6b7280">fio, 4 jobs, iodepth 32, direct I/O, per-guest sequential runs — Spinifex v1.20.0</text>
<line x1="74" y1="386.0" x2="734" y2="386.0" stroke="#e5e7eb" stroke-width="1"/>
<text x="64" y="390.0" font-size="12" text-anchor="end" fill="#6b7280">0</text>
<line x1="74" y1="305.0" x2="734" y2="305.0" stroke="#e5e7eb" stroke-width="1"/>
<text x="64" y="309.0" font-size="12" text-anchor="end" fill="#6b7280">100</text>
<line x1="74" y1="224.0" x2="734" y2="224.0" stroke="#e5e7eb" stroke-width="1"/>
<text x="64" y="228.0" font-size="12" text-anchor="end" fill="#6b7280">200</text>
<line x1="74" y1="143.0" x2="734" y2="143.0" stroke="#e5e7eb" stroke-width="1"/>
<text x="64" y="147.0" font-size="12" text-anchor="end" fill="#6b7280">300</text>
<line x1="74" y1="62.0" x2="734" y2="62.0" stroke="#e5e7eb" stroke-width="1"/>
<text x="64" y="66.0" font-size="12" text-anchor="end" fill="#6b7280">400</text>
<text x="22" y="224.0" font-size="12.5" fill="#6b7280" transform="rotate(-90 22 224.0)" text-anchor="middle">MiB/s</text>
<rect x="111.4" y="81.4" width="45.4" height="304.6" rx="2.5" fill="#2563eb"/>
<text x="134.1" y="75.4" font-size="11" text-anchor="middle" fill="#1f2937">376</text>
<rect x="159.8" y="198.1" width="45.4" height="187.9" rx="2.5" fill="#0d9488"/>
<text x="182.5" y="192.1" font-size="11" text-anchor="middle" fill="#1f2937">232</text>
<rect x="208.2" y="299.3" width="45.4" height="86.7" rx="2.5" fill="#f59e0b"/>
<text x="230.9" y="293.3" font-size="11" text-anchor="middle" fill="#1f2937">107</text>
<text x="184.0" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">GPU</text>
<rect x="331.4" y="85.5" width="45.4" height="300.5" rx="2.5" fill="#2563eb"/>
<text x="354.1" y="79.5" font-size="11" text-anchor="middle" fill="#1f2937">371</text>
<rect x="379.8" y="202.9" width="45.4" height="183.1" rx="2.5" fill="#0d9488"/>
<text x="402.5" y="196.9" font-size="11" text-anchor="middle" fill="#1f2937">226</text>
<rect x="428.2" y="321.2" width="45.4" height="64.8" rx="2.5" fill="#f59e0b"/>
<text x="450.9" y="315.2" font-size="11" text-anchor="middle" fill="#1f2937">80</text>
<text x="404.0" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">CPU1</text>
<rect x="551.4" y="84.7" width="45.4" height="301.3" rx="2.5" fill="#2563eb"/>
<text x="574.1" y="78.7" font-size="11" text-anchor="middle" fill="#1f2937">372</text>
<rect x="599.8" y="201.3" width="45.4" height="184.7" rx="2.5" fill="#0d9488"/>
<text x="622.5" y="195.3" font-size="11" text-anchor="middle" fill="#1f2937">228</text>
<rect x="648.2" y="321.2" width="45.4" height="64.8" rx="2.5" fill="#f59e0b"/>
<text x="670.9" y="315.2" font-size="11" text-anchor="middle" fill="#1f2937">80</text>
<text x="624.0" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">CPU2</text>
<line x1="74" y1="386" x2="734" y2="386" stroke="#1f2937" stroke-width="1.5"/>
<rect x="74" y="444" width="13" height="13" rx="2.5" fill="#2563eb"/>
<text x="93" y="455" font-size="12.5" fill="#1f2937">128K random read</text>
<rect x="216.8" y="444" width="13" height="13" rx="2.5" fill="#0d9488"/>
<text x="235.8" y="455" font-size="12.5" fill="#1f2937">128K mixed read</text>
<rect x="352.3" y="444" width="13" height="13" rx="2.5" fill="#f59e0b"/>
<text x="371.3" y="455" font-size="12.5" fill="#1f2937">16K random read</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 3.7 KiB

.github/assets/images/cisco-ucs-platform-benchmark/network-goodput-vs-streams.svg
+35
View File
@@ -0,0 +1,35 @@
<svg xmlns="http://www.w3.org/2000/svg" width="760" height="470" viewBox="0 0 760 470" font-family="-apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif">
<rect width="760" height="470" fill="#ffffff"/>
<text x="74" y="30" font-size="19" font-weight="700" fill="#1f2937">Network throughput: overlay vs physical fabric</text>
<text x="74" y="49" font-size="12.5" fill="#6b7280">iperf3 TCP receiver goodput, median across node pairs — Spinifex v1.20.0</text>
<line x1="74" y1="386.0" x2="734" y2="386.0" stroke="#e5e7eb" stroke-width="1"/>
<text x="64" y="390.0" font-size="12" text-anchor="end" fill="#6b7280">20</text>
<line x1="74" y1="62.0" x2="734" y2="62.0" stroke="#e5e7eb" stroke-width="1"/>
<text x="64" y="66.0" font-size="12" text-anchor="end" fill="#6b7280">25</text>
<text x="22" y="224.0" font-size="12.5" fill="#6b7280" transform="rotate(-90 22 224.0)" text-anchor="middle">Gbit/s</text>
<text x="140.0" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">1</text>
<text x="272.0" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">2</text>
<text x="404.0" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">4</text>
<text x="536.0" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">8</text>
<text x="668.0" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">16</text>
<path d="M140.0,164.4 L272.0,160.5 L404.0,160.5 L536.0,160.5 L668.0,160.5" fill="none" stroke="#2563eb" stroke-width="3"/>
<circle cx="140.0" cy="164.4" r="4.5" fill="#fff" stroke="#2563eb" stroke-width="2.5"/>
<circle cx="272.0" cy="160.5" r="4.5" fill="#fff" stroke="#2563eb" stroke-width="2.5"/>
<circle cx="404.0" cy="160.5" r="4.5" fill="#fff" stroke="#2563eb" stroke-width="2.5"/>
<circle cx="536.0" cy="160.5" r="4.5" fill="#fff" stroke="#2563eb" stroke-width="2.5"/>
<circle cx="668.0" cy="160.5" r="4.5" fill="#fff" stroke="#2563eb" stroke-width="2.5"/>
<text x="664.0" y="149.5" font-size="11.5" text-anchor="end" font-weight="600" fill="#2563eb">23.48</text>
<path d="M140.0,255.1 L272.0,222.1 L404.0,222.1 L536.0,222.1 L668.0,228.5" fill="none" stroke="#0d9488" stroke-width="3"/>
<circle cx="140.0" cy="255.1" r="4.5" fill="#fff" stroke="#0d9488" stroke-width="2.5"/>
<circle cx="272.0" cy="222.1" r="4.5" fill="#fff" stroke="#0d9488" stroke-width="2.5"/>
<circle cx="404.0" cy="222.1" r="4.5" fill="#fff" stroke="#0d9488" stroke-width="2.5"/>
<circle cx="536.0" cy="222.1" r="4.5" fill="#fff" stroke="#0d9488" stroke-width="2.5"/>
<circle cx="668.0" cy="228.5" r="4.5" fill="#fff" stroke="#0d9488" stroke-width="2.5"/>
<text x="664.0" y="217.5" font-size="11.5" text-anchor="end" font-weight="600" fill="#0d9488">22.43</text>
<line x1="74" y1="386" x2="734" y2="386" stroke="#1f2937" stroke-width="1.5"/>
<text x="404.0" y="430.0" font-size="12.5" text-anchor="middle" fill="#6b7280">parallel TCP streams</text>
<rect x="74" y="444" width="13" height="13" rx="2.5" fill="#2563eb"/>
<text x="93" y="455" font-size="12.5" fill="#1f2937">Host physical fabric (25GbE)</text>
<rect x="304.4" y="444" width="13" height="13" rx="2.5" fill="#0d9488"/>
<text x="323.4" y="455" font-size="12.5" fill="#1f2937">Guest overlay (OVN)</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 3.2 KiB

.github/assets/images/cisco-ucs-platform-benchmark/predastore-s3-read-by-client.svg
+39
View File
@@ -0,0 +1,39 @@
<svg xmlns="http://www.w3.org/2000/svg" width="760" height="470" viewBox="0 0 760 470" font-family="-apple-system, Segoe UI, Roboto, Helvetica, Arial, sans-serif">
<rect width="760" height="470" fill="#ffffff"/>
<text x="74" y="30" font-size="19" font-weight="700" fill="#1f2937">Predastore S3 read throughput by client</text>
<text x="74" y="49" font-size="12.5" fill="#6b7280">1 GiB objects, same presigned-curl client on every path — Spinifex v1.20.0</text>
<line x1="74" y1="386.0" x2="734" y2="386.0" stroke="#e5e7eb" stroke-width="1"/>
<text x="64" y="390.0" font-size="12" text-anchor="end" fill="#6b7280">0</text>
<line x1="74" y1="278.0" x2="734" y2="278.0" stroke="#e5e7eb" stroke-width="1"/>
<text x="64" y="282.0" font-size="12" text-anchor="end" fill="#6b7280">200</text>
<line x1="74" y1="170.0" x2="734" y2="170.0" stroke="#e5e7eb" stroke-width="1"/>
<text x="64" y="174.0" font-size="12" text-anchor="end" fill="#6b7280">400</text>
<line x1="74" y1="62.0" x2="734" y2="62.0" stroke="#e5e7eb" stroke-width="1"/>
<text x="64" y="66.0" font-size="12" text-anchor="end" fill="#6b7280">600</text>
<text x="22" y="224.0" font-size="12.5" fill="#6b7280" transform="rotate(-90 22 224.0)" text-anchor="middle">MiB/s</text>
<rect x="102.0" y="202.4" width="51.5" height="183.6" rx="2.5" fill="#2563eb"/>
<text x="127.8" y="196.4" font-size="11" text-anchor="middle" fill="#1f2937">340</text>
<rect x="156.5" y="136.5" width="51.5" height="249.5" rx="2.5" fill="#f59e0b"/>
<text x="182.2" y="130.5" font-size="11" text-anchor="middle" fill="#1f2937">462</text>
<text x="156.5" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">Host</text>
<rect x="267.1" y="170.5" width="51.5" height="215.5" rx="2.5" fill="#2563eb"/>
<text x="292.8" y="164.5" font-size="11" text-anchor="middle" fill="#1f2937">399</text>
<rect x="321.5" y="174.9" width="51.5" height="211.1" rx="2.5" fill="#f59e0b"/>
<text x="347.2" y="168.9" font-size="11" text-anchor="middle" fill="#1f2937">391</text>
<text x="321.5" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">GPU guest</text>
<rect x="432.1" y="102.5" width="51.5" height="283.5" rx="2.5" fill="#2563eb"/>
<text x="457.8" y="96.5" font-size="11" text-anchor="middle" fill="#1f2937">525</text>
<rect x="486.5" y="197.0" width="51.5" height="189.0" rx="2.5" fill="#f59e0b"/>
<text x="512.2" y="191.0" font-size="11" text-anchor="middle" fill="#1f2937">350</text>
<text x="486.5" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">CPU1 guest</text>
<rect x="597.0" y="106.8" width="51.5" height="279.2" rx="2.5" fill="#2563eb"/>
<text x="622.8" y="100.8" font-size="11" text-anchor="middle" fill="#1f2937">517</text>
<rect x="651.5" y="196.5" width="51.5" height="189.5" rx="2.5" fill="#f59e0b"/>
<text x="677.2" y="190.5" font-size="11" text-anchor="middle" fill="#1f2937">351</text>
<text x="651.5" y="408.0" font-size="13" text-anchor="middle" font-weight="600" fill="#1f2937">CPU2 guest</text>
<line x1="74" y1="386" x2="734" y2="386" stroke="#1f2937" stroke-width="1.5"/>
<rect x="74" y="444" width="13" height="13" rx="2.5" fill="#2563eb"/>
<text x="93" y="455" font-size="12.5" fill="#1f2937">Single-stream GET</text>
<rect x="224.1" y="444" width="13" height="13" rx="2.5" fill="#f59e0b"/>
<text x="243.1" y="455" font-size="12.5" fill="#1f2937">Chunked/concurrent GET</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 3.3 KiB

docs/hardware/cisco/llm-serving/README.md
-2
View File
@@ -22,8 +22,6 @@ resources:
url: "/hardware/cisco/platform-benchmark"
- title: "Vision pipeline (companion document)"
url: "/hardware/cisco/vision-pipeline"
- title: "Benchmark and workload materials (CISCO-refarch)"
url: "https://github.com/tomnewton-mulga/CISCO-refarch"
- title: "vLLM"
url: "https://docs.vllm.ai"
- title: "Qwen2.5-7B-Instruct"
docs/hardware/cisco/platform-benchmark/README.md
+31 -29
View File
@@ -22,8 +22,6 @@ resources:
url: "/docs/launching-instances"
- title: "GPU Passthrough"
url: "/docs/gpu-passthrough"
- title: "cisco-ucs-platform-benchmark (scripts and raw results)"
url: "https://github.com/tomnewton-mulga/CISCO-refarch"
sections:
- overview
- prerequisites
@@ -239,42 +237,46 @@ sequential repetitions per guest.
| Guest | 16K mixed read/write | 16K random read | 128K mixed read/write | 128K random read |
|---|---:|---:|---:|---:|
| GPU | 75 / 32 MiB/s | 75 MiB/s | 210 / 91 MiB/s | 332 MiB/s |
| CPU1 | 74 / 32 MiB/s | 76 MiB/s | 218 / 94 MiB/s | 333 MiB/s |
| CPU2 | 69 / 30 MiB/s | 107 MiB/s | 226 / 98 MiB/s | 405 MiB/s |
| GPU | 107 / 46 MiB/s | 107 MiB/s | 232 / 101 MiB/s | 376 MiB/s |
| CPU1 | 80 / 35 MiB/s | 80 MiB/s | 226 / 98 MiB/s | 371 MiB/s |
| CPU2 | 79 / 34 MiB/s | 80 MiB/s | 228 / 98 MiB/s | 372 MiB/s |
The 128K guest random-read numbers (332–405 MiB/s) sit between the [AWS EBS gp3](https://docs.aws.amazon.com/ebs/latest/userguide/general-purpose.html)
baseline (125 MiB/s) and its provisioned maximum (1,000 MiB/s) — a meaningful result
given that gp3 baseline is what most AWS operators treat as the default floor for
general-purpose block storage. The 16K numbers (75–107 MiB/s) land near that baseline.
The gap to the raw NVMe host performance (the [KIOXIA CD8P](https://americas.kioxia.com/en-us/business/ssd/data-center-ssd/cd8p-r.html) delivers over 4,000 MiB/s
host-side at comparable queue depths) is not architectural — it is the current
single-queue virtio-blk attach path and unthreaded NBD backend, both of which have
a clear source-level fix. These numbers are the baseline to improve against in future
releases, not the ceiling.
<img src="../../../../.github/assets/images/cisco-ucs-platform-benchmark/guest-storage-throughput.svg" alt="Guest block storage throughput by block size and access pattern">
The 128K guest random-read numbers (371–376 MiB/s) sit between the [AWS EBS gp3](https://docs.aws.amazon.com/ebs/latest/userguide/general-purpose.html)
baseline (125 MiB/s) and its provisioned maximum (1,000 MiB/s). The 16K numbers (79–107 MiB/s) land near that baseline.
128K mixed I/O improves throughput but carries a high p99 latency envelope (roughly
0.47–0.75 s); size queues and write patterns accordingly for latency-sensitive
0.45–0.80 s); size queues and write patterns accordingly for latency-sensitive
applications.
Predastore S3 validation used 1 GiB objects with three sequential and three
distributed-concurrent repetitions per host:
Predastore S3 validation used 1 GiB objects. The AWS CLI splits any download above 8 MiB into concurrent ranged GETs while a plain client issues a single streaming GET, so the sweep measures both patterns with the same client (presigned `curl`) on the host and on each guest — a single-stream PUT/GET, and a chunked/concurrent Range GET (8 MiB parts, 10 in flight) that mirrors the AWS CLI's default download path — with three repetitions each:
| Workload | Write, median | Read, median |
| Client | Single-stream read | Chunked/concurrent read |
|---|---:|---:|
| One host client | 121.9 MiB/s | 213.2 MiB/s |
| Three-host aggregate | 184.2 MiB/s | 381.7 MiB/s |
| Host | 340 MiB/s | 462 MiB/s |
| GPU guest | 399 MiB/s | 391 MiB/s |
| CPU1 guest | 525 MiB/s | 350 MiB/s |
| CPU2 guest | 517 MiB/s | 351 MiB/s |
The single-host sequential read (213 MiB/s) is a credible result for a single-node
S3-compatible store at low-to-moderate concurrency with 1 GiB objects — throughput at
this scale is typically network- and per-request-latency-bound rather than disk-bound,
and 213 MiB/s uses roughly 17% of the 25GbE storage fabric. The three-node distributed read (381.7 MiB/s, 1.79× single-host) confirms
Predastore is distributing reads across the cluster. The write scaling ratio (184.2 vs
121.9 MiB/s, 1.51×) is narrower than read, so worth tracking as a leading indicator of
backend contention as the cluster grows and write load increases.
<img src="../../../../.github/assets/images/cisco-ucs-platform-benchmark/predastore-s3-read-by-client.svg" alt="Predastore S3 read throughput by client, single-stream versus chunked/concurrent GET">
All sequential S3 checksum validations passed. Full methodology, raw fio JSON and S3
metrics are in the [benchmark repository](https://github.com/tomnewton-mulga/CISCO-refarch).
Single-stream writes land at 282–304 MiB/s across the host and all three guests. Under three simultaneous clients — one per physical node — the cluster sustains 640 MiB/s aggregate read and 538 MiB/s aggregate write from the host, and 665 / 515 MiB/s from the guests.
With the client held constant, single-stream host and guest reads fall in the same 340–525 MiB/s band, and aggregate read reaches 640–665 MiB/s across three concurrent clients. The chunked/concurrent Range-GET results vary by individual client rather than tracking host-versus-guest; that variation has no confirmed explanation yet and remains under investigation. All sequential S3 checksum validations passed.
### 6. Network benchmark
Network throughput was measured with iperf3 over TCP, with a 10-second run after a 3-second warmup, five repetitions at 1, 2, 4, 8 and 16 parallel streams, on two paths: host-to-host across the dedicated 25GbE storage fabric (VLAN 1336), and guest-to-guest across the OVN overlay that connects instances.
| Path | Single stream | 2–16 streams |
|---|---:|---:|
| Host physical fabric (25GbE) | 23.4 Gbit/s | 23.5 Gbit/s |
| Guest overlay (OVN) | 22.0 Gbit/s | 22.5 Gbit/s |
<img src="../../../../.github/assets/images/cisco-ucs-platform-benchmark/network-goodput-vs-streams.svg" alt="Network TCP goodput versus parallel stream count, host physical fabric versus guest overlay">
TCP goodput on a 25GbE link settles near 23.5 Gbit/s once framing and protocol overhead are removed, so the host path runs at effectively line rate with zero retransmissions. The guest overlay sustains 22.0 Gbit/s on a single stream, rising to 22.5 Gbit/s at two streams and holding flat through sixteen — 95–96% of the physical fabric across the full range.
## What this architecture unlocks
docs/hardware/cisco/vision-pipeline/README.md
-2
View File
@@ -14,8 +14,6 @@ tags:
- predastore
- terraform
resources:
- title: "Benchmark and workload materials (CISCO-refarch)"
url: "https://github.com/tomnewton-mulga/CISCO-refarch"
- title: "Multi-Node Install"
url: "/docs/install-multi-node"
- title: "VPC Networking"