From c4b380b2edd8ec61fd47a38da0f0cd5124872d8b Mon Sep 17 00:00:00 2001 From: Tom Newton Date: Thu, 17 Sep 2026 13:27:58 +1000 Subject: [PATCH] chore: update cisco docs --- .../guest-storage-throughput.svg | 44 ++++++++++++++ .../network-goodput-vs-streams.svg | 35 +++++++++++ .../predastore-s3-read-by-client.svg | 39 ++++++++++++ docs/hardware/cisco/llm-serving/README.md | 2 - .../cisco/platform-benchmark/README.md | 60 ++++++++++--------- docs/hardware/cisco/vision-pipeline/README.md | 2 - 6 files changed, 149 insertions(+), 33 deletions(-) create mode 100644 .github/assets/images/cisco-ucs-platform-benchmark/guest-storage-throughput.svg create mode 100644 .github/assets/images/cisco-ucs-platform-benchmark/network-goodput-vs-streams.svg create mode 100644 .github/assets/images/cisco-ucs-platform-benchmark/predastore-s3-read-by-client.svg diff --git a/.github/assets/images/cisco-ucs-platform-benchmark/guest-storage-throughput.svg b/.github/assets/images/cisco-ucs-platform-benchmark/guest-storage-throughput.svg new file mode 100644 index 000000000..74b8cc08b --- /dev/null +++ b/.github/assets/images/cisco-ucs-platform-benchmark/guest-storage-throughput.svg @@ -0,0 +1,44 @@ + + +Guest block storage throughput (Viperblock) +fio, 4 jobs, iodepth 32, direct I/O, per-guest sequential runs — Spinifex v1.20.0 + +0 + +100 + +200 + +300 + +400 +MiB/s + +376 + +232 + +107 +GPU + +371 + +226 + +80 +CPU1 + +372 + +228 + +80 +CPU2 + + +128K random read + +128K mixed read + +16K random read + \ No newline at end of file diff --git a/.github/assets/images/cisco-ucs-platform-benchmark/network-goodput-vs-streams.svg b/.github/assets/images/cisco-ucs-platform-benchmark/network-goodput-vs-streams.svg new file mode 100644 index 000000000..d81fa2624 --- /dev/null +++ b/.github/assets/images/cisco-ucs-platform-benchmark/network-goodput-vs-streams.svg @@ -0,0 +1,35 @@ + + +Network throughput: overlay vs physical fabric +iperf3 TCP receiver goodput, median across node pairs — Spinifex v1.20.0 + +20 + +25 +Gbit/s +1 +2 +4 +8 +16 + + + + + + +23.48 + + + + + + +22.43 + +parallel TCP streams + +Host physical fabric (25GbE) + +Guest overlay (OVN) + \ No newline at end of file diff --git a/.github/assets/images/cisco-ucs-platform-benchmark/predastore-s3-read-by-client.svg b/.github/assets/images/cisco-ucs-platform-benchmark/predastore-s3-read-by-client.svg new file mode 100644 index 000000000..e0bbefe65 --- /dev/null +++ b/.github/assets/images/cisco-ucs-platform-benchmark/predastore-s3-read-by-client.svg @@ -0,0 +1,39 @@ + + +Predastore S3 read throughput by client +1 GiB objects, same presigned-curl client on every path — Spinifex v1.20.0 + +0 + +200 + +400 + +600 +MiB/s + +340 + +462 +Host + +399 + +391 +GPU guest + +525 + +350 +CPU1 guest + +517 + +351 +CPU2 guest + + +Single-stream GET + +Chunked/concurrent GET + \ No newline at end of file diff --git a/docs/hardware/cisco/llm-serving/README.md b/docs/hardware/cisco/llm-serving/README.md index 257b013f5..61bcbb33f 100644 --- a/docs/hardware/cisco/llm-serving/README.md +++ b/docs/hardware/cisco/llm-serving/README.md @@ -22,8 +22,6 @@ resources: url: "/hardware/cisco/platform-benchmark" - title: "Vision pipeline (companion document)" url: "/hardware/cisco/vision-pipeline" - - title: "Benchmark and workload materials (CISCO-refarch)" - url: "https://github.com/tomnewton-mulga/CISCO-refarch" - title: "vLLM" url: "https://docs.vllm.ai" - title: "Qwen2.5-7B-Instruct" diff --git a/docs/hardware/cisco/platform-benchmark/README.md b/docs/hardware/cisco/platform-benchmark/README.md index e8c0ced38..47329c776 100644 --- a/docs/hardware/cisco/platform-benchmark/README.md +++ b/docs/hardware/cisco/platform-benchmark/README.md @@ -22,8 +22,6 @@ resources: url: "/docs/launching-instances" - title: "GPU Passthrough" url: "/docs/gpu-passthrough" - - title: "cisco-ucs-platform-benchmark (scripts and raw results)" - url: "https://github.com/tomnewton-mulga/CISCO-refarch" sections: - overview - prerequisites @@ -239,42 +237,46 @@ sequential repetitions per guest. | Guest | 16K mixed read/write | 16K random read | 128K mixed read/write | 128K random read | |---|---:|---:|---:|---:| -| GPU | 75 / 32 MiB/s | 75 MiB/s | 210 / 91 MiB/s | 332 MiB/s | -| CPU1 | 74 / 32 MiB/s | 76 MiB/s | 218 / 94 MiB/s | 333 MiB/s | -| CPU2 | 69 / 30 MiB/s | 107 MiB/s | 226 / 98 MiB/s | 405 MiB/s | +| GPU | 107 / 46 MiB/s | 107 MiB/s | 232 / 101 MiB/s | 376 MiB/s | +| CPU1 | 80 / 35 MiB/s | 80 MiB/s | 226 / 98 MiB/s | 371 MiB/s | +| CPU2 | 79 / 34 MiB/s | 80 MiB/s | 228 / 98 MiB/s | 372 MiB/s | -The 128K guest random-read numbers (332–405 MiB/s) sit between the [AWS EBS gp3](https://docs.aws.amazon.com/ebs/latest/userguide/general-purpose.html) -baseline (125 MiB/s) and its provisioned maximum (1,000 MiB/s) — a meaningful result -given that gp3 baseline is what most AWS operators treat as the default floor for -general-purpose block storage. The 16K numbers (75–107 MiB/s) land near that baseline. -The gap to the raw NVMe host performance (the [KIOXIA CD8P](https://americas.kioxia.com/en-us/business/ssd/data-center-ssd/cd8p-r.html) delivers over 4,000 MiB/s -host-side at comparable queue depths) is not architectural — it is the current -single-queue virtio-blk attach path and unthreaded NBD backend, both of which have -a clear source-level fix. These numbers are the baseline to improve against in future -releases, not the ceiling. +Guest block storage throughput by block size and access pattern + +The 128K guest random-read numbers (371–376 MiB/s) sit between the [AWS EBS gp3](https://docs.aws.amazon.com/ebs/latest/userguide/general-purpose.html) +baseline (125 MiB/s) and its provisioned maximum (1,000 MiB/s). The 16K numbers (79–107 MiB/s) land near that baseline. 128K mixed I/O improves throughput but carries a high p99 latency envelope (roughly -0.47–0.75 s); size queues and write patterns accordingly for latency-sensitive +0.45–0.80 s); size queues and write patterns accordingly for latency-sensitive applications. -Predastore S3 validation used 1 GiB objects with three sequential and three -distributed-concurrent repetitions per host: +Predastore S3 validation used 1 GiB objects. The AWS CLI splits any download above 8 MiB into concurrent ranged GETs while a plain client issues a single streaming GET, so the sweep measures both patterns with the same client (presigned `curl`) on the host and on each guest — a single-stream PUT/GET, and a chunked/concurrent Range GET (8 MiB parts, 10 in flight) that mirrors the AWS CLI's default download path — with three repetitions each: -| Workload | Write, median | Read, median | +| Client | Single-stream read | Chunked/concurrent read | |---|---:|---:| -| One host client | 121.9 MiB/s | 213.2 MiB/s | -| Three-host aggregate | 184.2 MiB/s | 381.7 MiB/s | +| Host | 340 MiB/s | 462 MiB/s | +| GPU guest | 399 MiB/s | 391 MiB/s | +| CPU1 guest | 525 MiB/s | 350 MiB/s | +| CPU2 guest | 517 MiB/s | 351 MiB/s | -The single-host sequential read (213 MiB/s) is a credible result for a single-node -S3-compatible store at low-to-moderate concurrency with 1 GiB objects — throughput at -this scale is typically network- and per-request-latency-bound rather than disk-bound, -and 213 MiB/s uses roughly 17% of the 25GbE storage fabric. The three-node distributed read (381.7 MiB/s, 1.79× single-host) confirms -Predastore is distributing reads across the cluster. The write scaling ratio (184.2 vs -121.9 MiB/s, 1.51×) is narrower than read, so worth tracking as a leading indicator of -backend contention as the cluster grows and write load increases. +Predastore S3 read throughput by client, single-stream versus chunked/concurrent GET -All sequential S3 checksum validations passed. Full methodology, raw fio JSON and S3 -metrics are in the [benchmark repository](https://github.com/tomnewton-mulga/CISCO-refarch). +Single-stream writes land at 282–304 MiB/s across the host and all three guests. Under three simultaneous clients — one per physical node — the cluster sustains 640 MiB/s aggregate read and 538 MiB/s aggregate write from the host, and 665 / 515 MiB/s from the guests. + +With the client held constant, single-stream host and guest reads fall in the same 340–525 MiB/s band, and aggregate read reaches 640–665 MiB/s across three concurrent clients. The chunked/concurrent Range-GET results vary by individual client rather than tracking host-versus-guest; that variation has no confirmed explanation yet and remains under investigation. All sequential S3 checksum validations passed. + +### 6. Network benchmark + +Network throughput was measured with iperf3 over TCP, with a 10-second run after a 3-second warmup, five repetitions at 1, 2, 4, 8 and 16 parallel streams, on two paths: host-to-host across the dedicated 25GbE storage fabric (VLAN 1336), and guest-to-guest across the OVN overlay that connects instances. + +| Path | Single stream | 2–16 streams | +|---|---:|---:| +| Host physical fabric (25GbE) | 23.4 Gbit/s | 23.5 Gbit/s | +| Guest overlay (OVN) | 22.0 Gbit/s | 22.5 Gbit/s | + +Network TCP goodput versus parallel stream count, host physical fabric versus guest overlay + +TCP goodput on a 25GbE link settles near 23.5 Gbit/s once framing and protocol overhead are removed, so the host path runs at effectively line rate with zero retransmissions. The guest overlay sustains 22.0 Gbit/s on a single stream, rising to 22.5 Gbit/s at two streams and holding flat through sixteen — 95–96% of the physical fabric across the full range. ## What this architecture unlocks diff --git a/docs/hardware/cisco/vision-pipeline/README.md b/docs/hardware/cisco/vision-pipeline/README.md index edb5a92b8..75424c5fe 100644 --- a/docs/hardware/cisco/vision-pipeline/README.md +++ b/docs/hardware/cisco/vision-pipeline/README.md @@ -14,8 +14,6 @@ tags: - predastore - terraform resources: - - title: "Benchmark and workload materials (CISCO-refarch)" - url: "https://github.com/tomnewton-mulga/CISCO-refarch" - title: "Multi-Node Install" url: "/docs/install-multi-node" - title: "VPC Networking"